iterative.ai

Under the hood DataChain combines power of warehouses with distributed clusters with proper data access patterns to process millions of video, images audio files: ☁️ Never copy data. Store references to files instead. (while still preserving versioning, data loading, efficient processing) ⚡ Use warehouses under the hood (e.g. ClickHouse) to store metadata and perform as many operations inside it (e.g. filters). ⚙️ Distributed compute that runs close to the data to compute Python-based UDFs 🤏 Data access. Pre-fetch, batching, caching, streaming - different workloads require different ways of using data. #unstructured #datachain #dvc #machinelearning #opensource

A quick glimpse from our CEO, Dmitry Petrov, into ETL and data governance aspects of the DataChain and our SaaS for unstructured data processing: ✅ Each dataset is immutable, versioned, and has fingerprints for all data objects to reproduce; ✅ All dependencies are tracked and saved: code, datasets, raw data sources; ✅ ETL can be run automatically or on schedule to produce new versions of the datasets; Interested to learn more? Contact us here https://datachain.ai/ Open source version is available here to try: https://lnkd.in/emFvJD84 #unstructured #dvc #datachain #machinelearning

DataChain got hand-picked on `r/Python` as one of the top 2024 tools in the "AI / ML / Data" category 👌. Thanks folks, we are also super convinced that we need better tools for unstructured / AI data management. It is still a very hard problem and existing platforms don't address all the needs. Meanwhile there is a very strong and growing demand from AI companies, from all the companies that now do RAGs and other apps that tap into unstructured data. We are working hard on DataChain and DVC to make the whole data processing for images, audio, texts, pdfs, etc scalable, faster, and pleasant experience. Stay tuned, more to come! Quote: "Our selection criteria remain focused on innovation, active maintenance, and broad impact potential. ...." #datachain #dvc #unstructured #machinelearning #opensource

Dealing with a lot of unstructured or multimodal (audio, pdfs, images, videos) data is hard. We clearly need new tools for unstructured data: processing, governance, analytics, preparing it for RAGs, etc, etc. This small video by Ivan Shcheklein is a glimpse into how our DataChain SaaS helps with those aspects: - stream audio files from tar or wds archives! - enrich, prepare, version, publish datasets ... 🚀 - bonus! 🤗  is now natively integrated as a storage provider! Colab notebook: https://lnkd.in/g4W4qF4i Jupyter Notebook: https://lnkd.in/gTbj8ZG2 DataChain Repo: https://lnkd.in/emFvJD84 #huggingface #machinelearning #unstructured #dvc #datachain

DataChain hit 2000 stars ⭐ on GitHub a week ago. Thank you for your interest and support 🤗 It was built to address those needs and pain points we saw in the DVC community when people have to deal with millions of files (e.g. images, pdfs, audio, etc). ❓How to "query" them to find similar, deduplicate, based on some insights, etc ❓What if those are tar or WebDataset archives ... 🤯 ➡️ How to apply transformations (e.g. LLMs or any other models) at scale to get insights and do analytics on top of that? 🧑🏻🤝🧑🏻 How to collaborate - share datasets with those insights? Version and reproduce those 💰What about ETLs with granular updates (it's expensive to run GPUs to get embeddings) ... And many, many more questions ... We've just scratched the surface and more features to come, but DataChain (open source and enterprise SaaS) is already saving many many data engineering and ML researchers hours. https://lnkd.in/emFvJD84 https://datachain.ai How do you manage your unstructured data? #unstructured #machinelearning #opensource #dataengineering #dvc #datachain

DataChain hit 2000 stars ⭐ on GitHub a week ago. Thank you for your interest and support 🤗 It was built to address those needs and pain points we saw in the DVC community when people have to deal with millions of files (e.g. images, pdfs, audio, etc). ❓How to "query" them to find similar, deduplicate, based on some insights, etc ❓What if those are tar or WebDataset archives ... 🤯 ➡️ How to apply transformations (e.g. LLMs or any other models) at scale to get insights and do analytics on top of that? 🧑🏻🤝🧑🏻 How to collaborate - share datasets with those insights? Version and reproduce those 💰What about ETLs with granular updates (it's expensive to run GPUs to get embeddings) ... And many, many more questions ... We've just scratched the surface and more features to come, but DataChain (open source and enterprise SaaS) is already saving many many data engineering and ML researchers hours. https://lnkd.in/emFvJD84 https://datachain.ai How do you manage your unstructured data? #unstructured #machinelearning #opensource #dataengineering #dvc #datachain

DataChain hit 2000 stars ⭐ on GitHub a week ago. Thank you for your interest and support 🤗 It was built to address those needs and pain points we saw in the DVC community when people have to deal with millions of files (e.g. images, pdfs, audio, etc). ❓How to "query" them to find similar, deduplicate, based on some insights, etc ❓What if those are tar or WebDataset archives ... 🤯 ➡️ How to apply transformations (e.g. LLMs or any other models) at scale to get insights and do analytics on top of that? 🧑🏻🤝🧑🏻 How to collaborate - share datasets with those insights? Version and reproduce those 💰What about ETLs with granular updates (it's expensive to run GPUs to get embeddings) ... And many, many more questions ... We've just scratched the surface and more features to come, but DataChain (open source and enterprise SaaS) is already saving many many data engineering and ML researchers hours. https://lnkd.in/emFvJD84 https://datachain.ai How do you manage your unstructured data? #unstructured #machinelearning #opensource #dataengineering #dvc #datachain

Mikhail Rozhkov shares insights from his talk at DSC Europe 2024 and DataChain. Read key highlights in his post https://lnkd.in/gSRAVcar

Boom! 💥 DataChain is trending on HN - come, join the discussion 🤗 We have been working really hard to rethink how AI changes data processing space - a lot of cool decisions and tech inside! #datachain #dvc #ai #machinelearning #mlops

🚀 Why JSON Metadata is Your Secret Weapon in Gen AI Development As AI developers, we often focus on model architecture and hyperparameters, but here's a game-changer: proper JSON metadata management for your training files. Here's why it matters: ✅ Structured Organization: Standardize your data labeling and categorization ✅ Smart Training Control: Filter datasets based on quality and attributes ✅ Version Control: Track changes and ensure reproducibility ✅ Performance Boost: Pre-filter datasets efficiently ✅ Quality Assurance: Maintain data integrity and provenance 💡 Pro Tip: Start implementing JSON metadata early in your project. It's much harder to retrofit it later! The following example is a way to select files using JSON metadata with DataChain. Try out Open-source DataChain at the repo in the comments. Who else is using JSON metadata in their Gen AI pipelines? Share your experiences below! 👇 #ArtificialIntelligence #GenerativeAI #DataScience #TechTips