There are 37 repositories under data-processing topic.
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
A light-weight, flexible, and expressive statistical data testing library
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Extract Transform Load for Python 3.5+
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
A list about Apache Kafka
All-in-one text de-duplication
Machine Learning notebooks for refreshing concepts.
Harmonious distributed data analysis in Rust.
Scalable data pre processing and curation toolkit for LLMs
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
Elastic data processing with Apache Pulsar and Apache Flink
🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.