data-processing

There are 51 repositories under data-processing topic.

pathway
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
batch-processing data-analytics data-pipelines data-processing dataflow etl etl-framework iot-analytics kafka machine-learning-algorithms pathway python real-time rust stream-processing streaming time-series-analysis
Language:Python 49343
Bash-Oneliner
onceupon / Bash-Oneliner
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
bash data-processing grep hardware linux linux-administration one-liners oneliner-commands shell shell-oneliner system terminal variables xargs xwindow
10589
miller
johnkerl / miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
command-line command-line-tools csv csv-format data-cleaning data-processing data-reduction data-regression devops devops-tools json json-data miller statistical-analysis statistics streaming-algorithms streaming-data tabular-data tsv unix-toolkit
Language:Go 9516
TomWright / dasel
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
json yaml configuration selector data-structures config parser yaml-processor json-processing devops-tools golang cli toml query update xml go data-processing data-wrangling
Language:Go 7673
DALI
NVIDIA / DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
fast-data-pipeline image-augmentation data-augmentation image-processing data-processing deep-learning machine-learning python neural-network gpu gpu-tensorflow audio-processing pytorch mxnet paddle
Language:C++ 5545
modelscope / data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
data-analysis data-science large-language-models llm data-visualization llms instruction-tuning pre-training multi-modal synthetic-data data data-pipeline data-processing foundation-models
Language:Python 5264
deepseek-ai / smallpond
A lightweight data processing framework built on DuckDB and 3FS.
data-processing duckdb
Language:Python 4835
unionai-oss / pandera
A light-weight, flexible, and expressive statistical data testing library
pandas validation schema dataframes testing pandas-validation pandas-dataframe data-validation data-cleaning data-check testing-tools assertions data-assertions data-verification dataframe-schema hypothesis-testing pandas-validator data-processing
Language:Python 4078
cocoindex
cocoindex-io / cocoindex
Data transformation framework for AI. Ultra performant, with incremental processing. 🌟 Star if you like it!
ai change-data-capture data data-indexing etl indexing pipeline python rag real-time rust semantic-search data-engineering data-infrastructure data-processing help-wanted knowledge-graph llm context-engineering hacktoberfest
Language:Rust 3218
broadway
dashbitco / broadway
Concurrent and multi-stage data ingestion and data processing with Elixir
elixir genstage data-ingestion data-processing concurrent broadway
Language:Elixir 2585
microsoft / DialoGPT
Large-scale pretraining for dialogue
data-processing dialogpt dialogue gpt-2 machine-learning pytorch text-data text-generation transformer
Language:Python 2413
asyml / texar
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
bert casl-project data-processing deep-learning dialog-systems gpt-2 machine-learning machine-translation natural-language-processing python tensorflow texar text-data text-generation xlnet
Language:Python 2390
numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
data-processing hacktoberfest k8s kubernetes map-reduce pipeline stream-processing
Language:Rust 2359
bytewax / bytewax
Python Stream Processing
python stream-processing rust data-engineering data-processing data-science dataflow machine-learning streaming-data
Language:Python 1879
python-bonobo / bonobo
Extract Transform Load for Python 3.5+
data-processing extract-transform-load bonobo python3 automation parallelization
Language:Python 1605
pyper-dev / pyper
Concurrent Python made simple
asyncio concurrency python threading data-pipelines data-processing multiprocessing parallel-computing data data-collection data-engineering
Language:Python 1506
OpenDCAI / DataFlow
Easy Data Preparation with latest LLMs-based Operators and Pipelines.
data data-agent data-cleaning data-pipelines data-processing data-science data-synthesis gradio-interface llms operators quick-data-processing sglang-bankend vllm-backend
Language:Python 1452
GoogleCloudPlatform / data-science-on-gcp
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering
Language:Jupyter Notebook 1399
dolma
allenai / dolma
Data and tools for generating and inspecting OLMo pre-training data.
data-processing large-language-models llm machile-learning nlp
Language:Python 1341
NVIDIA-NeMo / Curator
Scalable data pre processing and curation toolkit for LLMs
data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication
Language:Python 1204
microsoft / GODEL
Large-scale pretrained models for goal-directed dialog
data-processing dialogue dialogue-systems machine-learning text-data text-generation transformers conversational-ai language-grounding grounded-generation dialogpt language-model pretrained-model pytorch transformer
Language:Python 885
GoogleCloudPlatform / DataflowJavaSDK
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
big-data data-analysis data-mining data-processing data-science google-cloud-dataflow
853
benibela / xidel
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
xquery xml html json xpath cli command-line http web rest css-selector wget curl httpie xmlstarlet webscraper webscraping scraper datascraping data-processing
Language:Pascal 818
asyml / texar-pytorch
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
bert casl-project data-processing deep-learning dialog-systems gpt-2 machine-learning machine-translation natural-language-processing python pytorch roberta texar texar-pytorch text-data text-generation xlnet
Language:Python 746
hstreamdb / hstream
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
streaming database iot stream-processing haskell real-time sql hstreamdb scale distributed-systems distributed-database materialized-view kafka financial-analysis realtime-database iot-database data-processing streaming-data streaming-database
Language:Haskell 727
ChenghaoMou / text-dedup
All-in-one text de-duplication
data-processing de-duplication nlp text-processing
Language:Python 726
SebKrantz / collapse
Advanced and Fast Data Transformation in R
r cran rstats statistics data-science scientific-computing time-series panel-data weights weighted econometrics high-performance data-transformation data-aggregation data-processing data-manipulation data-analysis
Language:C 692
synthBTC
jofpin / synthBTC
A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.
bitcoin data-processing monte-carlo-simulation nodejs prediction synthetic-data turbit
Language:JavaScript 671
infoslack / awesome-kafka
A list about Apache Kafka
apache-kafka apache-spark data-pipeline data-processing infrastructure kafka kafka-streams stream-processing streaming-data
583
eternal
kousun12 / eternal
👾~ music, eternal ~ 👾
glsl node-based music 3d-graphics webaudio creative-coding webgl art midi data-processing
Language:JavaScript 565
maykulkarni / Machine-Learning-Notebooks
Machine Learning notebooks for refreshing concepts.
machine-learning python python-machine-learning machine-learning-algorithms deep-learning deep-learning-tutorial neural-networks data-science-notebook data-processing regression-models classification-trees clustering-methods reinforcement-learning natural-language-processing dimensionality-reduction model-evaluation deep-learning-algorithms machine-learning-tutorials
Language:Jupyter Notebook 533
Puchaczov / Musoq
SQL Syntax without any database
ai-assisted-queries cross-platform csharp csv data-analysis-sql data-exploration data-processing dotnet dotnet-core dotnetcore file-system plugin-architecture query-language sql text-processing
Language:C# 492
constellation-rs / amadeus
Harmonious distributed data analysis in Rust.
data-processing data-analysis stream-processing parallel-computing distributed-computing rust
Language:Rust 482
polyaxon / haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
tensorflow deep-learning jupyter python pytorch machine-learning models ui visualization matplotlib plotly bokeh mlops data-science data-visualization data-processing data-profiling tracking lineage serving
Language:Python 451
keithorange / PatternPy
📈 PatternPy: A Python package revolutionizing trading analysis with high-speed pattern recognition, leveraging Pandas & Numpy. Effortlessly spot Head & Shoulders, Tops & Bottoms, Supports & Resistances. For experts & beginners. #TradingMadeEasy 🔥
algorithmic-trading data-processing financial-analysis investment numpy open-source pandas pattern-recognition python stock-market strategies technical-analysis trading wealth-creation ohlcv-analysis
Language:Python 394
msamogh / nonechucks
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch
Language:Python 378