data-processing

There are 52 repositories under data-processing topic.

pathway
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
batch-processing kafka pathway python streaming machine-learning-algorithms real-time data-analytics data-pipelines data-processing dataflow etl etl-framework iot-analytics rust stream-processing time-series-analysis
Language:Python 49344
Bash-Oneliner
onceupon / Bash-Oneliner
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
oneliner-commands bash data-processing linux-administration terminal linux variables grep xargs system xwindow hardware shell shell-oneliner one-liners
10589
miller
johnkerl / miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
data-processing data-cleaning csv csv-format streaming-data streaming-algorithms tsv json json-data data-reduction data-regression statistics statistical-analysis devops devops-tools tabular-data command-line command-line-tools unix-toolkit miller
Language:Go 9516
TomWright / dasel
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
json yaml configuration selector data-structures config parser yaml-processor json-processing devops-tools golang cli toml query update xml go data-processing data-wrangling
Language:Go 7673
DALI
NVIDIA / DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
fast-data-pipeline image-augmentation data-augmentation image-processing data-processing deep-learning machine-learning python neural-network gpu gpu-tensorflow audio-processing pytorch mxnet paddle
Language:C++ 5545
datajuicer / data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
data-analysis data-science large-language-models llm data-visualization llms instruction-tuning pre-training multi-modal synthetic-data data data-pipeline data-processing foundation-models
Language:Python 5488
deepseek-ai / smallpond
A lightweight data processing framework built on DuckDB and 3FS.
data-processing duckdb
Language:Python 4835
unionai-oss / pandera
A light-weight, flexible, and expressive statistical data testing library
assertions data-assertions data-check data-cleaning data-processing data-validation data-verification dataframe-schema dataframes hypothesis-testing pandas pandas-dataframe pandas-validation pandas-validator schema testing testing-tools validation
Language:Python 4078
cocoindex
cocoindex-io / cocoindex
Data transformation framework for AI. Ultra performant, with incremental processing. 🌟 Star if you like it!
ai change-data-capture data data-indexing etl indexing pipeline python rag real-time rust semantic-search data-engineering data-infrastructure data-processing help-wanted knowledge-graph llm context-engineering hacktoberfest
Language:Rust 3219
broadway
dashbitco / broadway
Concurrent and multi-stage data ingestion and data processing with Elixir
elixir genstage data-ingestion data-processing concurrent broadway
Language:Elixir 2585
microsoft / DialoGPT
Large-scale pretraining for dialogue
dialogue machine-learning pytorch transformer text-generation dialogpt gpt-2 text-data data-processing
Language:Python 2413
asyml / texar
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
machine-learning natural-language-processing tensorflow deep-learning text-generation python machine-translation dialog-systems texar bert gpt-2 xlnet text-data data-processing casl-project
Language:Python 2390
numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
kubernetes stream-processing data-processing k8s pipeline map-reduce hacktoberfest
Language:Rust 2359
bytewax / bytewax
Python Stream Processing
python stream-processing rust data-engineering data-processing data-science dataflow machine-learning streaming-data
Language:Python 1879
python-bonobo / bonobo
Extract Transform Load for Python 3.5+
automation bonobo data-processing extract-transform-load parallelization python3
Language:Python 1605
pyper-dev / pyper
Concurrent Python made simple
asyncio concurrency python threading data-pipelines data-processing multiprocessing parallel-computing data data-collection data-engineering
Language:Python 1506
OpenDCAI / DataFlow
Easy Data Preparation with latest LLMs-based Operators and Pipelines.
data data-cleaning data-pipelines data-processing data-science data-synthesis llms operators data-agent sglang-bankend vllm-backend gradio-interface quick-data-processing
Language:Python 1452
GoogleCloudPlatform / data-science-on-gcp
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering
Language:Jupyter Notebook 1399
dolma
allenai / dolma
Data and tools for generating and inspecting OLMo pre-training data.
data-processing large-language-models llm machile-learning nlp
Language:Python 1341
NVIDIA-NeMo / Curator
Scalable data pre processing and curation toolkit for LLMs
data-curation llm data data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llmapps python llm-data-quality semantic-deduplication
Language:Python 1204
run-house / kubetorch
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
artificial-intelligence aws data-processing data-science distributed evaluation gcp inference infrastructure kubernetes machine-learning observability python pytorch ray serverless training
Language:Python 1086
microsoft / GODEL
Large-scale pretrained models for goal-directed dialog
data-processing dialogue dialogue-systems machine-learning text-data text-generation transformers conversational-ai language-grounding grounded-generation dialogpt language-model pretrained-model pytorch transformer
Language:Python 885
GoogleCloudPlatform / DataflowJavaSDK
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
google-cloud-dataflow data-science data-analysis data-mining big-data data-processing
853
benibela / xidel
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
xquery xml html json xpath cli command-line http web rest css-selector wget curl httpie xmlstarlet webscraper webscraping scraper datascraping data-processing
Language:Pascal 817
asyml / texar-pytorch
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
machine-learning natural-language-processing pytorch deep-learning text-generation python machine-translation dialog-systems texar bert gpt-2 xlnet roberta text-data data-processing texar-pytorch casl-project
Language:Python 746
hstreamdb / hstream
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
streaming database iot stream-processing haskell real-time sql hstreamdb scale distributed-systems distributed-database materialized-view kafka financial-analysis realtime-database iot-database data-processing streaming-data streaming-database
Language:Haskell 727
ChenghaoMou / text-dedup
All-in-one text de-duplication
text-processing de-duplication nlp data-processing
Language:Python 726
SebKrantz / collapse
Advanced and Fast Data Transformation in R
r cran rstats statistics data-science scientific-computing time-series panel-data weights weighted econometrics high-performance data-transformation data-aggregation data-processing data-manipulation data-analysis
Language:C 692
synthBTC
jofpin / synthBTC
A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.
bitcoin data-processing monte-carlo-simulation nodejs prediction synthetic-data turbit
Language:JavaScript 671
infoslack / awesome-kafka
A list about Apache Kafka
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing
583
eternal
kousun12 / eternal
👾~ music, eternal ~ 👾
glsl node-based music 3d-graphics webaudio creative-coding webgl art midi data-processing
Language:JavaScript 565
maykulkarni / Machine-Learning-Notebooks
Machine Learning notebooks for refreshing concepts.
machine-learning python python-machine-learning machine-learning-algorithms deep-learning deep-learning-tutorial neural-networks data-science-notebook data-processing regression-models classification-trees clustering-methods reinforcement-learning natural-language-processing dimensionality-reduction model-evaluation deep-learning-algorithms machine-learning-tutorials
Language:Jupyter Notebook 533
Puchaczov / Musoq
SQL Syntax without any database
sql csharp csv dotnet cross-platform data-analysis-sql data-exploration data-processing file-system plugin-architecture query-language text-processing ai-assisted-queries dotnet-core dotnetcore
Language:C# 492
constellation-rs / amadeus
Harmonious distributed data analysis in Rust.
data-processing data-analysis stream-processing parallel-computing distributed-computing rust
Language:Rust 482
polyaxon / haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
tensorflow deep-learning jupyter python pytorch machine-learning models ui visualization matplotlib plotly bokeh mlops data-science data-visualization data-processing data-profiling tracking lineage serving
Language:Python 451
keithorange / PatternPy
📈 PatternPy: A Python package revolutionizing trading analysis with high-speed pattern recognition, leveraging Pandas & Numpy. Effortlessly spot Head & Shoulders, Tops & Bottoms, Supports & Resistances. For experts & beginners. #TradingMadeEasy 🔥
algorithmic-trading data-processing financial-analysis investment numpy open-source pandas pattern-recognition python stock-market strategies technical-analysis trading wealth-creation ohlcv-analysis
Language:Python 394