data-processing

There are 50 repositories under data-processing topic.

pathway
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
batch-processing kafka pathway python streaming machine-learning-algorithms real-time data-analytics data-pipelines data-processing dataflow etl etl-framework iot-analytics rust stream-processing time-series-analysis
Language:Python 43151
Bash-Oneliner
onceupon / Bash-Oneliner
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
bash data-processing grep hardware linux linux-administration one-liners oneliner-commands shell shell-oneliner system terminal variables xargs xwindow
10539
miller
johnkerl / miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
data-processing data-cleaning csv csv-format streaming-data streaming-algorithms tsv json json-data data-reduction data-regression statistics statistical-analysis devops devops-tools tabular-data command-line command-line-tools unix-toolkit miller
Language:Go 9456
TomWright / dasel
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
json yaml configuration selector data-structures config parser yaml-processor json-processing devops-tools golang cli toml query update xml go data-processing data-wrangling
Language:Go 7604
DALI
NVIDIA / DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
fast-data-pipeline image-augmentation data-augmentation image-processing data-processing deep-learning machine-learning python neural-network gpu gpu-tensorflow audio-processing pytorch mxnet paddle
Language:C++ 5508
modelscope / data-juicer
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
data data-analysis data-pipeline data-processing data-science data-visualization foundation-models instruction-tuning large-language-models llm llms multi-modal pre-training synthetic-data
Language:Python 5192
deepseek-ai / smallpond
A lightweight data processing framework built on DuckDB and 3FS.
data-processing duckdb
Language:Python 4772
unionai-oss / pandera
A light-weight, flexible, and expressive statistical data testing library
pandas validation schema dataframes testing pandas-validation pandas-dataframe data-validation data-cleaning data-check testing-tools assertions data-assertions data-verification dataframe-schema hypothesis-testing pandas-validator data-processing
Language:Python 4014
cocoindex
cocoindex-io / cocoindex
Data transformation framework for AI. Ultra performant, with incremental processing.
ai change-data-capture data data-indexing etl indexing pipeline python rag real-time rust semantic-search data-engineering data-infrastructure data-processing dataflow help-wanted knowledge-graph llm context-engineering
Language:Rust 2805
broadway
dashbitco / broadway
Concurrent and multi-stage data ingestion and data processing with Elixir
broadway concurrent data-ingestion data-processing elixir genstage
Language:Elixir 2576
microsoft / DialoGPT
Large-scale pretraining for dialogue
dialogue machine-learning pytorch transformer text-generation dialogpt gpt-2 text-data data-processing
Language:Python 2404
asyml / texar
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
machine-learning natural-language-processing tensorflow deep-learning text-generation python machine-translation dialog-systems texar bert gpt-2 xlnet text-data data-processing casl-project
Language:Python 2389
numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
kubernetes stream-processing data-processing k8s pipeline map-reduce hacktoberfest
Language:Rust 2261
bytewax / bytewax
Python Stream Processing
python stream-processing rust data-engineering data-processing data-science dataflow machine-learning streaming-data
Language:Python 1812
python-bonobo / bonobo
Extract Transform Load for Python 3.5+
data-processing extract-transform-load bonobo python3 automation parallelization
Language:Python 1600
pyper-dev / pyper
Concurrent Python made simple
asyncio concurrency python threading data-pipelines data-processing multiprocessing parallel-computing data data-collection data-engineering
Language:Python 1474
GoogleCloudPlatform / data-science-on-gcp
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering
Language:Jupyter Notebook 1394
dolma
allenai / dolma
Data and tools for generating and inspecting OLMo pre-training data.
data-processing large-language-models llm machile-learning nlp
Language:Python 1313
OpenDCAI / DataFlow
Easy Data Preparation with latest LLMs-based Operators and Pipelines.
data data-cleaning data-pipelines data-processing data-science data-synthesis llms operators data-agent sglang-bankend vllm-backend gradio-interface quick-data-processing
Language:Python 1292
NVIDIA-NeMo / Curator
Scalable data pre processing and curation toolkit for LLMs
data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication
Language:Python 1145
microsoft / GODEL
Large-scale pretrained models for goal-directed dialog
data-processing dialogue dialogue-systems machine-learning text-data text-generation transformers conversational-ai language-grounding grounded-generation dialogpt language-model pretrained-model pytorch transformer
Language:Python 880
GoogleCloudPlatform / DataflowJavaSDK
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
google-cloud-dataflow data-science data-analysis data-mining big-data data-processing
852
benibela / xidel
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
xquery xml html json xpath cli command-line http web rest css-selector wget curl httpie xmlstarlet webscraper webscraping scraper datascraping data-processing
Language:Pascal 816
asyml / texar-pytorch
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
machine-learning natural-language-processing pytorch deep-learning text-generation python machine-translation dialog-systems texar bert gpt-2 xlnet roberta text-data data-processing texar-pytorch casl-project
Language:Python 746
hstreamdb / hstream
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
streaming database iot stream-processing haskell real-time sql hstreamdb scale distributed-systems distributed-database materialized-view kafka financial-analysis realtime-database iot-database data-processing streaming-data streaming-database
Language:Haskell 726
ChenghaoMou / text-dedup
All-in-one text de-duplication
text-processing de-duplication nlp data-processing
Language:Python 714
SebKrantz / collapse
Advanced and Fast Data Transformation in R
r cran rstats statistics data-science scientific-computing time-series panel-data weights weighted econometrics high-performance data-transformation data-aggregation data-processing data-manipulation data-analysis
Language:C 689
synthBTC
jofpin / synthBTC
A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.
bitcoin data-processing monte-carlo-simulation prediction synthetic-data turbit nodejs
Language:JavaScript 669
infoslack / awesome-kafka
A list about Apache Kafka
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing
581
eternal
kousun12 / eternal
👾~ music, eternal ~ 👾
glsl node-based music 3d-graphics webaudio creative-coding webgl art midi data-processing
Language:JavaScript 559
maykulkarni / Machine-Learning-Notebooks
Machine Learning notebooks for refreshing concepts.
machine-learning python python-machine-learning machine-learning-algorithms deep-learning deep-learning-tutorial neural-networks data-science-notebook data-processing regression-models classification-trees clustering-methods reinforcement-learning natural-language-processing dimensionality-reduction model-evaluation deep-learning-algorithms machine-learning-tutorials
Language:Jupyter Notebook 520
Puchaczov / Musoq
SQL Syntax without any database
sql csharp csv dotnet cross-platform data-analysis-sql data-exploration data-processing file-system plugin-architecture query-language text-processing ai-assisted-queries dotnet-core dotnetcore
Language:C# 488
constellation-rs / amadeus
Harmonious distributed data analysis in Rust.
data-processing data-analysis stream-processing parallel-computing distributed-computing rust
Language:Rust 482
polyaxon / haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
bokeh data-processing data-profiling data-science data-visualization deep-learning jupyter lineage machine-learning matplotlib mlops models plotly python pytorch serving tensorflow tracking ui visualization
Language:Python 452
msamogh / nonechucks
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch
Language:Python 379
keithorange / PatternPy
📈 PatternPy: A Python package revolutionizing trading analysis with high-speed pattern recognition, leveraging Pandas & Numpy. Effortlessly spot Head & Shoulders, Tops & Bottoms, Supports & Resistances. For experts & beginners. #TradingMadeEasy 🔥
algorithmic-trading data-processing financial-analysis investment numpy open-source pandas pattern-recognition python stock-market strategies technical-analysis trading wealth-creation ohlcv-analysis
Language:Python 378