data-processing

There are 38 repositories under data-processing topic.

Bash-Oneliner
onceupon / Bash-Oneliner
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
bash data-processing grep hardware linux linux-administration one-liners oneliner-commands shell shell-oneliner system terminal variables xargs xwindow
10286
miller
johnkerl / miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
command-line command-line-tools csv csv-format data-cleaning data-processing data-reduction data-regression devops devops-tools json json-data miller statistical-analysis statistics streaming-algorithms streaming-data tabular-data tsv unix-toolkit
Language:Go 9007
TomWright / dasel
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
cli config configuration data-processing data-structures data-wrangling devops-tools go golang json json-processing parser query selector toml update xml yaml yaml-processor
Language:Go 7171
DALI
NVIDIA / DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
audio-processing data-augmentation data-processing deep-learning fast-data-pipeline gpu gpu-tensorflow image-augmentation image-processing machine-learning mxnet neural-network paddle python pytorch
Language:C++ 5160
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
batch-processing kafka pathway python streaming machine-learning-algorithms real-time data-analytics data-pipelines data-processing dataflow etl etl-framework iot-analytics rust stream-processing time-series-analysis
Language:Python 4329
unionai-oss / pandera
A light-weight, flexible, and expressive statistical data testing library
pandas validation schema dataframes testing pandas-validation pandas-dataframe data-validation data-cleaning data-check testing-tools assertions data-assertions data-verification dataframe-schema hypothesis-testing pandas-validator data-processing
Language:Python 3404
broadway
dashbitco / broadway
Concurrent and multi-stage data ingestion and data processing with Elixir
elixir genstage data-ingestion data-processing concurrent broadway
Language:Elixir 2431
asyml / texar
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
bert casl-project data-processing deep-learning dialog-systems gpt-2 machine-learning machine-translation natural-language-processing python tensorflow texar text-data text-generation xlnet
Language:Python 2389
microsoft / DialoGPT
Large-scale pretraining for dialogue
data-processing dialogpt dialogue gpt-2 machine-learning pytorch text-data text-generation transformer
Language:Python 2361
python-bonobo / bonobo
Extract Transform Load for Python 3.5+
data-processing extract-transform-load bonobo python3 automation parallelization
Language:Python 1588
bytewax / bytewax
Python Stream Processing
data-engineering data-processing data-science dataflow machine-learning python rust stream-processing streaming-data
Language:Python 1568
GoogleCloudPlatform / data-science-on-gcp
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
cloud-computing data-analysis data-engineering data-pipeline data-processing data-science data-visualization machine-learning
Language:Jupyter Notebook 1318
numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
data-processing hacktoberfest k8s kubernetes map-reduce pipeline stream-processing
Language:Go 1292
dolma
allenai / dolma
Data and tools for generating and inspecting OLMo pre-training data.
data-processing large-language-models llm machile-learning nlp
Language:Python 996
synthBTC
jofpin / synthBTC
A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.
bitcoin data-processing monte-carlo-simulation nodejs prediction synthetic-data turbit
Language:JavaScript 910
microsoft / GODEL
Large-scale pretrained models for goal-directed dialog
conversational-ai data-processing dialogpt dialogue dialogue-systems grounded-generation language-grounding language-model machine-learning pretrained-model pytorch text-data text-generation transformer transformers
Language:Python 857
GoogleCloudPlatform / DataflowJavaSDK
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
big-data data-analysis data-mining data-processing data-science google-cloud-dataflow
854
asyml / texar-pytorch
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
bert casl-project data-processing deep-learning dialog-systems gpt-2 machine-learning machine-translation natural-language-processing python pytorch roberta texar texar-pytorch text-data text-generation xlnet
Language:Python 745
hstreamdb / hstream
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
data-processing database distributed-database distributed-systems financial-analysis haskell hstreamdb iot iot-database kafka materialized-view real-time realtime-database scale sql stream-processing streaming streaming-data streaming-database
Language:Haskell 708
benibela / xidel
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
cli command-line css-selector curl data-processing datascraping html http httpie json rest scraper web webscraper webscraping wget xml xmlstarlet xpath xquery
Language:Pascal 685
SebKrantz / collapse
Advanced and Fast Data Transformation in R
r cran rstats statistics data-science scientific-computing time-series panel-data weights weighted econometrics high-performance data-transformation data-aggregation data-processing data-manipulation data-analysis
Language:C 658
ChenghaoMou / text-dedup
All-in-one text de-duplication
data-processing de-duplication nlp text-processing
Language:Python 622
NVIDIA / NeMo-Curator
Scalable data pre processing and curation toolkit for LLMs
data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication
Language:Jupyter Notebook 615
infoslack / awesome-kafka
A list about Apache Kafka
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing
579
eternal
kousun12 / eternal
👾~ music, eternal ~ 👾
3d-graphics art creative-coding data-processing glsl midi music node-based webaudio webgl
Language:JavaScript 535
maykulkarni / Machine-Learning-Notebooks
Machine Learning notebooks for refreshing concepts.
classification-trees clustering-methods data-processing data-science-notebook deep-learning deep-learning-algorithms deep-learning-tutorial dimensionality-reduction machine-learning machine-learning-algorithms machine-learning-tutorials model-evaluation natural-language-processing neural-networks python python-machine-learning regression-models reinforcement-learning
Language:Jupyter Notebook 497
constellation-rs / amadeus
Harmonious distributed data analysis in Rust.
data-analysis data-processing distributed-computing parallel-computing rust stream-processing
Language:Rust 473
polyaxon / haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
bokeh data-processing data-profiling data-science data-visualization deep-learning jupyter lineage machine-learning matplotlib mlops models plotly python pytorch serving tensorflow tracking ui visualization
Language:Python 455
msamogh / nonechucks
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch
Language:Python 377
etl
flow-php / etl
PHP - ETL (Extract Transform Load) data processing library
data-engineering data-processing etl flow-php
Language:PHP 349
fondant
ml6team / fondant
Production-ready data processing made easy and shareable
data-processing fine-tuning foundation-models machine-learning pipeline python
Language:Python 342
lithops-cloud / lithops
A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
python big-data data-processing multicloud big-data-analytics serverless distributed object-storage parallel kubernetes multiprocessing cloud-computing serverless-computing serverless-functions
Language:Python 317
Puchaczov / Musoq
SQL Swiss Army Knife - Engine for Diverse Data Sources
ai-assisted-queries cross-platform csharp csv data-analysis-sql data-exploration data-processing dotnet dotnet-core dotnetcore file-system plugin-architecture query-language sql text-processing
Language:C# 317
matousc89 / padasip
Python Adaptive Signal Processing
adaptive-filtering signal-processing data-processing data-analysis machine-learning
Language:Python 303
alttch / rapidtables
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
data-processing dictionary-data library python python3 text-formatting
Language:Python 287
streamnative / pulsar-flink
Elastic data processing with Apache Pulsar and Apache Flink
apache-pulsar apache-flink data-processing batch-processing stream-processing flink flink-connector schema-registry schema sql flink-stream-processing catalog pulsar
Language:Java 278