data-processing

There are 37 repositories under data-processing topic.

Bash-Oneliner
onceupon / Bash-Oneliner
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
bash data-processing grep hardware linux linux-administration one-liners oneliner-commands shell shell-oneliner system terminal variables xargs xwindow
10095
miller
johnkerl / miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
command-line command-line-tools csv csv-format data-cleaning data-processing data-reduction data-regression devops devops-tools json json-data miller statistical-analysis statistics streaming-algorithms streaming-data tabular-data tsv unix-toolkit
Language:Go 8839
TomWright / dasel
Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.
json yaml configuration selector data-structures config parser yaml-processor json-processing devops-tools golang cli toml query update xml go data-processing data-wrangling
Language:Go 6852
DALI
NVIDIA / DALI
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
audio-processing data-augmentation data-processing deep-learning fast-data-pipeline gpu gpu-tensorflow image-augmentation image-processing machine-learning mxnet neural-network paddle python pytorch
Language:C++ 5040
unionai-oss / pandera
A light-weight, flexible, and expressive statistical data testing library
assertions data-assertions data-check data-cleaning data-processing data-validation data-verification dataframe-schema dataframes hypothesis-testing pandas pandas-dataframe pandas-validation pandas-validator schema testing testing-tools validation
Language:Python 3212
pathwaycom / pathway
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
batch-processing data-analytics data-pipelines data-processing dataflow etl etl-framework iot-analytics kafka machine-learning-algorithms pathway python real-time rust stream-processing streaming time-series-analysis
Language:Python 3053
asyml / texar
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
machine-learning natural-language-processing tensorflow deep-learning text-generation python machine-translation dialog-systems texar bert gpt-2 xlnet text-data data-processing casl-project
Language:Python 2384
broadway
dashbitco / broadway
Concurrent and multi-stage data ingestion and data processing with Elixir
broadway concurrent data-ingestion data-processing elixir genstage
Language:Elixir 2381
microsoft / DialoGPT
Large-scale pretraining for dialogue
data-processing dialogpt dialogue gpt-2 machine-learning pytorch text-data text-generation transformer
Language:Python 2341
python-bonobo / bonobo
Extract Transform Load for Python 3.5+
data-processing extract-transform-load bonobo python3 automation parallelization
Language:Python 1582
bytewax / bytewax
Python Stream Processing
data-engineering data-processing data-science dataflow machine-learning python rust stream-processing streaming-data
Language:Python 1440
GoogleCloudPlatform / data-science-on-gcp
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
data-analysis data-visualization cloud-computing machine-learning data-pipeline data-processing data-science data-engineering
Language:Jupyter Notebook 1310
numaproj / numaflow
Kubernetes-native platform to run massively parallel data/streaming jobs
data-processing hacktoberfest k8s kubernetes map-reduce pipeline stream-processing
Language:Go 1033
synthBTC
jofpin / synthBTC
A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.
bitcoin data-processing monte-carlo-simulation prediction synthetic-data turbit nodejs
Language:JavaScript 932
dolma
allenai / dolma
Data and tools for generating and inspecting OLMo pre-training data.
data-processing large-language-models llm machile-learning nlp
Language:Python 889
GoogleCloudPlatform / DataflowJavaSDK
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
big-data data-analysis data-mining data-processing data-science google-cloud-dataflow
855
microsoft / GODEL
Large-scale pretrained models for goal-directed dialog
conversational-ai data-processing dialogpt dialogue dialogue-systems grounded-generation language-grounding language-model machine-learning pretrained-model pytorch text-data text-generation transformer transformers
Language:Python 846
asyml / texar-pytorch
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
machine-learning natural-language-processing pytorch deep-learning text-generation python machine-translation dialog-systems texar bert gpt-2 xlnet roberta text-data data-processing texar-pytorch casl-project
Language:Python 744
hstreamdb / hstream
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.
streaming database iot stream-processing haskell real-time sql hstreamdb scale distributed-systems distributed-database materialized-view kafka financial-analysis realtime-database iot-database data-processing streaming-data streaming-database
Language:Haskell 704
benibela / xidel
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
xquery xml html json xpath cli command-line http web rest css-selector wget curl httpie xmlstarlet webscraper webscraping scraper datascraping data-processing
Language:Pascal 674
SebKrantz / collapse
Advanced and Fast Data Transformation in R
cran data-aggregation data-analysis data-manipulation data-processing data-science data-transformation econometrics high-performance panel-data r rstats scientific-computing statistics time-series weighted weights
Language:C 637
infoslack / awesome-kafka
A list about Apache Kafka
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing
575
ChenghaoMou / text-dedup
All-in-one text de-duplication
text-processing de-duplication nlp data-processing
Language:Python 572
eternal
kousun12 / eternal
👾~ music, eternal ~ 👾
glsl node-based music 3d-graphics webaudio creative-coding webgl art midi data-processing
Language:JavaScript 525
maykulkarni / Machine-Learning-Notebooks
Machine Learning notebooks for refreshing concepts.
classification-trees clustering-methods data-processing data-science-notebook deep-learning deep-learning-algorithms deep-learning-tutorial dimensionality-reduction machine-learning machine-learning-algorithms machine-learning-tutorials model-evaluation natural-language-processing neural-networks python python-machine-learning regression-models reinforcement-learning
Language:Jupyter Notebook 485
constellation-rs / amadeus
Harmonious distributed data analysis in Rust.
data-analysis data-processing distributed-computing parallel-computing rust stream-processing
Language:Rust 473
polyaxon / haupt
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
tensorflow deep-learning jupyter python pytorch machine-learning models ui visualization matplotlib plotly bokeh mlops data-science data-visualization data-processing data-profiling tracking lineage serving
Language:Python 453
NVIDIA / NeMo-Curator
Scalable data pre processing and curation toolkit for LLMs
data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication
Language:Jupyter Notebook 431
msamogh / nonechucks
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch
Language:Python 374
etl
flow-php / etl
PHP - ETL (Extract Transform Load) data processing library
data-engineering data-processing etl flow-php
Language:PHP 345
fondant
ml6team / fondant
Production-ready data processing made easy and shareable
data-processing fine-tuning foundation-models machine-learning pipeline python
Language:Python 333
lithops-cloud / lithops
A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
python big-data data-processing multicloud big-data-analytics serverless distributed object-storage parallel kubernetes multiprocessing cloud-computing serverless-computing serverless-functions
Language:Python 315
matousc89 / padasip
Python Adaptive Signal Processing
adaptive-filtering data-analysis data-processing machine-learning signal-processing
Language:Python 298
alttch / rapidtables
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
data-processing dictionary-data library python python3 text-formatting
Language:Python 287
streamnative / pulsar-flink
Elastic data processing with Apache Pulsar and Apache Flink
apache-pulsar apache-flink data-processing batch-processing stream-processing flink flink-connector schema-registry schema sql flink-stream-processing catalog pulsar
Language:Java 278
Yord / pxi
🧚 pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.
csv data-processing deserializer dsv json marshaller parser pixie pxi serializer ssv tsv
Language:JavaScript 269