pyspark

There are 77 repositories under pyspark topic.

SynapseML
microsoft / SynapseML
Simple and Distributed Machine Learning
spark pyspark azure scala microsoft ml machine-learning databricks cognitive-services lightgbm http model-deployment deep-learning ai apache-spark data-science synapse big-data onnx opencv
Language:Scala 4981
ibis-project / ibis
the portable Python dataframe library
python impala pandas database clickhouse postgresql sqlite mysql datafusion sql pyspark dask duckdb bigquery pyarrow mssql polars snowflake trino sqlalchemy
Language:Python 4289
spark-nlp
JohnSnowLabs / spark-nlp
State of the Art Natural Language Processing
nlp natural-language-processing spark pyspark named-entity-recognition sentiment-analysis lemmatizer spell-checker entity-extraction part-of-speech-tagger bert transformers albert tensorflow language-detection machine-translation text-classification language-model llm question-answering
Language:Scala 3713
apache / linkis
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
sql spark hive pyspark livy linkis engine storage resource-manager application-manager context-service scriptis udf hive-table rest-api jobserver thrift-server jdbc presto impala
Language:Java 3239
uber / petastorm
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
tensorflow pytorch deep-learning machine-learning sysml pyspark pyarrow parquet parquet-files
Language:Python 1755
awesome-spark / awesome-spark
A curated list of awesome Apache Spark packages and resources.
apache-spark pyspark awesome sparkr
Language:Shell 1625
jadianes / spark-py-notebooks
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
spark python pyspark data-analysis mllib ipython-notebook notebook ipython data-science machine-learning big-data bigdata
Language:Jupyter Notebook 1610
hi-primus / optimus
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
spark pyspark data-wrangling bigdata big-data-cleaning data-science data-cleansing data-cleaner data-transformation machine-learning data-profiling data-extraction data-exploration data-analysis data-preparation cudf dask dask-cudf data-cleaning
Language:Python 1447
AlexIoannides / pyspark-example-project
Implementing best practices for PySpark ETL jobs and applications.
pyspark etl-job python data-engineering spark data-science etl etl-pipeline
Language:Python 1404
jupyter-incubator / sparkmagic
Jupyter magics and kernels for working with remote Spark clusters
spark kernel cluster livy magic sql-query pandas-dataframe jupyter pyspark kerberos notebook jupyter-notebook
Language:Python 1287
mahmoudparsian / pyspark-tutorial
PySpark-Tutorial provides basic algorithms using PySpark
big-data big-data-analytics data-algorithms pyspark spark spark-dataframes spark-rdd
Language:Jupyter Notebook 1116
ptyadana / SQL-Data-Analysis-and-Visualization-Projects
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
sql mysql mysql-notes exercises mysqlworkbench mysql-database data-analysis postgresql postgres sqlite pgadmin tableau challenges digital-music-store sql-queries sql-data-analysis python pyspark apache-spark
Language:Jupyter Notebook 1089
logicalclocks / hopsworks
Hopsworks - Data-Intensive AI platform with a Feature Store
feature-store aws azure data-science feature-engineering feature-management gcp governance kserve machine-learning mlops model-serving pyspark python serverless ml hopsworks
Language:Java 1086
mahmoudparsian / data-algorithms-book
MapReduce, Spark, Java, and Scala for Data Algorithms Book
hadoop-mapreduce java distributed-computing scala mapreduce data-algorithms python machine-learning pyspark distributed-algorithms mappers reducers apache-hadoop apache-spark design-patterns partitioning
Language:Java 1061
h2oai / sparkling-water
Sparkling Water provides H2O functionality inside Spark cluster
h2o spark machine-learning integration pysparkling rsparkling big-data pyspark scala
Language:Scala 952
WeBankFinTech / Scriptis
Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.
hue zeppelin spark hive sql pyspark scala ide hql hive-table udf resouce-management errorcode linkis
Language:Vue 797
kuwala
kuwala-io / kuwala
Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
data data-integration data-science open-data spatial-analysis elt kuwala open-source scraping dbt postgres pyspark python jupyter admin-boundaries google-trends population react no-code react-flow
Language:JavaScript 768
lyhue1991 / eat_pyspark_in_10_days
pyspark🍒🥭 is delicious，just eat it!😋😋
spark pyspark
Language:Python 741
HariSekhon / DevOps-Python-tools
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
cloudformation python hbase json avro parquet spark pyspark travis-ci elasticsearch solr hadoop hdfs dockerhub docker linux aws devops gcp gcf
Language:Python 733
ankurchavda / SparkLearning
A comprehensive Spark guide collated from multiple sources that can be referred to learn more about Spark or as an interview refresher.
spark big-data pyspark
619
MrPowers / quinn
pyspark methods to enhance developer productivity 📣 👯 🎉
pyspark apache-spark
Language:Python 582
MrPowers / chispa
PySpark test helper methods with beautiful error messages
pyspark testing
Language:Python 513
ericxiao251 / spark-syntax
This is a repo documenting the best practices in PySpark.
pyspark best-practices
Language:Jupyter Notebook 456
spark-standalone-cluster-on-docker
cluster-apps-on-docker / spark-standalone-cluster-on-docker
Learn Apache Spark in Scala, Python (PySpark) and R (SparkR) by building your own cluster with a JupyterLab interface on Docker. :zap:
spark docker python scala pyspark jupyter r sparkr
Language:Jupyter Notebook 418
capitalone / datacompy
Pandas and Spark DataFrame comparison for humans and more!
python pandas spark data data-science compare dataframes numpy pyspark fugue dask polars
Language:Python 396
ekampf / PySpark-Boilerplate
A boilerplate for writing PySpark Jobs
python apache-spark pyspark boilerplate
Language:Python 391
commoncrawl / cc-pyspark
Process Common Crawl data with Python and Spark
spark warc-files wet commoncrawl sparksql pyspark wat-files common-crawl
Language:Python 381
CamDavidsonPilon / tdigest
t-Digest data structure in Python. Useful for percentiles and quantiles, including distributed enviroments like PySpark
python percentile quantile estimate pyspark distributed-computing mapreduce
Language:Python 376
cartershanklin / pyspark-cheatsheet
PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
spark apache-spark pyspark big-data
Language:Python 356
awesome-spark / spark-gotchas
Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
apache-spark book guide pyspark
355
huseinzol05 / Gather-Deployment
Gathers Python deployment, infrastructure and practices.
tensorflow airflow docker python docker-compose kafka pyflink pyspark
Language:Jupyter Notebook 351
kevinschaich / pyspark-cheatsheet
🐍 Quick reference guide to common patterns & functions in PySpark.
pyspark pyspark-tutorial cheatsheet cheat cheatsheets reference references documentation docs data-science data spark spark-sql guide guides quickstart
347
Spark-with-Python
tirthajyoti / Spark-with-Python
Fundamentals of Spark with Python (using PySpark), code examples
pyspark spark apache-spark dataframe mlib machine-learning big-data database map-reduce python hdfs analytics hadoop distributed-computing parallel-computing sql apache
Language:Jupyter Notebook 321
MrPowers / mack
Delta Lake helper methods in PySpark
deltalake pyspark
Language:Python 271
quintoandar / butterfree
A tool for building feature stores.
python package data-engineering etl-framework etl feature-store data-science pyspark
Language:Python 269
databrickslabs / dbldatagen
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
datagen pyspark python data-generation faker spark spark-streaming delta-live-tables deltalake databricks synthetic-data datageneration datagenerator
Language:Python 268

pyspark

microsoft / SynapseML

ibis-project / ibis

JohnSnowLabs / spark-nlp

apache / linkis

uber / petastorm

awesome-spark / awesome-spark

jadianes / spark-py-notebooks

hi-primus / optimus

AlexIoannides / pyspark-example-project

jupyter-incubator / sparkmagic

mahmoudparsian / pyspark-tutorial

ptyadana / SQL-Data-Analysis-and-Visualization-Projects

logicalclocks / hopsworks

mahmoudparsian / data-algorithms-book

h2oai / sparkling-water

WeBankFinTech / Scriptis

kuwala-io / kuwala

lyhue1991 / eat_pyspark_in_10_days

HariSekhon / DevOps-Python-tools

ankurchavda / SparkLearning

MrPowers / quinn

MrPowers / chispa

ericxiao251 / spark-syntax

cluster-apps-on-docker / spark-standalone-cluster-on-docker

capitalone / datacompy

ekampf / PySpark-Boilerplate

commoncrawl / cc-pyspark

CamDavidsonPilon / tdigest

cartershanklin / pyspark-cheatsheet

awesome-spark / spark-gotchas

huseinzol05 / Gather-Deployment

kevinschaich / pyspark-cheatsheet

tirthajyoti / Spark-with-Python

MrPowers / mack

quintoandar / butterfree

databrickslabs / dbldatagen