About dagster-feedstock
Feedstock license: BSD-3-Clause
About dagster
Home: https://github.com/dagster-io
Package license: Apache-2.0
Summary: The data orchestration platform built for productivity.
Documentation: https://dagster.readthedocs.io
Dagster is a system for building modern data applications. Combining an elegant programming model and beautiful tools, Dagster allows infrastructure engineers, data engineers, and data scientists to seamlessly collaborate to process and produce the trusted, reliable data needed in today's world.
About dagster-pipes
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-pipes
Package license: Apache-2.0
Summary: Toolkit for Dagster integrations with transform logic outside of Dagster
About dagster
Home: https://github.com/dagster-io/dagster
Package license: Apache-2.0
Summary: The data orchestration platform built for productivity.
Documentation: https://dagster.readthedocs.io
Dagster lets you define pipelines in terms of the data flow between reusable, logical components, then test locally and run anywhere. With a unified view of pipelines and the assets they produce, Dagster can schedule and orchestrate Pandas, Spark, SQL, or anything else that Python can invoke.
Dagster is designed for data platform engineers, data engineers, and full-stack data scientists. Building a data platform with Dagster makes your stakeholders more independent and your systems more robust. Developing data pipelines with Dagster makes testing easier and deploying faster.
About dagster-airbyte
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-airbyte
Package license: Apache-2.0
Summary: Package for integrating Airbyte with Dagster.
About dagster-airflow
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-airflow
Package license: Apache-2.0
Summary: Airflow plugin for Dagster
About dagster-aws
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-aws
Package license: Apache-2.0
Summary: Package for AWS-specific Dagster framework solid and resource components.
About dagster-celery
Home: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-celery
Package license: Apache-2.0
Summary: Package for using Celery as Dagster's execution engine.
About dagster-census
Home: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-census
Package license: Apache-2.0
Summary: Package for integrating Census with Dagster.
About dagster-dask
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-dask
Package license: Apache-2.0
Summary: Package for using Dask as Dagster's execution engine.
About dagster-datadog
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-datadog
Package license: Apache-2.0
Summary: Package for datadog Dagster framework components.
About dagster-dbt
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-dbt
Package license: Apache-2.0
Summary: A Dagster integration for dbt
About dagster-docker
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-docker
Package license: Apache-2.0
Summary: A Dagster integration for docker
About dagster-duckdb
Home: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-duckdb
Package license: Apache-2.0
Summary: Package for DuckDB-specific Dagster framework op and resource components.
About dagster-fivetran
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-fivetran
Package license: Apache-2.0
Summary: Package for integrating Fivetran with Dagster.
About dagster-github
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-github
Package license: Apache-2.0
Summary: Github plugin for Dagster
About dagster-graphql
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-graphql
Package license: Apache-2.0
Summary: The GraphQL frontend to python dagster.
About dagster-k8s
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-k8s
Package license: Apache-2.0
Summary: A Dagster integration for k8s
About dagster-managed-elements
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-managed-elements
Package license: Apache-2.0
Summary: Package for Managed elements with Dagster.
About dagster-mlflow
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-mlflow
Package license: Apache-2.0
Summary: Package for mlflow Dagster framework components
About dagster-msteams
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-msteams
Package license: Apache-2.0
Summary: A Microsoft Teams client resource for posting to Microsoft Teams
About dagster-mysql
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-mysql
Package license: Apache-2.0
Summary: A Dagster integration for MySQL
About dagster-pagerduty
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-pagerduty
Package license: Apache-2.0
Summary: Package for pagerduty Dagster framework components.
About dagster-pandas
Home: https://github.com/dagster-io/dagster
Package license: Apache-2.0
Summary: Utilities and examples for working with pandas and dagster, an opinionated framework for expressing data pipelines
About dagster-pandera
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-pandera
Package license: Apache-2.0
Summary: Integration layer for dagster and pandera.
About dagster-papertrail
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-papertrail
Package license: Apache-2.0
Summary: Package for papertrail Dagster framework components.
About dagster-postgres
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-postgres
Package license: Apache-2.0
Summary: A Dagster integration for postgres
About dagster-prometheus
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-prometheus
Package license: Apache-2.0
Summary: A Dagster integration for prometheus
About dagster-shell
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-shell
Package license: Apache-2.0
Summary: Package for Dagster shell ops.
About dagster-slack
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-slack
Package license: Apache-2.0
Summary: A Slack client resource for posting to Slack
About dagster-snowflake
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-snowflake
Package license: Apache-2.0
Summary: Package for Snowflake Dagster framework components.
About dagster-spark
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-spark
Package license: Apache-2.0
Summary: Package for Spark Dagster framework components.
About dagster-ssh
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-ssh
Package license: Apache-2.0
Summary: Package for ssh Dagster framework components.
About dagster-twilio
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-twilio
Package license: Apache-2.0
Summary: A Dagster integration for twilio
About dagstermill
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagstermill
Package license: Apache-2.0
Summary: A Dagster integration for papermill
About dagster-celery-docker
Home: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-celery-docker
Package license: Apache-2.0
Summary: A Dagster integration for celery-docker
About dagster-celery-k8s
Home: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-celery-k8s
Package license: Apache-2.0
Summary: A Dagster integration for celery-k8s-executor
About dagster-duckdb-pandas
Home: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-duckdb-pandas
Package license: Apache-2.0
Summary: Package for storing Pandas DataFrames in DuckDB.
About dagster-duckdb-polars
Home: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-duckdb-polars
Package license: Apache-2.0
Summary: Package for storing Polars DataFrames in DuckDB.
About dagster-duckdb-pyspark
Home: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster-duckdb-pyspark
Package license: Apache-2.0
Summary: Package for storing PySpark DataFrames in DuckDB.
About dagster-gcp
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-gcp
Package license: Apache-2.0
Summary: Package for GCP-specific Dagster framework solid and resource components.
About dagster-ge
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-ge
Package license: Apache-2.0
Summary: Package for GE-specific Dagster framework solid and resource components.
About dagster-pyspark
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-pyspark
Package license: Apache-2.0
Summary: Package for PySpark Dagster framework components.
About dagster-snowflake-pandas
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-snowflake-pandas
Package license: Apache-2.0
Summary: Package for integrating Snowflake and Pandas with Dagster.
About dagster-snowflake-pyspark
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-snowflake-pyspark
Package license: Apache-2.0
Summary: Package for integrating Snowflake and PySpark with Dagster.
About dagster-webserver
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-webserver
Package license: Apache-2.0
Summary: Web UI for dagster.
About dagit
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/dagit
Package license: Apache-2.0
Summary: Dagster UI
About dagster-gcp-pandas
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-gcp-pandas
Package license: Apache-2.0
Summary: Package for storing Pandas DataFrames in GCP.
About dagster-gcp-pyspark
Home: https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-gcp-pyspark
Package license: Apache-2.0
Summary: Package for storing PySpark DataFrames in GCP
Current build status
All platforms: |
|
Current release info
Name | Downloads | Version | Platforms |
---|---|---|---|
Installing dagster
Installing dagster
from the conda-forge
channel can be achieved by adding conda-forge
to your channels with:
conda config --add channels conda-forge
conda config --set channel_priority strict
Once the conda-forge
channel has been enabled, dagit, dagster, dagster-airbyte, dagster-airflow, dagster-aws, dagster-celery, dagster-celery-docker, dagster-celery-k8s, dagster-census, dagster-dask, dagster-datadog, dagster-dbt, dagster-docker, dagster-duckdb, dagster-duckdb-pandas, dagster-duckdb-polars, dagster-duckdb-pyspark, dagster-fivetran, dagster-gcp, dagster-gcp-pandas, dagster-gcp-pyspark, dagster-ge, dagster-github, dagster-graphql, dagster-k8s, dagster-managed-elements, dagster-mlflow, dagster-msteams, dagster-mysql, dagster-pagerduty, dagster-pandas, dagster-pandera, dagster-papertrail, dagster-pipes, dagster-postgres, dagster-prometheus, dagster-pyspark, dagster-shell, dagster-slack, dagster-snowflake, dagster-snowflake-pandas, dagster-snowflake-pyspark, dagster-spark, dagster-ssh, dagster-twilio, dagster-wandb, dagster-webserver, dagstermill
can be installed with conda
:
conda install dagit dagster dagster-airbyte dagster-airflow dagster-aws dagster-celery dagster-celery-docker dagster-celery-k8s dagster-census dagster-dask dagster-datadog dagster-dbt dagster-docker dagster-duckdb dagster-duckdb-pandas dagster-duckdb-polars dagster-duckdb-pyspark dagster-fivetran dagster-gcp dagster-gcp-pandas dagster-gcp-pyspark dagster-ge dagster-github dagster-graphql dagster-k8s dagster-managed-elements dagster-mlflow dagster-msteams dagster-mysql dagster-pagerduty dagster-pandas dagster-pandera dagster-papertrail dagster-pipes dagster-postgres dagster-prometheus dagster-pyspark dagster-shell dagster-slack dagster-snowflake dagster-snowflake-pandas dagster-snowflake-pyspark dagster-spark dagster-ssh dagster-twilio dagster-wandb dagster-webserver dagstermill
or with mamba
:
mamba install dagit dagster dagster-airbyte dagster-airflow dagster-aws dagster-celery dagster-celery-docker dagster-celery-k8s dagster-census dagster-dask dagster-datadog dagster-dbt dagster-docker dagster-duckdb dagster-duckdb-pandas dagster-duckdb-polars dagster-duckdb-pyspark dagster-fivetran dagster-gcp dagster-gcp-pandas dagster-gcp-pyspark dagster-ge dagster-github dagster-graphql dagster-k8s dagster-managed-elements dagster-mlflow dagster-msteams dagster-mysql dagster-pagerduty dagster-pandas dagster-pandera dagster-papertrail dagster-pipes dagster-postgres dagster-prometheus dagster-pyspark dagster-shell dagster-slack dagster-snowflake dagster-snowflake-pandas dagster-snowflake-pyspark dagster-spark dagster-ssh dagster-twilio dagster-wandb dagster-webserver dagstermill
It is possible to list all of the versions of dagit
available on your platform with conda
:
conda search dagit --channel conda-forge
or with mamba
:
mamba search dagit --channel conda-forge
Alternatively, mamba repoquery
may provide more information:
# Search all versions available on your platform:
mamba repoquery search dagit --channel conda-forge
# List packages depending on `dagit`:
mamba repoquery whoneeds dagit --channel conda-forge
# List dependencies of `dagit`:
mamba repoquery depends dagit --channel conda-forge
About conda-forge
conda-forge is a community-led conda channel of installable packages. In order to provide high-quality builds, the process has been automated into the conda-forge GitHub organization. The conda-forge organization contains one repository for each of the installable packages. Such a repository is known as a feedstock.
A feedstock is made up of a conda recipe (the instructions on what and how to build the package) and the necessary configurations for automatic building using freely available continuous integration services. Thanks to the awesome service provided by Azure, GitHub, CircleCI, AppVeyor, Drone, and TravisCI it is possible to build and upload installable packages to the conda-forge Anaconda-Cloud channel for Linux, Windows and OSX respectively.
To manage the continuous integration and simplify feedstock maintenance
conda-smithy has been developed.
Using the conda-forge.yml
within this repository, it is possible to re-render all of
this feedstock's supporting files (e.g. the CI configuration files) with conda smithy rerender
.
For more information please check the conda-forge documentation.
Terminology
feedstock - the conda recipe (raw material), supporting scripts and CI configuration.
conda-smithy - the tool which helps orchestrate the feedstock.
Its primary use is in the construction of the CI .yml
files
and simplify the management of many feedstocks.
conda-forge - the place where the feedstock and smithy live and work to produce the finished article (built conda distributions)
Updating dagster-feedstock
If you would like to improve the dagster recipe or build a new
package version, please fork this repository and submit a PR. Upon submission,
your changes will be run on the appropriate platforms to give the reviewer an
opportunity to confirm that the changes result in a successful build. Once
merged, the recipe will be re-built and uploaded automatically to the
conda-forge
channel, whereupon the built conda packages will be available for
everybody to install and use from the conda-forge
channel.
Note that all branches in the conda-forge/dagster-feedstock are
immediately built and any created packages are uploaded, so PRs should be based
on branches in forks and branches in the main repository should only be used to
build distinct package versions.
In order to produce a uniquely identifiable distribution:
- If the version of a package is not being increased, please add or increase
the
build/number
. - If the version of a package is being increased, please remember to return
the
build/number
back to 0.