hubmapconsortium / ingest-pipeline

Data ingest pipeline(s) for QA/metadata etl/post-processing

hubmapconsortium/ingest-pipeline Issues

Deprecate the use of `hubmap_commons.type_client`
Closed 3 months ago3
Create script to auto-deploy source code changes (and regenerate venv)
Closed a year ago
Modify split_and_create to handle comma separated list of tissue_id values
Closed a year ago
Parallelize fastq.gz validation
Closed a year ago
Dev/tech effort to facilitate data ingestion (specifically for flagship papers)
Closed a year ago
Fix failure to launch ingest of New datasets from launch_multi_analysis
Closed a year ago1
Deploy Python 3.6 through conda environment
Closed 2 years ago
Bulk Upload support for versioned primary datasets
Updated 2 years ago1
fix instance/app.cfg.example
Closed 2 years ago
ingest-pipeline contains systemd scripts and sysconfig that have not been updated for the new airflow_wrapper
Closed 2 years ago
Conda equivalent of regenerate_venv.sh
Closed 2 years ago
Modify ingest-pipeline systemctl files to point to new python paths
Closed 2 years ago
Set up test version of Airflow upgraded to 2.latest
Closed 2 years ago2
Timeout Troubleshooting
Closed 2 years ago
Test python changes on TEST and a001
Closed 2 years ago
Create conda environment on vm198 for airflow/python changes
Closed 2 years ago
Confirm TEST and a001 availability
Closed 2 years ago
Puppetization
Updated 2 years ago
Ingest-pipeline deployed "manually"
Updated 2 years ago
Move to Jira
Updated 2 years ago
Python test for ingest validation errors
Closed 2 years ago
Plan level of backups and exporting data
Updated 2 years ago
Write scripts for running Airflow on gpu002 & otherwise general Airflow_wrapper
Closed 2 years ago1
Figure out directory structure version number in ingest-validation-tools
Closed 2 years ago1
Merge assay types (CODEX & CODEX2); connect with Ivan
Updated 2 years ago1
Create a DAG that can submit New primary datasets
Updated 2 years ago1
modify new_dataset_survey to use dataset info from REST API
Closed 2 years ago1
Airflow 2.2.5 API Plugin
Closed 2 years ago
Airflow 2.latest login migration
Closed 2 years ago
Airflow 2.2.5 DAG import review
Closed 2 years ago
Create backup for Slurm REST API VM.
Closed 2 years ago
Create backup for Pitt/CMU dev instances
Closed 2 years ago
Restore software stack from backup
Closed 2 years ago
Create backup for Airflow postgres DB
Closed 2 years ago
Happy Birthday Joel!
Closed 2 years ago2
Brainstorm ideas for improving AWS efficiency
Updated 2 years ago
Continue off-boarding/hand-off (07/15)
Closed 2 years ago
Enhance workflow audit logging
Updated 2 years ago
Data Ingestion SOP
Updated 2 years ago1
Schedule deployment
Closed 2 years ago
Configure gpu002 with python
Closed 2 years ago
Create a python utility to extract tabular data to aid TMCs reviewing datasets for publication
Closed 2 years ago
In ingest-pipeline, modify reporting pipeline for new report format
Updated 2 years ago
Modify new_dataset_survey.py to get entities list from REST endpoint
Updated 2 years ago
Modify utils._get_type_client() to accept the trick form of https connection
Closed 2 years ago1
Modify plugin test of GZIP file
Closed 2 years ago
Why isn't set_dataset_error firing for salmon_rnaseq_10x_sn?
Updated 2 years ago2
Reprocess snareseq kidney data with annotations from Azimuth
Updated 2 years ago
Do a series of experiments to determine what tasks the P100 GPUs can handle
Updated 2 years ago
Off-boarding/Hand-off Work
Updated 2 years ago