Not able to generate dynamic dags more than 200
kuldeep27396 opened this issue · comments
I have 20 folders and all have 100+ dags in each one of them.
One of the folders have 201 dags, but airflow dagfactory is creating an issue while parsing more than 200 dags.
What is the solution for this?
@kuldeep27396 i'm using AWS MWAA, I have a similar use case, got a question for you:
i have defined the boilerplate python file and the corresponding yml file under
s3://bucket-name/dags/folder1/dag.py
s3://bucket-name/dags/folder1/dagyaml.yml
The dag.py file is recognized and MWAA throws error saying not able to find /usr/local/airflow/dags/gagyam.yml
but if i place s3://bucket-name/dags/dagyaml.yml, it works.
Any inputs or suggestion is appreciated