quanted / hem_app

django app for the human exposure model

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

address speed when loading files into dose table in DB

kdionisio opened this issue · comments

It is taking 8+ hrs to load a file into the dose table in the DB, depending on how large the file is. Is there any way we can speed this up? The query being used to load the file into the DB is:

LOAD DATA LOCAL INFILE '/home/kdionisio/S2D_daily_all.csv' INTO TABLE hem_app_dose FIELDS TERMINATED BY ',' ENCLOSED BY '"' ignore 1 lines (person_id, @chemical, day, dir_derm_exp, dir_derm_max, dir_derm_abs, dir_inhal_exp, dir_inhal_mass, dir_inhal_max, dir_inhal_abs, dir_ingest_exp, dir_ingest_abs, hem_app_dose.release, ind_derm_exp, ind_derm_max, ind_derm_abs, ind_inhal_exp, ind_inhal_max, ind_inhal_mass, ind_inhal_abs, ind_ingest_exp, ind_ingest_abs, out_sur, out_air, drain, waste) SET created_at = NOW(), updated_at = NOW(), chemical_id = (SELECT id FROM hem_app_chemical WHERE dtxsid = @chemical), runparams_id=6;