address speed when loading files into dose table in DB
kdionisio opened this issue · comments
It is taking 8+ hrs to load a file into the dose table in the DB, depending on how large the file is. Is there any way we can speed this up? The query being used to load the file into the DB is:
LOAD DATA LOCAL INFILE '/home/kdionisio/S2D_daily_all.csv' INTO TABLE hem_app_dose FIELDS TERMINATED BY ',' ENCLOSED BY '"' ignore 1 lines (person_id, @chemical, day, dir_derm_exp, dir_derm_max, dir_derm_abs, dir_inhal_exp, dir_inhal_mass, dir_inhal_max, dir_inhal_abs, dir_ingest_exp, dir_ingest_abs, hem_app_dose.release, ind_derm_exp, ind_derm_max, ind_derm_abs, ind_inhal_exp, ind_inhal_max, ind_inhal_mass, ind_inhal_abs, ind_ingest_exp, ind_ingest_abs, out_sur, out_air, drain, waste) SET created_at = NOW(), updated_at = NOW(), chemical_id = (SELECT id FROM hem_app_chemical WHERE dtxsid = @chemical), runparams_id=6;