V&A: run load, NER, NEL; train NEL
kdutia opened this issue · comments
- check data quality (joins etc)
- run load
- fix NEL for large datasets
- label NEL training data
- test model
- (after #342) add disambiguating descriptions to training data
- rewrite load function so more memory efficient
(doesn't rely on full dataframe)-> ended up usingpd.DataFrame(series.tolist())
instead ofseries.apply(pd.Series)
, and using categorical instead of string types for descriptions - run with trained model
load with a threshold of 0.8