ValueError: fill value must be in categories
swemeshy opened this issue · comments
When I use pg.aggregate_matrices to aggregate two H5AD files that have been preprocessed with Pegasus, I get the following error:
>>> pg.aggregate_matrices('aggregate_matrices.csv')
2021-02-03 01:54:28,996 - pegasusio.readwrite - INFO - h5ad file '/projects/ascites/2020-11-06_analysis_Pilot1/data/preprocessed_filtered_singlets.h5ad' is loaded.
2021-02-03 01:54:28,996 - pegasusio.readwrite - INFO - Function 'read_input' finished in 1.10s.
2021-02-03 01:54:30,306 - pegasusio.readwrite - INFO - h5ad file '/projects/ascites/2020-11-25_analysis_Pilot2/data/preprocessed_filtered_singlets.h5ad' is loaded.
2021-02-03 01:54:30,306 - pegasusio.readwrite - INFO - Function 'read_input' finished in 1.31s.
2021-02-03 01:54:30,466 - pegasusio.qc_utils - INFO - After filtration, 10302 out of 10302 cell barcodes are kept in UnimodalData object GRCh38-rna.
2021-02-03 01:54:30,559 - pegasusio.qc_utils - INFO - After filtration, 14836 out of 14836 cell barcodes are kept in UnimodalData object GRCh38-rna.
Traceback (most recent call last):
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3417, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-19-83d8f3ba4acc>", line 1, in <module>
pg.aggregate_matrices('aggregate_matrices.csv')
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pegasusio/decorators.py", line 12, in wrapper_timer
result = func(*args, **kwargs)
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pegasusio/data_aggregation.py", line 216, in aggregate_matrices
aggregated_data = aggrData.aggregate()
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pegasusio/decorators.py", line 12, in wrapper_timer
result = func(*args, **kwargs)
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pegasusio/aggr_data.py", line 200, in aggregate
unidata = self._aggregate_unidata(self.aggr.pop(key))
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pegasusio/decorators.py", line 30, in wrapper_run_gc
result = func(*args, **kwargs)
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pegasusio/aggr_data.py", line 138, in _aggregate_unidata
barcode_metadata.fillna(value=fillna_dict, inplace=True)
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pandas/core/frame.py", line 4321, in fillna
return super().fillna(
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pandas/core/generic.py", line 6078, in fillna
obj.fillna(v, limit=limit, inplace=True, downcast=downcast)
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pandas/core/series.py", line 4530, in fillna
return super().fillna(
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pandas/core/generic.py", line 6061, in fillna
new_data = self._mgr.fillna(
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 594, in fillna
return self.apply(
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 409, in apply
applied = getattr(b, f)(**kwargs)
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 1779, in fillna
values = values.fillna(value=value, limit=limit)
File "/home/sramesh/software/miniconda3/lib/python3.8/site-packages/pandas/core/arrays/categorical.py", line 1721, in fillna
raise ValueError("fill value must be in categories")
ValueError: fill value must be in categories
I'm not sure how to fix this error. Can you please help?
I'm using pegasuspy version 1.2.0 and pegasusio version 0.2.9.
Thanks!
I see that you were aggregating files in h5ad format, which stores string-type attributes in category type. This can be possible cause of the error, as the default NA fill value for categorical variable is "", which probably doesn't exist unless being added as a new level to that categorical attribute.
I'll see what the best way of handling this situation, fix it, and let you know in this thread.
