[Bug]: requires pyarrow >=14.0.0

[Nina Montana-Brown]

Bug description

Hi team - thanks for releasing this!

I was trying to use chug to test using a large dataset as follows:

import chug

task_cfg = chug.DataTaskDocReadCfg(
    page_sampling='all',
)
data_cfg = chug.DataCfg(
    source='pixparse/pdfa-eng-wds',
    split='train',
    batch_size=None,
    format='hfids',
    num_workers=0,
)
data_loader = chug.create_loader(
    data_cfg,
    task_cfg,
)
sample = next(iter(data_loader))

However, I ran into issues with my local env, specifically around the pyarrow dependency. Originally, in my env I had pyarrow==13.0.0, however encountered the following bug when running the above code snippet:

ArrowInvalid                              Traceback (most recent call last)
Cell In[1], [line 13](vscode-notebook-cell:?execution_count=1&line=13)
      [3](vscode-notebook-cell:?execution_count=1&line=3) task_cfg = chug.DataTaskDocReadCfg(
      [4](vscode-notebook-cell:?execution_count=1&line=4)     page_sampling='all',
      [5](vscode-notebook-cell:?execution_count=1&line=5) )
      [6](vscode-notebook-cell:?execution_count=1&line=6) data_cfg = chug.DataCfg(
      [7](vscode-notebook-cell:?execution_count=1&line=7)     source='pixparse/pdfa-eng-wds',
      [8](vscode-notebook-cell:?execution_count=1&line=8)     split='train',
   (...)
     [11](vscode-notebook-cell:?execution_count=1&line=11)     num_workers=0,
     [12](vscode-notebook-cell:?execution_count=1&line=12) )
---> [13](vscode-notebook-cell:?execution_count=1&line=13) data_loader = chug.create_loader(
     [14](vscode-notebook-cell:?execution_count=1&line=14)     data_cfg,
     [15](vscode-notebook-cell:?execution_count=1&line=15)     task_cfg,
     [16](vscode-notebook-cell:?execution_count=1&line=16) )
     [17](vscode-notebook-cell:?execution_count=1&line=17) sample = next(iter(data_loader))

File [~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:48](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:48), in create_loader(data_cfg, task_cfg, task_pipeline, is_training, start_interval, seed, distributed)
     [37](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:37)     loader = create_loader_from_config_wds(
     [38](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:38)         data_cfg=data_cfg,
     [39](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:39)         task_cfg=task_cfg,
   (...)
     [44](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:44)         distributed=distributed,
     [45](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/chug/loader.py:45)     )
...
File [~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:144](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:144), in pyarrow.lib.pyarrow_internal_check_status()

File [~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:100](https://file+.vscode-resource.vscode-cdn.net/Users/nin/repos/ocr/~/miniconda3/envs/ocr_eval/lib/python3.10/site-packages/pyarrow/error.pxi:100), in pyarrow.lib.check_status()

ArrowInvalid: Unable to merge: Field json has incompatible types: struct<pages: list<item: struct<images_bbox: list<item: list<item: double>>, images_bbox_no_text_overlap: list<item: list<item: double>>, lines: struct<bbox: list<item: list<item: double>>, score: list<item: double>, text: list<item: string>, word_slice: list<item: list<item: int64>>>, words: struct<bbox: list<item: list<item: double>>, line_pos: list<item: list<item: int64>>, score: list<item: double>, text: list<item: string>>>>> vs struct<pages: list<item: struct<images_bbox: list<item: null>, images_bbox_no_text_overlap: list<item: null>, lines: struct<bbox: list<item: list<item: double>>, score: list<item: double>, text: list<item: string>, word_slice: list<item: list<item: int64>>>, words: struct<bbox: list<item: list<item: double>>, line_pos: list<item: list<item: int64>>, score: list<item: double>, text: list<item: string>>>>

Updating this to pyarrow==14.0.0 resolved this bug. Since pyarrow version is not specifically outlined in the requirements, it may be useful to update this?

[Ross Wightman]

@NMontanaBrown hmm, that's not a direct dependency of this project, it's downstream dependency of datasets and would only be used if using a dataset via HF datasets, so seems like something that should be resolved in the datasets version constraints.

[Ross Wightman]

will make note though, perhaps add something to the README, there are also some gotchas related to datasets in terms of fsspec versions that don't appear to be fully spelled out in their contraints.