BUG: record_api.line_counts raises TypeError on unexpected keyword arguments to dump()

[Marc Garcia]

I'm getting the next error when calling record_api.line_counts. I'm using version 1.1.1.

/tmp/record_api_results.jsonl contains the output generated by python -m record_api, here there is a sample:

$ head -n 5 record_api_results.jsonl
{"location":"/api_stats/scripts/10002306.py:22","function":{"t":"builtin_function_or_method","v":"getattr"},"params":{"args":[{"t":"module","v":"pandas"},"read_csv"]}}
{"location":"/api_stats/scripts/10002306.py:22","function":{"t":"function","v":{"module":"pandas.io.parsers","name":"_make_parser_function.<locals>.parser_f"}},"bound_params":{"pos_or_kw":[["filepath_or_buffer","../input/train.csv"],["index_col",null]]}}
{"location":"/api_stats/scripts/10002306.py:23","function":{"t":"builtin_function_or_method","v":"getattr"},"params":{"args":[{"t":{"module":"pandas.core.frame","name":"DataFrame"}},"shape"]}}
{"location":"/api_stats/scripts/10002306.py:25","function":{"t":"method","v":{"self":{"t":{"module":"pandas.core.frame","name":"DataFrame"}},"name":"head"}},"bound_params":{}}
{"location":"/api_stats/scripts/10002306.py:27","function":{"t":"builtin_function_or_method","v":"getattr"},"params":{"args":[{"t":"module","v":"pandas"},"read_csv"]}}

Call to line_counts:

export PYTHON_RECORD_API_INPUT=/tmp/record_api_results.jsonl
export PYTHON_RECORD_API_OUTPUT=/tmp/record_api_results_line_counts.jsonl
python -m record_api.line_counts

Result:

Counting lines...
reading /tmp/record_api_results.jsonl: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 478985/478985 [00:02<00:00, 217632.24it/s]
writing:   0%|                                                                                                                                                                                                      | 0/13169 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/mgarcia/miniconda3/envs/pydata/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/mgarcia/miniconda3/envs/pydata/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/mgarcia/miniconda3/envs/pydata/lib/python3.7/site-packages/record_api/line_counts.py", line 42, in <module>
    __main__()
  File "/home/mgarcia/miniconda3/envs/pydata/lib/python3.7/site-packages/record_api/line_counts.py", line 38, in __main__
    write(row_)
  File "/home/mgarcia/miniconda3/envs/pydata/lib/python3.7/site-packages/record_api/jsonl.py", line 45, in write_line
    buffer.write(orjson.dumps(o, **kwargs))
TypeError: dumps() got an unexpected keyword argument

[Saul Shanabrook]

Do you have the same orjson version? I could pin the later one if that's the issue:

$ pip freeze | grep orjson
orjson==3.0.1

[Marc Garcia]

I had orjson==3.2, but I downgraded to 3.0.1 and still the same problem.

There is something very weird I don't understand:

Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import orjson
>>> orjson.dumps({})
b'{}'
>>> orjson.dumps({}, *[])
b'{}'
>>> orjson.dumps({}, *[], **{})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: dumps() got an unexpected keyword argument

For now I'm going to remove the kwargs from the call, since it seems that in this case it's not used anyway.

[Saul Shanabrook]

That looks really weird... We do use it one place to change the default kwarg passed in to change how some things I serialized I believe...

[Saul Shanabrook]

That seems like some sort of Python bug? Or an issue with your python install? I don't understand how f() and f(**{}) would be different.

[Marc Garcia]

I upgraded to the latest Python version, and it's working. It's weird that Python has this bug, but who knows... Closing, it works now.