KeyError when removing long examples after removing duplicate rows

[Serina Grill]

Error:

openai tools fine_tunes.prepare_data -f training_data_2022-09-14.jsonl
Analyzing...

- Your file contains 2446 prompt-completion pairs
Based on the analysis we will perform the following actions:
- [Recommended] Remove 1155 duplicate rows [Y/n]: y
- [Recommended] Remove 49 long examples [Y/n]: y
Traceback (most recent call last):
  File "/Users/ser/project/project-venv/bin/openai", line 8, in <module>
    sys.exit(main())
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/openai/_openai_scripts.py", line 63, in main
    args.func(args)
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/openai/cli.py", line 531, in prepare_data
    apply_validators(
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/openai/validators.py", line 851, in apply_validators
    df, optional_applied = apply_optional_remediation(
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/openai/validators.py", line 578, in apply_optional_remediation
    df = remediation.optional_fn(df)
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/openai/validators.py", line 171, in optional_fn
    return x.drop(long_indexes)
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/pandas/core/frame.py", line 4957, in drop
    return super().drop(
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/pandas/core/generic.py", line 4267, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/pandas/core/generic.py", line 4311, in _drop_axis
    new_axis = axis.drop(labels, errors=errors)
  File "/Users/ser/project/project-venv/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 6661, in drop
    raise KeyError(f"{list(labels[mask])} not found in axis")
KeyError: '[330, 352, 377, 378, 422, 424, 435, 1172, 1194, 1219, 1220, 1264, 1266, 1277, 1468, 1498, 1549, 1641, 1648, 1714, 1741, 1816, 1859, 1984] not found in axis'

I believe that since the duplicate rows were removed, many of the long examples are missing, throwing this error. And thus I end up needing to apply the first recommendation and not the second one, and then use the resulting file to apply the second recommendation.

It would be great to be able to apply both changes to the same file.

[hallacy]

Hi @serinamarie! Thanks for the issue!

Yep, that sure looks like what's happening. We should be able to take a look at this soon and dive into why this is happening (or if you want to take a shot at it, I certainly won't stop you)

[Serina Grill]

Sounds good, I believe I know what the issue is so I can get to it this weekend 👍

[Serina Grill]

Hi @hallacy, I have created a new branch locally with my changes but am unable to push them as I do not have write access. Would it be possible to get it? Thanks in advance.

[hallacy]

Right now the best way to submit a PR to us is to fork the repo, make your changes, and then open up a PR to merge your fork back in. It's a bit clunky but also seems to be the best way for us to maintain the right security controls over the repo unfortunately

[Serina Grill]

@hallacy Should I provide tests alongside the fix?

[hallacy]

I'm absolutely not going to say no to tests though I've already made you go through one round of edits. I can merge your PR in later today

[Serina Grill]

@hallacy Great, I've just added a test if you can review!