ValidationError thrown instead of InstructorRetryException in instructor 1.3.3

[Francis Irving]

• This is actually a bug report.
• I am not getting good LLM Results
• I have tried asking for help in the community on discord or discussions and have not received a response.
• I have tried searching the documentation and have not found an answer.

What Model are you using?

• gpt-3.5-turbo
• gpt-4-turbo
• gpt-4
• Other (please specify)

Describe the bug

When a model validation fails after the final attempt, instructor now throws a pydantic ValidationError. It used to throw an InstructorRetryException.

This is an unannounced API change, and also means that extra details like last_completion are no longer available.

I think it is a regression caused by this PR https://github.com/jxnl/instructor/pull/713/files - it probably also happens with JSONDecodeError. That PR has a new test in it that catches an exception, when the intent is probably expecting an exception to be thrown with pytest.raises.

To Reproduce

Example program instructor-bug-retry-exception.py:

import instructor
from pydantic import BaseModel, model_validator
from openai import OpenAI

class UserInfo(BaseModel):
    name: str
    age: int

    @model_validator(mode="after")
    def always_fail(self, value):
        raise ValueError("Example model that always fails validation")

client = instructor.from_openai(OpenAI())
try:
    user_info = client.chat.completions.create(
        model="gpt-3.5-turbo",
        response_model=UserInfo,
        messages=[{"role": "user", "content": "John Doe is 30 years old."}],
    )
except Exception as e:
    print("Caught " + e.__class__.__name__)

Running it with two versions of instructor:

(.venv) pangolin:fiddle francis$ pip list | grep instructor
instructor           1.3.3
(.venv) pangolin:fiddle francis$ python3 instructor-bug-retry-exception.py
Caught ValidationError
(.venv) pangolin:fiddle francis$ pip install instructor==1.3.2
Collecting instructor==1.3.2
  Using cached instructor-1.3.2-py3-none-any.whl.metadata (13 kB)
...
Using cached instructor-1.3.2-py3-none-any.whl (48 kB)
Installing collected packages: instructor
  Attempting uninstall: instructor
    Found existing installation: instructor 1.3.3
    Uninstalling instructor-1.3.3:
      Successfully uninstalled instructor-1.3.3
Successfully installed instructor-1.3.2
(.venv) pangolin:fiddle francis$ python3 instructor-bug-retry-exception.py
Caught InstructorRetryException
(.venv) pangolin:fiddle francis$

Expected behavior
An InstructorRetryException to be thrown by instructor.

Screenshots

n/a

[JP]

@ivanleomk Any thoughts on this? I saw you made a new 1.3.4 release today but we are locked at 1.3.2 because of this bug. You will see this issue also reported by someone else at #773.