The agent protocol compliance check fails for the minimal reference implementation

Question

The agent protocol compliance check fails for the minimal reference implementation

ATheorell opened this issue 8 months ago · comments

The compliance test suggested here: https://agentprotocol.ai/compliance fails with the below error for the minimal reference implementation https://github.com/AI-Engineers-Foundation/agent-protocol-sdk-python/blob/main/examples/minimal.py and identically for the api implementation at in PR #698 in gpt-engineer-org/gpt-engineer#698 .

It looks unsurprising that the list check fails, given that the json has both the task_ids and pagination properties.

/home/axel/Software/gpt-engineer/venv/bin/python /home/axel/Software/gpt-engineer/venv/bin/agent-protocol test --url http://127.0.0.1:8000
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.3.0 -- /home/axel/Software/gpt-engineer/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/axel/Software
plugins: anyio-3.7.1, asyncio-0.21.1
asyncio: mode=strict
collecting ... collected 7 items

../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_create_agent_task[http:/127.0.0.1:8000] PASSED [ 14%]
../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_agent_tasks_ids[http:/127.0.0.1:8000] FAILED [ 28%]
../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_get_agent_task[http:/127.0.0.1:8000] PASSED [ 42%]
../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_agent_task_steps[http:/127.0.0.1:8000] FAILED [ 57%]
../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_execute_agent_task_step[http:/127.0.0.1:8000] PASSED [ 71%]
../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_artifacts[http:/127.0.0.1:8000] PASSED [ 85%]
../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_get_agent_task_step[http:/127.0.0.1:8000] FAILED [100%]

=================================== FAILURES ===================================
_______ TestCompliance.test_list_agent_tasks_ids[http://127.0.0.1:8000] ________

self = <agent_protocol.utils.compliance.main.TestCompliance object at 0x7f6b24c05e10>
url = 'http://127.0.0.1:8000'

def test_list_agent_tasks_ids(self, url):
    response = requests.get(f"{url}/ap/v1/agent/tasks")
    assert response.status_code == 200

  assert isinstance(response.json(), list)

E AssertionError

../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py:20: AssertionError
_______ TestCompliance.test_list_agent_task_steps[http://127.0.0.1:8000] _______

self = <agent_protocol.utils.compliance.main.TestCompliance object at 0x7f6b24c06560>
url = 'http://127.0.0.1:8000'

def test_list_agent_task_steps(self, url):
    # Create task
    response = requests.post(f"{url}/ap/v1/agent/tasks", json=self.task_data)
    task_id = response.json()["task_id"]
    response = requests.get(f"{url}/ap/v1/agent/tasks/{task_id}/steps")
    assert response.status_code == 200

  assert isinstance(response.json(), list)

E AssertionError

../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py:36: AssertionError
________ TestCompliance.test_get_agent_task_step[http://127.0.0.1:8000] ________

self = <agent_protocol.utils.compliance.main.TestCompliance object at 0x7f6b24c06b00>
url = 'http://127.0.0.1:8000'

def test_get_agent_task_step(self, url):
    # Create task
    response = requests.post(f"{url}/ap/v1/agent/tasks", json=self.task_data)
    task_id = response.json()["task_id"]
    # Get steps
    response = requests.get(f"{url}/ap/v1/agent/tasks/{task_id}/steps")

  step_id = response.json()[0]

E KeyError: 0

../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py:61: KeyError
=========================== short test summary info ============================
FAILED ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_agent_tasks_ids[http:/127.0.0.1:8000]
FAILED ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_agent_task_steps[http:/127.0.0.1:8000]
FAILED ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_get_agent_task_step[http:/127.0.0.1:8000]
========================= 3 failed, 4 passed in 0.14s ==========================
Traceback (most recent call last):
File "/home/axel/Software/gpt-engineer/venv/bin/agent-protocol", line 8, in
sys.exit(cli())
File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/cli.py", line 25, in _check_compliance
check_compliance(url, args)
File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py", line 104, in check_compliance
assert exit_code == 0, "Your Agent API isn't compliant with the agent protocol"
AssertionError: Your Agent API isn't compliant with the agent protocol

Sasha Dog · Answer 1 · Thu Sep 21 2023 05:22:51 GMT+0800 (China Standard Time)

We actually fixed it! Please checkout our new SDK in this repo: https://github.com/AI-Engineers-Foundation/agent-protocol-sdk-python/tree/main

Sasha Dog · Answer 2 · Thu Sep 21 2023 05:23:08 GMT+0800 (China Standard Time)

Closing, feel free to reopen if the issue persists. :)

ATheorell · Answer 3 · Thu Sep 21 2023 14:55:10 GMT+0800 (China Standard Time)

For me, following the steps:

$ git clone git@github.com:AI-Engineers-Foundation/agent-protocol-sdk-python.git

$ cd agent-protocol-sdk-python

$ poetry install

$ poetry run python examples/minimal.py

And in a separate terminal

$ pip install --upgrade agent-protocol
...
Successfully installed agent-protocol-1.0.1

$ agent-protocol test --url http://127.0.0.1:8000

still yields the error in my previous post. Please reopen and let me know what I'm doing wrong @hackgoofer

ATheorell · Answer 4 · Fri Sep 22 2023 18:05:13 GMT+0800 (China Standard Time)

Please reopen or comment :) @hackgoofer

Sasha Dog · Answer 5 · Tue Sep 26 2023 02:55:51 GMT+0800 (China Standard Time)

Hi all,

So, here is how we are testing your compliance on agent-protocol:
URL=http://127.0.0.1:8000 bash -c "$(curl -fsSL https://raw.githubusercontent.com/AI-Engineers-Foundation/agent-protocol/main/testing_suite/test.sh)"

ATheorell · Answer 6 · Tue Sep 26 2023 15:50:20 GMT+0800 (China Standard Time)

Hi,
Thanks for pointing me to the new test and I can confirm that the minimal example is working