CT Error
glenn-jocher opened this issue · comments
Search before asking
- I have searched the HUB-SDK issues and found no similar bug report.
HUB-SDK Component
Other
Bug
@yogendrasinghx 3 failing CI tests here introduced in #116
See https://github.com/ultralytics/hub-sdk/actions/runs/8650461977/job/23719232082
=================================== FAILURES ===================================
___________________________ TestModel.test_model_001 ___________________________
self = <functional.test_model.TestModel object at 0x7fe415d86120>
@pytest.mark.smoke
def test_model_001(self):
"""Verify successful retrieval of a model by ID."""
model_id = TestData().get_models_data()["valid_model_ID"]
object_manager = ObjectManager(self.client)
model_obj = object_manager.get_model()
log = self.getLogger()
log.info(f"Attempting to retrieve model with ID: {model_id}")
model = model_obj.get_model_by_id(model_id)
log.info(f"Model retrieved successfully. Model data: {model.data}")
> assert "config" in model.data, "Config information not found in the model data"
E AssertionError: Config information not found in the model data
E assert 'config' in {}
E + where {} = <hub_sdk.modules.models.Models object at 0x7fe413bfc770>.data
tests/functional/test_model.py:23: AssertionError
----------------------------- Captured stderr call -----------------------------
2024-04-11 17:05:01,098 - hub_sdk.helpers.logger - ERROR - Unknown error occurred.
2024-04-11 17:05:01,099 - hub_sdk.helpers.logger - ERROR - Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
------------------------------ Captured log call -------------------------------
INFO test_model_001:test_model.py:17 Attempting to retrieve model with ID: pmcIKukVlTQfnkk3FxgJ
ERROR hub_sdk.helpers.logger:api_client.py:111 Unknown error occurred.
ERROR hub_sdk.helpers.logger:models.py:87 Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
INFO test_model_001:test_model.py:21 Model retrieved successfully. Model data: {}
___________________________ TestModel.test_model_008 ___________________________
self = <functional.test_model.TestModel object at 0x7fe415d872c0>
clear_export_model = None
@pytest.mark.smoke
def test_model_008(self, clear_export_model):
"""Verify successful export of a model."""
log = self.getLogger()
# Retrieve necessary data
model_id = TestData().get_models_data()["valid_model_ID"]
object_manager = ObjectManager(self.client)
model_obj = object_manager.get_model()
desired_format = TestData().get_models_data()["desired_model_format"]
# Export the model
log.info(f"Exporting model {model_id} in {desired_format} format")
model_obj.export_model(model_id, format_name=desired_format)
# Check if the model is successfully exported
export_status = model_obj.is_model_exported(model_id, format_name=desired_format)
log.info(f"Model export status: {'Success' if export_status else 'Failure'}")
> assert export_status
E assert False
tests/functional/test_model.py:207: AssertionError
----------------------------- Captured stderr call -----------------------------
2024-04-11 17:05:34,109 - hub_sdk.helpers.logger - ERROR - Unknown error occurred.
2024-04-11 17:05:34,110 - hub_sdk.helpers.logger - ERROR - Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
2024-04-11 17:05:35,269 - hub_sdk.helpers.logger - ERROR - Unknown error occurred.
------------------------------ Captured log call -------------------------------
INFO test_model_008:test_model.py:200 Exporting model pmcIKukVlTQfnkk3FxgJ in ncnn format
ERROR hub_sdk.helpers.logger:api_client.py:111 Unknown error occurred.
ERROR hub_sdk.helpers.logger:models.py:87 Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
ERROR hub_sdk.helpers.logger:api_client.py:111 Unknown error occurred.
INFO test_model_008:test_model.py:205 Model export status: Failure
___________________________ TestModel.test_model_009 ___________________________
self = <functional.test_model.TestModel object at 0x7fe415d874a0>
@pytest.mark.smoke
def test_model_009(self):
"""Verify successful retrieval of model storage URL."""
log = self.getLogger()
model_id = TestData().get_models_data()["valid_model_ID"]
log.info(f"Attempting to retrieve the storage URL for model with ID: {model_id}")
object_manager = ObjectManager(self.client)
model_obj = object_manager.get_model()
# Get the model storage URL
link = model_obj.get_model_download_link(model_id)
log.info(f"Storage URL retrieved successfully. URL: {link}")
> assert f"{model_id}/best.pt" in link, f"Model ID not found in the storage URL: {link}"
E TypeError: argument of type 'NoneType' is not iterable
tests/functional/test_model.py:227: TypeError
----------------------------- Captured stderr call -----------------------------
2024-04-11 17:06:37,[62](https://github.com/ultralytics/hub-sdk/actions/runs/8650461977/job/23719232082#step:6:63)8 - hub_sdk.helpers.logger - ERROR - Unknown error occurred.
2024-04-11 17:06:37,629 - hub_sdk.helpers.logger - ERROR - Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
------------------------------ Captured log call -------------------------------
INFO test_model_009:test_model.py:217 Attempting to retrieve the storage URL for model with ID: pmcIKukVlTQfnkk3FxgJ
ERROR hub_sdk.helpers.logger:api_client.py:111 Unknown error occurred.
ERROR hub_sdk.helpers.logger:models.py:87 Received no response from the server for model id pmcIKukVlTQfnkk3FxgJ
INFO test_model_009:test_model.py:225 Storage URL retrieved successfully. URL: None
=============================== warnings summary ===============================
<frozen importlib._bootstrap>:488
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.MessageMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
<frozen importlib._bootstrap>:488
<frozen importlib._bootstrap>:488: DeprecationWarning: Type google._upb._message.ScalarMapContainer uses PyType_Spec with a metaclass that has custom tp_new. This is deprecated and will no longer be allowed in Python 3.14.
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/functional/test_model.py::TestModel::test_model_001 - AssertionError: Config information not found in the model data
assert 'config' in {}
+ where {} = <hub_sdk.modules.models.Models object at 0x7fe413bfc7[70](https://github.com/ultralytics/hub-sdk/actions/runs/8650461977/job/23719232082#step:6:71)>.data
FAILED tests/functional/test_model.py::TestModel::test_model_008 - assert False
FAILED tests/functional/test_model.py::TestModel::test_model_009 - TypeError: argument of type 'NoneType' is not iterable
======= 3 failed, 21 passed, 1 skipped, 2 warnings in 162.00s (0:02:42) ========
Environment
No response
Minimal Reproducible Example
No response
Additional
Also a few deprecation warnings here, but lower priority.
Are you willing to submit a PR?
- Yes I'd like to help by submitting a PR!
@glenn-jocher The issue with the model ID in the test data was causing the test cases to fail. I've updated the test data, so if you rerun the CT, it should work now.
@yogendrasinghx yes, I see the latest scheduled CI from today is now passing. Nice work!!!
@
@yogendrasinghx ah nevermind, I was looking at the HUB-SDK CI action, I thought you had placed the tests in there.
Ok so what we want to do is take your new YAML that you created and place it in the HUB SDK CI, which is the reason this yaml exists in the first place.
@yogendrasinghx the HUB SDK CI is the offical repo CI and follows exactly the template from Ultralytics, with all the correct run criteria like schedule, manual run capability etc, so the other yml that you have there is duplicating code and effort and adding confusion.
@glenn-jocher Sure, I've added continuous testing to our existing CI pipeline and removed the ct.yml file. This will make things simpler. I've created a PR #120 for this; please check it out.
@yogendrasinghx awesome, nice work on #120, closing this issue as completed :)