Drain service attempts to create bucket that already exists
kralicky opened this issue · comments
It appears that the drain service is attempting to create a bucket that already exists. Here are the relevant logs:
2023-05-26 17:23:49,669 - INFO - Connected to S3 client
2023-05-26 17:23:49,683 - INFO - opni-drain-model bucket does not exist so creating it now
Traceback (most recent call last):
File "/app/./drain_modules.py", line 11, in <module>
drain_training_inferencing.main()
File "/app/drain_training_inferencing.py", line 377, in main
persistence = FilePersistence("workload_drain_model.bin")
File "/app/drain3/file_persistence.py", line 26, in __init__
self.connect_to_s3()
File "/app/drain3/file_persistence.py", line 56, in connect_to_s3
self.s3_client.create_bucket(Bucket=S3_BUCKET)
File "/opt/venv/lib64/python3.9/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/opt/venv/lib64/python3.9/site-packages/boto3/resources/action.py", line 83, in __call__
response = getattr(parent.meta.client, operation_name)(*args, **params)
File "/opt/venv/lib64/python3.9/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/opt/venv/lib64/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket name can not be an existing collection, and the bucket namespace is shared by all users of the system. Please select a different name and try again.
Restarting the pods does not fix the issue.
This appears to be an issue with Seaweed when nodes are rebooted. The plan is once the Python nats wrapper is updated to include object storage, then that will be used in place of Seaweed.