Compactor: Clean up file is not block visit marker log when using S3 bucket

[Ben Ye]

Describe the bug
When using S3 bucket client, there are excessive logs in Compactor saying file is not block visit marker during iter operation.

err="file is not block visit marker" operation="Iter

The error is thrown at https://github.com/cortexproject/cortex/blob/master/pkg/compactor/blocks_cleaner.go#L466. When trying to clean up partial blocks, we return this error when compactor finds a non visit marker file to terminate iteration early.

However, even though the error is expected, the error still got returned from the bucket client. When using S3 client, it will be retried and eventually the error will be logged at https://github.com/cortexproject/cortex/blob/master/pkg/storage/bucket/s3/bucket_client.go#L135.

To Reproduce
Steps to reproduce the behavior:

1. Start Cortex 1.16.0 Compactor

Expected behavior
There shouldn't be any logs for file is not block visit marker as it is not an error.

[Ben Ye]

I opened thanos-io/objstore#103 to fix this issue