AwsS3V3Adapter::visibility throws Unable to retrieve the visibility for file at location UnableToRetrieveMetadata exception

[terrafrost]

AwsS3V3Adapter::visibility throws Unable to retrieve the visibility for file at location UnableToRetrieveMetadata exception

Q	A
Flysystem Version	3.24.0
Adapter Name	AwsS3V3
Adapter version	3.24.0

Here's my code:

$client = new \Aws\S3\S3Client([
    'version' => 'latest',
    'region' => 'us-east-2',
    'credentials' => [
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
    ],
]);

// The internal adapter
$adapter = new \League\Flysystem\AwsS3V3\AwsS3V3Adapter(
    // S3Client
    $client,
    // Bucket name
    'whatever'
);

$filesystem = $adapter;

$config = new \League\Flysystem\Config();

try {
    $filesystem->visibility('RandomFolder');
} catch (\Exception $e) {
    echo $e::CLASS . '<br>';
    echo $e->getMessage();
}

When I run it I get a "Unable to retrieve the visibility for file at location" UnableToRetrieveMetadata exception.

The directory does exist, per echo $filesystem->directoryExists('RandomFolder') - it's just that the visibility() method is throwing an exception.

The IAM user that I'm using to connect to the S3 bucket has AmazonS3FullAccess and AmazonSESFullAccess policies. The JSON for the AmazonS3FullAccess policy is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*",
                "s3-object-lambda:*"
            ],
            "Resource": "*"
        }
    ]
}

That's not exactly what https://flysystem.thephpleague.com/docs/adapter/aws-s3-v3/#iam-permissions says but it seems close enough?

Anyway, this is an issue because AwsS3V3Adapter.php's copy() method is calling $this->visibility() and it's throwing an exception.

Any ideas?

[Frank de Jonge]

@terrafrost that's very weird. The visibility functions are not really intended for directories though, any particular reason why you're checking the visibility of a dir?

[terrafrost]

I'm more specifically trying to make it so that the folders can be moved (eg. $filesystem->move('RandomFolder', 'RandomFolder2', $config)), however, from that, I'm getting a Unable to copy file UnableToMoveFile exception. move() calls $this->copy($source, $destination, $config) which, in turn, does this:

        try {
            $visibility = $config->get(Config::OPTION_VISIBILITY);

            if ($visibility === null && $config->get(Config::OPTION_RETAIN_VISIBILITY, true)) {
                $visibility = $this->visibility($source)->visibility();
            }
        } catch (Throwable $exception) {
            throw UnableToCopyFile::fromLocationTo(
                $source,
                $destination,
                $exception
            );
        }

So that's where the error is ultimately coming from. I did $filesystem->visibility('RandomFolder') vs $filesystem->move('RandomFolder', 'RandomFolder2', $config) just because that was the depth of how far into the problem I dug.

[terrafrost]

Digging into it some more I see I'm getting a Aws\S3\Exception\S3Exception with this as the message:

Error executing "GetObjectAcl" on "https://whatever.s3.us-east-2.amazonaws.com/RandomFolder?acl"; AWS HTTP error: Client error: `GET https://whatever.s3.us-east-2.amazonaws.com/RandomFolder?acl` resulted in a `404 Not Found` response: NoSuchKeyThe specified key does not exist. (truncated...) NoSuchKey (client): The specified key does not exist. - NoSuchKeyThe specified key does not exist.

The folder definitely exists on S3 (as $filesystem->directoryExists('RandomFolder') is testament to). Maybe it'd be good not to check the visibility in copy() if it's trying to copy a folder?

[Frank de Jonge]

@terrafrost can you try setting the global option retain_visibility to false?

[Frank de Jonge]

Just for clarity, it's the options that you pass into the Filesystem constructor:

$filesystem = new Filesystem($adapter, ['retain_visibility' => false]);

[terrafrost]

That got me past the visibility error! Altho now it's failing here:

        try {
            $this->client->copy(
                $this->bucket,
                $this->prefixer->prefixPath($source),
                $this->bucket,
                $this->prefixer->prefixPath($destination),
                $this->visibility->visibilityToAcl($visibility ?: 'private'),
                $this->createOptionsFromConfig($config)['params']
            );
        } catch (Throwable $exception) {
            throw UnableToCopyFile::fromLocationTo($source, $destination, $exception);
        }

If I do exit($exception->getMessage()) I get this back:

Error executing "HeadObject" on "https://whatever.s3.us-east-2.amazonaws.com/RandomFolder"; AWS HTTP error: Client error: `HEAD https://whatever.s3.us-east-2.amazonaws.com/RandomFolder` resulted in a `404 Not Found` response NotFound (client): 404 Not Found (Request-ID: ...) -

It's just weird because $filesystem->directoryExists('RandomFolder') returns true. If I do $filesystem->directoryExists('FakeFolder') it returns false.

My current code:

$client = new \Aws\S3\S3Client([
    'version' => 'latest',
    'region' => 'us-east-2',
    'credentials' => [
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
    ],
]);

// The internal adapter
$adapter = new \League\Flysystem\AwsS3V3\AwsS3V3Adapter(
    // S3Client
    $client,
    // Bucket name
    'mybucket'
);

//$filesystem = $adapter;
$filesystem = new \League\Flysystem\Filesystem($adapter, ['retain_visibility' => false]);

//$config = new \League\Flysystem\Config();
$config = [];

//echo $filesystem->directoryExists('FakeFolder') ? 't' : 'f'; exit;

try {
    $filesystem->move('RandomFolder', 'RandomFolder2', $config);
} catch (\Exception $e) {
    echo $e->getMessage(); exit;
}

[terrafrost]

So I was able to isolate this down to an issue with the S3 PHP SDK:

$client = new \Aws\S3\S3Client([
    'version' => 'latest',
    'region' => 'us-east-2',
    'credentials' => [
        'key' => env('AWS_ACCESS_KEY_ID'),
        'secret' => env('AWS_SECRET_ACCESS_KEY'),
    ],
]);

$result = $client->headObject([
	'Bucket' => 'mybucket',
	'Key' => 'NewFolder' // an empty folder
]);

echo '<pre>';
var_dump($result);

That reproduced the issue so I created an issue with AWS to get to the bottom of the issue.

Here was their response:

As I understand from the case notes, you have a PHP code where you are performing a head object API call on a folder that exists but you are witnessing 404 object not found error. Thus, you want us to investigate the same. Please do correct me If I have misunderstood your query.

To begin with, I would like to elaborate on the nature of prefixes/folders. In S3, buckets and objects are the primary resources. S3 has a flat structure instead of a hierarchy like you would see in a file system. However, for the sake of organizational simplicity, the S3 console supports the folder concept as a means of grouping objects.

Now, when you create a folder from S3 console, S3 creates a 0-byte object with a key that's set to the folder name that you provided. For example, if you create a folder named 'newfolder' in your bucket, the Amazon S3 console creates a 0-byte object with the key newfolder/. The console creates this object to support the idea of folders as informed in the below attached document. Hence, here newfolder has its own existence as a 0 byte object.

[+] https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-folders.html

Based on that I decided to try this instead:

$result = $client->headObject([
	'Bucket' => 'mybucket',
	'Key' => 'NewFolder/' // an empty folder
]);

And that worked!