[Bug]: `create_derivatives` uploads files already in the store

[Renaud Chaput]

Report

I am using an external provider to process some video files. I calls back a webhook on my side when the files are processed and uploaded to the bucket used as the store for the derivatives.

Upon receiving this webhook, I call a custom derivatives provider to generate some thumbnails and store both the generated thumbnails and the already existing video versions:

# file_uploader.rb
Attacher.derivatives :from_provider,
                      download:
                        false do |_original, video_versions:, video_thumbnail:|
  versions = {
    # … the versions for the image thumbnail are computed here, from the `video_thumbnail` arg
  }

  version_storage = derivative_storage(nil)

  versions.merge!(
    video_versions.transform_values do |vid|
      Shrine::UploadedFile.new(
        id: File.basename(vid['url']),
        storage: version_storage,
        metadata: {
          size: vid['size'],
          mime_type: 'video/mp4',
        }.compact,
      )
    end,
  )

  versions
end

# webhook processor
asset.file_attacher.create_derivatives(
  :from_coconut,
  video_versions:,
  video_thumbnail: video_thumbnail_url,
)

asset.file_attacher.promote

asset.file_attacher.add_metadata(
  {
    filename: asset.filename,
    size: original_metadata['size'],
  }.compact,
)

asset.save!

When create_derivatives is called, upload_derivative is called for every version, even if the version is an UploadedFile with the same storage as the expected one.

Expected Behavior

UploadedFile with the correct storage are not uploaded again

Actual Behavior

UploadedFile returned by derivatives processor are uploaded

Steps to Reproduce the Problem

2. Ruby code to demonstrate the problem.

Ruby Version

3.2.2

Shrine Version

3.4.0

Anything else?

No response

[Janko Marohnić]

Yes, upload_derivatives will always upload the received files, even if they're already uploaded to a storage. This is by design, it allows you to re-upload files somewhere else if needed (or copy, in case of S3).

You probably want to use merge_derivatives or set_derivatives if you just want to assign already uploaded files. You can combine that with process_derivatives, since as you said create_derivatives calls upload_derivatives internally:

files = asset.file_attacher.process_derivatives(
  :from_coconut,
  video_versions:,
  video_thumbnail: video_thumbnail_url,
)

asset.file_attacher.set_derivatives(files)

[Renaud Chaput]

Thanks, I did something similar.

I thought that if the existing storage was the same as the target they would not be duplicated, but I guess there are use-cases for re-uploading to the same storage.