generate_location: context[:record] does not expose the full model if not using :cache store

Question

generate_location: context[:record] does not expose the full model if not using :cache store

bmedici opened this issue 4 years ago · comments

Brief Description

I need to personalise object location/path when uploaded to a S3 or Minio bucket.
To do this, I've added a "generate_location" method in my ImageUploader which inherits from the Shrine class.
In this location, I'd like to insert object's context and espacially attributes of its relations/parents. These objects are not accessible on the record passed in context[:record] from generate_location, when we decide to directly push to the final storage.
If I activate uploading to :cache and then promoting to the final storage, I see two uploads and only the second is made in the right context, with access to model's attributes.

Expected behavior

I'd expect context[:record] to be the exact (mongoid) record so I can use its attribute to generate the right location.

Actual behavior

When not using :cache storage, context[:record] is not the real model object and lacks attributes that were initialised in the controller.

Simplest self-contained example code to demonstrate issue

class ImageUploader < Shrine
  plugin :mongoid, storages: [:cache, :minio_gallery_pics]
  plugin :download_endpoint, prefix: "shrine", storages: [:minio_gallery_pics]
  plugin :model, cache: true

  def generate_location(io, context = {})
    record = context[:record]
    Rails.logger.debug "DEBUG | ImageUploader/generate_location context[:record].name #{context[:record].name.inspect}"
    Rails.logger.debug "DEBUG | ImageUploader/generate_location context[:record].gallery_id #{context[:record].gallery_id.inspect}"

    case record
    when GalleryPic
      sprintf(
        'gallery-%s/pic-%s/%s',
        record.gallery_id.inspect,
        record._id,
        super
        )
    else
      pretty_location(io, **context)
    end
  end

end

class GalleryPicsController < IdentifiedController
  def create
    uploaded_files.each do |file|
      Rails.logger.debug "DEBUG | GalleryPicsController/create file #{file.inspect}"
      created = gallery.gallery_pics.create(image: file)
      Rails.logger.debug "DEBUG | GalleryPicsController/create created.gallery_id #{created.gallery_id.inspect}"
      Rails.logger.debug "DEBUG | GalleryPicsController/create created.image_data #{created.image_data.filename}"
    end
  end
end

class Gallery
  include Mongoid::Document
  has_many :gallery_pics#, dependant: :destroy
end

class GalleryPic
  include Mongoid::Document
  belongs_to :gallery

  field :image_data, type: Hash # or `type: Hash`
  include ImageUploader::Attachment(:image, store: :minio_gallery_pics)
end

When skipping :cache storage

class ImageUploader < Shrine
  plugin :model, cache: false
end

DEBUG | GalleryPicsController/create file #<ActionDispatch::Http::UploadedFile:sn22m0000gn/T/RackMultipart20200415-70101-1160anl.jpg>, @original_filename="20190804_202749.jpg", @content_type="image/jpeg", @headers="Content-Disposition: form-data; name=\"files[]\"; filename=\"20190804_202749.jpg\"\r\nContent-Type: image/jpeg\r\n">

DEBUG | ImageUploader/generate_location context[:record].gallery_id nil
DEBUG | ImageUploader/generate_location location "gallery-/pic-5e975a1262e9d211d528b44b/gallerypic/5e975a1262e9d211d528b44b/image/dcb37d8ebfb7c262cdb51c21e0a22683.jpg"

DEBUG | GalleryPicsController/create created.gallery_id BSON::ObjectId('5e974c8862e9d249a128dc90')

We see the name is built with gallery-/ meaning that record.gallery_id.to_s is empty as we see it in the debug output just above.

When activating :cache storage (two-phase promote)

class ImageUploader < Shrine
  plugin :model, cache: true
end

DEBUG | GalleryPicsController/create file #<ActionDispatch::Http::UploadedFile:0x00007fbf239ff058 @tempfile=#<Tempfile:/var/folders/lk/x4brcx0x1g74530jsx0sn22m0000gn/T/RackMultipart20200415-70101-xxn9f7.jpg>, @original_filename="20190804_132248.jpg", @content_type="image/jpeg", @headers="Content-Disposition: form-data; name=\"files[]\"; filename=\"20190804_132248.jpg\"\r\nContent-Type: image/jpeg\r\n">

DEBUG | ImageUploader/generate_location context[:record].gallery_id nil
DEBUG | ImageUploader/generate_location location "gallery-/pic-5e975b3662e9d211d528b44c/gallerypic/5e975b3662e9d211d528b44c/image/28861da885b5e7291fd4312c0a68c57d.jpg"

DEBUG | ImageUploader/generate_location context[:record].gallery_id BSON::ObjectId('5e974c8862e9d249a128dc90')
DEBUG | ImageUploader/generate_location location "gallery-5e974c8862e9d249a128dc90/pic-5e975b3662e9d211d528b44c/gallerypic/5e975b3662e9d211d528b44c/image/563374f0a6b005bdc84294302487c096.jpg"

DEBUG | GalleryPicsController/create created.gallery_id BSON::ObjectId('5e974c8862e9d249a128dc90')

Here, we see the same behaviour on the first upload (to :cache storage), and everything works as expected when promoted to the final storage, the second time.

System configuration

Ruby version: ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin19]

Shrine version: shrine (3.2.1)

Jonathan Rochkind · Answer 1 · Tue May 12 2020 02:55:37 GMT+0800 (China Standard Time)

Here's my guess. I'm just guessing, haven't tried out the code, and don't use mongoid myself.

I believe it is in fact "the real model object" -- you will probably find it is in fact the very same ruby object as the one you have in the controller. (It's object_id is the same).

But it "lacks attributes that were initialised in the controller."

I think when you do gallery.gallery_pics.create(image: file), at the point Rails/mongoid are choosing to assign the file object to the GalleryPic object, the GalleryPic object simply doesn't have it's gallery_id set yet. It probably doesn't have gallery_id. I think you'd find the same thing if you tried to have a title in GalleryPic that automatically defaulted to have gallery_id in it -- at the point the GalleryPic is instantiated, it doesn't have a gallery_id yet.

I don't think there's anything shrine can do about it, if I'm understanding properly.

I suppose you could work around this by saying something like:

created = gallery.gallery_pics.create(image: file, gallery_id: gallery._id)

But I don't think I'd recommend that.

I think I'd probably just use the two state cache/store. The "cache" is considered temporary storage, periodically clean it out of old data. The "store" is permanent storage, where files will have their permanent desired locations. There are all sorts of gotchas that using two-state cache/store save you from, I think this may be another.

Abraham Petrillo · Answer 2 · Mon Jul 06 2020 22:19:50 GMT+0800 (China Standard Time)

Also witnessed a related issue to this. I bypass the activerecord callbacks as advised elsewhere:

class FileAttachmentUploader < Shrine
  plugin :default_url
  plugin :activerecord, callbacks: false

Then manually promote the file in an after_commit like so:

class FileAttachment < Activerecord::Base
after_commit :promote_to_s3
...
def promote_to_s3
    attachment_attacher.promote(action: :store) if attachment_attacher.cached?
  end

This seems to have occurred after upgrading shrine from 2.16.0 to 3.2.1

Also given we run promote in an after_commit, the promotion seems to intially work but the result is not saved into activerecord:

[7] pry(main)> a.attachment_attacher.cached?
=> true
[8] pry(main)> a.attachment_attacher.promote(action: :store)
[9] pry(main)> a.attachment_attacher.cached?
=> false
[10] pry(main)> a.reload
[11] pry(main)> a.attachment_attacher.cached?
=> true
[12] pry(main)>

Janko Marohnić · Answer 3 · Sun Oct 04 2020 00:38:30 GMT+0800 (China Standard Time)

Apologies for the late response, I've finally picked up Shrine again and am working on resolving the open issues.

@bmedici What @jrochkind described is accurate. When using :cache storage, then assigning the attribute will upload it straight to temporary storage, and only what the model has assigned at that point will be available in the uploader. Then at the point when the cached file is being promoted to permanent storage, the model has already been persisted, so everything is set. Similar when disabling :cache, attribute assignment will upload straight to permanent storage.

I would have expected Mongoid to already set :gallery_id at the time of attribute assignment, but it seems that's not the case. I believe you could work around it by building the associated model first, and then assigning the attribute (I don't know if Mongoid associations have #build as well):

gallery_pic = gallery.gallery_pics.build
gallery_pic.gallery_id # <= hopefully it's already set here
gallery_pic.image = file # now it should be available

As @jrochkind explained, that's just how Shrine works, uploading to the storage happens immediately on attribute assignment.

@abepetrillo I'm not sure what was the issue there, but you are correct, Shrine::Attacher#promote doesn't persist the record anymore in Shrine 3.x. This was a deliberate backwards incompatible change, separation of promotion and persistence made much more sense in terms of design. You can always use #atomic_promote if you want persistence as well.