Image.open with autorotate option fails

Question

Image.open with autorotate option fails

wmnnd opened this issue a year ago · comments

Opening any image with insufficient metadata (e.g. JPEGs with missing metadata or any PNG image) with Image.open returns an error when using the autorotate: true flag:

"/path/to/image.png"
|> Image.open(autorotate: true)
# {:error, "Failed to read image"}

However, using open and the autorotate operation separately, does not cause an error:

"/path/to/image.png"
|> Image.open!(autorotate: true)
|> Image.autorotate()
# {:ok,
#  {%Vix.Vips.Image{
#    ref: #Reference<0.1233456174.4567425959.78980>
#  }, %{angle: 0, flip: false}}}

Is this the intended behavior? Maybe it would be possible to still open an image when autorotate: true is set and the corresponding metadata is missing? Because it’s possible for JPEG files to not have this information and then there’s no way to know beforehand whether it’d be safe to include the flag or not.

I tried this using elixir-image 0.38.4. elixir -v: Elixir 1.15.7 (compiled with Erlang/OTP 26)

Kip Cole · Answer 1 · Thu Dec 07 2023 13:04:47 GMT+0800 (China Standard Time)

Thanks for the report @wmnnd and my apologies for the slow response.

I think libvips has some optimisations for opening jpeg images with autorotate: true and therefore it chooses an error return if it can't comply.

Secondly, :autorotate isn't a valid option for png files so I'm surprised it's working at all. And even more surprised its working in the case you show above.

I'll dig into this now.

Kip Cole · Answer 2 · Thu Dec 07 2023 13:21:50 GMT+0800 (China Standard Time)

I can see that I had :autorotate as a valid type for png, webp and some others - which is not correct so I will fix that for sure. And I will make the options handling for Image.open/2 be image-type sensitive so that it does't allow invalid options.

I still have no idea why your second example doesn't error. I can't reproduce your example, and all my test images fail with that code sequence.

Any chance you can share the image from that example?

Philipp · Answer 3 · Thu Dec 07 2023 23:28:10 GMT+0800 (China Standard Time)

Thanks for looking into this! So what would happen when you make the options type-sensitive? Will autorotate only be applied to JPEG images or would it throw an error immediately for other formats?

I just tried reproducing it with the JPEG image where I had initially encountered this, but the error no longer occurs. No idea why, I haven’t changed the file nor updated elixir-image 🤷

Kip Cole · Answer 4 · Sat Dec 09 2023 08:41:53 GMT+0800 (China Standard Time)

I've been doing some more experiments on this. If I dive down to the libvips layer (via Vix) I see the following:

# This is what `Image.open/2` calls and hence the error
iex> Vix.Vips.Image.new_from_file "./test/support/images/jose.png[autorotate=true]"
{:error, "Failed to read image"}

# This is what `new_from_file` ultimately calls and it just ignores the invalid
# option
iex> Vix.Vips.Operation.pngload "./test/support/images/jose.png", autorotate: true
{:ok,
 {%Vix.Vips.Image{ref: #Reference<0.1813262147.304218137.195396>},
  %{flags: [:VIPS_FOREIGN_PARTIAL, :VIPS_FOREIGN_NONE, :VIPS_FOREIGN_NONE]}}}

Therefore I think the right behaviour is to ignore :autorotate on file types that don't support it. However that risks a poor devx since no error is returned.

What do you think?

Kip Cole · Answer 5 · Sat Dec 09 2023 09:14:18 GMT+0800 (China Standard Time)

Another alternative, which would take more work (which is fine, but it does mean time) is to not call the generic Vix.Vips.Image.new_from_file/1 but instead call each loader specific to each file type. For example Vix.Vips.Operation.pngload/2 for png files and so on.

I can generate most of that code at compile time but its not completely straight forward to map suffixes to the right loader function.

@akash-akya, do you have any suggestions or recommendations?

Akash Hiremath · Answer 6 · Sun Dec 10 2023 16:27:28 GMT+0800 (China Standard Time)

Hey @kipcole9, agree that current behavior is a bit confusing. Just thinking what would be the correct way to handle it.

How about getting rid of accepting options as filename suffix string, and make new_from_file to accept options as keyword list. And ignore the invalid options is passed.

Then you can just forward the filename and options as a keyword list, and that should work.

Kip Cole · Answer 7 · Wed Dec 13 2023 11:29:35 GMT+0800 (China Standard Time)

@akash-akya I think that would be a welcome addition. vips_new_from_file does allow list of options to hopefully you are able to introspect to derive the valid parameters for a given image type?

Kip Cole · Answer 8 · Wed Dec 13 2023 11:35:14 GMT+0800 (China Standard Time)

On further reading I think the most appropriate solution for image is to manage autorotate: true quite differently than today.

Current behaviour is to encode is as a parameter to Vix.Vips.Image.new_from_file/2. But this parameter is only valid for a few file types. And according to this discussion, setting this option does not remove the metadata field from the image - potentially causing multiple rotations if Image.autorotate/1 is called later.

Proposed approach - breaking change

Therefore I think the right behaviour is to disallow autorotate: true in Image.open/2 and ask users to call Image.autorotate/1 after the image is open - this function works on a wide range of image types and it has the additional benefit of signalling if the image was in fact rotated and by how much.

Alternative approach - non-breaking

And alternative is to continue to support autorotate: true but actually call Image.autorotate/1 after the image is opened. This is non-breaking - but it doesn't signal if any rotation was performed.

Comments welcome - I'm ready to implement whatever makes the most sense. For me thats the first option, even though it's a breaking change.

Kip Cole · Answer 9 · Thu Dec 14 2023 13:29:17 GMT+0800 (China Standard Time)

I've implemented the breaking change above (disallow :autorotate as an option to Image.open/2) since this seems more consistent across all image types. I'm still open to discussion. I'm aiming to have a new release out by the end of the week (pending work on #114 as well).

Akash Hiremath · Answer 10 · Thu Dec 14 2023 15:02:11 GMT+0800 (China Standard Time)

Hi @kipcole9, sorry for the late reply.

vips_new_from_file does allow list of options to hopefully you are able to introspect to derive the valid parameters for a given image type?

Yes, but it is passed as variadic arguments. I am not sure how to dynamically expand keyword list to pass it though. Have to check.

Regarding autorotate. I agree with your approach, it is better to make it explicit to avoid confusion I guess.

Kip Cole · Answer 11 · Thu Dec 14 2023 16:18:29 GMT+0800 (China Standard Time)

I am not sure how to dynamically expand keyword list to pass it though. Have to check.

All things considered @akash-akya I'd say there is little reason to change your current implementation.

Regarding autorotate. I agree with your approach, it is better to make it explicit to avoid confusion I guess.

Thats my feeling too. With the added benefit that Image.autorotate/1 returns the metadata describing any rotation or flipping that occurred.

Philipp · Answer 12 · Thu Dec 14 2023 19:44:49 GMT+0800 (China Standard Time)

Thank you @kipcole9 and @akash-akya for looking into this and for building this wonderful library! 😊