Flexible and convenient texture loading

Question

Flexible and convenient texture loading

emilk opened this issue 9 months ago · comments

Intro

Showing images in egui is currently very cumbersome. There is egui_extras::RetainedImage, but it requires you to store the RetainedImage image somewhere and is obviously not very immediate mode.

Ideally we want users to be able to write something like:

ui.image("file://image.png");
ui.image("https://www.example.com/imag.png");

We also want 3rd party crates like egui_commonmark to be able to use the same system, without having to implement their own image loading.

Desired features

egui is designed to have minimal dependencies and never doing any system calls, and I'd like to keep it that way. Therefor the solution needs to be some sort of plugin system with callbacks. This is also the best choice for flexibility.

There is three steps to loading an image:

Getting the image bytes
- file://
- https://
- From a static list of include_bytes!("my_icon.png")
- …
Decoding the image
- png, jpg, …
- svg rasterization
- …
Uploading it to a texture

In most cases we want the loaded image to be passed to egui::Context::load_texture, which wil hand the image off to whatever rendering backend egui is hooked up to. This will allow the image loaded to work with any egui integration.

We should also allow users to have more exotic ways of loading images. At Rerun we have a data store where we store images, and it would be great if we could just reference images in that store with e.g. ui.image(rerun://data/store/path);.

In some cases however, the user may want to refer to textures that they themselves have uploaded to the GPU, i.e. return a TextureId::User. We do this at Rerun.

Proposal

I propose we introduce three new traits in egui:

BytesLoader
ImageLoader
TextureLoaader

The user can then mix-and-match these as they will.

Users can register them with ctx.add_bytes_loader, ctx.add_image_loader, ctx.add_texture_loader,
and use them with ctx.load_bytes, ctx.load_image, ctx.load_texture.

We will supply good default implementations for these traits in egui_extras. Most users will never need to look at the details of these.

All these traits are designed to be immediate, i.e. callable each frame. It is therefor up to the implementation to cache any results.

They are designed to be able to suppor background loading (e.g. slow downloading of images).
For this they return Pending when something is being loaded. When the loading is done, they are responsible for calling ctx.request_repaint so that the now-loaded image will be shown.

They can also return an error.

Pending will be shown in egui using ui.spinner, and errors with red text.

Common code

enum LoadError {
    /// This loader does not support this protocol or image format.
    ///
    /// Try the next loader instead!
    NotSupported,

    /// A custom error string (e.g. "File not found: foo.png")
    Custom(String),
}

/// Given as a hint. Used mostly for rendering SVG:s to a good size.
///
/// All variants will preserve the original aspect ratio.
///
/// Similar to `usvg::FitTo`.
pub enum SizeHint {
    /// Keep original size.
    Original,

    /// Scale to width.
    Width(u32),

    /// Scale to height.
    Height(u32),

    /// Scale to size.
    Size(u32, u32),
}

`BytesLoader`

// E.g. from file or from the web
trait BytesLoader {
    /// Try loading the bytes from the given uri.
    ///
    /// When loading is done, call `ctx.request_repaint` to wake up the ui.
    fn load(&self, ctx: &egui::Context, uri: &str) -> Result<BytesPoll, LoadError>;

    /// Allow implementations to evict the cache
    fn end_frame(&self, frame_index: usize) { }
}


enum BytesPoll {
    /// Data is being loaded,
    Pending {
        /// Set if known (e.g. from a HTTP header, or by parsing the image file header).
        size: Option<Vec2u>,
    },

    /// Bytes are loaded.
    Ready {
        /// Set if known (e.g. from a HTTP header, or by parsing the image file header).
        size: Option<Vec2u>,

        /// File contents, e.g. the contents of a `.png`.
        bytes: Arc<u8>,
    },
}

`ImageLoader`

// Will usually defer to an `Arc<dyn BytesLoader>`, and then decode the result.
trait ImageLoader {
    fn load(
        &self,
        ctx: &egui::Context,
        uri: &str,
        size_hint: SizeHint,
    ) -> Result<TexturePoll, TextureLoadError>;

    /// Allow implementations to evict the cache
    fn end_frame(&self, frame_index: usize) { }
}

trait ImagePoll {
    /// Image is loading.
    Pending {
        /// Set if known (e.g. from parsing the image header).
        size: Option<Vec2u>,
    },

    Ready {
        image: epaint::ColorImage, // Yes, only color images for now. Keep it simple.
    }
}

`TextureLoader`

// Will usually defer to an `Arc<dyn ImageLoader>`,
// and then just pipe the results to `egui::Context::laod_texture`.
trait TextureLoader {
    fn load(
        &self,
        ctx: &egui::Context,
        uri: &str,
        texture_options: TextureOptions,
        size_hint: FitTo,
    ) -> Result<TexturePoll, TextureLoadError>;

    /// Allow implementations to evict the cache
    fn end_frame(&self, frame_index: usize) { }
}

struct SizedTexture {
    pub id: TextureId,
    pub size: Vec2u,
}

enum TexturePoll {
    /// Texture is loading.
    Pending {
        /// Set if known (e.g. from parsing the image header).
        size: Option<Vec2u>,
    },

    /// Texture is ready.
    Ready(SizedTexture),
}

Common usage

egui_extras::install_texture_loader(ctx);

ui.image("file://foo.png");

Advanced usage:

egui_ctx.add_texture_loader(Box::new(MyTextureLoader::new(…)));

let texture: Result<TexturePoll, _> = egui_ctx.load_texture(uri);

ui.add(egui::Image::from_uri("file://example.svg").fit_to_width());

Implementation

For version one, let's ignore cache eviction, and lets parse all images inline (on the main thread). We can always improve this in future PRs.

in `egui`

We just have the bare traits here, plus:

impl Context {
    pub fn add_bytes_loader(&self, loader: Arc<dyn BytesLoader>) { … }
    pub fn add_image_loader(&self, loader: Arc<dyn ImageLoader>) { … }
    pub fn add_texture_loader(&self, loaded: Arc<dyn TextureLoaader>) { … }

    // Uses the above registered loaders:
    pub fn load_bytes(&self, uri: &str) { 
        for loaders in &self.bytes_loaders {
            let result = loader.load(uri);
            if matches!(result, Err(LoadError::NotSupported)) {
                continue;  // try next loader
            } else {
                return result;
            }
        }
    }
    pub fn load_image(&self, uri: &str, size_hint: FitTo) { … }
    pub fn load_texture(&self, uri: &str, texture_options: TextureOptions, size_hint: FitTo) { … }
}


/// a bytes loader that loads from static sources (`include_bytes`)
struct IncludedBytesLoader {
    HashMap<String, &'static [u8]>,
}

impl DefaultTextureLoader { } 

impl TextureLoaader for DefaultTextureLoader {
    fn load(
        &self,
        ctx: &egui::Context,
        uri: &str,
        texture_options: TextureOptions,
        size_hint: FitTo,
    ) -> Result<TexturePoll, TextureLoadError>
    {
        let img = ctx.load_image(uri, size_hint)?;
        match img {
            ImagePoll::Pending { size } => Ok(TexturePoll::Pending { size }),
            ImagePoll::Ready { image } => Ok(ctx.load_texture(image, texture_options)),
        }
    }    
}

in `egui_extras`

fn install_texture_loader(ctx: &egui::Context) {
    #[cfg(not(target_os == "wasm32"))]
    ctx.egui_add_bytes_loader(Arc::new(FileLoader::new()));

    #[cfg(feature = "ehttp")]
    ctx.egui_add_bytes_loader(Arc::new(EhttpLoader::new()));

    #[cfg(feature = "image")]
    ctx.add_texture_loader(Arc::new(ImageLoader::new()));

    #[cfg(feature = "svg")]
    ctx.add_texture_loader(Arc::new(SvgLoader::new()));
}

/// Just uses `std::fs::read` directly
#[cfg(not(target_os == "wasm32"))]
struct FileLoader {
    cached: HashMap<String, Arc<[u8]>,
}

/// Uses `ehttp` to fetch images from the web in background threads/tasks.
#[cfg(feature = "ehttp")]
struct EhttpLoader {
    cached: HashMap<String, poll_promise::Promise<Result<Arc<[u8]>>>>
}

#[cfg(feature = "image")]
struct ImageCrateLoader {
    cached: HashMap<String, ColoredImage>, // loaded parsed with `image` crate
}
impl ImageLoader for ImageCrateLoader { … }

#[cfg(feature = "svg")]
struct SvgLoader {
    cached: HashMap<(String, SizeHint), ColoredImage>,
}
impl ImageLoader for SvgLoader { … }

Implementation notes

A lot of the necessary code can be found in https://github.com/lampsitter/egui_commonmark/pull/8/files?diff=split&w=0
For how to handle SVGs, see egui_extras::RetainedImage, or the above url

YgorSouza · Answer 1 · Sat Sep 02 2023 15:01:33 GMT+0800 (China Standard Time)

Pending will be shown in egui using ui.spinner, and errors with red text.

Maybe these could have overrides as well, for added flexibility.

pub fn pending_ui(self, add_contents: impl FnOnce(&mut Ui)) -> Self;
pub fn error_ui(self, add_contents: impl FnOnce(&mut Ui, LoadError)) -> Self;

But these would have to be evaluated before anything is drawn, so it would probably require having to call a show() method after all the builders. I suppose if users want more customization (like displaying specific error icons, downloading images only once and caching them in the data directory, logging errors etc), it's not that hard to write your own, currently. I tried my hand at that in this project.

Jan Procházka · Answer 2 · Wed Sep 06 2023 19:03:28 GMT+0800 (China Standard Time)

Most of the proposal is implemented, but there's work left to do:

Address the last comments on #3297
Make egui_extras::FileLoader optional
Make load error messages shorter
egui::Image should support a number of different sources
- URI
- URI + SizedTexture - equivalent to current .image API
- URI + Bytes (include_image! shorthand)
Replace old APIs with new ones, and deprecate old APIs
- ui.image2 -> ui.image
- Image2 -> Image
- load_texture -> allocate_texture + deprecate load_texture
Loading spinner should be positioned nicely within the allocated space
Loading policy: Images may take a long time to load, but they shouldn't always use background threads.
- Options:
  - Async (always use a background thread if possible)
  - Sync (always load synchronously if possible)
  - Auto (size heuristic)
- Default should be set on Context, with override on Image.
Better image type detection
- Add mime: Option<String> to BytesPoll::Ready
  - Read from Content-Type in EhttpLoader
  - Read from file extension in FileLoader
- ImageCrateLoader should fall back to image::guess_format if mime is None
Add more examples
- Add egui logo to the demo widget gallery
- Image sizing options example
- Combine a few examples into one image example (download_image, svg, retained_image)
Deprecate RetainedImage and rework examples using it to use ui.image instead
Try it in practice by making a PR to https://github.com/lampsitter/egui_commonmark and rerun
- lampsitter/egui_commonmark#11
- rerun-io/rerun#3311
Image sizing API. Images are currently always ImageFit::ShrinkToFit, we want a more flexible API.
- size_range, width_range, height_range, respect_aspect_ratio(bool)
- size_fraction, width_fraction, height_fraction
- size, width, height
- original_size + scale
Improve error messages on CORS errors on web emilk/ehttp#33
Improve error message when there are no image loaders installed
Improve the "error" image UI (perhaps show a red ⚠️ with details on hover?)
Remove the newly added #![allow(deprecated)]
Deprecate and replace Button::image_and_text and similar functions

Jan Procházka · Answer 3 · Tue Sep 12 2023 15:50:56 GMT+0800 (China Standard Time)

Improve error messages on CORS errors on web (maybe a ehttp PR)

When the server sends invalid CORS headers, then the browser will output only an opaque network error. This typically shows up as TypeError: Failed to fetch. According to the spec, a network error is a Response object with:

type set to "error"
status set to 0
an empty status message
an empty header list
a null body

Browsers may output more information to the developer console or through other means, but they may not expose this information to the page. We can't tell the user it's "likely" a CORS error, because there are a whole bunch of other possible causes for a TypeError: Failed to fetch:

No internet connection
DNS resolution failed
Firewall or proxy blocked the request
Server is not reachable
The URL is invalid (yes, this is just an opaque "failed to fetch"...)
Server's SSL cert is invalid
The initial get which returned HTML contained CSP headers to block access to the resource
A browser extension blocked the request (e.g. ad blocker)

I don't know how to improve the situation. We could list all of the possible causes, but that's a lot of stuff to throw at a user. I also don't think it really needs to be improved, because the problem is clear once you look at the devtools console, which you do often anyway.

Emil Ernerfeldt · Answer 4 · Tue Sep 12 2023 16:38:25 GMT+0800 (China Standard Time)

We can improve the situations like so:

The current error message from ehttp is huge, and contains a callstack. That callstack is unhelpful imho.
We can direct the users eyes to the development console

So, I suggest in ehttp we detect the TypeError and return a shorter error message, something like: "Failed to fetch. Check the developer console for details"

lucasmerlin · Answer 5 · Thu Sep 14 2023 07:08:14 GMT+0800 (China Standard Time)

This is awesome!
If I understand this correctly, with these changes it should be possible to implement a, say, WgpuWasmTextureLoader, that uses createImageBitmap to decode images, removing the need to include the image crate in the wasm builds, which should reduce the wasm payload size and improve image decoding performance?

In my project I'm loading a lot of images so I implemented this with my own image abstraction, but it'd be awesome if I could switch to using the new loader traits in the future. I'd be happy to implement a WgpuWasmTextureLoader, either as part of the egui_wgpu crate or as a separate external crate.

Emil Ernerfeldt · Answer 6 · Thu Sep 14 2023 14:44:00 GMT+0800 (China Standard Time)

createImageBitmap could be used by a new WebImageLoader that I think would fit well into egui_extras. It should implement trait ImageLoader, and be responsible for converting raw bytes into an egui::ColorImage. That would indeed remove the need for the image crate.

That's all you would need to implement though, and it would work for any egui running on web.
The default egui texture loader will just pass the loaded ColorImage to whatever the egui backend is, i.e. wgpu, glow, miniquad, … and it will Just Work™️.

wangxiaochuTHU · Answer 7 · Mon Sep 18 2023 16:16:27 GMT+0800 (China Standard Time)

It's great!

Btw, is it also neccessary/possible that directly passing a grayscale image buffer (like &[u16]) to GPU, to avoid expanding it to ColorImage on CPU? In the current state of version 0.22.0, 16bit-per-pixel grayscale images are not supported. Also, 8bit-per-pixel W x H grayscale images need to be expanded to a rgba Vec<u8> with length W x H x 4 on CPU firstly and then be sent to GPU. When you capture 4K/8K images from a camera device at 30 fps and want to show them on App in realtime, too much overhead is paid for this casting.

Emil Ernerfeldt · Answer 8 · Tue Sep 19 2023 03:53:09 GMT+0800 (China Standard Time)

@wangxiaochuTHU For now, only ColorImage is supported in egui, i.e. 32-bit sRGBA. Adding more image types (Gray8, Gray16, etc) is definitely possible, but I see that as outside the scope of this issue (it is an optimization, after all).

Flexible and convenient texture loading

Intro

Desired features

Proposal

Common code

BytesLoader

ImageLoader

TextureLoader

Common usage

Implementation

in egui

in egui_extras

Implementation notes

`BytesLoader`

`ImageLoader`

`TextureLoader`

in `egui`

in `egui_extras`