[Proposal] OnScreenSize Markup

Question

[Proposal] OnScreenSize Markup

carolzbnbr opened this issue 2 years ago · comments

Carol Zambon commented 2 years ago

OnScreenSize Markup

Link to Discussion

Please link to the completed/approved Discussion

Summary

OnScreenSize Markup: A Markup for controlling Views according to a category a screen size fits in (Small, Medium, Large, ExtraLarge, etc).

Detailed Design

ScreenCategories.cs enum:
Depending on the screen-size (Width/Heigh), or a device model is, a ScreenCategories enum is used.

 public enum ScreenCategories
 {
        ExtraSmall = 1,
        Small = 2,
        Medium = 3,
        Large = 4,
        ExtraLarge = 5,
        NotSet = 6,
    }
  }

ICategoryFallbackHandler.cs:
This interface is used to determine the category a device fits in, during first execution we attempt to execute TryGetCategoryByDeviceModel and wether it returns false, we attempt to execute TryGetCategoryByPhysicalSize

    public  interface ICategoryFallbackHandler
    {
        bool TryGetCategoryByDeviceModel(string deviceModel, out ScreenCategories category);

        bool TryGetCategoryByPhysicalSize(double deviceWidth, double deviceHeight, out ScreenCategories category);
     }

OnScreenSize.cs:

The markup itself.

During the first run, the methods ICategoryFallbackHandler.TryGetCategoryByDeviceModel and ICategoryFallbackHandler.TryGetCategoryByPhysicalSize are called to determine the Device Category a device fits in, and after that, its result is cached to optimize next runs.
Depending on the Device Category a device was categorized, a value will be obtained from it's corresponding Markup's property.
For instance: If device category was categorized a Large, the markup will attempt to get the value from property OnScreenSize.Large.
In case a markup property is not defined, it will attempt to use the value from the OnScreenSize.DefaultSize.

Note: Before returning, each markup property's value are attempted to be converted to its property-type, making it possible to have a batter user experience, but allowing values to be converted to GridLength, Colors, Thinkness, RowDefinition, ColumnDefinition, and etc.

   public class OnScreenSize : IMarkupExtension
    {
        public OnScreenSize();
        public object DefaultSize { get; set; }
        public object ExtraSmall {get;set;}
        public object Small {get;set;}
        public object Medium {get;set;}
        public object Large {get;set;}
        public object ExtraLarge {get;set;}
        public object ProvideValue(IServiceProvider serviceProvider);
    }

Usage Syntax

XAML Usage

    <Label 
            Padding="{markups:OnScreenSize 
                       Medium='15, 15, 0, 0', 
                       Large='20, 20, 0, 0', 
                       DefaultSize='10, 10, 0, 0'}"
            Text="{Binding Location.Name}" TextColor="White" />

 <ContentPage.Resources>

            <Style x:Key="MainGridStyle" TargetType="Grid">
               <Setter Property="RowDefinitions">
                <markups:OnScreenSize>
                   <markups:OnScreenSize.Small>0.25*, 0.13*, 0.08*, 230, *</markups:OnScreenSize.Small>
                   <markups:OnScreenSize.Large>0.15*, 0.1*, 0.01*, 290, *</markups:OnScreenSize.Large>
                    <markups:OnScreenSize.DefaultSize>0.15*, 0.1*, 0.01*, 290, *</markups:OnScreenSize.DefaultSize>
                 </markups:OnScreenSize>
               </Setter>
                <Setter Property="ColumnDefinitions">
                <markups:OnScreenSize>
                   <markups:OnScreenSize.Small>*, 230, *</markups:OnScreenSize.Small>
                   <markups:OnScreenSize.Large>*, 290, *</markups:OnScreenSize.Large>
                   <markups:OnScreenSize.DefaultSize>*, 290, *</markups:OnScreenSize.DefaultSize>
                 </markups:OnScreenSize>
               </Setter>
            </Style>
 </ContentPage.Resources>

  <Grid  RowSpacing="0"  Padding="0" Style="{StaticResource MainGridStyle}">
      ...
  </Grid>

C# Usage

There is no code-equivalent for instantiating a Markup, since IServiceProvider instance from OnScreenSize.ProvideValue(IServiceProvider serviceProvider) is only available via XAML.

Shaun Lawrence · Answer 1 · Wed May 25 2022 04:25:15 GMT+0800 (China Standard Time)

First off I really like the idea.

I would like to explore the C# Usage option a little, in theory anything that we can build in XAML we can do in C#. The MarkupExtension might not how we build the C# UI but it would be nice to work out how it would be done in C#. I am not at my machine right now but I would be happy to take a look at some possible suggestions if that helps?

Another option I would like to check and I have to admit I don't know much about other approaches like OnIdiom, but is there a way to make things a little more type safe? Currently Small, Medium, etc. can take any value given it's object type but a GridLength can't be used for a Labels HeightRequest as an example. It would be amazing if we could add some safety here to prevent developers from doing silly things

Carol Zambon · Answer 2 · Wed May 25 2022 04:50:00 GMT+0800 (China Standard Time)

Shaun,

About your concern regarding the code-behind, I know that there are some static methods on class Device that people uses when attempting to return markup values, such as OnPlatform (Device.OnPlatform), OnIdiom (Device.OnIdiom), etc, but I don't know exactly how it is implemented, and wether they also use IServiceProvider for it. I need to take a look on xamarin source code, Maui it's probably the same.

About type safety, I also don't feel very confortable on having "object" instead of the actual type the markup should return. I need to explore a little more how the markups were implemented on Xamarin (and also Maui). I Must confess I didin't take a closer look on it.
I know that there is also a generic interface IMarkupExtension <T> for markups, but I don't know how a user would define a that generic type "T" on XAML.

About your concern on GridLength being used for Label.HeightRequest, it might happen by a user mistake, as it happens on the OnPlatform markup.

Let me get a closer look on how a code-behind extensions are implemented and I will bring some news back here.

Thanks

Carol Zambon · Answer 3 · Wed May 25 2022 05:40:16 GMT+0800 (China Standard Time)

Ok, I just implemented a static class (Markup) which behaves just like Device.OnPlatform (which I know it is obsolete on Xamarin, but on Xamarin it was replaced by RuntimePlatform which is a totally different thing), and also guarantees type safety.

Here is the code for c#

                var label = new Label();
                label.Padding = Markup.OnScreenSize<Thickness>(defaultSize: new Thickness(10, 10, 0, 0),
                                                            medium: new Thickness(15, 15, 0, 0),
                                                            large: new Thickness(20, 20, 0, 0));

Tell me what you think.

Pedro Jesus · Answer 4 · Thu May 26 2022 07:25:38 GMT+0800 (China Standard Time)

@carolzbnbr, thanks again for moving this forward❣️ I took a look into your implementation and I have something that I want to highlight.

Here you cache a bunch of device models and their Screen category (Small, Large, Medium, etc). And other Dictionary with the Size and their categories.

At the first point, I would like to avoid this approach because means that we need to populate the models Dictionary and update it when we have new device models. On .NET MAUI we have the Xamarin.Essentials APIs so we can look at how big is the screen size and use it to define the Screen category.

On Maui that will be

var height = DeviceDisplay.Current.MainDisplayInfo.Height;
var width = DeviceDisplay.Current.MainDisplayInfo.Width;

With that, we will solve the issue of having to keep a large list of devices and screen sizes, in the happy path we will need to perform this calculation just once.

But we may have other scenarios, where having that collection and calculating the size against the device display info will not help too much. Scenarios where developers will be interested in the size of the Window and not the device's screen size .

1. Having two apps opened at the same time Android

For example, in the image below I have two apps opened at the same time on my Motorola One device

So even if it's considered a Large Screen Category (1080 x 1920)in this scenario my app will execute in a Small Screen Category, in this case, should the markup change the size of the controls?

2. Changing the Window size in a desktop app

Also, we can think of desktop scenarios where the screen size can change (if the dev allows the user to do so), in that case, should the Markup be triggered and change the control's size?

3. Using iPad as a second screen

And one more scenario, now apple allows you to have your iPad as a secondary screen, so if you open a macOS application and drag and drop it inside the iPad how the markup we will handle that? For this one, I don't even know how the DeviceDisplay.Current.MainDisplayInfo will handle that it will update it to use the iPad values or the macOS values.

We can think of the answer for those scenarios together, and also if we should care about them (maybe I'm thinking too big).

Carol Zambon · Answer 5 · Thu May 26 2022 08:51:16 GMT+0800 (China Standard Time)

Hello @pictos,

Here are my answers regarding the questions/issues you raised previously.

Here you cache a bunch of device models and their Screen category (Small, Large, Medium, etc). And other Dictionary with the Size and their categories.

At the first point, I would like to avoid this approach because means that we need to populate the models Dictionary and update it when we have new device models. On .NET MAUI we have the Xamarin.Essentials APIs so we can look at how big is the screen size and use it to define the Screen category.

On Maui that will be
var height = DeviceDisplay.Current.MainDisplayInfo.Height;
var width = DeviceDisplay.Current.MainDisplayInfo.Width;
With that, we will solve the issue of having to keep a large list of devices and screen sizes, in the happy path we will need to perform this calculation just once.

In the past, during development and tests on some iOS simulators, I had issues regarding getting the correct width/height on some devices, leading to a mismatch screen size (Width/Height) between iOS simulator devices, and actual devices, so thats why I keep a list of devices "hard-coded". The user is also able to extend the DefaultFallbackHandler class and add new devices in case they need. I know it is a work-around, a hacking, but that was the only alternative I considered reliable at the time I first implemented it.

I agree that it is not an elegant solution, and yes, you are correct, that way either the maintainers, or the toolkit users (developers) would need to maintain that list up to date.

That's why I've chose to implement two methods to the class you mentioned:

TryGetCategoryByDeviceModel - attempt to get the Screen Category by a device model
TryGetCategoryByPhysicalSize - attempts do get a Screen Category by screen size

On my code we first attempt to execute method 1 and if that returns False, we fallback to execute method 2.

We may need to think about other solution, but in eiher case, we will end up having a kind of cache for Screen sizes versus Screen Categories because there is a bunch of screen sizes out there with minor differences on sizes among them, that can be incorrectly categorized.

But we may have other scenarios, where having that collection and calculating the size against the device display info will not help too much. Scenarios where developers will be interested in the size of the Window and not the device's screen size .

We could add a two-option-enum property on the markup for the user to choose from, for instance: ScreenSize versus WindowSize, and calculate accordingly. Or maybe - if possible - we could detect if the app is a Window-based app, ou a Screen-based app.

1. Having two apps opened at the same time Android

For example, in the image below I have two apps opened at the same time on my Motorola One device

So even if it's considered a Large Screen Category (1080 x 1920)in this scenario my app will execute in a Small Screen Category, in this case, should the markup change the size of the controls?

I don't have Android here for testing, but on the above sample screen you sent, does it resizes automatically when you open two apps at the same time? I mean, both app's view control page's gets resized when you put them in say dual mode?
By seeing the screen, it seems to keep the same size as if you were using in a single mode (in app fiting the whole screen). So I don't think that is something we would need to worry about.

2. Changing the Window size in a desktop app

Also, we can think of desktop scenarios where the screen size can change (if the dev allows the user to do so), in that case, should the Markup be triggered and change the control's size?

Well, I'm not expert on Maui, but on Xamarin, Markups are not involved (do not trigger) page lifecycle events. Actually is the opposide - page lifecycle events are the one responsible for triggering Xaml/Bindings/Markups/Converters, and etc.

So I don't think it is possible to start a "recalculation" for the whole screen/window views from within the markup by forcing all the controls on a page which uses the markup to change their sizes.
If the lifecycle events were raised when a page/window gets resized, that could trigger the UI elements to be recalculated, and wether that occurs, the Xaml UI would respond accordingly along with the Markup extension.

3. Using iPad as a second screen

And one more scenario, now apple allows you to have your iPad as a secondary screen, so if you open a macOS application and drag and drop it inside the iPad how the markup we will handle that? For this one, I don't even know how the DeviceDisplay.Current.MainDisplayInfo will handle that it will update it to use the iPad values or the macOS values.

We will need to test that, but I suspect that it keeps the aspect ratio of the macos's screen size when mirroing on iPad, so we might not need to worry about that too.

In my opnion, we could start by having the markup compatible with mobile phones screens, and over the time we could evolve to desktop support.

We can think of the answer for those scenarios together, and also if we should care about them (maybe I'm thinking too big).

No, you are not thinking too big. You are absolutely right, you raised important issues that we need to keep an eye on.

By now, my main concern is related on the screen categorization versus it's size, how to make it less "hard coded" and more smart? I don't have an answer yet.

Pedro Jesus · Answer 6 · Sun May 29 2022 04:52:49 GMT+0800 (China Standard Time)

@carolzbnbr sorry for the late reply, rush week on my side.

By now, my main concern is related on the screen categorization versus it's size, how to make it less "hard coded" and more smart? I don't have an answer yet.

So let's focus on this first, with MAUI we have the concept of Window, maybe we can use it to define the size that we're. In my head this also solves the issue for desktop applications (where the user can change the size of the screen)

Pedro Jesus · Answer 7 · Sun May 29 2022 04:54:28 GMT+0800 (China Standard Time)

I'm working on other issues, but let me know if you need any help. If you want/need a more real time conversation, you can join our discord server in the xamarin-communitytoolkit channel

Shaun Lawrence · Answer 8 · Sun May 29 2022 05:01:06 GMT+0800 (China Standard Time)

Sorry I do also intend on responding on the bits of our discussion but I haven't found time to properly read through it and come up with a response yet. Hopefully in the next few days I will

Carol Zambon · Answer 9 · Tue May 31 2022 03:27:26 GMT+0800 (China Standard Time)

Hi @pictos,

Hope you are doing great.

By now, my main concern is related on the screen categorization versus it's size, how to make it less "hard coded" and more smart? I don't have an answer yet.

So let's focus on this first, with MAUI we have the concept of Window, maybe we can use it to define the size that we're. In my head this also solves the issue for desktop applications (where the user can change the size of the screen)

Sorry, but I didn't understand what you have in mind.

Is this proposal approved?

Are we ready to start implementing it's MAUI version, or not?

Also, As I told you, I can't think on a different approach on how to implement that categorization without having the screen-sizes (window-size) either "cached" in a dictionary or directly hard-coded on method. Do you have another idea on how to implement that?

In case this Proposal is approved, which approach do you consider more reliable to implement? Honestly I would rely in the dictionary (Width/Height versus Screen-Category) as I've done before, but you seemed uncomfortable with that approach, so I would like to hear from you first.

Pedro Jesus · Answer 10 · Tue May 31 2022 04:37:37 GMT+0800 (China Standard Time)

Hey, @carolzbnbr I'm doing fine, how about you?

Oh, I'm sorry for the confusion. So this proposal isn't approved yet, we need to vote on it first. But before voting, I would like to have something on the path to discuss with the other members and the community.

About the screen sizes, you mentioned that you moved to the Dictionary approach due to issues on the iOS simulator, maybe on MAUI these issues are fixed, so I would like to confirm first. I can try to reserve some time to do a PoC and validate that if you don't have time right now.

I also noticed that there's no implementation to desktop targets, is that right?

Shaun Lawrence · Answer 11 · Tue May 31 2022 16:17:54 GMT+0800 (China Standard Time)

Ok, I just implemented a static class (Markup) which behaves just like Device.OnPlatform (which I know it is obsolete on Xamarin, but on Xamarin it was replaced by RuntimePlatform which is a totally different thing), and also guarantees type safety.

Here is the code for c#

                var label = new Label();
                label.Padding = Markup.OnScreenSize<Thickness>(defaultSize: new Thickness(10, 10, 0, 0),
                                                            medium: new Thickness(15, 15, 0, 0),
                                                            large: new Thickness(20, 20, 0, 0));

Tell me what you think.

I do like the look of that C# example but as you have pointed out I am not sure what could be done in XAML.

Also have you pointed out the OnPlatform extension and actually other MarkupExtensions are not type safe. It's a shame if we can't achieve type safety but it wouldn't be the end of the world.

Carol Zambon · Answer 12 · Thu Jun 02 2022 01:26:19 GMT+0800 (China Standard Time)

Hi @bijington, hope you are doing great!

Ok, I just implemented a static class (Markup) which behaves just like Device.OnPlatform (which I know it is obsolete on Xamarin, but on Xamarin it was replaced by RuntimePlatform which is a totally different thing), and also guarantees type safety.
Here is the code for c#
                var label = new Label();
                label.Padding = Markup.OnScreenSize<Thickness>(defaultSize: new Thickness(10, 10, 0, 0),
                                                            medium: new Thickness(15, 15, 0, 0),
                                                            large: new Thickness(20, 20, 0, 0));
Tell me what you think.
I do like the look of that C# example but as you have pointed out I am not sure what could be done in XAML.

That class is meant to be used on a code-behind only.
On Xaml, people will use the OnScreenSize (which implements IMarkupExtension) Markup. Check the sample usage for XAML samples.

Also have you pointed out the OnPlatform extension and actually other MarkupExtensions are not type safe. It's a shame if we can't achieve type safety but it wouldn't be the end of the world.

Yeap, thats a shame! OnPlatform is not type safe, but i know there is the IMarkupExtension<T> interface (which is by the way is not implemented by OnPlarform) which may help us. I only need some days off on my work to take a look and maybe implement PoC.

Shaun Lawrence · Answer 13 · Fri Jun 03 2022 02:31:08 GMT+0800 (China Standard Time)

@carolzbnbr I am good and I hope the same is true with yourself!

I like the idea of making things type safe where we can live you have suggested with the C# side. Sorry I missed that this would be C# only. I guess this IMarkupExtension<T> implementation for C# could use the underlying IMarkupExtension implementation for XAML to aid with code reuse. Do you think this should be possible?

Brandon Minnick · Answer 14 · Fri Jun 03 2022 03:29:04 GMT+0800 (China Standard Time)

Discussion from June Standup: https://www.youtube.com/watch?v=t3g_NrQfE8g

How do we set default values? Should we have different defaults for Mobile vs Desktop?
Are the values overridable?
Can we include BindableProperty that updates when the screen size changes?
- For example, on Android splitscreen or Windows desktop window sizing

Vladislav Antonyuk · Answer 15 · Fri Jun 03 2022 08:00:10 GMT+0800 (China Standard Time)

I propose to name it OnWindowsSize.

Because .NET MAUI supports windows
Because we layout components in Application, not in Screen. On phones, it makes sense because you cannot (almost) resize the app.

As for default values, I propose such options: https://mudblazor.com/features/breakpoints#breakpoints

As for the Window resize, we can implement some kind of this: https://mudblazor.com/components/hidden#listening-to-browser-window-resize-events

Ieuan Walker · Answer 16 · Fri Jun 03 2022 16:42:48 GMT+0800 (China Standard Time)

Not sure if its worth considering TV sizes too.

I.e. you may want to display things differently on a 55" tv compared to a 32" tv

Carol Zambon · Answer 17 · Fri Jun 03 2022 17:07:01 GMT+0800 (China Standard Time)

Hi @VladislavAntonyuk thanks for the info.

As for default values, I propose such options: https://mudblazor.com/features/breakpoints#breakpoints

This approach has some drawbacks:

1 - It is pixel-based, instead of "dp", or "dip" (Density-independent Pixels as used on Android) that could lead to issues like this. I know, i'm talking about "dp" which is not used on iPhones, Windows, mac, and etc, but wether we implement an algorithm similar to "dp" for all platforms we would avoid problems related to screen resolutions.

2 - They don't distinguish mobile phone sizes, they categorizes all mobile devices as ExtraSmall Devices, that could lead to problems, because we have many different mobile devices with many different sizes out there See here.

As for the Window resize, we can implement some kind of this: https://mudblazor.com/components/hidden#listening-to-browser-window-resize-events
Sorry but I didn't understand, maybe I missed something here.

As far as I know, markups in Xamarin/Maui cannot initiate UI events in which would force the Views that are using a markup to be also triggered/re-rendered .

Actually it is exactly the opposite. The markups are triggered by the Xaml (UI) during the rendering phase.

Please see here how a markup is created. They do not say that a markup can initiate UI events and that it could raise a Xaml (Views) changes by itself.

A Markup is much similar to a Converter.

Carol Zambon · Answer 18 · Fri Jun 03 2022 17:13:12 GMT+0800 (China Standard Time)

Hello @brminnick,

How do we set default values? Should we have different defaults for Mobile vs Desktop?

Are the values overridable?

We can change my implementation to maybe expose a couple of properties where the user could only sets their own values. On my current implementation this can be done by overriding methods of a class that is responsible for the categorization of devices. Please take a look here.

Can we include BindableProperty that updates when the screen size changes?

It is just like a regular Markup, so it already works on Bindable Properties.

Carol Zambon · Answer 19 · Fri Jun 03 2022 17:14:56 GMT+0800 (China Standard Time)

Removed, duplicate

Carol Zambon · Answer 20 · Fri Jun 03 2022 17:50:18 GMT+0800 (China Standard Time)

Hi @bijington,

Sorry I missed that this would be C# only.

No problem at all. :D

I guess this IMarkupExtension<T> implementation for C# could use the underlying IMarkupExtension implementation for XAML to aid with code reuse. Do you think this should be possible?

I'm almost sure of that.
My only concern regarding that is wether the user (Maui Community toolkit users who uses it) would need to explicitly declare a type (of a bindable property it is using the markup) in Xaml in the same way we need to do when using ArrayExtension. I would like to have something that could auto-infere the type. But I need to think better on how to do that (if possible).

developer9969 · Answer 21 · Fri Jun 03 2022 18:13:55 GMT+0800 (China Standard Time)

Just to say that this would be a fantastic addition - In the real world we have to support multiple device sizes - This is a real pain!!!
We usually test on iPhone SE 1st generation vs the iPhone max pro and similar for android, this would solve so much crappy code we had to write to handle both in a very clean fashion If does not cover .all scenarios you can think of, that is absolutely fine , its something that can evolve -otherwise it will never see the light of day, If it supports ios-Android only at the beginning it would be a fantastic start.

This should really be embedded in Maui but that will never happen in the short time.

Confused if this is approved already or not.. Cant wait to try it out.

prabhavmehra · Answer 22 · Wed Nov 22 2023 12:59:08 GMT+0800 (China Standard Time)

I propose to name it OnWindowsSize.

Because .NET MAUI supports windows

Because we layout components in Application, not in Screen. On phones, it makes sense because you cannot (almost) resize the app.

As for default values, I propose such options: https://mudblazor.com/features/breakpoints#breakpoints

As for the Window resize, we can implement some kind of this: https://mudblazor.com/components/hidden#listening-to-browser-window-resize-events

I am trying to have something similar but I am unable to get the resize events in IMarkupExtension, any suggestions?