Drawing HTML on top of a UIView is possible
telemakhos opened this issue · comments
Q: Can I place HTML elements (buttons and so on) on top of active
Is this really accurate?
Not sure if it'll work with a video layer, but for the Google Maps phonegap plugin drawing HTML on top of a native UIView is not a problem. Actually it works wonders...
So just passing by and putting this on your radar guys.
Hope it helps!
The aim of the cordova-plugin-iosrtc is to provide the WebRTC API without requiring custom stuff. If the video element is placed behind the Cordova HTML UIView and the background of the HTML view is not transparent, then the video would not be shown. How would that work if a <body> background is desired (for example a bitmap texture or a gradient)?
Said that, I may add an option to set this behavior per <video> element. Any suggestion? Perhaps a custom CSS field? a custom data-xxxx="true" attribute?
Not sure if it'll work with a video layer
Yes, sure.
How would that work if a background is desired (for example a bitmap texture or a gradient)?
Users can still set a desired background. Here's how the gmaps plugin solves the issue:
Yes, sure.
That's great news then. Might be worth exploring more in depth this option...
Any suggestion?
Not really, you know better than anyone what solution would fit better in your project and I just know the basic internals of the gmaps plugin. As a user I think that gmaps makes a great job providing a dummy transparent layer where you can draw freely any HTML element. That'd be undoubtedly the best solution since the users could draw a proper UI on top of the vid elements...
In any case I'm going to continue researching about this issue. I'll get back to you if I find anything worth looking at or if I have any interesting idea...
How about?
z-index: -1;
Using "z-index: -1" is what I consider a non appropriate usage of the CSS property. This is, setting "z-index: -1" may have very different results when in a web browser with native WebRTC support and when in an iOS Cordova device.
I would prefer having a custom data-xxxx attribute that would be ignored in the desktop browser or Android browser:
<video data-iosrtc-rear="true" autoplay></video>Thoughts?
We are already using the z-index in a non-appropriate way to place the videos relative to other videos, so " may have very different results when in a web browser with native WebRTC" is already happening.
Also, that way we ensure that the app has to put a transparent background in order the video to be seen.. ;)
The issue targeting CSS to only iOS (or web or Android) has several solutions. Ionic uses classes like…
.platform-browser video_div { z-index: x; }
.platform-android video_div { z-index: y; }
.platform-ios video_div { z-index: z; }
and also allows you to use dynamic targeted CSS using platform and ng-class (this is for angular but can be done with vanilla js).
The main issue here is abstracting everything and letting the users easily draw custom controls. I think this could be a very valuable differentiation feature for iosrtc. I’ve thought that an interesting option could be providing default CSS controls with the plugin (like the HTML5 video tag does) and allow the users to enable/customise them via a JSON object and overriding CSS properties… something like
{
"enableDefaultControls": true,
"controls": {
"startCall": true,
"stopCall": true,
"holdCall": true,
"fullScreen": true,
"switchCamera": false,
"mute": true,
etc...
}
}
Then provide default CSS classes (.stopCallButton, .startCallButton, .fullScreen, .fullScreenButton, etc) for both iOS and Android (xwalk):
SCSS example:
.fullScreenButton {
position: absolute;
bottom: 0;
float: right;
/* PNG Icon */
background-image: url('path/to/fullScreenButton.png');
background-repeat: no-repeat;
background-position: 25px 0px;
/* Scalable Vector Icon */
i {
font-size: 25px;
color: white;
vertical-align: middle;
}
/* Platform Specifics */
.platform-android {
// Android Stuff
}
.platform-ios {
// iOS Stuff
}
/* Modify appearance depending on the resolution */
@media (max-width: $screen-sm-max) {
display: none;
etc…
}
}
The defaults could result in a badass UI like:
This way the users would have a straightforward out-of-the-box solution to use controls on top of the video, and also a starting point with simple logic in case they want to go further customising their UI’s.
Just an idea...
I’ve thought that an interesting option could be providing default CSS controls with the plugin (like the HTML5 video tag does) and allow the users to enable/customise them via a JSON object and overriding CSS properties… [...]
Then provide default CSS classes (.stopCallButton, .startCallButton, .fullScreen, .fullScreenButton, etc) for both iOS and Android
There are no start/stop/hold/fullscreen "buttons" in the WebRTC spec, and there shouldn't be. That assumes "softphone vision" and WebRTC is beyond that.
I agree that rear video UIView is useful, but just that. It's up to the app/developer to place "control" buttons on top of the video view when needed.
Regarding the usage of CSS classes to tell the plugin whether to put the video UIView in front/rear, I don't like that approach. CSS classes are up to the application and they are supposed to provide custom style/design to the app. I'd rather custom data-xxxxx attributes as suggested above, as that is exactly the purpose of data-xxxx attributes: to add custom properties to the HTML element. Or the negative z-index...
Said that, I will implement the feature this morning XD
There are no start/stop/hold/fullscreen "buttons" in the WebRTC spec, and there shouldn't be. That assumes "softphone vision" and WebRTC is beyond that.
Yeah I know, I was just trying to get around the UI problem and the question of placing HTML on top. My suggestion was just an idea for convenience.
Said that, I will implement the feature this morning XD
Fair enough... 😁
Hi, first thanks for the wonderful work @ibc (and eface2face for supporting it), where's the BUY_ME_BEER.md in the repo? ;)
To shime on this issue (if I may):
- The "rear" video UIView was the first thing I personally looked for when reading through the docs yesterday. A gazillion +1 from me on this one I believe it's a fundemental thing to have.
- I feel everything-CSS that is otherwise for the purpose of demonstration (and you already have a an external demo repo) should remain outside the scope of this project. Sorry @telemakhos, nice screenshot though :)
- I personally perfer
data-xxxx, but that's just taste imho. If you already rely onz-index:-1kind-of-thingie then might just as well stick to that, both approaches are equally fine, as long as the functionality is there :)
Gload to see someone did actually nail this project for a change :)
I feel everything-CSS that is otherwise for the purpose of demonstration (and you already have a an external demo repo) should remain outside the scope of this project. Sorry @telemakhos, nice screenshot though :)
I didn't know about the demo app!... I guess that would be definitely a better option for the reference client with controls that I was suggesting.
Guys, I tried yesterday with zero success. Don't know why but I was not be able to place the video UIView behind the web view and make the web view transparent. I also had issues when using the zPosition attribute of the layer video layer.
I will try it again next week.
No worries. Are you trying to replicate the gmaps technique stacking the subviews programatically or are you trying to come up with a custom solution?
Let's say I must expend more time looking into phonegap-googlemaps-plugin...
This would be super useful. Right now I've had to manually hide the video view when using the phonertc plugin whenever I wanted to render anything over the video.
It would be really useful, but my time is limited. PR are welcome.
@ibc I could try to help on that issue. This is one of the few issues I have to fix before using your project. Did you actually create a branch and started working on it?
Thanks
I started working on it with no success. I will upload a branch on Monday and notify it here.
Thanks a lot.
I've upload a new branch "rear-video" to implement this feature. For now it has the same content as master since nothing that I tried did work so I've removed it (it was useless).
Notes and problems I remember:
- My idea is that a negative z-index makes the video element
UIViewto be placed behind the HTML view (the CordovaUIView). - Stuff must be done in
src/MediaStreamRenderer.swift. - Note that I use two
UIView:elementView: It has same size as its corresponding HTML<video>element.videoView: It is placed over theelementView. Its purpose is to have a different size (which is cropped by the parent view) so CSSobject-fitcan be implemented (already done).
- So we must work on the
elementViewleaving thevideoViewuntouched. - In my attempts I was not able to place the
elementViewbehind the Cordova view (it became invisible even if the HTML has no background color, not sure if something else must be done in the HTMLUIView). - I expect issues due the usage of
self.elementView.layer.zPosition = CGFloat(zIndex)since it seems thatzPositionmay alter how different layers are rendered. Currently if a negativez-indexis set for the video element, theelementViewjust disappears.
Hope somebody can help on this :)
If so, please fork the "rear-video" branch and let work on it.
This is the last outstanding issue preventing us from being able to use this in production. If you need anything at all, I'm down to help.
How to help is explained in my previous comment.
Hi IBC, in previous projects, we implemented an RTMP (flash) Video component for iOS as a native video component (without Cordova though), which runs in the background of a WebView component.
With transparent body settings it was possible to use it in the background and it worked pretty well vor us.
We want to move away from flash now by using WebRTC for future projects. I don't know if I can help you with this, but all we had to do in our project was this:
- (void)viewDidLoad
{
/*** other stuff ***/
// set WebView Transparent
webView.opaque = NO;
webView.backgroundColor = [UIColor clearColor];
}
In the HTML, i used
body {
background-color: transparent
}
If you knew this already, sorry for bothering you ;) I just came along this thread and thought this could help
@ibc Did you ever get the chance to look into derMani's workaround?
Not yet, sorry.
Maybe this can work: Include a local HTTP server in the webRTC plugin to serve the video content to the video tag - this can hopefully allow both overlayed HTML elements along with full CSS styling control. We used this technique when we created an RTMP plugin...
@acylum That would produce a really uncomfortable delay. Video player in browsers (including Safari) do cache.
This is in the works, so far the video successfully playing the background (HTML body must have transparent background), z-index support with negative values will be completed this week so that multiple videos (UIViews) are stacked on top of each other in order. Anyone that wishes to help testing this prior to sending a PR please feel free to PM me.
Sweet! I'd like to test it. Currently, my work around is to launch a wizviewmanager child window over the main view - so basically the webrtc video is "sandwiched" in-between. You have to use the window messaging to allow button interaction from wizview back to your main view.
@acylum awesome thanks! just replied you :)
+1
+1 are good, but pull requests are even better :)
@ibc I have this working in my fork, see my last commit here: https://github.com/1N50MN14/cordova-plugin-iosrtc/commit/30ccdaa583b1f33f790bb8db3a5789fcdd8088e4 I'm not able to send a PR just yet, but feel free to take it for a spin in the meantime
Great! Let me handle that on next week.
Take your time
Hey guys, just wanted to know if you have an update on this.
@derMani I submitted a PR for this one, waiting for ibc to look into it and eventually approve it, otherwise you can use my fork for now.
Guys, I won't be able to merge the PR until the second week of the new year.
@ibc No worries, Merry Christmas! ;)
Same! :)
@derMani Du bist willkommen und Frohes Neues Jahr :)
@telemakhos Yup I can make that work once @saghul submits his PR for WKWebView support
Hi, I'm sorry I couldn't properly check the status of the PR #98. I commented some issues regarding the PR (for example: /lib/libWebRTC-LATEST-Universal-Release.a can NOT be removed from the repo) and I'm not sure whether that PR handles all the required use cases without breaking working code (in fact I've read some users reporting wrong behavior).
So, if somebody can spend time to test #98 and provide proper conclusions I will merge it.
Sorry to bother you again @ibc, but are there any news regarding this issue?
Not directly related to this repo, but a plugin I work on just released support for rendering the camera preview layer behind a transparent UIWebView or WKWebView: bitpay/cordova-plugin-qrscanner
It's just a simple plugin, but might help if someone is trying to implement the same thing here: https://github.com/bitpay/cordova-plugin-qrscanner/blob/master/src/ios/QRScanner.swift
I think the status regarding this PR is already explained in comments above, but I'll try to summarize here:
- The author of the PR has already said that he's working in his own project so it's expected that he's no longer maintaining the code in this PR.
- There are also comments from some users that tested the PR and they reported some issues in some cases.
- I don't have time to test all the required use-cases nor to look at how other projects implement a similar feature.
So, if someone wants to contribute, take this PR, apply it into a fork of the official master branch of the plugin, check all the use-cases and document them, then I'll be very happy to accept a new PR.
Just upgraded to 3.0.0 of the Plugin and the PR still works for me.
The steps are basically the same as before (and suggested in the PR)
- Hide Video Controls for video element in Javascript
- Edit PluginMediaStreamRenderer.swift to make sure the webview is transparent.
I think I could provide a small example project, if you wish.
@ibc What exactly are "all use cases" ? :-)
I just created a web page, made sure everything had the correct CSS and after that I was able to see the video through my transparent elements.
Maybe that's the problem for some people here, you have to make everything transparent on your page, if you want to look through it. It's easy to forget about a few elements, which might still have a background-color (like body, html, ...)
Actually is not working for me.
Using 3.0.0;
Having all transparant. (verified several times).
body {
background-color: transparent;
}
html {
background-color: transparent;
}
.pane, .view {
background-color: transparent;
}
[nav-view-transition="ios"][nav-view-direction="forward"], [nav-view-transition="ios"][nav-view-direction="back"] {
background-color: transparent;
}
#myMiniVideo {
width:30%;
height:30%;
position:absolute;
right:0;
bottom:0;
z-index:0;
}
#videoscreen .video_controls {
position:absolute;
right:50px;
bottom:50px;
z-index:15;
}
Video overlays the .video_controls... :(
no matter what z-index the myMiniVideo have. When setting -1 z-index to myMiniVideo, the video is not visible.
Changes in PluginMediaStreamRenderer.swift :
// The browser HTML view.
self.webView = webView
self.eventListener = eventListener
self.webView.opaque = false
self.webView.backgroundColor = UIColor.clearColor()
// The video element view.
self.elementView = UIView()
// The effective video view in which the the video stream is shown.
// It's placed over the elementView.
self.videoView = RTCEAGLVideoView()self.webView.superview?.addSubview(self.elementView) self.elementView.userInteractionEnabled = false self.elementView.hidden = true self.elementView.backgroundColor = UIColor.clearColor() self.elementView.addSubview(self.videoView) self.elementView.layer.masksToBounds = true self.videoView.userInteractionEnabled = false
What exactly are "all use cases" ? :-)
I just created a web page, made sure everything had the correct CSS and after that I was able to see the video through my transparent elements.
I mean how it works. Which values do z-index accept and what happens if z-index is negative. And what happens if a <video> is placed on top of another <video> with different z-index, etc.
To be clear: I don't have an iOS device so I won't check all the cases, that's why I need complete documentation regarding how this PR works.
BTW as you can see, a comment after your says that it fails. If I accept this PR as it is, I'll become responsible of such a problem.
Done in #179
Is there anyway to redraw the rendering video onto a HTML5 canvas?
Is it possible to hide the video tag from view? This way I imagine it would be possible to show the video in canvas while the original video is playing in the background, hidden from the viewport