Accelerating HLS via CDN?
user080975 opened this issue · comments
Hi,
Is it possible to use a CDN to accelerate or cache live streams proxied by HLS proxy? I tried pointing a CDN to the server I've installed HLS-Proxy on and tried accessing the new link but I'm unable to access it somehow.
If it's possible, what would be the steps to get CDN working?
Thank You!
not sure that I understand the purpose for doing so..
you say "accelerate", but that doesn't make any sense..
since pushing video segments to a CDN takes time..
and is only meaningful if the actual purpose was to copy a live stream for distribution to a wide audience for vod (video on demand).
- is it possible? yes
- would need a client to re-request the HLS manifest peridically, in the same way that a video player does
- best to use custom hooks to implement specialized logic
- is it a very unusual use-case? yes
- are there better tools for this purpose? definitely
again, I'm not 100% sure of what you mean..
if by "delay", you mean "time to first byte", then there's nothing you can do.. since you can't speed up the video host server. And, of course, by adding a proxy (ie: man in the middle).. you're slowing down the speed to first byte, rather than speeding it up.
if by "delay", you mean that there's stutter during playback because there's a delay between when the current video segment finishes playback and when the next video segment has been downloaded (by the video player) and is available for playback.. creating a noticeable break in the video playback, then using the "prefetch" option of the proxy should be able to help. There are caveats to this, such as the server being configured to return "HTTP 429 Too Many Requests" for a very brief period of time following each response.. which I've observed in the wild.. which can be configured incorrectly.. such that when the video player re-requests the manifest to obtain the next batch of video segments in anticipation that the currently playing batch of video segments are nearly finished playing.. the server refuses for too long.. and forces this type of "delay".. in which case, there's nothing to be done.. since you can't reconfigure the video host server.
I've attached a screen recording below:
https://we.tl/t-ooUJte3KAg
In the recording, you can see that the when loading the proxied M3U8 link, the .ts file takes several seconds to load. This occurs each time the link is played, and sometimes the delay can be 6 - 8 seconds.
Would be possible to cache this .ts file periodically using a CDN so that the loading time for the .ts file is reduced? This is what I'm trying to achieve.
I just watched the video that shows the network waterfall diagram in Chrome DevTools..
presumably as you load an HLS video into an HTML5 video player via the HLS Proxy.
Your performance looks pretty great to me..
- 81.4 ms to complete the download of the m3u8 manifest
- 1.85 sec (average) to complete the download of each of 3x ts video segments (sequentially)
- each of which is approx 10MB and probably contains about 10 secs of video
I'm not sure if you've configured the proxy to prefetch all of the ts video segments in the m3u8 manifest,
which could protentially cause the time to download the first ts video segment to increase slightly,
since the proxy immediately begins to download --max-segments in parallel,
which (depending on your network) could slow how long it takes to proxy the first one,
but.. based on your numbers, there was no such slow down; everything looks good to me.
Which aspect of this waterfall do you think is experiencing a "delay"?
I've attached a screenshot of the relevant portion of the final video frame
(as context for anybody else who may subsequently read this conversation)..
I cheated, and used an online download time calculator..
You're downloading the ts video segments at approx 43 Mbit/sec.
My neighborhood is waiting (like Godot) for fiber.. still stuck on DSL.
Even that speed would be a big improvement for me.
Yay!.. American infrastructure..
I've attached a screen recording of what currently happens when proxied streams are loaded:
https://we.tl/t-ZurgOjWZ5w
As you can see from the video, there's about a five second delay before the video starts to play, where the first .ts file needs to be loaded. Even with the prefetch option turned on, I've noticed that when accessing the same link but from a different device or just via 4G, this delay still occurs.
If the prefetch and cache options are turned on, in theory shouldn't accessing the same link but via different devices no longer require five seconds to load the initial .ts file, since it's already cached?
What I was referring to "accelerating via CDN", is that would it be possible to store this initial .ts file in a CDN, rather than fetching it from my origin server each time, since user's locations won't always be near to my origin server.
Storing on CDN means that user's get the cached .ts file from their nearest CDN node rather than waiting to fetch from my server.
These screenshots are illustrative.. and helpful to visualize the expected behavior.
The earlier screenshot shows how a video player would load an HLS stream through the proxy without prefetching/caching any of the ts video segments.
This 2nd screenshot (taken from the final frame of the 2nd video):
..illustrates exactly what I described in an earlier response:
I'm not sure if you've configured the proxy to prefetch all of the ts video segments in the m3u8 manifest,
which could protentially cause the time to download the first ts video segment to increase slightly,
since the proxy immediately begins to download --max-segments in parallel,
which (depending on your network) could slow how long it takes to proxy the first one
so.. this 2nd waterfall diagram shows:
- 127 ms to complete the download of the m3u8 manifest
- 5 sec latency until the first byte of the 1st ts video segment is received from the proxy
- 275 ms (average) to complete the download of each of 3x ts video segments (sequentially)
- each of which is approx 10MB
- these files previously took 1.85 sec (average) without using prefetch/caching on the proxy
How long these 3x ts video segments remain cached on the proxy depends upon the value of --max-segments.
While they remain cached, then any additional clients that request these same 3x ts video segments from the HLS proxy will download them directly from RAM used by the proxy.. and the response will take very little time (ex: 275 ms).
Regarding using a CDN..
- live video streams would be unavailable until the ts video segments that you publish to the CDN become fully available
- this seems very unrealistic
- video on demand streams are static files and can be hosted anywhere
- if you want to publish them to a broad audience, then a CDN makes sense
- likewise, peer-to-peer is also a good strategy to geographically distribute the file hosting
- both considerations are completely outside the scope of this utility.. and are very edge use-cases, but doable.. I suppose
This is my settings for the proxy:
hlsd --host mydomain.com --port 8080 --prefetch --cache-timeout 3600 --max-segments 100
When using the same client / browser to access the target link after the first attempt, the loading speed is indeed very fast.
But when I access the same target link with a different client on a different network with comparable speeds, immediately after the above step, then it still takes about five seconds for the video to load.
Isn't the second client also supposed to receive the video very fast since it's already cached in RAM?
If both clients are using a single/common proxy server, then identical requests will be processed in an identical way. The proxy is stateless; there are no cookies or sessions.
Things to consider:
- does the URL for the m3u8 include any querystring parameters that differ between clients (ex: tokens, expiration, timestamps, etc)?
- likewise, do the URLs for the ts video segments within the m3u8 manifest change in any way each time that the m3u8 is generated by the origin server?
- of course, a live stream will constantly be adding new ts video segments, and removing old ones
- what I mean is.. for any particular ts video segment.. is its URL static every time it's included in the manifest?
PS: --max-segments 100 when each ts video segment is 10MB means that your cache can grow to use 1GB of RAM.. for each unique m3u8 manifest URL
I checked and the parameters and URL remain exactly the same each time. Since I'm using the proxied link, shouldn't the same link be pushed for both clients?
If I would like to push the proxy stream to a CDN, is there any modification to the HLS Proxy code that I need to make or is just a server configuration?
hmm..
--cache-timeout 3600means that your cache will remain in RAM for 1 hour of inactivity/nonuse before being garbage collected.. so presumably that won't be influencing your testing of 2x clients- since you're using the exact same URL to request content from the proxy.. then the base64 encoded URL for the m3u8 manifest on the origin server is static.. and doesn't change when requested by the clients
- I'd need to inspect the network traffic to figure out what's actually going on
In any case, an improvement that I could make to the proxy would be..
- when the cache (for a manifest) is empty and the proxy is about to perform its first initial prefetch..
- rather than initiating a parallel download of several segments (ie: up to
--max-segments)..- only download the 1st segment..
- when that download is complete, then initiate a parallel download of the remainder
This would allow the first chunk to download at full speed (rather than sharing bandwidth with a ton of other simultaneous download requests), and return it to the video player much faster.
Using the numbers from your 2x examples above:
- 81.4 ms to complete the download of the m3u8 manifest
- 1.85 sec to complete the download of the 1st ts video segment
- approx 3 sec latency until the remaining 2x ts video segments are available from the proxy
- 275 ms (average) to complete the download of each of 2x ts video segments (sequentially)
fwiw, personally.. some initial latency doesn't bother me;
my primary concern (as a user) is that once the stream begins to play.. that its playback is smooth and stutter-free.
to answer your question..
If I would like to push the proxy stream to a CDN, is there any modification to the HLS Proxy code that I need to make or is just a server configuration?
seems to me that:
- you'd make an .m3u8 manifest that is for vod, that uses relative URLs to reference each of the .ts video segments
- better to avoid using any subdirectories.. and just keep all files in the same bucket
- a good convention is to give the .ts video segments filenames that are sequentially numbered
- you'd push all files to the CDN
- 1x .m3u8
- many .ts
- you'd access the .m3u8 manifest through the CDN
- this is your "origin" server
- you can proxy the URL for the .m3u8 hosted by the CDN through the HLS Proxy
Though, to be clear.. I wouldn't recommend doing this.
The CDN is a server-side solution to improve the stream.
The HLS Proxy is a client-side solution to improve the stream.
Only one solution should be needed.
I just took a look at the code.. and implemented the improvement that I speculated about (above);
it's published to npm in v0.20.0.
I'd be curious to see your waterfall diagram loading the same live stream with prefetch (as the earlier screenshot)..
using this updated version.. for comparison sake.
I updated to the latest version and tried with this command:
hlsd --host mydomain.com --port 80 --prefetch --cache-timeout 3600 --max-segments 10
It seems that the prefetch option isn't doing anything anymore. Previously after the first initial playback, any subsequent reloads on the same client results in the video playing also immediately.
But in the new screen recording attached below, each time the page is reloaded, the video doesn't play immediately. Is this normal?
https://we.tl/t-6BMhpYQfjC
could you try this:
hlsd --host mydomain.com --port 80 --prefetch --cache-timeout 3600 --max-segments 10 --cache-key 2 -v 3 >log.txt 2>&1
..and attach a copy of log.txt
I did test this update.. and confirmed that it works precisely as desired (by inspecting terminal output in combination with a few additional logging statements). So.. to really see what's going on with your sample stream.. I need to inspect a log file.
oh.. and to keep the size of your log file minimal.. please only load your stream once.. and allow it to run for about 45 seconds before killing the process. thanks.
Okay here is the link to the log file:
https://we.tl/t-b5lCsY8ijH
summary:
proxying: http://111.21.23.62/PLTV/88888893/224/3221226984/index.m3u8?icpid=88888893&from=1&hms_devid=463,939
redirecting: http://111.21.23.62/PLTV/88888893/224/3221226984/1665292368-1-1662772222.hls.ts
redirecting: http://111.21.23.62/PLTV/88888893/224/3221226984/1665292378-1-1662772223.hls.ts
redirecting: http://111.21.23.62/PLTV/88888893/224/3221226984/1665292388-1-1662772224.hls.ts
prefetch (start): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292368-1-1662772222.hls.ts
proxying: http://111.21.23.62/PLTV/88888893/224/3221226984/index.m3u8?icpid=88888893&from=1&hms_devid=463,939
redirecting: http://111.21.23.62/PLTV/88888893/224/3221226984/1665292368-1-1662772222.hls.ts
redirecting: http://111.21.23.62/PLTV/88888893/224/3221226984/1665292378-1-1662772223.hls.ts
redirecting: http://111.21.23.62/PLTV/88888893/224/3221226984/1665292388-1-1662772224.hls.ts
prefetch (start): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292378-1-1662772223.hls.ts
prefetch (start): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292388-1-1662772224.hls.ts
cache (pending prefetch): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292368-1-1662772222.hls.ts
cache (callback added): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292368-1-1662772222.hls.ts
prefetch (complete, 8467144 bytes): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292378-1-1662772223.hls.ts
prefetch (complete, 8540464 bytes): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292388-1-1662772224.hls.ts
prefetch (complete, 8754032 bytes): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292368-1-1662772222.hls.ts
cache (callback complete): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292368-1-1662772222.hls.ts
cache (hit): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292368-1-1662772222.hls.ts
cache (hit): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292378-1-1662772223.hls.ts
cache (hit): http://111.21.23.62/PLTV/88888893/224/3221226984/1665292388-1-1662772224.hls.ts
notes:
- the m3u8 manifest is requested repeatedly..
- which is typical for a live stream, but normally the video segments in the 1st response are played before the next subsequent request
- in this case, the first 2x requests are performed without any such delay
- because the first 2x m3u8 manifest requests happen at nearly the same time, they both contain the same 3x ts video segments
- which for the sake of brevity, I'll refer to as: 2.ts, 3.ts, and 4.ts
- the 1st m3u8 manifest request:
- causes 2.ts to prefetch
- queues 3.ts and 4.ts to prefetch once 2.ts is downloaded
- the 2nd m3u8 manifest request:
- causes 3.ts and 4.ts to prefetch immedietely
- effectively causing all 3x ts video segments to download in parallel
- of the 3x ts video segments, 2.ts happens to be the last to finish downloading
- once this 1st ts video segment is available, the remaining 2x ts video segments are also readily available
takeaway:
- had your video player not requested the m3u8 manifest twice when the stream initially started playing, then prefetch would've allocated more bandwidth toward downloading the 1st ts video segment
- since it did what it did, the new strategy was effectively nullified.. and all ts video segments were downloaded in parallel
I’m using Safari on Mac OS. I just paste the m3u8 link directly in browser and let it play. Does that mean using Safari and playing the proxied steam nullifies the cache?
I'm not an Apple/Mac user.. and I wasn't aware that Safari natively supports playing HLS streams; that's actually pretty nifty. That being said, my comments directly apply to their implementation.. which is problematic (for the sake of our demo).
In the browser, I always test with Clappr in Chrome. Specifically, ..here.
PS: to be clear, what I meant by "the new strategy was effectively nullified".. is that in this log.. the cache behaves the same as it did in v0.19.0 release. I wasn't saying that the cache was in any way disabled or not working. I only meant that download of the 1st ts video segment isn't accelerated.. as was the intended purpose for the v0.20.0 release.
Sorry for the late reply, is there anyway to push the proxied stream to an rtmp server?