Support raw AAC in HLS

[Joey Parrish]

Originally raised in #887 (comment)

Our HLS parser must guess container and codecs from the limited information provided in an HLS playlist. Currently, we don't understand raw AAC.

MediaSource seems to support raw AAC with the MIME type "audio/aac" (no codecs parameter), so we should be able to handle this.

Sample streams:
https://content.jwplatform.com/manifests/vM7nH0Kl.m3u8

[Joey Parrish]

Recognizing raw AAC is easy, but given the new timestamp parsing code we are using to fix #1011, we still won't be able to fully support raw AAC until we have a parser for it.

[Álvaro Velad Galván]

Hi @joeyparrish,

With this patch should be enough?

diff --git a/lib/hls/hls_parser.js b/lib/hls/hls_parser.js
index b684d280..602afa05 100644
--- a/lib/hls/hls_parser.js
+++ b/lib/hls/hls_parser.js
@@ -1372,6 +1372,8 @@ shaka.hls.HlsParser.prototype.getStartTime_ =
       // There is no standard way to embed a timestamp in an mp3 file, so the
       // start time is presumably 0.
       return 0;
+    } else if (mimeType == 'audio/aac') {
+      return 0
     } else if (mimeType == 'video/mp2t') {
       return this.getStartTimeFromTsSegment_(responses[0].data);
     } else if (mimeType == 'application/mp4' ||
@@ -1867,6 +1869,7 @@ shaka.hls.HlsParser.AUDIO_EXTENSIONS_TO_MIME_TYPES_ = {
   'm4s': 'audio/mp4',
   'm4i': 'audio/mp4',
   'm4a': 'audio/mp4',
+  'aac': 'audio/aac',
   // mpeg2 ts aslo uses video/ for audio: http://goo.gl/tYHXiS
   'ts': 'video/mp2t'
 };

[media-square]

Is there an update concerning this bug, since we are facing the same issue?

[Joey Parrish]

@media-square, no update. We have not worked on this at all yet.

Re: the patch suggested by @avelad, no, that would not work for live streams as far as I can tell.

[Brian Folts]

Will the work involved here be a function to getStartTimeFromAacSegment_ implemented to parse the aac segment and pull it's start time?

[Joey Parrish]

Yes, that's the idea.

[media-square]

So basically, there is a solution which should work for live streams but...
When can we expect a patch or will this issue not being fixed?

[Erik Živković]

We are interested in this patch as well

[James Peet]

+1

[ismena]

@bes @peet-j Thanks for expressing interest!

@joeyparrish Could you, please, advise if this is on our road map?

[Joey Parrish]

As far as I can tell, containerless formats like MP3 and raw AAC do not have their own timestamps. So long as the video and audio content are properly aligned and the content is VOD, we could just use the patch from @avelad. But I don't think there are any timestamps to parse out, so we're left with guessing at "0".

I'm not certain what will happen with that patch and any non-zero-start-time content or any live content, though. What happens for raw AAC in live depends on how MediaSource behaves with timestamp-less raw content. I'm reaching out to the Chrome team for clarification, but I suspect this will not work yet for live.

[Joey Parrish]

I heard from Chrome team. The way AAC and MP3 work in MediaSource is that timestampOffset determines the starting time of each new append in the SourceBuffer, and after each append, the browser updates timestampOffset to reflect the amount of data that was appended. This is 'sequence' mode, even if you didn't set it explicitly on the SourceBuffer.

If we had the right metadata, we could make that work, including for live. But it will require refactoring in our HLS implementation. That refactor would overlap significantly with #1558.

[Kevin Scroggins]

Hi @joeyparrish, are there any updates on when this feature might be implemented? Thanks!

[JPeMu]

I happened to see this issue in passing. Just to correct the previous comment about containerless formats not having timestamps - this isn't true. The HLS spec requires that all packed audio formats (such as raw AAC/MP3 etc) have an ID3 PRIV tag inserted at the very start with the timestamp information.

Please see: https://tools.ietf.org/html/draft-pantos-http-live-streaming-23#section-3.4

This is a "MUST" item in the spec, and it goes so far as to add that clients "SHOULD NOT" play back audio data like this unless it contains this timestamp!

BR
Tony

[Joey Parrish]

It turns out that the timestamps in the HLS parser are only a small part of the problem of supporting raw AAC. The bigger issue is support for containerless formats generally at the MediaSource level. Every time we clear the buffer or seek, the timestampOffset needs to be set before appending containerless segments.

I filed issue #2337 to track full containerless format support. I will close this issue as soon as the HLS parser has been taught to recognize raw AAC. Until we can play it correctly, though (#2337), we will reject raw AAC HLS content.

[Joey Parrish]

The change to explicitly reject raw AAC has been cherry-picked for v2.5.8.