Passing VP9 to ffmpeg

[Patryk Rogalski]

Hi!

How can I pass vp9 stream to ffmpeg?

Initially, I started working from saving to webm example with VP8. I modified the code to send samples to ffmpeg and it worked. Then I wanted to try with VP9 but I must be doing something wrong because ffmpeg decoder is unable to decode these samples and I'm only getting the following error "Not all references are available".

Here is my code for VP9:

sampleBuilder := samplebuilder.New(10, &codecs.VP9Packet{})
		var videoTimestamp uint32

		for !isCanceled {
			packet := <- c.rtpChan
			sampleBuilder.Push(packet)

			sample := sampleBuilder.Pop()
			if sample == nil {
				continue
			}

			vp9Packet := codecs.VP9Packet{}
			if _, err := vp9Packet.Unmarshal(packet.Payload); err != nil {
				panic(err)
			}

			videoKeyframe := !vp9Packet.P
			videoTimestamp += sample.Samples

			if videoKeyframe {
				width := int(vp9Packet.Width[0])
				height := int(vp9Packet.Height[0])
				c.writeInputInfoToMixer(inputCodecs.VP9, int32(width), int32(height))
			}

			if c.sentFirstKeyFrameToMixer || videoKeyframe {
				c.writeSampleToMixer(videoTimestamp, videoKeyframe, sample.Data)
			}
		}

and the same for VP8:

sampleBuilder := samplebuilder.New(10, &codecs.VP8Packet{})
		var videoTimestamp uint32

		for !isCanceled {
			sampleBuilder.Push(<- c.rtpChan)

			sample := sampleBuilder.Pop()
			if sample == nil {
				continue
			}

			videoTimestamp += sample.Samples
			videoKeyframe := sample.Data[0]&0x1 == 0
			if videoKeyframe {
				raw := uint(sample.Data[6]) | uint(sample.Data[7])<<8 | uint(sample.Data[8])<<16 | uint(sample.Data[9])<<24
				width := int(raw & 0x3FFF)
				height := int((raw >> 16) & 0x3FFF)

				c.writeInputInfoToMixer(inputCodecs.VP8, int32(width), int32(height))
			}

			if c.sentFirstKeyFrameToMixer || videoKeyframe {
				c.writeSampleToMixer(videoTimestamp, videoKeyframe, sample.Data)
			}
		}

Any idea what I'm doing wrong?

[Patryk Rogalski]

Solved

[Sean DuBois]

@digitalix Glad to hear you got it working! Would you be able to share how you did it/open a PR? Just so other people don't have to deal with the same issue :)

thanks!

[Patryk Rogalski]

@Sean-Der thanks!
Here is some background:

Well, turns out I was looking at the wrong part of my system. The code above is completely fine and the problem was located in my FFmpeg part of things. My project is divided into two parts WebRTC server (pion) and a video mixer (ffmpeg with bindings in nodejs (beamcoder)). In the mixing part, the first frames that I receive would produce "Not all references are available" and at the time I was just stopping the mixing - thinking that it was a problem with my go code.

I'm not posting any code since it's not related to pion.

In case if anyone is interested in learning more feel free to contact me.

[Sean DuBois]

Ah I understand, sounds like a really cool project.

If there is anything I can do to make Pion better reach out. I love seeing people making things with it.

[Patryk Rogalski]

@Sean-Der

Actually, I was wrong - there is a problem with sample builder or at least the part that I made to detect keyframes for VP9. The code above worked only so long as there was only one video feed to the mixer after the 2nd client joined the input was ignored by ffmpeg and the final video would freeze.
After some consulting with someone why knows a lot about video and ffmpeg he suggested that it might be an issue with keyframes, at first I was sceptical but it made a lot of sense. A few hours later I found an alternative implementation of sample builder here: https://github.com/hakobera/go-webrtc-decoder/blob/master/decoder/framebuilder.go. Using this implementation solved the problem which means that the previous code would generate false positives (I'm not sure but that's how it looks).

Maybe it would be worth to redo the sample builder to be more like the one from the link above.

Here is my updated code:

sampleBuilder := NewFrameBuilder(10, &codecs.VP9Packet{}, &codecs.VP9PartitionHeadChecker{})
		var videoTimestamp uint32
		for !isCanceled {
			packet := <- c.rtpChan
			sampleBuilder.Push(packet)

			sample := sampleBuilder.Pop()
			if sample == nil {
				continue
			}

			var a []byte
			for _, p := range sample.Packets {
				packet := p.(codecs.VP9Packet)
				a = append(a, packet.Payload...)
			}

			videoKeyframe := !sample.Packets[0].(codecs.VP9Packet).P
			videoTimestamp = sample.Timestamp

			if videoKeyframe {
				width := 0
				height := 0
				c.writeInputInfoToMixer(inputCodecs.VP9, int32(width), int32(height))
			}

			if c.sentFirstKeyFrameToMixer || videoKeyframe {
				c.writeFrameToMixer(videoTimestamp, videoKeyframe, a, frameType.FULL)
			}

And lastly, I really love Pion! I spent 2-3 weeks exploring what’s out there and this project is perfect for what we need.

Keep up the good work!