Fall back to just audio or just video if simultaneous audio & video unavailable

[Duane Johnson]

I haven't tested that it's the case here, but in other libraries I've written, getting audio & video at the same time can result in people with just a mic on their device (or just a camera) being unable to participate.

I'll probably be playing around with media-devices a bit more here (nice lib!) but I just wanted to see if you'd considered this case already, or if you're looking for / would consider a PR.

[Jesse Gibson]

Oh, that's a good point. The apps I've worked on in the past have had strict requirements on mics/cameras so I've never encountered that in practice, but you're right, that's definitely an edge case. I'd be open to extending the API with graceful fallbacks, or maybe we could always execute those requests in parallel? That would avoid the fallback scenario. It could always follow the same code path:

const stream = new MediaStream()

// query audio/video separately, join both responses into a single stream.
const gumRequests = Object.keys(mediaConstraints).map(async mediaKind => {
  const mediaResponse = await getUserMedia({ [mediaKind]: mediaConstraints[mediaKind] })
  mediaResponse.getTracks().forEach(track => stream.addTrack(track))
})

await Promise.allSettled(gumRequests)

What do you think?

[Duane Johnson]

I like the parallel method, however, it has an unfortunate side-effect of making the user click through two permission dialogs instead of one (maybe on just some browsers? I've tested Chrome & Firefox).

So the general flow that seems to work is:

1. Try requesting both (video & audio)
2. If (1) succeeds, great! We have two tracks.
3. If (1) fails, try requesting just (video) & just (audio) in parallel

For #3:

• if both succeed: this is a weird condition that shouldn't happen, but accept it(?) & join the responses into a single stream
• if audio succeeds but video fails: we have just an audio track
• if video succeeds but audio fails: we have just a video track
• if both fail: throw an error

[Duane Johnson]

A simplifying assumption could be that the "weird condition that shouldn't happen", won't. If that's the case, this could be a reasonable method:

  async function ask({ audio = true, video = true }) {
    try {
      return await MediaDevices.getUserMedia({ audio, video });
    } catch (err) {
      if (audio && video) {
        return await ask({ audio: true, video: false });
      } else if (audio) {
        return await ask({ video: true, audio: false });
      } else {
        return null;
      }
    }
  }

[Jesse Gibson]

it has an unfortunate side-effect of making the user click through two permission dialogs instead of one

Ah, you're right! In that case, I really like the fallback plan you outlined in step 3. The only hesitation would be loss of error information from step 3, bullets 2 & 3: there's no way to determine why a track is missing (e.g. no camera, facingMode not satisfiable, camera already in use) and hence no way to report it to the user. Maybe it could be implemented as a new method? Or perhaps the method with fallbacks is the default, and the stricter implementation has a different name? (just thinking aloud)

await MediaDevices.withLibraryFallbacks({ audio: true, video: true }) // MediaStream [Audio]
await MediaDevices.withBrowserBehavior({ audio: true, video: true }) // NotFoundError!
//                 ^^^^^^^^^^^^^^^^^^^  terrible placeholder names

Eh, I don't know. If ask(...) throws the error given only audio OR video, then the information is still available to the developer, it just requires you query them separately. I could go either way 🤷

Would you be down to create a PR?

[Duane Johnson]

I ended up using that ask function I outlined, and it seemed like a reasonable thing for the user of media-devices to have to do, so now I'm a bit split on whether a PR would be needed. It might be a nice "best practice" to embed in the library. OTOH, it isn't strictly necessary (since it can be achieved as outlined, separate from the library).

[Jesse Gibson]

it seemed like a reasonable thing for the user of media-devices to have to do

Yeah, I'm leaning that way myself.

I'm a bit split on whether a PR would be needed

Let's leave it for now. I'm realizing that a fallback path could create a triple GUM prompt, which would be pretty surprising unless you knew the context. For a feature that's relatively small, I think it makes sense to leave it to the developer.

I'm gonna close this. Thanks for posing the question though, it was interesting to think about.