Add sound capability

[bruno]

Hello,

I am trying to get sound capabilities from the NDI stream, but all I can hear are intermittent short samples noise. Can you help me figure out why?

NdiReceiver.cs

void Update()
    {
        RenderTexture DecodedFrame = TryReceiveFrame();
        if (DecodedFrame == null) return;

        // Material property override
        if (targetRenderer != null)
        {
            targetRenderer.GetPropertyBlock(_override);
            _override.SetTexture(targetMaterialProperty, DecodedFrame);
            targetRenderer.SetPropertyBlock(_override);
        }
        
        // External texture update
        if (targetTexture != null) Graphics.Blit(DecodedFrame, targetTexture);
        
        AudioClip DecodedAudioFrame = TryReceiveAudioFrame();
        if (DecodedAudioFrame == null) return;

        //audio update
        AudioSource speaker = GetComponent<AudioSource>();
        if (speaker.clip != DecodedAudioFrame)
        {
            speaker.clip = DecodedAudioFrame;
            speaker.Play();
        }

    }
      AudioClip TryReceiveAudioFrame()
      {
          PrepareReceiverObjects();
          if (_recv == null) return null;
      
          // Audio frame capturing
          AudioFrame? audio_frameOrNull = RecvHelper.TryCaptureAudioFrame(_recv);
          if (audio_frameOrNull == null) return null;
          AudioFrame audio_frame = (Interop.AudioFrame)audio_frameOrNull;
      
      
          //decode audio data
          int arrayLength = audio_frame.no_samples*audio_frame.no_channels;
          last_frame_audio_data = new float[arrayLength];
          Marshal.Copy(audio_frame.p_data, last_frame_audio_data, 0, arrayLength);
      
          AudioClip audio_clip = AudioClip.Create($"AudioClip NDI {ndiName}", audio_frame.no_samples, audio_frame.no_channels,
              audio_frame.sample_rate, false); 
      
          //set data
          audio_clip.SetData(last_frame_audio_data, 0); 
      
          // Audio frame release
          _recv.FreeAudioFrame(audio_frame);

          return audio_clip;
      }

RecvHelper.cs

    public static Interop.AudioFrame? TryCaptureAudioFrame(Interop.Recv recv)
    {
        Interop.VideoFrame video;
        Interop.AudioFrame audio;
        Interop.MetadataFrame metadata;
        var type = recv.Capture(out video, out audio, out metadata, 0);
        if (type != Interop.FrameType.Audio) return null;
        return (Interop.AudioFrame?)audio;
    }

Recv.cs

    [DllImport(Config.DllName, EntryPoint = "NDIlib_recv_capture_v3")]
    static extern FrameType _Capture
      (Recv recv, out VideoFrame video,
          out AudioFrame audio, out MetadataFrame metadata, uint timeout);

[Keijiro Takahashi]

You can't implement the audio capability in such a way. You should buffer the incoming data and stream it to the audio device. You'll get noise if the buffer is too short, but it introduces latency when it's too long. It's pretty hard to determine the proper balance. You'll also consider the audio buffer size in Unity and the device driver (OS). That's why I haven't implemented it.

Please refer the following thread for further discussion: #17