Unusual time stamps with AudioCaptureWin and Brain Products RDA

Question

Unusual time stamps with AudioCaptureWin and Brain Products RDA

aypaek3 opened this issue 6 years ago · comments

Hello!

We're working on a study related brain rhythms and speech.
We are collecting EEG data with Brain Products BrainAmps Hardware and audio through a USB microphone. For the EEG, we are using the "BrainVision RDA" app and for the audio, we're using the "audiocapturewin" app. We are also using the keyboard app to save markers.

We have tried to use LabRecorder to save the data, and after using the import_xdf function in MATLAB, we noticed a very large discrepancy between the EEG and the audio recordings. The actual data for the EEG and audio seem to be intact.

We collected some data with subjects and noticed this behavior, and I also collected a small ~15 second data segment. EEG time stamps appear to make sense where the time stamps have a 15 second range, but the audio time stamps only cover about 0.3 seconds of time and there is a a large 80 second offset between the audio time stamps and the EEG time stamps.

Is there a way to correct the audio time stamps?

Attached is a zipfile containint the XDF file of the 15 second segment.
EEGandAudioTest.zip

Is it possible to assume that the first sample of the EEG and audio files are synchronized? We have already collected a few datasets with a few subjects and it would be great if they were still usable.

Thank you,

Chadwick Boulay · Answer 1 · Sun Jul 08 2018 11:15:03 GMT+0800 (China Standard Time)

I didn't write the app and I don't have an appropriate setup to test it, but looking at the source code I can offer some tips.
Debug and make sure that the reported sampling rate is correct in pwfx->nSamplesPerSec here.
The timing code is here. I recommend putting a break point on the call to push_chunk (currently line 216) and then inspecting the variables above it to make sure that now, firstSampleAge, lastSampleAge, lastSampleTime, and numFramesAvailable all make sense.

David Medine · Answer 2 · Mon Jul 09 2018 16:33:54 GMT+0800 (China Standard Time)

Something funny is going on here. I loaded the data without jitter correction and plotted the time stamp series. It looks like audio capture win is marking timestamps completely wrong. The plot should be a line, not a sawtooth! This means that the timestamps are getting marked backwards in time. I have no idea how this could have happened. One suggestion is to lower your audio sampling rate. Writing all this data (1 channel audio at 44.1 kHz + 64 channels EEG at 500 Hz) on your laptop might be overwhelming the audio system somehow. You can set this in the windows audio settings. Have you checked the audio samples themselves? I.e. does it sound correct? In anycase, somewhere between windows' sound layer (which is quite horrible) and AudioCaptureWin (which can work quite well under ideal conditions on Windows 7, at least) there is a problem. Unfortunately, if all your data have this problem there is little hope of synchronizing it exactly. It will never be the case that the first sample of one stream recorded by labrecorder corresponds to the first sample of another. They are normally roughly the same, but each stream gets recorded on an independent thread so they start recording at random times. I attach the analysis script I wrote to look at the data for reference. s_corr = load_xdf('EEGandAudioTest.xdf'); %% s = load_xdf('EEGandAudioTest.xdf', 'HandleClockSynchronization', false, 'HandleJitterRemoval', false); %% for n=1:length(s) if(strcmp(s{n}.info.type, 'EEG')==1) eeg = s{n}.time_series; eeg_ts = s{n}.time_stamps; end if(strcmp(s{n}.info.type, 'Audio')==1) audio = s{n}.time_series; audio_ts = s{n}.time_stamps; end end %% figure plot(audio_ts) ylabel('time stamp values'); xlabel('sample number'); %% figure plot(eeg_ts, eeg(1,:)); %% figure plot(audio_ts, audio(1,:)); %% figure plot(diff(audio_ts(1,:))); ylabel('ts_diff (s)'); ylabel('ts diff (s)'); xlabel('sample number');

aypaek3 · Answer 3 · Tue Jul 10 2018 00:04:20 GMT+0800 (China Standard Time)

Hello again.

We checked the EEG and audio files, and the signals make sense. EEG signals look like EEG, and the audio signal comes out normal too. The length of the files are roughly the same time-wise, so it didn't seem like we lost any data. Our only issue is that the time stamps don't seem to be make sense for the audio.

We discovered that this problem seems to manifest when we used a USB microphone.
https://www.amazon.com/Microphone-ELEGIANT-Condenser-Recording-YouTube/dp/B075XNQ8JW

We tried additional recordings with another microphone that uses a simple pink audio input jack and that seems to yield sensible time stamps. Using the built-in microphone in the laptop also yields sensible time-stamps. We'll try to do further recordings with this microphone.

It would be great if we could salvage the data that we collected before. Is there a sense of the delay that would be present between the EEG and the audio recordings? I understand that the delay wouldn't be consistent, but if it's within ~100 ms perhaps the data is still usable. If the delay could be on the order of seconds, we might just drop the data. Of course, we'll try to validate this ourselves.

If there are any other ideas that would help, that would be great.

Chadwick, we'll try playing with the source code and see if that helps with this particular microphone.

Thank you,

Matthew Grivich · Answer 4 · Tue Jul 10 2018 06:19:52 GMT+0800 (China Standard Time)

I concur with the previous posts that the audio timestamps are garbage. I don't know why, and would be difficult to figure out without having the same hardware setup (at least the microphone) on hand.

It would be great if we could salvage the data that we collected before. Is there a sense of the delay that would be present between the EEG and the audio recordings? I understand that the delay wouldn't be consistent, but if it's within ~100 ms perhaps the data is still usable. If the delay could be on the order of seconds, we might just drop the data. Of course, we'll try to validate this ourselves.

It kinda depends on how desperate you are. Did the audio data pick up the keyboard presses? If yes, you can use those as reference points to manually rescale the timestamps. The audio data should be a fairly stable frequency, so you can do linear interpolation/extrapolation. The audio data sounds like real data to me (though of course I don't know what I'm listening to).

For a less accurate measure, you can assume that the BrainVision and the Audio data start and end at the same time, but I'm not sure how accurate that would be. I'd guess between 50 ms and 1 second error. You could test to determine the typical error (do multiple tests, it may vary).

Tristan Stenner · Answer 5 · Tue Jul 10 2018 14:45:19 GMT+0800 (China Standard Time)

It kinda depends on how desperate you are. Did the audio data pick up the keyboard presses? If yes, you can use those as reference points to manually rescale the timestamps.

Or if you have a trusty lab physicist connect a clicky switch between two EEG electrodes and apply a tiny voltage to pick up both the click on the microphone and in the EEG recording.

I've also started a Qt port some time ago (repository, Windows binaries) but I haven't gotten around to test the timing yet.

David Medine · Answer 6 · Tue Jul 10 2018 16:14:55 GMT+0800 (China Standard Time)

This issue is most certainly caused by this particular setup. Your Windows machine hates that microphone for whatever reason. As far as salvaging the timestamps, you can't really be sure when the audio vs eeg recordings begin. I would note the difference in first timestamps between eeg and audio where those times aren't completely mangled by the usb mic. That will give an average time between stream acquisition and hopefully it will be within 100ms.