compute_features error: 'Error computing transition bandwidth of the filter. Defined filter length may be too short.'
michellekloc opened this issue · comments
I am very interested in using bycycle to analyze my EEG data, but I am having an issue using compute_features and I was hoping for some advice about what I could be doing wrong. I keep receiving the following error message:
"raise warnings.warn('Error computing transition bandwidth of the filter. Defined filter length may be too short.')"
The error message points to 'filter_kwargs', which directs to narrowband_kwargs. I am using the following parameters for my data analysis, where the file name is csc1_low (lowpass_filter was applied at 30Hz):
Fs = 30000
f_range=(5,12)
narrowband_kwargs = {'N_seconds': x}
x=0.1,0.5,0.75,1,5,10,50,100,500 -> I have used trial and error to try and find the approximate range for a filter length but have thus far not been successful
I have also set narrowband_kwargs to be equal to N_cycles = for different values with the same error.
burst_kwargs = {'amplitude_fraction_threshold': .0001,
'amplitude_consistency_threshold': .0005,
'period_consistency_threshold': .0005,
'monotonicity_threshold': .0008,
'N_cycles_min': 5}
df = compute_features(csc1_low, Fs, f_range,
center_extrema='T',
burst_detection_method='cycles',
burst_detection_kwargs=burst_kwargs,
find_extrema_kwargs={'filter_kwargs': narrowband_kwargs},
hilbert_increase_N=True)
My burst_kwargs values change depending on whether or not I have scaled my data. It is currently in Volts, which is why the burst_kwarg values are so small. When I am scaled to mV or uV the thresholds are changed according to the y-axis of the filtered data, and I have been starting my analysis with the same values used in the examples as a starting point so that I can play with the values as needed. Unfortunately, I receive the above error message regardless of whether the data is in V, mV, or uV.
I have checked my data structure several times, and the data filters beautifully using lowpass_filter with Fs=30000, f_lowpass=30, N_seconds=.5; the issue I'm running into is only with compute_features. Is there anything obvious that I am missing here? Thank you so much for your help!
Thanks for posting the issue!
Sorry you've been having issues. I totally should make the transition band check optional (it's like that in neurodsp, but I neglected to change it here).
Can you tell me what you get for:
type(csc1_low)
len(csc1_low)
or if you can replicate this issue with a snippet of code that I can completely run myself (your snippet is almost there, just missing the data, mainly) , that would be useful too :)
Hi! Thanks so much for your response, I didn't realize you were still minding the tool :) I'm relieved that this might have a solution.
I get the following, which is consistent with all of my attempts. My files are quite a bit longer than anything you show in your examples, but I was hoping that wouldn't be a problem. I threw in .dtype and .shape for good measure, because all of this data is being converted from .mat and it has not brought me joy:
type(csc1_low)
<class 'numpy.ndarray'>
len(csc1_low)
14798848
csc1_low.dtype
dtype('float64')
csc1_low.shape
(14798848,)
I wish I could upload the .npy file here to send you some example data, which is annoying. Please let me know what you think, thank you so much for your attention to this!!!
Can you try replacing your data with just random numbers like
csc1_low = np.random.randn(14798848)
so we can see, like you said, if it's something weird in the data? (It's probably not... but is a quick easy check)
And if you could post a self-contained code snippet that reproduces the problem, that'll be easier for me to reproduce!
Hey Scott,
Thanks so much for that suggestion as a test, I should've thought to do so on my own. Using a random number generated array of that size threw the same error:
raise warnings.warn('Error computing transition bandwidth of the filter. Defined filter length may be too short.')
Here is what I ran:
import numpy as np
import matplotlib.pyplot as plt
import bycycle
from bycycle.features import compute_features
csc1_low = np.random.randn(14798848)
plt.plot(csc1_low)
Fs=30000
f_t=(5,12)
narrowband_kwargs = {'N_seconds': .75}
burst_kwargs = {'amplitude_fraction_threshold': .2,
'amplitude_consistency_threshold': .5,
'period_consistency_threshold': .5,
'monotonicity_threshold': .8,
'N_cycles_min': 5}
df = compute_features(csc1_low, Fs, f_t,
center_extrema='T',
burst_detection_method='cycles',
burst_detection_kwargs=burst_kwargs,
find_extrema_kwargs={'filter_kwargs': narrowband_kwargs},
hilbert_increase_N=True)
I increased the burst_kwargs values to suit the y-axis range of the randomly generated values (which is why I plotted the random data). I removed plt.show() from the snippet to avoid creating an unnecessary window. For N_seconds I tried an increasing series of values as I had done before - all of the filter lengths I defined kicked back the same error. I hope this is what you needed! Please let me know if not. Thank you again!!! I hope you have a fantastic weekend.
Thanks for providing the snippet!
The code that it's breaking on is to check that the narrowband filter (used for peak/trough localization) has reasonable cutoff frequencies.
It uses scipy.signal.freqz to estimate this. The default of freqz is to define the frequency response over 512 points. Because the sampling rate is so high here (30k), that means there is only 1 sample every 30000/512/2 Hz (29.3 Hz). However, we are trying to check the transition between 5 and 12 Hz (theta), so it breaks.
What should really be done is that the code should be changed so that we do not require checking for the transition band (there should be a kwarg like check_transition_band=True which the user can turn off with False.
It would also be good to build a work-around so that we can check the transition band of a 30kHz filter in the theta range. I think we should do this by setting the kwarg in freqz: worN=Fs*2, so that we have the frequency response defined at intervals of 1Hz.
Looking at neurodsp right now, it looks like this would also break. Someone should test filtering a signal in neurodsp at 30kHz in the theta range. I'll be surprised if someone hasn't done it already, so maybe it does work?
So @michellekloc , for you to get a solution quickly, I would use a local version of this repo, if you're not already (as opposed to using a pip installed version). And implement one or both of the changes I mentioned above.
It would be great if you could also submit these changes as a pull request too. Since I don't really feel like doing it, and idk who in the Voytek lab will, unless someone who is happening to follow this volunteers.
I realized the change would be super quick, so I implemented it in PR #35
Can you try cloning the repo version of the code and checking to make sure your code runs now? Your snippet worked for me (I decreased the signal length so it would take forever. If you want your code to run faster, I would expect your results wouldn't change much if you downsampled from 30kHz to 1kHz.
$ git clone https://github.com/bycycle-tools/bycycle
then in the bycycle directory:
$ pip install .
IT WORKED!!!!!!! I ran my data through and have a complete output :) :) :) I was steeling myself for a deep dive into the code on Monday, thank you SO much for implementing this change so quickly (and on a Saturday). I seriously appreciate your help with this hangup, I don't know how to thank you enough! I'm very excited to start combing through the data with this and seeing what comes out of it - myself and the rest of the lab were thrilled when we found out that a tool like this exists. I hope you have an excellent weekend!
glad it worked! I wish I would have developed it better in the first place so this problem wouldn't have existed, so happy to help!