Request for Actiware 6.0 csv file compatibility
mcmahonmc opened this issue · comments
Hi Gregory,
I stopped by your demo at OHBM and was so impressed by your work on this! Unfortunately I don't think it's compatible with the files we have. We use Actiware 6.0 which exports individual files as csv files and I haven't been able to get the files to read in. I attached an example of one of our data files (GitHub won't let me upload csv files). Could you let me know if I'm missing something? If not, would you consider adding compatibility with this file type?
Hi Megan,
thank you for the message. Glad to know that the package can be useful for others.
Concerning your file, the latest release does contain an improvement in the handling of csv files from Actiwatch 2 (Respironics, Inc.) . Is this the device you use? If so, your file does not look like the ones I had access to, so far. There usually is a "Subject properties" section followed by a "Actiwatch Data Properties" section (that contains the sampling frequency, for example) before the actual "Epoch-by-epoch" section" (that I see in your screen-shot). Did you edit the file by hand? Maybe it is an option during the export process from Actiware.
If you have a raw file, anonymized, don't hesitate to send it to me: gregory.hammad@hotmail.fr
Hope to hear from you soon.
Best regards,
Grégory
HI Megan,
for the records, I continue our discussion here.
To summarize, for RPX files to be read by the pyActigrpahy package, it is required that the files contain the following sections;
- "Subject properties" (contains the 'name' of the pyActigraphy object to be created)
- "Actiwatch Data Properties" (contains the acquisition start time and frequency, as well as the device id).
- "Epoch-by-epoch" (contains the actual activity data... so most likely it is there by default).
In the file you then sent me, with the necessary sections, there was a new issue;
- sometimes, on Windows system, for a reason unknown to me, the 'newline' character that marks the end of a line (useful when parsing the file), '\r\n', is re-interpreted as '\n', which in turns is converted to '\r\r\n' when the file is written to disk. Making the file full of ill-formatted newline characters...
I modified the rpx reader to deal with this kind of situation.
@mcmahonmc : could you try to read your files with the latest version of the code?
git clone git@github.com:ghammad/pyActigraphy.git
cd pyActigraphy/
git checkout develop
pip install -e .
Hi Grégory,
Thank you!
Our lab only has the Actiware software for Windows so unfortunately I am unable to produce an exported data file on a Mac for comparison purposes.
I checked out the development branch and attempted to read a file, but got the following error:
In [2]: raw = pyActigraphy.io.read_raw_rpx('30426_3_26_2018_6_10_00_PM_New_Analy
...: sis.csv')
ValueError Traceback (most recent call last)
in
----> 1 raw = pyActigraphy.io.read_raw_rpx('30426_3_26_2018_6_10_00_PM_New_Analysis.csv')
~/Applications/pyActigraphy/pyActigraphy/io/rpx/rpx.py in read_raw_rpx(input_fname, language, start_time, period, data_dtype, light_dtype, delimiter)
284 data_dtype=data_dtype,
285 light_dtype=light_dtype,
--> 286 delimiter=delimiter
287 )
~/Applications/pyActigraphy/pyActigraphy/io/rpx/rpx.py in init(self, input_fname, language, start_time, period, data_dtype, light_dtype, delimiter)
105 uuid = self.__extract_rpx_uuid(header, delimiter)
106 start = self.__extract_rpx_start_time(header, delimiter)
--> 107 frequency = self.__extract_rpx_frequency(header, delimiter)
108 axial_mode = 'Unknown'
109
~/Applications/pyActigraphy/pyActigraphy/io/rpx/rpx.py in __extract_rpx_frequency(self, header, delimiter)
214 int(re.sub(r'([^\s\w])+', '', line.split(delimiter)[1])
215 .replace('\xa0', ' ').strip()),
--> 216 unit='second'
217 )
218 break
pandas/_libs/tslibs/timedeltas.pyx in pandas._libs.tslibs.timedeltas.Timedelta.new()
pandas/_libs/tslibs/timedeltas.pyx in pandas._libs.tslibs.timedeltas.convert_to_timedelta64()
pandas/_libs/tslibs/timedeltas.pyx in pandas._libs.tslibs.timedeltas.cast_from_unit()
ValueError: cannot cast unit second
Hi @mcmahonmc
does this file differ from the previous one?
I did check the parsing of the header (retrieving the name, the acquisition frequency, etc) and it works.
Would you mind to send me this new file so that I can take a look?
Thank you.
Grégory
Hi Megan,
"the plot thickens"... I develop and test pyActigraphy on Mac and Linux... And, just to be extra sure, I retested the latest version of the pyActigraphy (currently the head of the develop branch) and it reads your file, both on Mac OS (Sierra, 10.12.6) and Linux (Ubuntu 16.04). I'll try on Windows as soon as I can get my hand on a Windows machine... (Might take a while, I've never worked on Windows machine before).
Concerning the read_raw method, to read multiple files at once, depending of the reader_type input value, it uses one of the read_raw_XXX methods. So the fix for the read_raw_rpx method is automatically propagated. To be sure, I also just tested it and with reader_type='RPX', it works fine with your file.
Could you give me access to your '30426' file so that I can check what the problem is?
Thank you for your feedback and help.
Regards,
Grégory
Hi @mcmahonmc
I had a look at your new file and it seems identical (from the format point of view) to the first one.
And I am able to read and analyse these files with pyActigraphy, both on Mac OS and Linux Ubuntu. A check with a Windows machine would be useful too but I don't have access to such machine at the moment. I'll try to do this asap.
In the meantime, I'm glad you have found a way to read your files with pyActigraphy. If you have questions/suggestions when you start the analysis, do hesitate to contact me.
Good luck.
Grégory
Hi @mcmahonmc
If you still have any problem, trying to read your data files, please feel free to re-open this issue or fill a new one.
Regards,
Grégory