ABF1 file details differ between ABFInfo and pyABF

[Thomas Braun]

Looking at the abf1 file from https://github.com/youngseok-seo/NWB/blob/master/data/abf1/19122043.abf I get the following header output from pyabf
19122043.abf.zip.

The output from ABFInfo, https://moleculardevices.app.box.com/s/6pazy183dvq7yrxx5l6wx1sdxr4j57wf, is
abfinfo-output.txt.

Obvious differences:

Created: Jan 22, 2019, at 22:22:10.750 [10:56:42]
Created by: Clampex 9.2
Modified by: Clampex 9.2

vs

abfDateTime = 2019-09-25 16:34:59

which just looks like the file creation date.

$ ls -l data/abf1/19122043.abf
-rw-r--r-- 1 thomas 197121 6615552 Sep 25 16:34 data/abf1/19122043.abf

I can't seem to find the string "ClampEx" in the pyabf HTML file as well.

File GUID:       {EA774D20-4346-4219-ACCF-DB3B6DD89ABD}.

vs

fileGUID = 

And now a question :)
Where can I find nExperimentType = 1 in pyabf?

I need to figure out if I have resources to tackle that on my own.

[Scott W Harden]

Hi @t-b, thanks for looking into this! I'll look into these topics in more detail within the next few days. For now, here are some quick notes which you may find helpful.

Date calculation

ABF1 files don't store date information, so the file creation date is "invented" from the file creation time

pyABF/src/pyabf/abfHeader.py

Lines 193 to 196 in d91e6bc

 # format creation date based on when file was created 
 abfFilePath = fb.name 
 self.abfDateTime = round(os.path.getctime(abfFilePath)) 
 self.abfDateTime = datetime.datetime.fromtimestamp(self.abfDateTime) 

ABF2 files have uFileStartDate and uFileStartTimeMS header variables which are used to create the date like this

pyABF/src/pyabf/abfHeader.py

Lines 268 to 273 in d91e6bc

 # format creation date from values found in the header 
 startDate = str(self.uFileStartDate) 
 startTime = self.uFileStartTimeMS / 1000 
 startDate = datetime.datetime.strptime(startDate, "%Y%m%d") 
 timeStamp = startDate + datetime.timedelta(seconds=startTime) 
 self.abfDateTime = timeStamp 

It may be possible that the code in this block is incorrect.

GUID

ABF1 files don't contain a GUID.

ABF2 file GUIDs are created like this

pyABF/src/pyabf/abfHeader.py

Lines 260 to 266 in d91e6bc

 # format GUID 
 guid = [] 
 for i in [3, 2, 1, 0, 5, 4, 7, 6, 8, 9, 10, 11, 12, 13, 15, 15]: 
     guid.append("%.2X" % (self.uFileGUID[i])) 
 for i in [4, 7, 10, 13]: 
     guid.insert(i, "-") 
 self.sFileGUID = "{%s}" % ("".join(guid)) 

It may be possible that the code in this block is incorrect.

Access nExperimentType

abf = pyabf.ABF("test.abf")
print(abf._protocolSection.nExperimentType)

[Scott W Harden]

Hi @t-b, I somewhat misunderstood this issue in my original response. Most of the suggestions I gave only related to ABF2 files.

I have since added your example ABF1 file to the collection of demo ABFs. Its header values (as assessed by pyABF) can be easily viewed here: 19122043.md

My understanding before today was that ABF1 files do not contain acquisition date/time or GUID information. However, you found that the ABFInfo program is able to display these pieces of information from ABF1 files.

Finding file date in ABF1 files SOLVED

It looks like file time is a known integer at byte 24:

pyABF/src/pyabf/abfHeader.py

Line 128 in d91e6bc

 self.lFileStartTime = readStruct(fb, "i", 24) 

I'd bet the date, if there, is a few bytes in either direction.

Finding the GUID in ABF1 files SOLVED

The GUID object is 16 bytes. In ABF2 files I think it's located at byte 40. You could try checking out the ABF1 header with a hex editor and modifying bytes until you find the section that corresponds to the GUID. I'd guess it's within the first 100 bytes of the file.

EDIT: Here are the first few bytes of the GUID. I spot a "Clampex" too!

File GUID:       {EA774D20-4346-4219-ACCF-DB3B6DD89ABD}.

nExperimentType SOLVED

... I'd suggest the same hex editor technique for figuring out where nExperimentType is stored in ABF1 files, but this is not easy work! If you find a more comprehensive list of ABF file structure which may help avoid all this work, let me know =]

EDIT:

The nExperimentType is a short at byte 260

The struct map is a hint

The values don't always match the memory layout of structs in ABFHEADR.H, but they might be close! I'll bet the values you're looking for clump together.

[Thomas Braun]

@swharden Thanks for the quick response. I'll look into that.

[Scott W Harden]

ABF1 date time detection works now:

abf = pyabf.ABF(PATH_DATA+"/19122043.abf")
print(f"abfDateTime: {abf.abfDateTime}")

abfDateTime: 2019-01-22 22:22:10

GUID and nExperimentType remain unsolved...

[Scott W Harden]

GUID works in ABF1 files now too.

abf = pyabf.ABF(PATH_DATA+"/19122043.abf")
print(f"fileGUID: {abf.fileGUID}")

fileGUID: {EA774D20-4346-4219-ACCF-DB3B6DD8BDBD}

[Scott W Harden]

poking around in a hex editor I figured out nExperimentType is at byte 260 in ABF1 files. This now works:

abf = pyabf.ABF(PATH_DATA+"/19122043.abf")
print(f"nExperimentType: {abf._headerV1.nExperimentType}")

nExperimentType: 1

I think this addresses all of the issues you raised in your original issue. You can see how arduous this process is!

If you think additional variables are useful, let me know and I'll try to implement them. My guess is that the ones in pyABF now are are suitable for most users (and also that most people use ABF2 files these days too).

I'll publish this on pypi this weekend so these new features can be installed with pip

[Thomas Braun]

@swharden Thanks a lot for handling that so quickly. I've checked the changes and they do fix the issues I had.

[Scott W Harden]

Note that the GUID calculation has been incorrect by one byte this whole time. This has been fixed, and tests added comparing the GUID to that generated by ABFINFO.exe, but the GUID displayed by pyABF will change by two characters as a result. To make it easier to identify correct vs. incorrect GUIDs produced by pyABF, the Microsoft-style curly braces have been removed.

This is discussed in #95