Add support for reading bcf file containing EBSD dataset
minhtamhoang opened this issue · comments
Hello,
I am trying to load bruker bcf file, but I got errors as attached below:
ERROR:hyperspy.io:If this file format is supported, please report this error to the HyperSpy developers.
Traceback (most recent call last):
File "EBSD_analysis.py", line 4, in
file = hs.load("APMT_200x_EBSD_2x4_indent_single_indent_top4.bcf")
File "/anaconda3/lib/python3.7/site-packages/hyperspy/io.py", line 467, in load
for filename in filenames]
File "/anaconda3/lib/python3.7/site-packages/hyperspy/io.py", line 467, in
for filename in filenames]
File "/anaconda3/lib/python3.7/site-packages/hyperspy/io.py", line 525, in load_single_file
return load_with_reader(filename=filename, reader=reader, **kwds)
File "/anaconda3/lib/python3.7/site-packages/hyperspy/io.py", line 545, in load_with_reader
file_data_list = reader.file_reader(filename, **kwds)
File "/anaconda3/lib/python3.7/site-packages/hyperspy/io_plugins/bruker.py", line 1247, in file_reader
return bcf_reader(filename, *args, **kwds)
File "/anaconda3/lib/python3.7/site-packages/hyperspy/io_plugins/bruker.py", line 1281, in bcf_reader
obj_bcf = BCF_reader(filename, instrument=instrument)
File "/anaconda3/lib/python3.7/site-packages/hyperspy/io_plugins/bruker.py", line 913, in init
header_file = self.get_file('EDSDatabase/HeaderData')
File "/anaconda3/lib/python3.7/site-packages/hyperspy/io_plugins/bruker.py", line 451, in get_file
item = item[i]
KeyError: 'EDSDatabase'
How can I resolve these errors?
Thanks,
Hi @freshcrystal,
HyperSpy cannot ready EBSD patterns from Brukers BCF files. Nor can kikuchipy, extending HS for EBSD analysis. We can however read Bruker's HDF5 EBSD format. I've read about a BCF->HDF5 converter script provided by Bruker, can you use this?
Thanks for your response. Can you tell where I can find the conversion script? Thanks.
I do not, unfortunately.
It is already documented in http://hyperspy.org/hyperspy-doc/current/user_guide/io.html#bruker-s-formats (and https://hyperspy.org/rosettasciio/supported_formats/bruker.html#bruker-format) but it would be good to tweak the error message and also to make the comment in the dosctring a note.
Hello, I had not implemented EBSD reading, as HyperSpy has no EBSD signal, so the mapping of metadata is not clear. bcf is container (think about it like zip or rar file, just different format (AidAim SingleFileSystem)), and reading the internal files is implemented. AFAIK and had looked into this, bcf's generated from EBSD workflow contains readable xml and binary files. Structures of these needs to be reverse engineered, albeit brief skimming over data looked for me that it would not be very challenging (no challenge compared to what I had reverse engineered hitherto). XML content also needs mapping to some other expected data structures. If there would be anyone who would provide me with sensible EBSD datasets, and would lead with metadata mapping requirements, I would gladly implement this functionality.
BTW, implementing the EBSD reading would improve coverage of bruker._api as it contains some code already in place which is not being tested due to file tables (of max 4 files in those bcf of EDS only) are not crossing block boundaries.
during EMAS workshop I had discussed this a bit. This needs to be implemented. Albeit I also realized that I considerably am very bad in EBSD. I mean I more less understand how it works, but I am completely ignorant which pieces of preserved information is important and how it should be mapped to (what is just metadata, and what is data).
Are you aware that there is a .bcf -> .emd converter provided by Thermo-Fisher?
In the main folder of Velox start SIConverter.exe
The .emd files created by the converter are however currently not supported by HSpy. Supporting those might probably be a sortcut for the implementation of a .bcf reader.
Maybe useful for people trying to get their EBSD data from Bruker bcf: I wrote a guide how one can use Mike Jackson's bcf2hdf5 program to extract the patterns into an hdf5: https://lukmuk.github.io//posts/2024/05/blog-ebsd-bcf/
This worked for EBSD maps acquired with Esprit 2.1.5 (with the "Save raw patterns" option during saving of the bcf). The bcf2hdf5.exe on the latest release is not up to date with the repo, hence the direct compiling of the code.
The generated hdf5 can be opened, cleaned up and saved as a HyperSpy/Kikichypy hdf5, which I want to share as a notebook soon...
I hope it helps!
Edit: The converter notebook I currently use is here: https://github.com/lukmuk/ebsd-processing-bruker/tree/main/kikuchipy-bcf2hdf5-converter
This should be useful, until there is a better way to open the data in python, which should be much easier to install and run.
For anyone interested to do the implementation, the source code is at https://github.com/BlueQuartzSoftware/BCFTools. The readme says that the SFS Reader has been inspired by the bcf reader of rosettasciio (formelly hyperspy of course!), so there should be some similarities between the rosettasciio code and bcftools.
On a side note, it would be nice to see rosettasciio be inspired by code which was in part inspired by rosettasciio! 😄
Maybe useful for people trying to get their EBSD data from Bruker bcf: I wrote a guide how one can use Mike Jackson's bcf2hdf5 program to extract the patterns into an hdf5: https://lukmuk.github.io//posts/2024/05/blog-ebsd-bcf/
This worked for EBSD maps acquired with Esprit 2.1.5 (with the "Save raw patterns" option during saving of the bcf). The bcf2hdf5.exe on the latest release is not up to date with the repo, hence the direct compiling of the code.
The generated hdf5 can be opened, cleaned up and saved as a HyperSpy/Kikichypy hdf5, which I want to share as a notebook soon...
I hope it helps!
Edit: The converter notebook I currently use is here: https://github.com/lukmuk/ebsd-processing-bruker/tree/main/kikuchipy-bcf2hdf5-converter
Hello, I tried to install this bcf2hdf5 on a Linux system. But the compiling stopped at 51% and complained something. Any idea to resolve this? Thank you.
@yuanh9, you need to ask at https://github.com/BlueQuartzSoftware/BCFTools - this repository is for RosettaSciIO.
@yuanh9 I think we initially had the same issue when compiling on Ubuntu WSL (I think 22.04) and Mike Jackson from BCFTools fixed it actually in this commit: BlueQuartzSoftware/BCFTools@369becc
I may need to check also my guide but can you make sure that you cloned the latest development branch from BCFTools? I hope it works.
@lukmuk I followed your guide and cloned https://github.com/BlueQuartzSoftware/BCFTools.git.
@yuanh9 Compiling worked on my PC with Pop_OS 22.04 (ignoring all the warnings) following the commands on the blog post, so it should work at least with Ubuntu 22.04. I could not test the compiled executable, since I don't have a bcf at hand right now. Please see attached screenshot for the package versions and kernel.
If it is still not working, please consider opening an issue on the BCFTools repository.