Support view attributes

Question

Support view attributes

constantinpape opened this issue 4 years ago · comments

The bigdataviever attributes are stored in the metadata like this:

<ViewSetups>

  <Attributes name="channel">
    <Channel>
       <id>0</id>
       <name>0</name>
    </Channel>
    <Channel>
       <id>1</id>
       <name>1</name>
    </Channel>
  </Attributes>

  <Attributes name="illumination">
    <Illumination>
       <id>0</id>
       <name>0</name>
    </Illumination>
  </Attributes>
   
 <ViewSetup>
    <id>0</id>
    ....
   <attributes>
      <channel>0</channel>
      <illumination>0</illumination>
   </attributes>
  </ViewSetup>

 <ViewSetup>
    <id>1</id>
    ....
   <attributes>
      <channel>1</channel>
      <illumination>0</illumination>
   </attributes>
  </ViewSetup> 

</ViewSetups>

So there is a group of existing attributes specified by <Attributes name=...> and each ViewSetup then maps to the corresponding attributes in <attributes>.

In order to support this, I will add a dictionary argument attributes to make_bdv,
that maps the ViewSetup that is written to its attributes.
To be compatible to the old code, this will default to attributes={"channel": None}, which translates to: the attribute for this ViewSetup is channel and increase the counter by 1.

If a user wants to add custom attributes (i.e. other attributes than channel), the first call to make_bdv must have all the attribute names in the attributes dict that is passed.
Otherwise subsequent calls that introduce new attribute names will fail.

cc @martinschorb

Martin Schorb commented 4 years ago

exactly !

Martin Schorb · Answer 1 · Fri Mar 27 2020 18:58:55 GMT+0800 (China Standard Time)

As far as I know make_bdv does not report if a ViewID already exists in a data set (it would just append a "Channel").
I would like to be able to run a converter script multiple times without having to write the data over and over again if it is already present.
So if I call make_bdv with a set of attributes that already exists, the default should be to skip writing it. However, there should also be a possibility to overwrite an existing entry.

Could you implement this s well?

Constantin Pape · Answer 2 · Fri Mar 27 2020 20:24:09 GMT+0800 (China Standard Time)

As far as I know make_bdv does not report if a ViewID already exists in a data set (it would just append a "Channel").

No it does report it, and asks the user if the view-id should be over-written, see https://github.com/constantinpape/pybdv/blob/master/pybdv/converter.py#L38-L42.

I think what you propose is a better solution though:
Add argument overwrite=False to make_bdv, if False, just skip existing setup-ids, if True over-write them.

Would this work for you?

Constantin Pape · Answer 3 · Fri Mar 27 2020 20:53:09 GMT+0800 (China Standard Time)

exactly !

Great, will implement it and let you know once all is there.

Martin Schorb · Answer 4 · Tue Mar 31 2020 21:38:21 GMT+0800 (China Standard Time)

I have another one, quite important for multiple-view BDV:

displaysettings

I think your code as it is now should already support it. I will have a closer look and play a bit...

Constantin Pape · Answer 5 · Tue Mar 31 2020 21:41:38 GMT+0800 (China Standard Time)

I have another one, quite important for multiple-view BDV:

displaysettings

What do you mean by displaysettings? Is this part of the bdv.xml spec?

I think your code as it is now should already support it. I will have a closer look and play a bit...

Sounds good, let me know what you find.

Martin Schorb · Answer 6 · Fri Apr 03 2020 20:15:04 GMT+0800 (China Standard Time)

Hi Constantin,

displaysettings was implemented by Nico bigdataviewer/bigdataviewer-playground#111 (comment)

It basically stores the display state in BDV, so you can pre-load the contrast adjustment, LUT assignments etc. Probably there will be a few more attributes coming. So let's keep it flexible:

Could you simply make type-dependent assignments to the attribute tags?

int -> set the id and name as it is right now.
dict -> fill the attribute field with the tags defined in that dict.

I guess there should be easy ways to create xml fileds and tags from a dict...

So for displaysettings, I would provide: {"channel": 0, "tile": 1, "displaysettings": {"min": 55, "max":234}}

Martin Schorb · Answer 7 · Fri Apr 03 2020 22:38:57 GMT+0800 (China Standard Time)

And you would also need to change all functions in transformations.py to accept the setupID as a parameter to access the proper values.

This would be extremely helpful to be able just to update the transform and not write a new data container. Or, even better make_bdv's overwrite mode should by default just rewrite the xml entry for the ViewSetup not the data container.

Constantin Pape · Answer 8 · Sat Apr 04 2020 00:32:16 GMT+0800 (China Standard Time)

Could you simply make type-dependent assignments to the attribute tags?
* int -> set the id and name as it is right now.

* dict -> fill the attribute field with the tags defined in that dict.
I guess there should be easy ways to create xml fileds and tags from a dict...

So for displaysettings, I would provide: {"channel": 0, "tile": 1, "displaysettings": {"min": 55, "max":234}}

Yes, this is easy to implement, I will give it a go later.

This would be extremely helpful to be able just to update the transform and not write a new data container. Or, even better make_bdv's overwrite mode should by default just rewrite the xml entry for the ViewSetup not the data container.

I would rather provide a new function for this then change make_bdv; let's discuss this in person though.

Constantin Pape · Answer 9 · Sat Apr 04 2020 02:51:05 GMT+0800 (China Standard Time)

Ok, thought about this a bit more and it's a bit more tricky.
We need some way of uniquely identifying an attribute setup.
When we just pass an int (=id) this is easy.
If we pass full dicts and don't require any specific fields, this is more difficult.
I would opt for always requiring the dict to contain {'id': <some number>}

Constantin Pape · Answer 10 · Mon Apr 06 2020 21:20:54 GMT+0800 (China Standard Time)

Ok, will support passing dicts for attributes with two restrictions:

needs to contain the field id
cannot be nested, i.e. no other dicts or lists etc as values

Constantin Pape · Answer 11 · Sun Apr 12 2020 22:30:24 GMT+0800 (China Standard Time)

I have implemented this now in #15. However, I only allow passing attributes that are dict of dicts, e.g.

attributes = {'channel': {'id': 0, 'name': 'Channel1'}, 'displaysettings': {'id': 0, 'contrast_minimum': 0., ...}}

(Supporting both int and dict attributes made the code much more difficult to understand and is not necessary, I think).

@martinschorb could you give it a try and see if this works for you?

Martin Schorb · Answer 12 · Tue Apr 14 2020 17:29:59 GMT+0800 (China Standard Time)

hmm,

I give it:

In[8]: attributes
Out[8]: {'channel': {'id': 0}}

and end up with

     <Attributes name="channel">
        <Channel>
          <id>{'id': 0}</id>
          <name>{'id': 0}</name>
        </Channel>

then it crashes...


  File "C:\Software\Anaconda3\envs\pyEM\lib\site-packages\pybdv\converter.py", line 322, in make_bdv
    attributes_ = validate_attributes(xml_path, attributes, setup_id, overwrite_)

  File "C:\Software\Anaconda3\envs\pyEM\lib\site-packages\pybdv\metadata.py", line 371, in validate_attributes
    xml_ids = [int(child.find('id').text) for child in attribute]

  File "C:\Software\Anaconda3\envs\pyEM\lib\site-packages\pybdv\metadata.py", line 371, in <listcomp>
    xml_ids = [int(child.find('id').text) for child in attribute]

ValueError: invalid literal for int() with base 10: "{'id': 0}"

Martin Schorb · Answer 13 · Tue Apr 14 2020 18:57:24 GMT+0800 (China Standard Time)

Found the problem:

print(child.tag) for child in attribute
Channel

Capital C in Channel in the Attributes definition, whereas smallcaps channel in the ViewSetup attributes

Martin Schorb · Answer 14 · Tue Apr 14 2020 19:02:02 GMT+0800 (China Standard Time)

hmm OK, the 'name' attrib of the xml tag 'Attributes' should be correct (smallcaps)...
But the tag clearly isn't. So I guess that's why the find does not find anything...?

Constantin Pape · Answer 15 · Fri Apr 17 2020 14:52:29 GMT+0800 (China Standard Time)

@martinschorb sorry I don't quite understand what is not working for you.
I just tried this example:

import numpy as np
from pybdv import make_bdv
from pybdv.metadata import get_attributes

data = np.random.rand(1, 128, 128)
attrs1 = {'channel': {'id': 0}, 'tile': {'id': 0}}
make_bdv(data, 'test', attributes=attrs1)

data = np.random.rand(1, 128, 128)
attrs2 = {'channel': {'id': 1}, 'tile': {'id': 0}}
make_bdv(data, 'test', attributes=attrs2)

attrs_out1 = get_attributes('test.xml', 0)
print("Read 1:", attrs_out1)

attrs_out2 = get_attributes('test.xml', 1)
print("Read 2:", attrs_out2)

and it works fine.

Could you post a minimal example of the issue you are having please?

Martin Schorb · Answer 16 · Fri Apr 17 2020 17:27:04 GMT+0800 (China Standard Time)

Oh, I am calling make_bdv with overwrite = False in this case. Maybe this helps locating the problem. But this will change anyway with the new overwrite options.

Constantin Pape · Answer 17 · Fri Apr 17 2020 17:32:07 GMT+0800 (China Standard Time)

Oh, I am calling make_bdv with overwrite = False in this case

Ok, I think this explains it: You probably wrote the original file with an older pybdv version where the attributes were not properly supported yet. It does not over-write the attributes and so you have the broken xml.

For me, it works:

import h5py
from pybdv import make_bdv
from pybdv.metadata import get_attributes

with h5py.File('./fm_r1.h5', 'r') as f:
    data = f['t00000/s00/0/cells'][:]

make_bdv(data, 'martin', attributes={'channel': {'id': 0},
                                     'displaysettings': {'id': 0, 'alpha': 0.5}})
attrs = get_attributes('martin.xml', 0)
print(attrs)

But this will change anyway with the new overwrite options.

Yes, I will update the code for this now and let you know once it's there.

Martin Schorb · Answer 18 · Fri Apr 17 2020 17:36:18 GMT+0800 (China Standard Time)

It also happens when I delete all files and rewrite them...

Now even earlier (with the displaysettings)

fm_r1.zip

Constantin Pape · Answer 19 · Fri Apr 17 2020 18:09:01 GMT+0800 (China Standard Time)

Can you try with the code snippet I have posted above and can you make sure you are using the latest commit: 7d9f987

Martin Schorb · Answer 20 · Fri Apr 17 2020 18:12:48 GMT+0800 (China Standard Time)

Same problem


  File "<ipython-input-5-5ecb2a962fcb>", line 11, in <module>
    attrs = get_attributes('martin.xml', 0)

  File "C:\Software\Anaconda3\envs\pyEM\lib\site-packages\pybdv\metadata.py", line 455, in get_attributes
    attributes = {att.tag: int(att.text) for att in attributes}

  File "C:\Software\Anaconda3\envs\pyEM\lib\site-packages\pybdv\metadata.py", line 455, in <dictcomp>
    attributes = {att.tag: int(att.text) for att in attributes}

ValueError: invalid literal for int() with base 10: "{'id': 0}"

Martin Schorb · Answer 21 · Fri Apr 17 2020 18:14:43 GMT+0800 (China Standard Time)

If I install it through pip it should be the latest commit... right?

pip install git+https://github.com/constantinpape/pybdv

Constantin Pape · Answer 22 · Fri Apr 17 2020 18:16:19 GMT+0800 (China Standard Time)

I am never sure what pip does ....
Can you try the following:

clone the repository
navaigate into the root folder of the repo
run pip install -e .

This should install the latest version

Martin Schorb · Answer 23 · Fri Apr 17 2020 18:18:18 GMT+0800 (China Standard Time)

ran a manual install using python setup.py install

Same result

Martin Schorb · Answer 24 · Fri Apr 17 2020 18:20:37 GMT+0800 (China Standard Time)

OK

now it runs

Martin Schorb · Answer 25 · Fri Apr 17 2020 18:21:28 GMT+0800 (China Standard Time)

Also the initial make_bdv needs to be the most recent version.

I still had an old file in there...

Constantin Pape · Answer 26 · Fri Apr 17 2020 18:21:39 GMT+0800 (China Standard Time)

ran a manual install using python setup.py install

Yeah, that's even worse then pip :D

(Python installing is pretty broken ...)

Constantin Pape · Answer 27 · Fri Apr 17 2020 18:22:22 GMT+0800 (China Standard Time)

OK

now it runs

Ok, great :)

Also the initial make_bdv needs to be the most recent version.

I still had an old file in there...

Yes, this has all changed quite a bit recently.

Constantin Pape · Answer 28 · Fri Apr 17 2020 20:00:53 GMT+0800 (China Standard Time)

So I think this is working now :).
Feel free to reopen if any other issues come up.

Martin Schorb · Answer 29 · Tue Apr 21 2020 16:31:56 GMT+0800 (China Standard Time)

Hi,

I just realized that Displaysettings is capitalized in the attributes definition while in Nico's exported files it is in smallcaps.

bigdataviewer/bigdataviewer-playground#111

This is a problem with XML, correct?

Maybe you can align with him to make sure there is a consistent definition of the attributes (small/capitalized/...)

Constantin Pape · Answer 30 · Tue Apr 21 2020 17:06:13 GMT+0800 (China Standard Time)

I just realized that Displaysettings is capitalized in the attributes definition while in Nico's exported files it is in smallcaps.

bigdataviewer/bigdataviewer-playground#111

This is a problem with XML, correct?

This is just in the attrib (at least that's how it is called in python), so I am not really sure if this will cause any major issues. But it should still be done consistently!
I capitalize it, because that's the way it's done for all the other BDV attribute names.

I will comment in the playground issue to see if we can coordinate this.

Constantin Pape · Answer 31 · Tue Apr 21 2020 17:08:34 GMT+0800 (China Standard Time)

Sorry my bad, this is actually different for the field names.
So yes, this will def. cause incompatibilities in XML and we need to do it the same way.