Running inference for report reconstruction?
peiqi-sui opened this issue · comments
Really interesting work! I'm trying to test the effectiveness of PRIOR's report reconstruction on some clinical data, and I'm wondering if there's an efficient way to run inference without making significant changes to the repo?
Thank you so much for your help!!
Thanks for your attention. I will try my best to push these codes in Dec once I have finished my recent workload. It is worth noting that the "report reconstruction" only reconstructs the high-level prototypes rather than the linguistic reports. We only can visualize the prototype distribution over the memory bank.
It means the scores without Gumbel softmax in this line
https://github.com/QtacierP/PRIOR/blob/086bf2e0d379d7dacc47a6ad738ffabc1ad6e25f/codes/prior/modules/spb.py#L29C9-L29C45
Thank you for clarifying. This is the sentence from the paper that threw me off: "each sentence embedding is updated by a learnable sentence-wise prototype memory bank (SPB) for the final linguistic representation". It left me with the impression that the reports are being updated through the prototype memory bank.
In that case, it'd be amazing if you could make it possible to run inference on subregion-sentence alignment at some point. Much appreciated!
Right~ Aligning fine-grained features is our ultimate goal. To be honest, the results of VLP models in this regard are not yet promising for clinical applications. However, they do have the potential to enhance the explanations provided by these models. I will re-organize the related codes in the next update.