mxochicale / RATCHET

RAdiological Text Captioning for Human Examined Thoraxes

RATCHET: RAdiological Text Captioning for Human Examined Thoraxes

RATCHET is a Medical Transformer for Chest X-ray Diagnosis and Reporting. Based on the architecture featured in Attention Is All You Need. This network is trained and validated on the MIMIC-CXR v2.0.0 dataset.

Architecture

Run the code

1. Download pretrained weights, unzip them and put them in ./checkpoints folder.

• ratchet_model_weights_202303111506.zip
  Size: 1.5G
  MD5: 26ab19cf18908841320205e192dabe9f
  

2. Start streamlit to run the webapp:

streamlit run web_demo.py

Environment:

Python 3.9.10

Packages:

imageio                  2.26.0
matplotlib               3.7.1
numpy                    1.23.5
pandas                   1.5.3
scikit-image             0.20.0
streamlit                1.20.0
tensorflow               2.11.0
tokenizers               0.13.2
tqdm                     4.64.1

Docker Container

Build the docker container:

docker build -t ratchet ./Dockerfile

Run the docker image on CXR images:

docker run --user $(id -u):$(id -g) \
-v /path/to/image_input_folder:/code/RATCHET/inp_folder \
-v /path/to/report_output_folder:/code/RATCHET/out_folder:rw \
-i -t ratchet python run_model.py

Each image in inp_folder would have a corresponding .txt report saved in out_folder.

Results

                   

Generated Text:

In comparison with the study of ___, there is little overall change. Again there is substantial enlargement of the cardiac silhouette with a dual-channel pacer device in place. No evidence of vascular congestion or acute focal pneumonia. Blunting of the costophrenic angles is again seen.

More Examples

References

Hou, Benjamin, Georgios Kaissis, Ronald M. Summers, and Bernhard Kainz. "Ratchet: Medical transformer for chest x-ray diagnosis and reporting." In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VII 24, pp. 293-303. Springer International Publishing, 2021. https://arxiv.org/abs/2107.02104 google-scholar: https://scholar.google.com/scholar?cites=6324608147072853701