aman-sawarn / Medical-Report-Generation-Using-X-Ray-Images

Taking X-Ray images and generating reports on it

Medical-Report-Generation-Using-X-Ray-Images

Overview

The aim of this project is to generate medical reports from X-ray images. This problem holds a great role when there is a lack of Quality doctors in remote parts of the world, and just lab technicians could take an X-Ray and send it to doctors anywhere across the globe. It would bring in quality healthcare services at cheaper rates to parts of world where doctors would take weeks to reach. This is my final year Major Project.

Data Source: http://academictorrents.com/details/66450ba52ba3f83fbf82ef9c91f2bde0e845aba9

How Data Looks?

Sample Outputs

Notebooks Layout

1. References have the references used
2. Workflow has notebooks which were used for experiments and tuning purposes
3. Loss plots has different plots for training and validation purposes
4. Training has various training notebooks
5. Outputs have output generated from the model
6. Checkpoints has model weights
7. Tokenizer has a .pickle file containing key value pairs from the tokenizer's vocabulary

Folders Layout

1. Outputs- Has sample outputs
2. Tokenizer- Has Tokenizer weights stored in .pickle file
3. Workflow- Has all experiments while before reaching the final model
4. checkpoints_2/train-Has all encoder decoder layer weights
5. training- Has notebook training notebooks and evaluation using BLEU and Rouge scores
6. losses_plots- Has all training and evaluation plots while training

Approach To Solve the problem

1. Use pretrained weights from Inception V3 network. Extract features from the last CNN layer, as we would be using attention in this model. Remove the last layer of the model, and transform all the input images using these weights.

2. Define an encoder layer. It has a fully connected layer. Pass the pre trained vector from Inception V3 network to featurize images at encoder stage.

3. Now, Lets talk about the impressions. Tokenize the text data. Use top k words, and replace all other words with tokens. Pad sequences to the maximum length.

Model and Training Details

1. Start: Use the vectors that we got by transforming all images on Inception V3 weights. Pass it through encoder. We define a LSTM/GRU(not bidirectional) decoder. This decoder attends the image to predict the next word.

2. Train: To train the model, pass the saved vectors from images through encoder. Pass the encoder output, hidden states, and token to the decoder network. The decoder's hidden state is passed back to model, and the output is used to calculate the loss. Now pass the target word as the next input to decoder. It is different and worth noting that we do not pass the predicted output but the original output as the next input. This is known as Teacher Forcing. Finally, calculate gradients and backpropagate.

3. While Generating the output, do the same as step above. But here we pass the previous output as input to the next time step of the decoder. Stop at token. Store the weights of attention at every timestep of generating the output.

4. Plot the Attention weight and the part it is focusing at every timestep of input.

Training loss

Validation loss

Research-Papers/Solutions/Architectures/Kernels/References

1. https://github.com/wisdal/diagnose-and-explain

2. https://towardsdatascience.com/deep-learning-for-detecting-pneumonia-from-x-ray-images-fc9a3d9fdba8

3. https://towardsdatascience.com/deep-learning-for-detecting-pneumonia-from-x-ray-images-fc9a3d9fdba8

4. https://www.youtube.com/watch?v=MgrTRK5bbsg&list=PLQY2H8rRoyvxcmHHRftsuiO1GyinVAwUg&index=21&t=0s

5. https://conferences.oreilly.com/tensorflow/tf-ca-2019/public/schedule/proceedings

6. https://github.com/wangleihitcs/Papers/blob/master/medical%20report%20generation/2019(AAAI)%20Knowledge-Driven%20Encode%2C%20Retrieve%2C%20Paraphrase%20for%20Medical%20Image%20Report%20Generation.pdf

7. https://github.com/wangleihitcs/Papers/tree/master/medical%20report%20generation

8. https://github.com/shreyanshrs44/Medical-Report-Generation

9. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8867873

10. https://github.com/zhjohnchan/awesome-radiology-report-generation

11. https://github.com/omar-mohamed/X-Ray-Report-Generation/tree/master

12. https://www.tensorflow.org/tutorials/text/image_captioning

13. https://arxiv.org/pdf/1502.03044.pdf

14. https://gist.github.com/UdiBhaskar/9854346018d151d38e6772cbf8f42bba

15. https://gist.github.com/UdiBhaskar/070e666dbafbe35f011528b748b7d4b0

16. https://www.appliedaicourse.com/course/11/Applied-Machine-learning-course