Video Caption using S2VT
Requirements
For running my code and reproducing the results, the following packages need to be installed first. I have used Python 3.6 for the whole of this project.
- PyTorch
- Caffe
- NumPy
- cv2
- imageio
- scikit-image
Running instructions
1.Install all the packages mentioned in the 'Requirements' section for the smooth running of this project.
2.Download the MSVD dataset to Data/YouTubeClips
3.Change all the path in these python files to point to directories in your workspace
4.Run extract_feats.py to extract the RGB features of videos
5.Run train.py to train the model
6.Run test.py to generate the caption of test videos
or you can directly extract features from a video and generate captions using test_from_video.py
DataSet
You can download the MSVD dataset here
You can download the extracted video features at Features_VGG, and
unzip it to "Data/Features_VGG"
Result
The val.json is the grundtruth of test dataset, the result.json is the generate result. We use the metrics of WangLei(https://github.com/wangleihitcs/CaptionMetrics)
Acknowledgement
Some code copy from vijayvee(https://github.com/vijayvee/video-captioning)