PhamThinh31 / src-for-videosummarization

extract-feaute.py (Convolutional Neural Network Feature Extractor)

Version: 0.3

Requirements

The main library

tensorflow
tensorflow-gpu
keras
torch
torchvision
opencv-python
and more . . . 

---

Usage on another script

Two class ExtractFeatureVideo and ExtractFeatureDataSet

Import to use

from CNN import ExtractFeatureVideo, ExtractFeatureDataSet

Extract Feature Video

module = ExtractFeatureVideo(video_path,sampling_rate=int,device_name='str')

feature = module.VGG16(output_layer='str')

feature.save(path_to_save)

This code support VGG16, VGG19, ResNet50, ResNet152, InceptionV1, InceptionV3. To use anoher architecture read tutorial below.

--

Extract Feature DataSet

This class is inheritance of ExtractFeatureVideo.

module = ExtractFeatureDataSet(dataset_name,output_path,from_id=int,to_id=int,sampling_rate=int,device_name='str'):

module.VGG19(output_layer='str')

---

Usage CNN.py script

Use command python CNN.py --help or python CNN.py -h for see full arguments

Positional arguments

python CNN.py name_of_method x_str

name_of_method is name of CNN architecture. Support: VGG16, VGG19, ResNet50, ResNet152, InceptionV1, InceptionV3

x_str is Video(V) or Dataset(D). Choose Video or DataSet for run

Example

python CNN.py inceptionv1 v | python CNN.py inceptionv1 d

python CNN.py Resnet50 video | python CNN.py resnet50 dataset

Optional arguments

Include positional arguments, and have two optional +Video and +Dataset

+Video

• VideoPath* : Path to video

• -sr or --samplingRate: Sampling rate, default = 1.

• -l or --layer: Name of hiden layer on CNN architecture, default=None.

• -d or --device: Device use to run, default='0'.

• -s or --save: Path to save feature data to file, default=None.

  • Example: python CNN.py inceptionv1 video +Video /path/to/video.mp4 -sr 2 -d 1 -s path/to/folder/save/ -l fc1

+DataSet

• DataSetName* : Name of Dataset. Support three dataset: BBC EastEnders, TVSum50, SumMe. To use on another dataset, read tutorial below.

• OutputPath* : Path to folder save feature data file.

• -f or --fromid : ID begin-th video, default=None.

• -e or --endid : ID end-th video. Two args use for big dataset. Run from video begin-th to video end-th, default=None.

• -sr or --samplingRate: Sampling rate, default = 1.

• -d or --device: Device use to run, default='0'.

  • Example: python CNN.py resnet152 dataset +DataSet bbc /path/folder/out -f 12 -e 35 -sr 2 -d 2

To use on another dataset. Try to use file utilities/make_csv_from_dataset.py with command

python make_csv_from_dataset.py +newdataset dataset_name /path/to/folder/video /path/to/folder/out

---

Add new CNN architector

There are two famous frameworks used for extract feature is tensorflow and pytorch. I make two template for them.

Tensorflow

Read this template code.

Brief code:

def NAME_OF_CNN_HERE(self,output_layer=OUTPUT_LAYER_HERE):
    ...
    base_model = tensorflow_model_here.TENSORFLOW_MODEL_HERE(weights='imagenet')
    ...
    self.imageSize = (XXX_HERE,YYY_HERE) #size of image
    ... 

Change the content of the lines that have the words *HERE. Read comment on template code

Pytorch

Read this template code.

Brief code:

class NAME_OF_CNN_HERE(nn.Module):
        ---
        change_pytorch_model_here = models.CHANGE_PYTORCH_MODE_HERE(pretrained=True)
        ...
        change_pytorch_model_here.float()
        change_pytorch_model_here.cuda()
        change_pytorch_model_here.eval()
        module_list = list(CHANGE_PYTORCH_MODEL_HERE.children())
        self.conv5 = nn.Sequential(*module_list[: -XXX_HERE]) 
    ---

Change the content of the lines that have the words *HERE. Read comment on template code