MarcBS/keras Multimodal Learning fork
This fork of Keras offers the following contributions:
- Caffe to Keras conversion module
- Layer-specific learning rates
- New layers for multimodal data
Contact email: marc.bolanos@ub.edu
GitHub page: https://github.com/MarcBS
MarcBS/keras is compatible with: Python 2.7 and Theano only.
Caffe to Keras conversion module
This module allows to convert Caffe models to Keras for their later training or test use. See this README for further information.
Layer-specific learning rates
This functionality allows to add learning rates multipliers to each of the learnable layers in the networks. During training they will be multiplied by the global learning rate for modifying the weight of the error on each layer independently. Here is a simple example of usage:
x = Dense(100, W_learning_rate_multiplier=10.0, b_learning_rate_multiplier=10.0) (x)
New layers for sequence-to-sequence learning and multimodal data
Recurrent layers
LSTM layers:
- LSTMCond: LSTM conditioned to the previously generated word (additional input with previous word).
- AttLSTM: LSTM with Attention mechanism.
- AttLSTMCond: LSTM with Attention mechanism and conditioned to previously generated word.
And their corresponding GRU version:
- GRUCond: GRU conditioned to the previously generated word (additional input with previous word).
- AttGRUCond: GRU with Attention mechanism and conditioned to previously generated word.
Convolutional layers
- ClassActivationMapping: Class Activation Mapping computation used in GAP networks.
- CompactBilinearPooling: compact version of bilinear pooling for merging multimodal data.
Projects
You can see more practical examples in projects which use this library:
ABiViRNet for Video Description
Installation
In order to install the library you just have to follow these steps:
- Clone this repository:
git clone https://github.com/MarcBS/keras.git
- Include the repository path into your PYTHONPATH:
export PYTHONPATH=$PYTHONPATH:/path/to/keras
Keras
For additional information on the Deep Learning library, visit the official web page www.keras.io or the GitHub repository https://github.com/fchollet/keras.