A list of resources dedicated to compositionality.
Contributions most welcome! Please check the Contribution guideline for an example.
- David Ha, Andrew Dai, Quoc V. Le. HyperNetworks, arXiv:1609.09106, ICLR, 2017
- The main idea is to use a hypernetwork to generate the weights for another network. An “embedding vector” is generated by a hyper RNN. This embedding vector is used to dynamically scale the weights of a RNN at every timestep. The approach is evaluated on different language tasks: character-level language modeling (PTB, Wiki), machine translation and handwriting prediction. The results outperform the baselines on these tasks marginally.
-
Lake, Brenden, and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks ICML. 2018.
- Study on the ability of RNNs to generalize in a compositional way on a data set for sequence-to-sequence modelling called SCAN, which requires models to create compositional solutions and generalize on new data based on related concepts.
-
A Santoro, F Hill, D Barrett, A Morcos, T Lillicrap. Measuring abstract reasoning in neural networks ICML, 2018.
- A dataset for abstract reasoning inspired by human-like IQ tests. To succeed in the task the model must be able to generalize in various ‘regimes’ in which the training and test data differ in clearlydefined ways.
-
Atzmon, Yuval, et al. Learning to generalize to new compositions in image understanding arXiv:1608.07639 (2016).
- A compositional split of the MS COCO dataset. This alternative split for training and test data can be used to test whether image captioning models can generalise to new (unseen) compositions.
- Singh, Chandan, et al. Hierarchical Interpretations for Neural Network Predictions arXiv:1806.05337 (2018).
- Provides a method for creating a hierarchical representation of the contributions to the classification of both RNNs and CNNs.
- Christiansen, Morten H., and Nick Chater. The Now-or-Never bottleneck: A fundamental constraint on language. Behavioral and Brain Sciences 39 (2016).
- Combining psycholinguistic evidence to hypothesize that the way humans process language is highly constrained by limited time and memory resources, therefore implying a very particular process in which the brain chunks incoming input and processes it into increasingly abstract representations.
- Lake, Brenden M., et al. Building machines that learn and think like people. Behavioral and Brain Sciences 40 (2017).
- Highlighting the differences (and missing pieces) between human cognition and current Deep Learning architectures.