zhangxiaohan123 / pytorch-fixed-point-analysis

Implementation of fixed point analysis for Recurrent Neural Network.

Fixed Point Analysis

This is the implementation of fixed point analysis for Recurrent Neural Network by PyTorch.

Sussillo, D., & Barak, O. (2013). Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks.

Niru Maheswaranathan. et al. (2019) Universality and individuality in neural dynamics across large populations of recurrent networks.

This repository contains the code for the analysis on the canonical task Frequency-cued sine wave, which is studied on these papers.

Experiments

First, train your model by train.py.

Trajectories and topology of fixed points

• Plot trajectories and fixed points

$ python plot_trajectories.py --activation relu

• Different points in the same trajectory correspond to one fixed point, and different trajectories correspond to different fixed point.

$ python compare_fixed_point.py --activation relu

distance between 2 fixed point start from different IC; different time of same trajectory.
2.2076301320339553e-07
distance between 2 fixed point start from different IC; same time of different trajectories.
0.13503964245319366

Eigenvalue decomposition of Jacobian around fixed points.

$ python linear_approximation.py --activation relu

• Distribution of eigenvalues

• There is the correlation with the frequencies of trajectories and the values of the imaginary part of the maximum eigenvalue of Jacobians.