lindermanlab / S5

I'm currently looking to reproduce the results from 6.3 of the S5 paper, but it seems like the experiment is missing in both the S5 model & the dataloading.

Would it be correct to 'hijack' the log_step (

S5/s5/ssm.py

Line 216 in bdb9eda

self.log_step = self.param("log_step",

) and substitute if for log(dt) of the observations?

Hi there,

Thanks for raising the issue. Yes, the public version of the repository doesn't currently contain the implementations for the pendulum experiments (or any variable observation interval scenarios more generally). You are mostly correct, though, in that you also provide the vector of intervals that are then multiplied with the self.log_step. There is a bit of extra plumbing to pass it all the way through, which makes the code a little bit messy, and so we didn't include this in the initial release.

We are in the process of rolling forward the public repository for reproduction of these experiments. We are hoping to update the repository shortly (~ a month?).

If it's urgent, then feel free to reach out over email and we can talk about getting you a MVP in the meantime.

Thanks,
A

Thank you for the clarification. I will do some tinkering on my side, and wait for the official update in the meantime.

Hello!

Bit of an update on this: We spent some time rolling forward the variable timestep code to be "more compatible" with the current public code. We are still in the process of testing it, and so we are not merging the changes on the branch pendulum into main just yet. I have run it though, and it reproduces the results from the paper.

The code for running the CRU baseline is in a fork of the original code, available here. I use this modified code to generate and save out the pendulum data, and to run the CRU experiment while also logging to WandB.

To prepare the directory structure and generate the data:

git clone https://github.com/lindermanlab/S5.git
cd S5
pip install -r requirements.txt
mkdir -p cache_dir
git checkout pendulum
git clone https://github.com/andrewwarrington/Continuous-Recurrent-Units.git
cd Continuous-Recurrent-Units
pip install -r requirements.txt
wandb login  % If using wandb, follow the on-screen instructions.
python run_experiment.py --dataset pendulum --task regression -lsd 30 --sample-rate 0.5 --USE-WANDB 0 --dir_name ./../cache_dir --exit_after_generation 1  % Generate the data, then exit.

All the pendulum experiments, with all the correct hyperparameters, are contained in ./bin/run_experiments/run_pendulum.sh (which you should be able to fill in with your WandB credentials and then just run).

I wouldn't consider this a ``release'' of the pendulum experiments just yet, we still need to double check that the code still executes the LRA/SC35 experiments correctly, make the code a little cleaner (the extra plumbing for variable intervals is a bit cumbersome), and tidy up how some things are logged to WandB (the main performance metrics are correct, though). However, if you are just looking for how to go about implementing variable intervals, then all of that should be in there.

I will ping this thread again whenever we merge the two implementations.

Let me know if there are any problems!
Andy

Hey, seems like it works!

I also have my repro model (mostly, conj_sym is still missing) done with variable observations. I think some of the extra plumbing can be prevented by making log_step a (H,) shape rather than (H,1), but I could be missing the reason for it. I'll give them both a go once I have some GPU compute available. I'll let you know how it goes.

Hey, It's been a while

I was able to reproduce the variable-time, I ended up recreating the S5 model in pytorch (https://github.com/i404788/s5-pytorch) to integrate with other existing modules but it seemed to take too much memory for my experiment in the end.

Thanks for the all the help

Reproducing "6.3 Variable Observation Interval"