Giters

pymc-devs / pymc-experimental

Home Page:https://pymc-experimental.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Standard deviation parameters are incorrectly treated as variances in statespace covariance matrices

rklees opened this issue · comments

commented

I simulate data from a simple integrated random walk model defining the initial state, the measurement disturbance standard deviation (here, I set it equal to 0), and the trend disturbance standard deviation. Given these parameters, the simulated data should be close to a straight line. However, is looks completely different. Therefore, I used statsmodel to simulate the data using the same initial state and disturbance standard deviations. Also the random generator is initialized the same in both simulations. The simulate dataset from statsmodel is close to a straight line as expected. Therefore, I wonder whether there is something wrong in simulate_from_numpy_model. Here is the code:

measurement_error = st.MeasurementError(name="obs")
IRW = st.LevelTrendComponent(order=2, innovations_order=[0, 1])

param_dict = {
"initial_trend": [0., 0.01], "sigma_trend": np.array([np.sqrt(5e-7)]),
"sigma_obs": np.array([0.]),
}

mod = IRW + measurement_error
nobs = 1024 # we simulate a time series of length Nobs

x, y = simulate_from_numpy_model(mod, np.random.default_rng(2163), param_dict, steps=nobs)

And here is the code from statsmodel:
model = sm.tsa.UnobservedComponents(y, level=True, stochastic_level=False, trend=True, stochastic_trend=True, irregular=True)
res = model.fit()
y_sim = model.simulate(initial_state = [0., 0.01], params=[0., 5e-7], nsimulations=nobs, random_state=np.random.default_rng(2163))

y and y_sim have nothing common. Whereas y_sim represents a slight deviation from the straight line, y is completely different, with a strong negative slope, though the undisturbed slope is positive. Attached the simulated data in both cases.
simulate_from_numpy_model
statsmodel

commented

I can take a look, but this is just a debug function. Why are you using it? There is a large test suite that checks that the statespace models match the statsmodels outputs.

commented

I use it because it makes data generation for a given state-space model easy, much better than statsmodel (as far as I know).

commented

In this specific case, statsmodels expects sigma squared, whereas the structural model expects sigma. If you remove the square root from the structural model (but not the statsmodel argument), you will get the same output, modulo random draws. I don't think setting the seed is enough to guarantee the same outputs between the two functions, because there might be differences in how/when each random number is generated (all at once vs one-by-one in loop, for example).

I'm not sure what your exact use-case in, but in general I suggest you use the usual PyMC API to generate draws from the statespace models, rather than the testing function.

commented
commented 
measurement_error = st.MeasurementError(name="obs")
IRW = st.LevelTrendComponent(order=2, innovations_order=[0, 1])

# Change sigma_trend to remove the square root
param_dict = {
"initial_trend": [0., 0.01], "sigma_trend": np.array([5e-7]),
"sigma_obs": np.array([0.]),
}

mod = IRW + measurement_error
nobs = 1024 # we simulate a time series of length Nobs

x, y = simulate_from_numpy_model(mod, np.random.default_rng(2163), param_dict, steps=nobs)

# sm part unchanged 

fig, ax = plt.subplots()
ax.plot(y, label='Structural')
ax.plot(y_sim, label='Statsmodels')
ax.legend()
plt.show()

Untitled

This now looks correct to me?

commented
commented

Yeah that's a fair point, the name is not consistent with what you get. Currently, whatever number you pass in is directly plugged into the Q matrix at whatever position on the main diagonal, so either it should be squared internally (this is what statsmodels does) or the name should be changed.

My preference would be to square things internally and keep the name the same, because no other PyMC distributions are parameterized by the variance directly. Does that seem reasonable?

commented
commented
commented

It applies to all shocks in all structural models I believe. I will have to check them all.

commented
commented

You'll have to pip install the main branch from github. We haven't done a new release yet, so just doing pip install pymc-experimental won't have the patch

commented
commented

It's similar to what you did before when you were installing from my branch, but now just targeting the main branch. Should be something like:

pip install git+https://github.com/pymc-devs/pymc-experimental.git

commented
commented

Make sure you pip uninstall pymc-experimental first

commented
commented
commented

I just tested the command on my computer and it seems like it works. There hasn't been a release, so the version won't be bumped by this command.

You can check the source code by e.g. st.MeasurementError?? (in a Jupyter notebook) and check that you see the sigma being squared in the make_symbolic_graph method, like this:

    def make_symbolic_graph(self) -> None:
        error_sigma = self.make_and_register_variable(f"sigma_{self.name}", shape=(self.k_endog,))
        diag_idx = np.diag_indices(self.k_endog)
        idx = np.s_["obs_cov", diag_idx[0], diag_idx[1]]
        self.ssm[idx] = error_sigma**2
commented
commented
commented

Everything works as it appears now. You put a prior on sigma (the standard deviation), which is then squared and placed on the diagonal of the covariance matrix.

commented