Scaling of parameter space representations
ksnxr opened this issue · comments
Many thanks for this interesting library!
Comparing with analytical expressions, I think the provided dense representation of Fisher information matrix is calculated as the expectation over the data points in the train loader. Are the other representations, e.g. KFAC and EKFAC, on the same scale? Or, is there a constant scaling, e.g. by the batch size, that we should be aware of?
Everybody is at the very same scale. I am curious why are you asking?
…On Sun, Sep 3, 2023, 00:26 ksnxr ***@***.***> wrote:
Many thanks for this interesting project!
Comparing with analytical expressions, I think the provided dense
representation of Fisher information matrix is calculated as the
expectation over the data points in the train loader. Are the other
representations, e.g. KFAC and EKFAC, on the same scale? Or, is there a
constant scaling, e.g. by the number of batch size, that we should be aware
of?
—
Reply to this email directly, view it on GitHub
<#68>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AALTMWKBY5RC55P45YEC2QTXYOXAVANCNFSM6AAAAAA4I3TQQM>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
Thanks for your quick response. I have a use case where it is necessary to have approximations that are supposed to be of the same scale as the analytical Fisher; Since I couldn't find something about this in the project, I figure it might be better to verify this