tfjgeorge / nngeometry

{KFAC,EKFAC,Diagonal,Implicit} Fisher Matrices and finite width NTKs in PyTorch

Home Page:https://nngeometry.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Montecarlo Sampling Question

simomagi opened this issue · comments

Looking at the implementation of the Monte Carlo Sampling for computing the Fisher Information I see that the sampled probabilities are divided by the square root of the number of trials. Is there a specific reason for this? Why not simply dividing by the number of trials?

Hi, if you look at the formula:
Screenshot from 2021-10-01 15-05-50

In order to get a 1/n on the left hand side, you need to factorize by 1/sqrt(n) in the parenthesis on the right hand side