How to set max_compositions
hlzhang109 opened this issue · comments
Hi Chen, do you know how to set the max_compositions/steps param? The default at
is 0 but would raise an error
private-transformers/private_transformers/privacy_utils/accounting/gdp_accounting.py:33: RuntimeWarning: invalid value encountered in double_scalars return norm.cdf(-eps / mu + mu / 2) - np.exp(eps) * norm.cdf(-eps / mu - mu / 2)
rv_accountant/accountant.py:55: RuntimeWarning: divide by zero encountered in double_scalars mesh_size = 2*eps_error / np.sqrt(2*max_compositions*np.log(2/eta0))
Great point. Ideally, we'd only call the privacy accounting function after we've done a gradient update, in which case we'd get strictly positive self.steps (note self.steps is updated in the step function).
My code runs a prelim. eval. before any parameter update just to check the initial performance model of the model. This produces a warning you'd get, since the privacy accounting code is also ran. This doesn't really affect the correctness however.
By the way, self.steps is not the max composition steps, but rather the total number of updates already performed by the optimizer.
Thanks Chen for the explanation! I set the dataset size too small when debugging so self.steps didn't get updated as gradient_accumulation_steps was never reached.