Normal Inverse Gaussian Outputs Positive log_prob

[i418c]

While training a model with a NIG output, I noticed that the loss for the negative log likelihood would go negative. At first I thought that perhaps the values of verify in the model were outside of support for the NIG function, but fixing these values to what (I think) should be valid still would yield a negative loss.

Any help on why this occurs and how to fix it would be appreciated.

While creating an example, I noticed that the quickest way to trigger this was to set the verify values to single number. A gist reproducing the issue can be found here.

[Christopher Suter]

This is fine. Probability densities can be large and positive, yielding positive log prob and negative NLL loss.

…

On Sun, Jan 14, 2024 at 23:44 i418c ***@***.***> wrote: While training a model with a NIG output, I noticed that the loss for the negative log likelihood would go negative. At first I thought that perhaps the values of verify in the model were outside of support for the NIG function, but fixing these values to what (I think) should be valid still would yield a negative loss. Any help on why this occurs and how to fix it would be appreciated. While creating an example, I noticed that the quickest way to trigger this was to set the verify values to single number. A gist reproducing the issue can be found here <https://gist.github.com/i418c/8e407534558ede92e39e2fbfc5e9c91f#file-normal-inverse-gaussian-negative-log_prob-ipynb> . — Reply to this email directly, view it on GitHub <#1782>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABG2GM3JJBB2E5B63AR5NDYOSX37AVCNFSM6AAAAABB2VQMFOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA4DCMRUGA4DANQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[i418c]

Let me ask a different question then. If the loss of my model can be negative, how would I know if it's anywhere near optimal? Experimentation?

[Christopher Suter]

Hehe, welcome to the wild world of high dimensional nonconvex optimization 🙂. No idea! I think most people abandon hope of provable optimality in most cases and focus on whether a particular method solves a specific problem satisfactorily.

…

On Mon, Jan 15, 2024 at 09:28 i418c ***@***.***> wrote: Let me ask a different question then. If the loss of my model can be negative, how would I know if it's anywhere near optimal? Experimentation? — Reply to this email directly, view it on GitHub <#1782 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABG2GJJ7MKJJBPARNDK36TYOU4J3AVCNFSM6AAAAABB2VQMFOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJSGI3TQMZUGE> . You are receiving this because you commented.Message ID: ***@***.***>

[i418c]

Thanks for the help. It may be helpful for the project to have a tutorial or example to highlight that a negative loss isn't abnormal when training with NLL. Those of us who did poorly in statistics would be grateful.