Giters

rbonghi / jetson_stats

📊 Simple package for monitoring and control your NVIDIA Jetson [Orin, Xavier, Nano, TX] series

Home Page:https://rnext.it/jetson_stats

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Mem graph doesn't update after crash of the program when used all memory

martincerven opened this issue · comments

commented

Describe the bug

When using models that consume all available memory and eventually crash, the jtop doesn't update, only after ctrl+c, jtop it shows available memory correctly.

To Reproduce

Steps to reproduce the behavior:

  1. use audiocraft on Orin Nano with monitor
  2. When running jupyter cells, it will eventually run out of memory and crash container
  3. Mem won't update, only after closing and reopening

Expected behavior

I would expect I won't have to restart jtop

Additional context

When running any (pytorch) models that consume more memory than available, the jtop doesn't update after they eventually crash.

Board

Happened to both Orin Nano 8GB devkit, when trying audiocraft, and AGX 32GB devkit when running larger models (in containers)