Giters

fbcotter / py3nvml

Python 3 Bindings for NVML library. Get NVIDIA GPU status inside your program.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Question: Retrieve Peak GPU memory usage

patrickvonplaten opened this issue · comments

commented

Hello @fbcotter,

Thanks a lot for the great library!
Do you know how one could retrieve the peak GPU memory usage of a process?

commented

Hi @patrickvonplaten sorry I didn't get a notification about your question! I'm sure you'd have figured something out by now, but if you want it to be access programmatically, you could spin up a monitoring process that queries the gpu state regularly using py3nvml and keeps track of the peaks.

If you just want to see the peaks maybe something like nvtop is better. Or even simpler would be watch -n 0.5 py3smi (if you have py3nvml installed or watch -n 0.5 nvidia-smi if you don't)