Giters

microsoft / nn-Meter

A DNN inference latency prediction toolkit for accurately modeling and predicting the latency on diverse edge devices.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

can nn-meter be used for pytorch/tensorflow on the server device?

g-crabapple opened this issue · comments

commented

will there be the code guidence for custom backend for pytorch/tensorflow on the server device？because I had ever seen you say you test nn-meter on V100 in the answer under another issues.thanks

commented

Sorry for the late reply. As of now, we do not have plans to provide server-side guidance. In our preliminary experiments, we took a quick look at V100 inference and found that latency can be calculated through a lookup table, indicating that almost all operators run sequentially during inference. However, we have not conducted further exploration on this topic. In fact, backends on the server-side can be registered in the same way as on the edge-side, but it would require users to build their own interface.

commented

@JiahangXu thanks for your time! I'll try it