Giters

Qengineering / Rock-5-image

Rock Pi 5 image with OpenCV, deep learning frameworks and NPU drivers

Home Page:https://qengineering.eu/deep-learning-examples-on-raspberry-32-64-os.html

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Rock 5 image on Orange pi 5

FeAlias opened this issue · comments

commented

Hi, just wondering if this would work on the Orange pi 5? Thanks.

commented

@FeAlias ,

You can always try, but I don't think it will work. Undoubtedly there will be low level hardware differences which prevent the image to function properly.

commented
commented

You need to install the RKNPU framework on your Orange Pi. It comes with examples.

To get home brewed DNN models working you need the RKNN Toolkit. It can be installed on a Ubuntu machine.

Note, the Toolkit will install/use Tensorflow, Pytorch, PaddlePaddle and other major frameworks to convert their models to NPU models. It means a massive load on your machine. Use a PC, not the Orange Pi if you want to use the Toolkit.

commented
commented

I think I must warn you. Don't set your expectations too high!
Biggest problem with converting models are the supported operations. Not every operation can be done in a NPU.
Second problem is the 8-bit conversion. NPUs are working fast with 8 or 16 bit integers. They are not so keen on floats. Not every model can be converted into another format without a penalty. (Loss of accurateness, for instance).
You may try converting the model on a PC and see if it succeeds. If so, your lucky. If not , too bad.
There is almost no difference between the Orange and the Rock. No need for a Rock, if you have the Orange.

commented
commented

You (Python or C++) program must evoke the NPU. See it as a slave. It only execute calculations given by another "master" program, like your Python program. Don't try to modify low level drivers or software.
The correct procedure goes as follows:

  1. You get your model.
  2. See if the model is supported by the RK toolkit (Tensorflow, Pytorch etc).
  3. If not, see if you can convert it to ONNX. (Here, things can get complicated, as mentioned earlier)
  4. Try to convert your model to RKNPU.
  5. Transfer the RKNPU model to your Orange Pi. (So far you were working on the Ubuntu PC)
  6. Write a Python program that uses the model.
  7. Once you are happy, you can call the program during booting the Pi.
commented

Showstopper.
I just read that RedPajama-INCITE-7B-Chat is a LLM model.
Most of these models consist over a billion parameters. That is far too much for a tiny device like Orange or Rock.
The NPU hasn't the capacity to hold even a fraction of the parameters.

Better to use BERT base, BERT Squad, MobileBERT or DistilBERT. They are designed for edge devices.

commented