`detector` in binary mode gives false result
RomainJennes opened this issue · comments
Detectors that allow multiclass detection can't be used in binary mode. The result is that they classify all executables as packed.
Preparation steps
$ dataset make baseline -n 100 -f PE -p upx
Issue
PEiD, for example, gives correct results in multiclass mode:
$ detector baseline -d PEiD
<<snipped>>
Detection results:
──────── ───────── ────── ─────────
Accuracy Precision Recall F-Measure
83.00% 100.00% 63.04% 77.33%
──────── ───────── ────── ─────────
But fails in binary mode because classes everything as packed:
$ detector baseline -b -d PEiD
<<snipped>>
Detection results:
──────── ───────── ─────── ─────────
Accuracy Precision Recall F-Measure
29.00% 29.00% 100.00% 44.96%
──────── ───────── ─────── ─────────
In contrary, detectors that can only work in binary mode give good results:
$ detector baseline -b -d REMINDer
<<snipped>>
Detection results:
──────── ───────── ─────── ─────────
Accuracy Precision Recall F-Measure
97.00% 93.88% 100.00% 96.84%
──────── ───────── ─────── ─────────