NVIDIA / garak

the LLM vulnerability scanner

Home Page:https://discord.gg/uVch4puUCs

Repository from Github https://github.comNVIDIA/garakRepository from Github https://github.comNVIDIA/garak

Add probe tier to `--list_probes`

erickgalinkin opened this issue · comments

Summary

When enumerating probes, also print the tier of the probe. Alternately/additionally, allow --list_probes to accept an optional argument to list only probes of that tier.

Basic example

garak --list_probes yields something like:

probes: ansiescape 🌟                       
probes: ansiescape.AnsiEscaped.      Tier 1
probes: ansiescape.AnsiRaw.              Tier 2
...

garak --list_probes 1 yields something like:

probes: ansiescape.AnsiEscaped.      Tier 1
probes: dan.Ablation_Dan_11_0.         Tier 1

etc.

Motivation

Help people identify/search for probes a little more cleanly.

@erickgalinkin May I be assigned this role? My plan for the issue is as follows:

  1. Read the existing Garak CLI + update to allow for second parameter with function
  2. Extend the logic to allow tier to be printed based on the second parameter.
  3. Add help text and filtering