Triage runtime crash via LLM
DonggeLiu opened this issue · comments
Our current triage rules cannot identify all errors caused by our fuzz target.
Meanwhile, LLM can help identify the cause of a crash when I test it manually.
We can automate what I did manually to reduce the workload in the current manual triaging.
For each crash NOT ruled out by our current semantic checks, we can pass its 1) source code, 2) error stack trace, and 3) fuzzing log to LLM and ask LLM to judge the cause of the error.
Then, we can read LLM's response from the report during manual triaging.
This can also be extended by appending the source code of all files in the stack trace to the triage prompt, given Vertex AI has a sufficient 32k context window.
We can only add the relevant functions if some files are too long. Function names and line numbers from the call stack can help identify which function to include.
Once we save some triaging prompts and their manual judgments, we can use them as examples in the training prompt.