fine tune
rgarcia opened this issue · comments
I feel like the success rate of a given objective will always be significantly less than 100%. E.g., I've been testing Tarsier to try and get it to make a reservation at a restaurant using resy.com, but it fails at the login step because it incorrectly picks the label on the "password" input to type text into (and wants to type my email in that field no less).
I wonder if there's some system that could be built that has a human in the loop correcting mistakes made by the AI, and those corrections are then used to fine tune the model for that specific task.
Anyway this is more of a "idea" than an issue, but wondering what you think! (Feel free to close)
Are you using GPT-4-Vision or the text based approach?
In the text based case, you could also add a critic loop for the LLM to validate that it made the correct choice. Another option is to first have the LLM choose candidate elements and then provide more context on the selected elements. (Such as HTML attributes)