How well does GPT-4V perform Visual Question Answering (VQA) on Data Structures?
There is no dataset for VQA on graph and tree data structures in previous work, so we must create one. We create a standard, repeatable process for selecting and obtaining VQA tasks that must fall under a certain criteria.
Technical overview of creation and evaluation of dataset.
![lmm-graph-tree-vqa](https://private-user-images.githubusercontent.com/44552816/298180395-828ceb7c-a682-4ab2-ad47-7ba972c7ab53.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjUwNTMsIm5iZiI6MTcyMTI2NDc1MywicGF0aCI6Ii80NDU1MjgxNi8yOTgxODAzOTUtODI4Y2ViN2MtYTY4Mi00YWIyLWFkNDctN2JhOTcyYzdhYjUzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDU1M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI1NDgyZmJiMjdkNjlmYTcyMjQ4MTJiODg5M2ZiMTk4OWU3NWE4MzQyYTc2N2JjMmFkOGIxNTAxZGIxMWEyMDImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.QkTFHpckV0z2yaf8KKBJ1XJac0JcI0LsxpgETnQhIGA)
We elect GPT-4V as the primary model to observe for this evaluation.
![model](https://private-user-images.githubusercontent.com/44552816/298180138-9fa31867-6af8-4db7-858c-21efc3e3e199.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjUwNTMsIm5iZiI6MTcyMTI2NDc1MywicGF0aCI6Ii80NDU1MjgxNi8yOTgxODAxMzgtOWZhMzE4NjctNmFmOC00ZGI3LTg1OGMtMjFlZmMzZTNlMTk5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDU1M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg2YWIyODkwNmM0MDU5MzM0Nzg1OTRkYTI5ZmIzYTkyZWJjNDhkZWZhZTM5ODI4NmY1OWIzZDc1YTlkOGU3MTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.lxSD-mM-9tfVZEiF6Vkq0CMhPRaycHiMKuf3jeOl7yc)
Overview of the dataset architecture.
![dataset analysis](https://private-user-images.githubusercontent.com/44552816/298180313-df2150f8-a86c-4f14-bd5d-a42ff58b7d0a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyNjUwNTMsIm5iZiI6MTcyMTI2NDc1MywicGF0aCI6Ii80NDU1MjgxNi8yOTgxODAzMTMtZGYyMTUwZjgtYTg2Yy00ZjE0LWJkNWQtYTQyZmY1OGI3ZDBhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE4VDAxMDU1M1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTIzY2YxNjY5ZDUxMDE5MmEyMTQxNDQ0YTMzYjk5NGMxMjBiNjMxYTA1YTZiMjNkZGVlZDE5YWE3MmExMjBjY2YmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.FS8GL_mzv3eROvUCyFNHEapIT69PYL3U7TwgNmxD_jU)