single dimension running problem

[Mr. Feng]

Hi, the path "prompts/prompts_per_dimension" contains only a part of evaluation dimensions. So, if i want to evaluation the dimension of "background_consistent", and there is no proper prompt text file (maybe background_consistent.txt) for me to generation videos. So, what can i do to perform evaluation on "background_consistent"?

[Mr. Feng]

And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?

[Yinan He]

Hi, the path "prompts/prompts_per_dimension" contains only a part of evaluation dimensions. So, if i want to evaluation the dimension of "background_consistent", and there is no proper prompt text file (maybe background_consistent.txt) for me to generation videos. So, what can i do to perform evaluation on "background_consistent"?

@GFENGG Hi , you can refer to this file. On the left are the corresponding evaluation dimensions, and on the right is the prompt list of left dimension.

For example, if you want to test background_consistency, you only need to sample the video of scene.

VBench/dimension_to_folder.json

Lines 2 to 17 in 8046fcb

 "subject_consistency": "subject_consistency", 
 "background_consistency": "scene", 
 "aesthetic_quality": "overall_consistency", 
 "imaging_quality": "overall_consistency", 
 "object_class": "object_class", 
 "multiple_objects": "multiple_objects", 
 "color": "color", 
 "spatial_relationship": "spatial_relationship", 
 "scene": "scene", 
 "temporal_style": "temporal_style", 
 "overall_consistency": "overall_consistency", 
 "human_action": "human_action", 
 "temporal_flickering": "temporal_flickering", 
 "motion_smoothness": "subject_consistency", 
 "dynamic_degree": "subject_consistency", 
 "appearance_style": "appearance_style" 

[Yinan He]

And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?

@GFENGG all_category is not a standalone dimension, so it will not appear in the json.

prompts/all_category.txt is a collection of video prompts for all dimensions that need to be tested.
By sampling all prompts from prompts/all_category.txt into videos, VBench will automatically assign corresponding evaluation dimensions to each sampled video for evaluation and provide results.

[Mr. Feng]

And the "prompts/all_category.txt" is used to evaluation the model on different categories? But in "VBench_full_info.json", there seems no prompts of "prompts/all_category.txt"?

@GFENGG all_category is not a standalone dimension, so it will not appear in the json.

prompts/all_category.txt is a collection of video prompts for all dimensions that need to be tested. By sampling all prompts from prompts/all_category.txt into videos, VBench will automatically assign corresponding evaluation dimensions to each sampled video for evaluation and provide results.

Thanks for your reply. In which file (maybe a .json file like "VBench_full_info.json") can i find the relation between the prompts in "prompts/all_category.txt" and corresponding evaluation dimensions?

[Yinan He]

@GFENGG Sorry for the late reply. In all_category, there are approximately 100 prompts for each category. We conducted VBench evaluation for all prompts in subject consistency, background consistency, motion smoothness, dynamic degree, aestheic quality, image quality and overall consistency without further screening corresponding relationships.

[Ziqi Huang]

Hi, we're closing this issue as it appears your questions have been addressed. However, feel free to open a new issue or reopen this one, if you have further questions or if anything else comes up related to this issue.