René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Cannot reproduce Action-Prediction Score","articleBody":"Thank for a great benchmark!!\r\n\r\n\r\nI tried to reproduce the results of VisualWebBench using `lava-v1.6-vicuna-7b` model.\r\n\r\nThe  results are as follows\r\n```\r\n| Task type         | Metric   |    Score |\r\n|-------------------+----------+----------|\r\n| web_caption       | rouge_1  | 29.985   |\r\n| webqa             | f1       | 39.599   |\r\n| heading_ocr       | rouge_1  | 57.33    |\r\n| element_ocr       | rouge_1  | 55.5956  |\r\n| element_ground    | accuracy | 31.477   |\r\n| action_prediction | accuracy |  1.06762 |\r\n| action_ground     | accuracy | 10.6796  |\r\n```\r\n\r\n**This results differ significantly from the paper's score (30.6) with respect to the “Action Prediction” score (my experiment's score is 1.06762).**\r\nAs for the scores for the other tasks, they seem to be well reproduced.\r\n\r\nI would like to know in what environment you experimented.\r\nAlso, has anyone else obtained similar results?","author":{"url":"https://github.com/Ryosuke0104","@type":"Person","name":"Ryosuke0104"},"datePublished":"2024-08-23T08:19:18.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/7/VisualWebBench/issues/7"}