Visual Commonsense Reasoning
On VCR, a model must not only answer commonsense visual questions, but also provide a rationale that explains why the answer is true.
Submitting to the leaderboard
Submission is easy! You just need to email Rowan with your predictions. Formatting instructions are below:
What kinds of submissions are allowed?
The only constraint is that your system must predict the answer first, then the rationale. (The rationales were selected to be highly relevant to the correct Q,A pair, so they leak information about the correct answer.)
- To deter this, the submission format involves submitting predictions for each possible rationale, conditioned on each possible answer.
- A simple way of setting up the experiments (used in the paper) is to consider a task with query and four response choices. For Q->A the query is the question, and the response choices are the answers. For QA->R, the query is the question and answer, concatenated together, and the response choices are the rationales.
Questions?
If it's not about something private, check out the google group below:
VCR Leaderboard
There are two different subtasks to VCR:
- Question Answering (Q->A): In this setup, a model is provided a question, and has to pick the best answer out of four choices. Only one of the four is correct.
- Answer Justification (QA->R): In this setup, a model is provided a question, along with the correct answer, and it has to justify it by picking the best rationale out of four choices.
We combine the two parts with the Q->AR metric, in which a model only gets a question right if it answers correctly and picks the right rationale. Models are evaluated in terms of accuracy (%). How well will your model do?
| Rank | Model | Q->A | QA->R | Q->AR |
|---|---|---|---|---|
| Human Performance University of Washington (Zellers et al. '18) |
91.0 | 93.0 | 85.0 | |
📼 Feb 19, 2019 | FAIR Facebook AI Research | 65.7 | 70.1 | 46.3 |
2 Feb 9, 2019 | CKRE Peking University | 66.1 | 68.5 | 45.5 |
3 Nov 28, 2018 | Recognition to Cognition Networks University of Washington https://github.com/rowanz/r2c | 65.1 | 67.3 | 44.0 |
4 Nov 28, 2018 | BERT-Base Google AI Language (experiment by Rowan) https://github.com/google-research/bert | 53.9 | 64.5 | 35.0 |
5 Nov 28, 2018 | MLB Seoul National University (experiment by Rowan) https://github.com/jnhwkim/MulLowBiVQA | 46.2 | 36.8 | 17.2 |
| Random Performance | 25.0 | 25.0 | 6.2 |