Visual Commonsense Reasoning

On VCR, a model must not only answer commonsense visual questions, but also provide a rationale that explains why the answer is true.


Submitting to the leaderboard

Submission is easy! You just need to email Rowan (rowanz at cs.washington.edu) with your predictions. Formatting instructions are below:

Please include in your email 1) a name for your model, 2) your team name (including your affiliation), and optionally, 3) a github repo or paper link.

If you submit an ensemble, please tell me how many models you used in your ensemble.

I'll try to get back to you within a few days, usually sooner. Teams can only submit results from a model once every 7 days.

I reserve the right to not score any of your submissions if you cheat -- for instance, please don't make up a bunch of fake names / email addresses and send me multiple submissions under those names.


What kinds of submissions are allowed?

The only constraint is that your system must predict the answer first, then the rationale. (The rationales were selected to be highly relevant to the correct Q,A pair, so they leak information about the correct answer.)

  • To deter this, the submission format involves submitting predictions for each possible rationale, conditioned on each possible answer.
  • A simple way of setting up the experiments (used in the paper) is to consider a task with query and four response choices. For Q->A the query is the question, and the response choices are the answers. For QA->R, the query is the question and answer, concatenated together, and the response choices are the rationales.

Questions?

If it's not about something private, check out the google group below:

VCR Leaderboard

There are two different subtasks to VCR:

  • Question Answering (Q->A): In this setup, a model is provided a question, and has to pick the best answer out of four choices. Only one of the four is correct.
  • Answer Justification (QA->R): In this setup, a model is provided a question, along with the correct answer, and it has to justify it by picking the best rationale out of four choices.

We combine the two parts with the Q->AR metric, in which a model only gets a question right if it answers correctly and picks the right rationale. Models are evaluated in terms of accuracy (%). How well will your model do?

Rank Model Q->A QA->R Q->AR
Human Performance

University of Washington

(Zellers et al. '18)
91.0 93.0 85.0

šŸ“¼

September 11, 2023
GPT4RoI

Anonymous

89.491.081.6

2

January 19, 2024
ViP-LLaVa

Anonymous

89.290.981.3

3

March 14, 2024
LLME-VCR

Advanced Data Engineering and Real-time Computing Laboratory (ADE), School of Computor Science and Technology, Huazhong University of Science and Technology (HUST).

87.086.675.7

4

November 28, 2022
HunYuan_vcr

Tencent Data Platform

85.888.075.6

5

January 28, 2023
SP-VCR (ensemble of 4 models)

Shopee MMU

83.688.674.4

6

April 8, 2022
KS-MGSR

KDDI Research and SNAP

85.386.974.3

7

January 17, 2022
VLUA+

Kuaishou MMU

84.887.074.0

8

February 21, 2022
VQA-GNN + MerlotReserve-Large (ensemble of 2 models)

Anonymous

85.286.674.0

9

April 8, 2022
VLMT-VCR

Anonymous

84.785.972.9

10

December 14, 2021
VL-RoBERTa

Joint Laboratory of HIT and iFLYTEK Research (HFL)

84.286.472.8

11

August 25, 2022
SP-VCR (single model)

Shopee MMU

82.387.472.2

12

June 28, 2021
VLUA (single model)

Kuaishou MMU

82.387.072.0

13

November 1, 2021
šŸ·MerlotReserve-Large

University of Washington / AI2

https://rowanzellers.com/merlotreserve
84.084.971.5

14

May 10, 2021
UNIMO+ERNIE(ensemble of 7 models)

Baidu NLP

https://arxiv.org/abs/2012.15409
82.386.571.4

15

November 19, 2020
BLENDER (single model)

WeSee AI team, Tencent

81.686.470.8

16

June 24, 2020
ERNIE-ViL-large(ensemble of 15 models)

ERNIE-team - Baidu

https://arxiv.org/abs/2006.16934
81.686.170.5

17

January 12, 2024
EventLens-large

Advanced Data Engineering and Real-time Computing Laboratory (ADE), School of Computor Science and Technology, Huazhong University of Science and Technology (HUST).

82.782.768.5

18

October 28, 2020
MMCNet (ensemble of 4 models)

UC Berkeley

80.083.166.9

19

September 30, 2019
UNITER-large (ensemble of 10 models)

MS D365 AI

https://arxiv.org/abs/1909.11740
79.883.466.8

20

April 7, 2022
RobustNet

Anonymous

78.884.066.5

21

June 24, 2020
ERNIE-ViL-large(single model)

ERNIE-team - Baidu

https://arxiv.org/abs/2006.16934
79.283.566.3

22

November 15, 2021
ADVL (single model, formerly CLIP-TD)

Anonymous

79.682.966.2

23

June 8, 2021
DELV

Intel Labs Cognitive AI & MSRA NLC

79.482.565.8

24

May 22, 2020
VILLA-large (single model)

MS D365 AI

https://arxiv.org/pdf/2006.06195.pdf)
78.982.865.7

25

January 12, 2024
EventLens-large

Advanced Data Engineering and Real-time Computing Laboratory (ADE), School of Computor Science and Technology, Huazhong University of Science and Technology (HUST).

81.080.365.5

26

May 28, 2021
MERLOT (single model)

University of Washington / AI2

https://rowanzellers.com/merlot
80.680.465.1

27

October 30, 2020
VitsNet

Carnegie Mellon University

78.881.864.6

28

July 16, 2020
gnimix

anonymous

78.981.064.1

29

June 5, 2021
SEITU

Anonymous

https://github.com/MyLittleChange/SEITU
77.980.763.0

30

September 23, 2019
UNITER-large (single model)

MS D365 AI

https://arxiv.org/abs/1909.11740
77.380.862.8

31

February 15, 2022
VQA-GNN

Anonymous

77.980.062.8

32

November 1, 2021
šŸ·MerlotReserve-Base

University of Washington / AI2

https://rowanzellers.com/merlotreserve
79.378.762.6

33

March 15, 2022
VCR-test

Anonymous

77.380.262.4

34

June 24, 2020
ERNIE-ViL-base(single model)

ERNIE-team - Baidu

https://arxiv.org/abs/2006.16934
77.080.362.1

35

September 10, 2021
Kam-net

Anonymous

77.479.261.8

36

May 22, 2020
VILLA-base (single model)

MS D365 AI

https://arxiv.org/pdf/2006.06195.pdf)
76.479.160.6

37

October 22, 2020
MMCNet

UC Berkeley

76.079.360.6

38

February 17, 2021
Test_VILLA

CCR

76.679.060.6

39

December 3, 2022
VILLA_TEST

Alignment, Shandong University

76.679.060.6

40

December 3, 2022
VILLA_GIST

Alignment, Shandong University

76.479.060.4

41

April 23, 2020
KVL-BERT

Beijing Institute of Technology

76.478.660.3

42

August 9,2019
ViLBERT (ensemble of 10 models)

Georgia Tech & Facebook AI Research

https://arxiv.org/abs/1908.02265
76.478.059.8

43

March 23, 2019
VVT

runningcat

75.378.959.7

44

September 23,2019
VL-BERT (single model)

MSRA & USTC

https://arxiv.org/abs/1908.08530
75.878.459.7

45

August 2, 2021
PVL (single model)

PVL-team (UCLA)

76.977.159.7

46

September 10, 2021
SGEITL

Anonomous

76.078.059.6

47

April 1, 2023
UNITER_kd

WalkingDog, Shandong University

75.777.759.0

48

January 8, 2021
vlt (single model)

Anonymous

75.377.858.9

49

August 9,2019
ViLBERT (ensemble of 5 models)

Georgia Tech & Facebook AI Research

https://arxiv.org/abs/1908.02265
75.777.558.8

50

December 14, 2022
UNITER_joint

WalkingDog, Shandong University

75.677.558.8

51

June 16, 2021
GITRL

Anonomous

75.577.558.7

52

September 4,2019
Unicoder-VL (ensemble of 2 models)

MSRA & PKU

https://arxiv.org/abs/1908.06066
76.077.158.6

53

March 5, 2022
PEVL

Anonymous

76.076.758.6

54

December 14, 2022
UNITER_independent

WalkingDog, Shandong University

75.577.358.6

55

September 23, 2019
UNITER-base (single model)

MS D365 AI

https://arxiv.org/abs/1909.11740
75.077.258.2

56

November 5, 2020
TDN

VARMS (Sun Yat-sen University)

75.776.458.0

57

May 13, 2019
B2T2 (ensemble of 5 models)

Google Research

http://arxiv.org/abs/1908.05054
74.077.157.1

58

March 13, 2023
GPT4

OpenAI

73.575.456.2

59

September 11, 2021
YTX

Harbin Institute of Technology

73.874.855.4

60

April 1, 2023
VL-BERT_kd

WalkingDog, Shandong University

73.275.755.4

61

May 13, 2019
B2T2 (single model)

Google Research

http://arxiv.org/abs/1908.05054
72.675.755.0

62

August 27,2019
Unicoder-VL (single model)

MSRA & PKU

https://arxiv.org/abs/1908.06066
73.474.554.9

63

July 30,2019
ViLBERT (single model)

Georgia Tech & Facebook AI Research

https://arxiv.org/abs/1908.02265
73.374.654.8

64

November 29, 2022
VL-BERT_prec

WalkingDog, Shandong University

73.474.554.8

65

March 5, 2022
ALBEF

Salesforce Research (experiment by Anonymous)

https://arxiv.org/abs/2107.07651
72.974.554.7

66

May 26, 2021
ViLBERT_GCN

Beijing Institute of Technology

73.374.454.6

67

April 7, 2021
DCGR

Harbin Institute of Technology

72.874.254.3

68

March 11, 2021
CARC

Harbin Institute of Technology

72.973.854.1

69

November 14, 2019
HGL

Sun Yat-sen University

72.273.453.2

70

May 22, 2019
TNet (ensemble of 5)

FlyingPig (Sun Yat-sen University)

72.772.653.0

71

March 31, 2021
CMR

flying melon

72.372.852.8

72

July 5,2019
VisualBERT

UCLA & AI2 & PKU

https://arxiv.org/abs/1908.03557
71.673.252.4

73

February 22, 2022
TAB-KD

Anonymous

71.672.852.4

74

March 8,2021
SAC

SAC

71.772.852.2

75

January 5, 2022
GTEHG

MIC-Tongji University

71.272.451.7

76

April 13, 2021
A3 Net

A3 Net

71.272.051.4

77

January 17, 2022
JCL

Tinjin University

70.771.650.9

78

November 29, 2022
TAB-VCR_attribute

WalkingDog, Shandong University

70.571.650.8

79

September 4, 2019
MKDN

Peking University

70.771.550.5

80

October 25, 2019
TAB-VCR

UIUC

https://arxiv.org/abs/1910.14671
70.471.750.5

81

June 3, 2020
RobustCL

NeurIPS 2020 submission ID1218

70.471.550.5

82

October 30, 2021
YXY

Peking University

70.671.250.5

83

May 22, 2019
TNet (single model)

FlyingPig (Sun Yat-sen University)

70.970.650.4

84

March 26, 2021
UABE

WalkingDog from Qilu University of Technology

70.371.350.4

85

November 29, 2019
WWR-Net

Tianjin University

70.870.750.2

86

October 13, 2019
transformer-r2c

SYSU NEW BANNER (Sun Yat-sen University)

70.770.550.0

87

May 13, 2019
HGL

HCP

https://arxiv.org/abs/1910.11475
70.170.849.8

88

September 4,2019
CAR

Sun Yat-sen University

70.570.449.8

89

December 6,2020
SIA V1

SIA

69.671.349.8

90

September 3, 2020
SFW1

Jiangsu University

70.270.749.7

91

January 20, 2022
CCN-KD

Anonymous

70.070.649.7

92

September 3, 2020
SFW2

Jiangsu University

69.671.149.6

93

May 3, 2021
RKB

Hefei University of Technology

69.670.749.3

94

June 10, 2021
PUV (Pretrain UNITER by VC feature)

Anonymous

69.870.249.3

95

February 3, 2021
vlb (single model)

Anonymous

69.869.848.9

96

June 10, 2021
BLU

Anonomous

69.370.148.9

97

May 16, 2019
MRCNet

MILAB (Seoul National University)

68.470.548.4

98

May 18, 2019
CCD

Anonymous

68.570.548.4

99

May 17, 2019
MUGRN

NeurIPS 2019 submission ID 21

68.269.447.5

100

May 13, 2019
SGRE

UTS

https://github.com/AmingWu/Multi-modal-Circulant-Fusion/
67.569.746.9

101

January 30, 2022
R2C-KD

Anonymous

66.969.446.6

102

February 19, 2019
FAIR

Facebook AI Research

65.770.146.3

103

January 20, 2022
CCN-NKD

Anonymous

67.268.246.1

104

November 28, 2019
DAF

Beijing Institute of Technology

66.968.746.0

105

February 25, 2019
CKRE

Peking University

66.968.245.9

106

October 30, 2021
MIE

flying melon

66.268.845.5

107

August 8, 2022
ATGAN

MMT, Tianjin University

63.471.945.5

108

May 14, 2019
emnet

DCP

66.668.045.4

109

Nov 28, 2018
Recognition to Cognition Networks

University of Washington

https://github.com/rowanz/r2c
65.167.344.0

110

May 20, 2019
DVD

SL

66.365.043.3

111

March 27, 2019
GS Reasoning

UC San Diego

65.761.041.1

112

May 17, 2019
R2R (text only)

Anonymous

58.469.140.5

113

May 28, 2021
R2CC

Arjun Singh

60.761.737.6

114

Nov 28, 2018
BERT-Base

Google AI Language (experiment by Rowan)

https://github.com/google-research/bert
53.964.535.0

115

June 18, 2021
MUR

Anonymous

46.546.025.6

116

September 15, 2021
SG-QA-model

Anonymous

70.924.917.7

117

Nov 28, 2018
MLB

Seoul National University (experiment by Rowan)

https://github.com/jnhwkim/MulLowBiVQA
46.236.817.2

118

October 23, 2021
BERT-base-vc-ft

LUKA-Axe

58.624.914.5

119

November 15, 2019
Visual-Lang-base (ensemble)

Tiny-Group

17.774.113.3
Random Performance 25.0 25.0 6.2