Sample stimuli

sample 0 sample 1 sample 2 sample 3 sample 4 sample 5 sample 6 sample 7 sample 8 sample 9

How to use

from brainscore_vision import load_benchmark
benchmark = load_benchmark("Rajalingham2018-i2n")
score = benchmark(my_model)

Model scores

Min Alignment Max Alignment

Rank

Model

Score

1
.664
2
.652
3
.646
4
.625
5
.623
6
.617
7
.608
8
.607
9
.606
10
.600
11
.600
12
.596
13
.594
14
.593
15
.590
16
.588
17
.585
18
.585
19
.584
20
.584
21
.581
22
.575
23
.573
24
.573
25
.573
26
.573
27
.570
28
.564
29
.564
30
.564
31
.563
32
.563
33
.562
34
.561
35
.561
36
.561
37
.560
38
.560
39
.560
40
.560
41
.558
42
.555
43
.555
44
.555
45
.554
46
.554
47
.554
48
.551
49
.549
50
.549
51
.549
52
.549
53
.549
54
.546
55
.546
56
.546
57
.546
58
.545
59
.545
60
.545
61
.545
62
.543
63
.543
64
.542
65
.541
66
.541
67
.541
68
.540
69
.540
70
.539
71
.538
72
.537
73
.537
74
.537
75
.537
76
.536
77
.536
78
.536
79
.535
80
.535
81
.534
82
.534
83
.534
84
.534
85
.533
86
.533
87
.532
88
.532
89
.531
90
.530
91
.528
92
.528
93
.528
94
.528
95
.528
96
.528
97
.527
98
.527
99
.527
100
.526
101
.526
102
.524
103
.524
104
.524
105
.523
106
.523
107
.523
108
.522
109
.522
110
.521
111
.521
112
.521
113
.521
114
.520
115
.520
116
.520
117
.520
118
.519
119
.518
120
.518
121
.517
122
.517
123
.515
124
.515
125
.515
126
.515
127
.515
128
.514
129
.513
130
.513
131
.513
132
.512
133
.512
134
.512
135
.511
136
.511
137
.511
138
.511
139
.510
140
.509
141
.509
142
.508
143
.508
144
.507
145
.507
146
.506
147
.505
148
.505
149
.504
150
.503
151
.503
152
.503
153
.503
154
.502
155
.502
156
.502
157
.500
158
.500
159
.500
160
.500
161
.499
162
.499
163
.499
164
.499
165
.499
166
.498
167
.498
168
.497
169
.496
170
.496
171
.495
172
.494
173
.493
174
.492
175
.491
176
.490
177
.488
178
.488
179
.488
180
.488
181
.487
182
.485
183
.484
184
.481
185
.481
186
.480
187
.480
188
.479
189
.478
190
.478
191
.477
192
.477
193
.477
194
.476
195
.475
196
.475
197
.475
198
.474
199
.474
200
.474
201
.474
202
.473
203
.472
204
.472
205
.471
206
.470
207
.470
208
.469
209
.466
210
.465
211
.464
212
.462
213
.461
214
.461
215
.458
216
.458
217
.456
218
.456
219
.454
220
.454
221
.452
222
.451
223
.451
224
.449
225
.449
226
.448
227
.448
228
.448
229
.447
230
.447
231
.447
232
.446
233
.446
234
.445
235
.445
236
.445
237
.444
238
.443
239
.443
240
.441
241
.440
242
.438
243
.438
244
.437
245
.437
246
.437
247
.435
248
.434
249
.433
250
.433
251
.428
252
.428
253
.427
254
.425
255
.425
256
.424
257
.424
258
.419
259
.415
260
.413
261
.413
262
.410
263
.410
264
.410
265
.408
266
.407
267
.406
268
.405
269
.396
270
.395
271
.392
272
.386
273
.383
274
.381
275
.376
276
.375
277
.373
278
.372
279
.371
280
.370
281
.370
282
.370
283
.370
284
.370
285
.370
286
.370
287
.370
288
.370
289
.370
290
.370
291
.370
292
.370
293
.367
294
.366
295
.365
296
.363
297
.362
298
.360
299
.359
300
.356
301
.354
302
.351
303
.348
304
.346
305
.344
306
.341
307
.341
308
.335
309
.334
310
.333
311
.333
312
.332
313
.330
314
.330
315
.324
316
.322
317
.320
318
.311
319
.310
320
.307
321
.305
322
.291
323
.286
324
.286
325
.285
326
.284
327
.283
328
.279
329
.276
330
.276
331
.270
332
.270
333
.265
334
.263
335
.261
336
.256
337
.256
338
.256
339
.256
340
.256
341
.256
342
.256
343
.256
344
.256
345
.255
346
.254
347
.250
348
.245
349
.243
350
.243
351
.231
352
.226
353
.225
354
.220
355
.219
356
.216
357
.211
358
.211
359
.209
360
.209
361
.208
362
.187
363
.186
364
.185
365
.177
366
.167
367
.165
368
.161
369
.160
370
.157
371
.150
372
.148
373
.144
374
.137
375
.131
376
.129
377
.127
378
.116
379
.114
380
.113
381
.112
382
.108
383
.108
384
.104
385
.103
386
.103
387
.102
388
.101
389
.098
390
.096
391
.095
392
.092
393
.090
394
.083
395
.083
396
.083
397
.082
398
.078
399
.076
400
.075
401
.067
402
.065
403
.061
404
.060
405
.054
406
.049
407
.046
408
.045
409
.032
410
.027
411
.023
412
.020
413
.020
414
.014
415
.014
416
.010
417
.009
418
.009
419
.004
420
-0.002
421
-0.006
422
-0.006
423
-0.010
424
-0.016
425
-0.053
426
-0.053
427
-0.055
428
-0.067
429
X
430
X
431
X
432
X
433
X
434
X
435
X
436
X
437
X
438
X
439
X
440
X
441
X
442
X
443
X

Benchmark bibtex

@article {Rajalingham240614,
                author = {Rajalingham, Rishi and Issa, Elias B. and Bashivan, Pouya and Kar, Kohitij and Schmidt, Kailyn and DiCarlo, James J.},
                title = {Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks},
                elocation-id = {240614},
                year = {2018},
                doi = {10.1101/240614},
                publisher = {Cold Spring Harbor Laboratory},
                abstract = {Primates{	extemdash}including humans{	extemdash}can typically recognize objects in visual images at a glance even in the face of naturally occurring identity-preserving image transformations (e.g. changes in viewpoint). A primary neuroscience goal is to uncover neuron-level mechanistic models that quantitatively explain this behavior by predicting primate performance for each and every image. Here, we applied this stringent behavioral prediction test to the leading mechanistic models of primate vision (specifically, deep, convolutional, artificial neural networks; ANNs) by directly comparing their behavioral signatures against those of humans and rhesus macaque monkeys. Using high-throughput data collection systems for human and monkey psychophysics, we collected over one million behavioral trials for 2400 images over 276 binary object discrimination tasks. Consistent with previous work, we observed that state-of-the-art deep, feed-forward convolutional ANNs trained for visual categorization (termed DCNNIC models) accurately predicted primate patterns of object-level confusion. However, when we examined behavioral performance for individual images within each object discrimination task, we found that all tested DCNNIC models were significantly non-predictive of primate performance, and that this prediction failure was not accounted for by simple image attributes, nor rescued by simple model modifications. These results show that current DCNNIC models cannot account for the image-level behavioral patterns of primates, and that new ANN models are needed to more precisely capture the neural mechanisms underlying primate object vision. To this end, large-scale, high-resolution primate behavioral benchmarks{	extemdash}such as those obtained here{	extemdash}could serve as direct guides for discovering such models.SIGNIFICANCE STATEMENT Recently, specific feed-forward deep convolutional artificial neural networks (ANNs) models have dramatically advanced our quantitative understanding of the neural mechanisms underlying primate core object recognition. In this work, we tested the limits of those ANNs by systematically comparing the behavioral responses of these models with the behavioral responses of humans and monkeys, at the resolution of individual images. Using these high-resolution metrics, we found that all tested ANN models significantly diverged from primate behavior. Going forward, these high-resolution, large-scale primate behavioral benchmarks could serve as direct guides for discovering better ANN models of the primate visual system.},
                URL = {https://www.biorxiv.org/content/early/2018/02/12/240614},
                eprint = {https://www.biorxiv.org/content/early/2018/02/12/240614.full.pdf},
                journal = {bioRxiv}
            }

Ceiling

0.48.

Note that scores are relative to this ceiling.

Data: Rajalingham2018

240 stimuli match-to-sample task

Metric: i2n