Troubleshooting

1) My submission didn't work. What happened?

Description: This is the most common question we get, and our FAQ provides an overview of what could have happened.

Cause: See FAQ page.

Fix: If your submission is not successful, you should get an email with a link that details what went wrong in your build. This is called the Console Log (or Build Log), and it details exactly what happened to cause your submission to fail. We included common Console Log errors below (along with their resolutions), but if your error is not on that list, feel free to reach out to the Brain-Score Community or directly to the Brain-Score Team.

2) SSL Error when trying to run models locally

Description: After implementing a Pytorch model and running score for the first time, you get:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)
        

Cause:: This can sometimes occur if Pytorch is currently offline, or having issues

Fix: Add the following lines in the very beginning of your code and try again:

import ssl
ssl._create_default_https_context = ssl._create_unverified_context
        

3) Console Log error: Server is unable to locate model weights (or another file)

Description: When checking your console (build) log for what might have gone wrong, you see an error akin to this:

ERROR:root:Could not run model mikes_model on benchmark Geirhos2021sketch-error_consistency because of [Errno 2] No such file or directory: '/rdma/vast-rdma/scratch/Sun/score_plugins_vision_env_142/vision/brainscore_vision/models/mikes_model/mikes_model.pth'
Traceback (most recent call last):
        

Cause: You are trying to save your weights to a local (relative) path, or for some reason the server cannot locate your .pth (weights) file. Sometimes, your working path on your machine might not be the same as the working path our server uses to execute runs.

Fix: We have written code that allows users to download from S3 buckets here, specifically the load_file_from_s3 function. Please use this code in place of your own if you are hosting your weights on S3. We hope to eventually add support for other cloud storage, like Google Drive.

4) Console Log error: No registrations found

Description: When checking your console (build) log for what might have gone wrong, you see an error akin to this:

ERROR:root:Could not run model mikes_model on benchmark ImageNet-C-noise-top1 because of No registrations found for mikes_model
        

Cause: You are trying to score a model (plugin) that does not exist yet in the Brain-Score ecosystem.

Fix: Please add your model to the model_registry in your submission package's __init__.py. See here (specifically Part 4) for more details.

5) Console Log error: Issue with Tensorflow 1.15

Description: When checking your console (build) log for what might have gone wrong, you see an error akin to this:

ERROR: Could not find a version that satisfies the requirement tensorflow==1.15
        

Cause: Usually this is caused by users trying to use Tensorflow >1.15. This error occurs when our package installer attempts to reconcile conflicting TF versioning requirements (server needs <=1.15, user requests >1.15). This can also happen if you force your model to use a specific Python version that is >3.7.

Fix: First, make sure you are not forcing your model to use a Python version that is >3.7. Currently, Brain-Score does not support Python version >3.7 or TF versions >1.15, but we will very soon, so stay tuned.

I am very confused as to how to add a benchmark into Brain-Score. What do I do?

Description: We recommend starting with our tutorial series, specifically tailored for benchmark contributors here. If you still are unsure, the more in depth documentation here is always a great place to visit as well.

Cause: General apprehension for Brain-Score's benchmark submission process.

Fix: See benchmark tutorial page for an overview. These tutorials should cover everything that is needed for contributing a benchmark, but in general the process is this: submit your stimuli and data, and then submit a file that outlines how you want your model to use Brain-Score in evaluating your subject data. Brain-Score will take care of everything else; you just can call our methods!

2) I am getting a 403/404/400 Error when trying to upload data to BrainIO.

Description: After trying to upload stimuli or data to BrainIO, you get an error saying something has failed with AWS, usually a "403 Access Denied" error or similar.

Cause:: You currently do not have the BrainIO credentials needed to upload.

Fix: Reach out to a Brain-Score team member to get your credentials.