In contrast, the IEA code indicates that an annotation is based entirely on computed heuristics. IEA coded annotations are very common, but relatively untrustworthy, so these are left out of our default selection.

For convenience, we organize evidence codes into three groups based on level of trustworthiness. These can be selected as default options in our form, and can be roughly described as "experimentally based annotations", "human reviewed annotations", and "all annotations". Based on our own experiments, most deep learning models get the highest performance using "human reviewed annotations".

Split Method

The split method for proteins determines how your dataset was split into training, validation, and testing. Our recommended setting is cluster50, in which proteins with over 50% similarity are always grouped within the same dataset. This is standard for most protein classification problems, and makes the problem setting more challenging. Random is the alternative setting in which proteins are divided blindly. A major benefit of random is that because splits are made truely randomly, class counts have a more even distribution between training, validation and testing. The cluster50 and random splits are pre-generated, and consistent between dataset requests. Other settings, such as for annotation quality, may change the number of proteins in a dataset, but only by filtering training, validation, and testing seperately. As long as the split method is consistent between datasets, models trained on one can be fairly evaluated on the other.

Date Filtering

We allow GO annotations to be filtered by date, largely so that users can build datasets compatible with previous CAFA competitions.

Namespace

Gene Ontology terms are divided into three distinct catagories of Molecular Functions, Biological Processes, and Cellular Components. These represent different aspects of gene properties, and are almost always modeled separately. GO Bench outputs separate datasets for each of these namespaces.

Propogate Annotations

Our Gene Ontology dataset maps proteins to the associated GO terms, but only actually stores the most specific term associated with a given protein. For instance, we might store an association between the protein Q13219 and the GO term "hexose biosynthetic process", but we wouldn't explicitely store an association between Q13219 and the more general term "metabolic process".

The propogate annotations option augments our dataset with these implicit associations by propogating known annotations up the Gene Ontology tree. This greatly expands the number of annotations provided, but is also somewhat redundant.

Files

GO_Bench returns a gzipped tar file containing training, validation, and testing data for all selected namespaces. The output can be unzipped with "tar -xzvf GO_benchmark_data.tar.gz", as shown in our ipython notebook tutorial. Once unzipped, it should have the following structure.

.tsv files within the dataset contain two tab-separated columns, one representing Uniprot protein IDs and the other listing GO IDs associated with the protein, separated by commas.
.json files list the GO terms found in training, validation, and testing sets. For completeness, testing sets always contain every GO ID associated with the given prot ids, but training and validation sets may have rare terms filtered out to improve training.

Benchmark Compatibility

Training and validation sets returned by our form are designed to any overlap with datasets from "Benchmarking Gene ontology Function Predictions Using Negative Annotations" (Vesztrocy, Dessimoz 2020). Datasets may also be made compliant with the CAFA Challenge by filtering by the proper maximum date.

Model Comparison