ProtCompass_Embeddings

Frozen protein embeddings, input-audit records and label files for the paper ProtCompass: Biological Information and Predictive Utility in Protein Representations (anonymous submission to ICLR 2027).

Protein encoders are usually evaluated by structural probing, downstream prediction and concept erasure. These measurements can mix signal learned by the encoder with information that is already present in the input or that the evaluation adds. ProtCompass separates these sources. It audits which inputs each encoder actually reads, compares structural probe scores with non-learned descriptions of the same inputs, compares downstream scores with simple sequence baselines, and controls concept-erasure experiments for the representation capacity that is removed together with the target property. The paper evaluates 33 encoders on a probing suite of 21 probes built from CATH 4.3 domains and on ten downstream tasks.

• Code, result tables and analysis scripts: https://anonymous.4open.science/r/ProEmbedEva-C4F6
• Predicted structures for four downstream tasks: https://huggingface.co/datasets/Anonymoususer2223/ProtEnv

This repository holds the release that backs the paper: embeddings, labels, the input audit and the pretrained weights of the structure encoders. Its paths mirror the working directory of the code (for example extract/embeddings/<task>/<encoder>/ and checkpoints/). An earlier release was removed; see Earlier release (removed).

Contents

Sizes are in GB (10^9 bytes).

The current release has 3,040 files and 161.2 GB in total. The appendix names are section and subsection titles of the paper.

Encoders

The paper evaluates 33 encoders. They are declared as 9 sequence-based, 19 structure-based and 5 multimodal models. The analyses group them by a measured input audit, which finds that 10 encoders do not read coordinates (sequence-only) and 23 do (coordinate-reading). The only encoder whose measured group differs from its declared class is ProstT5, which is declared multimodal and read on its amino-acid track only.

The directory name is the key used in every path. Width is the width of the stored array. Atoms read comes from the atom audit (extract/encoder_atom_audit.csv); "side chain" marks the three encoders that the side-chain audit (extract/encoder_sidechain_audit.csv) finds to read atoms beyond CB.

Randomly initialised encoders use one fixed seed for all proteins; the hash of their weights is recorded in meta.json. Three further encoders were audited and are not released: EGNN (its embeddings do not change when the coordinates change), GeoBPE (its output changes under a rigid motion) and dMaSIF (all-zero embeddings). FoldVision has no extraction on remote homology, so it is scored on nine tasks.

Which layer is read

Every embedding is taken from the last representation the encoder computes before any task head or decoder. Where an encoder has residue states, a protein is the masked mean of its residue states. The rule of each encoder is recorded in the rule field of its meta.json.

Pretrained weights

checkpoints/ holds the local checkpoint files that six of the released structure encoders read. Every file is byte-identical to its upstream release, and loading it with the project's wrapper on CPU reproduces the weight hash (weights_sha256) recorded in the meta.json of the released embeddings. The upstream licence is stored next to each file, and checkpoints/SHA256SUMS lists the full hashes. checkpoints/README.md has a script that links the files to the paths the code reads.

The code paths are relative to the root of the code repository, whose protcompass/ folder mirrors this repository. The upstream repositories are github.com/dauparas/ProteinMPNN (commit 8907e66), github.com/BioinfoMachineLearning/GCPNet (commit 172733b) and github.com/KeatingLab/PottsMPNN (commit a6ccac3); a clone of GCPNet or PottsMPNN at that commit already holds its files at the paths the code reads.

Weights loaded from a public source, which this repository does not copy:

The first twelve are Hugging Face hub ids; the revisions are those recorded in the checkpoint_revision field of each meta.json.

• ESM3. The esm3-sm-open-v1 weights are distributed by EvolutionaryScale under the Cambrian Non-Commercial License Agreement, which does not allow redistribution, so they are not in this repository. Accept the licence and download them from the gated Hugging Face repository EvolutionaryScale/esm3-sm-open-v1. The code expects them under protcompass/esm3-sm-open-v1/ (weights file data/weights/esm3_sm_open_v1.pth).
• TFN-pre and IPA-pre. Their weights were taken from the models' released repositories and are not redistributed here. The meta.json of their embeddings records the SHA-256 prefix of the file that was used (17af58de61a4674c for TFN-pre, cd4ca8e7f61daba2 for IPA-pre). The code expects them at protcompass/checkpoints/tfn_pretrained/best.pt and protcompass/checkpoints/ipa_pretrained/best.pt.
• Randomly initialised encoders (SchNet, TFN, DimeNet, MACE, GearNet, IPA, GVP, GCPNet, CDConv, PottsMPNN) have no weights file. They are built with the fixed seed 20260919, and the hash of the resulting weights is in each meta.json.

Downstream tasks

Fourteen tasks were assembled. Four (contact prediction, mutation effect, PPI sites and metal binding) were excluded because their labels or metric were unsuitable (appendix "Tasks and analyses the paper evaluated and does not use"), and ten are scored. Every encoder reads the same rows of a task. For each task the embeddings cover the rows of the manifest with stage1 == 1.

The structure sources and split sizes are those of the paper's table "Structures and splits of the ten scored tasks". Every extracted row has a structure file whose sequence equals the task sequence (structure_coverage in meta.json).

File formats

Downstream embeddings: extract/embeddings/<task>/<encoder>/

• Row order equals the order of the manifest rows with stage1 == 1 and the same split. The id files are identical for every encoder of a task.
• Labels are in the label column of extract/manifest/<task>.csv: an integer class index (EC, localisation), a fold index (remote homology), a list of term indices written as a string (GO-MF, GO-BP), 0 or 1 (solubility, membrane versus soluble) or a real value (stability, fluorescence).
• PPI affinity: extract/pairs/ppi_affinity_pairs.csv has columns id, split, chain_a, chain_b, label. The paper represents a pair by concatenating the embeddings of chain_a and chain_b in that order.
• ref_composition/ in every task directory is not an encoder. It is the non-learned composition reference of the paper: 20 amino-acid frequencies, log length, length/600 and the fraction of residues outside the 20-letter alphabet (23 values).
• extract/manifest/<task>.csv has columns id, split, sequence, structure_path, length, label, stage1, plus align_path, struct_coverage (experimental and AlphaFold DB structures) or mean_plddt, chain_break (ESMFold structures).

Fields of meta.json (not every field is present for every encoder):

Probing embeddings: probing_embeddings/

The probing set is 19,461 CATH 4.3 domains with 2,823,525 residues. The same row convention holds for every probing directory in this repository.

FoldVision has no residue states, so probing_embeddings/foldvision/ holds pooled_fp32.npy only, and only the protein-level probes apply to it. Language models were run on the sequence of the residues kept in labels.h5; a discontinuous CATH domain is passed as one concatenated sequence.

probing_data/labels.h5 holds three groups. protein/ has one row per domain (id, length, CATH labels, split codes, protein-level targets). residue/ has one row per residue (amino acid, N, CA, C, O and CB coordinates with CB built from ideal geometry where it is absent, DSSP 3- and 8-state, relative solvent accessibility, B-factor, backbone angles, contact counts and other residue labels). pair/ has the sampled residue pairs with their distances. The split codes are split_superfamily (15,565 train, 1,948 validation, 1,948 test; no superfamily is shared between splits) and split_topology (15,378 / 2,159 / 1,924; the topology-disjoint split of the confirmation experiments), with 0 = train, 1 = validation and 2 = test. probing_data/splits/ lists the same splits as domain ids. The CATH domain structure files are public and are not redistributed here.

probing_embeddings_vframe/<encoder>_vframe/ follows the same layout. It stores the scalar channels together with the vector channels projected onto a per-residue frame built from N, CA and C (width 148 for GVP and the two GCPNets, 176 for MACE). The two GCPNet directories also hold residue_atommean_fp16.npy and pooled_atommean_fp32.npy, the same representation averaged over the heavy atoms of each residue. The frame definition is in each meta.json.

Modality ablation: extract/ab_modality/

Three extra sources hold the weights fixed and change only the input.

extract/ab_modality/embeddings/<task>/<source>/ follows the downstream format for all ten tasks, and extract/ab_modality/probing_embeddings/<source>/ follows the probing format. The audit rows of these sources are in extract/ab_modality/encoder_input_audit_ab.csv and encoder_input_audit_ab_shuf.csv. The full-input ESM3 and SaProt embeddings they are compared with are the esm3 and saprot entries of the main release.

Confirmation sets

Residue-level interface task: masif_site/

Interface residues rebuilt from MaSIF-site with a label defined from geometry: a residue of the target chain is an interface residue if any of its heavy atoms lies within 5 Å of a heavy atom of another chain. There are 3,327 chains (2,677 train, 297 validation, 353 test) and 647,332 residues. embeddings/<encoder>/ follows the probing format for 28 of the 33 encoders (SaProt, Foldseek, both GCPNets and FoldVision are not scored on this task). labels.h5 holds the residue labels (residue/interface) and the split (protein/split_masif); exclusions.csv lists the 35 entries that were removed.

Latent-space atlas: atlas/data_v4/

atlas.json holds every layer; the other files hold one task layer each (ec, go_mf, go_bp, localisation, homology, solubility, membrane, stability, fluorescence, affinity). Each layer is a stratified sample of about 2,000 proteins of the task's own rows. For every protein it gives the id, split and label; for every encoder the UMAP coordinates (k = 10, seed 42, Procrustes-aligned to ESM2-150M), the k-NN purity with its chance level and a five-fold linear probe accuracy; and between encoders the debiased linear CKA and k-NN overlap, both computed on the full embeddings. summary.json holds the per-layer summary. The atlas is built from the embeddings in extract/embeddings/.

Input audit

extract/encoder_input_audit.csv has one row per encoder and dataset (probing for the CATH set, stage1 for the downstream set). Each row gives the relative change of the pooled embedding over the same 32 proteins under a 1 Å coordinate jitter, a coordinate swap with another protein, a sequence shuffle and a rigid motion, the measured uses_coordinates, uses_residue_identity and invariant, the declared inputs, and a verdict. The file also holds the failing rows of EGNN and GeoBPE. extract/encoder_atom_audit.csv gives the change when each of the N, CA, C, O and CB coordinates is perturbed and the resulting atom_set. extract/encoder_sidechain_audit.csv gives the same for side-chain atoms.

Loading example

import numpy as np
import pandas as pd
from huggingface_hub import snapshot_download

repo = "Anonymoususer2223/ProtCompass_Embeddings"
task, encoder = "subcellular_localization", "esm2_650m"

# Download one encoder on one task and the task manifest (about 65 MB).
local = snapshot_download(repo, allow_patterns=[f"extract/embeddings/{task}/{encoder}/*",
                                                f"extract/manifest/{task}.csv"])
d = f"{local}/extract/embeddings/{task}/{encoder}"
splits = ("train", "valid", "test")
X = {s: np.load(f"{d}/{s}.npy") for s in splits}
ids = {s: open(f"{d}/ids_{s}.txt").read().split() for s in splits}

manifest = pd.read_csv(f"{local}/extract/manifest/{task}.csv")
manifest = manifest[manifest.stage1 == 1].set_index("id")
y = {s: manifest.loc[ids[s], "label"].to_numpy() for s in splits}

# A quick check with a logistic regression. The paper's heads (linear and MLP-2, selected on validation,
# three seeds) are in extract/score2.py of the code repository.
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler().fit(X["train"])
clf = LogisticRegression(max_iter=1000).fit(scaler.transform(X["train"]), y["train"])
print("test accuracy", clf.score(scaler.transform(X["test"]), y["test"]))

Per-residue probing embeddings are large (up to 14.5 GB for ESM2-3B), so open them with mmap_mode="r":

import h5py
# In allow_patterns a * also matches '/', so the top-level files are named one by one.
local = snapshot_download(repo, allow_patterns=["probing_embeddings/esm2_650m/*", "probing_embeddings/offsets.npy",
                                                "probing_embeddings/lengths.npy", "probing_embeddings/ids.txt",
                                                "probing_data/labels.h5"])
R = np.load(f"{local}/probing_embeddings/esm2_650m/residue_fp16.npy", mmap_mode="r")
offsets = np.load(f"{local}/probing_embeddings/offsets.npy")
lengths = np.load(f"{local}/probing_embeddings/lengths.npy")
with h5py.File(f"{local}/probing_data/labels.h5", "r") as f:
    k = 10
    rows = slice(offsets[k], offsets[k] + lengths[k])
    residue_embeddings = R[rows]          # (lengths[k], 1280)
    dssp3 = f["residue/dssp3"][rows]      # 0 = H, 1 = E, 2 = C
    split = f["protein/split_superfamily"][k]

Structures and datasets

The manifests give, for every row, the structure file the extraction read (structure_path, relative to the directory that holds the code repository's protcompass/ folder). The ESMFold structures of stability, fluorescence, solubility and PPI affinity are in the ProtEnv dataset under predicted_structures/; the last two components of structure_path (for example predictions_fluorescence/fluor_20897.pdb) are the member name in the corresponding archive. The experimental (PDB), AlphaFold DB and CATH structures are public and are not redistributed. The scripts that fetch them and build the manifests are in the code repository.

Earlier release (removed)

An earlier release, produced by an earlier extraction pipeline for a previous version of this work, was stored under embeddings/ and probing_IF/. The current paper does not use it. An audit in September 2026 found defects in it: structure encoders received placeholder or all-zero coordinates on several tasks, some entries were degenerate or mislabelled, and four of its tasks had defects in their labels or metric. It was removed from this repository. Every file the paper uses is in the current release.

License

The embeddings, labels, manifests, audit files and atlas data produced for this work are released under the MIT licence. The files under checkpoints/ keep their upstream licences: MIT for the ProteinMPNN, GCPNet and PottsMPNN files (the licence text is next to each file) and CC BY 4.0 for the GearNet weights (checkpoints/gearnet_pretrained/NOTICE.md).