\section*{List of Figures}
S1 The ESM3 architecture . . . . . . . . . . . 22
S2 Geometric Attention . . . . . . . . . . . . 25
S3 Structure tokenizer reconstruction quality . 32
S4 Visualization of structure tokenizer reconstructions
33
S5 Visualization of local neighborhoods which map to the same learned structure token . . 34
S6 pTM and pLDDT calibration . . . . . . . . 35
S7 Schematic of function tokenization . . . . . 35
S8 Function prediction benchmarking results . 36
S9 Visualization of noise schedules used . . . . 41
S10 Scaling curves for structure prediction . . . 43
S11 Conditional and unconditional Scaling behavior for each track 45
S12 Distribution of pTM and pLDDT . . . . . . 45
S13 Unconditional generation of high-quality and diverse proteins using ESM3 . . . . . . 47
S14 Generation of sequences using chain-of- thought . . . . . . . . . . . . . . . . . . . 48
S15 Prompting ESM3 to generalize beyond its training distribution 50
S16 Multimodal protein editing with ESM3 . . . 54
S17 Alignment improves model generations . . 57
S18 Randomly selected successful generations from the base model and finetuned model . 58
S19 Flow cytometry data confirms cells expressing esmGFP can be detected at the single cell level
63
S20 B8 Replication ….. 63
S21 Chromophore knockout mutations ….. 63 S22 Sequence identity of esmGFP ….. 64 S23 ESM3-open is a powerful predictor of struc- ture and function trained for open release ….. 66
\section*{List of Tables}
S1 Parameter details for different model configurations
24
S2 Training details for stage 2 training of an all-atom structure token decoder ….. 31 S3 Pre-training dataset statistics ….. 40 S4 Pre-training unique token statistics ….. 40 S5 Data augmentation and conditioning infor- mation applied to each dataset ….. 40 S6 Noise Schedules and Dropout Probabilities ….. 41 S7 Precision @ L ….. 44 S8 Protein structure prediction results ….. 44 S9 Negative log-likelihood of each track condi- tioned on other tracks ….. 44 S10 Functional motif definitions for conserved region ….. 50
S11 InterPro tags extracted from CAMEO test set proteins for prompting with fold specification
S12 Novelty and designability metrics. . . . . . 52
S13 Atomic coordination dataset . . . . . . . . 56
S14 Multiple sequence alignment of select GFP designs (B8, esmGFP) and reference proteins 65