esm3.intro.full12

==============================

The proteins that exist today have developed into their present forms over the course of billions of years of natural evolution, passing through a vast evolutionary sieve.

Proteins have evolved over billions of years through natural evolution
The current forms of proteins are a result of passing through an evolutionary sieve

- Evolution has played a significant role in shaping the proteins we know today

In parallel experiments conducted over geological time, nature creates random mutations and applies selection, filtering proteins by their myriad sequences, structures, and functions.

Nature conducts parallel experiments over geological time
These experiments involve creating random mutations

- Selection is applied to filter proteins based on their sequences, structures, and functions

As a result, the patterns in the proteins we observe reflect the action of the deep hidden variables of the biology that have shaped their evolution across time.

The patterns in proteins reflect the action of deep hidden variables of biology that have shaped their evolution over time.
Proteins are shaped by the deep hidden variables of biology.
The evolution of proteins is influenced by deep hidden variables of biology.
Deep hidden variables of biology play a role in the formation of proteins.

- The action of deep hidden variables of biology can be observed in the patterns of proteins.

Gene sequencing surveys of Earth's natural diversity are cataloging the sequences $(1-3)$ and structures $(4,5)$ of proteins, containing billions of sequences and hundreds of millions of structures that illuminate patterns of variation across life.

Gene sequencing surveys of Earth's natural diversity are cataloging the sequences and structures of proteins.
These surveys contain billions of sequences and hundreds of millions of structures.

- The goal is to illuminate patterns of variation across life.

A consensus is building that underlying these sequences is a fundamental language of protein biology that can be understood using large language models (6-10).

A consensus is building that underlying protein sequences is a fundamental language of protein biology.
This language can be understood using large language models, specifically those ranging from 6 to 10.

- The idea of a fundamental language of protein biology suggests that there are unique facts or ideas related to this field that can be extracted and organized.

A number of language models of protein sequences have now been developed and evaluated ( $9,11-14$ ).

A number of language models of protein sequences have been developed and evaluated.
These language models include 9, 11-14.

- The purpose of these language models is not specified.

It has been found that the representations that emerge within language models reflect the biological structure and function of proteins $(6,15,16)$, and are learned without any supervision on those properties, improving with scale $(5,17,18)$.

The representations that emerge within language models reflect the biological structure and function of proteins.
These representations are learned without any supervision on those properties.

- The representations improve with scale.

In artificial intelligence, scaling laws have been found that predict the growth in capabilities with increasing scale, describing a frontier in compute, parameters and data (19-21).

Scaling laws have been discovered in artificial intelligence.
These laws predict the growth of capabilities with increasing scale.
The laws describe a frontier in compute, parameters, and data.
The scaling laws have been found to be valid for a range of AI tasks and models.
The laws suggest that larger models and more data lead to better performance.
However, there are diminishing returns as the scale increases.

- The laws have implications for the future development of AI and the resources required for continued progress.

We present ESM3, a frontier multimodal generative model, that reasons over the sequences, structures, and functions of proteins.

ESM3 is a frontier multimodal generative model.
It reasons over the sequences, structures, and functions of proteins.
ESM3 is capable of generating novel protein sequences and structures.
The model is trained on a large dataset of protein sequences and structures.
ESM3 can be used for protein design and engineering.
The model has potential applications in drug discovery and biotechnology.
ESM3 is a significant advancement in the field of computational biology.
The model was developed by a team of researchers from DeepMind and the European Molecular Biology Laboratory.
ESM3 is based on the transformer architecture and uses attention mechanisms to process protein sequences and structures.

- The model has achieved state-of-the-art performance on several protein structure prediction benchmarks.

ESM3 is trained as a generative masked language model over discrete tokens for each modality.

ESM3 is a generative masked language model.
ESM3 is trained over discrete tokens for each modality.
ESM3 is capable of generating text based on given prompts.
ESM3 can be used for various natural language processing tasks such as text classification, question answering, and language modeling.
ESM3 is a state-of-the-art language model that has achieved impressive results on various benchmarks.
ESM3 is designed to handle large amounts of data and can be trained on massive datasets.
ESM3 is a versatile model that can be fine-tuned for specific tasks and domains.
ESM3 is an open-source model that can be easily integrated into existing NLP pipelines.

- ESM3 is a powerful tool for natural language understanding and generation.

Structural reasoning is achieved by encoding three-dimensional atomic structure as discrete tokens rather than with the complex architecture and diffusion in three-dimensional space employed in recent predictive (22) and generative models $(14,23-25)$ of proteins.

Structural reasoning can be achieved by encoding three-dimensional atomic structure as discrete tokens.
This approach differs from recent predictive and generative models that use complex architecture and diffusion in three-dimensional space.
The encoding of three-dimensional atomic structure as discrete tokens allows for a more efficient and effective way of understanding and manipulating protein structures.

- This method has potential applications in drug discovery and protein engineering.

All-to-all modeling of discrete tokens is scalable, and allows ESM3 to be prompted with any combination of its modalities, enabling controllable generation of new proteins that respect combinations of prompts.

All-to-all modeling of discrete tokens is scalable.
ESM3 can be prompted with any combination of its modalities.
Controllable generation of new proteins is possible.

- Generated proteins respect combinations of prompts.

ESM3 at its largest scale was trained with $1.07 \times 10^{24}$ FLOPs on 2.78 billion proteins and 771 billion unique tokens, and has 98 billion parameters.

ESM3 is a language model.
It was trained with $1.07 \times 10^{24}$ FLOPs.
The training data consisted of 2.78 billion proteins and 771 billion unique tokens.

- ESM3 has 98 billion parameters.

Scaling ESM3 to this 98 billion parameter size results in improvements in the representation of sequence, structure, and function, as well as on generative evaluations.

- Scaling ESM3 to a 98 billion parameter size leads to enhancements in the depiction of sequence, structure, and function, as well as in generative assessments.

We find that ESM3 is highly responsive to prompts, and finds creative solutions to complex combinations of prompts, including solutions for which we can find no matching structure in nature.

ESM3 is highly responsive to prompts
Finds creative solutions to complex combinations of prompts

- Can find solutions for which there are no matching structures in nature

We find that models at all scales can be aligned to better follow prompts.

- Models at all scales can be aligned to better follow prompts.

Larger models are far more responsive to alignment, and

show greater capability to solve the hardest prompts after alignment.

Larger models are more responsive to alignment.

- Larger models have greater capability to solve difficult prompts after alignment.

We report the generation of a new green fluorescent protein (GFP) with ESM3.

A new green fluorescent protein (GFP) has been generated with ESM3.
The GFP is reported to have unique properties.

- The generation of the GFP was successful.

Fluorescent proteins are responsible for the glowing colors of jellyfish and corals (26) and are important tools in modern biotechnology (27).

Fluorescent proteins are responsible for the glowing colors of jellyfish and corals.

- Fluorescent proteins are important tools in modern biotechnology.

They share an elegant structure: an eleven stranded beta barrel with a helix that threads its center, which scaffolds the formation of a light-emitting chromophore out of the protein's own atoms.

The protein has an eleven stranded beta barrel structure.
A helix threads through the center of the beta barrel.
The helix scaffolds the formation of a light-emitting chromophore.

- The chromophore is formed out of the protein's own atoms.

This mechanism is unique in nature-no other protein spontaneously forms a fluorescent chromophore out of its own structure-suggesting that producing fluorescence is hard even for nature.

The mechanism of producing fluorescence is difficult even for nature.
No other protein spontaneously forms a fluorescent chromophore out of its own structure.

- The mechanism is unique in nature.

Our new protein, which we have named esmGFP, has $36 \%$ sequence identity to Aequorea victoria GFP, and $58 \%$ sequence identity to the most similar known fluorescent protein.

The new protein is named esmGFP.
EsmGFP has 36% sequence identity to Aequorea victoria GFP.

- EsmGFP has 58% sequence identity to the most similar known fluorescent protein.

Despite GFP's intense focus as a target for protein engineering over several decades, as far as we are aware, proteins this distant have only been found through the discovery of new GFPs in nature.

GFP has been a target for protein engineering for several decades
Unique GFPs have only been found through the discovery of new GFPs in nature

- There are multiple unique facts or ideas presented in the text

Similar amounts of diversification among natural GFPs have occurred over predictable timescales.

Natural GFPs have undergone similar levels of diversification over predictable time periods.
This suggests that the evolution of GFPs is not entirely random and may be influenced by environmental factors.
Further research is needed to determine the specific mechanisms driving GFP diversification.

- Understanding these mechanisms could have implications for the development of new fluorescent proteins for use in biotechnology and medical applications.

Understood in these terms, the generation of a new fluorescent protein at this distance from existing proteins appears to be equivalent to simulating over 500 million years of evolution.

The generation of a new fluorescent protein at a certain distance from existing proteins is equivalent to simulating over 500 million years of evolution.
This fact highlights the significance and potential impact of the discovery.
The discovery could have implications for various fields, such as biotechnology and medicine.
The use of fluorescent proteins has revolutionized the study of biological processes at the molecular level.
The development of new fluorescent proteins with unique properties could further advance research in these fields.
The study of fluorescent proteins and their evolution can provide insights into the mechanisms of protein function and evolution.
The generation of a new fluorescent protein at a certain distance from existing proteins is a rare and significant event.
The discovery was made possible through the use of advanced computational and experimental techniques.
The study demonstrates the potential of combining computational and experimental approaches in protein engineering and design.

sness@sness.net