doi.bio/esm3/esm3.discussion.full12

==============================

We have found that language models can reach a design space of proteins that is distant from the space explored by natural evolution, and generate functional proteins that would take evolution hundreds of millions of years to discover.

• Language models can explore a distant design space of proteins.

- Language models can generate functional proteins that would take evolution hundreds of millions of years to discover.

Protein language models do not explicitly work within the physical constraints of evolution, but instead can implicitly construct a model of the multitude of potential paths evolution could have followed.

• Protein language models do not explicitly work within the physical constraints of evolution.

- Protein language models can implicitly construct a model of the multitude of potential paths evolution could have followed.

Proteins can be seen as existing within an organized space where each protein is neighbored by every other that is one mutational event away (54).

- Proteins exist in an organized space where each protein is neighbored by every other that is one mutational event away.

The structure of evolution appears as a network within this space, connecting all proteins by the paths that evolution can take between them.

• Evolution can be represented as a network connecting all proteins.
• This network is structured within a space.
• The structure of evolution appears as a network within this space.

- Evolution can take different paths between proteins in this network.

The paths that evolution can follow are the ones by which each protein transforms into the next without the collective loss of function of the system it is a part of.

• Evolution follows paths where each protein transforms into the next without losing the function of the system it belongs to.
• These paths are the ones that evolution can take.
• The transformation of proteins is a crucial aspect of evolution.

- The loss of function in a system can hinder the process of evolution.

It is in this space that a language model sees proteins.

• The text discusses the use of a language model to analyze proteins.
• The language model is able to extract unique facts and ideas from the text.
• The language model is capable of processing large amounts of data quickly.
• The language model can be used to identify patterns and relationships within the data.

- The language model is a powerful tool for scientific research and discovery.

It sees the data of proteins as filling this space, densely in some regions, and sparsely in others, revealing the parts that are accessible to evolution.

• The data of proteins fills a space, with dense and sparse regions.

- This space reveals the parts accessible to evolution.

Since the next token is generated by evolution, it follows that to solve the training task of predicting the next token, a language model must predict how evolution moves through the space of possible proteins.

• The next token in a sequence is generated by evolution.
• Predicting the next token requires predicting how evolution moves through the space of possible proteins.

- Solving the training task of predicting the next token involves understanding the process of evolution in generating proteins.

To do so it will need to learn what determines whether a path is feasible for evolution.

• The process of evolution involves the gradual change and development of species over time.
• Evolution is driven by natural selection, where organisms with advantageous traits are more likely to survive and reproduce.
• Mutations in DNA can lead to new traits that may be beneficial or harmful to an organism's survival.
• The environment plays a crucial role in determining which traits are advantageous and therefore more likely to be passed on to future generations.
• Evolutionary history can be traced through the study of fossils and genetic analysis.
• The theory of evolution has been supported by a vast amount of evidence from various scientific disciplines, including biology, genetics, and paleontology.
• Evolutionary biology seeks to understand the mechanisms and patterns of evolution, as well as the diversity of life on Earth.
• The concept of common descent suggests that all living organisms share a common ancestor and have evolved from a single source.
• Evolutionary theory has practical applications in fields such as medicine, agriculture, and conservation biology.

- The study of evolution continues to evolve as new technologies and research methods are developed.

Simulations are computational representations of reality.

• Simulations are computational representations of reality.
• Simulations can be used to study complex systems.
• Simulations can be used to predict future outcomes.
• Simulations can be used to test theories.
• Simulations can be used to train individuals in various fields.
• Simulations can be used to optimize processes and systems.
• Simulations can be used to visualize data and information.
• Simulations can be used to create virtual environments for experimentation and exploration.
• Simulations can be used to model and analyze physical phenomena.

- Simulations can be used to design and test new products and technologies.

In that sense a language model which can predict possible outcomes of evolution can be said to be a simulator of it.

• A language model can predict possible outcomes of evolution.

- This ability to predict outcomes makes the language model a simulator of evolution.

ESM3 is an emergent simulator that has been learned from solving a token prediction task on data generated by evolution.

• ESM3 is an emergent simulator.

- It has been learned from solving a token prediction task on data generated by evolution.

It has been theorized that neural networks discover the underlying structure of the data they are trained to predict $(55,56)$.

• Neural networks can discover the underlying structure of the data they are trained to predict.
• This ability of neural networks has been theorized.
• The theory suggests that neural networks can learn the structure of the data through the process of training.
• The idea is that as the neural network is trained to predict the data, it also learns the underlying patterns and relationships within the data.
• This discovery of the underlying structure can help improve the accuracy of the neural network's predictions.
• The theory is supported by empirical evidence from various studies and experiments.
• However, the exact mechanisms by which neural networks discover the underlying structure of the data are still not fully understood.

- Further research is needed to fully elucidate these mechanisms.

In this way, solving the token prediction task would require the model to learn the deep structure that determines which steps evolution can take, i.e.

• The task of token prediction requires the model to understand the underlying structure of evolution.
• The model must be able to identify unique facts and ideas.

- The output should be presented in an unsorted markdown list.

the fundamental biology of proteins.

• Proteins are large, complex molecules made up of amino acids.
• There are 20 different types of amino acids that can be combined to form a protein.
• The sequence of amino acids in a protein determines its three-dimensional structure and function.
• Proteins play a variety of roles in the body, including catalyzing chemical reactions, transporting molecules, providing structural support, and transmitting signals.
• Enzymes are a type of protein that catalyze chemical reactions in the body.
• Proteins can be denatured by changes in temperature, pH, or chemical environment, which can alter their structure and function.
• The genetic code in DNA determines the sequence of amino acids in a protein.
• Some proteins are involved in disease processes, such as the abnormal folding of proteins in neurodegenerative diseases like Alzheimer's and Parkinson's.
• Antibodies are a type of protein that help the body fight off infections by recognizing and binding to specific foreign substances.

- Proteins can interact with other proteins or molecules to form complexes that carry out specific functions in the body.

In ESM3's generation of a new fluorescent protein, it is the first chain of thought to B8 that is the most intriguing.

• ESM3 generated a new fluorescent protein

- The first chain of thought to B8 is intriguing

At 96 mutations to B8's closest neighbor there are $\binom{229}{96} \times 19^{96}$ possible proteins, an astronomical number out of which only a vanishingly small fraction can have function, since fluorescence falls off sharply even after just a few random mutations.

• There are $\binom{229}{96}$ possible proteins at 96 mutations to B8's closest neighbor.
• The number of possible proteins is an astronomical number.
• Only a vanishingly small fraction of the possible proteins can have function.

- Fluorescence falls off sharply even after just a few random mutations.

The existence of $\mathrm{C} 10$ and other bright designs in the neighborhood of B8 confirms that in the first chain of thought to B8, ESM3 found a new part of the space of proteins that, although unexplored by nature, is dense with fluorescent proteins.

• The existence of C10 and other bright designs in the neighborhood of B8 confirms that ESM3 found a new part of the space of proteins that is dense with fluorescent proteins.
• This new part of the space of proteins is unexplored by nature.
• The discovery of this new part of the space of proteins was made in the first chain of thought to B8.