esm3.discussion.full5

==============================

We have found that language models can reach a design space of proteins that is distant from the space explored by natural evolution, and generate functional proteins that would take evolution hundreds of millions of years to discover. Protein language models do not explicitly work within the physical constraints of evolution, but instead can implicitly construct a model of the multitude of potential paths evolution could have followed.

Language models can help us explore a wider range of proteins than what has been discovered through natural evolution. They can generate functional proteins that would take evolution a long time to find. These models don't have to follow the same physical constraints as evolution, but they can still create a model of the many possible paths evolution could have taken.

User:

Proteins can be seen as existing within an organized space where each protein is neighbored by every other that is one mutational event away (54). The structure of evolution appears as a network within this space, connecting all proteins by the paths that evolution can take between them. The paths that evolution can follow are the ones by which each protein transforms into the next without the collective loss of function of the system it is a part of.

Proteins can be thought of as existing in a specific area where they are surrounded by other proteins that are only one small change away from them. The way that proteins evolve can be visualized as a network within this space, connecting all proteins through the paths that evolution can take between them. These paths represent the ways in which one protein can transform into another without causing harm to the larger system they are a part of.

User:

It is in this space that a language model sees proteins. It sees the data of proteins as filling this space, densely in some regions, and sparsely in others, revealing the parts that are accessible to evolution. Since the next token is generated by evolution, it follows that to solve the training task of predicting the next token, a language model must predict how evolution moves through the space of possible proteins. To do so it will need to learn what determines whether a path is feasible for evolution.