OpenAI's latest model Astra uses a byte-based architecture: the Transformer layer is repeatedly computed on the same data.

Perceiving AI News Flash: OpenAI's upcoming release Astra has adopted a recurrent depth architecture, also known as a looped transformer. In a standard Transformer, each token generation passes through a fixed set of layers; Astra allows the same piece of information to go through the same group of layers repeatedly, undergo additional rounds of computation, and then produce an output.

This architectural approach was previously explored last year. The Seed team introduced Ouro, a Looped Language Model that similarly loops a set of Transformer layers to embed more computation within the model. Instead of consistently elongating the chain of thought, the model can increase the inference computation. As a result, smaller models can leverage more computation to achieve performance close to that of larger models.

However, this architecture also presents a security concern. Since some inference occurs within internal hidden states that are not visible to humans in the complete text record, it becomes more challenging to inspect the model for violations through the chain of thought. Therefore, OpenAI has restricted the extent to which Astra utilizes recurrent depth, ensuring that it maintains a readable chain of thought and is preparing to incorporate additional CoT monitoring.