https://arxiv.org/pdf/2609.12303
This paper proposes an architecture that bypasses tokenization, allowing the model to directly process raw bytes—essentially reading text character by character.
What they did:
- Trained small models (~1 billion / 1B parameters), some token-based and others byte-based
- Conducted part of the training by borrowing from/replicating larger models (ll