Scan QR code to follow QbitAI

< img id=“wx_img” src=“https://www.qbitai.com/wp-content/uploads/imgs/qbitai-logo-1.png” width=“400” height=“400”>
Luyu 2026-10-20 20:50:34 Source: QbitAI
Hardwiring Native Memory into Transformers
Luyu reporting from the Office
QbitAI | Public Account QbitAI
The AI community has recently set its collective sights on three letters: RSI.
Hinton’s RSI paper features 22 co-authors, with Yoshua Bengio, OpenAI Chief Scientist Jakub Pachocki, Anthropic co-founder Jack Clark on board—practically half the AI elite in one room.
While everyone turns their gaze toward AI self-improvement, a more fundamental question surfaces:
How does one round of improvement become the starting point for the next?
Effective strategies must be preserved, and past missteps must be recognized. The linchpin holding all this together is memory.

Conventional approaches are straightforward: hook AI up to an external experience repository, allowing past operations and lessons learned to be stored and retrieved.
MemTensor’s exploration in this domain, however, offers a completely different answer: native memory.
What is native memory? It means turning the retention, updating, and retrieval of experience into an intrinsic, learnable capability of the model itself.
When memory takes root directly within the model, it forms a memory-native model.
Their newly open-sourced MemT 1.0 series demonstrates a concrete path:
It encodes experiences into internal dynamic states, learns how to write, update, and read through training, and finally routes these states through a dedicated memory branch into downstream computations.
Compared to external memory, it preserves full capability while offering a much higher ceiling for future scaling.
This also marks the first engineering validation of a memory-native model scaling near 300B parameters.
(Paper link:
https://arxiv.org/abs/2610.26825)
Memory may be intangible, but it serves as the rock-solid foundation for model growth.
From early demands for models to merely retain information to effectively leveraging past experience, the demands on memory keep escalating, pushing it from behind the scenes directly into the spotlight.
Putting Memory Inside the Model
MemT 1.0 has launched with two distinct variants:
MemT-1.0-Nano packs 36B total parameters with 3B activated, built on the base model Qwen3.6-35B-A3B; MemT-1.0-Flash pushes the envelope further to 288B parameters with 17B activated, powered by a post-trained DeepSeek-V4-Flash foundation.
Their shared architectural shift is the addition of a parallel memory computation pathway alongside existing attention mechanisms in selected Transformer blocks.

Let’s illustrate with a simple example.
When tackling a new task, external memory approaches typically inject raw text memories, retrieved evidence, or abstract methods back into the prompt context, letting the model reuse experience.
The retrieval and synthesis pipeline can also incorporate learning mechanisms.
In contrast, memory-native models like MemT 1.0 delve deeper into the model’s inner workings:
Experiences are encoded into updateable matrix states; during answer or action generation, the model reads these states via a dedicated memory branch, fusing the retrieved output with native attention outputs to steer downstream token generation.
The distinction lies in how experience is represented and through what computational process it exerts influence.
The benefits are substantial: integrating read/write operations into training facilitates smoother procedural reuse. The trade-off, however, is that precise deletion, interpretability, and state recovery require extra mechanisms.
External text remains much easier to inspect and trace back. For now, each approach has its own strengths and weaknesses.

The real difference becomes apparent when evaluating how models actually perform after leveraging experience.
Take a developer’s daily bread and butter: code bug fixing.
Receive bug → locate code → apply fix—that’s the standard loop. But if the issue isn’t reproduced early or boundary conditions aren’t verified, fixes fall flat and spawn cascade regressions from one generation to the next.
Take SWE-bench Verified as an example, where one task involves Django’s parse_http_date function, requiring a fix for two-digit year parsing logic.
The research team compared execution traces across three setups: No Experience, Text Experience, and Memory-Native.
Without experience, once the agent located the bug, it quickly began altering parsing logic, only attempting targeted reproduction and validation in later steps. Both experienced groups, on the other hand, constructed reproduction tests prior to editing code.
Notably, MemT-1.0-Nano re-examined an edge condition after its initial modification, refined the implementation, and ran extra validation.
Clearly, experience altered the execution sequence and actively participated in in-task verification and correction.
Across a broader sample of 500 SWE-bench Verified tasks (each run independently three times), MemT-1.0-Nano achieved an average success rate of 69.73% at critical checkpoints, compared to 66.07% for the baseline without experience and 69.13% for the text experience group.
The percentage of tasks succeeding across all three runs rose from 54.00% (no experience) to 62.80%, compared to 61.80% for text experience.

On SWE-bench Multilingual, under identical trigger configurations, the native memory group achieved an average success rate of 63.56%, versus 60.00% without experience and 63.00% with text experience.
In other words, procedural experience retained through native memory performs on par with explicit text memory, with advantages manifesting primarily in average success rate and repeated execution reliability.
To visualize this, consider a pixel-art maze exploration demo.
Same maze, same movement speed: on the left sits an agent without native memory; on the right is MemT 1.0 equipped with native memory.
In Round 1, both agents explore uncharted paths, and both hit dead ends.
The difference emerges once round one finishes: successful paths and dead ends to avoid are collated, archived, and loaded into MemT 1.0’s memory to guide the next exploration.
In Round 2, starting from the same point, the left agent continues to redundantly probe dead-end branches, while the right agent leverages previous experience to bypass known dead ends and follow the viable route straight to the goal.

Video link: https://mp.weixin.qq.com/s/DTdaGUk8o0dwkx1wxnnbYw
Paths carved out in the previous round directly shape how the next round unfolds—a tangible demonstration of what native memory brings to the table.
Consider an information updating scenario closer to daily enterprise workflows.
An exhibition originally scheduled for Saturday at 3:00 PM was officially rescheduled to Sunday at 4:00 PM. Later, someone casually suggested moving it to Monday, though no official decision was finalized.
Such fluctuating, incremental information poses a notorious challenge for AI.
Merely memorizing the latest timestamp isn’t enough; the model must distinguish between official decisions and casual suggestions while preserving unchanged details.
Across multiple conversational turns, MemT-1.0-Nano successfully tracked the nuances, accurately reporting the original schedule, the officially updated time, and confirming that other arrangements remained unchanged.

Across a comprehensive evaluation spanning 11 memory benchmarks, both Flash and Nano demonstrated standout performance.
Achieving average scores of 50.6 and 47.0 respectively, they claimed the top two spots among evaluated parametric memory methods, leading other models by a wide margin.

Frankly speaking, text-based memory systems still hold an overall edge, particularly in granular detail retrieval over long histories.
What MemT 1.0 presents today is an initial computational realization of this memory-native paradigm; yet, its real-world performance already rivals text memory, pointing to immense scaling potential ahead.
What Makes Memory “Native”?
This raises the question: if experience isn’t fed into the task context, where does the model draw upon it?
This is precisely the core of MemT 1.0’s architectural innovation.
The native approach in MemT 1.0 first encodes relevant experience into internal dynamic states. When generating downstream actions, incoming queries read from this state via dedicated memory branches, fusing the retrieved signals with the standard attention output.
In its parametric design, MemT 1.0 preserves the base model’s original compute pipeline while integrating a native memory branch into selected Transformer layers, spanning three tiers:
- Backbone Weights: House the model’s existing general capabilities and remain frozen during memory training;
- Hyper-Memory: Trainable parameters like read/write projections and gates that dictate precisely how memory is encoded, updated, and retrieved;
- Local Memory: Dynamic matrix states that store newly ingested information across interactions.
With this decoupled hierarchy, the model can assimilate fresh experiences without needing full-backbone fine-tuning every time; only the dynamic states corresponding to the task trajectory change.

The read/write mechanics are similarly sophisticated.
During writes, hidden states undergo projection to yield memory keys and values; erase, write, and forget gates collectively modulate existing matrix associations. During reads, the current input maps into memory queries to fetch relevant information, which then passes through normalization and output gating to influence subsequent generation.
The update mechanism builds upon Gated DeltaNet-2 with token-by-token generation and custom stabilization.
(Insider note: The author of Gated DeltaNet-2 also participated in relevant discussions.)
It triggers at the end of each turn, adjusting stored memory associations via gating, while maintaining token-order state updates within turns.
Ablation studies validate this design: replacing token-level updates with turn-level pooling caused Nano’s average benchmark score to drop from ~47.0 to 41.0.
This six-point drop clearly demonstrates that continuous token-level updates handle memory decay more gracefully and promptly correct within-turn associations.

Yet as memory content continuously expands, what happens when model capacity saturates?
Here, MemT 1.0 introduces MoS (Mixture-of-Storage), built upon MoE concepts.
MoS expands a single memory-native module into multiple memory experts, learning specialized storage and utilization routing. Crucially, it employs dense fusion: all memory experts participate in computation, with routing weights governing individual contributions.
This fundamentally differs from the sparse expert routing seen in conventional MoE backbones.

However, architecture is merely the starting line.
To teach the model how to effectively leverage this pathway, the team curated a 34-billion-token memory training corpus, comprising 20B tokens for continual pre-training and 14B tokens for post-training.
The format for continual pre-training is standardized: long texts are partitioned into sequential chunks written into memory, after which the model relies on the memory state to answer queries.
After processing each chunk, the model updates its state for the next segment.
Post-training further aligns task outcomes across three data categories: memory-native language modeling data, memory storage and utilization data (e.g., summarization, continuation, compression), and specialized task data designed to reinforce long-horizon memory.
A distinctive technique here is that the previously frozen backbone serves as a teacher model, having access to full context history, while MemT 1.0 acts as the student, observing only its own memory states. Off-policy distillation then aligns the student’s distribution with the teacher’s.

Taken together, the core tenets of native memory come into focus: encoding history into states is merely step one; mastering how to retain, update, and retrieve completes the picture of a memory-native model.
The ultimate value of this pathway will need to be judged holistically across experience reuse efficiency, computational overhead, and long-term operating conditions.
From Continual Learning to RSI: Memory Unlocks New Horizons for Capability Growth
Some may ask: Why pursue native memory at all?
Existing external memory solutions are already mature, backed by robust practical use cases. In contrast, native memory is in its infancy, and its current performance is roughly comparable.
Not so fast! Native memory possesses a unique attribute:
It turns the representation and read/write mechanics of experience into explicit targets for model capability scaling—what the MemT team terms “Memory Scaling,” establishing a third growth curve alongside inference compute scaling.

Image generated by AI
In external setups, one judges the value of experience by checking whether retrieved text improves task outcomes. With MemT 1.0, one can ask further: for the exact same experience, can an alternate internal representation, update rule, or read mechanism unlock superior performance?
This equips memory optimization with a dedicated set of trainable and architecturally tunable variables.
Admittedly, this does not yet prove that native memory’s long-term ceiling exceeds that of external memory, but training and expanding capacity around internal states clearly yields incremental capability gains.
The next step is verifying whether these gains generalize beyond the current training distribution to sustain performance across longer interactions and novel tasks.
Returning to RSI mentioned at the outset:
Cracking native memory brings substantial dividends to continual learning and RSI.
As is widely recognized, RSI consists of three core building blocks: memory, feedback, and verification/update mechanisms. Memory ensures effective outcomes carry forward, while the latter two evaluate which adaptations warrant inclusion in the next iteration.
MemT 1.0 directly tackles experience retention and reuse, validating that internal memory can guide ongoing action—completing the first, and arguably most critical, piece directly within the model architecture.

Image generated by AI
Looking ahead, experience accumulated in local memory might crystallize into hyper-memory parameters via subsequent training, which in turn enhances the model’s native memory management.
Once this feedback loop closes, systems could potentially leverage logged outcomes and methodologies to spark cross-task intelligence explosion—the ultimate frontier of RSI.
Beyond theoretical benefits, native memory dramatically empowers enterprises in accumulating institutional knowledge.
Current discussions around enterprise AI predominantly obsess over early deployment phases—whether the current model is usable and what immediate tasks it can handle.
While valid, companies quickly discover through prolonged adoption that institutional knowledge accumulation is equally paramount.
Long-term enterprise deployment invariably involves project rules, operational sequences, and validation protocols. If every new task requires restating identical constraints and relearning identical pitfalls, prior investments fail to translate into compounding returns.
For these repetitive workflows, how many times validated experience can be reused should stand alongside single-run model performance as a cornerstone of enterprise AI evaluation.
Native memory provides the model-level substrate for this continuity.
Under an identical 128K context window, Nano achieves approximately 1.86x the end-to-end inference speed of full-context baselines; for equivalent history lengths, the persistent state footprint per trajectory is just 1/66th that of standard KV caches.

In plain terms, this delivers immediate, tangible cost reduction.
State budgets for extended multi-turn sessions become much easier to forecast, and multiple concurrent queries can reuse pre-built states, amortizing upfront preparation overhead.
To be fair, native memory does not guarantee superior quality on every task compared to external memory. However, at comparable quality thresholds, it systematically trims rework through low-level procedural reuse—a tangible bottom-line benefit for enterprise operations and maintenance.
In essence, native memory and external memory are not mutually exclusive. A synergistic division of labor may yield even greater dividends.
External text logs remain indispensable for querying exact verbatim transcripts, sources, or version diffs; meanwhile, native mechanisms can be deployed whenever procedural experience needs to be invoked during downstream execution.
Ultimately, when effective progress can be preserved and iteratively validated across novel tasks, continual self-improvement finally gains an accumulative foundation.
Through comprehensive hierarchical memory management and scheduling, global read/write efficiency can be maximized. Today, the release of memory-native models makes such engineering deployments and implementations a practical reality.
The value of memory in driving recursive self-improvement will only become more evident through sustained, long-term validation.
GitHub link:
https://github.com/MemTensor/MemT-1.0
HuggingFace link:
https://huggingface.co/collections/MemTensor/memt-1
All rights reserved. Unauthorized reproduction or use in any form is prohibited; violators will be prosecuted.
- Breaking! DeepSeek Releases New Model, Hyper-Cost-Efficiency Obliterates Everything2026-10-17
- This Double 11, Bailing’s New Model Gets Down to Work Without the Burnout2026-10-15
- Claude Solves 20-Year Cryptographic Cold Case in 30 Minutes: Academia Went Down the Wrong Track for Decades2026-10-17
- Silicon Valley AI Reality Check: Whoever Has Compute Calls the Shots2026-10-11
Scan QR code to share on Moments
Related Reads
PaddlePaddle Framework Steps Up to 3.0! Deep Dive into Five Major Features Including “Unified LLM Training and Inference”
Providing robust support across industry, academia, and research
Shisan 2024-08-01
New Move in LLMs by Tencent Cloud! Vector Database Launched as Standalone Product, Available in August
Delivering billion-scale vector retrieval capabilities
Mingmin 2023-07-05
Testing Personality, Depression, and Cognitive Patterns with LLMs! Measuring Psychological Traits via Game Plot Progression | Tsinghua Research
Customized generation of interactive narrative games
Xifeng 2024-03-13
Meituan Acquires 100% of Light Year Away for 2 Billion RMB! Wang Xing Backs Tsinghua Bunkmate, with $280M Cash on Hand
Current team size around 70 people
Baijiao 2023-06-29
Hardest Dialect Wenzhounese Conquered! China Telecom Speech LLM Supports 30 Dialects—Something GPT-4o Can’t Do
Speech LLM competition reaches new heights
Baijiao 2024-05-28
World-Class Multimodal LLM Open-Sourced! 01.AI and Kai-Fu Lee Deliver Again
Leading across major Chinese and English benchmarks
Xifeng 2024-01-22
Popular Articles
Tracking new AI trends, reporting breakthrough moments in tech
Copyright © Beijing Geek Partner Technology Co., Ltd. QbitAI. All Rights Reserved. Beijing ICP Filing No. 17005886-1