Skip to content
Main Site News Console

First Audio-Visual Dual-Stream Memory for Agents: openJiuwen Agent-Memory Masters Long Videos Without Missing a Single Key Event

· 量子位
国内AI

Scan QR code to follow QbitAI

Image 1

< img id=“wx_img” src=“https://www.qbitai.com/wp-content/uploads/imgs/qbitai-logo-1.png” width=“400” height=“400”>

Image 2Simiao 2026-11-04 10:49:02 Source: QbitAI

Long-Video Memory Accuracy Surges, QA Latency Cut by up to 97%

Yunzhong reporting from Office

QbitAI | WeChat Official Account: QbitAI

openJiuwen AgentMemory has officially launched its multimodal memory capability, OmniMem-Lens: blind second-by-second chunking is the root cause of memory fragmentation—structuring memory around real-world events is the true solution, boosting recall accuracy by up to 45.7% and slashing online QA latency by up to 97%.

A Problem Plaguing Every Agent: How to “Remember” a 3-Hour Video?

With the rapid evolution of AI agents and vision-language models (VLMs), we are no longer satisfied with agents handling simple text-only tasks; we want them to take on much more complex work.

For example, “watch a 3-hour product launch event and find all product highlights,” or “analyze a full day of surveillance footage and flag anomalous events.”

However, when facing ultra-long videos, current agents are instantly overwhelmed by the sheer volume of information exceeding their context windows, causing a steep decline in processing capacity and accuracy.

The common industry practice today is brute-force slicing: cutting long videos into small chunks using fixed time windows (e.g., one minute) and having the agent memorize them separately.

This approach is like the blind men describing an elephant, introducing two fatal flaws to agent memory:

1. Fragmented Memory — Fragmented Evidence

A coherent lecture by a teacher gets sliced down the middle, with the first and second halves processed and stored separately by the agent. When a user asks a question, the agent retrieves only an out-of-context half of the evidence, leading to fabrications and frequent hallucinations.

2. Diluted Memory — Submerged Evidence

A key action lasting just a few seconds is forcibly bundled with several minutes of irrelevant background footage before and after it. When the agent tries to retrieve fine details, critical information is buried under immense noise, causing recall accuracy to plummet.

The root of the problem is that agents memorize videos by the second, whereas humans understand videos by events.

For example, when you recall a movie you watched yesterday, you remember it by plot points like “protagonists meet → misunderstanding arises → reconciliation,” rather than using a stopwatch to store a snippet every 60 seconds. Yet current agents do exactly that—they record movies with a stopwatch.

Even worse, most video event segmentation solutions on the market rely on a single modality—looking only at video frames or listening only to audio. Both paths can fail: when the visuals remain static for a long time, the system cannot detect that the topic has shifted; when there is no dialogue, it hears zero clues.

This is not a lack of compute; it is memory losing its “focus” right at the point of segmentation.

OmniMem-Lens: Empowering Agents to Memorize “by Events” Like Humans

To enable agents to process long videos just like the human brain, the team officially rolled out OmniMem-Lens (Multimodal Memory Lens) in the openJiuwen AgentMemory 0.2.0 release.

As the core technology of AgentMemory for multimodal video memory processing, OmniMem-Lens innovatively utilizes both visual and auditory tracks to understand events within videos and organize memory content accordingly. For the first time, it enables agent video memory to truly store by events and retrieve by events.

The team positions it as equipping agents with a super hippocampus capable of “autofocusing” and “hierarchical organization.”

It gets three key things right: segmenting by real events, storing memory hierarchically, and building offline for online querying.

Image 3

Audio-Visual Synergy: Memorizing by Real Events Like the Human Brain

OmniMem-Lens equips agents with both “eyes” and “ears”: eyes observe the visuals, identifying and tagging scene transitions; ears listen to speech, distinguishing topic changes via ASR transcription even when visuals remain static.

By cross-referencing both sources of evidence, long videos are segmented along actual event boundaries, freeing agent comprehension from out-of-context misinterpretations.

Three-Tier Memory Architecture: Hierarchical Storage Across Granularities for Both Fine Details and Global Context

Different user queries require different granularities of evidence. OmniMem-Lens builds a three-tier memory unit under the hood:

Image 4

Outlines when you need high-level context, fine details when you need precision—retrieved on demand without interference.

Think of it like a book with a table of contents and an index: the table of contents (L1) guides you to the general chapter, the index (L2) points you to a specific sentence, and the original text (L0) is always available for verification.

Moreover, L1 and L2 memory units are fully textualized and vectorized, allowing seamless integration into mainstream vector databases—enabling developers with existing RAG or Agent stacks to adopt it with virtually zero modifications.

Build Memory Offline, Query Answers Online: Heavy Lifting Upfront, Instant Responses for Users

If an agent had to re-watch hours of video for every query from a different angle, it would not only be excruciatingly slow but also consume massive compute.

OmniMem-Lens handles the heavy memory organization offline in advance; when faced with queries, the agent looks up the pre-built memory directly, delivering sub-second responses with targeted precision.

The online retrieval pipeline is straightforward: the user query first undergoes semantic matching against L1/L2 memory units to quickly pinpoint refined multimodal memories; only when visual detail verification is explicitly required does the system fetch the exact video frames from the L0 layer on demand. This “lightweight text indexing + lazy fetching of raw video” architecture drastically improves video memory retrieval efficiency.

The design philosophy of OmniMem-Lens is to dual-focus on event boundaries like a professional camera and hierarchically refract memory structures like a prism, transforming agent comprehension of long videos from blind slicing into sharp, precise insight.

Benchmark Results: Leading Accuracy Across the Board, Latency Cut by up to 97%

Claims mean little without benchmarks. The team evaluated performance across three mainstream long-video QA benchmarks, comparing OmniMem-Lens paired with Qwen3-VL-8B against directly querying the VLM model Qwen3-VL-8B on raw video content.

How to read the charts below:

Higher “Recall Accuracy” = more accurate retrieved evidence;

Lower “Online Latency” = shorter wait time for answers.

1. Memory Recall Accuracy: Clear Lead Across All Three Benchmarks

Image 5

Accuracy increased by 8% on the general long-video benchmark Video-MME-L and by 6.4% on the long-video understanding benchmark LVbench.

The most notable jump was +45.7% on EgoLife-QA—a benchmark capturing a full day of daily life from a first-person perspective. Packed with dense events and trivial details, it represents the exact scenario where fixed-interval chunking fails hardest, and where the value of “event-based memory organization” shines brightest.

Image 6

2. Online QA Latency: From “Re-watching the Video” to “Querying Memory Directly”

Image 7

The longer the video, the more frames the direct VLM approach must digest, and the higher the latency—on LVbench, direct Qwen3-VL-8B took 34.5 seconds, while OmniMem-Lens took just 1.72 seconds, a full 20x speedup.

In real-world terms: Direct VLM answering is like re-watching a 3-hour movie from beginning to end before answering every question; OmniMem-Lens is like a film critic who took detailed notes while watching, instantly checking their notes to answer whenever you ask.

Two Examples: Understanding the Gap Between “Blind Slicing” and “Precise Insight”

Example 1: Broken Causal Relationships

Image 8

Take a 2-hour animated film as an example: on a train platform, a girl leaves while a boy watches her go; after she leaves, the boy imagines her transforming into a butterfly and fluttering away.

Fixed 1-minute blind slicing divides this 120-minute video into 120 chunks. However, the complete event—“girl leaves → boy watches → butterfly fantasy”—runs from 14:20 to 16:50, spanning across minutes 14, 15, and 16, severed twice by window boundaries.

This fragments the event into three disconnected records: Minute 14 “Girl leaves”, Minute 15 “Boy watches”, and Minute 16 “Butterfly flies away”.

When the user queries “What was the boy imagining after the girl left?”, the retrieved clips only show “boy watches” and “butterfly flies away,” with zero context indicating that the butterfly was merely a fantasy—the agent treats it as a real event and delivers an incorrect answer.

OmniMem-Lens, on the other hand, stores this event as a single coherent segment (14:20–16:50) in memory: the ASR stream detects the shift in dialogue, the visual stream captures scene transitions, and both streams align to cut precisely along the true event boundary. The cause-and-effect relationship—“fantasizing after the girl leaves”—is fully preserved, clearly distinguishing reality from fantasy to produce an accurate answer.

The exact same video with different memory structures makes all the difference between failing to answer and answering accurately.

Example 2: When Visuals Are Static, Audio Is Key

The previous example could largely rely on visual cues. But in the next scenario, eyes alone are not enough.

Image 9

During a 2-hour product review meeting, the exact same architecture diagram stays on screen for 40 minutes without any visual change. Yet during those 40 minutes, the speaker discusses three distinct topics: architectural design, performance stress testing, and next quarter’s roadmap.

A vision-only approach (as well as fixed-window slicing) sees only “a static slide displayed for 40 minutes,” classifying it as a single chunk and bundling all three topics into one memory entry. When a user asks “What was the QPS in the stress testing section?”, the key metric is buried inside 40 minutes of noise and cannot be retrieved.

OmniMem-Lens’s ASR audio stream hears the difference: the visuals haven’t changed, but the speaker has shifted topics. As soon as the speech pivots, the system segments the video—splitting the three topics into separate entries for pinpoint retrieval.

This is precisely the value of dual audio-visual streams: when visuals fail, audio takes over; when audio fails (such as wordless camera cuts), visuals take over. Any single modality can miss crucial boundaries, but cross-verifying both ensures reliability.

WorkSwarm will soon support integration with openJiuwen AgentMemory, helping you remember video content from daily conversations and recall it instantly during discussions.

Conclusion

For AI agents to truly take on complex human tasks, execution capability alone is not enough—they also need a reliable memory.

By enabling agents to “memorize by real events” and “retrieve hierarchically with high efficiency,” OmniMem-Lens resolves the issues of fragmented memory, inaccurate retrieval, and sluggish response times in long-video scenarios. It establishes a high-precision, low-latency memory foundation for multimodal agents, allowing them to truly “never forget and answer effortlessly” when dealing with long videos.

OmniMem-Lens is now officially available in openJiuwen AgentMemory 0.2.0—feel free to try it out and contribute.

If you found this article helpful, feel free to like, share, and subscribe, and let us know in the comments: How does your agent handle long-video memory?

Project URL: https://gitcode.com/openJiuwen/agent-memory/

*This article is published by QbitAI under authorization. The views expressed herein belong solely to the original author.

All rights reserved. Reproduction or use in any form without authorization is strictly prohibited.

Simiao

Scan QR code to share to WeChat MomentsImage 11: Scan me!

@CEOs, Why Does Your Next Personal Assistant Have to Be Human?

Locally deployed, out of the box

Wenle2025-09-17

Build Your Own Agent in Just 4 Steps! DAMO Academy’s New Open-Source Framework Is Beginner-Friendly

Open-source LLMs as core components

Baijiao2023-09-16

In the Auto Research Era, 47 Open-Ended Tasks Become the Must-Test Benchmark for Agent Capabilities

Officially entering the era of “iterative optimization”

Deng Simiao2026-05-13

AI-Driven High-Quality Brand Growth as the Key Breakthrough | Alimama Future Commerce Awards Ceremony Concludes

In the AI era, every category is worth reinventing.

QbitAI2025-03-20

Beijing Says Agents Can Already Build Worlds, While Hangzhou Calls Them Newly Invented Lightbulbs

Industry and academia singing different tunes

Shisan2026-07-25

End of the Shrimp-Farming Era? Zero-Deployment, 24/7 Online, Self-Evolving “Cyber Mules” Have Arrived!

Raising mules is genuinely better than farming shrimp

Mengyao2026-03-16

Taking #1! AgiBot’s Dual Models Sweep Multiple Leaderboards 2026-10-30

Scan QR code to follow QbitAI Image 22

Tracking new AI trends, reporting breakthroughs across the tech industry

Copyright © Beijing Geek Partner Technology Co., Ltd. All rights reserved by QbitAI Beijing ICP Filing No. 17005886-1