IEEE TPAMI 2026 · Memory-persistent embodied intelligence

Dream
to Recall.

Memoir queries memory with an imagined future rather than the present: a world model simulates where the agent expects to go, and those latent states retrieve the observations and behaviours that match.

Yunzhe Xu·Yiyuan Pan·Zhe Liu

Shanghai Jiao Tong University

xyz9911@sjtu.edu.cn PhD expected 2027 · seeking postdoc positions in computational neuroscience and world models

PPaper↗ AarXiv↗ ⌘Code↗ DData↗

IMAGINED FUTURE

ẑ_t+1→ẑ_t+2→ẑ_t+3

retrieves

PERSISTENT MEMORY

A graph of stored viewpoints, some highlighted as retrieved

Simulate where you expect to go.
Retrieve what you have seen.

01+5.4

SPL on IR2R

over the strongest memory-persistent baseline

028.3×

training speedup

against full-memory incorporation

03−74%

inference memory

while improving navigation performance

0420.1

gap to oracle

73.3 achieved vs. 93.4 with perfect retrieval — the open problem

01 · THE CENTRAL IDEA

Memory is queried by a
simulated future,
not by the present.

Past navigation becomes useful only when an agent can access the right part of it at the right moment.

Existing systems either absorb the entire memory or inspect a fixed window. Memoir instead learns a language-conditioned world model that imagines task-relevant future states. Those latent states become adaptive queries over an explicit experience store.

01Imaginepredict intent-aligned future states

02Retrieveselect observations and behaviours

03Navigateact with experience in context

Memoir overview comparing memory strategies and showing its adaptive hybrid retrieval pipeline — Figure 1 From persistent experience to adaptive retrieval. Memoir stores both environmental observations and navigation histories, then retrieves from each using the same imagined trajectory.

02 · HOW MEMOIR WORKS

One loop, three learned systems.

The world model, viewpoint-level memory, and navigation policy are trained together to turn imagination into selective, grounded recall.

WORLD MODEL

Prospective simulation

A language-conditioned contrastive model unfolds latent future states from the current episode and instruction.

HYBRID MEMORY

Dual-path retrieval

Imagined latents query a history bank for behavioural precedent and an observation bank for visual evidence.

NAVIGATION MODEL

Experience integration

Specialised encoders fuse coarse topology, fine visual detail, and retrieved navigation history into a single action.

Detailed Memoir architecture with world model, hybrid viewpoint-level memory, and navigation model — Figure 2 The complete architecture. Green denotes imagined and historical state; blue denotes observation memory.

03 · EMPIRICAL EVIDENCE

Retrieval quality—not memory volume—is the constraint.

Across ten scenarios in IR2R and GSA-R2R, Memoir navigates better while using memory more selectively. Replacing the query–retrieval pathway with ground truth reveals how much is still on the table.

Oracle retrieval93.4

Memoir73.3

GR-DUET67.9

No retrieval64.0

IR2R validation unseen · SPL ↑ · bars scaled from 60

Oracle retrieval results across three navigation backbones in seen and unseen environments — Figure 3 A consistent gap of roughly twenty SPL points appears across all three backbones in unseen environments. It bounds what better simulation could buy.

Navigation case study where Memoir retrieves a relevant prior episode and succeeds where DUET and GR-DUET fail — Case study A retrieved behavioural episode supplies the missing route structure without flooding the policy with every stored observation.

04 · A Neuroscience LENS

Could imagination be an
index into memory?

Memoir is an engineering testbed, not a biological model. But it makes a computationally precise proposal: prospective latent simulation can estimate which stored experience will be useful next.

Comparison of retrieval query types: random, instruction similarity, current-state similarity, and imagined future — What kind of query? Holding observation retrieval fixed, a prospective query outperforms cue-based ones — querying with the current state or the instruction. The ordering favours prospection, though the margins are small next to the oracle gap.

Multi-agent navigation results as the number of agents sharing one memory pool increases — Shared memory Agents navigate at the same time and share one pool, each querying with its own imagination. Unseen performance rises with the pool while seen performance stays flat.

Where the need term comes from

Prioritised replay weights memory access by expected future occupancy — the need term, computable from the successor representation over enumerable states under a fixed policy. Neither condition holds when observations are panoramic and the goal changes every episode. A goal-conditioned world model supplies the estimate instead.

Prediction

Disrupting prospective simulation while preserving stored memory should degrade performance in proportion to the oracle gap; improving simulation fidelity should yield gains of the same magnitude.

Prediction

Observing a demonstrator should move performance toward the oracle ceiling in proportion to the behavioural information the demonstration carries — a computational analogue for observationally biased replay.

05 · WHAT FAILURE TEACHES US

Recall is only as useful as the experience it retrieves.

Memoir can recover from sparse observation by borrowing the route structure of a prior successful episode. But when a retrieved episode itself failed, its visual overlap can mislead the current policy. Behavioural outcome therefore matters, not just state similarity.

A memory system should remember not only where it has been, but whether what it did worked.

Memoir failure analysis contrasting retrieval of failed and successful prior navigation episodes — Failure analysis A previously failed trajectory leads the agent astray; a successful behavioural memory enables recovery.

READ · BUILD · EXTEND

Take Memoir further.

Explore the implementation, download the released datasets and checkpoints, or cite the TPAMI paper.