Overview

ReMAP lets a multimodal model retrieve visual evidence while reasoning. Global memory retains scene and cross-image context; Local memory retrieves regions relevant to the current reasoning state. Both return compact latent tokens, and a learned policy decides when and which memory to access.

How it works

1. Preserve global context

   Question-conditioned Global memory summarizes individual images and their relationships before decoding.

2. Retrieve local evidence

   Local memory selects and re-encodes relevant image regions, using Global context to interpret the details.

3. Learn when to look again

   Reinforcement learning compares branched continuations to train the choice between continued reasoning and Global or Local memory access.

Evaluation

The paper evaluates ten benchmark families and reports gains over prior visual-memory methods on all four multi-image benchmarks. Cross-backbone comparisons and memory ablations examine where access helps; visual-context analysis measures the tokens entering the reasoning sequence. Full protocols and limitations are in the paper.

Paper and resources

ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory
arXiv:2610.05097

Figures and technical descriptions are based on the linked paper. The original author list and publication status are shown above.