CoChat
Sign Up
Cited by 5Embodied and Extended CognitionRobot Manipulation and LearningReinforcement Learning in Robotics

Object-Centric Scene Representations Using Active Inference

Open access
Van de Maele, Toon, Verbelen, Tim, Mazzaglia, Pietro, Ferraro, Stefano, Dhoedt, Bart·2024·Neural Computation
DOI 10.1162/neco_a_01637
Open this paper in CoChat — free accountOpen PDF

Abstract

Representing a scene and its constituent objects from raw sensory data is a core ability for enabling robots to interact with their environment. In this letter, we propose a novel approach for scene understanding, leveraging an object-centric generative model that enables an agent to infer object category and pose in an allocentric reference frame using active inference, a neuro-inspired framework for action and perception. For evaluating the behavior of an active vision agent, we also propose a new benchmark where, given a target viewpoint of a particular object, the agent needs to find the best matching viewpoint given a workspace with randomly positioned objects in 3D. We demonstrate that our active inference agent is able to balance epistemic foraging and goal-driven behavior, and quantitatively outperforms both supervised and reinforcement learning baselines by more than a factor of two in terms of success rate.

Integrity signals

Registry

Crossref · 10.1162/neco_a_01637

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

10 of 17 cited sources checked · 4 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    Cortical Column Networks (CCNs) are object-centric models that learn the approximate posterior over latent pose for views of a particular object by predicting observations of novel poses.

    “We ground our approach on Cortical Column Networks (CCN) (Van de Maele et al., 2022), which are object-centric models that learn the approximate posterior over the latent pose q θ (p k,t |o k,t ) for views of a particular object, by predicting observations of novel poses.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Van de Maele et al., 2022 source
  • Premisecitation

    To the best of the authors' knowledge, none of the existing scene representation benchmarks consider an active camera that can move in the scene.

    “To the best of our knowledge, none of these benchmarks consider an active camera that can move in the scene.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Johnson et al., 2016 source

      Section 2 (Related Work; page unavailable) discusses visual-understanding benchmarks but does not say whether any allow an active camera. Section 3 (The CLEVR Diagnostic Dataset; page unavailable) says camera positions are randomly jittered when images are generated, which does not establish whether a camera can move actively through a scene. Searches for active and movable cameras found no passage supporting the claim about all existing benchmarks. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
    • cites Yan et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Sajjadi et al., 2022b

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    The active inference agent outperforms supervised learning baselines by a large margin.

    “outperforms both supervised and reinforcement learning baselines by a large margin”
    • Also stated in: § 1. Introduction · plain
      “show that we perform better on the active vision benchmark”
    Quote in the paperSubstantial claim
  • Findingown data

    The active inference agent balances epistemic foraging and goal-driven behavior.

    “We demonstrate that our active inference agent is able to balance epistemic foraging and goal-driven behavior”
    Quote in the paperSubstantial claim
  • Findingown data

    The active inference agent outperforms reinforcement learning baselines by a large margin.

    “outperforms both supervised and reinforcement learning baselines by a large margin”
    • Also stated in: § 1. Introduction · plain
      “show that we perform better on the active vision benchmark”
    Quote in the paperSubstantial claim
  • May combine two claims. 17
  • Classified as probably not a substantial claim. 16

  • May combine two claims.

    Premisecitation

    Prior methods for object pose and segmentation typically operate on static inputs and do not allow the agent to interact with the scene.

    “However, these methods typically operate on static inputs and do not allow the agent to interact with the scene.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Minaee et al., 2021 source

      The Introduction describes segmentation as partitioning images or video frames, supporting the characterization of the surveyed methods as operating on visual inputs. It does not establish that prior object-pose methods typically use static inputs, or say that these methods prevent an agent from interacting with the scene. Section 3.11 also describes an interactive segmentation method, though it does not address agent–scene interaction. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Xiang et al., 2018a

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Sucar et al., 2020 source

      Section 6 describes NodeSLAM as building an object map incrementally from a stream of images, rather than operating only on static inputs. Section 7.4, “Robot Manipulation Application” (page unavailable), describes a robot using its reconstructions to pack and sort objects, contrary to the claim that the method does not allow interaction with the scene. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    The brain builds a generative model of the world, constantly explaining its observations (active inference).

    “A prevailing account of human perception is that the brain builds a generative model of the world, constantly explaining its observations (Friston et al., 2017), i.e. active inference.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Friston et al., 2017 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Humans learn by actively engaging and interacting with the world.

    “In contrast, humans learn by actively engaging and interacting with the world (James et al., 2014).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites James et al., 2014 source

      The opening section describes children’s manual exploration of objects as a possible step in learning to recognize them. The paper studies 18- to 24-month-olds and reports associations, so it does not establish the broader claim that humans generally learn through active engagement with the world. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    Vision is cast as inverting a generative model of the scene to infer its constituent objects, their appearance, and pose.

    “In this regard, vision is cast as inverting a generative model of the scene, in order to infer its constituent objects, their appearance, and pose (Parr et al., 2021).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Parr et al., 2021 source

  • May combine two claims.

    Premisereasoning

    CCNs can be efficiently trained in parallel on a limited dataset of views of one object, at the cost of poor generalization to new object categories.

    “This allows these to be efficiently trained in parallel on a limited dataset consisting of views of one particular object, at the cost of poor generalization (i.e. for new object categories a new CCN will need to be trained).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Van de Maele et al., 2022 source

      The Experimental Setup (§3.1; page unavailable) describes 10,000 rendered views per object, and the Discussion (§4; page unavailable) says current CCNs do not generalize even to different instances of the same category and are trained from scratch for each new category. The copy does not establish that training is efficient or performed in parallel; Classification (§3.2; page unavailable) also notes that training uses views of other categories as negative anchors, so the dataset is not exclusively views of one object. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premisecitation

    CCNs consist of a separate encoder-decoder model trained per object category, inspired by cortical columns, and reflect how toddlers interact with one object at a time.

    “These consist of a separate encoder-decoder model trained separately per object category, which is inspired by cortical columns in the brain (Hawkins et al., 2017), and reflect how toddlers typically interact with one object at a time (James et al., 2014).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Hawkins et al., 2017 source
    • cites James et al., 2014 source

      In Method > Tasks and Measures (page unavailable), the paper describes giving children one object at a time to explore, retrieving each before handing over the next. This supports the one-object-at-a-time aspect of the citation, but it describes an experimental procedure rather than establishing that toddlers typically interact that way outside the task. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    The LEXA baseline solves the task in 11 out of 100 cases, while the proposed approach solves it 62 times.

    “We observe that the LEXA baseline is only able to solve the task in 11 out of 100 cases, while our approach was able to solve this 62 times.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The azimuth error is highest for the most symmetric object (tomato soup can) and lowest for the least symmetric object (mustard bottle).

    “we observe that the azimuth error is the highest for the object with the most symmetry (i.e.tomato soup can), and the lowest for the object with the least symmetry (i.e. the mustard bottle), while this is not necessarily reflected by the elevation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    PoseCNN solves the task 17 times and 30 times with the added infogain term.

    “PoseCNN was only able to solve it 17 times and 30 times with the added infogain term.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The average success rate is 69%, more than double the PoseCNN models with and without info gain (28.4% and 16.2% respectively).

    “We obtain an average success rate of 69%, which is more than double than the PoseCNN models with and without info gain (28.4% and 16.2% respectively).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    At the first step, the belief over object positions has a large variance, but over time the agent iterates and attends to different objects, narrowing the variance over these specific object instances.

    “At the first step, it is clear that the belief over object positions has a large variance, but over time the agent iterates and attends to different objects, narrowing the variance over these specific object instances.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    LEXA has lower range and elevation error than the PoseCNN baselines.

    “While LEXA performs worse than the PoseCNN baselines in terms of success rate, the range and elevation error are lower.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Over multiple timesteps, the variance over the object position reduces, and the particle mean lies closer to the ground truth position of the objects.

    “We observe that over multiple timesteps the variance over the object position reduces, and the particles mean lies closer to the ground truth position of the objects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The LEXA agent sometimes reaches the goal almost perfectly but in other cases picks the wrong target object.

    “We can see that the LEXA agent sometimes is able to reach the goal almost perfectly (second and last column) but in other cases, it picks the wrong target object.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Humans decompose scenes into object-centric representations and infer information about objects, their appearance, parts, pose, and shape in 3D space.

    “Humans naturally decompose scenes into object-centric representations and infer information about objects, their appearance, their constituent parts, as well as their pose and shape in the 3D space (Hinton, 1979).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hinton, 1979 source

  • Classified as probably not a substantial claim.

    Interpretationown data

    The paper's benchmark requires an agent to find a target object by moving a camera in a 3D environment.

    “we designed a benchmark that casts the scene perception problem as a problem in which a goal observation has to be reached by a virtual camera.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    PoseCNN has access to depth information, whereas the proposed approach does not.

    “Note that PoseCNN has access to the depth information, whereas our approach has not.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    Minimizing expected free energy balances goal-directed behavior and epistemic foraging.

    “Hence, minimizing expected free energy balances goal-directed behavior and epistemic foraging.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Parr et al., 2022 source

  • Classified as probably not a substantial claim.

    Premisenone given

    Setting an agent's goal in active inference is done by specifying a prior preference in observation space.

    “Setting an agent's goal in active inference is done by specifying a prior preference in observation space, which the agent is expected to obtain.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The presented method is limited to static environments.

    “Currently, our method only considers static environments.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Because symmetry in object shape can cause a high overall translation error while actually having an accurate final observation, the authors opted for spherical coordinate metrics instead of an error in allocentric 6DOF space.

    “As symmetry in object shape can cause a high overall translation error while actually having an accurate final observation, opted for these metrics instead of an error in allocentric 6DOF space.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Using an infogain exploration strategy helps in finding the target object, but the object-centric model is also an important success factor.

    “This shows that using an infogain exploration strategy clearly helps in finding the target object, but the object-centric model is also an important success factor.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    For a Bernoulli variable, optimizing KL-divergence is the same as minimizing binary cross entropy error.

    “Similarly, optimizing the KL-divergence for a Bernoulli variable is equivalent to minimizing the binary cross entropy error over this variable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Currently, the method only considers static environments.

    “Currently, our method only considers static environments.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The scene is static and the entities do not change over time.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Visual inputs are processed by a dorsal 'where' stream representing object location and a ventral 'what' stream representing object identity.

    “This is consistent with findings in the brain, where visual inputs are processed by a dorsal (”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mishkin et al., 1983 source

  • Classified as probably not a substantial claim.

    Premisereasoning

    Actions are selected that minimize expected free energy, scoring how much expected observations realize preferred observations and how much information they bring about hidden variables.

    “Actions are then selected that minimize the expected free energy G, as defined in Equation 3, i.e. scoring how much expected observations will realize the preferred observations, and how much information these will bring over the hidden variables.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Active inference is a process theory of the brain stating that all neuronal processing and action is driven by minimization of a bound on surprise (free energy).

    “Active inference is a process theory of the brain which states that all neuronal processing and action is driven by the minimization of (a bound on) surprise, i.e. free energy (Parr et al., 2022).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Parr et al., 2022 source

  • May combine two claims.

    Interpretationreasoning

    The approach has the downside that it is more difficult to scale towards scenes with multiple objects of a single instance, or to deal with heavily cluttered environments of similar categories.

    “However, our approach comes the downside that it is more difficult to scale towards scenes with multiple objects of a single instance, or to deal with heavily cluttered environments of similar categories.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Observing objects from a fixed distance lets an agent use object scale to estimate distance to the object.

    “We only consider observing objects from a fixed distance, as this will allow an agent to use the scale of an object for estimating the distance to the object.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Minimizing the negative log-likelihood loss is equivalent to minimizing mean squared error between reconstructed and true observations.

    “In practice, optimizing the negative log likelihood terms is equivalent to minimizing the mean squared error between the reconstructed observation and the ground truth observation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    In active inference, perception is cast as approximate Bayesian inference over hidden variables.

    “In active inference, perception is cast as approximate Bayesian inference over the hidden variables.”
    Quote in the paperNot a substantial claim
31 of 31

  • The authors state this as an assumption.

    The scene is static and the entities do not change over time.

    “In the remainder, we assume the scene is static and therefore these entities do not change.”

  • The authors state this as an assumption.

    The identity and translation parameters of objects are consistent over the scene, do not change over time, and are not dependent on the agent's actions.

    “Note that the identity and translation parameters are consistent over the scene, do not change over time, and are therefore not dependent on the action of the agent.”

  • The paper relies on this without support.

    Using two randomly oriented rectangles in different colors as a background increases robustness to different environments.

    “Then two randomly oriented rectangles in different colors are used as a background, which increases robustness to different environments.”
3 of 3

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Initial camera position exceeds the stated height limit

    Minor

    The stated starting range and elevation place the camera about 46 cm above the table center, far above the stated maximum height of 0.6 cm.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Brockman, 2016, OpenAi gym
  2. Rezende, 2018, Taming VAEs
  3. Yan, 2021, CLEVR3D: Compositional language and elementary visual reasoning for question answering in 3D real-world scenes.

Cited by

Sorted by citations

Related

Authors
Van de Maele, T., Verbelen, T., Mazzaglia, P., Ferraro, S., & Dhoedt, B.
Affiliations
  • Ghent University, 9000 Ghent, Belgium toon.vandemaele@ugent.be
  • VERSES (United States) (US)
  • Ghent University (BE)
Corresponding author
  • Verbelen, Tim
  • Mazzaglia, Pietro
  • Ferraro, Stefano
  • Dhoedt, Bart
ORCID
Venue
Neural Computation
Publisher
MIT Press
Published
2024-03-21
Volume
36
Issue
4
Pages
677-704
ISSN
  • 0899-7667 (ISSN-L)
  • 1530-888X
DOI
10.1162/neco_a_01637
OpenAlex
W4392579507
Cited by
5
References
71
Citations by year
2026: 1 · 2025: 2 · 2024: 1 · 2023: 1
FWCI
0.97
Type
journal article
Language
en
Open access
Locations
  • Neural Computation (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Ghent University Academic Bibliography (Ghent University) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Embodied and Extended Cognition
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
active inference, scene understanding, object pose estimation, allocentric reference frame, active vision, generative models, robot perception

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-01.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-01.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review