Deconstructing Deep Active Inference: A Contrarian Information Gatherer

May combine two claims.

The DAI_HR approaches cannot be considered complete implementations of deep active inference because they are restricted to a single decoder and lack encoder, transition, and critic networks.

“Since, Sancaktar et al (2020); Lanillos et al (2020); Oliver et al (2019) restricted themself to the context of a single decoder, this approach can not be considered as a complete implementation of deep active inference.More precisely, the encoder, transition and critic (or policy) networks are missing.”

• cites Sancaktar et al (2020) sourcePartly supported

  Section 3.2 (p. 5) describes a convolutional decoder as the visual generative model, and Section 6 (p. 12) says actions are generated without learning an explicit policy. These passages support the architectural description, but the paper does not say that the absence of encoder, transition, and critic networks makes its approach an incomplete implementation of deep active inference. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Lanillos et al (2020) sourcePartly supported

  Section 2.3.2 (p. 4) describes a learned visual generative model implemented with a Mixture Density Network, supporting the description of a single learned decoder-like network. Section 2.3.1 (p. 3) describes inference and dynamics, but the paper does not assess whether its approach is a complete implementation of deep active inference or state that encoder, transition, and critic networks are required. That assessment is the citing authors’ own characterization. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Oliver et al (2019) sourceCould not verify

  The Technical Challenges section (p. 8) describes generative functions computed from forward kinematics, but does not characterize the implementation as a single decoder. The passages on the computational model and conclusions (pp. 5, 8) do not address encoder, transition, or critic networks. I searched for those network terms and for deep active inference; our copy does not establish the architectural claim attributed to Oliver et al. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: A complete implementation of deep active inference requires encoder, transition, and critic or policy networks.

May combine two claims.

The DAI_FA requires the modeller to encode prior preferences within the distributions predicted by the encoder and transition network, which can limit its applicability.

“Third, the DAI F A requires the modeller to encode the prior preferences within the distributions predicted by the encoder and transition network.This can limit the applicability of the approach.”

• cites Ueltzhöffer (2018) sourcePartly supported

  Methods (p. 14) says the modeller must specify prior expectations, and Discussion and Outlook (p. 30) describes hardcoding them as explicit prior distributions on the first dimension of the latent space as “ad-hoc and restricted.” This supports the claimed limitation, but the paper does not say that preferences must be encoded within distributions predicted by both the encoder and transition network. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

May combine two claims.

The DAI_FA approach unrolls the POMDP over time, building a large computational graph for each action-perception cycle, making it computationally intensive and potentially intractable for long horizons.

“Second, this approach unrolls the partially observable Markov decision process over time.In other words, the code builds a huge computational graph containing the encoder, decoder, transition and policy networks for each action-perception cycle.Therefore, the approach is computationally intensive and can become intractable for a large time horizon.”

• cites Ueltzhöffer (2018) sourcePartly supported

  The Methods describe a discrete-time recurrent model that carries information between timesteps (p. 7), supporting the temporal part of the claim. They do not describe a huge computational graph or say that long horizons become intractable. Instead, the Methods call the implementation fast (p. 14), and the Results report modest memory use for the tested run (p. 21). (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

In DAI_FA, clamping the first element of the mean and variance vectors predicted by the transition is uncontentious because it is simply how the generative model is defined.

“Note, clamping the value of the first element of the mean and variance vectors predicted by the transition is uncontentious, i.e., this is simply how the generative model is defined.”

• cites Ueltzhöffer (2018) sourceCould not verify

  The Methods (p. 11) describe a Gaussian state transition whose means and standard deviations are calculated by neural networks. The passage does not address clamping the first element of either vector or say that such clamping defines the generative model; searches for those terms found no passage that does. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

May combine two claims.

The DAI_FA agent has no expected free energy and is instead trained to minimize cumulated variational free energy over time.

“First, there is no expected free energy, instead the agent is trained to minimize the cumulated variational free energy over time.”

• cites Ueltzhöffer (2018) sourcePartly supported

  The Methods (p. 17) accumulate variational free energy across time steps, and the Discussion (p. 30) describes minimizing its path or time integral. But the Methods (pp. 17–18) also explicitly describe optimizing expected free energy under a population distribution over agent parameters, so the unqualified statement that there is no expected free energy is not supported. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

May combine two claims.

The DAI_RHI approach cannot be considered a complete implementation of deep active inference because it is restricted to a VAE and lacks transition and critic networks.

“Since, Rood et al (2020) restricted themself to the context of a VAE, this approach cannot be considered as a complete implementation of deep active inference.More precisely, the transition and critic (or policy) networks are missing.”

• cites Rood et al (2020) sourcePartly supported

  Section 2 (p. 2) describes a simplified model for the static RHI experiment, with no dynamics on the internal variables; this supports the claim that its scope is limited. But Section 2.1 (p. 4) explicitly compares a convolutional decoder with a VAE, so the approach is not restricted to a VAE. The paper does not state that missing transition and critic networks make it an incomplete implementation of deep active inference. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: A complete implementation of deep active inference requires encoder, transition, and critic or policy networks.

Classified as probably not a substantial claim.

The expected free energy as stated in Equation (7) needs to be rearranged because it cannot be easily evaluated.

“The EFE as stated in Equation (7) needs to be re-arranged because it cannot be easily evaluated.”

• cites Champion et al (2023) sourcePartly supported

  Section 2.2.3 (p. 9) says verbatim that the EFE needs to be rearranged because it cannot be easily evaluated. It identifies the expression as Equation (2), however, not Equation (7); our copy does not establish that the two equation numbers refer to the same expression. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

Although Millidge's mathematics uses a POMDP formalism, the code implements a fully observable MDP without an encoder/decoder architecture.

“Although the mathematics in the paper are based on the formalism of a partially observable Markov decision process (POMDP), the code does not implement an encoder/decoder architecture, that is, the code implements a fully observable decision process, i.e., MDP.”

• cites Millidge (2020) sourceSupported

May combine two claims.

The DAI_VPG implementation has no hidden states, which raises uncertainty about how the intrinsic value is computed.

“Now, remember that in the implementation, there is no encoder Q(s τ ) and no decoder P (o τ |s τ ).In other words, there are no hidden states s τ , raising some uncertainty about how the intrinsic value is computed.”

• cites Millidge (2020) sourceSupported

May combine two claims.

The DAI_VPG implementation is a mixture of POMDP and MDP settings, with an MDP generative model and expected free energy adapted from the POMDP setting.

“Thus, (as it stands) the implementation of the DAI V P G agent is a mixture between the POMDP and MDP setting, where the generative model corresponds to an MDP, and the expected free energy is adapted from the POMDP setting.”

• cites Millidge (2020) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The expected free energy formulation of Parr and Friston (2019) has become the standard definition used in deep active inference.

“The above EFE formulation was proposed by Parr and Friston (2019), and has become the typical EFE definition in deep active inference.”

• cites Parr and Friston (2019) sourcePartly supported

  The ‘Expected free energy’ section defines an EFE functional, supporting the attribution of a formulation to Parr and Friston. The Introduction says the paper revisits EFE and offers an alternative generalised formulation; our copy does not establish that its EFE definition became typical in deep active inference. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Using this definition of expected free energy in deep learning may account for why the CHMM minimizing EFE could not solve the task.

“The use of this definition of the expected free energy in the deep learning context may explain why the CHMM minimizing EFE was unable to solve the task”

• cites Champion et al (2023) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The approach of Schneider et al. (2022) is framed from a reinforcement learning perspective in which the agent seeks to maximize reward.

“The last paper that we review (Schneider et al, 2022) is motivated from a reinforcement learning perspective, where the agent aims to maximize reward.”

• cites Schneider et al, 2022 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Using this definition of expected free energy may also account for presentational uncertainties, such as minus sign additions, in the deep active inference literature.

“may also explain some of the presentational uncertainties (e.g., additions of minus signs) found in the deep active inference literature, see Champion et al (2023) for details.”

• cites Champion et al (2023) sourceSupported

May combine two claims.

Schneider et al.'s agent does not only maximize reward greedily; it also maximizes the information gain between the model parameters and the expected states and rewards.

“However, instead of greedily maximizing reward, the agent also maximizes the information gain between the model parameters and the expected states and rewards”

• cites Schneider et al, 2022 sourceSupported

May combine two claims.

Unlike the standard active inference objective, which is not Bellman-optimal for horizons longer than one step, the CHMM approach is Bellman-optimal and closely resembles sophisticated inference.

“Thus, in contrast to the standard active inference objective that is not Bellman-optimal for time horizons greater than one (Da Costa et al, 2023), our approach is Bellman-optimal and closely resembles sophisticated inference (Friston et al, 2020).”

• cites Da Costa et al, 2023 sourceSupported
• cites Friston et al, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The CHMM minimizing expected free energy almost always selected the same action, which prevented it from solving dSprites.

“On further inspection, we found that the CHMM minimizing expected free energy almost always picks the same action, which makes it unable to solve the dSprites environment.”

May combine two claims.

For the action down, the average reward was zero, while for all other actions the expected reward was negative.

“Additionally, when selecting the action down the average reward is zero, while for all the other actions, the expected reward will be negative.”

May combine two claims.

The CHMM maximizing reward continued selecting all actions and successfully solved the task.

“In contrast, the CHMM maximizing reward, keeps on selecting all the actions, enabling it to successfully solve the task.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For this case, the epistemic value is strongly negative.

“In this instance, the epistemic value will be strongly negative.”

May combine two claims.

In the second experiment, the likelihood is high-entropy and the state prior moves left to right.

“In the second experiment, the likelihood has high entropy (c.f., right-most graph in Figure 17), and the prior over states is shifting from left to right (c.f., Figure 18).”

May combine two claims.

The agent using ε-greedy action selection collected the most cumulative reward and learned to solve the task fastest.

“Importantly, according to our experiments, the agent using the epsilon-greedy algorithm received the highest amount of cumulated rewards and learned to solve the task the fastest.”

May combine two claims.

Because VAE and HMM agents act randomly, they failed the task and accumulated about -7K reward after 500K iterations.

“Since the VAE and HMM agents take random actions in the environment, the agents were unable to solve the task and accumulated a total amount of reward of around -7K (after 500K training iterations).”

Probably not a substantial claim: may not pass S4 (based).

Maximizing the epistemic objective leads to loss of information.

“maximizing the epistemic value causes information to be lost.”

May combine two claims. The quote only partly states this claim.

On this task DQN learns more slowly than A2C but more stably.

“In contrast, DQN has a slower but more stable learning on this task.”

May combine two claims.

A2C learns faster than DQN early on but shows a drop in performance between 180K and 280K training iterations.

“Also, A2C initially learns much faster than DQN, but suffers from a decrease in performance between 180K and 280K training iterations.”

May combine two claims.

The reward-maximizing CHMM keeps selecting the right and left actions, which allows it to move the shape toward the correct corner.

“In contrast, the CHMM maximizing reward, keeps on selecting the actions right and left, which enables it to drag the shape towards the appropriate corner, c.f., Figure 13(c).”

May combine two claims.

A2C and DQN reached roughly the same asymptotic reward, and PPO outperformed both of them.

“A2C and DQN obtained about the same asymptotic reward, and were outperformed by PPO.”

May combine two claims.

The hidden Markov model agent learned a good representation of the task and of its dynamics.

“The agent was able to learn a good represenation of the task and of its dynamics, but was not able to solve the task.”

Classified as probably not a substantial claim.

The definition of the expected free energy differed across the reviewed papers.

“Also, the definition of the expected free energy varied between papers.”

May combine two claims.

The authors struggled to replicate some claimed results, such as training DAI MC on the animal AI environment, and in some cases could not access the code.

“Yet, we struggled to replicate some of the results claimed, e.g.training DAI M C on the animal AI environment, and we were unable to access code in some instances.”

May combine two claims.

Every agent minimized its variational free energy, which declined smoothly over training.

“all the agents were able to minimize their variational free energy, and the VFE smoothly decreases during training.”

Classified as probably not a substantial claim.

Epistemic value rises as the prior and true posterior grow more similar to the approximate posterior.

“The right-most graph of Figure 14 shows that the epistemic value increases as the prior P (s τ ) and therefore the true posterior P (o τ |s τ ) becomes more similar to the approximate posterior Q(s τ |π).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The top-down attention parameter omega in the DAI_MC agent is part of the model specification and is not obtained through variational inference.

“Importantly, the above expression for ω t is part of the model specification, i.e., it is not obtained through variational inference.”

Classified as probably not a substantial claim.

The number of terms needed to compute the prior over actions in DAI_MC grows exponentially with the planning time horizon.

“Finally, note that the number of terms required to compute the prior over actions (defined in Equation 5) grows exponentially with the time horizon of planning.”

May combine two claims.

The CHMM minimizing expected free energy repeatedly chose one action and became an expert at predicting the future for that action.

“Thus, the CHMM minimizing expected free energy repeatedly picks a single action, and becomes an expert at predicting the future when selecting this action.”

Classified as probably not a substantial claim. May combine two claims.

Building a VAE and a hidden Markov model before the CHMM allows validation of individual components and better localization of potential problems.

“Importantly, building a VAE and a HMM first, we can validate the implementation of individual components (i.e., encoder, decoder, and transition network), before tackling the CHMM, which represents a complete implementation of active inference.This method also allows us to better localize the source of potential problems.”

Probably not a substantial claim: may not pass S4 (based).

The critical hidden Markov model (CHMM) is a complete implementation of active inference.

“which leads to the critical HMM (CHMM).Importantly, building a VAE and a HMM first, we can validate the implementation of individual components (i.e., encoder, decoder, and transition network), before tackling the CHMM, which represents a complete implementation of active inference.”

Classified as probably not a substantial claim. May combine two claims.

Schneider et al. show a relationship between their objective and the expected free energy, a relationship that rests on assumptions including that states and rewards are latent variables.

“Then, the authors demonstrate the relationship between Equation 18and the expected free energy.This relationship is explained in more details in (Schneider, N.D.), and relies on the following assumptions:”

• cites Schneider, N.D.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The EFE-minimizing CHMM settles on a single action, down, and becomes highly accurate at predicting the future for that action, which keeps the KL divergence between the transition and encoder outputs small.

“This suggests that the CHMM minimizing EFE is picking a single action (down), and becomes an expert at predicting the future when selecting this action.This effectively makes the KL divergence between the output of the transition and encoder networks small.”

• Also stated in: § 6. Conclusion · plain

  “Visualizing the distribution of actions selected as training progresses shows that the agent minimizing EFE almost exclusively picks action down.”

May combine two claims.

None of the reviewed deep active inference approaches is a complete and definitive realization of deep active inference, because they are incomplete, cannot scale to image-based tasks, or rely on restrictive assumptions.

“However, we do not feel that any of these approaches is a complete and definitive realization of deep active inference, because these approaches are either incomplete, unable to scale to image-based tasks, or make restrictive assumptions (see Section 2 for details).”

• Also stated in: § 6. Conclusion · plain

  “However, we did not compare with these methods as they are either incomplete, unable to scale to image-based tasks, or make restrictive assumptions.”

May combine two claims.

The information gain that is supposed to give an EFE-minimizing agent its exploratory behaviour instead prevents the agent from solving dSprites, because it reduces the agent to a single action and thus to a suboptimal policy.

“To conclude, the same information gain that is intended to give an EFE minimizing agent its exploration behaviour, also prevents the agent from solving the dSprites environment.This is because the agent is reduced to picking a single action, leading to a suboptimal policy.”

• AssumptionRests on an assumption: The dSprites environment is a suitable and informative testbed for evaluating deep active inference agents.

May combine two claims.

An almost uniform likelihood barely affects the joint distribution, leaving the approximate posterior dominant.

“as the likelihood is almost uniform, it will not have much impact on the joint distribution P (o τ |s τ )Q(s τ |π) and the approximate posterior Q(s τ |π) will dominate.”

May combine two claims.

For every action other than down, the expected reward is negative, because a trial is interrupted after 50 action-perception cycles without crossing the bottom line and the agent then receives a reward of -1.

“For all the other actions, the expected reward will be negative because after 50 action-perception cycles without crossing the bottom line, the trial is interrupted and the agent receives a reward of -1.”

May combine two claims.

In dSprites, choosing the action down yields an average reward of zero, because shapes are equally numerous on the left and right of the image and the reward rises linearly toward a corner and is zero at the image center.

“Additionally, when selecting the action down, the average reward is zero, because (in the dSprites dataset) there are as many shapes on the left of the image as on the right, and when crossing the bottom line, the agent receives a reward which is linearly increasing (or decreasing) as a corner is approached and is zero at the center of the image.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Alternative definitions of the expected free energy exist and are likely to perform better.

“However, there are alternative definitions, which are likely to behave better (Parr et al, 2022).”

• cites Parr et al, 2022 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The smooth L1 loss was selected because it is less sensitive to outliers than the mean squared error.

“Note, the SL1 was picked because it is less sensitive to outliers than”

May combine two claims.

Schneider et al.'s approach is a valuable contribution to reinforcement learning for robotic control, but it is not a complete deep active inference agent.

“To conclude, this approach is a great contribution to reinforcement learning applied to robotic control, but cannot be considered as a complete deep active inference agent.”

Probably not a substantial claim: may not pass S4 (based).

Under two Gaussian assumptions with identity covariance matrices, the KL-divergence becomes mean squared error.

“the KL-divergence reduces to the mean squared error (MSE) because of two Gaussian assumptions (with identity covariance matrices)”

The quote only partly states this claim.

If the state prior and approximate posterior differ, the true posterior nearly equals the state prior.

“the true posterior will almost be equal to the prior over states P (s τ |o τ ) ≈ P (s τ )”

Classified as probably not a substantial claim.

Failing to cross the bottom line within 50 actions yields a reward of -1.

“if the agent does not cross the bottom line within the first 50 actions, it will receive a reward of -1.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The deep active inference agent of Çatal et al. (2020) is built on a state space model, which is why it is named DAI SSM.

“The deep active inference agent proposed by (C ¸atal et al, 2020) is based on a state space model, and is therefore called DAI SSM .”

• cites C ¸atal et al, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Training stability was improved by adding a target network, structurally identical to the critic, whose weights are synchronized with the critic's every K learning iterations.

“Also, as for the DQN agent, we improved the training stability by implementing a target network Ĝθ a , which is structurally identical to the critic and whose weights are synchronized with the weights of the critic every K (learning) iterations.”

May combine two claims.

The assumptions behind Schneider et al.'s derivation of expected free energy place strong constraints on the model, and the proof fails when the likelihood mappings are not delta distributions.

“Importantly, while the above assumptions allow the authors to derive the expected free energy from Equation 18, these assumptions also impose a lot of constraints on the model.For example, the proof does not hold if the likelihood mappings are not delta distributions.”

Classified as probably not a substantial claim.

When the critic network's predicted values are very close, an agent using softmax sampling may choose actions randomly.

“if the values predicted by the critic network are very close to each other, then an agent using softmax sampling may perform random actions.”

May combine two claims.

Consequently, the log true posterior is very negative, whereas the log approximate posterior is less negative.

“This implies that the logarithm of true posterior will be very negative, while the logarithm of the approximate posterior will be less negative, i.e., closer to zero.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All experiments can be reproduced using code available on GitHub.

“The code that can be used to reproduce all the experiments can be found on GitHub at the following URL: https://github.com/ChampiB/Deconstructing_Deep_Active_Inference.”

Classified as probably not a substantial claim. May combine two claims.

Building the VAE and HMM agents before the CHMM validates the implementation of the individual components (encoder, decoder, transition network) and helps localize where problems arise.

“Importantly, building a VAE and a HMM first allows us to validate the implementation of individual components (i.e., encoder, decoder, and transition network), before tackling the CHMM, which represents a complete implementation of active inference.This method also allows us to better localize the source of potential problems.”

May combine two claims.

The CHMM that selects the best action according to the critic suffers from insufficient exploration, which slows its learning.

“Put simply, the latter suffers from a lack of exploration that slows down its learning.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Dropping a constant does not change the optimal policy.

“Removing a constant does not influence which policy is the best.”

May combine two claims.

With larger N, the estimator for Q(s t+1|a t) becomes a Gaussian mixture with N components, rendering the Equation 40 KL-divergence analytically intractable.

“for higher value of N , the estimator of Q(s t+1 |a t ) would be a mixture of Gaussian with N components, effectively rendering the KL-divergence in Equation 40analytically intractable.”

May combine two claims.

This behaviour is unexpected, since epistemic value should be larger when true and approximate posteriors are more similar.

“This behaviour should not be observed, as the epistemic value is bigger when the true posterior P (s τ |o τ ) and the approximate posterior Q(s τ |π) are more similar.”

Classified as probably not a substantial claim.

A Monte-Carlo estimate is typically employed.

“Generally, a Monte-Carlo estimate is used as follows:”