From memristors to compositional machine learning

The binary encoding of a collection of AHaH nodes operating the FF-RU instruction can be used as the basis of a clustering algorithm.

“As we have reported elsewhere [Nugent and W 2014], the binary encoding of a collection of AHaH nodes operating the FF-RU instruction can be used as the basis of a clustering algorithm.”

• cites Nugent and W 2014 sourcePartly supported

  The Methods section, “AHaH Clustering,” describes a collective of AHaH nodes whose binary output vector assigns cluster IDs, supporting the clustering claim. Our copy does not identify the nodes as operating an “FF-RU instruction,” so that specific operating condition is not verified. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Optimizing the variables in the synaptic access power equation can reduce dynamic power consumption by more than nine orders of magnitude.

“Optimization over these variables can reduce the dynamic power consumption by over nine orders of magnitude, as seen in Table 1.”

Lower training error for reflective learning compared to standard training for growth trees of the same size and growth rates provides evidence for this.

“The later is evidenced by lower training error for reflective learning when compared to standard training for growth trees of the same size and growth rates.”

• AssumptionRests on an assumption: The ability to autoencode data is useful for dimensionality reduction, noise tolerance and generalization in classification tasks and to automatically discover data regularities in unsupervised clustering tasks.

The FF-RU clustering method is susceptible to the null state for inputs that are not temporally and spatially sparse.

“We have found that this method is susceptible to the null state 3 for non temporally and spatially sparse inputs.”

A differential pair can create its own reference, with zero defined when the conductances of the two memristors are equal.

“a differential pair can create its own reference: zero is defined when the conductance of each memristor is equal.”

Unlike DNNs, the Competitive Growth Framework is more amenable to neuromorphic hardware implementation because it does not require derivative computation, backward propagation, or high-precision synaptic weights.

“Unlike DNN, the CGF is more amenable to neuromorphic hardware implementation as it does not require the computation of derivatives, backward propagation of information, or high-precision synaptic weightsfoot_0 .”

Anti-Hebbian operations can drive the state of a differential-pair synapse toward zero, providing a local passive mechanism to adapt to fabrication variations.

“Through anti-Hebbian operations, the state of a differential-pair synapse can be driven toward zero, providing a local passive mechanism to adapt to fabrication variations.”

kT-RAM adds a general-purpose adaptive synaptic resource to existing digital computing platforms.

“kT-RAM adds a general-purpose adaptive synaptic resource to existing digital computing platforms.”

• cites Nugent and Molter 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The conjugate G_a plus G_b can be used to encode evidence accumulation.

“While the conductance differential encodes state, the conjugate G a + G b can be used to encode evidence accumulation.”

A differential-pair synapse representation has advantages over a single memristor.

“A differential-pair synapse representation of a synapse has a number of advantages over single memristor.”

• AssumptionRests on an assumption: Memristive device state decay and variation in device properties during fabrication and operation are typical.

A single memristor cannot represent a negative weight without a reference because negative conductance is not physically possible.

“It is not physically possible to have a negative conductance, which means that a single memristor cannot represent a negative weight without a reference.”

• AssumptionRests on an assumption: Most machine learning models, including DNNs, require positive and negative multiplicative synaptic weights.
• AssumptionRests on an assumption: Conductance cannot be negative.

Because conductance decay is approximately a common-mode signal, differential representation provides a degree of isolation.

“Since conductance decay is (to a first approximation) a common-mode signal, differential representation provides a degree of isolation.”

• AssumptionRests on an assumption: Conductance decay is approximately a common-mode signal.

Adaptive learning rates can be achieved with a differential synapse representation through its initialization state.

“Adaptive learning rates can be achieved with a differential synapse representation through it's initialization state.”

Multiple levels of computational abstraction can be defined on top of kT-RAM's differential-memristor synaptic primitive.

“Like transistors and logic gates, multiple levels of computational abstraction can be defined on top of kT-RAM's differential-memristor synaptic primitive.”

Recent analog neural network accelerators require considerable chip real-estate for calibration to counteract circuit fabrication variations.

“recent attempts at analog neural network accelerators demonstrate that considerable chip real-estate is required for calibration to counteract circuit fabrication variations.[Miyashita et al. 2016]”

• cites Miyashita et al. 2016 sourceNot checked — no full text of the source

Tolerance to decay and material variations can be achieved with a differential-pair representation.

“Tolerance to decay and material variations can be achieved with a differential-pair representation.”

• AssumptionRests on an assumption: Memristive device state decay and variation in device properties during fabrication and operation are typical.

DNNs running on modern digital compute cores do not match the conditions that minimize synaptic access power, because they use high synaptic precision, continuous input activation, high synaptic access, separated memory and processing, and high operating voltages.

“None of these conditions match the operating regime of DNN on modern digital compute cores, where synaptic precision is high, input activation is continuous, synaptic access high, memory and processing are separated (and therefore d is high) and operating voltages are high.”

As the conjugate grows and evidence accumulates, the same change in the differential causes smaller changes in synaptic output.

“As the conjugate grows and evidence accumulates, the same change in the differential causes smaller changes in the synaptic output.”

Distributed representations are important for compositional learning because they enable generalization to unseen data.

“Distributed representations are important in compositional learning because they provide the ability to generalize on unseen data [Liaw et al. 2002].”

• cites Liaw et al. 2002Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The positive feedback inherent in Hebbian learning will force the AHaH node to the null state when there is no opposing distribution.

“The positive feedback inherent in Hebbian learning will then force the AHaH node to the null state.”

The risk of Hebbian learning occurs when there is no opposing data distribution, which can occur due to weight initialization or temporary data fluctuations.

“The risk of Hebbian learning occurs when there is no opposing data distribution, which can occur for a number of reasons including weight initialization or temporary data fluctuations.”

Modifying the input pattern from P to PP′ may change the binary evaluation of any node along path Q, including the root node.

“Modification of the input pattern P → PP ′ may therefore change the binary evaluation of any node along path Q, including the root node.”

If a tree's output is treated as an integer identifying its leaf or path, then a decision tree forest's output can be treated as a vector of such integers, one per tree.

“However, as Bengio pointed out [Bengio et al. 2010], if we consider the output of a tree to be an integer specifying the DT leaf or path, then we can consider the output of a decision tree forest (DTF) as the encoding of a vector whose elements are these integers, one per tree in the forest.”

• cites Bengio et al. 2010Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Spikes in a spike space are directly mapped to active synapses in a synaptic memory space via row and column addressing.

“That is, spikes in a spike space are directly mapped to active synapses in a synaptic memory space (a kT-RAM Core) via row and column addressing.”

Growth of upstream trees causes catastrophic loss of information for downstream trees.

“In other words, the growth of upstream trees results in catastrophic loss of information for down-stream trees.”

This has implications for transitioning from digital CMOS to alternative technologies such as memristors, because many hardware embodiments could implement the growth communication protocol.

“This has implications in making a transition from existing digital CMOS technology to alternate technologies like memristors, as numerous hardware embodiments could be created that would implement the growth communication protocol.”

• AssumptionRests on an assumption: The ability to learn a compositional representation is the underlying objective of deep learning and is responsible for its success in modeling intrinsically hierarchical data such as vision, speech, and language.

The ideal decision boundary can be accomplished by gating the application of AHaH plasticity to individual nodes based on information in the group.

“This ideal decision boundary can be accomplished by gating the application of AHaH plasticity to individual nodes based on information in the group.”

If DT1 adds one or more nodes to its encoding, all nodes of DT2 will see a different spike pattern.

“If DT1 adds one or more nodes to its encoding, all nodes of DT2 will see a different spike pattern.”

When a neuron's decision boundary falls between opposing data distributions, Hebbian learning acts to maximize the decision margin between the distributions, provided the Hebbian influence falls as neural activation increases.

“When a neuron's decision boundary falls between opposing data distributions, Hebbian learning acts to maximize the decision margin between the distributions, provided the Hebbian influence falls as the neural activation increases.”

A single decision tree's output is not a distributed representation.

“In the case of a single decision tree, it is clear that its output is not a distributed representation.”

Mistake reflection focuses growth in phase-space regions that cause classification errors.

“Through mistake reflection, growth is focused in regions of the phase-space causing classification errors.”

• AssumptionRests on an assumption: The ability to learn a compositional representation is the underlying objective of deep learning and is responsible for its success in modeling intrinsically hierarchical data such as vision, speech, and language.

A node operating pure anti-Hebbian learning cannot maximize its decision margin.

“However, a node operating pure anti-Hebbian learning cannot maximize its decision margin.”

Using a counter and a time limit prevents the situation where the set is never filled, such as when there are more AHaH nodes than data regularities.

“The use of a counter and time limit prevents the case where the set is never filled, for example if the number of AHaH nodes exceeds the number of data regularities.”

Using a set ensures that every AHaH node receives an equal total amount of positive Hebbian feedback, which prevents the runaway rich-get-richer cycle that leads to the null state.

“The use of a set insures that all AHaH nodes receive the same total amount of Hebbian (positive) feedback, thus preventing the run-away rich-get-richer scenario that leads to the null state.”

Mistake reflection and tree growth suggest a general learning approach that avoids computing derivatives of continuous functions or back-propagating error signals.

“Mistake reflection and tree growth hints at a general learning approach that does not rely on computing derivatives of continuous functions or back-propagating error signals.”

Collections of BinaryPartitioners can be used to increase the number of cluster centers, for instance by using a decision tree.

“Collections of BinaryPartitioners can be use used to increase the number of cluster centers, for example with the use of a decision tree.”

The decision tree forest output is a distributed representation that can express a number of configurations possibly exponential in the number of trees.

“This is a distributed representation, which can express a number of configurations possibly exponential in the number of trees.”

If each tree in a decision tree forest grows in response to node activations, the forest will add resolution to resolve finer details in the phase-space region where mistakes occur.

“If each tree in the DTF grows in response to node activations, then the DTF will add resolution to resolve finer details in the phase space region where mistakes occur.”

The BinaryPartitioner can be regarded as an improved version of a single AHaH node executing the FF-RU instruction.

“We call the result a BinaryPartitioner (Listing 2), which can be seen as an improved version of a single AHaH node operating the FF-RU instruction.”