Английская Википедия:Generalized Hebbian algorithm
Шаблон:Short description The generalized Hebbian algorithm (GHA), also known in the literature as Sanger's rule, is a linear feedforward neural network for unsupervised learning with applications primarily in principal components analysis. First defined in 1989,[1] it is similar to Oja's rule in its formulation and stability, except it can be applied to networks with multiple outputs. The name originates because of the similarity between the algorithm and a hypothesis made by Donald Hebb[2] about the way in which synaptic strengths in the brain are modified in response to experience, i.e., that changes are proportional to the correlation between the firing of pre- and post-synaptic neurons.[3]
Theory
The GHA combines Oja's rule with the Gram-Schmidt process to produce a learning rule of the form
- <math>\,\Delta w_{ij} ~ = ~ \eta\left(y_i x_j - y_i \sum_{k=1}^{i} w_{kj} y_k \right)</math>,[4]
where Шаблон:Math defines the synaptic weight or connection strength between the Шаблон:Mathth input and Шаблон:Mathth output neurons, Шаблон:Math and Шаблон:Math are the input and output vectors, respectively, and Шаблон:Math is the learning rate parameter.
Derivation
In matrix form, Oja's rule can be written
- <math>\,\frac{\text{d} w(t)}{\text{d} t} ~ = ~ w(t) Q - \mathrm{diag} [w(t) Q w(t)^{\mathrm{T}}] w(t)</math>,
and the Gram-Schmidt algorithm is
- <math>\,\Delta w(t) ~ = ~ -\mathrm{lower} [w(t) w(t)^{\mathrm{T}}] w(t)</math>,
where Шаблон:Math is any matrix, in this case representing synaptic weights, Шаблон:Math is the autocorrelation matrix, simply the outer product of inputs, Шаблон:Math is the function that diagonalizes a matrix, and Шаблон:Math is the function that sets all matrix elements on or above the diagonal equal to 0. We can combine these equations to get our original rule in matrix form,
- <math>\,\Delta w(t) ~ = ~ \eta(t) \left(\mathbf{y}(t) \mathbf{x}(t)^{\mathrm{T}} - \mathrm{LT}[\mathbf{y}(t)\mathbf{y}(t)^{\mathrm{T}}] w(t)\right)</math>,
where the function Шаблон:Math sets all matrix elements above the diagonal equal to 0, and note that our output Шаблон:Math is a linear neuron.[1]
Stability and PCA
Applications
The GHA is used in applications where a self-organizing map is necessary, or where a feature or principal components analysis can be used. Examples of such cases include artificial intelligence and speech and image processing.
Its importance comes from the fact that learning is a single-layer process—that is, a synaptic weight changes only depending on the response of the inputs and outputs of that layer, thus avoiding the multi-layer dependence associated with the backpropagation algorithm. It also has a simple and predictable trade-off between learning speed and accuracy of convergence as set by the learning rate parameter Шаблон:Math.[5]
See also
References
