Lab (2)

Neural Network – Perceptron Architecture

 

Objective:       Learn about Hebbian Learning

                        Set up a network to recognize simple letters.

 

In this lab we will try to review the Hebbian rule and then set a network for recognition of some English characters that are made in 4x3 pixel frame.

 

Linear Associator

Hebb’s learning law can be used in combination with a variety of neural network architectures.  Here we will use the Linear Associator.  This network consists of an input layer and a linear layer.  The output layer uses a purlin function to compute the activation.

 

a = Wp or .

 

The task of a Linear Associator network is to learn a Q pairs of prototype input/output vectors:

 

{p1, t1}, {p2, t2}, {p3, t3}, {p4, t4}, …{pq, tq}.

 

If the network receives a slight change in the input, there will be a slight change also in the output.

 

The Hebb’s Rule

In a very brief statement, Hebb’s rule applies when we want to update the weight matrix.  In this network both the input and output plays a rule in the updating values.

 

 

In matrix form, this can be written as:

 

 

By looking at this statement, one can simplify the first iterations by assuming the initial weight to be 0.  Thus, the weight matrix will become:

 

= TPT

 

We used normalization with this rule in class.  We will try that again here.  Sometimes, the network is unable to produce the correct results.  We can normalize the input and see how that affects the output.  There is also a method that we can use to help the network converged.

Pseudoinverse Rule

Using this rule, we will use a different procedure to compute the weight matrix:

 

 

Simple Classification Example

This is an auto association problem in which we have three prototype patterns.  We want to design an autoassociative network to recognize these patterns, and then run the network for the test inputs.  Here is what you need to do:

A)    Train the network using Hebb’s learning rule, without normalization.  If the network produced 0 error, then you are done.  Try the test cases.

B)     Train the network, but this time use the normalize input.  If the network produced 0 error, then try the test cases.  There are two commands that you can use to normalize either the rows or the columns of matrix, normr and normc.  If you wish to normalize p, then each column represents one of the inputs, so you want to normalize columns.  If you used the transpose of the input, then each row represents an input, so you need to normalize the rows.

C)    Use the pseudoinverse rule to train the network.  Your network should produce the correct outputs with the original input matrix, no normalization is needed. To compute inverse, you can use the inv command. 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Simple Letter Recognition

For this section of the lab you need to submit the list of all commands that you have used to produce the results in each part.  You can write the commands either by hand or type the commands in a file and then print the file out at the end.  Note: commands only.

 

In the last class period, we generated 10 characters using a 4x3 pixel frame.  These capital letters were: A, C, F, I, J, L, O, P, T, and U.  We also assumed the –1 is used for blank pixels and 1 for colored pixels to generate the codes representing each letters.

 

The first thing we want to do here is to duplicate that work again here and build the input matrix.

             A                         C                            F                          I                          J                   

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

             L                          O                            P                          T                         U                 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Build the codes for each letter and then write the matrix PT below.   (related load)

 

 

Using Hebb Rule in Matlab

Now that we have the matrix let’s go start the Matlab and perform the computations.

 

Start the Matlab and create p, one row for each input (relate load):

>> pt = [ 1 1 1 1 1 -1 1 -1 1 1 1 1 …..

               …..

 

>> pt    (to make sure it produces the correct p)

 

To distinguish between P and PT, let’s use p for P and pt for PT.  Since we want the network to produce the input, our target is the same as P.  Thus,

>> p = pt’

>> t = p

 

Now you have the target matrix t, the input matrix p, and the transpose of the input matrix, pt.

 

Find w ?

>> w = t * pt

 

Then test the input with this weight matrix. 

 

>> a = w * p

>> e = t – a

 

Do you have 0 for your error?

 

What can we do?  Let’s just for the sake of it, try the normalized pt. Find the pt normalized.  Find out how?  (related normc and normr)

>> pnt =

 

Once you fond the pt normalized, pnt, pn can be found transposing pnt, pn = pnt’. The target matrix is the same as pnt.   compute t.

>> pn = pnt’

>> t = pn. 

 

 

Then compute the w, W = TPn

 

>> w  = t * pnt

 

finally, compute a, a = WPn

>> a = w * p

>> e = t – a

 

Did you get 0 for error?

 

Pseudoinverse Rule

The last thing we want to do is to use the Pseudoinverse rule to solve this problem.  

 

 

Note that we need the inverse of a matrix for this case. (related inv)

 

We need to use the original P for this part.  You can compute the entire W with one command.  I leave it up to you to decide how you want to do it.  Compute that W and then find a = WP.  Find the error for this case. 

 

If by you didn’t get 0 for the error, try to use the normalize matrix to see if that helps.

 

Variations of Hebbian Learning

In case that the Hebbian didn’t produce the correct results, we can use an iterative method to update the weight matrix:

 

 

where is the learning rate, to begin, you can assume that to be 1, a is the output that you have produced and P is the input matrix.