PSYC 4032 PSYCHOLOGY OF LEARNING EXAM 4 STUDYGUIDE (Chapters 5, 6, 9, 10, 12)

---

Edward Lee Thorndike (1874-1949)

While Pavlov was developing a general model of learning involving "reflexes" and
classical conditioning (an approach that was becoming popular in Europe),
Thorndike was also carrying out experiments on animal learning. Thorndike was
interested in how animals learn to solve problems. His approach was fundamentally different than Pavlov's. While Pavlov was interested in how animals react to various stimuli, Thorndike was interested in how the animal responds to a situation in the environment in an effort to achieve some result.
If Thorndike had been in Pavlov's lab he would have wondered how dogs learn to
produce specific behaviour in order to get food. (For example, some dog owners
insist that their dog sit before being given food. Thorndike would have been
interested in how the animal learns this behaviour.)
Note that people had been interested in instrumental learning for a number of years before Pavlov and Thorndike started their experiments on learning. In particular, they were interested in showing that animals were capable of intelligent behaviour as a way of defending Darwin's theory of evolution. This was considered important because people who attacked the Theory of Natural Selection argued that humans were fundamentally different than other animals in terms of there ability to reason. What set Thorndike apart from his predecessors was that he was the first to investigate instrumental learning systematically using sound experimental methods.
Thorndike's Puzzle Box Procedure
Thorndike placed a hungry cat inside a "puzzle box" with food outside. Initially, the cat would become agitated and produce many different "random" behaviours in an attempt to get out of the cage. Eventually, the cat would press the paddle by chance, the door would open and the cat could escape and get the food. The cat would then be placed inside the box again and would again take a long time (on average) to escape after exhibiting many different behaviours.
Puzzle Box

Thorndike examined the time to escape (his operational definition of learning was thus latency) as a function of trials. The learning curve was gradual and uneven (see below).

There was little evidence of sudden insight (intelligence in the "Sherlock Holmes/CSI Miami" sense). Nevertheless, after about thirty trials, the cats
would press the paddle almost as soon as they were placed in the cage. Thorndike concluded that the animals learned by "trial and success". Based on observation such as these, Thorndike proposed a general theory of
learning which is called the Law of Effect. This law of effect states that:
The Law of Effect starts with the assumption that when an animal encounters a
new environment, it will initially produce largely random behaviours (e.g.,
scratching, digging, etc.). Over repeated trials, the animal will gradually associate some of these behaviours with good things (e.g., access to food) and these behaviours will be more likely to occur again. In Thorndike's terms, these
behaviours are "stamped in". Other behaviours that have no useful consequences are "stamped out" (see below).

Because, the more useful behaviours are more and more likely to be performed,
the animal is more and more likely to complete the task quickly. Thus, in the cat in the box example, the time to escape will tend to decrease. Note that according to Thorndike's view of learning, there is no need to postulate any further intelligent processes in the animal. There is no need to assume that the
animal notices the causal connection between the act and its consequence and no need to believe that the animal was trying to attain some goal. The animal simply learns to associate certain behaviours with satisfaction such that these behaviours become more likely to occur. Thorndike called this type of learning instrumental learning. The animal learns to produce a response that is intrumental in getting satisfaction.

---

Skinner and Operant Learning

                Burrhus Fredric Skinner (1904-1990)

Picking up on Thorndike's research on instrumental learning was another Harvard graduate student in Psychology --approximately 40 years after Thorndike left Harvard for Columbia University -- B. F. Skinner. Skinner replaced the term instrumental learning with the term operarant learning refined Thorndike's terminology and methodology  to fit the new paradigm in psychology -- Behaviorism as well as Ernst Mach's paradigmitic contribution to physics Operationalism. He began by elucidating differences between Pavlovian/Watsonian type classical conditioning and his operant learning--showing that they were fundamentally different processes.

In classical conditioning:

In operant learning:
Skinner replaced Thorndike's term instrumental responses with the term operant responses or simply operants because they operate on the world to produce a consequence (feedback from the world that has just been operated on). He also referred to instrumental learning as operant learning. Thus, operant learning is: The process through which the consequence of an operant (behavior) affects the likelihood that the behavior will be produced again in the future. Unlike reflexes, operant behaviors can be accomplished in a number of ways (compare an eyeblink to pressing a paddle) and are what we normally think of as voluntary actions. In operant learning, the emphasis is on the consequences of a motor act rather than the act in and of itself. Skinner, like Thorndike, believed in the Law of Effect. He believed that the
tendency to emit an operant behavior is strengthened or weakened by the consequences of the response. However, he avoided mentalistic terms and interpretations. Thus for example he used the term reinforcer, instead of reward, to refer to the stimulus change that occurs after a behavior and tends to make that behavior more likely to occur in the future. (The term "satisfaction" was distasteful to Skinner due to its anthropomorphizing and loose use of language . After all, how can we know if a cat is satisfied by the food is gets when it escapes from the cage? All we really know is that the behavior leading to the food will be more likely to occur again in a similar situation.) Thus Thorndike's law of effect becomes a logical contingency:
Skinner Box
Skinner developed a new method for studying operant learning using what is
commonly called a "Skinner box". Skinner boxes are also called operant chambers.
Operant Chamber Illustration

A Skinner box is a cage with a lever or some other mechanism that the animal can operate to produce some effect, such as the delivery of a small amount of juice. The advantage of the Skinner box over Thorndike's puzzle box is that the animal does not have to be replaced into the cage on each trial. With the Skinner box, the animal is left in the box for the experimental session and is free to respond whenever it wishes. The standard measurement used by Skinner to assess operant learning was the rate of responses. (This was Skinner's operational definition of learning.) Skinner and his followers argued that virtually everything we do can be understood as operant or instrumental responses that occur because of their past reinforcement and that this is independent of whether or not we are aware of the consequences of our behaviour. For example, if the students in a class all smiled when the Professor walks to the right side of the room but put on blank expressions when he or she walks to the left, there is a good chance that the Professor will end up spending most of the lecture on the right - even thought he or she is not aware of what is happening. A more simple effect you can have on your Professor is to simply be alert and enthusiastic. This will tend to make him or her more enthusiastic and you will get a better lecture.
Four Consequences of Behaviour
As mentioned above, Skinner believed that operant behaviour (i.e., operant
responses) is determined by its consequences. He identified four possible
consequences of behaviour:
      Any stimulus that increases the probability of a behaviour (e.g.,
      access to fish is a positive reinforcer for a cat).
      Familiar examples of positive reinforcement: studying and gambling.
      Any stimulus whose removal increases the probability of a behaviour.
      For example, bar pressing that turns off a shock.
      Any stimulus whose presence (as opposed to absence in -ve
      reinforcement) decreases the probability of behaviour. For example,
      bar press that leads to a shock.
    Any stimulus whose removal decreases the probability of a behaviour. Skinner thought that punishment was the least effective of the 4 possible consequences for learning.
Processes Associated with Operant Conditioning
As with classical conditioning, there are a number of processes in operant
conditioning.
Shaping
Imagine a rat in Skinner box where a pellet of food will be delivered whenever the animal presses a lever. What happens if the rat in a Skinner box never presses the lever? To deal with this problem, one can use a procedure known as "shaping". One might start by providing the reinforcement when the rat gets close to the lever. This increases the chance that the rat will touch the lever by accident. Then you provide reinforcement when the animal touches the lever but not when the animal is near the lever. Now you hope the animal will eventually press the lever and, when it does, you only reinforce pressing.
Thus, shaping involves reinforcing behaviours that are increasingly similar to
desired behavior. (Shaping is sometimes called the method of successive approximations.)


Extinction and Spontaneous Recovery
Extinction in operant conditioning is similar to extinction in classical conditioning. If the reinforcer is no longer paired with the response, the response decreases. e.g., people stop smiling if you do not smile back.
The response also exhibits spontaneous recovery some time after the extinction session.
Extinction of operant learning has two non-intuitive facts:
This may reflect the fact that the onset of extinction is more "obvious".
Stimulus Control
The instrumental or operant response in operant condition is not elicited by an external stimuli but is, in Skinner's terms, emitted from within. But this does not
mean that external stimuli have no effect. In fact they do exert considerable control over behaviour because they serve as discriminative stimuli.
Suppose a pigeon is trained to hop on a treadle to get some grain. When a green
light comes on, hopping on the treadle will pay off, but when a red light comes on it will not. In this case, the green light becomes a positive discriminative stimuli (S+) and the red light becomes a negative discriminative stimuli (S-).
Note that the S+ does not signal food in the way that the CS+ might in a Pavlov's
laboratory. (Recall the example with the black and gray squares where after
training the animal salivates in response to a black square, the CS+, but not a gray square.) Instead, the S+ signals a particular relationship between the instrumental response and the reinforcer telling the pigeon "if you jump now, you will get food." A variety of techniques have been used to study the role of discriminative stimuli in operant learning and many of the results mirror those of generalization of discrimination in classical conditioning.
For example, if a pigeon is trained to respond only when a yellow light appeared,
after training, it will also respond to lights of a different colour. However, there is a response gradient - the response decreases with the size of the difference
(measured in terms of wave frequency) between the test light and original yellow
light (i.e., the original discriminative stimulus).

The following cartoon illustrates an experiment in which a rat learns to discriminate between a triangle and a square in order to get food.
                    Discriminative Stimuli (after Lashley, 1930)


Reinforcement Schedules in Operant Conditioning
A major area of research in Operant Learning is on the effects of different
reinforcement schedules. The first distinction is between partial and continuous
reinforcement.

•          Continuous Reinforcement: every response is reinforced

•          Partial or Intermittent Reinforcement: only some responses are reinforced.

In initial training, continuous reinforcement is the most efficient but after a response is learned, the animal will continue to perform with partial reinforcement. Extinction is slower following partial reinforcement than following continuous reinforcement. Skinner and others have described four basic schedules of partial reinforcement which have different effects on the rate and pattern of responding.
We have fixed and variable interval and ratio schedules.
         Ratio schedules: reinforcer given after some number of responses.
         Interval schedules: reinforcer given after some time period.
         Fixed: the number of responses or time period is held constant.
         Variable: the number of responses or the time period is varied around a mean.
Typical Behaviour with the 4 Schedules:

•          Fixed-Ratio: bursts of responses.

•          Variable-Ratio: extremely high, steady rate of responding. (Slot machines work on a VR schedule). Known to produce extreme behavior patterns categorized as compulsive or addictive behavior patterns
  

•          Fixed-Interval: pauses with accelerating responses as the time approaches (the "scallop effect").

•          Variable-Interval: after training, a slow, steady pattern of responses is usually seen.
  

Response rate is generally higher with the ratio schedules.

Yes I know that Operant Learning Theory is:
laiden with terms, but we need this exact vocabulary to avoid misunderstandings that might occur if we relied on layman's terms. Example:  Although we commonly understand that a "reward" provides a subject with some psychological pleasure or satisfaction, the term has at least 3 significant drawbacks for the scientist:
Reinforcer, on the other hand, is more carefully defined.  Indeed, whether an event is a reinforcer depends only on the effect it has on behavior.
Primary vs. Conditioned Reinforcement:
The final part of the puzzle comes in the form of conditioned reinforcement (S^r) NOTICE THE LOWER CASE "r"!!!. This is a concept that seems to cause a fair amount of confusion, so follow along...imagine a rat in an operant chamber who has learned that if it presses a lever when a green light is on it will receive food (an example of a Primary Reinforcer--stimuli needed for survival by the organism --food, water, oxygen, etc.) But not when a red light is on. In the example of our friend the rat, above, the S+  component seems pretty straight-forward. But the green light, because it has been associated with food in the past, can be used as a reinforcer to maintain behavior as well. Like a CS, the reinforcing properties of the green light exist because of the unique past history of this individual rat (rats which have not been trained in the presence of green lights signaling the availability of food show little interest in green lights). Lever pressing can be maintained, at least to some extent, by making the appearance of the green light contingent upon some behavior of the rat, such as pressing another lever. Now, where S^r gets complex (sorry!) is that there are two basic types of conditioned reinforcement--one type (token reinforcement) in which the S^r is a necessary step along the way to receiving Primary reinforcement (S^R), and one in which S^r is not a necessary condition for obtaining reinforcement but has reinforcing properties because it has been associated with powerful primary/biologically significant reinforcing events in the past. Most discussions of S^r are about token reinforcement. The term refers to the fact that with this type of reinfrocement, the S^r must be accumulated (like money, or tokens) before the Primary reinforcement (S^R) can be obtained. For example, getting points on a test, a grade in a class, etc., are examples of token reinforcement in that a student needs to pass a test (get points, or tokens), get a passing grade in a class, etc., to eventually graduate and get a diploma (another S^r, but more on this later). In this case, S^r makes perfectly good sense that it will maintain behavior, since the bigger contingency out there can't be had without picking up this one along the way. That's not to say that the S^r is unimportant; it does maintain the behavior by providing a small, but immediate, reinforcement for the behavior.

The other kind of S^r is usually much more subtle. We don't need to get it, like we do with the "necessary step" type. But it is still very important in our lives. Getting a smile when we've said an encouraging word to a friend, or receiving a smile when we really need it, may not be necessary (in that it is not tied into a larger reinforcement system), but it still "feels good". And why does a smile have reinforcing properties? Because, like the green light for the rat, it has been present when other "good" things have happened to us.

Operant Learning (B.F. Skinner in his own words)

At the same time as this work was appearing in the USA the Polish psychologists Konorski and Miller began the first cognitive analyses of classical conditioning - the forerunners of the work of Rescorla, Wagner, Dickinson and Mackintosh. In case you had forgotten here is a very basic review of the Rescorla/Wagner reinterpretation of Pavlovian conditioning as Cognitive Neuroscience in the Information Processing tradition: According to Rescorla and Kamin, associations are only learned when a surprising event accompanies a CS. In a normal simple conditioning experiment the US is surprising the first few times it is experienced so it is associated with salient stimuli which immediately precede it. In a blocking experiment once the association between the CS (CS1) presented in the first phase of the procedure and the US has been made the US is no longer surprising (since it is predicted by CS1). In the second phase, where both CS1  and CS2 are experienced, as the US is no longer surprising it does not induce any further learning and so no association is made between the US and CS2. This explanation was presented by Rescorla and Wagner (1972) as a formal model of conditioning which expresses the capacity a CS has to become associated with a US at any given time. This associative strength of the US to the CS is referred to by the letter V and the change in this strength which occurs on each trial of conditioning is called dV. The more a CS is associated with a US the less additional association the US can induce. This informal explanation of the role of US surprise and of CS (and US) salience in the process of conditioning can be stated as follows:
dV = ab(L - V)
where a is the salience (intensity) of the US, b is the salience (intensity) of the CS and L is the amount of processing given to a completely unpredicted US. In words: when the US is first encountered the CS has no association to it so V is zero. On the first trial the CS gains a strength of abL in its association with the US which is proportional to the saliences of the CS and the US and to the initial amount of processing given to the US. As we start trial two the associative strength is V is abL so the change in strength that occurs with the second pairing of the CS and US is ab(L - abL). It is smaller than the amount learned on the first trial and this reduction in amount that is learned reflects the fact that the CS now has some association with the US, so the US is less surprising (cute...very cute--oops I'm not supposed to impose my opinions). As more trials ensue, the equation predicts a gradually decreasing rate of learning which reaches an asymptote at L.
However, the diagram below shows: this is not what is seen when the development CS-US associations is measured over time. Instead the learning curve is sigmoidal. Rescorla has argued that the equation is consistent with observed behavior if one assumes that very small changes in associative strength are undetectable and that there is a limit to the amount of effect that very large changes can have on behavior.

CS-US aquisition
There are other respects, however, where the model performs better in predicting experimental outcomes. It can also be applied to a number of CSs each of which contributes to an overall associative strength V of the US in the right hand side of the equation. It is reasonably clear that the presence of the CS salience term b in the equation lets it account for overshadowing. The meaning of the equation is clearest if the specific dVs on the left hand side are seen as referring to the increments in association between specific CSs while V on the right hand side is referring to the predictability of the US and so is the sum of all the different CS-US associations. If the conditioning strength accrued to CS1 is denoted by dV1 and that to CS2 by dV2 then our equations are:
dV1 = ab1(L - V)
dV2 = ab2(L - V)
and both dV1 and dV2 accrue to V on each trial. The amount of association directed to each CS is proportional to their salience.
The equation also models blocking well. During the initial phase of a blocking experiment the associative strength of the US is increased so later, when a second CS is presented the amount of associative strength it can gain has been reduced.
The critical question is, however, does the model predict experimental outcomes it was not explicitly divised for, i.e. can it be generalized? In one example the model predicts the effects of pairing two previously learned CSs on learning about a third new stimulus. If on separate occasions (not as compound stimuli) two CSs of equal salience have both been completely associated with a US then V=L for both stimuli and dV on subsequent trials is zero for both. Now a third CS in conjunction with the original pair is presented so three CSs are presented together whereas only two of them were presented singly in the past. The overall associative strength of the US is now 2L, a contribution of L from both of the original CSs. The equation predicts that there will be a negative change in associative strength on this trial proportional to the salience of the CSs:
dV = ab(L - 2L)
dV = -abL
Conducting the experiment shows: the third stimulus becomes a conditioned inhibitor of the US - it provokes a CR of the opposite quality to that produced by the other two CSs.