🚀 go-pugleaf

Dave,

I am posting the responses I have received so far. Some of
the responses provide a great deal of insight on connectionist
learning and neuroscience and their interactions (see, in
particular, the second note by Dr. Peter Cariani). I have also
tried to provide answers to two frequently asked questions.

I hope all of this will generate more interest in the questions
being raised about connectionist learning. As you can see below,
perhaps other questions need to be raised. The original posting is
attached below for reference.

Asim Roy
Arizona State University
=========================================================

ANSWERS TO SOME FREQUENTLY ASKED QUESTIONS:

a) Humans get stuck in local minima all the time. So what is
wrong with algorithms getting stuck in local minima?

RESPONSE:

We can only claim that humans are sometimes "unable to learn."
We cannot make any claim beyond that. And so this phenomenon of
"unable to learn" does not necessarily imply "getting stuck in a
local minima." Inability to learn maybe due to a number of
reasons, including insufficient information, inability to extract
the relevant features of a problem, insufficient reward or
punishment, and so on. Again, to reiterate, "inability to learn"
does not imply "getting stuck in a local minima."

Perhaps this misconception has been promoted in order to
justify certain algorithms and their weak learning
characteristics.


b) Why do you say classical connectionist learning is
memoryless? Isn't memory actually in the weights?

RESPONSE:

Memoryless learning implies there is no EXPLICIT storage of any
learning example in the system in order to learn. In classical
connectionist learning, the weights of the net are adjusted
whenever a learning example is presented, but it is promptly
forgotten by the system. There is no EXPLICIT storage of any
presented example in the system. That is the generally accepted
view of "adaptive" or "on-line learning systems."

Imagine such a system "planted" in some human brain. And suppose
we want to train it to learn addition. So we provide the first
example - say, 2 + 2 = 4. This system then uses the example to
promptly adjust the weights of the net and forgets the particular
example. It has done what it is supposed to do - adjust the
weights, given a learning example. Suppose, you then ask this
"human", fitted with this learning algorithm: "How much is
2 + 2?" Since it has only seen one example and has not yet fully
grasped the rule for adding numbers, it probably would give a
wrong answer. So you, as the teacher, perhaps might ask at that
point: "I just told you 2 + 2 = 4. What do you mean you don't
remember?" And this "human" might respond: "Very honestly, I
don't recall you ever having said that! I am very sorry." And
this would continue to happen after every example you present
to this "human"!!!

So do you think there is memory in those "weights"? Do you think
humans are like that?

Please send any comments on these issues directly to me
(asim.roy@asu.edu). All comments/criticisms/suggestions are
welcome. And all good science depends on vigorous debate.

Asim Roy
Arizona State University

=========================================================

From Prof. Robert Fry <FRYRL@f1groups.fsd.jhuapl.edu>

I found your commentary in the recent Connectionists posting very
interesting.  I have been working on the logical basis for neural
computation for well over a decade now and have what I feel are
many exciting results that I think help provide some focus
regarding how to approach the problem of the understanding and
designing artificial neural systems.  I have enclosed some
references -- not to impress you, but to give you a flavor for
the logical foundation that I have struggled with ultimately to
get it into an extremely simple and insightful explanation of
neural computation.

Anyway, I am very interested in what you talked about and am
interested in trying to attend the described panel. I think that
finally, researchers are beginning to ask the right questions.
I believe that learning itself is the computational search for
the "right" questions and so perhaps we collectively, are about
to really learn something about how the computational
objective(s) of the brain and neural computation in general.

    Sincerely,
     Prof. Robert L. Fry

Relevant Publication List

 "A logical basis for neural network design," invited book
chapter in Theory and application of neural networks, Academic
Press, to be published 1997.

 "Neural Mechanics," invited paper, Proc. Int. Conf. Neur. Info.
Proc., Hong Kong, 1996.

 "Rational neural models based on information theory," invited
paper, Post-conference workshop on Neural Information
Processing, NIPS'95.

 "Observer-participant models of neural processing," IEEE Trans.
Neural Networks, July 1995

 "Rational neural models based on information theory," 1995
Workshop on Maximum Entropy and Bayesian Methods, Sante Fe, NM,
July 1995, sponsored by the Sante Fe Institute and Lawrence
Livermore Laboratory

 "Rational neural models," Information theory and the brain
workshop, Stirling, Scotland, Sept 1995.

 "Neural processing of information," R. L. Fry, paper
presentation at 1994 IEEE International Symposium on
Information Theory, Norway.

 "A mathematical basis for an entropy machine that makes rational
decisions," APL Internal Memo F1F(1)90-U-094, 1990.

 "Neural models for the distributed processing of information,"
APL IRAD report,  1991.

 "The principle of cross-entropy applied to neural networks and
autonomous target recognition," APL Internal Memo F1F(1)90-U-005,
1990.

 "Maximized mutual information using macrocanonical probability
distributions," 1994 IEEE/IMS Workshop on Information Theory and
Statistics, Arlington, VA.

=============================================================

From Prof. Danny Silver <dsilver@csd.uwo.ca>

I have been following your discussion and summarizations with
some interest over the last few months and would like to provide
some input at this point. Many of the issues you raise are quite
important - in particular your call for a clear set of external
objectives for any "brain-like" learner.  However, the general
context in which you have framed the question is somewhat limited.
As I understand it, you are suggesting that the classical context
of local learning and predefined structure is unrealistic.  In
many ways I agree .. I beleive the larger and more important
context involves the issues of what has been called "life-long
learning" and "learning to learn" and "consolidation and
transfer of knowledge". I have no idea why so many researchers
continue to pursue the development of the next best inductive
algorithm or archiecture (be it ANN or not)  when many of them
understand that the percentage gains on predictive accuracy
based solely on an example set is marginal in comparison to the
use of prior knowledge (selection of inductive bias).  The
research community has looked very carefully at how we induce
ANN models from examples ... but in comparision there has been
very little work on the consolidation and transfer of the
knowledge contained in previously learned ANN models. Most of
the questions you have posed are subsumed by this larger =0Acontext.Primari=
ly .. what type of mechanism(s)  is required
to (1) learn a task taking advantage of previous learning
(prior knowledge), and  (2) consolidate this new task knowledge
for use in future learning. I do not think that the CNS (central
nervous system) is successful in doing this - just by chance -
there is something unique about it's  architecture that  allows =0Athis too=
 occur.   Recent work by myself, Lorien Pratt, Sebatian =0AThrun, Rich Caru=
ana, Tom Mitchell, Mark Ring, Johnathon Baxter,
and others [NIPS 95 work shop on learning to learn] have provided =0Afounda=
tions on which to build in this area.   I encourage to
review some of this material if you are interested and feel it
applies.

You can start by cheking out my homepage at
ww.csd.uuwo.ca/~dsilver It will lead you to other related areas.

=================================================================
Daniel L. Silver    University of Western Ontario, London, Canada
                     N6A 3K7 - Dept. of Comp. Sci.
  dsilver@csd.uwo.ca  H: (902)582-7558   O: (902)494-1813
  WWW home page ....  http://www.csd.uwo.ca/~dsilver   
=================================================================

From Prof. Vassilis G. Kaburlasos <vgkabs@amphipolis.ee.auth.gr>

Regarding the panel discussion on "connectionist learning : is it
time to reconsider the foundations ?", I read the arguments with
interest and I would like to pose, in addition, another issue.

That is, to simulate convincingly a biological system we should
not probably be dealing solely with vectors of real numbers. The
capacity to deal with symbols and other types of data merits also
attention. In other words, besides memory and more global
learning capabilities, it will be advantageous to be able to
handle jointly disparate data such as real numbers, fuzzy sets,
propositional statements, etc.

Therefore from a model development point of view it might be
quite advantageous to consider working on less structured
spaces than the conventional N-dimensional Euclidean space.
Such a space could be a (mathematical) lattice. Note that all
the previously mentioned data are in effect elements of a
mathematical lattice. That is, not only the conventional
Euclidean space is a lattice but also the set of propositional
statements, the collection of fuzzy sets on a universe of
discourse, etc.

My tentative proposion is this : For machine learning purposes
only, replace the Euclidean-space by a Lattice-space.

Just imagine how much the learning and decision making robustness
of a system would be enhanced if in addition to memory, the
capability to design the net on its own, polynomial complexity,
generalization capability, etc., the system in question in also
able to handle jointly dsparate data.

 With considerations,

Vassilis G. Kaburlasos
Aristotle University of Thessaloniki, Greece

================================================================

From Prof. Weber Martins - weber@eee.ufg.br

Just to congratulate you for organizing such an important panel.
On the foundations, I would like to remind you that even the
point that learning must occur by synaptic adjustments is another
question that should be discussed in my opinion.

===============================================================

An additional note from Prof. Weber Martins - weber@eee.ufg.br

As a researcher with Computer and Electrical Engineering
background, I was not talking about biological plausibility. I
am interested in the resolution of complex problems in a
efficient way. I work mainly with weightless (Boolean) neural
networks. In those models, we usually deal with the adjusting
of neuronal "behavior", not synapses. In my point of view, the
Neural Network community uses homogeneous networks (with the
same type of neuron throughtout the network) to simplify
mathematical analysis and algorithms. However, from many books
I read on NN, it seems that if you don't adjust synapses, you're
not doing NN... From an computational point of view, this doesn't =0Alook r=
ight to me.

===============================================================

From Dr. Peter Cariani <peter@epl.meei.harvard.edu>

I presume that the panel organizers really want to focus on these
two assumptions of connectionist learning rather than looking at
other assumptions that are embedded in the whole connectionist
approach. We should recognize, however, that the two assumptions
listed above do not begin to exhaust the theoretical commitments
that are bound up in most connectionist models.

Some of these other assumptions are:
1) Place-coding.
Inputs to the network are encoded by spatial patterns of
activation of elements in the input layer rather than in the
time-structure of the inputs (i.e. rate-place or labelled-line
neural codes rather than temporal pattern codes). To the extent
that input patterns are encoded temporally, in the std view a
time-delay layer is used to convert this into a place pattern
that is handled by a conventional network. However, examples of
fine time structure have been found in many parts of the brain
where there has been a concerted effort to look for it.

2) Scalar signals.
There is one signal conveyed per element (no multiplexing of
multiple signals onto the same lines), so that inputs to and
outputs from each element are scalar quantities (1 and 2 are
closely related).

3) Synchronous or near-synchronous operation.
The inputs are summed together to produce an output signal for
each time step. The std. neurobiological assumption is that
neurons function as rate-integrators with integration times
comparable to the time-steps of the inputs rather than
coincidence detectors. But examples of coincidence detection
in the brain are many, and there has been an ongoing debate
about whether the cortical pyramidal cell is best seen in terms
of an integrating element or as a coincidence detector.

4) Fan-out of the same signals to all target elements.
Connection weights may differ, but the same signals are being
fed to all of their targets. The std. neurobiological assumption
is that impulses of spike trains invade all of the daughter
branches of axonal trees. However, there may be conduction blocks
at branchpoints that mean that some spikes will not propapagate
into some terminals (so that the targets do not receive all of
the spikes that were transmitted through the axon trunk).
There are examples of this in the crayfish.

To the extent that one or more of these assumptions are altered,
one can potentially get neural architectures with different
topologies, and with potentially different capacities.

I'm not one to quibble over what the topic of a discussion should
be -- that's up to the participants. I'd just like to suggest
that if we (a very general and inclusive "we") are going to
"reconsider the foundations" of connectionism, we might think
more broadly about ALL the assumptions, tacit and explicit,
that are involved.

My own sense is that, in lieu of a much deeper understanding of
exactly what kinds of computational operations are being carried
out by cortical structures and how these are carried out, we
should probably avoid labels like "brain-like" that give the
false sense that we understand more than we do about how the
brain works.

If one is seriously interested in how "brain-like" a given
network architecture is, then one needs to get real, detailed
neuroanatomical and/or neurophysiological data and make the
comparison more directly. Comparisons with what's in the
textbooks just doesn't do. Things get messy very fast when the
neural responses are phasic, nonmonotonic, and have a multitude
of different kinds of stimuli that produce them.

Peter Cariani

Peter Cariani, Ph.D.
Eaton Peabody Laboratory
Massachusetts Eye & Ear Infirmary
243 Charles St, Boston MA 02114
tel       (617) 573-4243
FAX      (617) 720-4408
email    peter@epl.meei.harvard.edu
============================================================

Asim Roy's note to Peter Cariani:

If I understand you correctly, you are saying that we need to
broaden the set of questions. I am sure this issue will come
up as we grapple with these questions. And one of the issues
from the artificial neural network point of view is how exactly
do you replicate the detailed biological processes. You
certainly want to extract the clever biological ideas, but at
some point, say 50 years from now, we might do better than
biology with our artificial systems. And we might do things
differently than in biology. An example is our flying machines.
We do better than the birds out there. The functionality is
there, but we do it differently and do it better. I think the
point I am making is that we need not be tied to every biological
detail, but we certainly want to pick up the good ideas and then
develop them further. And in the end, we would have a system far
superior to the biological ones, but not exactly like it in all
the details.

==============================================================

Peter Cariani's reply to the above note:

Yes, I think the most important decisions we make regarding
how to construct neural nets in the image of the brain are to
determine exactly which aspects of the real biological system
are functionally relevant and which ones are not. I definitely
agree with you that every biological detail is not important,
and I myself am trying to work out highly abstracted schemes
for how temporally-structured signals might be processed by
arrays of coincidence detectors and delay lines. (Usually
the standard criticisms from biologists are that not enough
of the biological details are included.) What I am saying,
however, is that the basic functional organization that is
assumed by the standard connectionist models may not be the
right one (or it may not be the only one or the canonical
one). There are many aspects of connectionist models that
really don't seem to couple well to the neurophysiology,
so maybe we should go back and re-examine our assumptions
about neural coding. I myself think that temporal pattern
codes are far more promising than is commonly thought,
but that's my idee fixe. (I could send you a review
that I wrote on them if you'd like).

I definitely agree with you that once we understand the
functional organization of the brain as an information
processing system, then we will be able to build devices
that are far superior to the biological ones. My motto
is: "keep your hands wet, but your mind dry" -- it's
important to pay attention to the biology, to not project
one's preconceptions onto the system, but it's equally
important to keep one's eyes on the essentials, to avoid
getting bogged down in largely irrelevant details. I've
worked on both adaptive systems and neurophysiology, and
I know that the dry people tend to get their neural models
from textbooks (that present an overly simplified and uncritical
view of things), and the wet people tend not to say much about
what kinds of general mechanisms are being used (they look
to the NN people for general theories). There is a cycling of
ideas between the two that we need to be somewhat wary of ---
many neuroscientists begin to believe that information processing
must be done in the ways that the neural networks people suggest,
and consequently, they cast the interpretation of the data in
that light. The NN people then use the physiological evidence
to suggest that their models are "brain-like". Physiological
data gets shoe-horned into a connectionist account (and
especially what aspects of neural activity people decide to go
out and observe), and the connectionist account is then held
up as being an adequate theory of how the brain works. There
are very few strong connectionist accounts that I know of that
really stand up under scrutiny -- that are grounded in the data, =0Athat pr=
edict important aspects of the behavior, and that cannot
be explained through other sets of assumptions. In these
discussions you really need both the physiologists, who
understand the complexities and limitations of the data, and
the theorists, who understand the functional implications, to
interact strongly with each other.

So, anyway, I wish you the best of luck with your session.

==============================================================

From Dr. Stefan Kremer <Stefan.Kremer@crc.doc.ca >

Your abstract is very interesting.  It sounds like it will be a
great discussion.  The idea of using a different kind of learning
which explicitly stored training data is one I've worked on in
the past.  A few questions that crossed my mind while reading
the abstract are listed below:

On Wed, 23 Apr 1997, Asim Roy wrote:

> Classical connectionist learning is based on two key ideas.
>First, no training examples are to be stored by the learning
>algorithm in its memory (memoryless learning).

I'm a bit unclear about this.  Aren't the weights of the network
trained to implicitly store ALL the training examples?  I would
have said that that connectionist learning is based on the idea
that "ALL training examples are to be stored by the learning
algorithm in its memory"!

If I understand correctly, the first key idea is that the
training examples are not EXPLICITLY stored in a way in which
they could be retrieved or reconstructed.  Perhaps my confusion
lies in the word stored.  How would you define that?

I would further say that a number of dynamical recurrent
networks like those discussed at my NIPS workshop =0A(http://running.dgcd.d=
oc.ca/NIPS/) do explictly store presented
examples.  Infact, training algorithms like back-propagation
through time have been criticized for having to explicitly store
previous input and hidden unit patterns and thus consume extra
memory resources.  But, I guess you're probably aware of this
since you have Lee on your panel.

> The second key idea is that of local
> learning - that the nodes of a network are autonomous learners.
> Local learning embodies the viewpoint that simple, autonomous
> learners, such as the single nodes of a network, can in fact
> produce complex behavior in a collective fashion. This second
>idea, in its purest form, implies a predefined net being provided
>to the algorithm for learning, such as in multilayer perceptrons.

In what sense are the learners autonomous?  In the MLP each
learner requires a feedback error value provided by another
node (and ultimately an outside source) in-order to update.
I would say its NOT autonomous.

> Second, strict local learning (e.g. back propagation type
>learning) is not a feasible idea for any system, biological
>or otherwise.

If it is not feasible for "any" learning system then any system
which attempts to use it must fail.  Therefor, working
connectionist networks must not use strict local learning. 
Therefore, strict local learning cannot be one of the
fundamental ideas of the connectionist approach.  Therefore why
are we discussing it?

I must have misunderstood something here...any ideas where I
went off-track?
-------------
Dr. Stefan C. Kremer, Research Scientist, Artificial Neural Systems
Communications Research Centre, 3701 Carling Ave.,
P.O. Box 11490, Station H, Ottawa, Ontario   K2H 8S2

WWW: http://running.dgcd.doc.ca/~kremer/index.html
Tel: (613)990-8175  Fax: (613)990-8369 E-mail:
Stefan.Kremer@crc.doc.ca

===============================================================

From Prof. Andrew Coward <ANDREW.COWARD.0082863@nt.com>

In response to your invitation for comments on
cogpsy@coglab.psy.soton.ac.uk I have the following :

I believe it is past time to reconsider the foundations.  The
critical deficiency in current connectionism is lack of
understanding of the meaning of the term "system architecture".
Any system which performs a complex function using large numbers
of components experiences very severe constraints on its system
architecture if building, repairing or adding features to the
system are to be feasible. Current electronic systems are in
production which use billions of individual transistors.  Such
systems must have a simple functional architecture which can (for
example) relate failures experienced at the high functional level
to defects occurring at the device level. The reason the von
Neumann architecture is ubiquitous for electronic systems  is
that it provides the means for such a simple relationship. It
achieves this by partitioning functionality into consistent
elements at a number of levels of detail. The consistent element
is the instruction, and instruction based descriptions can be
seen at every level. Typical levels include device (instruction
'open gate');  assembly code (instruction 'jump'); software
(instruction 'do:[  ]');  procedure call (instruction 'disconnect
x'); through features (instruction 'carry out Y') to major system
functions (instruction 'periodically test for problems') and
overall function.

The requirement for a function to operate within a simple
architecture is crucial. To illustrate, if I needed to design a
function to connect telephones together, many designs would be
possible, and would carry out the function efficiently, in some
cases more efficiently than designs actually in use. However, the
vast majority of those designs would be useless once it was
necessary for them to interact with and support functions like
testing (a connection did not work, which component is
defective ?) or billing (who should be charged how much for
that call ?) or adding features (allow recipient of call to
identify caller).  Conclusions drawn about human cognition from
a simulation which performs (for example) face recognition
without considering how that function would fit within the total
cognitive/behavioral system are almost certainly invalid.

In my 1990 book I argued that although the brain obviously did
not have a von Neumann architecture, very similar pressures
exist for a simple functional architecture (for example, the
need to build many copies from DNA 'blueprints'). I went on to
develop a new and complete system architecture based on an
element of functionality appropriate to the brain, the pattern
extraction/action recommendation element.  This architecture
separates cognition into a few major functions, which can in
turn be partitioned further all the way down to neurons. The
functionality required in neurons is defined by the functional
partitioning at a higher level, and that functional partitioning
is in turn constrained by the information available to individual
neurons, the kind of changes which can be made in neuron
connectivity, and the timescale of such changes. (see a couple of
1997 papers).

This architecture shows phenomena which bear a remarkable
resemblance to human brain phenomena, including unguided
learning by categorization generating declarative memory;
dream sleep;  procedural memory;  emotional arousal;  and
even internally generated image sequences. All these phenomena
play a functional role in generating behavior from sensory
input, and have also been demonstrated by electronic simulation
(Coward 1996).

My response to the questions to be posed to the panelists
would be:

1.  Should memory  be used for learning? Is memoryless learning
an unnecessary restriction on learning algorithms?

In the pattern extraction hierarchy architecture,  which appears
to me to be the only option other than the obviously inapplicable
von Neumann,  one major type of learning (associated with a major
separation in the architecture) is the process of sorting
experience into categories and associating behaviors with those
categories. A category is established by extracting and recording
a set of patterns from one unfamiliar object, and developed by
adding patterns extracted from any subsequent object which
contains many of the patterns already in the category. Memory of
objects is thus the prerequisite to this type of learning,
which is associated with the cortex. Memoryless learning occurs
in other major functions and is an appropriate model in those
functions.

2.  Is local learning a sensible idea? Can better learning
algorithms be developed without this restriction?

The real issues here are first to identify what information
could feasibly be made available to a neuron (e.g.   past
firing of the neuron itself;  correlated firing of neurons
in its neighborhood;  correlated firing between the neuron and
another neuron;  correlated firing within a separate functional
group of neurons;  feedback from pleasure or pain;  or feedback
of some expected result).  The second issue is to identify the
nature of the feasible changes to the neuron which could be
produced (e.g.  assignment or removal of a neuron;  addition or
deletion of an input;  correlated addition or deletion of a set
of inputs;  changes in relative strength of inputs;  correlated
changes in the strength of a set of inputs;  general change in
effective input strengths (i.e. threshold change);  how long a
change lasts).  Only after these qualitative factors have been
defined by higher functional requirements can quantitative
algorithms can be developed.

3.  Who designs the network inside an autonomous learning system
such as the brain?

Within the pattern extraction hierarchy architecture it is
possible to start from random connectivity and sort experienced
objects into categories without guidance or feedback.

References:

Coward L.A. (1990),   'Pattern Thinking',  New York:  Praeger
(Greenwood)

Coward L.A. (1996),  'Understanding of Consciousness through
Application of Techniques for Design of Extremely Complex
Electronic Systems'  Towards a Science of Consciousness , Tucson,
Arizona.

Coward L.A. (1997),  'Unguided Categorization, Direct and
Symbolic Representation, and Evolution of Cognition in a
Modified Connectionist Theory',  to be published in Proceedings
of the Conference on New Trends in Cognitive Science, Austria
1997.

Coward L.A. (1997), 'The Pattern Extraction Architecture: a
Connectionist Alternative to the Von Neumann Architecture',
to be published in the Proceedings of International Workshop
on Artificial and Natural Neural Networks,  Canary Islands 1997.

================================================================

An additional note from Prof. Andrew Coward
<ANDREW.COWARD.0082863@nt.com>

I believe the brain has a functional architecture which I label
the pattern extraction hierarchy. At the highest level,
functionality separates into five major functions. The first
extracts constant patterns from the environment (e.g. object
color independent of illumination). The second allows the set
of patterns which have been extracted from one object to enter
the third function. The third function  generates a set of
alternative behavioral recommendations with respect to the
selected object. The fourth function selects one (or perhaps
none) of the alternatives to proceed to action, and the fifth
function implements the action. This functional separation can
be observed in the major physiology (roughly, the primary
sensory cortex, the thalamus, the cortex,  the basal ganglia,
and the cerebellum).  There are of course levels of detail and
complexity below this.  Within each function, the needs of that
function determine the functionality required from neurons
within the function, subject to what neuron functionality is
possible (which in turn is one factor forcing the use of the
architecture).

I mentioned in the earlier note that functional partitioning is
constrained by the possible neuron functionality given
limitations in the areas of the information available to
individual neurons, the kind of changes which can be made in
neuron connectivity, and the timescale of such changes.
Expanding on this somewhat:

The source of information which controls changes to extracted
pattern could be: feedback from comparison with expected result; 
feedback from pleasure or pain;  past firing of neuron itself;
correlated firing of neurons in neighbourhood;  correlated
firing between neuron and another neuron;  correlated firing
within a separate functional group of neurons

The nature of the changes to the extracted pattern produced
in the neuron could be:  assignment or removal of a neuron; =0Aaddition or =
deletion of an input;  correlated addition or
deletion of a set of inputs;  changes in relative strength of
inputs;  correlated changes in the strength of a set of inputs; =0Ageneral =
change in effective input strengths (i.e. threshold
change);  changes in sensitivity to other parameters.

The permanence of the changes to the extracted pattern could be:
change only at time source of information is present;  change
for limited time following source of information being present;
change for long period following source of information being
present.

In each high level function, the particular combination of
information used, changes which occur, and timescale is dictated
by the type of high level functionality. For example, in the
behavioral alternative generation region the changes which are
required include assignment of neurons, biased random assignment
of inputs, setting sensitivity to the arousal factor for the
assigned region, deletion of inactive inputs, and threshold
reduction. The sources of information which ultimately control
these changes are correlated firing of neurons in neighbourhood,
correlated firing of neurons in the neighbourhood with firing
of an input, correlated firing of a separate functional group
of neurons, and firing of the neuron itself.  Timescale is short
and long for different operations. Each type of change has an
associated source(s) of information and timescale.  This
combination of functionality at the neuron level gives rise
to all the phenomena of declarative memory at the higher level,
including dream sleep.

A different combination of neuron parameters is required in the
behavioral alternative selection function, and gives rise to
learning which is not declarative. In this function, pleasure
and pain act on recently firing neurons to modulate the ability
of similar firing in the future to gain control of action. I
apply the term 'memoryless' to this learning because no record
of prior states is preserved (although the memory of pleasure
and pain may be preserved in the alternative generation function).

I regard the perceptron and even the adaptive resonance neurons
as simplistic in the system sense, although it turns out that
the Hebbian neuron plays an important system role.

The above is a summary of some discussion in the papers which
have been accepted for publication in the proceedings of a
couple of conferences in the next month or so. I could send
copies if you are interested.

I appreciate the opportunity to discuss.

Andrew Coward.

================================================================
APPENDIX:

1997 International Conference on Neural Networks (ICNN'97)
Houston, Texas (June 8 -12, 1997)
----------------------------------------------------------------
Further information on the conference is available on the
conference web page:

http://www.mindspring.com/~pci-inc/ICNN97/
------------------------------------------------------------------
PANEL DISCUSSION ON

"CONNECTIONIST LEARNING: IS IT TIME TO RECONSIDER THE FOUNDATIONS?"

-------------------------------------------------------------------
This is to announce that a panel will discuss the above question
at ICNN'97 on Monday afternoon (June 9). Below is the abstract
for the panel discussion broadly outlining the questions to be
addressed. I am also attaching a slightly modified version of a
subsequent note sent to the panelist. I think the issues are
very broad and the questions are simple. The questions are not
tied to any specific "algorithm" or "network architecture" or
"task to be performed." However, the answers to these simple
questions may have an enormous effect on the "nature of
algorithms" that we would call "brain-like" and for the design
and construction of autonomous learning systems and robots. I
believe these questions also have a bearing on other brain
related sciences such as neuroscience, neurobiology and
cognitive science.

Asim Roy
Arizona State University

-------------------------
PANEL MEMBERS

1. Igor Aleksander
2. Shunichi Amari
3. Eric Baum
4. Jim Bezdek
5. Rolf Eckmiller
6. Lee Giles
7. Geoffrey Hinton
8. Dan Levine
9. Robert Marks
10. Jean Jacques Slotine
11. John G. Taylor
12. David Waltz
13. Paul Werbos
14. Nicolaos Karayiannis (Panel Moderator, ICNN'97 General Chair)
15. Asim Roy

Six of the above members are plenary speakers at the meeting.
-------------------------

PANEL TITLE:

"CONNECTIONIST LEARNING: IS IT TIME TO RECONSIDER THE FOUNDATIONS?"

ABSTRACT

Classical connectionist learning is based on two key ideas.
First, no training examples are to be stored by the learning
algorithm in its memory (memoryless learning). It can use and
perform whatever computations are needed on any particular
training example, but must forget that example before examining
others. The idea is to obviate the need for large amounts of
memory to store a large number of training examples. The second
key idea is that of local learning - that the nodes of a network
are autonomous learners.Local learning embodies the viewpoint
that simple, autonomous learners, such as the single nodes of a
network, can in fact produce complex behavior in a collective
fashion. This second idea, in its purest form, implies a
predefined net being provided to the algorithm for learning,
such as in multilayer perceptrons.

Recently, some questions have been raised about the validity
of these classical ideas. The arguments against classical ideas
are simple and compelling. For example, it is a common fact that
humans do remember and recall information that is provided to
them as part of learning. And the task of learning is
considerably easier when one remembers relevant facts and
information than when one doesn=92t. Second, strict local learning
(e.g. back propagation type learning) is not a feasible idea for
any system, biological or otherwise. It implies predefining a
network "by the system" without having seen a single training
example and without having any knowledge at all of the
complexity of the problem. Again, there is no system that can
do that in a meaningful way. The other fallacy of the local
learning idea is that it acknowledges the existence of a
"master" system that provides the design so that autonomous
learners can learn.

Recent work has shown that much better learning algorithms, in
terms of computational properties (e.g. designing and training
a network in polynomial time complexity, etc.) can be developed
if we don=92t constrain them with the restrictions of classical
learning. It is, therefore, perhaps time to reexamine the ideas
of what we call "brain-like learning."

This panel will attempt to address some of the following
questions on classical connectionists learning:

1.  Should memory  be used for learning? Is memoryless learning
an unnecessary restriction on learning algorithms?
2.  Is local learning a sensible idea? Can better learning
algorithms be developed without this restriction?
3.  Who designs the network inside an autonomous learning system
such as the brain?

-------------------------

A SUBSEQUENT NOTE SENT TO THE PANELIST


The panel abstract was written to question the two pillars of
classical connectionist learning - memoryless learning and pure
local learning. With regards to memoryless learning, the basic
argument against it is that humans do store information
(remember facts/information) in order to learn. So memoryless
learning, as far I understand, cannot be justified by any
behavioral or biological observations/facts. That does not mean
that humans store any and all information provided to them. They
are definitely selective and parsimonious in the choice of
information/facts to collect and store.

We have been arguing that it is the "combination" of memoryless
learning and pure local learning that is not feasible for any
system, biological or otherwise. Pure local learning, in this
context, implies that the system somehow puts together a set of
"local learners" that start learning with each learning example
given to it (e.g. in back propagation) without having seen a
single training example before and without knowing anything
about the complexity of the problem. Such a system can be
demonstrated to do well in some cases, but would not work in
general.

Note that not all existing neural network algorithms are of
this pure local learning type. For example, if I understand
correctly, in constructive algorithms such as ART, RBF,
RCE/hypersphere and others,  a "decision" to create a new node
is made by a "global decision-maker" based on evidence on
performance of the existing system. So there is quite a bit of
global coordination and "decision-making" in those algorithms
beyond the simple "local learning".

Anyway, if we "accept" the idea that memory can indeed be used
for the purpose of learning (Paul Werbos indicated so in one of
his notes), the terms of the debate/discussion change
dramatically. We then open the door to the development of far
more robust and reliable learning algorithms with much nicer
properties than before. We can then start to develop algorithms
that are closer to "normal human learning processes". Normal
human learning includes processes such as (1) collection and
storage of information about a problem, (2) examination of the
information at hand to determine the complexity of the problem,
(3) development of trial solutions (nets)for the problem, (4)
testing of trial solutions (nets), (5)discarding such trial
solutions (nets) if they are not good enough, and (6) repetition
of these processes until an acceptable solution is found. And
these learning processes are implemented within the brain,
without doubt, using local computing mechanisms of different
types. But these learning processes cannot exist without
allowing for storage of information about the problem.

One of the "large" missing pieces in the neural network field
is the definition or characterization of an autonomous learning
system such as the brain. We have never defined the external
behavioral characteristics of our learning algorithms. We have
largely pursued algorithm development from an "internal
mechanisms" point of view (local learning, memoryless learning)
rather than from the point of view of "external behavior or
characteristics" of these resulting algorithms. Some of these
external characteristics of our learning algorithms might be:(1)
the capability to design the net on their own, (2) polynomial
time complexity of the algorithm in design and training of the
net, (3) generalization capability, and (4) learning from as
few examples as possible (quickness in learning).

It is perhaps time to define a set of desirable external
characteristics for our learning algorithms. We need to
define characteristics that are "independent of": (1) a
particular architecture, (2) the problem to be solved
(function approximation, classification, memory, etc.),
(3)local/global learning issues, and (4) issues of whether
to use memory or not to learn. We should rather argue about
these external properties than issues of global/local learning
and of memoryless learning.

With best regards,
Asim Roy
Arizona State University