🚀 go-pugleaf

In article <3018@crete.cs.glasgow.ac.uk> gilbert@cs.glasgow.ac.uk (Gilbert Cockton) writes:

>Just what sort of Science did young Mr. Turing have in mind when he
>decided that subjective opinion could ever be a measure of system
>performance?

Young Mr Turing suggested in his 1950 paper "Computing Machinery and
Intelligence" that it would be 50 years before what we now call "the
Turing Test" and he called "the imitation game" could be applied. I see
no reason to revise that estimate downwards.  He also made it clear that
he considered that everyday notions of "machinery" and "thinking" then
carried too much excess semantic baggage for reasonable discussion of
the question "can machines think" to be profitable, and consequently
suggested the "imitation game" as a Gedanken experiment with which to
clear the philosophical air.

While there have been a few occasions in AI research when people were
asked to play a Turing-like game to assess a program whose specific
purpose was to articulate a model of a certain kind of of human
behaviour, such as Colby's "PARRY", or the ranking of MYCIN's diagnostic
responses compared to a panel of experts, it remains true that the
primary purpose of the Turing Test in AI is as a gedanken experiment.
As Turing pointed out in the paper in question:

  "The popular view that scientists proceed inexorably from
  well-established fact to well-established fact, never being
  influenced by any improved conjecture, is quite mistaken.
  Provided it is made clear which are proved facts and which are
  conjectures, no harm can result. Conjectures are of great
  importance since they suggest useful lines of research."

The Turing Test was suggested by its originator as a source of useful
conjecture. It still serves that purpose, and that tradition in AI is
continued by Searle's Chinese Room argument, and Harnad's Total Turing
Test, to name two examples recently ventilated on comp.ai.

>How do AI types *REALLY* test their systems?

This question seems to suggest that AI types _pretend_ to be using
something like the Turing Test, but actually in the privacy of their
labs are up to something quite different. I thought, Gilbert, that you
had once in your career suffered some education in AI? The answer to how
we AI types test our systems is as various as the kinds of system we
build, and the reasons we build them. In many cases, as I am sure you
know, what is interesting about our systems is why they don't work :-)
That is not the kind of thing which is established by testing a system.

-- 
Chris Malcolm    cam@uk.ac.ed.edai   031 667 1011 x2550
Department of Artificial Intelligence, Edinburgh University
5 Forrest Hill, Edinburgh, EH1 2QL, UK