Summary on The Winograd Schema Challenge and Common Sense, The Turing Test, and The Quest for Real AI by Hector J. Levesqueand et. al.
Introduction
In this post I have tried to point out the key ideas presented in the amzing paper The Winograd Schema Challenge by Hector J. Levesqueand, Ernest Davis and Leora Morgenstern and the book Common Sense, The Turing Test, and The Quest for Real AI by Hector J. Levesqueand. As the book serves a very detail background knowledge (!) for the mentioned paper, I find it more suitable to put together the summary as one. Anyone looking for a quick intro on effective evaluation methods for AI and a possible AI system itself would be an ideal audience.
This is part of a paper review assignment of the course CSE6369.
Thank you for taking the time to read this article! ❤
My Summary
The rules of arithmetic allow us to deal with abstract numbers in terms of concrete symbols. The manipulation of those symbols mirros the realations among the numbers being represented.
The rules of some sort of logic allow us to deal with abstract ideas in terms of concrete symbols. The manipulation of these symbols mirros the relations among the ideas being represented.
The paper of interest presents an improved alternative to Turing’s Imitation Game namely as “The Winograd Challenge” and the book lays the background to with paper and then some with interestinf discussions on the difference between the current AI and the Good-Old-Fashioned-AI (GOFAI), how to dissect a GOFAI, ways to test a GOFAI system, possible features of a GOFAI system etc. The summary presented here tries to follow the order that the book itself has followed.
What is GOFAI
The author categorizes the conventional machine learning algorithms as Adaptive Machine Learning (AML). These are usually systems that learn some intelligent behaviour by training on massive amount of data. However, there is a different category of AI that is built upon common sense, not learning from massive amounts of data. Common sense brings the key ability of processing situations that are sufficiently unlike the patterns seen before into the play. About this McCarthy has said that:
We shall therefore say that a program has common sense if it automatically deduces for itself a sufficiently wide class of immediate consequences of anything it is told and what it already knows.
How to Approach GOFAI
One single viewpoint such as from psychologoy and neuro-science would not be enough to solve the mystery behind a human brain. It can discover occational phenomena and patterns to discover a few details, but understanding the whole concept would be a whole new ball-game. So, as possible solutions, we can either study the objects that produce the phenomenon (i.e. human brain) or we can attemp to study the phenomenon itself more directly. An example would be to study the aerodynamics behind flying and observing birds to fly.
Hence, instead of getting caught up in the details of how human brain does its job, we need to shift our attention to the job itselt, and wonder how it might get done at all. This way, we might get to see what is and what is not essential in producing the phenomenon itself.
Inter-relation between Knowledge and Behaviour
How does our thinkings come into existaence and how do they progress? Psychologist SIr Frederic Bartlett says:
The important characteristics of the thinking process, as I am proposing to treat it, now can be stated: The process begins, when evidence or information is available which is treated as processing gaps, as being incomplete. The gaps are then filled up, or that part of the information which is incomplete is completed. This is done by an extention or suplemenation of the evidence, which remains in accord with the evidence (or claims to do so), but carries it further by utilizing other sources of information besides those which started the who;e process going, and, in many instances, in addition to those which can be direcrtly identified in the external surroundings. (Thinking, 1958, p. 75)
It is these other sources of information that is also known as the background knowledge is the key.
What is Winograd Challenge and why
Previously proposed tests like Turing’s Imitiation Game and Searle’s Chinese Room argument has drawbacks when it comes to test the existence of intelligence within an agent/subject. Like the turning test allows a natural, freeflowing conversation between the interogator and the examinee and the coverage of such a test is simply too huge. Also it fails to reproduce the result effectively. The Chinese room argument does not hold in a number of ground, like, the rule book would be so large that it can never exist, or if it did, the agent could memorize it to learn the language Chinese etc.
Winograd Schema Challenge is proposed here as an alternative, is a small reading comprehension test involving a single binary question. In each of the questions, we have the following four features:
- Two parties are mentioned in a sentence by noun phrases. They can be two males, two females, two inanimate objects or two groups of people or objects.
- A pronoun or possessive adjective is used in the sentence in reference to one of the parties, but is also of the right sort for the second party. In the case of males, it is “he/him/his”; for females, it is “she/her/her” for inanimate object it is “it/it/its,” and for groups it is “they/them/their.”
- The question involves determining the referent of the pronoun or possessive adjective. Answer 0 is always the first party mentioned in the sentence (but repeated from the sentence for clarity), and Answer 1 is the second party.
- There is a word (called the special word) that appears in the sentence and possibly the question. When it is replaced by another word (called the alternate word), everything still makes perfect sense, but the answer changes.
Example: The town councillors refused to give the angry demonstrators a permit because they feared violence. Who feared violence?
Answer 0: the town councillors, Answer 1: the angry demonstrators
Ending Thoughts
Surprisingly model like GPT-3 has already solved winograd challenge with human level accuracy or may be even more. Does that mean the quest for AI is done? May be not. Concepts like “Embodied AI” or “Embodied Control” requires a two-way connection between the body and the mind. But, the current models mostly provide a learning system or brain, we are far away from the embodied AI. Parhaps, that would be the next big step.