The AI revolution has outgrown the Turing Test: Introducing a new framework

March 14, 2023 ndowd

Chris Saad is a startup and product builder, angel investor and strategic adviser. Previously, he was head of product for the Uber Developer Platform.

As AI becomes a transformative part of our technology landscape, a common vocabulary about the capabilities of each new tool and technique is essential. Common vocabularies create shared intellectual spaces allowing all stakeholders to accelerate understanding, increase adoption, facilitate collaboration, benchmark progress and drive innovation.

So far, the most widely known tool for benchmarking AI is the Turing Test.

AI Classifcation Framework - General Diagram

Image Credits: Chris Saad

However, the field of artificial intelligence (AI) has come a long way since the inception of the Turing Test in 1950. As such, it is becoming increasingly clear that the Turing Test is insufficient for evaluating the full range of AI capabilities that are emerging today — or are likely to emerge in the future.

The Turing Test operates on a simplistic pass/fail basis and focuses heavily on chat/linguistic capability, which is only one aspect of human intelligence. This narrow focus on language ignores many other critical dimensions of intelligence, such as problem-solving, creativity and social awareness. Additionally, the Turing Test presupposes a level of human-like intelligence that may not be relevant or useful for evaluating AI.

The framework

To address these limitations, there is an urgent and critical need to develop a more nuanced and comprehensive framework for evaluating AI capabilities across multiple dimensions of intelligence.

This insight led me to develop the “AI Classification Framework.” The ACF is a new approach to evaluating AI capabilities based on the Theory of Multiple Intelligences.

The Theory of Multiple Intelligences was first proposed by psychologist Howard Gardner in 1983. Gardner argued that intelligence was not a single, unified entity but rather a collection of different abilities that could manifest in a variety of ways. Gardner identified eight different types of intelligence: According to Gardner, individuals may excel in one or more of these areas, and each type of intelligence is independent of the others. The theory challenged the traditional view of intelligence as a singular, fixed entity and opened up new avenues for exploring the diversity of human cognition. While the theory of multiple intelligences has been subject to some criticism and debate over the years, it has had a significant impact on the field of psychology and education, particularly in the development of alternative approaches to teaching and learning.

This seemed perfect as a basis for the AI Classification Framework. Following the theory, the framework supports evaluating AI tools across multiple dimensions of intelligence, including linguistic, logical-mathematical, musical, spatial, bodily-kinesthetic, interpersonal and intrapersonal intelligence.

For each dimension of intelligence, the framework provides a scale from 1–5 with 1 being “No Capability” or the equivalent of a human infant and 5 being “Self-agency” or capability that might be considered “Super Intelligence” — beyond human ability.

Examples

The framework itself is a detailed table of descriptions that can be found here. I’ve also created a simple visual representation for easy high-level reference. What follows are two simple examples of the segments that respond to the capabilities of ChatGPT and DALL-E 2.