Is that AI doing what you think it’s doing?
I can remember my first experience using image search on my photo library, a long time ago. I noticed that a new tool had appeared in the UI, prompting me to search for things that might appear in my photos. It seemed to work like magic: when I searched for castles, it found castles (I have visited a lot of castles). When I searched for seascapes it found seascapes (I live near the sea).
Years later, image search doesn’t seem like magic any more: it’s just another everyday miracle that has become mundane. Because it has become mundane, it is easy to fall into the trap of believing that we understand how it works: that it understands and interprets the world in the same way that we do.
Last week I experienced a great corrective to this, when reading the great book, You Look Like a Thing and I Love You by Janelle Shane, who also writes the blog AI Weirdness. (Go read both: you’ll find out much more than I can share in this article.) Of the many insights and ideas in the book, two gave me pause for thought.
The first was the observation that AI image recognition has a tendency to find giraffes in pictures that don’t contain giraffes. Why should this be? Are algorithms cunningly trying to avoid being fooled by the giraffe’s camouflage? Are there giraffes all around us in disguise?
The answer is much more simple and revealing: image recognition algorithms are trained on large sets of photos taken by humans. Most things in the world are boring. But we don’t take photos of those things: we take photos of interesting things, like giraffes. So, an image recognition algorithm hasn’t been trained to understand the world as it is, but a highly selective representation of the world. One that has a lot of giraffes in it.
This example reinforced to me that, whatever language we use, AI algorithms do not learn in exactly the way that we do. We can’t tell an algorithm about the world and we can’t explain about the relative scarcity of giraffe sightings: all we can do is show more examples (and even if we manipulate the data set, we are not explaining). When working with AI, we need a different mental model to when we work with our fellow humans, who are capable of reasoning and argument.
The second idea comes later in the book, in a more sobering chapter, where Shane describes the practice of using adversarial learning to figure out which parts of an image an AI pays most attention, and how to grab that attention in unexpected ways. This produces some surprising and counter-intuitive results: inserting a small patch of what looks like random pixels into a photo, makes no material difference to the human perception of the image, but completely changes what the AI reports, from a submarine to a bonnet, or from a brown bear to a tree frog.
Again, this reinforced to me that, even though we talk of AI’s perceiving, seeing and understanding, they are doing something different to what we mean when we use these words in everyday language. And, not only can their ‘perception’ be scrambled, we may not even discover this potential for scrambling until we have put the AI to work in real use cases.
Shane does a much better job of explaining and exploring these topics than me (again, go read the book and the blog), but I wanted to share these lessons as a reminder to those of us working in technology of our duty to understand and our duty to explain. Understanding how AI actually works is essential to using it in responsible ways: discovering that you’ve accidentally invented an over-sensitive giraffe detector seems amusing; discovering that you’ve accidentally invented an algorithm that sees fraud or threats in populations where they don’t exist, or fails to see need where it does exist, is rather more disturbing and dangerous. One of the promises to act in BCG’s AI Code of Conduct concerns transparency, and transparency means more than just demonstrating the operations of a particular algorithm: it means understanding and communicating the nature of AI as a category, and the ways it differs from human reasoning.
Even though, as Janelle Shane shows, AI is often more dumb, narrow and lazy than we think it is, it is still astonishingly powerful: the history of computing is the history of putting small things together to achieve large things. But we’ll need to make sure that we truly understand the small things to get the large things that we want.
