From flying an airplane to forecasting the weather: people use computers for very different things. But is a computer able to see and perhaps even judge whether something is beautiful or ugly? Informatician Jan-Mark Geusebroek of the Intelligent Systems Lab Amsterdam (ISLA), Informatics Institute, investigates how to he can teach a computer to look at things like people do.

Geusebroek works on various lines of research within informatics. At present he is working on very compact representation of images, whereby only the most important characteristics remain. ‘I'm looking for those features which describe an object', explains Geusebroek. ‘The computer needs to learn these features in order to achieve a level of understanding of things.' To give insight into his research, Geusebroek is teaching a robot dog to recognize objects, whereby a worldwide network of computers forms the ‘brain' of the dog.

View things as humans do

Geusebroek is of the opinion that a lot can be learned from how people look at things. ‘A human is born with a brain, an extremely complicated structure which has evolved over millions of years. I look at the world outside and assume that humans are adapted to it. Suppose you view your head as a large calculator, what do you need to calculate in order to think something about the image you are looking at? This is how I try to make my computer view things, and so I steal some parts of human perception, such as three colour channels. More channels would probably work better, but that would cost more pixel space; more colour is, thus, less resolution. Fewer colour channels wouldn't be a good choice, since you would be colour blind. Humans are also very sensitive to movement; we notice movements in both the intensity difference as well as the colour contrast, so we can see moving edges. I then use these facts in the computer algorithmics.'

It remains to be seen whether the computer will ever be able to view things as humans do. ‘In twenty years computers will probably have the same computing power as the human brain. However, the algorithmics are still missing. But we are making progress. Thanks to new fMRI techniques, neurobiologists and psychologists know approximately which area in the brain plays a role at a particular moment. However, we have no idea what is being calculated. I sometimes compare neurologists' work to examining a pentium-4 processor under the microscope, and then trying to figure out how it works. It is super-complex.'

Geusebroek is looking for the direct effect of physics on statistics. ‘When the suns sets, how does the contrast in grass change? Can an algorithm describe exactly how the structures change as a result of the setting sun?'  Thanks to his research, Geusebroek has developed another manner of observation. ‘I have learned to look at things more carefully, I look for a structure in a photograph. I probably study a piece of tree-bark longer than other people.' Geusebroek's main aim, teaching the computer to observe things as a human being, stems in the first instance from curiosity. He admits laughingly that ‘it would be useful if my computer really was able to see things. I'd sit in the back of the car, and let the computer drive.'

Erotic pictures

Geusebroek is now supervisor of three PhD students. In collaboration with researchers from Groningen, one of them is investigating how the eye moves when looking at a picture. ‘When you show somebody a picture of a person, many people look at the face first. But it takes about 100 milliseconds before you even see a face at all. What do you see in those first milliseconds? And does this depend on the content of the picture?'

Geusebroek also collaborates with the University of Bonn. ‘We show people pictures with an emotional content, for example pitiful, violent or erotic pictures. The fMRI-scanner shows which areas of the brain are activated. We then examine to what extent the content of the image determines the emotions. Do we still see emotions if, for example, we standardise the colour setting?' In collaboration with the department of Psychophysics in Utrecht, Geusebroek also investigates the sensitivity of the eye for distinguishing colours and structures. ‘How does a human see a tree? You don't absorb each leaf separately, yet you get the impression ‘tree'. How does this impression come about?

Ultimately, the informatician hopes to be able to forecast what people look at in a picture. When this is known, this information can be used to teach a computer to look at things better. ‘We hope that we will also be able to teach the system to make subjective judgements, so that the robot dog doesn't just learn to recognise things, but also to distinguish between what is beautiful and what isn't'