Computers Learn To Identify Signs In TV Footage
Three PhD candidates in the Netherlands have spent the last few years developing technology to recognise sign language in real-time.
One researcher, Jeroen Arendsen, said, “Very little is known about the perception of sign language. The aim of this research was to expand our knowledge of human observation of signing. For instance, it turns out that sign language users only need to see a small part of a hand movement to know it is a sign and what it means.”
This means technology can be developed to recognise signs. With a Dutch foundation for children who are deaf or hard of hearing, the researchers developed a computer to do just this. The signs are recognised in real-time and can be used to teach children new signs more quickly.
Children aged three to six years old were shown a picture and asked to make the sign. The computer then had to assess the sign. Arendsen says, “And that is quite difficult, because a computer is more easily confused than people by irrelevant fidgeting.”
Fellow PhD candidate Jeroen Lichtenauer then converted the signs into information that the computer could read. Gineke Ten Hold, the third PhD candidate, worked as a bridge for these two processes.
Compared to a control group, the children who practiced with the computer improved their vocabulary.
This is not the first time computers have been used to recognise sign language. Earlier this year, software created by researchers from Oxford andLeeds Universities in the UK worked out a way for computers to learn sign language by absorbing TV shows with both subtitles and signs.
The software uses the arms to work out the rough location of the hands and identifies flesh colouring to reveal exact hand shapes. Once the researchers were satisfied, the computer was exposed to over 10 hours of TV footage.
In this time, the computer learnt over 200 nouns and adjectives that were used multiple times in the footage. During the initial project the computer achieved 65% accuracy, which has since been improved with more practice and exposure.
The success of this project means that in time the computer could produce digital avatars to fluently sign alongside any TV programme.
More information:
(compiled by Miriam Walsh)