Accueil - Portfolio d'Emmanuelle ABOAF

Transcription de la conférence Empowering Communication: The quest for automatic solutions in sign language

Transcript

Slide 1

Hello everyone,

Welcome to my talk "Empowering Communication: The quest for automatic solutions in sign language".

If you need a transcript, feel free to scan the barcode.
The video also contains captions. Feel free to activate the captions on this video.

Slide 2

My name is Emmanuelle, I'm French and I'm a developer at SHODO, a French company. I'm also Deaf with two cochlear implants.

I'm part of the CNCF Deaf and Hard of Hearing Working Group, which brings together Deaf web professionals from all over the world. I'm also part of 3 feminist associations in France: Duchess France, Yeeso and Compositech to represent women in tech and by extension disabled women. And finally, I'm proud to be part of the Women Techmakers programme to amplify my message about inclusion, disability and accessibility.

Slide 3

Before we talk about sign language, you need to know that around 466 million people worldwide are affected by disabling hearing loss according to World Health Organization (WHO).

This represents 5% of the world's population. WHO estimates that 700 million people will be affected by 2050, i.e. one person in 10.

Slide 4

We have 72 million Deaf and Hard of Hearing people around the world use sign language. Contrary to popular myth, not all Deaf and Hard of Hearing (DHH) people know sign language.

There are over 300 different sign languages.

Slide 5

This means that sign language is NOT universal. Each country has its own sign language. For example :

  • In the United States, we have American Sign Language.
  • In the UK, we have British Sign Language.
  • In France, we have French Sign Language.
  • In India, Indian Sign Language.
  • And so on, so on...

However, there is an international sign language that enables Deaf people all over the world to communicate with each other.

International Sign Language (ISL) is different from American Sign Language.

Seeing that sign language is not universal, we realise that creating an automatic tool in sign language will only be accessible to a local language and population.

Sign language varies considerably from one region to another, even within the same country, due to cultural and regional differences. Variations can include differences in vocabulary, grammar, intonation and body movements.

Slide 6

This makes research more complicated. When you create a tool in ASL, it won't be understandable by people who use French sign language or another sign language. If you want the tool to be available in another sign language, you also have to integrate the language. Let me remind you that signed words are different depending on the language. Just the "Hello" sign differs depending on the country.

For example. Here is "Hello" in ASL and "Bonjour" in French Sign Language.

Thanks to Sign Language Processing (SLP), which has been around for a few years now, many sign language projects are flourishing because the data is public. This data comes from American Sign Language (ASL). What hearing people don't know is that sign language is not universal. So, for example, I can see professionals in France using these data to create an application with a 3D avatar. But ASL is not the language used in France. As a result, it's completely pointless to create the app in France because the French Deaf population doesn't know ASL.

Slide 7

That's why it's so important to get to know the Deaf and Hard of Hearing community, Deaf culture and, above all, learn the local sign language with Deaf people.

If you don't know the language, how will you know that the language you are using is the right one? That the vocabulary is right? That the sign movements and facial and body expressions are right?

Hence the importance of involving Deaf people in the process. Ask for their opinion on your project and even involve them from start to finish.

Slide 8

Every year for years, students at engineering schools have been constantly trying to invent a new prototype designed to facilitate communication between Deaf people who speak sign language and hearing people.

Students in search of technological prowess think they are doing the right thing by creating these prototypes without the approval of the community and without even trying to understand how this community works.

The most obvious example is gloves.

The gloves are the perfect example of the "Disability Dongle", a term invented by Liz Jackson.

Slide 9

Liz Jackson created the term "Disability Dongle" in 2019 to draw attention to the phenomenon of design and engineering students and practitioners prototyping "innovative" solutions for disabled people. The definition satirises an outcome in which designs or technologies "for" disabled people attract mainstream attention and praise despite disabled people's legitimate concerns about them.

This term refers to a technological solution designed for disabled people but which, in reality, is often useless or poorly adapted to their real needs. Take gloves, for example. The gloves attracted the attention of the public and therefore potential funders, but did not meet with the approval of the Deaf community.

The gloves have therefore distracted from the real needs of the community, which is calling for greater accessibility with sign language. In fact, access to education, healthcare, administrative procedures, the media and so on using sign language is still very difficult for Deaf people whose mother tongue is sign language. To meet this need, more sign language interpreters are needed, and they need to be trained. Even today, there are too few interpreters to meet all the needs.

What's more, do you see people walking around with these gloves that obstruct their movements, especially as they are heavy and huge?

Slide 10

There is also the question of responsibility. If the machine translation is bad, the person receiving the message will misinterpret it.

A sign can mean several words and this sign is defined according to the context.

Watch Jason's video showing the same sign meaning several words.

If you don't know sign language at all, this can have consequences of more or less impact.

That's why you need to work with the people concerned to ensure that the tool is reliable.

Slide 11

I'm going to make a bit of a connection with automatic captions. Today, automatic captions are not 100% reliable. There can be a lot of errors depending on the environment, the context and the person speaking.

In the USA, the quality is better than in France. Being French, when I see that there are a lot of errors in the automatic captions, I have to do some mental substitution to fill in the gaps, especially during live broadcasts. When the AI makes a mistake, I have to make up for it by reading lips or hearing the missing word if I've understood it correctly. It's very exhausting.

So imagine with automatic sign language? That's why we need real Sign Language interpreters to help the community in their daily lives.

Automated solutions are not intended to replace them, but they can help in areas where interpreters cannot intervene. Digital content, for example. Having digital content or audio announcements in public transport translated by a 3D avatar can be an effective solution.

Slide 12

The problem with 3D avatars is their quality. At present, it is difficult to reproduce facial and body expressions. Two things that are very important in sign language.

Just signing is not enough. You also need expressions to transmit the tone of the message and the context.

3D avatars work very well, but more often than not they look like robots and are devoid of human expression. That's the challenge today. I have no doubt that in a few years' time, avatars will be more fluid.

Slide 13

Watch a great video from KEIA, a French compagny, to raise awareness of AI and sign language.

Thanks to Object Detection API, it's easy to capture the sign and recognise it. However, the program only manages to capture sign by sign, and slowly at that. But sign language isn't about signing sign by sign, it's a whole language that's constructed and alive through facial and bodily expressions. You can sign at high speed, but the AI will have trouble capturing the signs because they go so fast that it's hard to recognise them.

Slide 14

I'm going to take the example of a French company because that's what I know best. Being French, I follow technological developments in France. IVèS, a French company, has created its first SignBot. A conversational chat in French sign language.

From my own experience with SignBot, during a demonstration, I signed very quickly and the AI didn't capture what I said. I had to sign very slowly for it to understand me.

It's very promising. The advantage of this SignBot lies in the collaboration with Deaf signers. And it works! But there's still a lot of work to do.

The American company "Sign Speak" is also building technology for Deaf people with them.

Why is it so important to work with Deaf people?

Did you know that, to enrich the avatar's vocabulary, you need to have the sign in motion capture so that the AI can learn it? Unlike automatic captionning, where you just need to provide a verbatim, the process is more complex and slower.

As we know, language evolves. Every day, new signs appear in the sign language repertoire. Each country sees its local sign language grow. So the database has to be constantly updated.

Slide 15

That's why it's so important to work with the Deaf and Hard of Hearing community.

Nothing about us without us. It's a movement, a slogan created by the disabled community in the United States. Nothing about us without us shows the importance of involving disabled people in product design.

You can collaborate with Deaf and Hard of Hearing people in tech to create a tool with AI Sign Language. For exemple, there is the CNCF Deaf and Hard of Hearing Working Group, which brings together Deaf and Hard of Hearing people in tech all over the world. You can contact them.

If you're looking for someone local, you can be sure to find them via this group or contact a local Deaf association.

In conclusion, if you want to create a tool that involves sign language with AI in any country, get the community involved.

Slide 16

Thank you very much for listening to me.