DeepMind's Newest Model Allows Pixel 11 Devices To Transcribe Sign Language Into Text

Key Points:

• Google has introduced DeepMind's new sign-language-to-text (SL2T) model with the launch of the Pixel 11 family, integrating it into Gboard and Live Transcribe to enable deaf and hard of hearing users to sign directly to their phones.
• The SL2T model currently supports American Sign Language (ASL) to English transcription, having been trained on over 100,000 hours of multilingual sign language data, with plans to expand to more languages and devices.
• Unlike previous systems, SL2T converts sign language video into geometric wireframes on-device to protect privacy, then translates these wireframes directly into text without relying on intermediate glosses, improving translation quality.
• DeepMind highlights the importance of supporting diverse communication modalities for deaf individuals and aims to bridge communication gaps between Deaf and hearing communities through this technology.
• Although initial support is limited to ASL, the model's multilingual training provides a foundation for future expansion to cover the 200+ sign languages used by over 70 million deaf and hard of hearing people worldwide.