eVoice: AI-Powered Assistive Communication via FSL & Silent Lip Reading

Problem Statement

Over 4 million Filipinos live with hearing, speech, or language impairments. While the Filipino Sign Language Act (RA 11106) recognizes FSL as the national sign language and the Magna Carta for Disabled Persons (RA 7277) mandates equal service access, day-to-day communication remains broken. Certified FSL interpreters are scarce (fewer than 1,000 nationwide) and largely concentrated in Metro Manila.

Furthermore, thousands of non-verbal Filipinos—such as stroke survivors, aphonia patients, trauma victims, and individuals with vocal cord paralysis—can mouth words silently but cannot produce audible sound, leaving them isolated in hospitals, local health centers (RHUs), schools, and everyday public transactions. Dedicated hardware kiosks or eye-tracking boards cost tens of thousands of pesos, putting them far out of reach for ordinary Filipino families and public facilities.

Proposed Solution

eVoice (“Your voice, without the sound”) turns any standard Android smartphone into an accessible, bidirectional assistive communication device without requiring external sensors, specialized gloves, or expensive kiosks.

The application unifies five multimodal features:

1. FSL-to-Text: The front- or rear-facing smartphone camera tracks hand gestures, finger articulation, and upper-body poses in real time. A lightweight sequence classification model maps recognized sign sequences into readable Filipino/English text.
2. Lip Reading-to-Text (Visual Speech Recognition): Analyzes silent mouth, lip, and jaw movements using MediaPipe Face Mesh contours. A compact spatio-temporal model predicts intended phonemes/visemes and turns silently mouthed words into text.
3. Text-to-Speech (TTS): Instantly synthesizes confirmed text into clear spoken audio in Tagalog or English, allowing users to communicate naturally with people who cannot read FSL or lip movements.
4. Text-to-FSL: Translates incoming written or spoken messages back into FSL through an indexed library of verified video recordings and sign animations, respecting FSL spatial grammar and glossing rather than verbatim word-by-word substitution.
5. Unified Communication Mode: Provides an intuitive toggle between signing, silent mouthing, quick phrase buttons, and synthetic speech so users can adapt to varying lighting conditions, fatigue, or social environments.

Target Market

• Primary Citizens: Deaf and hard-of-hearing Filipinos who use FSL; individuals with vocal cord damage, post-stroke speech loss, or aphonia; and neurodivergent non-verbal communicators.
• Frontline Service & Public Health: Rural Health Units (RHUs), barangay health workers (BHWs), hospital triage desks, emergency triage responders, and government social welfare desks (DSWD/PDAO).
• Inclusive Classrooms & Families: DepEd Special Education (SPED) teachers, hearing parents of Deaf children, and inclusive workplace managers.

How AI Is Used

• On-Device Landmark Tracking (MediaPipe Hands & Face Mesh): Extracts 468 3D facial landmarks (focusing on lip perimeter coordinates 61, 291, 0, 17) and 21 3D hand keypoints at 30 FPS directly on the smartphone GPU/NPU.
• Visual Speech Recognition (Viseme Classifier): A quantized temporal convolutional network (TCN) or GRU on normalized mouth contour geometry that maps silent visual phonemes into high-probability words.
• FSL Sequence Recognition (LSTM / Mobile Transformer): Evaluates temporal hand-pose transitions and non-manual facial markers to predict discrete FSL glosses.
• FSL Grammar & Syntax Adaptation: Lightweight on-device rule engine and lookup model that maps linear Tagalog/English sentences into natural FSL gloss sequences and vice versa.
• Android Text-to-Speech: Low-latency native TTS engine delivering natural spoken voice output in Tagalog and English without external API calls.

PLDT / Smart / ePLDT / Rokid Angle

• Smart Communications: Zero-rated mobile data synchronization for community-curated FSL dialect video packs and emergency medical gloss libraries.
• Rokid AR Glasses: Pairs with Rokid smart glasses to display silent lip-reading captions and FSL transcriptions directly inside the user’s heads-up display during doctor consultations or counter transactions.
• ePLDT Vitro: Secure Philippine cloud hosting for anonymized model training pipelines, federated dictionary updates, and open-source FSL video repository contributions.
• PLDT Enterprise: Accessible CSR bundling for LGU Persons with Disability Affairs Offices (PDAO), DepEd inclusive learning resource centers, and partner hospital networks.

Demo Scenario (3 minutes for judges)

1. The Silent Clinic Triage: A volunteer acts as a non-verbal patient arriving at an RHU triage desk.
2. Silent Lip Reading in Action: Facing the smartphone camera, the user silently mouths “Masakit ang dibdib ko” (no audible sound). eVoice tracks lip contours, outputs the text in real-time, and speaks it aloud through the phone’s speaker.
3. Bidirectional FSL Signing: The user switches to FSL mode and signs “May allergy ako sa gamot”. eVoice’s camera captures hand coordinates and displays the translated text for the triage nurse.
4. Reverse Response (Text-to-FSL): The nurse speaks into the phone: “Maupo ka muna dito”. eVoice converts the speech to text and displays a verified FSL animation clip of the phrase.
5. Closing Statement: “eVoice gives voice without sound—making accessibility as ubiquitous as a smartphone.”

30-Hour Build Plan

Suggested Stack: Flutter / Android (Dart & Kotlin), Google MediaPipe, TensorFlow Lite / ONNX Runtime, Android TTS Engine, Local SQLite. Public Data / Datasets: Open FSL educational datasets (Deaf Resource Center / KWF public resources), consent-based self-recorded clips, LRW (Lip Reading in the Wild) open benchmark subsets.

Risks & Privacy Notes

• Strict On-Device Privacy (RA 10173 Compliance): All camera frames, facial geometry, and hand coordinates are processed in volatile memory on the phone and discarded immediately. No camera feeds or biometric recordings are transmitted or stored remotely.
• Homophene & Lighting Limitations: Many words produce identical mouth movements (“pat”, “bat”, “mat”). The system uses language context and a bounded emergency/service vocabulary to mitigate mispredictions under varied lighting.
• Cultural & Community Respect: FSL is a complete language with its own grammar, not signed English/Tagalog. eVoice is designed as an assistive communication aid co-designed with Deaf stakeholders, not a full replacement for human certified interpreters.

AI Use Disclosure (draft)

eVoice utilizes Google MediaPipe for real-time facial and hand landmark detection, a custom quantized TensorFlow Lite model for temporal viseme and sign sequence classification, and the native Android TTS engine for speech synthesis. Generative AI tools assisted in prototyping UI layouts and dictionary structuring. All model evaluations in the demo rely on transparent, consent-backed video calibration.

Rubric Scoring