Kamay: Filipino Sign Language and voice access to every government counter

Problem Statement

The Magna Carta for Disabled Persons (RA 7277) and the Filipino Sign Language Act (RA 11106, 2018) require government offices to be accessible and recognize FSL as the national sign language of the Filipino Deaf. In practice, most LGU offices, hospitals, and agency branches have no FSL interpreter on site. Deaf citizens end up writing notes, bringing relatives, or giving up. Certified interpreters are few and concentrated in Metro Manila. Government kiosks and online portals are also mostly text-heavy and touch-only, which shuts out blind and low-vision users and seniors. Persons with disabilities number in the millions in the Philippines, and many lose access to services meant for them, including their own PWD ID and benefits.

Proposed Solution

Kamay is a tablet, kiosk, and web app with three modes:

1. FSL → text/speech. A camera tracks hand, face, and body landmarks. A sign-recognition model turns signs into glosses, and an LLM rewrites the glosses into proper Filipino or English sentences that are shown and spoken to the clerk.
2. Speech → captions + FSL. The clerk speaks in Filipino or English. Speech-to-text shows live captions, and an avatar signs key phrases using a library of FSL clips and simple LLM-generated gloss sequences.
3. Voice navigation. Blind or low-literacy users can ask for things by voice (“Gusto kong mag-renew ng PWD ID”). Kamay guides them through kiosk menus and forms, reads fields aloud, and fills them in.

The MVP focuses on a limited vocabulary of about 100–200 high-value service signs (PWD ID, cedula, birth certificate, payment, schedule, numbers, yes/no). Anything it doesn’t recognize falls back to typed chat or a remote interpreter. Rokid AR glasses can show live captions for a Deaf user while the clerk speaks naturally.

Target Market

• Primary users: Deaf and hard-of-hearing Filipinos, blind and low-vision citizens, seniors, and PWD ID holders.
• B2G buyers: LGU Persons with Disability Affairs Offices (PDAO), city halls, PhilHealth, SSS, GSIS, LTO, PSA, DFA, public hospitals, and NCDA. Coverage could reach about 1,700 LGUs and thousands of agency branches.
• B2B: Banks, telcos, malls, and hospitals that need accessible customer service. This also helps them meet accessibility and corporate social responsibility goals.

How AI Is Used

• Pose and landmark estimation (MediaPipe Holistic): Real-time hand, face, and body keypoints on a regular camera. Without it, sign input would need special gloves or sensors.
• Sign recognition model (LSTM/Transformer on keypoints): Classifies FSL signs. This is the core capability, and there is no non-AI substitute.
• LLM gloss-to-sentence and sentence-to-gloss: FSL grammar is different from Filipino and English. The LLM turns sign sequences into natural sentences and simplifies clerk speech into glossable phrases.
• Speech-to-text and text-to-speech (Whisper and Filipino TTS): Enables live captions and spoken output.
• Voice agent: Understands intent (“magpa-renew”) and controls the kiosk flow. Without it, blind users can’t navigate visual menus.

PLDT / Smart / ePLDT / Rokid Angle

• Rokid: A strong fit. AR glasses show live captions or the clerk’s translated speech in the Deaf user’s field of view, or let a clerk see FSL translations hands-free. This could be a showcase use case for Rokid in the Philippines.
• PLDT Enterprise: Bundle Kamay with fiber and managed kiosks for LGUs and agency branches. Offer a remote FSL interpreter fallback through PLDT video connectivity.
• ePLDT: Low-latency inference and model hosting in Philippine data centers, plus a “Kamay Cloud” API for other government apps.
• Smart: 5G enables mobile Kamay units for barangay outreach and disaster evacuation centers.

Demo Scenario (3 minutes for judges)

1. Introduce a Deaf team member or a consenting Deaf volunteer who wants to renew a PWD ID. Normally they would stand at the counter with a pen and paper.
2. They sign “PWD ID – renew – how much?” to the tablet camera. The screen shows the recognized glosses, then the sentence “Gusto ko pong i-renew ang PWD ID ko. Magkano po?” which is spoken aloud.
3. A teammate acting as the clerk replies verbally. The Deaf user sees live captions on Rokid glasses plus a short avatar sign clip: “Libre po. Kailangan ng 1x1 picture at medical certificate.”
4. Switch to voice navigation: a blindfolded judge volunteer says “Kumuha ng cedula” and Kamay walks them through the kiosk form by voice.
5. Close with: “RA 11106 made FSL a national language. Kamay makes it a working language at every counter.”

30-Hour Build Plan

Suggested stack: Python, MediaPipe Holistic, PyTorch/TensorFlow (LSTM or small Transformer), OpenAI Whisper, Gemini or other LLM, Coqui/Google TTS (Filipino), React web app, Rokid SDK or Android display. Public data / APIs: FSL video datasets on Hugging Face / Kaggle (e.g., community FSL alphabet and word sets), the Philippine Deaf Resource Center and the Komisyon sa Wikang Filipino’s FSL materials (licensing to be checked), self-recorded clips with consent, MediaPipe models, NCDA accessibility guidelines.

Risks & Privacy Notes

• Accuracy and vocabulary limits: FSL has regional variants and depends on facial grammar. Keep the vocabulary limited, show confidence, always offer typed or remote-interpreter fallback, and co-design with Deaf organizations.
• Cultural respect: Deaf community members are part of the design and testing. Kamay supports human interpreters and does not replace them.
• Data Privacy Act 2012: Video of people signing is personal data, and disability status is sensitive personal information. Process on-device where possible, don’t store video by default, and get explicit consent for any training recordings.
• Clause 24 / competition rules: Use only consenting volunteers’ recordings and openly licensed datasets. No real client data goes into AI tools.
• Latency: Run keypoint extraction on the edge device and send only keypoints, not video, to the cloud.

AI Use Disclosure (draft)

Kamay uses MediaPipe for pose landmarks, a team-trained LSTM/Transformer for sign classification, OpenAI Whisper for speech-to-text, a Filipino TTS model, and an LLM (e.g., Google Gemini) for gloss-to-sentence translation and voice-navigation intent. AI coding assistants helped with UI scaffolding. Training clips were recorded by consenting volunteers. Sign vocabulary and translations were checked by Deaf and FSL-fluent reviewers, and every demo output was checked by a human.

Rubric Scoring