KumpuniAR: Hands-free Rokid AR and voice copilot for barangay utility repair and rural health triage

Problem Statement

Most AI assistants assume the user is sitting at a desk with free hands to type prompts and read paragraphs on a screen. Frontline workers who keep Philippine municipalities running cannot work that way. When a typhoon knocks out a barangay water chlorinator, solar micro-grid inverter, or fiber distribution box, or when a Rural Health Unit midwife manages a high-risk delivery hours away from a provincial hospital, both of their hands are occupied with tools or patients. Flipping through PDF manuals on a wet smartphone screen causes costly mistakes, equipment damage, and delayed emergency care, while certified senior engineers and medical specialists are concentrated in Metro Manila, Cebu, and Davao.

Proposed Solution

KumpuniAR shifts human-AI interaction from text chat to zero-UI, egocentric spatial guidance on Rokid AR smart glasses, with a hands-free phone-stand mode for barangays without glasses. The forward-facing camera on the Rokid glasses captures what the worker’s hands are doing from a first-person perspective.

A fast instance-segmentation model tracks physical components such as terminal blocks, fiber splice trays, breaker switches, or medical triage instruments alongside the worker’s hand positions. Simultaneously, a multimodal vision-language model grounded in verified TESDA, PLDT Enterprise, and DOH procedural checklists watches each step. If the technician reaches for the wrong terminal or skips a safety lock, KumpuniAR speaks an immediate warning in Tagalog or Bisaya and highlights the correct part with a directional indicator on the Rokid optical display. When a case requires human sign-off, a remote specialist in Manila views the live first-person feed over Smart 5G and drops spatial pins that lock onto the physical object in the wearer’s view.

Target Market

• Primary users: PLDT and Smart field installation and restoration technicians, rural electric cooperative linemen across 121 electric co-ops, LGU water and disaster equipment technicians, and Rural Health Unit midwives and nurses.
• B2B and B2G buyers: PLDT Enterprise field operations, TESDA regional training centers, DOH Telemedicine and Provincial Health Offices, and municipal engineering offices.
• Hardware ecosystem partner: Rokid Philippines as the flagship enterprise wearable platform bundled with Smart 5G Enterprise SIM plans.

How AI Is Used

• Egocentric object and hand-state tracking: YOLOv11-seg and MediaPipe Hands detect tools, equipment parts, and hand-object contact in the first-person camera stream at 25 frames per second.
• Proactive mistake detection and intervention: Following the HoloAssist paradigm, a multimodal model monitors task progress against a structured state graph and speaks up proactively when the user deviates from the safe sequence, rather than waiting for the user to ask a question.
• Multimodal procedural RAG: Retrieves the exact diagram and torque, wiring, or dosage specification from ePLDT-hosted DOH, TESDA, and PLDT technical manuals based on the component visible in the camera frame.
• Tagalog and Bisaya voice interface: Hands-free speech recognition and voice synthesis allow the worker to say “Susunod na hakbang” or “Tama ba itong kabit ko?” over generator or rain noise.

PLDT / Smart / ePLDT / Rokid Angle

• Rokid Philippines: Directly showcases Rokid AR glasses as an essential field productivity and life-saving device rather than a novelty screen, creating an immediate B2B and B2G hardware-plus-software bundle.
• PLDT Enterprise and Smart 5G: Requires high-uplink, low-latency Smart 5G and LTE connectivity to stream annotated egocentric video between field workers and central command centers.
• ePLDT: Hosts the sovereign enterprise manual repository, vision-language inference servers, and automated video audit logs for regulatory compliance and TESDA skills certification.
• De La Salle University: Aligns with DLSU computer vision and human-computer interaction research on wearable assistive systems and industrial maintenance.

Demo Scenario (3 minutes for judges)

1. Place a physical fiber distribution box or breaker panel and a simulated Rural Health Unit emergency kit on the demo table. A teammate puts on the Rokid AR glasses, and the glasses’ live first-person view is mirrored onto the big screen for the judges.
2. Task 1, Utility Repair: The wearer says, “KumpuniAR, i-reset at i-reconnect ang barangay comms box.” The HUD highlights the main breaker first. The wearer intentionally reaches for the live secondary wire without flipping the breaker.
3. KumpuniAR immediately interrupts in Tagalog: “Hinto. Buhay pa ang linya. Patayin muna ang pulang breaker sa kaliwa.” A red warning frame pulses over the breaker until the wearer flips it.
4. Task 2, Remote Specialist Pin: A judge clicks on a specific port on the web console laptop. That exact port lights up with a persistent marker inside the wearer’s Rokid view as the wearer moves their head.
5. Automated Audit Log: Once the 4-step repair finishes, KumpuniAR generates a timestamped photo-verified completion report on ePLDT in 3 seconds.

30-Hour Build Plan

Suggested stack: Android Kotlin + Rokid Glass SDK (with WebXR / mobile browser fallback), Ultralytics YOLOv11-seg, MediaPipe Hands, LiveKit or WebRTC, Python FastAPI, Gemini Multimodal Live API, Next.js specialist console. Public data / APIs: HoloAssist egocentric interaction benchmark patterns, TESDA public competency checklists (Electrical Installation and Maintenance NC II, Broadband Installation NC II), DOH BEmONC and primary care clinical flowcharts.

Research & Literature References

1. Xin Wang, Taein Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Felipe Vieira Frujeri, Neel Joshi, and Marc Pollefeys, “HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World,” Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20270-20281, 2023. DOI: 10.1109/ICCV51070.2023.01854
   • How KumpuniAR uses this study: Adopts the HoloAssist proactive intervention architecture where an egocentric AI copilot tracks hand-object manipulation across physical equipment tasks and speaks up immediately when the user makes an execution mistake.
2. Andreas Follmann, Marc Ohligs, Nadine Hochhausen, Stefan K. Beckers, Rolf Rossaint, and Michael Czaplik, “Technical Support by Smart Glasses During a Mass Casualty Incident: A Randomized Controlled Simulation Trial on Technically Assisted Triage and Telemedical App Use in Disaster Medicine,” Journal of Medical Internet Research, Vol. 21, Issue 1, e11939, 2019. DOI: 10.2196/11939
   • How KumpuniAR uses this study: Validates our triage HUD checklist and remote specialist tele-mentoring workflow, which demonstrated significantly higher triage accuracy when field responders used AR smart glasses compared to unassisted control groups.
3. Zhan Zhang, Katelyn Joy, R. Upadhyayula, M. Ozkaynak, R. Harris, and K. Adelgais, “A Smart Glass Telemedicine Application for Prehospital Communication: User-Centered Design Study,” JMIR Human Factors, Vol. 11, e53157, 2024. DOI: 10.2196/53157
   • How KumpuniAR uses this study: Guides our low-cognitive-load visual overlay design so that HUD markers and voice prompts do not obstruct the field worker’s central vision during high-stress emergencies.

Risks & Privacy Notes

• Field safety and human override: KumpuniAR never actuates physical switches automatically. High-voltage and clinical steps display a mandatory safety confirmation and require human verification.
• Bystander and patient privacy under RA 10173: Automatic face blurring runs on the edge before any frame leaves the device for remote mentoring or audit logging. Medical triage sessions require patient or guardian consent when conscious.
• Connectivity drops in rural sites: The core step-by-step checklist and quantized YOLOv11n component detector run locally on the Android host device so basic guidance continues even if 5G drops to offline mode.
• Clause 24 compliance: All hackathon demonstrations use synthetic maintenance props and volunteer roleplay rather than real patient feeds or proprietary telco schematics.
• Strict 30-hour feasibility constraint: Achieving 25fps egocentric tool-and-hand segmentation with low-latency VLM voice interruption on Rokid AR hardware is brittle under changing room lighting and requires scoping the demo to 3 fixed steps.

AI Use Disclosure (draft)

KumpuniAR uses YOLOv11-seg and MediaPipe Hands for real-time component and hand tracking, a multimodal vision-language model for procedural step verification and proactive error warnings, and speech-to-text / text-to-speech models for Tagalog voice interaction. AI coding assistants helped generate WebRTC signaling boilerplate and HUD overlay shaders. All procedural checklists were verified by the team against public TESDA and DOH standards.

Rubric Scoring