Mitra Robot
Formerly Invento Robotics
Overview
Mitra Robot, formerly Invento Robotics, builds commercial humanoid robots with large language models built in, for customer engagement, healthcare assistance and visitor management.
- Role
- Software Engineer Intern
- Employment type
- Internship
- Dates
- Aug – Dec 2022
- Team
- Two interns, reporting to the Chief Software Architect
- Company
- Mitra Robot (formerly Invento)
Delivered
Fall detection
< 20 sto alert medical staff
- Real-time fall detection model: fine-tuned vision transformers with OpenAI CLIP, in TensorFlow and PyTorch
- Live on a ROS2 robot in a senior care facility in Texas, USA
- Recognises falls in low light, too
Gesture recognition
Older residents' voices can be soft or broken, so speech alone fails. Simple gestures say what they need.
36 gestures in all, for example:“I want food”“Show me my family's latest photos”“Play some music”
- Ran a data collection event with young and older people
- Collected, cleaned and labelled every training video
- Fine-tuned CLIP-based models and other vision transformers, with OpenCV and MediaPipe, to recognise each gesture
20,000+gesture videos
36gestures
25+participants, young and old
The ML pipeline
- Worked across the whole AI/ML pipeline, from data to deployment
- Ran model evaluations, testing each model against how the robot actually performed
- Wrote storage optimisation algorithms to store large amounts of robot-collected video data
Robot operations
- Fixed WebRTC bugs
- Handled AWS and GCP
- Kept the robots running smoothly
Stack: Python, PyTorch, TensorFlow, ROS2, Computer vision, OpenCV, MediaPipe, CLIP, Vision transformers, Statistical modelling, Model evaluation, Dataset collection, WebRTC, AWS, Google Cloud, Human–robot interaction