Project 02 · 2024
Therapy exercises you play with your hand, and a webcam
Hand therapy after a stroke or an injury is repetitive, and it is done at home where nobody is watching. So it gets skipped, and there is no record of what was actually done. This turns the exercises into games driven by a webcam, with the difficulty calibrated to the patient's own range of motion and the results going back to their therapist.
This is the Free Drawing exercise, ported from the original to run in the browser. Pinch your index finger and thumb together to put the pen down, open them to lift it. The hand tracking model runs inside this tab, so the video never leaves your device and nothing is uploaded anywhere.
Pinch your index finger and thumb together to draw, and open them to lift the pen. The video never leaves your device: the model runs in this tab.
ratio no hand
A webcam gives 21 landmarks per hand and nothing else. Everything clinical has to be derived from them, and the hard part is not detecting the gesture once, it is detecting it the same way tomorrow, in a different room, with the patient sitting closer to the camera.
A pinch is the distance between the index tip and the thumb tip, but in pixels that distance halves when the patient sits twice as far away. So it is divided by the length of the thumb's proximal phalanx, a bone the hand carries with it. The ratio is scale-invariant, which is what lets one calibrated threshold keep meaning the same gesture across sessions.
That ratio is compared against a number measured during calibration from the patient's own range of motion. A hand that cannot close fully gets a threshold that its best pinch can still reach, so the exercise stays winnable and the progression is a number the therapist can move deliberately rather than a difficulty setting.
Pinch and lateral pinch come out of fingertip distances. Wrist flexion, extension, abduction and adduction come out of the angle between the forearm vector and the hand vector. Pronosupination comes out of how the palm's landmark spread collapses as the hand rotates, which is the awkward one, since rotation about the camera axis is exactly what a single 2D view sees worst.
Each game trains a movement, but the reason there are seven is adherence. The same pinch repeated four hundred times is the thing patients stop doing, so the movement stays fixed and the game around it changes.
Every exercise takes the same configuration block, so the therapist tunes the session rather than picking a preset: level, repetitions or elapsed time as the stopping condition, which hand, whether to calibrate first, which fingers are in play, collision tolerance, and a cognitive load flag that adds a decision on top of the movement. High contrast, sound and negative feedback are separate switches, because the population includes patients for whom each of those is the wrong choice.
A rehabilitation exercise that no clinician can prescribe or read back is a toy. So the other half of the project is a desktop application for the therapist: patients and professionals with their relationship between them, routines assembled from configured exercises, stored calibrations, a calendar of scheduled sessions, and export to PDF for the clinical record. Credentials are stored encrypted, and the patient never sees this side of it at all.
Next project
Quadruped Walking Robot