AI Typing Tutor
I set out to build an AI-assisted typing tutor to help supercharge typing skills. Hoping to also make it fun, I did so in the format of a retro game!
Why Typing Still Matters
Let's be honest: in a world of voice assistants and AI that can write entire essays from prompts, does typing still matter? The answer is yes—at least until Neuralink or some other mind-reading tech becomes more efficient.
The reality is that typing remains one of the most fundamental ways we interface with computers. Whether you're coding, writing, or just communicating, the speed and accuracy of your typing directly impacts your productivity. And here's the thing: most people never actually learned how to type properly. They just… figured it out over time, hunting and pecking their way through life.
In the AI era, typing skills are more important than ever with natural language (or "prompt") being the conversation between humans and AI. The faster and more accurately you can input your thoughts, the more effectively you can work with AI tools rather than being slowed down by them.
The Breakthrough: Seeing Under Your Knuckles
I've been experimenting with different forms of AI vision for multiple projects recently, but this one caught me by surprise. The breakthrough came when I realized hand tracking technology can be used to determine—from your webcam, in real-time—which keys your fingertips are touching.
Not which keys you're pressing. Which keys you're touching.
This might sound like a small distinction, but it's huge. Traditional typing tutors can only see what you type after you press a key. They can't see your hand position, your finger placement, or whether you're hovering over the wrong key before you make a mistake.
But with computer vision and hand tracking? We can see everything.
The moment I realized it might be something special was during testing with different hand positions. I tried various angles, different postures, even awkward positions where my palm was blocking the view. And the system could still track my fingertips. It could literally "see" under my knuckles.
The core hand detection utilized is from Google MediaPipe. Combining this technology with other forms of deterministic and LLM-assisted features, I can hopefully make a fun game that is actually useful.
What's Next?
This is still in the early stages, but the potential here is enormous. Imagine a typing tutor that:
- Sees your hand position in real-time
- Provides instant, visual feedback (look at screen, not keyboard)
- Identifies user typing patterns
- Provides coaching for user improvement areas
- Works entirely in your browser using your existing webcam
No special hardware. No expensive equipment. Just AI-powered computer vision and a webcam.
Check out this demo of me running through an early iteration of the calibration process that maps the virtual keyboard. In the future it may be calibration-free, but using this approach allows me to move on to core gameplay and tutor features. After calibration it is able to detect my fingertips on the key caps in real time!
Right now, I'm doing some family and friends testing to refine the experience and work out issues. If you're interested in helping test or collaborating on this project, reach out! I'd love to hear from educators, typing instructors, developers, or anyone curious about the intersection of AI and education.
More to come on this project soon. Stay tuned.