Studio Xia update: Handwriting recognition demo
The Studio Xia Chinese program currently asks students to grade themselves on writing hanzi (Chinese characters). It shows what the hanzi is supposed to look like and then asks you to click right or wrong; the score is entirely on the honor system.
Which works pretty well, as long as everybody is there to learn. If you cheat, after all, you're only cheating yourself.
But as I try to develop this software from a set of drill activities towards something more gamelike (the eventual goal is "Learn Chinese: The RPG"), it would be really nice to have some automatic scoring of hanzi input. Maybe the amount of damage you do to a monster depends on how fast and accurately you draw one of the hanzi from your current working set, for example.
I started out a couple weeks ago by trying to compile an open-source OCR (optical character recognition) package called Tesseract which supports the simplified Chinese character set. Ran into a bunch of dumb Linux linker dependency problems, but finally got it working -- only to discover that its accuracy was really awful.
- We already know which hanzi the student is supposed to be drawing, so we only have to compare against that one, not the entire corpus.
- We can observe the student writing the hanzi stroke by stroke in an HTML Canvas element, and analyze as they go; this is a much easier starting point than, say, a bitmap - no need to do edge detection when we can get the start and endpoints of each line directly.
- Observing the student write each stroke also gives us a rich set of data that would not be available from a static bitmap - such as which direction they were moving the "pen" when they drew each stroke, and the points at which they changed direction.
Better yet, by detecting where the students' strokes differ from the strokes of the hanzi model, we can offer feedback like "These lines are not supposed to cross here" or "You messed up the stroke order" or "You missed one stroke", etc.
After a couple of weeks of experimentation, I have an algorithm that I think is ready for wider testing.
Try the demo here! (For now this demo includes only a single character, 你, which means "you".)
If you want to help me out, please try it and let me know:
- Does that demo work on your browser? If not, which browser do you have?
- When you write the character, does the feedback that it gives seem accurate? Or is it marking something wrong that you think is correct?
(If you find bugs, a screenshot would be very helpful!)
A single-character demo isn't very impressive, but I have also created a page where a teacher can input new hanzi to the system. The teacher draws five or more correct examples of the hanzi, and my program uses them to construct what I call a "fuzzy model" to capture the range of acceptable variation in the shape of the hanzi.
Final note: Writing hanzi with a mouse (or worse, trackpad) really sucks. I know that. This program will work with a mouse, but touchscreen users are the main target audience. Touchscreens (whether phone, tablet, or laptop) are the wave of the future and in a few years mice will start to look very old-fashioned. So I'm especially interested in getting this to work well on iOS Safari, Android stock browser, Android Firefox, Amazon Kindle stock browser, Windows 8 tablets, etc. etc. If you have one of those, please use it to try out the demo and let me know what problems you run into!