What if a toy could help us understand how young children are developing early math skills? That question sits at the center of a final-year mechatronic engineering project by Camryn Ylonde Abrahamson, supervised by Professor Herman Kamper. The project brought together speech technology, embedded systems, and mechanical design to build an interactive toy car that asks children simple math questions, listens to their spoken answers, and responds in real time.

The educational problem behind the project is serious. In South Africa, many children struggle with foundational numeracy, which is often not picked up early. Large class sizes can make individual attention difficult, while formal assessments and specialist evaluations are often expensive, stressful, or only used once a problem has already become obvious. 

This project explored whether a speech-based toy could offer a more accessible and child-friendly way to support early math assessment. It wasn’t designed to replace teachers or formal clinical assessment. Instead, it was developed as a research prototype that could help make early screening more interactive and easier to use in low-resource settings.

Why Child Speech is Hard for Machines to Process

That sounds simple until the speech part enters the room and starts causing trouble. Most automatic speech recognition systems, which convert speech into text, work best when they’ve been trained on very large amounts of transcribed audio. That creates a problem here for two reasons. Many South African languages don’t have large speech datasets. Child speech is also harder for standard systems to process than adult speech. Children speak with higher pitch, more variation, and less stable articulation, and collecting child speech data comes with clear ethical and practical limits. The report describes this as a double low-resource problem, since both the speakers and the languages come with limited training data.

A Focused Speech Task Instead of Full Speech-To-Text

To deal with that, Abrahamson didn’t build a full speech-to-text system. She narrowed the task to something more focused and realistic for the setting: recognising spoken digits from 0 to 9. The system follows a template-based approach. In plain terms, it compares a child’s recording with a small set of example recordings and predicts which number sounds most similar. Before that comparison happens, the audio is cleaned up and standardised. Recordings are resampled to a common format, trimmed with voice activity detection, which identifies where speech starts and stops, and normalised so that the system pays less attention to differences in loudness or recording conditions.

The project then tested different ways of representing and comparing speech. One option used mel-frequency cepstral coefficients, or MFCCs, which are compact measurements that capture the shape of a speech signal in a way that roughly reflects how humans hear sound. Another used WavLM embeddings, which are learned speech features produced by a large pre-trained model. After comparing alternatives, the strongest system used WavLM Base+ embeddings, dynamic time warping, and k-nearest neighbors. Dynamic time warping is a method that lines up two speech signals even when they’re spoken at different speeds. K-nearest neighbors then classifies the input based on the closest matching examples. In this case, the best result came from comparing each spoken answer with nearby examples and letting the three closest matches vote on the answer.