The main idea of PoliLips arose during conversations with deaf students, at the MultiChancePoliTeam (the service for students with disability at Politecnico di Milano). Students reported a strong difficulty during class attendance (and their scores, got at the end of courses, confirmed the problem).
Several deaf students provided us an interesting clue, reporting that lipreading was their preferred compensation mechanism, sometimes mixed with aural information, but several factors affected lipreading effectiveness: some words are inherently hard to lip-read, some people can be particularly hard to understand (for example, who talks very fast), or, finally, the position of the speaker can prevent good observation of her/his facial movements. Thus, we started thinking whether a specific device could have increased the effectiveness of lipreading during classrooms.
PoliLips mixes the three information modalities we can collect from the teacher: visual (lipreading), aural, and (ASR generated) textual. In doing so our goal was twofold: first, we argued that each modality could have compensated for errors present in or induced by the others (for example, if the ASR had failed to transcribe a word, the student could have used lipreading to correct the error and understand the correct word); second, the resulting system could have been able to handle different degrees of hearing loss (especially profound deafness), and students’ preferences in compensation mechanisms.
PoliLips captures and sends to students’ laptops, via wired or wireless network, an audio/video/textual stream composed of a video of the teacher’s face, her/his voice, and a textual transcription performed by an ASR. PoliLips facilitates class attendance when the student cannot see the teacher’s face (for example, whenever the teacher writes on the blackboard) or the teacher is too far, or she/he not in front of the student. The device could be useful not only in university classrooms, but in whatever context where a speaker talks to a large audience, and network connections are available.
PoliLips is a hardware/software solution; the teacher wears a hardware device, while specific software applications are installed on teacher’s and students’ laptops. We designed and built the hardware, relying on off-the-shelf components, and developed the applications. ASR functionalities were provided by a commercial application.
PoliLips is part of the CATS project (see the Projects section above).
Designed and developed by:
L. Sbattella and R.Tedesco
A short demo of PoliLips.
The source code will be released soon, as open-source software.