Dutch-CAPT (NWO PROO) project
Implications of Potentially Erroneous Feedback in CALL Systems
(In Dutch: “Implicaties van potentieel foutieve feedback voor CALL-systemen”)
The ultimate objective of this project is to advance our understanding of the effects on learning of the frequency and the seriousness of feedback errors in advanced Computer Assisted Language Learning (CALL) systems. The domain in which we work is pronunciation training in the acquisition of Dutch as a second language. Pronunciation training is an outstanding example of a situation in which the need for interactive multimedia environments is obvious. At the same time it is obvious that state-of-the-art speech and language technology will make occasional assessment errors when it comes to evaluating the correctness of responses in interactive student-system dialogs, both with respect to contents and pronunciation quality or fluency. Empirical investigation of the impact of feedback errors in interactive multimedia courses requires that such courses are available.
For this purpose, we have developed Computer-Assisted Pronunciation Training (CAPT) program based on Automatic Speech Recognition for learners of Dutch with different mother tongues (Dutch-CAPT). The program offers realistic language input in the form of short movies with Dutch actors and audio recordings of several native speakers. It contains a wide range of pronunciation exercises including minimal pairs and dialogues and it provides automatic, easy-to-understand, and reliable feedback on the student’s pronunciation, by focussing on eleven problematic Dutch sounds. The program, whose design has been based on a thorough analysis of typical segmental errors made by non-native speakers of Dutch, has been tested in a series of experiments. The subjects of these experiments were beginner learners from various countries across the world, who were following a Dutch course. One group of subjects used this program for a few hours over a period of four weeks; another group used a program that did not provide any feedback on pronunciation for approximately the same amount of time; a third group who did not train with any program was used as control. The pronunciation of these subjects before and after training was scored along the dimension ‘segmental quality’ by a number of experts. The results of these experiments, which are currently being examined, will reveal whether this program, which has the advantage of providing immediate and automatic feedback, does indeed lead to significant improvements in the learner’s pronunciation, and whether it is conducive to higher learning gains than more traditional forms of training without feedback.
This project has been carried out within the framework of the NWO ‘Programma voor onderwijsonderzoek’ (PROO).
Publications
- A. Neri, C. Cucchiarini, H. Strik
The effectiveness of computer-based corrective feedback for improving segmental quality in L2-Dutch
To appear in ReCALL. [PDF] - C. Cucchiarini, A. Neri, F. de Wet, H. Strik
ASR-based pronunciation training: Scoring accuracy and pedagogical effectiveness of a system for Dutch L2 learners
Proceedings of Interspeech-2007, Antwerp, Belgium, pp. 2181-2184. [PDF] - A. Neri, C. Cucchiarini & H. Strik
Pronunciation training in Dutch as a second language on the basis of automatic speech recognition.
Stem-, Spraak- en Taalpathologie, Vol. 15, No. 2, pp. 157-167. [PDF] - A. Neri, C. Cucchiarini, H. Strik
Selecting segmental errors in L2 Dutch for optimal pronunciation training.
IRAL – International Review of Applied Linguistics, 44, pp. 357–404, 2006. [PDF] - A. Neri, C. Cucchiarini, H. Strik
Improving segmental quality in L2 Dutch by means of Computer Assisted Pronunciation Training with Automatic Speech Recognition.
Proceedings of CALL 2006, Antwerp, Belgium, pp. 144-151. [PDF] - A. Neri, C. Cucchiarini, H. Strik
ASR-based corrective feedback on pronunciation: Does it really work?
Proceedings of Interspeech-2006, Pittsburg, USA, pp. 1982-1985. [PDF] - K. Truong, A. Neri, F. de Wet, C. Cucchiarini, H. Strik
Automatic detection of frequent pronunciation errors made by L2-learners.
Proc. of InterSpeech 2005 (IS2005), Lisbon, 4-8 Sept. 2005, pp. 1345-1348. [PDF] - K. Truong, A. Neri, F. de Wet, C. Cucchiarini, H. Strik
Automatic pronunciation error detection: an acoustic-phonetic approach.
Acoustic Society of America Conference, workshop on Second Language Speech Learning, Vancouver 14-15 May 2005. - K. Truong, A. Neri, C. Cucchiarini, H. Strik (2004)
Automatic pronunciation error detection: an acoustic-phonetic approach.
Proceedings of the InSTIL/ICALL Symposium, 17-19 June, Venice, Italy, pp. 135-138. [PDF] - A. Neri, C. Cucchiarini, H. Strik (2004)
Segmental errors in Dutch as a second language: How to establish priorities for CAPT.
Proceedings of the InSTIL/ICALL Symposium, 17-19 June, Venice, Italy, pp. 13-16. [PDF] - A. Neri, C. Cucchiarini & H. Strik (2003)
Automatic Speech Recognition for second language learning: How and why it actually works.
Proceedings of 15th ICPhS, Barcelona, Spain, pp. 1157-1160. [PDF] - A. Neri, C. Cucchiarini, H. Strik, L. Boves (2002)
The pedagogy-technology interface in Computer Assisted Pronunciation Training.
Computer Assisted Language Learning, 15:5, pp. 441-467. [PDF] - A. Neri, C. Cucchiarini, H. Strik (2002)
Feedback in Computer Assisted Pronunciation Training: when technology meets pedagogy.
Proc. of the 10th Int. CALL Conference on “CALL professionals and the future of CALL research”, University of Antwerp, pp. 179-188. [PDF] - A. Neri, C. Cucchiarini, H. Strik (2002)
Feedback in computer assisted pronunciation training: Technology push or demand pull?
Proc. of ICSLP-2002, Denver, USA, pp. 1209-1212. [PDF] - A. Neri, C. Cucchiarini & H. Strik (2001)
Effective feedback on L2 pronunciation in ASR-based CALL.
Proc. of the workshop on Computer Assisted Language Learning, Artificial Intelligence in Education Conference, AIED-2001, San Antonio, Texas, pp. 40-48. [PDF]
Presentations
- Uitspraakevaluatie & training met behulp van spraaktechnologie.
Pronunciation assessment & training by means of speech technology.
Symposium Virtuele communicatieondersteuning, Gasthuisberg, K.U. Leuven, 28-04-2007. [PPT] [PDF] - ASR-based corrective feedback on pronunciation: Does it really work?
Special Session Wed3A3O: Technologies for Specific Populations: Learners and Challenged
Oral presentation Wed3A3O.2, Interspeech 2006, Pittsburgh, 20-09-2006. [PPT] [PDF]
The pedagogical effectiveness of ASR-based computer assisted pronunciation training
Ambra Neri
PhD Thesis, University of Nijmegen [PDF]
Abstract:
Computer Assisted Pronunciation Training (CAPT) systems with Automatic Speech Recognition (ASR) technology have become increasingly popular to train pronunciation in the second language (L2). The advantage of these systems is the provision of a self-paced, stress-free type of training with automatic feedback on pronunciation quality. Despite this popularity, little is known on the actual pedagogical effectiveness of these systems. In other words, little empirical evidence is available as to whether and to what extent the use of these systems can improve pronunciation quality for a learner, while it is well-known that ASR-based feedback on non-native pronunciation quality is not yet 100% error-free. The research reported on in this thesis investigates the pedagogical effectiveness of ASR-based feedback on segmental quality. The thesis starts by identifying pedagogical requirements for pronunciation training in L2. Existing CAPT systems are then critically examined to establish which pedagogical requirements can be achieved with current ASR-based CAPT technology. Some of these suggestions are subsequently implemented to develop a customized ASR-based CAPT system (Dutch-CAPT) for teaching Dutch pronunciation to adult immigrants. First, a method is presented to select important segmental errors made by learners of Dutch with different mother tongues. By means of auditory analyses of different speech databases, an inventory of eight Dutch phonemes that appear to be particularly problematic tp learn is obtained. This inventory is subsequently implemented in Dutch-CAPT, which offers a simple form of feedback on segmental errors. The improvement made by a group of immigrants who used Dutch-CAPT is measured and compared to that of controls. The results indicate that the ASR-based feedback provided yielded the largest improvements in the pronunciation of the targeted phonemes, despite occasional errors in the feedback. The thesis ends with suggestions to design pedagogically sound and technologically reliable ASR-based CAPT, and to evaluate these systems.
.