Department of Computer Science
Northern Illinois University
Dekalb, IL 60116

This presentation describes a framework for conducting language learning research with materials presented and assessment conducted over the World Wide Web. Three pilot studies to explore this model are described, and an informal summary of the results of conducting these studies is presented, as well as conclusions and recommendations for future research of this type.

Motivation for the project

o increasing the mix of student characteristics
o increasing the number of subjects for statistically significant findings (important for LCT languages)

Languages and pilot studies

Technology

Results

Summary/Conclusions

Language learning research, especially for the less commonly taught languages, often suffers from two problems. The first is the "generalizability" of a study: are a certain set of results due to something special about the teacher or the students who participated in the study? The second is the usually small number of subjects in a given study. A typical beginning class in any of the Southeast Asian languages at U.S. universities consists of around 10 students (often less; sometimes more). This sample size is not likely to lead to statistically significant findings.

The Internet would seem to provide a way to reduce the severity of these problems. If students at different universities could combine to participate in a language learning study, we would at once have a more diverse student/subject population and a larger number of subjects. Language research for the less commonly taught languages could be done in a significant and meaningful way.

Finally, the ability to easily collect and store data on student performance on a central database would be a great advantage. Although the tests, quizzes, etc. can be done at any computer connected to the internet, anywhere in the country (or world), the results can be automatically stored in one central location for later analysis. Manual transmission of data and/or manual data entry of results from disparate locations is entirely eliminated.

For this project, we designed three pilot studies in which instruction and assessment would be done over the Internet: one each for Thai, Indonesian, and Tagalog. These studies would serve as proofs-of-concept for this model of a distributed study. Students at Northern Illinois University would participate in the second year of the project to test the software, database, and network technologies. In the third year, students at other universities would join NIU students. A summary of the studies follows:

Indonesian: does listening-while-reading improve comprehension of new reading materials? This study was a moderately complex design, since we did not want to disadvantage any students by giving them different learning materials over the course of time. That is, we did not want to have an experimental group (who got audio) and a control group (who did not). Rather, students were divided into two groups; on alternate weeks each group was given readings with or without accompanying audio. A within/without groups analysis will be performed to determine if the availability of audio led to better scores on a post-test.

Thai: does practice with feedback improve tone discrimination ability? This study is a classic pre-test, instruction (in this case, practice with feedback), post-test design. Students were asked to listen to three words and tell which has the different tone (or if they are all the same). The practice mode allows them to see if their answer was correct and to see the correct answer; it also allows them listen to the words any number of times. The three sessions were to be conducted within a single week, but with at least one day between sessions.

Tagalog: What learning strategies are used by students to acquire new vocabulary in a second language? What strategies facilitate longer retention, depth of word knowledge, and appropriate word use? What are the implications of these results in the teaching and learning of vocabulary words in a foreign language classroom, particularly when the medium for presenting instruction is, in part, the Internet? Weekly vocabulary lessons presented on the Internet contained Web-based activities that supported five different learning strategies. The students were also asked to keep a personal log that recorded time spent studying vocabulary on the Web as opposed to other locales (e.g., home, language lab, etc.)

All computer-based materials were presented to students over the World Wide Web. Our Web servers are Microsoft IIS servers. Data on student behaviors (answers, timings, etc.) was stored in Microsoft SQL databases (Tagalog in one database; Thai and Indonesian shared a single database structure). All three studies used Java technology to present materials and record and store student answers; for Tagalog, Microsoft Active Server Page (ASP) technology was also used to track and log which Web pages students visited and to record the amount of time they spent on each page.

The Thai study used a Sun-provided Java extension called Java Media Framework (JMF) to present high-quality audio. Because high-quality audio is critical to this study and because there can be delays in loading audio over the Internet, we needed a system that could

JMF provided the only Web-based system with these features.

The Indonesian materials were standard: Real Audio was used to present the accompanying audio, Web pages displayed the text, and a Java applet presented quiz questions and sent the results to the database. The course instructor configured versions with and without audio each week, and the quiz applet had a login mode to accept only students in the appropriate group each week.

The Tagalog study used Java applets to support five learning strategies that were presented in weekly vocabulary lessons. Weekly on-line surveys asked students about their preferences and which strategies they found to be most useful. ASP was used to keep track of the Web pages the students visited and to record the amount of time they spent on each learning strategy exercise. The students also kept a personal log in which they were asked to record how much time they spent studying new vocabulary at different locations. Weekly on-line quizzes were administered and the results were saved in a database and emailed to the language professors.

Technology: in general, the technology worked well. With the exception of a difficult month resulting from a major hacking intrusion, the Web servers worked without problems, the database system (SQL server) was trouble-free, ASP technologies worked as desired, as did the Java applets (with one exception).

The single unresolved Java problem was with the JMF extensions necessary to deliver high-quality audio and enable precise timings of student responses. While these extensions worked well over the Internet within our university, problems did occur with remote sites (Alabama). In some cases, the practice exercises would not start or would terminate prematurely at the remote site. The error messages claimed that the audio format was not recognized, yet at other times the same audio would load and play successfully. We have not been able to resolve this problem. In addition, it appears that different versions of the Java Virtual Machine plug-ins may not be compatible with the Applet. These issues are difficult to overcome since we cannot control the computing environment at remote locations.

The use of a relational database (Microsoft SQL Server) proved to be more useful – and easier to use – than expected. In addition to operating trouble-free for the entire project (once configured and set up), the query language feature was useful in formulating ad hoc queries against the database. For example, we could easily form and execute queries such as

None of these queries (and others) was anticipated in the original project design, yet they were easy to pose as new needs for information emerged during the project.

Finally, our Graduate RA wrote an interactive query program that wrote query results to a comma delimited file; that file then was easily importable into spreadsheet and statistical programs for further analysis.

Project Management: coordination between project members during the project was a continuing problem. For example:

Clearly, most of our external consultants were not as deeply invested in the project as we were. Because of some of these problems, the validity of data from the remote institutions may be questionable.

Student Participation: in retrospect, it might have been wiser to design studies for intermediate or advanced students. Some beginning students are not highly motivated and regard any "extra work" as a burden and if the computer work does not contribute to their grade, they may not take it seriously. For example:

If the computer work does contribute to their grade, they may find ways to "beat" the system. For example:

Trying to prevent some of these behaviors rather than relying on voluntary conformance with our requests required a substantial amount of time and effort. In one study, quizzes were posted – each week - just before the class was to take the quiz (so that students could not preview the questions).

Required Resources: time, skills, and effort. In order for an internet-based research system to be practical, it must not place excessive demands on researchers. Of course, a reasonable amount of time and effort are expected. In the case of high-technology projects, one needs technical people (programmers, computer system administrators, etc.) as well as language and content experts. This immediately adds a layer of complexity and cost to any such project. In the case of this project, the technical demands varied:

The overall model proposed and tested by this project seems valid. The benefits of centralized data collection and the pooling of subjects at different locations can outweigh the disadvantages that were encountered in these pilot studies, especially if the following points are addressed in the design of future studies:

1. Increase initial efforts at the beginning of the project to ensure the interest and commitment of external project personnel. (Apparently monetary honoraria are not always sufficient motivation.) Ensure that they are given a clear statement of what is expected and what their contribution is to be. It may be helpful to bring them in at an early stage to help design the project. This will help them become "invested" in the project and will ensure that they understand it well.
2. Be wary of new, uncommon, or untested technologies. The Thai study’s use of JMF (new and uncommon) caused a problem. Stick to well-known and well-understood technologies. Use the simplest technologies appropriate to the problem.
3. Try to do studies with serious, well-motivated students. The confounding factors of lack of motivation and disinterest are difficult to quantify and deal with. For unmotivated students, the issue of rewards/punishments is difficult: should participation/performance on study tasks be linked with course grades or not?
4. Consider internet-based study topics and designs that do not constrain students in terms of time and place and communications. The nature of the internet makes it difficult to, for example,

o insist that all students take an exercise at a given time
o insist that students have no access to other on-line helps and aids
o insist that students take an on-line test just once (this is preventable, but is awkward and resented)
o dictate the order in which activities are done

5. Pure experimental studies, always problematic in education, are problematic when the medium is the Internet as well. In addition to the issues just mentioned, which make it difficult to control relevant variables, there is the question of preferential treatment: if we are comparing 2 treatments, one of which is supposed to be superior and will bring about enhanced outcomes, how can we justify not making this treatment available to the control group?
6. Ensure there is technical help available or consider the use of commercial software products. Although the technical requirements used in these studies would not be considered cutting-edge by technologists, they nonetheless required substantial expertise and time. Most language teachers have neither the background nor the time to develop and administer these kinds of projects. Access to technical help is vital. Alternately, some commercial software may be suitable for certain studies (e.g. Blackboard) – however, all such packages will have limitations which may make them unsuitable for specific projects (e.g. Thai audio and timing requirements).