Due to recent cuts in educational spending at the state level, school districts are under increased pressure to justify spending on technology. Many spend millions of dollars placing computers in classrooms, without addressing whether the new technology has positively effected student achievement. For the most part computer instruction in schools still maintains a focus on application software such as Microsoft Word, Power Point or web-based resources, though these programs have not been shown to improve student achievement (Cuban and Kirkpatrick, 1998). It is apparent that using software types that do not exploit the unique qualities of computer instruction may eventually lead to decreased technology funding. Two concerns for educators are: "Can computer instruction be more efficient than regular instruction?" and "Does a school's current arrangement have any worth?"
A study of basic skills instruction in West Virginia (Mann, Shakeshaft, Becker & Kottcamp, 1999), however, has already demonstrated the cost-effectiveness of computer-based instruction. The study further found that not only can computer-based instruction cost significantly less than implementing statewide initiatives of lowering class size, but also it can reduce traditional instructional time by as much as 30% (Solmon, 1999). Now, an even more recent study in New Jersey's Middletown Township School District further demonstrates that educational software has the potential to improve student test scores.
The Study
The Middletown intervention was conducted with fourth grade students, and the results reveal the achievement-improvement potential for computer-based instruction at the elementary school level. This study, implemented over a period of 8 months between September 2001-April 2002, compared results on student writing samples between two elementary schools. Both schools' student scores on three separate writing prompts were compared, while only one school was exposed to a writing software treatment that utilized two pieces of commercial software (Writing Fitness and Perfect Copy). It was found that the treatment school (Ocean Avenue School) had a 17.91% increase in writing achievement over the control school (Port Monmouth School) as measured by scores on the holistic scoring of writing prompts. It addition, 11.3% more of the students at Ocean Avenue School scored in the proficient range on the 2002 administration of the New Jersey Elementary School Proficiency Test (ESPA) than the students at Port Monmouth School. As this paper will reveal, the technology may have contributed to a 12.91% achievement effect.
Educational Attitudes
Choosing appropriate software for the study meant discovering how teachers in the Middletown Township School District, and at Ocean Avenue in particular, were using computer time. There was a need to know whether they were facilitators of the technology, or were merely trying to teach concepts with software. These teachers were well trained and knowledgeable, yet it was evident that they were freely interpreting what should be taught. In addition they did they have access to any yearly measure of student achievement in technology, and its resultant impact on the regular curriculum.
A district survey revealed that computer teachers regarded their role as that of developing student skills and familiarizing them with computers opening files and mastering the keyboard. They also confused vendor word processing programs with tutorial, and drill and practice programs. Over half (58%) of the teachers surveyed viewed their primary computer role as that of facilitator. They favored computer lab time primarily because their student's were energized by the experience. "They are more motivated than in the classroom," one teacher remarked. The teachers agreed that 40 minutes of computer time a week was not sufficient. However, they clung to the notion that computer instruction was a specialized area of instruction, such as music and art, and were not familiar with the potential of computer instruction supporting regular instruction.
There was a sense that although technology advocates wanted students to be computer literate, there was little agreement as to what elements constituted computer literacy. Prior to the study, technology instruction in the Middletown School District concentrated on application software. The district itself had been more concerned with providing high speed Internet access, rather than identifying appropriate educational applications. The study would instead expose both Ocean Avenue students and educators to tutorial software.
Literature
The research literature illustrates that computers have the potential to effect and create significant efficiencies that may increase student achievement on standardized testing. The factors that assist in this regard are class size, increased use of tutorial software, and increased student exposure time to the software. The following studies document these findings.
Cuban and Kirkpatrick (1998) found, in an examination of ten meta-analytic studies, that drill and practice as well as tutorial instruction produced positive achievement results on standardized testing, but the results with internet-enhanced applications were inconclusive.
A study of basic skills instruction in West Virginia (Mann, Shakeshaft, Becker, & Kottkamp, 1999), found that fifth grade achievement in basic skills increased significantly using a tutorial instructional model (11% score improvement in math and reading on the Stanford–9), and further found that students who had been taught with computers in the classroom scored significantly higher than students who were instructed in a lab setting or students who had a combination treatment of classroom and lab instruction. Students in this study (Mann, et al) were encouraged to interact in small groups, and cooperative learning with computer instruction in this study appears to have been an effective approach.
A study of mathematics achievement (Wenglinsky, 1998) found that there was a small positive effect (treatment students were only 3 to 5 weeks ahead of non-treatment students) on fourth grade achievement using mathematical drill and practice learning games; conversely, there was a negative effect with eighth grade students. It was further discovered at the eighth grade level that there was increased student achievement as a result of the use of tutorial and simulation software.
It should be noted that one of the difficulties in attempting to measure the effect of a software treatment is the problem of controlling for all of the variables as they relate to classroom instruction. If student achievement increases during an intervention, the result may be a technology effect, but that effect will invariably be combined with other factors such as outstanding classroom instruction. The above-mentioned studies have attempted to control some of the instructional variables in order to measure a technology involvement with increased achievement. One particular study by Van Groesback (1995) appeared not to have controlled for all variables and attributed the positive results of the study to the technology intervention. Due to this weakness, the study should be viewed with a skeptical lens, and is only mentioned here because the achievement gains were similar to this study and the Mann et al. (1999) study.
Software
For this study, two pieces of educational software were purchased and made available to the fourth grade teachers and the computer teacher at Ocean Avenue School. Writing Fitness from Merit Software and Renaissance Learning's Perfect Copy became the focus of the investigation. Tutorial software was chosen for the study, because it seeks to address higher order thinking skills by guiding students via interactive prompts, checking student understanding, and recording student progress for teacher use.
The intervention process focused on comparing fourth grade student achievement. Teachers at both schools cooperated by providing identical writing picture prompts to their students and jointly scoring the results. Embedded measures in the software were monitored to ascertain whether they provided a window relative to increased achievement. Teachers at the Ocean Avenue School were interviewed throughout the process to answer questions about the perceived effectiveness of the software.
Results
It was found that the treatment school (Ocean Avenue School) had a 17.91% increase in writing achievement over the control school (Port Monmouth School) as measured by scores on the holistic scoring of the writing prompts. In addition, 11.3% more of the students at Ocean Avenue School scored in the proficient range on the 2002 administration of the New Jersey Elementary School Proficiency Test (ESPA) than the students at Port Monmouth School. Ocean Avenue School experienced an increase in the percentage of proficient students by 3.95% over the previous year's results, while Port Monmouth School had a 1% decrease in the number of students proficient over the previous year's results.
Table 1 illustrates the percentage of students who had a sustained increase in the writing prompts over the duration of the study. For the purposes of the study, students were considered to have achieved a sustained increase if they had at least a one-point gain over their first writing prompt scores. It should be noted that classes at Ocean Avenue School are designated with the abbreviation OA, while classes at Port Monmouth are designated with the abbreviation PM
Table 1
Percentage of Students Who Achieved a Sustained Increase on the Writing Prompts
School
Percentage
Ocean Avenue School (OA)
59.73%
Port Monmouth School (PM)
41.82%
Table 2 delineates the average percent increase on each section of Writing Fitness 1. The results are suspect because of the loss of data due to the instability of the computer lab at the time when students were working on this section. The entire lab was upgraded to Windows 2000, and there was no loss of data with sections 2 and 3. The percentage increases on Writing Fitness 2 and 3 are similar except for class OA 3 on section 2. This achievement gain is curious because it is more than double the other five score increases in sections 2 and 3. With Writing Fitness 2, OA 3 achieved a score that was 19.6 percentage points higher than the average score on the same section achieved by all three classes.
Table 2
Average Percent Increase on Writing Fitness at Ocean Avenue School.
Classes
Writing Fitness 1
Writing Fitness 2
Writing Fitness 3
OA 1
0%
25.17%
26.90%
OA 2
0%
20.53%
16.59%
OA 3
13.18%
52.25%
17.90%
Average
4.39%
32.65%
20.46%
This result somewhat parallels a similar increase in Perfect Copy for OA 3. Table 3 illustrates the average score achieved by each class and the total number of student hours spent working with the Perfect Copy software application. Perfect Copy was attempted by all three classes when they were working on Writing Fitness 2. The results revealed that OA 3 had an average score that was 7.9% higher than the average score of the other two classes on Perfect Copy. The 7.9% score advantage was achieved at a time when OA 3 was spending 2.41 times the amount of time working with the Perfect Copy application than the other two OA classes. From these preliminary results it would appear that for OA 3 there was a correlation between the time spent on Perfect Copy and the above average scores on that application
Table 3
Average Score Achieved by Each Class and the Total Amount of Student Hours Spent Working on Perfect Copy
Class
Average Score
Total Hours
OA 1
45.72%
30.22 hrs.
OA 2
46.23%
23.72 hrs.
OA 3
49.60%
72.85 hrs.
School
47.18%
42.26 hrs
Discussion and Conclusion
One question for this discussion is whether the implementation components had an effect on student achievement and, if so, how large was the effect. The size of the sustained increase of the achievement on the writing prompts for Ocean Avenue students is evidence that student achievement did increase during the implementation. In addition, Ocean Avenue students had an 11.9% proficient advantage over Port Monmouth and raised the percentage proficient by 3.95% over the previous year's results. However, the results cannot be solely attributed to the technology intervention because of the inability to completely factor out the variables of inconsistent student writing and the effectiveness of classroom instruction.
The increased achievement results with tutorial software provide the most reliable evidence of increased student achievement during this implementation. It has been demonstrated that students in OA 3 experienced increased scores on Perfect Copy, probably as a result of increased time with that application. The increases occurred over a 3-month period, which suggests that the software had an effect over a relatively short period of time. While OA 3 students were outperforming OA 1 and OA 2 students on Perfect Copy, they were also outperforming the two classes on Writing Fitness. Because the instructional topics on both applications are not comparable, increased time with Perfect Copy probably had an effect on this increase. It has also been established that all three fourth grade classes were balanced along gender, ethnicity, and cognitive ability lines, which somewhat diminishes classroom demographics as a possible contributing factor.
What renders the increased achievement results for OA 3 compelling is that they parallel similar results on a quasi–experimental study (Van Groesbeck, 1995) that measured student achievement utilizing Perfect Copy. In this study Van Groesbeck created an experimental group and a control group of fourth, fifth, and sixth grade students and exposed the experimental group to the Perfect Copy treatment for 13 weeks. The results of the Van Groesbeck (1995) study revealed a 20.3% average mean score increase in achievement for the experimental group.
The Ocean Avenue implementation measured an increase in achievement over a similar time interval. The Ocean Avenue results somewhat replicate the results in the Van Groesbeck (1995) study and provide further proof that tutorial software might have a positive effect in achievement. The two studies are parallel in that there was increased achievement over a short period of time. The Ocean Avenue study further demonstrated the importance of increased time spent on the tutorial application in order to increase achievement.
If there was an attempt at measuring the technology effect of this implementation, the size of the gain would be contained in the difference between the two schools' average scores on the writing prompts and the percentage of students who scored in the proficient range on the ESPA. In the four years (1999-2002), a larger percentage of students at Ocean Avenue School scored in the proficient range on the ESPA than did the Port Monmouth students. An average of the years 1999 and 2001 was used and resulted in a 5% achievement advantage for Ocean Avenue students. During this implementation 59.73% of Ocean Avenue School students achieved a sustained improvement in scores on the writing prompts, wherein 41.82% of Port Monmouth students achieved a sustained improvement in scores on the same writing prompts. This resulted in a 17.91 percentage point difference in sustained improvement on the writing prompts between the two schools. If the 5 percentage point longitudinal ESPA differential in achievement between the two schools was subtracted from this amount, a 12.91 percentage point difference would still be realized.
In essence, the longitudinal difference in achievement between the two schools has been small, but with the difference in scores on the prompts, it would suggest that technology might have had an effect. What makes this result particularly compelling is that it is within the ranges of the Van Groesbeck (1995) and the West Virginia Basic Skills Study (Mann et al., 1999) studies. In the Mann et al. (1999) study it was found that the technology treatment accounted for 11% of the increased achievement in the West Virginia Basic Skills initiative, and Van Groesbeck (1995) reported a 20.3% effect. It seems reasonable to infer that tutorial software has the potential to increase achievement within a range of 11% to 20% with a variance that is yet to be determined. By means of the same inferential analysis, the Ocean Avenue implementation had a similar effect.
Implications
It appears that continuing to have students work with tutorial software such as Perfect Copy and Writing Fitness for long periods of time may assist in improving student achievement. The data do not clarify the issue of whether there was a single factor that assisted with increased achievement. The results do not support that Perfect Copy is superior to Writing Fitness, only that increased time with tutorial software appears to have an effect. Perfect Copy as a product is versatile and expandable in that articles for practice are grade appropriate and additional articles can be purchased. It appears that Writing Fitness may prove to be a good measure for writing mechanics improvement.
Email: Charles W. Bindig, Ed D.
Resources
Cuban, L., & Kirkpatrick, H. (1998). Computers make kids smarter right? Technos, 7, 26-31.
Kulik, J. A. (1994). Meta-Analytic studies of findings on computer based instruction. In E. L. Baker, & H. F. O'Neill, Jr. (Eds.), Technology assessment in education and training. (pp. 9-33). Hillsdale, NJ: Erlbaum.
Mann, D., Shakeshaft, C., Becker, J., & Kottkamp, R. (1999). West Virginia's basic skills/computer education program: an analysis of student achievement. Santa Monica, CA: Milken Family Foundation. Retrieved January 4, 2001, from the World Wide Web: http://www.mff.org/pubs/ME155.pdf
New Jersey State Department of Education. (1996). Core Curriculum Content Standards (New Jersey State Department of Education Publication No. PTM 1400.06). Trenton, NJ.
Perfect Copy [Computer software]. (1999). Wisconsin Rapids,WI: Advantage Learning Systems, Inc.
Solmon, L. C. (1999). Afterword; comparing technology with other policy initiatives. In Mann, D., Shakeshaft, C., Becker, J., & Kottkamp, R. (1999). West Virginia's basic skills/computer education program: an analysis of student achievement. Santa Monica, CA: Milken Family Foundation. Retrieved January 4, 2001, from the World Wide Web: http://www.mff.org/pubs/ME155.pdf
Van Groesbeck, R. (1995). "Mini" ILS improves students' language mechanics skills. Technological Horizons in Education, May, 1995. Retrieved April 5, 2002, fromthe World Wide Web: http://www.thejournal.com/magazine/vault/articleprint version
Wenglinsky, H. (1998). Does it compute? the relationship between educational technology and student achievement in mathematics. Educational Testing Service Policy Information Center. Retrieved January 8, 2001, from the World Wide Web: ftp://etsis1.ets.org/pub/res/technolog.pdf
Writing Fitness [Computer Software]. (1999). New York, NY: Merit Software [Producer and Distributor]
