In the April issue of Technology & Learning, part one of this article described the first two stages that schools move through as they learn to link data to higher student achievement (www.techlearning.com/story/showArticle.jhtml?articleID=160400818). In stage one, schools make initial efforts to contextualize the many data sources available, but their data analysis is mostly for its own sake. Few of the findings make their way into classroom practice, and the connection of data analysis to student achievement remains tenuous.
As individuals and stakeholder groups develop experience and sophistication in understanding the limitations, context, and implications of various data sources, they move into stage two — using data to improve educational efficiency. In this phase, the school does everything it can to maximize the performance of students who are on the verge of moving to the next level and thereby get as many as possible over the bar. Schools in stage two that are persistent in gathering formative assessment data and adjusting teaching to fill the identified gaps in student performance are able to achieve significant gains in school performance for a few years as they remove the slack in the system and push up the "bubble" students to the next level.
In the long run, however, schools that stay in stage two — dedicated to improving efficiency by focusing on the students at the margin — will run headlong into the ever rising NCLB standards. Data analysis that focuses on improving efficiency works mostly at the edges of the problem, and eventually the school will pull all of the slack out of the system.
Imagine an electronics company that has a strong history in the VCR business. A few years ago, this company noticed that their profits were beginning to slip. To address the problem, they reduced costs by improving the efficiency of VCR production. They modernized their plants, streamlined their efforts, and consequently were able to generate better margins for each VCR sold. For a few quarters, the company's profits rose. Soon, however, the pattern reversed itself and no matter how much effort was made to improve the efficiency of the production process, the company could not meet its profit targets.
The company had responded aggressively to dwindling profits, but its efforts to improve efficiency could only take it so far. The market had changed and the product they produced was no longer in demand, so the company's only hope would be to fundamentally change their work. They would need to develop new products — such as DVD players or DVRs — to meet the needs of the changed market.
Like our fictional electronics company, American schools are operating in a radically changing marketplace. Schools are being required to produce greater numbers of higher performing students. Improving educational efficiency is an absolutely necessary response to these changes, but schools that focus solely on improving efficiency will ultimately stall in their efforts to sustain growth. For schools to reach the ever increasing standards of NCLB, they must do more than push marginal students over the next NCLB threshold. Moreover, to reach a plane of sustained growth and improvement, they must develop transformational ways of working and produce fundamentally different kinds of results.
The move to stage three — analysis for sustained achievement — requires that a school adopt a core belief that all its students can perform to higher standards. While most educators will agree with this concept in principle, the challenge is establishing structures and systems that fundamentally change the practices of the school to support this philosophy. In stage three, schools combine data analysis and organizational reform to create an environment driven by an abiding belief that all students can and will meet the standards. This is a major shift in the way schools operate, and it requires careful collection and analysis of data to sustain its implementation and maintain its progress. Following are a number of strategies used by stage-three schools to deliver targeted instructional efforts for students at all levels, including the advanced and those thought incapable of reaching proficiency.
1. Focus on Individuals
Stage-two schools center their data analysis efforts and instructional interventions on fairly broad groups of students: classes, measurable sub-groups, "bubble" students, or whole tested grade levels. Typically, only those students who are performing at the margin of proficiency (and therefore can contribute most readily to improving the school's overall performance) get significant individual attention. The real, but less than comfortable, corollary of this approach is that stage-two schools give only modest attention to other groups, such as those students who are securely proficient performers and those students who are performing well below the proficiency threshold. Individualized attention is even less available for students in these groups, unless a student is eligible for Special Education services.
Stage-three schools shift the focus from groups to individual students — every single, individual student. At Jeremiah Gray and Rosa Parks elementary schools in Perry Township outside Indianapolis, principals Ann Puckett-Harpold and Gary Robinson have created structures to ensure that all students are seen and treated as individuals and not just members of a larger group. Classroom performance information is collected via an online grade book and combined with data from online formative assessments to create individualized Student Learning Profiles. Teachers, children, and parents meet quarterly to review this information and develop individualized Student Learning Agreements. These agreements — which include specific goals and actions for the student, parents, and teachers — are stored electronically and reviewed against student progress in subsequent meetings. As a result of such targeted efforts, these two schools have risen to having more than 90 percent of their students achieving above the state standard in reading.
Monthly Benchmark Review
Findings
25% of students picked "another country" (answer A) as a definition for "foreign." Although this is a correct meaning, it's not the correct meaning in this context.
Next Steps: Students
Students overall scored very high. Class average was 77.
Zachary, Crystal, Kelsey, Peter, and James all scored 90%. They will need more challenging material.
James and Emanuel scored 50 and 45 respectively. I don't believe this reflects their ability so I would have to question their attention to the test. Prior to the next test, we will have a conversation about how important it is for them to take benchmarks seriously.
Next Steps: Curriculum
Review context clues and the need for students to go back and find the word in context before choosing an answer, even if they feel they know what the word means.
Stage-three schools' accountability systems include regular teacher analysis of performance data and specific action plans at both the individual and classroom levels.
2. Personalized Learning
Stage-three schools understand that traditional school structures related to time and class organization limit their ability to address the individual needs of the students. While students are still organized into classes, schools committed to sustained improvement combine regular data collection and analysis with flexible grouping structures to personalize learning opportunities for students. At Thomas A. Edison Charter School in Wilmington, Delaware, students have a dedicated 90-minute reading period every morning. Students get additional individual attention because reading groups are smaller than normal classes and all teachers and many administrators are assigned to them. The groups are multi-age and levelled according to the learning needs of the students. Most importantly, the groups are flexible, and membership is adjusted based on the results of frequent assessments. The school uses an electronic version of the Scholastic Reading Inventory to measure reading performance at least every eight weeks. The results are analyzed by teachers, and students are regrouped to maximize the personalization of their learning experience.
3. Look at "Added Value"
NCLB's targets are based on a percentage of students in a particular cohort reaching a defined level of proficiency. Through 2014, incremental improvement is required in this percentage. Interestingly, these requirements do not require individual students to demonstrate improved performance over time. As long as a student scores above the proficiency threshold, her performance counts toward AYP. In fact, a student could score significantly lower in the 8th grade compared to the 5th grade, but not hurt her school's overall performance as long as she stayed above the proficiency level. Conversely, a student could make significant gains in performance from 3rd grade to 5th grade, but still not contribute to her school's AYP goal because her 5th-grade performance was still slightly proficient. This situation leads many stage-two schools to maintain their focus mostly on the students near the proficiency threshold and give little individual attention to those well above or below the cut-off point.
Stage-three schools look beyond the requirements of NCLB and track individual student performance over time with a goal of getting improvement from all students over the full time they are in the school. Some American schools have taken an important lesson from the United Kingdom, where the Department for Education and Skills includes "added value" data when reporting high-stakes assessments scores. Added value is reported in terms of the average gain in performance for same students from one high-stakes assessment to the next. This gain is then indexed against expected performance improvement targets. Schools are judged on same student improvement rather than increasing the number of students over a particular threshold. Consequently, the steady, continuous improvement of all students is required for the school to meet its performance targets.
4. Systems of Accountability
In stage-three schools, school leaders establish structures of accountability that ensure teachers regularly analyze student performance data, talk about it in functional units, and enact specific action plans at the classroom, team, grade, and school level. Sharif El-Mekki, principal of Anna T. Shaw Middle School in Philadelphia, conducts monthly ABCs or Administrative Benchmarks Conferences with his teachers. At these conferences, each teacher presents an analysis of the month's formative assessment data, a summary of their findings, and an action plan for addressing the student learning needs identified. By asking guiding questions about general trends and specific students, El-Mekki coaches his teachers through the difficult task of customizing teaching to meet the learning needs of all students. Following the ABCs, assistant principals and team leaders meet with groups of teachers to tackle thorny problems, share best practices, and collaborate to address general findings. The result is a school where every teacher is highly conversant about specific performance details of their students.
Challenges
Moving a school from stage two to stage three requires transformational change. Stage-three schools must not only adopt new core beliefs, they must be willing to radically restructure time and teams based on frequent data collection and analysis. This endeavor creates extraordinary organizational challenges. How can teachers hope to provide customized instruction to dozens or even hundreds of students? Where do you find the time to do additional assessments, analyze the data, develop action plans, and implement changes? While there are no instant solutions to these challenges and getting to stage three requires sustained hard work, the following suggestions can help to build the capacity to support this radical change.
- All Hands on Deck
Stage-three schools use all available resources to personalize learning. All qualified employees run reading groups. Students provide peer tutoring in structured sessions. Creative arts and physical education teachers are trained in data analysis and support the school's goals through integrated learning units. Pastoral advisors monitor and report on Student Learning Agreements. In short, no resource is wasted and all are focused on the same objective: helping all students achieve higher standards. - Integrated Technology Systems
Select student information systems, formative assessment systems, and student performance tracking tools that easily integrate. Teachers lose focus and efficiency when they have to learn and juggle several data systems simultaneously. - Students Take Ownership
The ultimate key in sustained improvement is getting the students to share in the responsibility for their own learning. When students have the skills to analyze their own formative assessment data, for example, they have the tools to take ownership of their own learning. This means training students to analyze data on their own performance. Specifically, they need to learn how to look beyond top-level scores to strand and skill-level details of their performance. Then, they must be given opportunities to practice setting goals and creating action plans to guide their own learning. When this is accomplished, the capacity of the organization to grow is nearly unlimited, and stage three of data utilization is fully embedded.
Todd McIntire is a vice president at Edison Schools, Inc. He is currently involved in developing Edison's new division in the United Kingdom.
Indiana Monthly Edison Benchmark Review
High-Stakes Test—ISTEP
Strand/Skill Analysis—
Patterns and Anomalies
- Which strands were overall weaknesses for your class? (Strands where the majority of the questions are 60% and lower)
- Of these strands, which ones did you focus on during the previous month of instruction? (Look back at lesson plans)
- Of these strands, which ones will you focus on during the next month of instruction? (Look forward at scope and sequence)
- Were there particular skills within weak strands in which students did perform well? What were they and what about those questions allowed students to do well?
- Which strands were overall strengths for your class? (Strands where the majority of the questions are 65% or higher)
- Of these strands, which ones did you focus on during the previous month of instruction? (Look back at lesson plans)
- Of these strands, which ones will you focus on during the next month of instruction? (Look forward at your scope and sequence)
- Were there particular skills within strong strands in which students did not perform well? What were they and what about those questions caused students to do poorly?
Planning for Instruction
- Looking forward to your next month of instruction, which skills will you emphasize with your students based on your data analysis and scope and sequence?
- What particular vocabulary words or phrasing of questions will you now use in your teaching to ensure student success the next time they encounter those words/ questions on the Benchmarks?
Asking the right questions about data gleaned from high stakes tests is central to spotting the performance trends, strengths, and weaknesses that will help teachers map a customized instructional course.
Data Analysis and Assessment Tools
The following products allow schools and districts to gather, analyze, and make the most of their data.
Apple's PowerSchool is a Web-based Student Information System that delivers real-time information via the Internet. The new PowerSchool Premiere was developed for schools that need a more scalable SQL database. Its Open Database Connectivity (ODBC) allows for integration with other applications, and its interoperability eliminates the need to enter duplicate data in multiple applications. www.apple.com/education/powerschool
Star Insight from Century Consultants is a decision-support service that provides tools for visually manipulating, graphing, and filtering large amounts of data on a desktop. www.centuryltd.com
Chancery's Student Management System is an SIF-compliant (Schools Interoperability Framework) application built on open standards that allow for easier integration of data with other applications. The system provides access to information based on the user's role, ensuring that confidential data is only seen by authorized users. www.chancery.com
Designed for non-technical users, ConfluentEDU automatically pulls data from disaggregate sources and presents the information in a graphical interface, allowing the user to visualize data through graphs and charts and generate basic reports. www.confluenttech.com/education.htm
Cognos Inc.'s solutions for K-12 education allow districts to analyze data to measure student achievement, track the educational qualifications of teachers in the district, and tie budgets to outcomes and results. www.cognos.com
The Exceptional Student Education module, part of EDmin.com's INFORM suite of applications, facilitates the process of creating Individual Education Plans. The Web-based system lets multiple people access the forms simultaneously, so they can work on different parts of the plan at the same time. www.edmin.com
EdSmart's data warehouse system and data-analysis tools, EdDiscover and EdExplore, allow for multiple views of data; charting, graphing, and drilling down; generating reports; and exporting to HTML and PDF formats. EdDiscover provides direct data-cube analysis while EdExplore provides a relational database approach to analysis. www.edsmartinc.com
EdSoft Assessment provides online and offline benchmark assessment, objective-based assessment, and reporting that can be organized by question, objective, or answer distribution. Administrators can view trends over entire districts and also drill down to monitor individual student performance. www.ed-soft.com
The eScholar Data Warehouse is a pre-built system that integrates a wide range of data from a number of sources to allow for easier reporting. It includes a data model, ETL (extract, transform and load) software, security structure, and standard reports. It supports Oracle, MS SQL Server, and IBM DB2 platforms. www.escholar.com
EduSoft's Web-based assessment program helps districts track student progress towards state exams, district benchmarks, and classroom tests in all core K-12 academic subjects. www.edusoft.com
Educational Testing Service's IDMS (Instructional Data Management System) uses a Web-based delivery system to send data directly to teachers' desktops. The Standard's-based approach allows educators to track student progress over time and plan instruction based on student strengths and weaknesses. www.etspulliam.org
Harcourt's Stanford Learning First measures student progress in reading and mathematics for grades 3-8. Instructional content is tailored for each state's standards, benchmark assessments are modeled on state tests, and professional development is included. The Web-based system provides online and paper-based assessments and delivers real-time results. www.harcourtassessment.com
IBM Insight at School allows users to analyze data from different sources and on different platforms, provides benchmarks against state and national standards, and generates customizable reports. www.ibm.com
Data Exploration: A Journey to Better Teaching and Learning, a video from Learning Point Associates, profiles two schools that have successfully used data analysis to improve student performance. The video is supplemented by a booklet that provides resources for data analysis and activities for educators. www2.learningpt.org
The Quality School Portfolio, a Web-based data analysis tool, was developed at the National Center for Research on Evaluation, Standards, and Student Testing at UCLA. The software is free, but districts or schools that wish to use it must first join the QSP User Group. qsp.cse.ucla.edu
Data Point from the National Study of School Evaluation is a Web-based data collection and management tool that incorporates statistical analysis tools such as t-tests and chi-square tests. It provides three options for developing a student improvement plan: the NSSE framework, a customized template based on state district requirements, or a customized template based on another education agency. www.nsse.org
NPower New York's Online Survey Tools PDF provides information on how survey tools work and how to evaluate them, listing several recommended choices for nonprofits and schools. www.npowerny.org/tools/online+survey+tools.pdf
OtisEd AssessMart is an assessment analysis platform that can be integrated with a district's existing student information system and customized for specific reporting needs. OtisEd School Data Bus Pro is a Web-based platform for data extraction, cleansing, and reporting. www.otised.com
The Pearson School Assessment series includes Pearson Benchmark, a Web-based assessment tool that allows for multiple measures of student performance against standards, test-creation and administration tools, and reporting capabilities. It also includes Pearson PASeries (Progress Assessment), which provides growth forecasting and progress measurements towards state standards in reading and math. www.pearsoned.com
PLATO Data Management & Analytics includes both data warehousing and synchronization software, working with a district's SIS to provide ongoing uploading of data to a central warehouse. It compares individual, classroom, and school data against state and national standards. www.plato.com
Sagebrush Analytics allows users to access multiple data sources at the school and district levels, design and run queries of this data on the fly, and publish and print reports. www.sagebrushcorp.com
Scantron's Achievement series is a content-neutral, Web-based assessment platform that allows users to develop, manage, and administer tests. Their Performance Series contains a diagnostic test that allows educators to assess student proficiency levels for class placement and to predict performance on high-stakes tests. www.scantron.com
SCHOLARinc's Scholar Suite Web-based assessment and data management system allows for the aggregation and disaggregation of data at the classroom, grade, school, district, and state level. It handles assessments at the classroom, school, or district level, and it allows for the generation of individual student profiles. www.scholarinc.com
SchoolNet's Web-based Instructional Management Solutions (IMS) are modular and content-neutral systems that allow districts to analyze data, assess performance, and individualize instruction. They allow for the creation and dissemination of district-wide or school-specific reports for analyzing AYP against NCLB targets. www.schoolnet.com
STI's Student Information Systems contain modular components, so schools and districts can only purchase the ones they need. The administrative data-management components are STI Office, STI Classroom, STI District, and STI State. STI Assessment integrates with an existing SIS, provides benchmarking throughout the year, and generates individual and aggregate reports. www.sti-k12.com
Sunguard's PerformancePLUS provides data extraction, aggregation, and analytic tools. Its out-of-the-box data warehousing solution for districts works with pre-existing information systems. www.pentamation.com
TetraData's Achievement Series, offered through a partnership with Scantron, is a Web-based test delivery and management system with real-time results and reporting capabilities. Collaboration features allow users to meet online to create or review tests. Results can be aggregated and disaggregated to meet state standards. www.tetradata.com
ThinkLink Learning's Predictive Assessment System uses a Continuous Assessment Model that measures student progress in the midst of instruction, allowing teachers to adjust instruction based on the data generated. The model is mapped to standards for Alabama, Kentucky, Mississippi, Tennessee, and West Virginia, but they are also applicable to other states. www.thinklinklearning.com
AMS Enterprise from TurnLeaf is a data collection and dissemination tool that works with any data source; allows administrators to identify achievement gaps and formulate AYP; and allows teachers to evaluate student progress and implement differentiated instruction. www.turnleaf.com
