You shouldn’t be. Used properly, the numbers can improve education immensely.
First, a confession: I hate big data.
There are too many factors influencing results to show anything useful, I often protest. Results of high-stakes test scores don’t measure my abilities as a teacher. My students’ abilities and readiness to learn vary randomly from class to class and year to year, so the data is skewed and unreliable. I know more about my students from daily observation than big data can teach me. Big data is overwhelming, and since I’m untrained in statistics, I have no idea how to interpret it. Data can be used to prove whatever you want it to prove.
I suspect that many teachers and administrators feel the way I do about our increasing reliance on big data.
And yet I’m endlessly fascinated by data. I teach AP Literature, and every year I pore through my students’ scores on the AP exam. I compare my students’ performance to that of my students in previous years. I compare their performance to their grades in class and their scores on practice tests. I look at how many students improved over the course of the year and whether that improvement was reflected in the final results. And I try to figure out if I’ve really made a difference and what techniques and activities introduced in a particular year made an impact in the long run. I know that even though the data is sometimes mystifying and will never provide me with absolute certainty, it also supplies me with a good way to test my assumptions about my teaching, reflect on my practices, and strategize about improvements I might make.
Big data also furnishes important information about individual students, as well as the strengths and weaknesses of particular schools and districts. In the future, we’ll have ever more data points available for analysis and interpretation, and so it makes sense that we all start developing effective systems for maintaining, reporting, analyzing, and responding to our data.
Our district is making great strides with our efforts to tame and use big data effectively, but, like all districts, we are learning as we go along. A key in the entire process is to create a culture where the stakeholders—teachers, administrators, counselors, parents, and students—are all invested in maintaining a continuing meaningful conversation about the data and its implications.
To this end, districts need to create a strategy to address four parts of the process: collection, reporting, analysis, and plan of action. First, it is extremely important that it is easy for teachers and administrators to capture the data. Next, stakeholders must have easy access to the data, presented in a way that allows them to quickly spot trends and patterns. Finally, frequent opportunities need to be provided where stakeholders look at the data, reflect on the implications, and then plan next-step approaches.
The first part, the collection of data, sounds easier than it actually turns out to be. A few years ago, we had a real problem in our district: we had a very expensive system to house all of our student scores and demographic data. However, few teachers were actually using the system. Despite many trainings and professional development sessions, many teachers did not take the time to import their benchmark assessments into the system nor did they run reports to show the efficacy of what we were doing. Our data was spotty and incomplete, and so not particularly useful.
Why weren’t teachers using this fancy system? Simply put, it was a lot of extra work. Teachers had to export scores on benchmark assessments from their gradebooks, follow a complex procedure in the analytics tool to receive those scores, and then import the scores into the system. Worse yet, reports were difficult to produce—one wrong selection in the report set-up, and no data would be returned. In addition—and this may sound dumb, but I know that I’m guilty too—teachers just didn’t want to remember yet another user name and password to access one more Web site.
Luckily, we had an excellent working relationship with JupiterEd, the company that supplied our gradebook system in our district a couple of years earlier. The company already had a strong student information system and had been thinking about introducing an analytics component. So we collaborated with them to produce a system that would meet not only our needs, but also the needs of other school systems across the country.
The system now in place has many advantages over what we’ve used in the past. The analytics component is integrated with each teacher’s gradebook. With this system, teachers can simply connect an assignment or assessment with the analytics module by clicking a checkbox and then choosing the right analytics template. Teachers don’t have to duplicate information, and they can name their assignments and assessments anything they choose.
At the same time, administrators receive the results of the state testing in an electronic format, which then can be imported into the system. In our district, we’re importing high school exit exam scores, annual state testing, CELDT scores, and even fitness tests. The highest priority will always be the data that’s used publicly to measure our schools’ performance against that of other schools and the data needed to measure our accreditation goals, but we have a real interest in gathering as much data as we can.
Once the scores are stored in the system, there also needs to be an easy process to present that data in clean and comprehensible reports. With our system, teachers can run a variety of reports about their students’ performance. For example, they can compare student performance on summative versus formative assessments, see how their students measure up compared with other students in the school and in the district, and compare student performance from year to year. Because our data warehouse system is part of the gradebook, teachers can compare a host of data points to performance on their own assessments.
Administrators can also view student scores and print reports that track trends and even compare schools and individual teachers. In our district, this feature has been helpful in providing support to the teachers who need to improve their methods and choose intervention programs for struggling students.
A system to warehouse and report on data will be useless if nobody looks at that data. In our district we pride ourselves on a culture where we continually strive to improve, and conversations about our data are now an important part of that process. During every collaboration—whether it is a staff development day, a department meeting, or a grade-level planning session—we spend some time looking at and discussing the data.
In order for our time working with data to be productive, all the stakeholders need to be invested in the process. While our district administrators ultimately create the game plan and give us directives, it is partly formulated as a result of collaboration with teachers and counselors.
It’s also critical for every district to recognize that there will be resistance to looking at data, but rather than steamrolling over what might be legitimate objections, it is important to address them head on, so that we all understand how our data might be useful in developing approaches to improving student achievement. We also need to address our natural confirmation bias when looking at data. Study after study shows that we pay attention to the evidence that supports our worldview, and we discount or ignore the evidence that challenges what we think we know. Often when we receive data that contradicts our views, we will find ways to explain why the data is unreliable. It is precisely at those moments that we have to resist the urge to go on the defensive and instead attempt to learn from the data.
We also need to provide everyone involved with better tools to understand the data we are receiving. When comparing scores, our new system does indicate whether differences are statistically significant. However, we all need better training and guidance on understanding the more complex issues with determining the significance of the data. In our district we haven’t completely solved this issue, but I suspect that ultimately all districts will need to create administrator positions for well-trained statisticians who would assume the role of data guru and guide.
In the meantime, we’ve found it best to position our explorations of data around questions we all want answered. All the stakeholders should start by figuring out what they need to know and then locating the right data and determining the best ways to examine that data to derive answers to our questions. How do we know if we’re effective? Are we reaching all the students? What methods work best? Tapping into the curiosity of teachers and other stakeholders (even students!) will make the data less threatening and provide us with opportunities to experiment with ways to improve. Once we develop a new plan of action, the process isn’t over. We will later look at how our new approach worked, ask new questions, and continue to refine our methods.
Because of our new abilities to gather data, we’re entering a brave new world where technology will give us great insight into student learning. It’s important, therefore, that we create systems and protocols that foster a culture that is receptive to this new knowledge.
Dr. Steven Rubenstein is the English Teacher & Technology TOSA (Teacher on Special Assignment) at the Beverly Hills Unified School District
Texas district uses real-time data to drive enrollment decisions
NWISD instructors use data from maps to manage campus populations at new and existing schools.
Challenge: Since 2003, the student population at Northwest (TX) Independent School District (NWISD) has nearly tripled, averaging 1,200 new students each year. With such rapid growth, the district had to find a way to conduct real-time data analysis for academic needs, boundary planning, capacity analysis, and resource placement. To effectively address this, school leadership sought a more effective way to map out the boundaries and allocation of resources to meet student needs.
Solution: The district selected GuideK12, geovisual analytic software that visualizes student data on an interactive map to allow for real-time analysis. “The dynamic nature of the software helps us streamline the planning processes for everything from academic needs, facilities, and boundaries to looking at wireless access points throughout the district for the next phase of our 1:1 program,” says Dr. Edward Chevallier, assistant superintendent of curriculum and instruction. “Being able to query our student data will allow faster, more effective decisions on aligning resources with the needs in the district, and [we can] better anticipate future needs to help us serve our community.”
By using GuideK12 to plan for the new building in 2015, the facilities, planning, and construction department was able to quickly and effectively map out multiple boundary scenarios. Seeing the data drastically improved the department’s ability to efficiently manage attendance zones and campus populations at new and existing sites.
How a district used data to improve student performance on assessments
Calvert County Public Schools can combine historical data with current data from more than 330district benchmarks.
Challenge: In 2004, Calvert County (MD) Public Schools wanted to provide quick and easy access to student data to support informed decision-making across the district. Specifically, administrators needed to be able to access students’ historical performance on state tests and current progress on district assessments so educators could improve teaching and learning. So the district set out to find a system that would provide an integrated solution for assessment and data management.
Solution: The district began using the Web-based Performance Matters assessment and data management system in 2004 in every school, across all grade levels. With the Web-based system, administrators can combine historical data (including the Maryland School Assessment (MSA) and High School Assessment scores) with current data from more than 330 district benchmarks administered from pre-K through 12th grade. The district also includes students’ grades, attendance, SAT scores, DIBELS scores, and other data in the system.
Since implementing Performance Matters, Calvert County elementary and middle schools have achieved steady gains on the MSA. “Before we began using Performance Matters, teachers didn’t have data that showed exactly which objectives their students were struggling with or what their misperceptions were. But it’s the individual objectives and items that make a big difference in student performance,” says says Matt Poteet, supervisor of the department of instructional and informational technology.
5 Best Practices to Create a Thriving Data-driven Culture
1 Set meaningful long- and short-term goals.
One of the first steps to creating a high-performing, data-driven culture is to set specific, meaningful year-long goals.
2 Make it easy to collect and analyze data.
After the goals are set, close the intervention cycle by measuring student growth.
3 Uncover and address the causes of problems.
Making observations about the data is often the first step in data analysis. One way to do this is with sentence starters, such as:
• I observed that…
• Some patterns and trends I noticed were…
• I am most interested in…
These sentence starters help teachers stay focused on what is happening instead of why it is happening, and build capacity to make insightful observations.
4 Allot sacred time for action planning.
A key part of developing a data cycle that works is ensuring teachers have time designated specifically for analysis and action planning.
5 Create a culture of collaboration, not compliance.
In addition to weekly data meetings, teachers can use data in grade level collaborative meetings to better address student needs and share best practices across a grade level or subject area. School leaders can also use data to identify areas for staff coaching and professional development.
Karina DiGirolamo is the Director of Curriculum and Instruction for Equitas Academy Elementary School in Los Angeles. Her school uses a Web-based system called Kickboard (www.kickboardforteachers.com).
