The Perils of Big Data

There's a name that will be on a lot of tongues at SXSWedu. That name is inBloom – and it will be invoked with a worrying shudder.

Launched in 2011, the nonprofit was a serious and well-connected endeavor. Originally called the Shared Learning Collaborative, funded by the Gates Foundation and the Carnegie Corporation to the tune of $100 million, and with education partners in nine states, it was intended to give educators and administrators easy access to student data. Districts and states could input around 400 different data fields: test scores, addresses, disciplinary records, disabilities, family members, even if a student wore colored lenses while taking a test. All that data, instantly accessible, with inBloom at the forefront of this education revolution.

Within two years, inBloom was gone. Parents, educators, and privacy experts had raised so many red flags about a private entity holding that much student information that it closed its doors. Even a massive last-ditch investment in additional encryption did nothing to placate those fears. When it finally imploded, CEO Iwan Streichenberger issued a mournful statement that "the unavailability of this technology is a real missed opportunity for teachers and school districts seeking to improve student learning."

Schools collect, hold, and attempt to analyze vast data sets about their students. So here's the big question: What are we collecting this data for?

The headline reasoning is to get kids through school and, increasingly, to college: Analyze the data, and you can see which kids need what assistance, or which programs work where. However, the current culture and policy looks at headline data, and nothing more. There's a simplistic, reductionist approach, a pass/fail grade for students: It's big data a mile wide, but an inch deep, and rather than helping, it has seemingly caused more problems. Patrick Briggs, Texas state director for AVID, explained, "You look at [No Child Left Behind (NCLB)]; when it was signed into law, it was a great goal. It said that by 2014, there would be no achievement gap. But in many places, the gap has gotten wider."

Defining Smart Goals

AVID (Advance­ment Via Individual Determination) is one of the horde of companies and nonprofits that claim they can help districts bridge that statistical gap. For Briggs, a former teacher with 25 years as a professional educator, the real issue is that while all this data is being gathered, the metrics aren't measuring anything useful. He said, "The educator has the privilege of having a child for 180 days, and I'm going to judge that educational system based on one test given to every child on one day of the year. No other profession does that. We don't go into a law firm on one day of the year and say, 'All of you who are here today and aren't absent, here's this survey,' and say, 'Here's the results for the whole year.'"

Michelle Riconscente of educational gaming software firm GlassLab argues that the concentration on high-stakes testing, and the data that it collects, actually is done with good intentions. She said, "We care about our kids doing well, and knowing that they're being educated well is important. Because of that, we're looking for information that will assure us that's happening. The question then is, how do you go about doing that?"

However, as is always the way, those good intentions have paved a road to hell – or, as Riconscente diplomatically puts it, "the unintended consequences of this approach have been less than ideal." She continues, "So we see a lot of time and expense on testing, instead of instruction, and research shows that curriculum will narrow to specifically meet the content that's on the test." It's not that testing is new, or inherently bad: She notes that education has always contained barometers and gauges to allow teachers to measure what their students do and don't know, and the skills they have and haven't acquired. She said, "Good learning means good assessments."

The darkest shadow of that data is the school accountability system. It's a simple model: The state sets certain metrics and tells districts and schools that as long as a certain percentage of students pass a test by a certain percentage, then those districts and schools are successful. This creates what Briggs calls "an acceptable level of failure," and he calls that unacceptable. "The system basically says, 'Reach this certain level and we'll give you a nice label.' ... Here's what we're really saying as a country: If you need 80 percent of your students to pass on a given day in the school year, then it's OK for 20 percent of students to fail."

That culture seeps down to the individual educator, and Briggs admits that, when he was a young teacher, he let that paradigm run his classroom. "Shame on me. When I was 21, I didn't know how to make all the kids in my class successful, so when 10 percent of my class failed, I slept at night." However, he's equally critical of how the system grades teachers. Using that simple marker doesn't discern what educators are really achieving in their classrooms. "It doesn't measure that I stayed up marking 120 essays so I could give students feedback, or that I paid $1,000 of my own money to have food for kids that haven't eaten."

Even worse, in true stable-door/horse-bolting fashion, Texas educators only get their rankings in the summer. According to Riconscente, the prevalence of research shows that, the more timely the feedback, the better a student learns; by extrapolation, the culture of a single metric issued on a single date is basically useless. In this case, it's the teachers who are hamstrung. Briggs said, "What good does it do me to get the results after my kids are gone?"

Don't get the impression that Briggs is saying throw out the tests and ignore all metrics. What we need, he argues, are "smart goals."

Knowledge vs. Security

Noelle Ellerson knows education data. As associate executive director of policy & advocacy for the American Association of School Administrators, part of her job is to allay concerns about what is being collected and how it is being stored, especially in the wake of the inBloom debacle. She said, "What's in a student's electronic record now? Grades, behavior incident, home phone number, street address. What was in a student's record in a filing cabinet in 1950? Grades, behavior incident, home phone number, street address." However, in the wake of data-driven legislation like [NCLB], districts are dealing with a greater volume of data than ever. Second, now it is all electronic. She said, "The idea of online data brings about the shadow of Big Brother. News flash: Your child's data has always been out there, and it's been safe and secure and right 99 times out of 100."

Post-inBloom, it's that security issue that has observers worried. For decades, secure school records meant having a better lock on the principal's filing cabinet. That all changed in 1974 with the passage of the Family Educational Rights and Privacy Act (FERPA), which created specific restrictions about who had access to student files, and why and when they could use them. For a data law that predates the World Wide Web by 17 years, "it's done a good job," said Eller­son. "I don't think it's perfect, but the fact that FERPA can be 40 years old and not need much overhaul, I think is a testament."

However, FERPA has its limits, and earlier this year President Obama announced a wave of new online legislation. That includes a new Student Digital Privacy Act (SDPA), based on California's Student Online Personal Information Protection Act. There are other student data privacy laws in other states, but the Golden State's version puts a greater responsibility – and more restrictions – on third-party vendors providing services to schools and school districts. In a preview speech to the Federal Trade Commission in January, Obama said, "Data collected on students in the classroom should only be used for educational purposes – to teach our children, not to market to our children."

That's what's changed since FERPA passed. Four decades ago, a lot of school districts didn't even have computers; now, like everyone else, they are dependent on electronics, and that means third-party vendors – server companies, technical support, software providers, testing services. That raised all kinds of legal questions. Will they simply store the data, or will they have access to it? How deeply involved will they be in the day-to-day educational process? Will they be allowed to share that information with others? How will school districts, with little to no technical expertise, know if the data is being mishandled? That's something Texas is already facing in other fields: After the Health and Human Services Com­mis­sion canceled a Medicaid contract with Xerox, the company simply retained about 2 million pages of patient files. As a result, the firm and the state could face millions in federal fines for violating the Health Insur­ance Portability and Account­ability Act.

There is also the fear of privatizing education. Publishing behemoth Pearson parlayed its control of the school book industry into massive influence over education policy. What's to stop a major cloud-server provider like Amazon from doing the same thing?

Ellerson's biggest concern is that, post-inBloom, any pending legislation driven solely by those concerns could be counterproductive. She supports ideas like requiring third-party vendors to destroy data when a contract expires. Yet she fears overly constraining the legitimate use of data by those same vendors to help teachers and students. "You're shutting the door before the mission has started." She also doesn't want legislation to add more expense to struggling districts, but instead find ways to proactively encourage best practices – not least making systems as secure as possible. She said, "That could include incentivizing vendors to invest in data security, or incentivizing school districts to use vendors with better security." So far, there is no draft of the SDPA to examine, so Ellerson is also concerned that it not be so over-defined as to become irrelevant. She said, "No one can anticipate what a student data record will look like in 30 years, never mind 10."

That there will be records is a given, and it's arguably more for the benefit of policymakers and administrators than students. Bret Cohen, an expert in data privacy law with D.C.-based attorneys Hogan Lovells, explained, "To figure out how well educational policy is working, you have to collect information about students." In the post-NCLB data explosion, he argues that school districts are too small and too poorly funded to try to handle their data management all internally, and they probably shouldn't even try. He said, "Even major corporations are turning to third-party vendors to secure their information. So if the most profitable businesses in the world can't do it better than someone who specializes in this, then a school has no chance."

Learning From inBloom

Those vendors with an eye to the lucrative educational market are learning from inBloom's collapse, especially in avoiding its blunders. Cohen says inBloom's first major mistake was in its legal documents, "the ones they post on their website that most people don't read. The way they had worded things was, for lack of a better term, tone deaf." They were larded with standard legalese, saying that inBloom did not control nor was it responsible for the data. "What they were intended to say was that the school was the ultimate entity with control, which is ultimately a good thing." How­ever, parents in client districts got concerned about an outside entity holding their kids' data, which became a political issue, which led to states canceling contracts, which put inBloom in an untenable position.

Whether pending legislation satisfies privacy experts, companies, or both, it should at least clarify the legal landscape for parents, school districts, and future inBlooms. However, the flip side is the fear of less scrupulous uses. It seems like every other day, there's another data breach or hacking scandal. Think someone getting hold of your credit card number is bad? Imagine someone getting hold of every student's disciplinary record. Worried about hackers being able to invade your privacy by turning on your laptop camera? How about a corporation getting to look at kids' medical records? Researchers and developers are still barely scratching the surface of the power of data-mining and predictive algorithms, and let's not pretend for one second that every corporation would be above commercially abusing those records. A core part of the California law bars firms from using those files to target advertising at individual students, and that's something FERPA's authors could never anticipate. For Cohen, the new legal challenge is "trying to figure out a way to allow those kinds of innovations in the classroom while protecting students against anything that might happen, based on breach or misuse of their information."

The legislative landscape is already shifting. In 2014, there were 110 new laws dealing with student data in 36 states, and most of them put more pressure on state boards of education. According to Amelia Vance, director of education and data technology for the National Association of State Boards of Education (NASBE), it's an increasingly technical demand on state boards' time, one that combines data security with opportunities for classroom innovation. "They want to find that core balance to allow more kids to achieve," she said. NASBE's aim is to promulgate best practices and case studies, as contemplated in West Virginia's Student Data Accessibility, Transparency and Account­a­bility Act in 2014. It set clear roles for the state's board of education and department of education, and defined what data can be collected and shared, and with whom. Just as importantly, it defined what data cannot be collected and stored – for example, political and religious affiliations, or sexual orientation. The West Virginia State Board of Education also adopted new policies, including rewriting the job of its chief technology officer. Vance said, "They have him work between the tech office and the acquisitions office and all the offices that a state board has, to bring people together to ask, 'How can we protect privacy in everything that we do?'"

Vance applauds states that are proactive, not just adopting better data policies, but letting people know about them. Take Colorado: The districts that had signed up for inBloom's services "took the brunt of public criticism, but the state board of education took that as a lesson." Without legislative prompting, it launched an aggressive public education campaign, including a new privacy policy website, and hosted public meetings focusing solely on student data policy. "The lesson from inBloom to everyone is that you have to, have to, have to communicate, and communication doesn't just mean putting all your policies in a 100-page PDF."

So let's say the new laws clarify everything from a legal standpoint. And let's imagine that third-party vendors can provide enhanced security and better data storage without costing schools a fortune. If it's still bad data being collected, or good data being misapplied, nothing is really fixed. Riconscente argues that education could learn from modern gaming, where massive amounts of data are accessed, stored, and analyzed in real time. There's even potential for a significant philosophical shift. High-stakes testing has put "a negative hue over failure," said Riconscente; however, in a gaming environment, mistakes become an opportunity. "When you play a game, failure is part of the process. It's not a negative. It's very much tied to how you get better. You try something, it works or it doesn't work, you respond to the feedback that you get, and you move towards the goal." If she's right, then the failure of inBloom could be the learning experience that data policymakers needed.

---

SXSWedu runs Mon.-Thu., March 9-12, at the Austin Convention Center, Downtown Hilton, and JW Marriott.