When New York City leaders passed comprehensive open data legislation in 2012, they set a high bar for open information and government transparency. Not only would the city open its data: it would open it all. Yesterday, the Open Data Law turned five years old. New York City’s open data team is celebrating with a suite of new announcements, including a new website, a new data literacy pilot, and the city’s inaugural Open Data Week.
Each of these developments was the direct result of user research, conducted to ensure that NYC Open Data reflects the unique needs and character of New Yorkers and their city. In that sense, these updates reflect the century-long local tradition of civil society protections from which the initiative emerges. As NYC Open Data steps into the future, it’s worth looking back at how it became what it is today.
OPEN DATA’S ANTECEDENTS
Before government information was stored in databases, the notion that information created in the public interest belongs to the public was fueling the development of modern urban civics. The idea that government activities should be “open” traces, in part, back to New York City in the early 20th century, when Progressive Era reformers fought for legal remedies to the corrupt ward politics and favors-trading power brokerage characterized by the Tammany Hall political machine. The resulting accountability measures, which included the creation of the modern civil service and rules on contract awards, aimed to build government transparency by ensuring that public officials did, in fact, act in the public interest. Half a century later, as public records legislation swept across the country, the public’s right to information grew from good policy to a legislated guarantee.In 1993, when the World Wide Web was still in its infancy, the recently-formed NYC Commission on Public Information and Communication (COPIC) published the first-of-its-kind “Public Data Directory.” The Directory inventoried the digital databases maintained by city agencies, and was created in direct response to the evolving tools and technologies that city government used to maintain information that was, in its words, “required by law to be accessible to the public.”
Throughout the early 2000s, this philosophy of public digital information gradually moved toward the mainstream. In 2007, on the opposite side of the country, Tim O’Reilly – the popularizer of such technology concepts as “Web 2.0” and “open source” – convened 30 good-government advocates, information law experts, and corporate technologists in Sebastopol, California. Their charge: reach consensus on a nationwide agenda for public information in the digital age. By the end of the convention, the group had issued “Eight Principles ofOpen Government Data,” which stipulated that government data should be “complete, primary, timely, accessible, machine processable, non-discriminatory, non-proprietary, and license-free.”
Many public institutions had a practice of disclosing data long before 2007. But the Sebastopol conference represented a unique historical moment when three related but distinct political and technological movements converged. Portions of the “sunshine” rhetoric of good government advocacy, open access methodology of scientific research, and reusability principles of open source software development merged into what we now know as “open data.” In the years that followed, these eight provisions echoed across the nation in the dozens of policy decisions at the state and local level that ultimately swelled into a nationwide movement.
At the federal level, Barack Obama issued the “Memorandum on Transparency and Open Government” on the first day of his presidency in 2009. The memo instructed his administration to devise an implementation strategy for more “transparent, participatory, and collaborative government.” The result – the Open Government Directive – established the federal government’s first open data policy.
That same year in New York City, the Mayor’s Office under Michael Bloomberg partnered with the Economic Development Corporation (EDC) and the Department of Information Technology and Telecommunication (DoITT) to launch the first NYC BigApps competition. BigApps, which was designed to spur community data science and app development using open data, represented an attitude that city data could be not just a way to disclose information, but an opportunity for direct engagement with the community. For the occasion, the city published 170 datasets, including information on citywide events, property sales, recreational facilities, and restaurant inspections.
NYC Open Data Timeline, adapted from opengovtimeline.com
IN NYC, OPEN DATA IS THE LAW
On the legislative side of city government, City Councilmember Gale Brewer (now the Borough President of Manhattan) was rallying support for a law that would codify open data requirements for all city agencies. After she introduced the legislation to City Council in February 2010, New Yorkers across the pillars of civil society joined Brewer’s fight to pass the bill, which ultimately became Local Law 11 of 2012 when signed by Bloomberg on March 7, 2012.More commonly known as the “Open Data Law,” this legislation draws from the 1974 New York State Freedom of Information Law (FOIL), but deploys the Open Government Data principles to push a step further. Rather than responding to specific requests for public information on an ad hoc basis, the city required itself to publish data – all data – outright.
In mid-2015, three years and 1,300 datasets into the program, the de Blasio administration committed to opening its data for all. Among the oft-cited benefits of open data is its potential to equip people with the knowledge to improve their lives. With Open Data for All, that potential became the program’s principal imperative: “New York City is dedicated to putting this power [to create opportunity and solve problems] into the hands of all New Yorkers,” the document states.
Open Data for All, July 2015
According to Amen Ra Mashariki, NYC Chief Analytics Officer, Director of the Mayor’s Office of Data Analytics (MODA), and leader of the NYC Open Data initiative, the fact that open data is legislated underlies the program’s focus on equity. In his words: “Our vision of Open Data for All — the idea that Open Data belongs to New Yorkers — naturally followed from the fact that elected officials, via the constituents they represent, were so committed to this idea that they added it to the administrative code.”
The Open Data Law mandates that the city release all of its public data online by the end of 2018. To accomplish this, MODA and DoITT require each city agency to submit annual timelines for publishing eligible datasets. This method resulted in the publication of more than 1600 datasets from 60+ city agencies, but recently, civic advocates, lawmakers, and the de Blasio administration felt that the city could do even better. Between November 2015 and January 2016, City Council passed and Mayor de Blasio signed a package of amendments to the Open Data Law that provided additional mechanisms to help agencies ensure that all qualifying datasets are disclosed. These new laws include:
- A Freedom of Information Law (FOIL) reporting requirement, which requires city agencies to review their FOIL disclosures for datasets that could be published publicly.
- A public nominations provision, which guarantees timely and thorough decisions on all public requests for new datasets.
- An “Examinations and Verifications” process, which requires MODA to examine three agencies each year to verify that all eligible datasets have been disclosed.
ENACTING AN OPEN DATA CULTURE
The “sunlight” rhetoric of government transparency, typified by the pioneering work of the Sunlight Foundation, operates through the architectural metaphor of the window. The thinking goes like this: by opening up a window into the walls of a government institution, sunlight shines through, shedding light on government activities and acting as a “disinfectant” to potential corruption.The promise of open data, though, goes further. It represents information that allows people to take action on their own behalf. Digital technology shifts government transparency from the optical metaphor of looking through a window to the navigational metaphor of entering through a portal.
Screenshot from nyc.gov/opendata
The city’s new website and pilot workshop on data analysis skills, both announced on March 7, lower the barrier to entry. The website features guides for data novices and an easier way to get in touch with the Open Data team. It also includes a framework, which borrows from human-centered design methodologies, and NYC’s Digital Playbook, for ensuring that every decision made by the open data team is done with New Yorkers in mind. These six “Open Data values" are:
- Start with users
- Treat the publication of the dataset as its debut
- Encourage purposeful and easy engagement
- Empower agencies
- Integrate open data into citywide processes
- Learn, test, standardize – and learn again
OPEN DATA IS A LOCAL AFFAIR
The open data legislation in New York City represents an effective way to sustain an open data program through political administrations. But Open Data for All, and the developments it inspired, has made open data valuable to actual New Yorkers. Open data doesn’t exist in NYC just because it’s a best practice. Five years after open data became law, the program thrives because it still enjoys support and pressure from a variety of community stakeholders.Take, for instance, the New York City Transparency Working Group. This coalition of local civic organizations represents among the strongest collective voices driving more and better open data for New York City. The group is co-chaired by John Kaehny, Executive Director of good government group Reinvent Albany, and Gene Russianoff, who in 1988 helped drive campaign finance laws in NYC that have been replicated around the nation. Its member institutions include the League of Women Voters, which has been active locally in NYC since its inception on the eve of women’s suffrage in 1920, and BetaNYC, a civic technology and open government group that has been connecting New Yorkers with city data since before the existence of the open data law.
Whether five or fifty years ago, the city’s information and data disclosure practices have been deeply rooted in New York City’s unique character and local political history. While open data programs in many municipalities share commonsense principles, policy verbiage, and portal technologies, open data in New York City is a profoundly local affair.
This post was updated at 8:45 am on March 10 to reflect then-NYC Councilmember Gale Brewer’s contributions to the NYC Open Data Law and to clarify the membership of the NYC Transparency Working Group.
