Former Federal CIO Vivek Kundra recently said, “this notion of thinking about data in a structured, relational database is dead, some of the most valuable information is going to live in video, blogs, and audio, and it is going to be unstructured inherently." Modern, 21st century tools have evolved to tackle unstructured information, yet a huge majority of federal organizations continue to try and use relational databases to solve modern information challenges. The question is this - what is keeping us from realizing the full potential of our data, structured and unstructured, which is (among other things) a vital necessity to national security?
Oracle and other legacy relational database systems are so pervasive in government today that when a new federal contract is awarded, these systems are unilaterally brought in as well – often ordered alongside the carpet, furniture, and wall paint. Largely, it's a result of inertia and the mistaken belief that because these systems have been around for so long, and so many organizations use them, they must be the right solution. And since many government agencies have enterprise licenses with these relational database vendors, it appears they are getting something for 'free', when in fact the frequent misapplication of relational technology to the wrong data challenge is costing far more in development and operations/maintenance costs. Fully two-thirds of any system lifecycle cost is in operations and maintenance. If you were to build the Golden Gate Bridge with wood versus steel, you can imagine the constant manpower and material costs of maintaining that bridge over time. The same principle applies to building IT systems. Using the wrong 'tools and materials' will bankrupt your IT budget by driving O&M costs sky high, putting serious pressure on budget for innovation and new development.
Relationertia – a Measurable Tax on National Productivity
This de facto implementation of legacy relational databases across government organizations, or "Relationertia", has the effect of imposing a measureable tax on national productivity.
These systems are no longer equipped to deal with the type of information governments must leverage. Today, more than 80 percent of information is unstructured – that is, data that doesn't fit into the traditional columns and rows of a relational database. Given the hardware, power, and people costs required to scale a database to deal with this type of information, it's no wonder that government database budgets have reached billions annually.
There are newer, more modern approaches that would allow government organizations to break free of this Relationertia tax and put their budgets back on track, while at the same time unlocking new innovations. For example, there is a wave of next-generation databases that are specifically designed to manage and leverage unstructured data in the modern era. And these systems can be up and running in weeks, cutting costs by up to 95 percent. At a time when recent budget battles have resulted in the Office of Management and Budget (OMB) asking federal agencies to cut 10 percent out of their FY13 budgets and to identify projects that demonstrate efficiencies, public agencies need to invest in technologies that allow them to fulfill their mission and spend the right dollars to save money.
Unstructured Information – Get Control now or Pay the Price
The growing amount of unstructured data that government organizations must manage is at a critical stage. And the government is not spared from this explosion of unstructured information – the massive volume of mission-critical government information has serious implications on everything from the budget to national security. This means selecting and implementing the right technology is more important than ever.
Outdated yet entrenched relational databases can sometimes be made to work for the 80 percent of data that now qualifies as unstructured, but they are so inefficient that projects can quickly escalate into the $100,000,000 range. In many cases, organizations will be forced to keep adding zeros as projects take years to complete or are simply abandoned because the customer tried to use the wrong tool for the job – more wasted money that is in effect a tax on all of us and has a measureable impact on the GDP.
Breaking Free of the Tax on Productivity
As an alternative to the costly proposition of relational databases, unstructured databases can cut implementation times from years to months, IT staffs from dozens to as little as one person, hardware requirements by up to ninety-five percent, maintenance costs by seventy percent, and total budget from an astounding nine or ten figures to just fractions of those numbers.
Organizations as diverse as the FAA, Department of State, Library of Congress, The Department of Agriculture, the Department of Transportation, the National Archives, and the US Army have implemented unstructured databases in a variety of functions, including search and discovery, open source intelligence, metadata catalogs, dynamic information delivery, and analytics, among others. These new vendors are also making implementation and ongoing maintenance much more efficient by providing an information centric architecture that makes it easy to develop new applications, mix and match content in new ways, and enable new delivery methods.
Government organizations suffering from Relationertia and the public that feels that impact in what amounts to a federal tax on productivity have two huge challenges: Exact benefits from the insight that this enormous mass of unstructured data could bring – from the preservation of historical documents to new insights on the war on terror – and doing so under ever-tightening budgets.
Enter the new wave of databases for unstructured data, which are ready to help organizations face these new challenges head on – a welcome option in a time of national debt and the continued budget crisis.
Randall Jackson is the Vice President of Public Sector at MarkLogic.
