They had gathered on a Saturday for final testing of the state's health insurance exchange. More than a website, it was a gateway to a new era of better, more affordable, more democratic health care.
But their high hopes turned to shock and alarm when the website didn't work. It crashed, it flashed repeated error messages, it couldn't process the most routine insurance enrollment scenario.
A gray-faced crew from Oracle Corp., the state's primary technology contractor, looked on helplessly.
"Shut it down," said Rocky King, Cover Oregon's executive director. "Let's not waste anybody else's time here."
It was just three days before the site was scheduled to go live.
More than two months later, the exchange continues to stymie the efforts of an army of software experts. Still plagued by as many as 40 critical programming defects -- any one of which could prevent the site from operating -- a functioning exchange remains weeks, perhaps months, away.
Cover Oregon leaders, meanwhile, have swung their attention away from the website to processing paper applications by hand, hiring more than 400 reinforcements.
How did this happen? How did Oregon, the darling of health care reformers and the recipient of more federal exchange money than all but two other states, go from health care pioneer to national laggard?
Dozens of interviews and the review of hundreds of documents and internal emails by The Oregonian reveal the roots of Oregon's headline-grabbing fiasco run deep:
- Despite a consultant's warnings, the exchange's information technology project disregarded basic cost and management controls until a year into the work -- until, arguably, it was too late.
- The state entered into time and materials contracts with its contractors with no fixed price, reducing the state's leverage.
- State leaders scrapped a plan to hire an IT general contractor to manage the work, choosing to oversee the project itself despite a consultant's finding that it lacked the expertise to do so.
- Oregon made an enormous and high-risk bet on California-based Oracle, handing over the lion's share of the developmental responsibility to the giant software firm and paying it to date $90 million. In return, the state got substandard software code, repeated broken promises and perhaps the least functional exchange site in the country.
The deadline to create a functioning exchange was always tight. Federal requirements repeatedly changed or were issued at the last minute. And the project was no simple task, requiring a site that interacted securely with state and federal agencies, tribes and a dozen insurance carriers.
But questionable management moves by the state and Cover Oregon also played an undeniable role.
Oracle Corp.
In November -- a full month after missing the exchange's Oct. 1 launch date -- Oracle executives reassured Oregon it was sending in some of the company's brightest stars to rescue the glitch-plagued exchange.
State officials were less than impressed. They'd heard it before.
It was the third promise from Oracle in five months that it would fix the problems with an infusion of new talent. By December, the original Oracle team of about 40 had swelled to 176, each of them billing between $177 and $374 per hour.
In response to the federal Affordable Care Act, Oregon's exchange was intended to be a one-stop shopping website for insurance consumers and small businesses, helping them enroll in insurance and qualify for income-based tax credits. People insured by Medicare and most large employers weren't affected.
Like no other state in the country, Oregon turned to Oracle to deliver a soup-to-nuts exchange site.
It's hard to argue against Oracle. It is one of the dominant technology companies in the world, earning more than $2.1 billion in its most recent quarter on sales in excess of $8 billion.
And two of the three finalists for the 2011 Oregon Health Authority exchange contract dropped out, leaving Oracle as the sole bidder.
The OHA, which oversees all of the state's health services, had ambitious plans. It wanted Oracle to build the exchange while replacing and modernizing several other state IT systems.
Carolyn Lawson, OHA chief information officer, said the Oracle contract was in place by the time she took her position in July 2011. But she vigorously defends the choice.
"If you take apart an insurance exchange, eligibility determination and e-commerce are key components," Lawson said. "They'd all been done as individual components by Oracle. There was no reason not to believe they could do this."
Oregon's initial funding for the project, a $48 million federal grant, called for a fully working exchange by Feb. 15, 2013.
In the end, what the exchange received in May 2013 was an unfinished, untested IT project with deep flaws hidden within -- and just four months to fix them.
The first hint of major trouble came in May 2013. A report penned by a Cover Oregon technologist raised dozens of "critical" problems with the Siebel customer relationship management system at the heart of the project devised by Oracle. The problems left the system prone to "data corruption and loss," "runtime errors" and "degraded performance."
Large quantities of code written to make the system work had been done poorly, filled with sloppy mistakes or unnecessary details, the report stated. "I was stunned; we all were," Lawson said.
Oracle declined repeated requests for comment.
Bob Cummings, an IT oversight analyst for the state Legislative Fiscal Office, says Cover Oregon didn't share the May internal report with him or Maximus, its quality consultant. He was alarmed by the thousands of lines of custom software code, suggesting the Oracle product simply wasn't "very well tailored to what Cover Oregon and OHA wanted to build."
Aaron Karjala, the Cover Oregon's chief information officer, said Oracle promised repeatedly it could get a handle on the technical failings. But he was losing faith. In August, he hired consulting giant Deloitte to build the exchange's public website where consumers could browse for health plans, in case Oracle didn't deliver.
Oracle's promises and missed deadlines continued right up until the state's humiliating failure to go live on Oct. 1.
Cover Oregon officials continued to promise the site would be fixed soon. But behind the scenes they soon realized the problems ran deeper than anyone had realized.
The poor coding work meant bugs -- including 157 of the most serious "blocker" variety -- continued to plague the site as recently as Thanksgiving. Cover Oregon claims that number has since been reduced to about 40.
Cover Oregon stopped paying Oracle in September. It is sitting on more than $18 million in invoices submitted by the company since then.
Nevertheless, Oregon has been good for Oracle. Between the OHA and Cover Oregon, the state has paid Oracle more than $90 million over the last two years and could pay the company another $30 million or more. Overall, the project has cost more than $160 million so far.
In August 2012, the consulting firm Maximus urged the public corporation overseeing the exchange, which became Cover Oregon, renegotiate several "time and materials" contracts to increase efficiency and ensure contractors produce "tangible" results. This was never done.
Systems Integrator
Washington state officials credit Deloitte, its systems integrator -- an overarching general contractor, of sorts -- for getting their health insurance exchange operational by Oct. 1. Kentucky does the same.
In fact, the vast majority of states that built their own exchange hired a systems integrator.
Oregon did not.
It was one of the only states that chose instead to oversee the project itself.
"It's not unlike a defendant serving as his own lawyer -- not a good idea," said Robert Booz, a technology analyst at the Gartner Group. "With some exceptions, states don't have the level of experience with these big system implementations."
What Deloitte did for Washington was to mercilessly narrow the scope of their project, said Michael Marchand, spokesman for the Washington exchange. Washington played small-ball and made its goals achievable.
Oregon took a different tack, opting for a cutting-edge exchange site that would not only allow consumers to shop for commercial health insurance, it would also handle small-business insurance and enroll eligible participants directly into the Medicaid-funded Oregon Health Plan.
The first contracts with Oracle and a handful of other private contractors to build the exchange were let by the OHA, the state agency that oversees health-related programs. Early in 2011, OHA officials decided they needed a systems integrator to oversee the exchange project.
OHA's advisors said a systems integrator was necessary. "The state does not have the development resources with the requisite experience," consulting firms Wakely Consulting Group and KPMG wrote in a joint March 2011 report. They added that the integrator could lead to "significantly lower" overall costs.
A report by CSG Government Solutions said Oregon lacked qualified managers and the systems integrator would provide "critical expertise ... The (state IT) team does not have experience with anything like the Oracle Solution which has been chosen."
In July 2011, OHA was poised to follow that advice. It solicited a systems integrator to oversee the exchange work.
But that month Oregon Health Authority Director Bruce Goldberg hired Carolyn Lawson to become chief technical officer of OHA.
Four months into the two-year exchange project, Lawson changed its entire makeup. She launched a sweeping reorganization of the team working on it, and canceled the solicitation for a systems integrator deciding it would save the state money, documents show. The state would be its own systems integrator.
Lawson defends the decision even as Oregon's exchange remains mired in technical difficulties.
"OK, let's be honest about this, there is no evidence that having a systems integrator is a slam-dunk path to success," she said. "There are plenty of IT projects that have gone horribly wrong that had a systems integrator. From what we understood at the time, I think it was the right decision."
Lawson based her reasoning, in part, on her faith in Oracle, a company she had worked closely with in previous posts in California. "The reason why you want a prime is you want to have access to the people who know the most about the platform or platforms you're using," Lawson said. "We had Oracle."
After missing its deadlines, Oregon is now adopting the narrower Washington model pushed so hard by Deloitte. Oregon's technology team is engaged in a massive "de-scoping," jettisoning many of the whistles and bells included in the state's original ambitious plan.
Technology analyst Booz said it's difficult for any IT client, and particularly a government client, to exercise the "absolute cold-hearted, cold-blooded objectivity," required of an effective systems integrator. "You're spending your political capital criticizing a contractor you chose. That's tough."
State Mismanagement
The health exchange project lost significant momentum and time when Lawson, Oregon Health Authority's new IT director, changed the project's approach, according to a consultant's report.
Crucial to the success of Lawson's plan was for the state and its consultants to diligently oversee Oracle's work while doing much of the programming itself. But the OHA technology team had no experience dealing with Oracle systems, and lacked software developers and managers with expertise, as consultants like Wakely warned.
In fact, three quarters of the necessary personnel for the exchange project still hadn't been hired four months after the Oracle contract was signed
The exchange team "has little experience and few, if any, technical resources to validate the work-in-progress by Oracle," stated a Nov. 2011 report from the quality consultant, Maximus. "The hiring of new personnel with sufficient expertise will be critical."
Lawson said she hired consultants to provide the expertise the exchange team lacked.
Whatever the reason, from the very start the Oracle team seemed to have broad leeway to run the project as it saw fit, according to several people who worked on the project in 2012.
"The state was sitting on their hands. It was management by consensus. I think that probably was why Oracle pretty much took over and said we're just going to move along," said Larry Ridgley, a senior business analyst who worked on the project through March 2012. "The (state's) project managers I spoke with felt completely overwhelmed. I don't think they knew how to manage a project of that scope."
An August 2012 Maximus report found the exchange project was disorganized, lacked basic management and budget controls to ensure contractor performance, and was poisoned by distrust and suspicion between executives at Cover Oregon and their counterparts at the Oregon Health Authority. The project essentially had two masters.
The report recommended a more unified management and better management controls. By October, the "window was closing" where these changes would have much effect on the success of the project, according to Maximus.
After lawmakers blasted the exchange for its lack of management controls in 2012 legislative hearings, OHA and Cover Oregon did make changes, but not quickly or decisively enough, interviews and documents show.
Lawson says the real problem was the tight timeline, as well as several federal rules that were changed or issued at the last minute. "You could have had the best management controls in the world and had a hard time managing through this," she said. She said narrow contract work orders helped keep Oracle on task. "We didn't just run wild with the money."
In October 2012, a consultant hired for a crucial part of the project quit after just eight days. Karthikeya Kala was alarmed by the fact that Oracle was essentially writing its own project specifications -- and poorly at that, he said. "It was obvious the project wasn't going to succeed," he said.
In his farewell letter he informed state managers the project wasn't following industry standards, resulting in a "chaotic" situation.
Adding to the problems was persistent infighting starting in 2012. Cover Oregon leaders believed OHA was preventing them from building a nimble, e-commerce site. Eventually, they went to Gov. John Kitzhaber to secure independence, exchange emails show.
In November 2012, Lawson accused Cover Oregon staff of "blindsiding" her in legislative hearings. The following month, Lawson urged King, Cover Oregon's director, to help "contain the emotion" of his subordinates after a "rough meeting. He responded that the real problem was the "competence" of her staff.
Bureaucratic backbiting continued into this October when Lawson accused Cover Oregon of instructing Oracle contractors not to talk to OHA. "The Oracle team at the highest levels believes they have been instructed by Rocky (King) not to talk to us," Lawson said in Oct. 7 email to her boss, Bruce Goldberg. "Since launch, the (Cover Oregon) side of the project has become a black hole."
While King feuded with Lawson, he also struggled to put an optimistic spin on the situation, repeatedly telling the public the site would be ready on Oct. 1. Privately, he worried that Oracle wouldn't deliver.
"We are in serious trouble," he told his leadership team in a Sept. 17 email. "This whole IT side has gotten away from us."
(c) 2013 The Oregonian (Portland, Ore.)
