The high-rise offices of the Los Angeles Information Technology Agency (ITA), which manages the IT systems used by 30,000 city employees, are a model of corporate efficiency -- a floor of cubicles ringed by window-facing rooms. Glass doors define a modest-size waiting room, where a flat-screen plays the city government TV channel on loop. A tall trophy case displays the department's victories. An organizational chart shows photos of CTO and ITA General Manager Randi Levin and her executive team.
It's all ordinary enough to make one temporarily forget that the iconic L.A. City Hall building, a tower made famous as a scene-setter in well known motion pictures, is across the street. Believe it or not, this Hollywood reference point is tangentially relevant, at least for Levin.
Watch Video: Los Angeles CTO Randi Levin is spearheading a first-of-its kind implementation of Gmail that's under way and set for a summer rollout.
Whether she likes it or not, Levin has become the star of her own story -- partly of her own doing, partly due to forces beyond her control. Levin's front-and-center introduction to the mainstream world came last year, when she led the ITA on a procurement that will replace the city's aging e-mail system with a new Web-based enterprise solution. At the core, Levin had two simple goals in mind: improve service and save money.
When the city pickedGoogle's productivity tools along with its popular e-mail service Gmail, what initially was thought to be a run-of-the-mill IT project quickly morphed into something bigger and more complex. The decision stoked a period of intense lobbying from L.A.'s existing e-mail provider (Novell) and Google's biggest competitor (Microsoft), rivals who likely saw the city's decision to adopt Google's hosted services as something that could potentially crack the state and local government market's inertia when it comes to cloud computing. Levin was unexpectedly pressured from within, as L.A. fire and police officials expressed concern that moving their sensitive data onto Google's off-site servers could pose a security problem. Levin said she has since quelled those concerns and the political pressure.
The script, if you will, continues to be written. Los Angeles is now slowly marching toward a full implementation of Gmail for the city work force. If successful, the project could open the floodgates for other governments that are awaiting a successful test case before entering the cloud computing environment.
A Missing Data Point
Ever since Levin began leading the ITA two and a half years ago, she repeatedly heard from employees who were dissatisfied with the unreliability of the city's existing e-mail system, Novell GroupWise. It had too much downtime, and users were frustrated by the lack of features and the user experience. The product itself wasn't inherently unreliable, Levin said, but the ITA lacked the necessary money or manpower for its proper upkeep. And like many IT departments, Levin was facing the prospect of shrinking budgets due to the recession's lingering effects. The problem would only get worse, she thought. On-premises e-mail just wasn't a cost-effective option anymore, in her mind.So the ITA put together an RFP with the option of a software-as-a-service product or a hosted solution. Levin said the agency received 10 responses, from the likes of Google, Microsoft and Yahoo. After mulling over the decision with an intradepartmental group of IT managers, last summer officials chose a proposal that would implement Gmail on more than 30,000 desktops, and later adopt the Google Apps productivity suite, which includes calendar, word processing, document collaboration, Web site support, video and chat capabilities, data archiving, disaster recovery and virus protection.
The five-year deal, valued at $17 million, made L.A. the first government of its scale to choose Gmail for the enterprise -- a somewhat surprising bit of information that made approving the project much more complex.
"We were under the assumption that Washington, D.C., had already fully implemented Google for its e-mail solution, which it had, but not in the way we're doing it. But we didn't really know that at the time," Levin said.
Watch Video: Los Angeles CTO Randi Levin reviews the decisions and considerations for implementing Google's Gmail citywide.
It turned out that Washington, D.C., was using Gmail for disaster recovery and giving employees the option to use it as their primary e-mail. During the decision-making period, Levin didn't think L.A. would be the first large government to fully adopt Gmail. "Nor did we think it was going to be as political as it turned out to be," she added. That knowledge wouldn't necessarily have changed the city's decision, Levin said, but it would have given the city a heads-up that lobbying and outside interest from the public was coming.
The lobbying was "extensive," said L.A City Council President Eric Garcetti, who presided over the Council's unanimous vote in October 2009 to adopt the plan. As many as five companies made their presence known in the corridors of City Hall, he said, as misinformation reigned and unfounded rumors flourished. Attempts at deal-making continued until minutes before the Council voted. Levin said those temptations were never a factor. "We tried to maintain a very rigorous [procurement] process, and we really wanted the integrity of the process to stay intact."
Crunching the Numbers, Squeezing the Budget
The incessant lobbying spurred troublesome misinformation, particularly about the solution's cost and security, Levin said.The cost and potential savings confused outside observers and elected officials because the ITA wanted to accurately reflect the city's deteriorating economic condition, Levin said. That meant the projections were changed more than once.
"It became more and more important to focus on cash as opposed to a true ROI [return on investment]," she explained. This changed the numbers. The ITA had, at different times, estimated savings of $8 million to $30 million. "From the cash perspective, we looked at what software and hardware would be removed as we went to a new solution -- what wouldn't we have to buy anymore or pay maintenance on."
Levin felt it was important to do an "apples-to-apples" comparison. Unfortunately some people didn't understand the difference between ROI and cash savings, she said. By the time the numbers were made clear, some people inaccurately believed Gmail would be more expensive than the existing solution. Although, in a limited sense, that was true because the city will pay for both GroupWise and Gmail for one year as the migration occurs. (Ironically the ITA will offset the added cost by using money from a prior anti-trust settlement with Microsoft.)
After a few attempts at numbers crunching, the city estimated $5.5 million in hard-cost savings from the Google adoption, and an additional $20 million savings in soft costs due to factors like better productivity. The ITA expects applications like Google Docs will help reduce some of the redundant paper pushing that plagues bureaucracies, and it hopes someday to utilize Gmail's mobile functionality and ease-of-use to drive further savings through increased collaboration.
Moving the city's data to Gmail will let the ITA reassign and/or cut nine employees who were working internally on the GroupWise system, Levin said, and it will eliminate 92 servers from the city's data center -- a sprawling basement-level facility in the ITA building. Those savings are significant, she said, because as of mid-November the ITA faced the prospect of losing 60 or 70 employees to early retirement, as well as additional cuts to the 800-person ITA organization.
"We have servers of every shape, size, brand and year here," she said, "and with diminished staffing, we're trying to figure out where's the best use of our resources, and we think it's really more in the applications area -- in public safety related to their radio systems and some of their other applications, and also for the other departments' Web sites -- doing a lot in terms of transparency and getting data out to the public, and more self-service."
Security Fears, Reliability Concerns
Data security was another contentious issue. The public at large continues to debate the security of cloud computing and hosted services, particularly as it relates to putting the public's data -- which may well include addresses, Social Security numbers and other sensitive information -- on servers in unknown locations that are managed by a corporation.After some officials from the L.A. police and fire departments expressed worry that their departments' sensitive data would be vulnerable if stored on off-premise servers, the ITA worked hard to ensure that the security parameters met California Department of Justice requirements, said Kevin Crawford, Levin's deputy in charge of the Gmail migration. Google employees who have access to L.A.'s data will be certified by the state Department of Justice. Google, for its part, is building a segregated "government cloud" that will house data owned by public-sector customers like Los Angeles. The government cloud will be on servers located somewhere within the contiguous 48 states, although L.A. won't know exactly where its data is -- the unknown location is part of Google's security model.
The government cloud will be up and running "sometime in 2010," according to David Mihalchik, business development executive for Google federal. Crawford said he's been told the new cloud will be ready by June, in time for L.A.'s full implementation. The company also is in the process of securing Federal Information Security Management Act (FISMA) certification.
L.A.'s agreement with Google is written so that it's clear the city owns the data at all times, Crawford said. "That's a very big deal for us. We've written [the contract] as ironclad as we can. We've also written into the nondisclosure that the data belongs to us in perpetuity; it will outlive the contract itself," he said. That means if the city wants to switch to another vendor after the contract ends, the city will be able to recall its archived data. Officials also negotiated unlimited and liquidated damages in the event that there's a breach of Google's servers.
Crawford said the bottom line is that Google's security apparatus is far superior to the ITA's for the simple fact that the company has the resources to devote many more people to it. In Google-speak, L.A.'s data will be "sharded," meaning it will be shredded into multiple pieces and stored on different hard drives -- a security encryption method the ITA can't do from its in-house data center. Garcetti too said he's comfortable with the security of cloud computing: "At the end of the day, I trust Google's security as much as any individual city, town or village to protect themselves because [Google] is that much more experienced."
Of course, reliability is part of security. Crawford said Gmail had only about 10 percent of the downtime in 2009 as the city's current e-mail. And if disaster strikes -- L.A. sits in earthquake country atop a fault zone -- two other data centers outside the geographical location will have a live version of the city's data that can be quickly retrieved in case of emergency. The ITA didn't have the means for such robust disaster recovery in its existing configuration.
Pilot to Gmail Control
A citywide pilot of Gmail launched in January, led by Crawford's team in partnership with CSC, a Virginia-based integrator that's partnering with Google on the implementation. The pilot will continue for 60 days with 3,000 participants -- about 10 percent of the city's work force. The citywide implementation is scheduled for mid-June, Crawford said. After the e-mail deployment is done, the ITA will focus on Google Apps, and all employees will be trained on video and chat sooner rather than later. Deployment at the police department will be phased in last so that Google has time to complete FISMA certification and the government cloud.Crawford said the city will save a significant amount of money by handling the transfer of e-mail archives to the new system, instead of having CSC perform the task. The city has approximately 14 terabytes of archived data, and seven terabytes of live e-mail data.
Watch Video: Kevin Crawford, assistant general manager of Los Angeles' Information Technology Agency, discusses technical considerations of the Google Gmail implementation.
The ITA also wrote a piggyback clause into its agreement with CSC that allows any public-sector agency in California to purchase off that same contract. "The kind of services and capabilities we're providing right now [to L.A.] are the same as any municipality can get," said Mark Kneidinger, managing partner for CSC's Federal Consulting Practice, who is a former public-sector CIO. He said the available services include e-mail capacity; e-mail migration; building the architecture and security; end-user services; training; and the system architecture, design and integration. The vendor can also migrate agencies' applications to the cloud.
Kneidinger said smaller municipalities could use the piggyback contract to form consortia that would further cut implementation costs. "What's happening currently among cities -- if you have a lot of small towns -- and I know there are a number of areas in the country where this occurs, is that they're sharing common e-mail systems. Then basically [Gmail] is just the same overlay," he said.
But so far, according to Crawford, no other governments have piggybacked on the L.A. contract -- he thinks that's because they're in a wait-and-see mode. It's just another example of the scrutiny and spotlight Levin admits is now omnipresent. "But I think more of the pressure actually is on CSC and Google to make this successful," Levin said, "because obviously they would like to increase their sales in the state and local government arena. I have been assured they will make it work here."
With L.A. in the fold, Google executives are bullish on the state and local government market.
"The interest is very strong, and I think it goes back to the benefits of cloud computing," said Google's Mihalchik. "Government spends too much on IT, and particularly as it pertains to e-mail and collaboration solutions that we're offering -- it's rare to have an opportunity to both reduce costs and improve performance. That's what happened in the city of Los Angeles, and that's why other government agencies are looking at this and are strongly interested."
Mihalchik said the customer base for Google Apps has quickly expanded to include pilots or implementations in governments across the United States, like Orlando, Fla., and within 12 federal agencies. Crawford said that as of mid-January, he had talked with more than 70 governments that were interested in moving data and services to cloud computing.
So Los Angeles might not be an outlier for long. And there's a sense that Levin already is writing the next part of this script -- and that Web-based e-mail is only Scene One. She's exploring options that would outsource city servers or put them under the management of a public-private partnership.
As for the Gmail decision, Levin is certain. "There was no question in our mind that this was the right thing to do," she said.
