It was Aug. 12, the first day of the 2014-15 school year. The second-largest district in the country started using a new integrated student information system dubbed MiSiS to serve more than 640,000 students in more than 900 schools, and it wasn't working right. In some cases, students were left stranded without a class schedule. The data the system did have wasn't always accurate. Soon, the system slowed to a crawl. Then, the software bugs multiplied.
"By the first day, it was just trouble from the beginning," said Mary Lu Camacho, who had worked on scheduling classes in previous roles in the district.
Along with an FBI probe into the district's troubled iPad procurement, problems with MiSiS cost the superintendent, CIO and project manager their jobs. But failure was not an option. Over the last three years, a hand-picked group of leaders have slowly brought LAUSD back from chaos. This is the story of how it happened.
Back story
Since 1993, the district has been involved in a class action lawsuit, negotiations and court orders regarding faulty special education practices, including tracking student records. Back then, the district had 26 student information-related systems that didn't integrate, including a centrally managed one for elementary schools, and other locally managed systems at the high school and middle school level.A revised court order in 2003 established measurable improvements the district needed to make so records wouldn't get lost. One of the key fixes was an integrated student information system that followed students throughout their school career from pre-kindergarten to 12th grade.
"The district has been trying for a very long time to get this done and get it done right," said Diane Pappas, a long-time leader in the district's legal department.
Eleven years later at a high-stake's time, the district launched a system modified with code from Fresno Unified's student information system, after spending millions of dollars and trying two different approaches. The system included enrollment, attendance, scheduling, grades, transcripts, GPA calculations, special ed, meal eligibility, transportation, health, counseling and discipline data. The district had received warnings from principals, teachers, administrators and Board Member Bennett Kayser that the system wasn't ready, but their warnings went unheeded. Inside the project team, a culture of fear meant that few spoke up.
"Part of the problem was that people were very aware of the problem, and they did not have the freedom to talk," said Shahryar Khazei, who was a deputy CIO at the time.
Leadership changes
After a few months of limping along with a broken system, Superintendent John Deasy and CIO Ron Chandler resigned under pressure the same week in October 2014; later that month, the district terminated the contract for MiSiS project leader Bria Jones. Almost immediately, the school board brought in 82-year-old Ramon C. Cortines for the third time as superintendent to sort out the mess.Cortines personally recruited leaders from other departments to turn around what had been dubbed the MiSiS Crisis. Pappas became the CEO of the project, while Camacho worked with a team on the scheduling piece of the system and Khazei provided technical expertise. The 300-person MiSiS recovery team was made up of school and district leaders from different areas along with staff from Microsoft. Together, they started the long process of fixing the broken system while schools were still using it.
"Before I came, we really didn't have very many school people that had experience to share how the system should work," said Camacho, who had worked with scheduling in previous roles as an assistant principal and district coordinator.
Pappas had spent time on projects with Cortines in his previous runs as superintendent, and she had a reputation for getting things done. She didn't have a technical background, but on Nov. 3, 2014, she started trying to understand what went so wrong with the system.
"Sometimes when you don't know things, you're not afraid to do the impossible because you don't know how impossible what you're doing really is," Pappas said.
The long-time leader was surprised to hear that the project team was afraid to speak up about their concerns, so she began rebuilding a culture of trust, openness and communication, not only within the project team, but also with the school board, the public and education leaders throughout the district.
"She's very open and honest, and I think that was needed because we had to regain the trust of our stakeholders," said Khazei, who's now the district's CIO. "She played such an amazing role in really bringing that confidence back on the team and buffering us from the politics and giving us that room we needed to be able to repair and fix the problem."
Fix-it year
The new project leadership team soon discovered that system requirements had never been established and users hadn't really been consulted about what the system should do, shortcomings that required a considerable amount of time and energy to fix."Not only did we have to fix the system, we had to create documentation requirements, and it has just been probably the hardest challenge I've had professionally," Pappas said.
Every morning at 9 a.m., the team met in the war room to talk about the issues of the day and how their work affected each other. They had a safe space to deal with challenges and come together as a team.
In her first week on the project, Camacho met with all the secondary assistant principals of counseling services. These assistant principals were responsible for creating their schools' master schedule of student and teacher course schedules. For example, teachers would need to be assigned different sections of algebra and geometry so that they could accommodate the 200 students who wanted a mix of those classes. And they needed these schedules before the first day of school, which didn't happen in 2014.
Camacho listened to the problems they were having with MiSiS and came out of that meeting with seven pages filled with requests for what needed to change in the most complex piece of the system.
"My heart hurt for them because I was in their shoes once, and I couldn't imagine myself going through that," Camacho said.
Camacho created a formal focus group of users who could give feedback to developers and prioritized the fix-it list based on what needed to happen at different points in the year. It was crucial to have the users' input because they understand the school system and processes.
Microsoft provided manpower and expertise for the round-the-clock task of fixing the broken system. They helped build the district team's capacity in terms of skills to address the challenges they were facing. Many LAUSD staff pulled hours of overtime to fix the bugs that kept popping up so seniors could get their transcripts, students could register for classes and teachers could take attendance, among other things.
"I think the complexity and challenge here was that the system was live, and the decision was made that we were too far into it and that we were at a point of no return," Khazei said. "We had to operate the system as we were fixing it, and that's why it was such a major effort and accomplishment."
Costly repair
Since June 2013, the district has spent approximately $173 million on MiSiS, but previously spent millions more on other attempts, Pappas said. In the recovery years, the team did 13,000 updates to the system, made 1,600 enhancements, closed out 100,000 help desk tickets and fixed 5,700 bugs. Call volume has gone down 35 percent, and 40,000 users log into the system every day. Now they've gone from daily update releases to one release every three weeks.By the end of the fix-it year, the system had been turned around. In 2015-16, the MiSiS team worked on making the system run and look better. In the 2016-17 school year, the system became more stable, though the district was still fixing things here and there. The new school year starts Aug. 15, and the team leaders hope to have another smooth opening where every student is enrolled in classes, every teacher can take attendance, every office clerk can enroll a student and all staff with report access can print reports as needed, Camacho said.
Progress to date
LAUSD's team leaders learned several key lessons from the project: Project management skills are important, especially in such a large district with projects that dwarf the size of what most districts have to manage. An environment of trust, communication and collaboration is extremely valuable. Project teams have to partner with internal business owners and users to ensure success."The key is making sure that the stakeholders are engaged and they're participating and they're partnering and that testing is done very carefully before going live," Khazei said.
To prevent scheduling problems in the future, Camacho led a collaborative effort between the MiSiS, local, central and school site support staff to create a playbook to standardize practices and processes for starting and ending the school year, as well as transitioning to summer school. It includes a monthly task calendar for each of these events, checklists, key milestones, district policies and responsibility charts. Previously, individual schools decided when they would put together their schedules and how they would do it, often leaving it until the last minute.
"That annual playbook that we have put together is very helpful, and it keeps everyone on their toes," Khazei said.
Microsoft has transitioned out of the picture now that the crisis is over, while Infosys, the IT consulting services firm, has come in to help manage the system into the future. While the system isn't perfect, it's working more smoothly, according to Pappas. With this system functioning and special education changes in place, the district is looking to prove to the court that it's done everything it's supposed to according to the court order to improve special education service.
The MiSiS team members are now in leadership positions that are visible within the district, Pappas said. "The leadership team knows them, their careers have changed. They've gone from some of them being mild mannered to being aggressive and strong and taking charge. It's like everybody grew up. I think I did too."
Team members will never forget they were part of the MiSiS recovery project. "It's really been great, painful, but great. It's like the real lows, and the real highs altogether," said Pappas. "I'm so proud of the team. These people are rock stars."
