The Alaska Department of Public Safety spent 11 years and $27.9 million trying to replace the state's aging criminal information database before the project was abandoned, derailed by problems with funding and internal management, according to documents about the project.
The database, the Alaska Public Safety Information Network, known as APSIN, has not been replaced nor fully upgraded since it launched in 1984 on what is now regarded as an outdated large-scale business computer system. Officials say they are increasingly nervous about a failure that could disrupt information for Alaska law enforcement agencies about warrants, prior arrests and fingerprints of suspected criminals. The public safety department spends hundreds of thousands of dollars a year to maintain the system and a shrinking pool of programmers are familiar with the technology.
But the 11-year effort to replace APSIN has been dormant since 2013, when a critical audit found the development of "New APSIN" was too troubled to succeed, according to documents obtained by Alaska Dispatch News.
The upgrade project began in 2002, when the Department of Public Safety hired a Virginia-based defense contractor to modernize the database. A critical goal was to move away from aging technology built on a "mainframe," the massive business computers in use when the APSIN system's code was written in the 1980s. Nine state departments, including the Department of Public Safety, still use the state's mainframe.
Designed to be built and funded over multiple years, the upgrade was a complex project that demanded substantial resources. From fiscal year 1998 through 2014, the Department of Public Safety paid $27.9 million to 35 different contractors, though most went to Science Applications International Corp. (now known as Leidos), a multibillion-dollar defense and security contractor, according to a summary of expenditures provided to Alaska Dispatch News.
When outside auditors from the Seattle-based firm MTG Management Consultants began looking at the project two years ago, they found a partially complete product that was largely untested and "not yet useful."
"This project profile is indicative of a seriously dysfunctional project with little to no likelihood of success if left on its current path," the auditors concluded. Despite the millions of dollars spent, they said they couldn't determine how much more was needed to complete the project.
"It is clear, however, that the current budget was half of what the vendor required to deliver something functional," the auditors said. And, they found, the then target date of July 2013 was an illusionary goal. "Even if something is delivered by this date, it will be nearly impossible to determine what (Department of Public Safety) requirements it does or does not meet."
Months later, the department quietly halted further spending on the project.
As a result, the original cost and technological problems not only still exist, but are 11 years more acute than they were when the upgrade was begun and the system "was thought to be nearing its end of life in terms of supportability," according to the audit.
Audit documents indicate that by most measures, the bungle was particularly large, taking significantly longer and costing far more money than similar projects in other U.S. states.
"While it is unfair to compare the capabilities and costs of today's marketplace to that in 2001," auditors wrote, "there are many recent examples of larger peer state agencies that have fully replaced their crime information center applications in a fraction of the elapsed time and budget compared to that used by Alaska."
The failed project exposed simmering layers of tension within the Department of Public Safety and its division that supports law enforcement technology and criminal history information statewide, documents show, and has triggered numerous restructuring efforts, including a new internal governance board.
The current deputy commissioner of the Department of Public Safety, Terry Vrabec, said in an interview that the public safety administration remains committed to upgrading the APSIN system. But he said he doesn't exactly know how, when or at what cost. The price tag is more acute now, given Alaska's particularly challenging fiscal climate.
"We do need to own up to the fact that we're not where we'd like to be," Vrabec said. "And there is a cost to that."
A good start
For law enforcement officials, the APSIN system has functioned for more than 30 years as a "one-stop-shop" for information such as a person's physical description, addresses, driver history, arrests and other criminal records, vehicle information, warrants and case involvement. The system ties into the National Crime Information Center, the FBI's massive network launched in 1967.
When a state trooper pulls over a speeding car on the Seward Highway, for example, the trooper radios a dispatcher back at headquarters. Using information pulled from APSIN, the dispatcher can tell the trooper if the driver's license is expired, or if the driver is wanted on a warrant.
The system has become deeply integrated with police technology and criminal background checks in the state in addition to connecting with federal and national databases. Over the years, elements and programs have been added to keep pace with state and federal requirements and improve the user-friendliness of the database.
At the user level, troopers no longer directly connect to APSIN to search for records on a daily basis. Instead, troopers use the Alaska Records Management System, an electronic records and field reporting system deployed in 2011 and fed by APSIN. Jess Carson, president of the Department of Public Safety chapter of the Public Safety Employees Union, said that in terms of day-to-day user interface, the department's police technology is "significantly better" than what existed in the early 2000s.
But officials and software developers with knowledge of the project say the underlying mainframe technology, now decades old, is the source of consternation.
When the Department of Public Safety, then under the administration of Gov. Tony Knowles and the late Public Safety Commissioner Glenn Godfrey, decided to replace APSIN in the late 1990s, the system was already about 15 years old. APSIN's launch in 1984 coincided with the introduction of the first Macintosh computer and the development of the first nationwide analog cellphone network, technology that has long been overtaken. But the criminal information database expanded around a core mainframe, which is physically located in Juneau and remains there to this day.
"Unfortunately, we once again find ourselves at a point where the demands of the criminal justice system as well as ever-increasing state and federal requirements make it necessary to design and build an entirely new system," the department wrote on its website. That history of APSIN, along with other content related to the project, was taken offline last fall.
In early 1999, the "APSIN Redesign Project" launched with a legislative document written by MTG Consulting Group, the same group that would eventually conduct the review that led to the project's end. In 2001, the search began for a contractor to build the new system. According to the archived "History of APSIN" page, the project was expected to take "approximately five years" to complete.
The replacement system was to be built from scratch and funded in phases.
In February 2002, records show, the Department of Public Safety awarded the redesign contract to Science Applications International Corp., or SAIC -- a multibillion-dollar government services contractor headquartered in McLean, Virginia. The company is also known for big-ticket software development contracts with federal government agencies, from the National Security Administration to the Department of Defense to the FBI.
In September 2013, SAIC spun off into a $4 billion company, and the parent company was renamed Leidos, which had the Alaska contract. In an email, a spokeswoman for Leidos would only say that "the Leidos system was delivered on time, on-scope and is fully functioning." The spokeswoman, Jennifer Gephart, referred further questions to the Department of Public Safety.
According to the APSIN redesign audit, the project apparently started off on the right foot. Auditors concluded that the original bid documents "appropriately specified phased implementation … clearly expressed budget tolerances, and had clear requirements."
As the years passed, those elements gradually became murkier, as the project evolved under uncertain funding and changing technology.
History of unmet expectations
In 2007, the department hired a contractor, Dean Barnes, to manage the APSIN project. In the department's quarterly magazine, Barnes wrote that his job was "ensure that DPS and our APSIN (contractor) building the software for DPS work together to produce a quality product that is useful for the users and makes it easier, faster and better for APSIN users to do their job."
By then the project had been in the works for five years. It was supposed to be wrapping up.
But in another five years, in November 2012, Barnes resigned from the project. His departure led Department of Public Safety management, "acting out of concern for the New APSIN project, which has a history of unmet expectations," to seek a third-party review, the documents show.
Within two months, a small team of auditors from the Seattle-based MTG Consulting Group, a management-consulting firm that focuses solely in clients in the public safety sector, arrived in Anchorage to interview people connected with the APSIN project.
Charles Collins Jr., the lead author of the MTG report, said in an interview that the consulting group had encountered SAIC in the past. He said he believed this to be MTG's first critical report on the contractor.
The preliminary 65-page audit, released in March 2013 and obtained through a public records request, reveals a dysfunctional project. Auditors found little evidence of a path to completion. The scope was unclear and the design too complicated. Funding, never fully secured, was piecemeal. Five of six phases of the new system were built, yet not fully tested by users. No one outside of the department was in charge of making sure the project was on the right track. So much time had elapsed that the technologies used to build the system in the first place were becoming outdated. And costs would continue to rise.
More than a decade of work, auditors found, had yielded a product that the agency couldn't use. Part of the problem was that the Department of Public Safety never signed off on the original design plans, according to the audit. As a result, when SAIC ran into a situation where enough resources weren't available, it simply stopped work on that part of the project and shifted elsewhere.
"This behavior … has resulted in a system that is partially complete across most dimensions and has expended significant funds for a system that is not yet useful," auditors wrote.
Over the duration of the project, the state paid SAIC, now Leidos, $22.6 million for the system, according to a breakdown in expenditures provided by the Department of Public Safety. Thirty-four other individual contractors and companies, including large companies like Hewlett Packard Inc., received payments totaling about $5.3 million.
Without a drastic change in direction, auditors said that the project stood little chance of gaining funding again.
"It is widely held that legislative approval for additional funds are at risk, based on the long history of funding requests and lack of demonstrable progress with New APSIN," the audit said.
The public safety commissioner's office was apparently largely uninvolved with the project until around the time of the audit. As was the traditional practice in the department, the commissioner and deputy commissioner mostly delegated involvement to mid-level managers. While that wasn't unusual, the MTG auditors wrote that it would have been "normal in today's environment to have a more hands-on, executive-level presence, especially given the history of the New APSIN effort."
Joseph Masters, who was the public safety commissioner from September 2008 to October 2013, said in an interview that he did not order major changes in scope until near the end of his five-year tenure. In 2012, he said, under pressure to move the agency away from the state mainframe, he started looking personally into the status of the project and set a firm completion deadline.
The deadline came and went. The project didn't go live. Around the same time, the project manager left, and Masters said he decided to bring in MTG. He said he "virtually stopped" the project several months before he resigned in October 2013, an event he said was unrelated to APSIN.
Masters said he didn't feel he was entirely abandoning the project, a decision he said would have to be made by the current administration. He said he was trying to halt completing further phases for a system that the department potentially couldn't use.
"That part was easy," Masters said."(That) we're not going to continue spending money on this until we know what's going on."
Division scrutinized
The problems detailed in the audit reports were flagrant enough to prompt Masters and other officials to launch a broad review of how information technology projects were managed within the Department of Public Safety.
That inquiry focused on parts of the Division of Statewide Services, the branch of the department that handles police technology and infrastructure, as well as criminal history information for federal, state and local law enforcement in Alaska.
In the December 2013 "organizational review" of the Division of Statewide Services, also conducted by MTG Consulting Group, auditors found evidence of what they called "a seriously dysfunctional IT support organization."
The review found that the division lacked governance and communication, its benefits to other divisions and departments were unclear, and there was a serious aversion to risk or change. Its leaders had trouble securing funding for multi-year projects. Auditors also found the failure of the New APSIN project had compounded a sense among units like Alaska State Troopers that the division could not take on new technology projects.
"Due to historical performance, the lack of business analysts, and the frustration with being told 'no,' stakeholders look for opportunities to solve the problems without involving DSS," the auditors wrote. Recent, successful software implementations, such as the Alaska Records Management System, were managed outside of the Division of Statewide Services, the report noted.
Vrabec, the deputy public safety commissioner, confirmed that the reports were a "wake-up call."
"Did we know we were struggling that much in some areas? No," said Vrabec, who became deputy commissioner in June 2011 in the administration of Gov. Sean Parnell and has continued under Gov. Bill Walker.
"So it's our obligation to recognize that. Own up to it and fix it and move on."
A larger struggle
The Department of Public Safety is not the only state agency trying to modernize its data systems and move away from the mainframe. Six other agencies have embarked or are preparing to embark on projects with similar goals, according to information provided by the state Department of Administration, which manages state mainframe technology.
On a technical basis, the APSIN audit documents show a connection between APSIN and a similar upgrade project by the Department of Motor Vehicles and its database for vehicle and license information. DMV fiscal manager Coleen Greenshields said $11 million has been appropriated to that project, and about $2.9 million had been spent through January.
The former manager of the APSIN project, Dean Barnes, signed a contract to work with the DMV shortly after resigning from APSIN, records show. Barnes declined to comment. His contract with DMV ended in July 2014.
SAIC/Leidos also signed a contract with the DMV and was paid $1.1 million between December 2012 and November 2014. The contract ended when DMV determined that the scope of the project and required work was "not defined well enough to move forward," Greenshields wrote in a Jan. 21 email. Greenshields said the DMV was preparing to select a new vendor and that the project was still on track.
With the APSIN project stalled, Masters, the former Public Safety Commissioner, said the goals that first prompted the redesign effort should remain.
"The state has to get off that mainframe," Masters said. "And they've got to upgrade to where that information is housed to be available to those that need it."
As the Department of Public Safety debates how and when to make its own move, other internal changes are underway. Vrabec said the commissioner's office is focusing on strengthening oversight in the Division of Statewide Services, where leadership has recently been in flux.
The longtime director, David Schade, left in November 2013. (Reached by phone, Schade said he wouldn't comment.) After a year of a transitional director, a new director, Katherine TePas, began work on Nov. 24. TePas was previously the sexual assault coordinator for Gov. Sean Parnell, where she was deeply involved in the Alaska National Guard scandal, and before that, a domestic violence and sexual assault services program coordinator for the Alaska State Troopers.
On Monday, after just two months on the job, TePas was replaced by Dan Spencer, a former public safety director of administrative services.
In emails to Alaska Dispatch News, TePas cited numerous examples of change that has already occurred within the division since the MTG audits, such as filling a new data processing manager position, setting up an internal tracking system for software developers, expanding help desk hours and creating an internal board of governance.
As for the future of APSIN, TePas wrote: "At this time DPS has no plans to restart the project."