The US government spent billions on a system for detecting hacks. The Russians outsmarted it.

WASHINGTON — When Russian hackers first slipped their digital Trojan horses into federal government computer systems, probably in the spring, they sat dormant for days, doing nothing but hiding. Then the malicious code sprang into action and began communicating with the outside world.

At that moment — when the Russian malware began sending transmissions from federal servers to command-and-control computers operated by the hackers — an opportunity for detection arose, much as human spies behind enemy lines are particularly vulnerable when they radio home to report what they’ve found.

Why, then, when computer networks at the State Department and other federal agencies started signaling to Russian servers, did nobody in the U.S. government notice that something odd was afoot?

The answer is part Russian skill, part federal government blind spot.

The Russians, whose operation was discovered this month by a cybersecurity firm that they hacked, were good. After initiating the hacks by corrupting patches of widely used network monitoring software, the hackers hid well, wiped away their tracks and communicated through IP addresses in the United States rather than ones in, say, Moscow, to minimize suspicions. The hackers also used novel bits of malicious code that apparently evaded the U.S. government’s multibillion-dollar detection system, Einstein, which focuses on finding new uses of known malware and detecting connections to parts of the Internet used in previous hacks.

But Einstein, operated by the Department of Homeland Security (DHS), was not equipped to find novel malware or Internet connections, despite a 2018 report from the Government Accountability Office suggesting that building such capability might be a wise investment. Some private cybersecurity firms do this type of “hunting” for suspicious communications — maybe an IP address to which a server has never before connected — but Einstein does not.

“It’s fair to say that Einstein wasn’t designed properly,” said Thomas Bossert, a top cybersecurity official in both the George W. Bush and Trump administrations. “But that’s a management failure.”

CISA spokeswoman Sara Sendek said the breaches stretch back to March and were not caught by any intrusion detection or prevention system. As soon as CISA received indicators of the activity it loaded them into Einstein to help identify breaches on agency networks, Sendek said. CISA is providing technical assistance to affected agencies, she said.

Russia has denied involvement in the intrusions.

The federal government has invested heavily in securing its myriad computers, especially since the extent of the devastating Chinese hack of the Office of Personnel Management was discovered in 2015, when more than 20 million federal employees and others had their personal information, including Social Security numbers, compromised.

But this year’s months-long hack of federal networks, discovered in recent days, has revealed new weaknesses and underscored some previously known ones, including the federal government’s reliance on widely used commercial software that provides potential attack vectors for nation-state hackers.

The FBI and Department of Homeland Security are investigating the scope and nature of the breaches, which intelligence officials believe were carried out by the Russian Foreign Intelligence Service (SVR). Sen. Richard Blumenthal, D-Conn., on Tuesday publicly acknowledged as much, tweeting that the Senate received a “classified briefing on Russia’s cyberattack [that] left me deeply alarmed, in fact downright scared.”

The Russians reportedly found their way into federal systems by first hacking SolarWinds, a Texas-based maker of network-monitoring software, and then slipped the malware into automatic updates that network administrators, in the federal government and elsewhere, routinely install to keep their systems current. The company reported that nearly 18,000 of its customers may have been affected worldwide.

More broadly, the hack highlighted the struggles of the government’s network-monitoring systems to detect threats delivered through newly written malicious code communicating to servers not previously affiliated with known cyberattacks. This is something advanced nation-state hackers, including from Russia, sometimes do — presumably because it makes intrusions harder to detect.

The full scope of the hack remains unknown, though it’s clear that a growing number of agencies have been penetrated, including the departments of State, Treasury, Homeland Security and Commerce and the National Institutes of Health. They are among victims that include consulting, technology, telecom, and oil and gas companies in North America, Europe, Asia and the Middle East.

The Pentagon was assessing Tuesday whether there had been intrusions at the sprawling department and what impact they may have had, a spokesman said.

The U.S. government has not publicly attributed the hacks to anyone.

Emails were one target of the hackers, officials said. Although it’s not yet clear what the Russians may be intending to do with the information, their victims, including a variety of State Department bureaus, suggest a range of motives.

At the State Department, they may want to know what policymakers’ plans are with respect to regions and issues that affect Russia’s strategic interests. At the Treasury, they may have sought insights into potential Russian targets of U.S. sanctions. At the National Institutes of Health (NIH), they may be interested in information related to coronavirus vaccine research.

As the investigative work continues, some lawmakers are focused on probing why and how federal cybersecurity efforts have fallen short despite years of damaging hacks by Russian and Chinese spies and major federal investments in defensive technologies.

Einstein, which was developed by the DHS and is operated by the department’s Cybersecurity and Infrastructure Security Agency (CISA), was made to be a backbone of federal protection of civilian agency computers, but the 2018 GAO report found significant weaknesses.

The capability to “identify any anomalies that may indicate a cybersecurity compromise” was planned for deployment by 2022, the report said. It also said network monitoring by individual agencies is spotty. Of 23 federal agencies surveyed, five “were not monitoring inbound or outbound direct connections to outside entities,” and 11 “were not persistently monitoring inbound encrypted traffic.” Eight “were not persistently monitoring outbound encrypted traffic.”

“DHS spent billions of taxpayer dollars on cyber defenses and all it got in return was a lemon with a catchy name,” said Sen. Ron Wyden, D-Ore., a member of the Senate Intelligence Committee. “Despite warnings by government watchdogs, this administration failed to promptly deploy technology necessary to identify suspicious traffic and catch hackers using new tools and new servers.”

It was not just this administration.

Bossert, who worked on the original Einstein concept in the George W. Bush administration, said the idea was to place active sensors at an agency’s Internet gateway that could recognize and neutralize malicious command-and-control traffic. “But the Bush, Obama and Trump administrations,” he said, “never designed Einstein to meet its full potential.”

CISA officials told congressional staff on a Monday evening call that the system did not have the capacity to flag the malware that was signaling back to its Russian masters.

The officials said federal agencies had not given CISA the information necessary to identify agency servers that should not be communicating with the outside world, said one congressional aide, who spoke on the condition of anonymity to discuss a sensitive matter.

“To CISA, all internal agency computers look the same, and so Einstein only flags samples of known malware or connections to ‘known bad’ IP addresses,” the aide said.

Other cybersecurity experts say the breaches highlight the “desperate” need for a government board that can conduct a deep investigation of an incident such as SolarWind’s, whose corrupted patches enabled the compromises — and crucially, make the report public.

“We need people to read the report, and say, ‘Oh, wow, we need to secure our (information technology) pipeline,’ ” said Alex Stamos, head of the Stanford Internet Observatory, a research group. He previously was chief security officer at Facebook and Yahoo.

He said there are “hundreds or thousands of companies” in this space that may have security flaws without knowing. These firms do network monitoring, IT management and log aggregation. “Enterprise IT is a $2 trillion market,” Stamos said. “There’s no agency in charge of ensuring its security.”