The perfect weapon: How Russian cyberpower invaded the U.S.

WASHINGTON — When Special Agent Adrian Hawkins of the FBI called the Democratic National Committee in September 2015 to pass along some troubling news about its computer network, he was transferred, naturally, to the help desk.

His message was brief, if alarming. At least one computer system belonging to the DNC had been compromised by hackers federal investigators had named "the Dukes," a cyberespionage team linked to the Russian government.

The FBI knew it well: The bureau had spent the last few years trying to kick the Dukes out of the unclassified email systems of the White House, the State Department and even the Joint Chiefs of Staff, one of the government's best-protected networks.

Yared Tamene, the tech-support contractor at the DNC who fielded the call, was no expert in cyberattacks. His first moves were to check Google for "the Dukes" and conduct a cursory search of the DNC computer system logs to look for hints of such a cyberintrusion. By his own account, he did not look too hard even after Hawkins called back repeatedly over the next several weeks — in part because he wasn't certain the caller was a real FBI agent and not an impostor.

"I had no way of differentiating the call I just received from a prank call," Tamene wrote in an internal memo, obtained by The New York Times, that detailed his contact with the FBI.

It was the cryptic first sign of a cyberespionage and information-warfare campaign devised to disrupt the 2016 presidential election, the first such attempt by a foreign power in American history. What started as an information-gathering operation, intelligence officials believe, ultimately morphed into an effort to harm one candidate, Hillary Clinton, and tip the election to her opponent, Donald Trump.

Like another famous American election scandal, it started with a break-in at the DNC. The first time, 44 years ago at the committee's old offices in the Watergate complex, the burglars planted listening devices and jimmied a filing cabinet. This time, the burglary was conducted from afar, directed by the Kremlin, with spear-phishing emails and zeros and ones.

An examination by The Times of the Russian operation — based on interviews with dozens of players targeted in the attack, intelligence officials who investigated it and Obama administration officials who deliberated over the best response — reveals a series of missed signals, slow responses and a continuing underestimation of the seriousness of the cyberattack.

The DNC's fumbling encounter with the FBI meant the best chance to halt the Russian intrusion was lost. The failure to grasp the scope of the attacks undercut efforts to minimize their impact. And the White House's reluctance to respond forcefully meant the Russians have not paid a heavy price for their actions, a decision that could prove critical in deterring future cyberattacks.

The low-key approach of the FBI meant that Russian hackers could roam freely through the committee's network for nearly seven months before top DNC officials were alerted to the attack and hired cyberexperts to protect their systems. In the meantime, the hackers moved on to targets outside the DNC, including Clinton's campaign chairman, John D. Podesta, whose private email account was hacked months later.

Even Podesta, a savvy Washington insider who had written a 2014 report on cyberprivacy for President Barack Obama, did not truly understand the gravity of the hacking.

By last summer, Democrats watched in helpless fury as their private emails and confidential documents appeared online day after day — procured by Russian intelligence agents, posted on WikiLeaks and other websites, then eagerly reported on by the American media, including The Times. Trump gleefully cited many of the purloined emails on the campaign trail.

The fallout included the resignation of Rep. Debbie Wasserman Schultz of Florida, the chairwoman of the DNC, and most of her top party aides. Leading Democrats were sidelined at the height of the campaign, silenced by revelations of embarrassing emails or consumed by the scramble to deal with the hacking. Though little-noticed by the public, confidential documents taken by the Russian hackers from the DNC's sister organization, the Democratic Congressional Campaign Committee, turned up in congressional races in a dozen states, tainting some of them with accusations of scandal.

In recent days, a skeptical president-elect, the nation's intelligence agencies and the two major parties have become embroiled in an extraordinary public dispute over what evidence exists that President Vladimir Putin of Russia moved beyond mere espionage to deliberately try to subvert American democracy and pick the winner of the presidential election.

[Dispute over Russia reflects growing GOP resistance to Trump]

Many of Clinton's closest aides believe that the Russian assault had a profound impact on the election, while conceding that other factors — Clinton's weaknesses as a candidate; her private email server; the public statements of FBI Director James B. Comey about her handling of classified information — were also important.

While there's no way to be certain of the ultimate impact of the hack, this much is clear: A low-cost, high-impact weapon that Russia had test-fired in elections from Ukraine to Europe was trained on the United States, with devastating effectiveness. For Russia, with an enfeebled economy and a nuclear arsenal it cannot use short of all-out war, cyberpower proved the perfect weapon: cheap, hard to see coming, hard to trace.

"There shouldn't be any doubt in anybody's mind," Adm. Michael S. Rogers, the director of the National Security Agency and commander of U.S. Cyber Command said at a postelection conference. "This was not something that was done casually, this was not something that was done by chance, this was not a target that was selected purely arbitrarily," he said. "This was a conscious effort by a nation-state to attempt to achieve a specific effect."

For the people whose emails were stolen, this new form of political sabotage has left a trail of shock and professional damage. Neera Tanden, president of the Center for American Progress and a key Clinton supporter, recalls walking into the busy Clinton transition offices, humiliated to see her face on television screens as pundits discussed a leaked email in which she had called Clinton's instincts "suboptimal."

"It was just a sucker punch to the gut every day," Tanden said. "It was the worst professional experience of my life."

The United States, too, has carried out cyberattacks, and in decades past the CIA tried to subvert foreign elections. But the Russian attack is increasingly understood across the political spectrum as an ominous historic landmark — with one notable exception: Trump has rejected the findings of the intelligence agencies he will soon oversee as "ridiculous," insisting that the hacker may be American, or Chinese, but that "they have no idea."

Trump cited the reported disagreements between the agencies about whether Putin intended to help elect him. On Tuesday, a Russian government spokesman echoed Trump's scorn.

"This tale of 'hacks' resembles a banal brawl between American security officials over spheres of influence," Maria Zakharova, the spokeswoman for the Russian Foreign Ministry, wrote on Facebook.

Over the weekend, four prominent senators, two Republicans and two Democrats, joined forces to pledge an investigation while pointedly ignoring Trump's skeptical claims.

"Democrats and Republicans must work together, and across the jurisdictional lines of the Congress, to examine these recent incidents thoroughly and devise comprehensive solutions to deter and defend against further cyberattacks," said Sens. John McCain, Lindsey Graham, Chuck Schumer and Jack Reed.

"This cannot become a partisan issue," they said. "The stakes are too high for our country."

A target for break-ins

Sitting in the basement of the Democratic National Committee headquarters, below a wall-size 2012 portrait of a smiling Barack Obama, is a 1960s-era filing cabinet missing the handle on the bottom drawer. Only a framed newspaper story hanging on the wall hints at the importance of this aged piece of office furniture.

"GOP Security Aide Among 5 Arrested in Bugging Affair," reads the headline from the front page of The Washington Post on June 19, 1972, with the bylines of Bob Woodward and Carl Bernstein.

Andrew Brown, 37, the technology director at the DNC, was born after that famous break-in. But as he began to plan for this year's election cycle, he was well aware that the DNC could become a break-in target again.

There were aspirations to ensure that the DNC was well protected against cyberintruders — and then there was the reality, Brown and his bosses at the organization acknowledged: The DNC was a nonprofit group, dependent on donations, with a fraction of the security budget that a corporation its size would have.

"There was never enough money to do everything we needed to do," Brown said.

The DNC had a standard email spam-filtering service, intended to block phishing attacks and malware created to resemble legitimate email. But when Russian hackers started in on the DNC, the committee did not have the most advanced systems in place to track suspicious traffic, internal DNC memos show.

Tamene, who reports to Brown and fielded the call from the FBI agent, was not a full-time DNC employee; he works for a Chicago-based contracting firm called The MIS Department. He was left to figure out, largely on his own, how to respond — and even whether the man who had called in to the DNC switchboard was really an FBI agent.

"The FBI thinks the DNC has at least one compromised computer on its network and the FBI wanted to know if the DNC is aware, and if so, what the DNC is doing about it," Tamene wrote in an internal memo about his contacts with the FBI. He added that "the Special Agent told me to look for a specific type of malware dubbed 'Dukes' by the U.S. intelligence community and in cybersecurity circles."

Part of the problem was that Hawkins did not show up in person at the DNC. Nor could he email anyone there, as that risked alerting the hackers that the FBI knew they were in the system.

Tamene's initial scan of the DNC system — using his less-than-optimal tools and incomplete targeting information from the FBI — found nothing. So when Hawkins called repeatedly in October, leaving voice mail messages for Tamene, urging him to call back, "I did not return his calls, as I had nothing to report," Tamene explained in his memo.

In November, Hawkins called with more ominous news. A DNC computer was "calling home, where home meant Russia," Tamene's memo says, referring to software sending information to Moscow. "SA Hawkins added that the FBI thinks that this calling home behavior could be the result of a state-sponsored attack."

Brown knew that Tamene, who declined to comment, was fielding calls from the FBI. But he was tied up on a different problem: evidence suggesting that the campaign of Sen. Bernie Sanders of Vermont, Clinton's main Democratic opponent, had improperly gained access to her campaign data.

Wasserman Schultz, then the DNC's chairwoman, and Amy Dacey, then its chief executive, said in interviews that neither of them was notified about the early reports that the committee's system had likely been compromised.

Shawn Henry, who once led the FBI's cyber division and is now president of CrowdStrike Services, the cybersecurity firm retained by the DNC in April, said he was baffled that the FBI did not call a more senior official at the DNC or send an agent in person to the party headquarters to try to force a more vigorous response.

"We are not talking about an office that is in the middle of the woods of Montana. We are talking about an office that is half a mile from the FBI office that is getting the notification," Henry said.

"This is not a mom-and-pop delicatessen or a local library. This is a critical piece of the U.S. infrastructure because it relates to our electoral process, our elected officials, our legislative process, our executive process," he added. "To me it is a high-level, serious issue, and if after a couple of months you don't see any results, somebody ought to raise that to a higher level."

[Russia and the U.S. election: What we know and don't know]

The FBI declined to comment on the agency's handling of the hack. "The FBI takes very seriously any compromise of public and private sector systems," it said in a statement, adding that agents "will continue to share information" to help targets "safeguard their systems against the actions of persistent cybercriminals."

By March, Tamene and his team had met at least twice in person with the FBI and concluded that Hawkins was really a federal employee. But then the situation took a dire turn.

A second team of Russian-affiliated hackers began to target the DNC and other players in the political world, particularly Democrats. Billy Rinehart, a former DNC regional field director who was then working for Hillary Clinton's campaign, got an odd email warning from Google.

"Someone just used your password to try to sign into your Google account," the March 22 email said, adding that the sign-in attempt had occurred in Ukraine. "Google stopped this sign-in attempt. You should change your password immediately."

Rinehart was in Hawaii at the time. He remembers checking his email at 4 a.m. for messages from East Coast associates. Without thinking much about the notification, he clicked on the "change password" button and half asleep, as best he can remember, he typed in a new password.

What he did not know until months later is that he had just given the Russian hackers access to his email account.

Hundreds of similar phishing emails were being sent to American political targets, including an identical email sent on March 19 to Podesta, chairman of the Clinton campaign. Given how many emails Podesta received through this personal email account, several aides also had access to it, and one of them noticed the warning email, sending it to a computer technician to make sure it was legitimate before anyone clicked on the "change password" button.

"This is a legitimate email," Charles Delavan, a Clinton campaign aide, replied to another of Podesta's aides, who had noticed the alert. "John needs to change his password immediately."

With another click, a decade of emails that Podesta maintained in his Gmail account — a total of about 60,000 — were unlocked for the Russian hackers. Delavan, in an interview, said that his bad advice was a result of a typo: He knew this was a phishing attack, as the campaign was getting dozens of them. He said he had meant to type that it was an "illegitimate" email, an error that he said has plagued him ever since.

During this second wave, the hackers also gained access to the Democratic Congressional Campaign Committee, and then, through a virtual private network connection, to the main computer network of the DNC.

The FBI observed this surge of activity as well, again reaching out to Tamene to warn him. Yet Tamene still saw no reason to be alarmed: He found copies of the phishing emails in the DNC's spam filter. But he had no reason, he said, to believe that the computer systems had been infiltrated.

One bit of progress had finally been made by the middle of April: The DNC, seven months after it had first been warned, finally installed a "robust set of monitoring tools," Tamene's internal memo says.

Honing stealthy tactics

The United States had two decades of warning that Russia's intelligence agencies were trying to break into America's most sensitive computer networks. But the Russians have always managed to stay a step ahead.

Their first major attack was detected on Oct. 7, 1996, when a computer operator at the Colorado School of Mines discovered some nighttime computer activity he could not explain. The school had a major contract with the Navy, and the operator warned his contacts there. But as happened two decades later at the DNC, at first "everyone was unable to connect the dots," said Thomas Rid, a scholar at King's College in London who has studied the attack.

Investigators gave it a name — Moonlight Maze — and spent two years, often working day and night, tracing how it hopped from the Navy to the Department of Energy to the Air Force and NASA. In the end, they concluded that the total number of files stolen, if printed and stacked, would be taller than the Washington Monument.

Whole weapons designs were flowing out the door, and it was a first taste of what was to come: an escalating campaign of cyberattacks around the world.

But for years, the Russians stayed largely out of the headlines, thanks to the Chinese — who took bigger risks, and often got caught. They stole the designs for the F-35 fighter jet, corporate secrets for rolling steel, even the blueprints for gas pipelines that supply much of the United States. And during the 2008 presidential election cycle, Chinese intelligence hacked into the campaigns of Obama and McCain, making off with internal position papers and communications. But they didn't publish any of it.

The Russians had not gone away, of course. "They were just a lot more stealthy," said Kevin Mandia, a former Air Force intelligence officer who spent most of his days fighting off Russian cyberattacks before founding Mandiant, a cybersecurity firm that is now a division of FireEye — and the company the Clinton campaign brought in to secure its own systems.

The Russians were also quicker to turn their attacks to political purposes. A 2007 cyberattack on Estonia, a former Soviet republic that had joined NATO, sent a message that Russia could paralyze the country without invading it. The next year cyberattacks were used during Russia's war with Georgia.

But American officials did not imagine that the Russians would dare try those techniques inside the United States. They were largely focused on preventing what former Defense Secretary Leon E. Panetta warned was an approaching "cyber Pearl Harbor" — a shutdown of the power grid or cellphone networks.

In 2014 and 2015, a Russian hacking group began systematically targeting the State Department, the White House and the Joint Chiefs of Staff. "Each time, they eventually met with some form of success," Michael Sulmeyer, a former cyberexpert for the secretary of defense, and Ben Buchanan, now both of the Harvard Cyber Security Project, wrote recently in a soon-to-be published paper for the Carnegie Endowment.

The Russians grew stealthier and stealthier, tricking government computers into sending out data while disguising the electronic "command and control" messages that set off alarms for anyone looking for malicious actions. The State Department was so crippled that it repeatedly closed its systems to throw out the intruders. At one point, officials traveling to Vienna with Secretary of State John Kerry for the Iran nuclear negotiations had to set up commercial Gmail accounts just to communicate with one another and with reporters traveling with them.

Obama was briefed regularly on all this, but he made a decision that many in the White House now regret: He did not name Russians publicly, or issue sanctions. There was always a reason: fear of escalating a cyberwar, and concern that the United States needed Russia's cooperation in negotiations over Syria.

"We'd have all these circular meetings," one senior State Department official said, "in which everyone agreed you had to push back at the Russians and push back hard. But it didn't happen."

So the Russians escalated again — breaking into systems not just for espionage, but to publish or broadcast what they found, known as "doxing" in the cyberworld.

It was a brazen change in tactics, moving the Russians from espionage to influence operations. In February, 2014 they broadcast an intercepted phone call between Victoria Nuland, the assistant secretary of state who handles Russian affairs and has a contentious relationship with Putin, and Geoffrey Pyatt, the U.S. ambassador to Ukraine. Nuland was heard describing a little-known American effort to broker a deal in Ukraine, then in political turmoil.

They were not the only ones on whom the Russians used the steal-and-leak strategy. The Open Society Foundation, run by George Soros, was a major target, and when its documents were released, some turned out to have been altered to make it appear as if the foundation was financing Russian opposition members.

[Russians hacked GOP but hid it, say U.S. intelligence agencies]

Last year, the attacks became more aggressive. Russia hacked a major French television station, frying critical hardware. Around Christmas, it attacked part of the power grid in Ukraine, dropping a portion of the country into darkness, killing backup generators and taking control of generators. In retrospect, it was a warning shot.

The attacks "were not fully integrated military operations," Sulmeyer said. But they showed an increasing boldness.

Cozy Bear and Fancy Bear

The day before the White House Correspondents' Association dinner in April, Dacey, the DNC's chief executive, was preparing for a night of parties when she got an urgent phone call.

With the new monitoring system in place, Tamene had examined administrative logs of the DNC's computer system and found something very suspicious: An unauthorized person, with administrator-level security status, had gained access to the DNC's computers.

"Not sure it is related to what the FBI has been noticing," said one internal DNC email sent on April 29. "The DNC may have been hacked in a serious way this week, with password theft, etc."

No one knew just how bad the breach was — but it was clear that a lot more than a single filing cabinet worth of materials might have been taken. A secret committee was immediately created, including Dacey, Wasserman Schultz, Brown and Michael Sussmann, a former cybercrimes prosecutor at the Justice Department who now works at Perkins Coie, the Washington law firm that handles DNC political matters.

"Three most important questions," Sussmann wrote to his clients the night the break-in was confirmed. "1) What data was accessed? 2) How was it done? 3) How do we stop it?"

Sussmann instructed his clients not to use DNC email because they had just one opportunity to lock the hackers out — an effort that could be foiled if the hackers knew that the DNC was on to them.

"You only get one chance to raise the drawbridge," Sussmann said. "If the adversaries know you are aware of their presence, they will take steps to burrow in, or erase the logs that show they were present."

The DNC immediately hired CrowdStrike, a cybersecurity firm, to scan its computers, identify the intruders and build a new computer and telephone system from scratch. Within a day, CrowdStrike confirmed that the intrusion had originated in Russia, Sussmann said.

The work that such companies do is a computer version of old-fashioned crime scene investigation, with fingerprints, bullet casings and DNA swabs replaced by an electronic trail that can be just as incriminating. And just as police detectives learn to identify the telltale methods of a veteran burglar, so CrowdStrike investigators recognized the distinctive handiwork of Cozy Bear and Fancy Bear.

Those are CrowdStrike's nicknames for the two Russian hacking groups that the firm found at work inside the DNC network. Cozy Bear — the group also known as the Dukes or APT 29, for "advanced persistent threat" — may or may not be associated with the FSB, the main successor to the Soviet-era KGB, but it is widely believed to be a Russian government operation. It made its first appearance in 2014, said Dmitri Alperovitch, CrowdStrike's co-founder and chief technology officer.

It was Cozy Bear, CrowdStrike concluded, that first penetrated the DNC in summer 2015, by sending spear-phishing emails to a long list of U.S. government agencies, Washington nonprofits and government contractors. Whenever someone clicked on a phishing message, the Russians would enter the network, "exfiltrate" documents of interest and stockpile them for intelligence purposes.

"Once they got into the DNC, they found the data valuable and decided to continue the operation," said Alperovitch, who was born in Russia and moved to the United States as a teenager.

Only in March 2016 did Fancy Bear show up — first penetrating the computers of the Democratic Congressional Campaign Committee, and then jumping to the DNC, investigators believe. Fancy Bear, sometimes called APT 28 and believed to be directed by the GRU, Russia's military intelligence agency, is an older outfit, tracked by Western investigators for nearly a decade. It was Fancy Bear that got hold of Podesta's email.

Attribution, as the skill of identifying a cyberattacker is known, is more art than science. It is often impossible to name an attacker with absolute certainty. But over time, by accumulating a reference library of hacking techniques and targets, it is possible to spot repeat offenders. Fancy Bear, for instance, has gone after military and political targets in Ukraine and Georgia, and at NATO installations.

That largely rules out cybercriminals and most countries, Alperovitch said. "There's no plausible actor that has an interest in all those victims other than Russia," he said. Another clue: The Russian hacking groups tended to be active during working hours in the Moscow time zone.

To their astonishment, Alperovitch said, CrowdStrike experts found signs that the two Russian hacking groups had not coordinated their attacks. Fancy Bear, apparently not knowing that Cozy Bear had been rummaging in DNC files for months, took many of the same documents.

In the six weeks after CrowdStrike's arrival, in total secrecy, the computer system at the DNC was replaced. For a weekend, email and phones were shut off; employees were told it was a system upgrade. All laptops were turned in and the hard drives wiped clean, with the uninfected information on them imaged to new drives.

Though DNC officials had learned that the Democratic Congressional Campaign Committee had been infected, too, they did not notify their sister organization, which was in the same building, because they were afraid that it would leak.

All this work took place as the bitter contest for the Democratic nomination continued to play out between Clinton and Sanders, and it was already causing a major distraction for Wasserman Schultz and the DNC's chief executive.

"This was not a bump in the road — bumps in the road happen all the time," she said in an interview. "Two different Russian spy agencies had hacked into our network and stolen our property. And we did not yet know what they had taken. But we knew they had very broad access to our network. There was a tremendous amount of uncertainty. And it was chilling."

The DNC executives and their lawyer had their first formal meeting with senior FBI officials in mid-June, nine months after the bureau's first call to the tech-support contractor. Among the early requests at that meeting, according to participants: that the federal government make a quick "attribution" formally blaming actors with ties to Russian government for the attack to make clear that it was not routine hacking but foreign espionage.

"You have a presidential election underway here and you know that the Russians have hacked into the DNC," Sussmann said, recalling the message to the FBI. "We need to tell the American public that. And soon."

The media’s role

In mid-June, on Sussmann's advice, DNC leaders decided to take a bold step. Concerned that word of the hacking might leak, they decided to go public in The Washington Post with the news that the committee had been attacked. That way, they figured, they could get ahead of the story, win a little sympathy from voters for being victimized by Russian hackers and refocus on the campaign.

But the very next day, a new, deeply unsettling shock awaited them. Someone calling himself Guccifer 2.0 appeared on the web, claiming to be the DNC hacker — and he posted a confidential committee document detailing Trump's record and half a dozen other documents to prove his bona fides.

"And it's just a tiny part of all docs I downloaded from the Democrats networks," he wrote. Then something more ominous: "The main part of the papers, thousands of files and mails, I gave to WikiLeaks. They will publish them soon."

It was bad enough that Russian hackers had been spying inside the committee's network for months. Now the public release of documents had turned a conventional espionage operation into something far more menacing: political sabotage, an unpredictable, uncontrollable menace for Democratic campaigns.

[A potential showdown looms between Donald Trump and the CIA]

Guccifer 2.0 borrowed the moniker of an earlier hacker, a Romanian who called himself Guccifer and was jailed for breaking into the personal computers of former President George W. Bush, former Secretary of State Colin L. Powell and other notables. This new attacker seemed intent on showing that the DNC's cyberexperts at CrowdStrike were wrong to blame Russia. Guccifer 2.0 called himself a "lone hacker" and mocked CrowdStrike for calling the attackers "sophisticated."

But online investigators quickly undercut his story. On a whim, Lorenzo Franceschi-Bicchierai, a writer for Motherboard, the tech and culture site of Vice, tried to contact Guccifer 2.0 by direct message on Twitter.

"Surprisingly, he answered right away," Franceschi-Bicchierai said. But whoever was on the other end seemed to be mocking him. "I asked him why he did it, and he said he wanted to expose the Illuminati. He called himself a Gucci lover. And he said he was Romanian."

That gave Franceschi-Bicchierai an idea. Using Google Translate, he sent the purported hacker some questions in Romanian. The answers came back in Romanian. But when he was offline, Franceschi-Bicchierai checked with a couple of native speakers, who told him Guccifer 2.0 had apparently been using Google Translate as well — and was clearly not the Romanian he claimed to be.

Cyberresearchers found other clues pointing to Russia. Microsoft Word documents posted by Guccifer 2.0 had been edited by someone calling himself, in Russian, Felix Edmundovich — an obvious nom de guerre honoring the founder of the Soviet secret police, Felix Edmundovich Dzerzhinsky. Bad links in the texts were marked by warnings in Russian, generated by what was clearly a Russian-language version of Word.

When Franceschi-Bicchierai managed to engage Guccifer 2.0 over a period of weeks, he found that his interlocutor's tone and manner changed. "At first he was careless and colloquial. Weeks later, he was curt and more calculating," he said. "It seemed like a group of people, and a very sloppy attempt to cover up."

Computer experts drew the same conclusion about DCLeaks.com, a site that sprang up in June, claiming to be the work of "hacktivists" but posting more stolen documents. It, too, seemed to be a clumsy front for the same Russians who had stolen the documents. Notably, the website was registered in April, suggesting that the Russian hacking team planned well in advance to make public what it stole.

In addition to what Guccifer 2.0 published on his site, he provided material directly on request to some bloggers and publications. The steady flow of Guccifer 2.0 documents constantly undercut Democratic messaging efforts. On July 6, 12 days before the Republican National Convention began in Cleveland, Guccifer released the DNC's battle plan and budget for countering it. For Republican operatives, it was insider gold.

Then WikiLeaks, a far more established outlet, began to publish the hacked material — just as Guccifer 2.0 had promised. On July 22, three days before the start of the Democratic National Convention in Philadelphia, WikiLeaks dumped out 44,053 DNC emails with 17,761 attachments. Some of the messages made clear that some DNC officials favored Clinton over her progressive challenger, Sanders.

That was no shock; Sanders, after all, had been an independent socialist, not a Democrat, during his long career in Congress, while Clinton had been one of the party's stars for decades. But the emails, some of them crude or insulting, infuriated Sanders delegates as they arrived in Philadelphia. Wasserman Schultz resigned under pressure on the eve of the convention where she had planned to preside.

Trump, by now the Republican nominee, expressed delight at the continuing jolts to his opponent, and he began to use Twitter and his stump speeches to highlight the WikiLeaks releases. On July 25, he sent out a lighthearted tweet: "The new joke in town," he wrote, "is that Russia leaked the disastrous DNC e-mails, which should never have been written (stupid), because Putin likes me."

But WikiLeaks was far from finished. On Oct. 7, a month before the election, the site began the serial publication of thousands of private emails to and from Podesta, Clinton's campaign manager.

The same day, the United States formally accused the Russian government of being behind the hackings, in a joint statement by the director of national intelligence and the Department of Homeland Security, and Trump suffered his worst blow to date, with the release of a recording in which he bragged about sexually assaulting women.

The Podesta emails were nowhere near as sensational as the Trump video. But, released by WikiLeaks day after day over the last month of the campaign, they provided material for countless news reports. They disclosed the contents of Clinton's speeches to large banks, which she had refused to release. They exposed tensions inside the campaign, including disagreements over donations to the Clinton Foundation that staff members thought might look bad for the candidate and Tanden's complaint that Clinton's instincts were "suboptimal."

"I was just mortified," Tanden said in an interview. Her emails were released on the eve of one of the presidential debates, she recalled. "I put my hands over my head and said, 'I can't believe this is happening to me.'" Though she had regularly appeared on television to support Clinton, she canceled her appearances because all the questions were about what she had said in the emails.

Tanden, like other Democrats whose messages became public, said it was obvious to her that WikiLeaks was trying its best to damage the Clinton campaign. "If you care about transparency, you put all the emails out at once," she said. "But they wanted to hurt her. So they put them out 1,800 to 3,000 a day."

The Trump campaign knew in advance about WikiLeaks' plans. Days before the Podesta email release began, Roger Stone, a Republican operative working with the Trump campaign, sent out an excited tweet about what was coming.

But in an interview, Stone said he had no role in the leaks; he had just heard from an American with ties to WikiLeaks that damning emails were coming.

Julian Assange, the WikiLeaks founder and editor, has resisted the conclusion that his site became a pass-through for Russian hackers working for Putin's government or that he was deliberately trying to undermine Clinton's candidacy. But the evidence on both counts appears compelling.

In a series of email exchanges, Assange refused to say anything about WikiLeaks' source for the hacked material. He denied that he had made his animus toward Clinton clear in public statements ("False. But what is this? Junior high?") or that the site had timed the releases for maximum negative effect on her campaign. "WikiLeaks makes its decisions based on newsworthiness, including for its recent epic scoops," he wrote.

Assange disputed the conclusion of the Oct. 7 statement from the intelligence agencies that the leaks were "intended to interfere with the U.S. election process."

"This is false," he wrote. "As the disclosing party we know that this was not the intent. Publishers publishing newsworthy information during an election is part of a free election."

But asked whether he believed the leaks were one reason for Trump's election, Assange seemed happy to take credit. "Americans extensively engaged with our publications," he wrote. "According to Facebook statistics WikiLeaks was the most referenced political topic during October."

Though Assange did not say so, WikiLeaks' best defense may be the conduct of the mainstream American media. Every major publication, including The Times, published multiple stories citing the DNC and Podesta emails posted by WikiLeaks, becoming a de facto instrument of Russian intelligence.

Putin, a student of martial arts, had turned two institutions at the core of American democracy — political campaigns and independent media — to his own ends. The media's appetite for the hacked material, and its focus on the gossipy content instead of the Russian source, disturbed some of those whose personal emails were being reposted across the web.

"What was really surprising to me?" Tanden said. "I could not believe that reporters were covering it."

Devising a government response

Inside the White House, as Obama's advisers debated their response, their conversation turned to North Korea.

In late 2014, hackers working for Kim Jong Un, the North's young and unpredictable leader, had carried out a well-planned attack on Sony Pictures Entertainment intended to stop the Christmastime release of a comedy about a CIA plot to kill Kim.

In that case, embarrassing emails had also been released. But the real damage was done to Sony's own systems: More than 70 percent of its computers melted down when a particularly virulent form of malware was released. Within weeks, intelligence agencies traced the attack back to the North and its leadership. Obama called North Korea out in public, and issued some not-very-effective sanctions. The Chinese even cooperated, briefly cutting off the North's internet connections.

As the first Situation Room meetings on the Russian hacking began in July, "it was clear that Russia was going to be a much more complicated case," said one participant. The Russians clearly had a more sophisticated understanding of American politics, and they were masters of "kompromat," their term for compromising information.

But a formal "attribution report" still had not been forwarded to the president.

"It took forever," one senior administration official said, complaining about the pace at which the intelligence assessments moved through the system.

In August a group that called itself the "Shadow Brokers" published a set of software tools that looked like what the NSA uses to break into foreign computer networks and install "implants," malware that can be used for surveillance or attack. The code came from the Tailored Access Operations unit of the NSA, a secretive group that mastered the arts of surveillance and cyberwar.

The assumption — still unproved — was that the code was put out in the open by the Russians as a warning: Retaliate for the DNC, and there are a lot more secrets, from the hackings of the State Department, the White House and the Pentagon, that might be spilled as well. One senior official compared it to the scene in "The Godfather" where the head of a favorite horse is left in a bed, as a warning.

The NSA said nothing. But by late August, Rogers, its director, was pressing for a more muscular response to the Russians. In his role as director of the Pentagon's Cyber Command, he proposed a series of potential counter-cyberstrikes.

While officials will not discuss them in detail, the possible counterstrikes reportedly included operations that would turn the tables on Putin, exposing his financial links to Russia's oligarchs, and punching holes in the Russian internet to allow dissidents to get their message out. Pentagon officials judged the measures too unsubtle and ordered up their own set of options.

But in the end, none of those were formally presented to the president.

In a series of "deputies meetings" run by Avril Haines, the deputy national security adviser and a former deputy director of the CIA, several officials warned that an overreaction by the administration would play into Putin's hands.

"If we went to DEFCON 4," one frequent participant in Haines' meetings said, using a phrase from the Cold War days of warnings of war, "we would be saying to the public that we didn't have confidence in the integrity of our voting system."

Even something seemingly straightforward — using the president's executive powers, bolstered after the Sony incident, to place economic and travel sanctions on cyberattackers — seemed too risky.

"No one was all that eager to impose costs before Election Day," said another participant in the classified meeting. "Any retaliatory measures were seen through the prism of what would happen on Election Day."

Instead, when Obama's national security team reconvened after summer vacation, the focus turned to a crash effort to secure the nation's voting machines and voter-registration rolls from hacking. The scenario they discussed most frequently — one that turned out not to be an issue — was a narrow vote in favor of Clinton, followed by a declaration by Trump that the vote was "rigged" and more leaks intended to undercut her legitimacy.

Donna Brazile, the interim chairwoman of the DNC, became increasingly frustrated as the clock continued to run down on the presidential election — and still there was no broad public condemnation by the White House, or Republican Party leaders — of the atttack as an act of foreign espionage.

Brazile even reached out to Reince Priebus, the chairman of the Republican National Committee, urging him twice in private conversations and in a letter to join her in condemning the attacks — an offer he declined to take up.

"We just kept hearing, the government would respond, the government would respond," she said. "Once upon a time, if a foreign government interfered with our election we would respond as a nation, not as a political party."

But Obama did decide that he would deliver a warning to Putin in person at a Group of 20 summit meeting in Hangzhou, China, the last time they would be in the same place while Obama was still in office. When the two men met for a tense pull-aside, Obama explicitly warned Putin of a strong American response if there was continued effort to influence the election or manipulate the vote, according to White House officials who were not present for the one-on-one meeting.

Later that day, Obama made a rare reference to the United States' own offensive cybercapacity, which he has almost never talked about. "Frankly, both offensively and defensively, we have more capacity," he told reporters.

But when it came time to make a public assertion of Russia's role in early October, it was made in a written statement from the director of national intelligence and the secretary of homeland security. It was far less dramatic than the president's appearance in the press room two years before to directly accuse the North Koreans of attacking Sony.

The reference in the statement to hackings on "political organizations," officials now say, encompassed a hacking on data stored by the Republicans as well. Two senior officials say the forensic evidence was accompanied by "human and technical" sources in Russia, which appears to mean that the United States' implants or taps in Russian computer and phone networks helped confirm the country's role.

But that may not be known for decades, until the secrets are declassified.

A week later Vice President Joe Biden was sent out to transmit a public warning to Putin: The United States will retaliate "at the time of our choosing. And under the circumstances that have the greatest impact."

Later, after Biden said he was not concerned that Russia could "fundamentally alter the election," he was asked whether the American public would know if the message to Putin had been sent.

"Hope not," Biden responded.

Some of his former colleagues think that was the wrong answer. A U.S. counterstrike, said Michael Morell, the former deputy director of the CIA under Obama, has "got to be overt. It needs to be seen."

A covert response would significantly limit the deterrence effect, he added. "If you can't see it, it's not going to deter the Chinese and North Koreans and Iranians and others."

The Obama administration says it still has more than 30 days to do exactly that.

The next target

As the year draws to a close, it now seems possible that there will be multiple investigations of the Russian hacking — the intelligence review Obama has ordered completed by Jan. 20, the day he leaves office, and one or more congressional inquiries. They will wrestle with, among other things, Putin's motive.

Did he seek to mar the brand of American democracy, to forestall anti-Russian activism for both Russians and their neighbors? Or to weaken the next American president, since presumably Putin had no reason to doubt American forecasts that Clinton would win easily? Or was it, as the CIA concluded last month, a deliberate attempt to elect Trump?

In fact, the Russian hack-and-dox scheme accomplished all three goals.

What seems clear is that Russian hacking, given its success, is not going to stop. Two weeks ago, the German intelligence chief, Bruno Kahl, warned that Russia might target elections in Germany next year. "The perpetrators have an interest to delegitimize the democratic process as such," Kahl said. Now, he added, "Europe is in the focus of these attempts of disturbance, and Germany to a particularly great extent."

But Russia has by no means forgotten the American target. On the day after the presidential election, the cybersecurity company Volexity reported five new waves of phishing emails, evidently from Cozy Bear, aimed at think tanks and nonprofits in the United States.

One of them purported to be from Harvard University, attaching a fake paper. Its title: "Why American Elections Are Flawed."