WASHINGTON - Cybersecurity experts knew Russian operatives might try to tamper with the nation's electronic voting systems. Many people inside the U.S. government and the Obama White House also knew.
In the summer of 2016, a cluster of volunteers on a federally supervised cybersecurity team crafting 2018 election guidelines felt compelled to do something sooner. Chatting online, they scrambled to draw up ways for state and local officials to patch the most obvious cyber vulnerabilities before Election Day 2016.
Their five-page list of recommendations focused on two gaping holes in the U.S. election system. It warned that internet voting by at least some citizens in 32 states was not secure and should be avoided. And, critically, it advised how to guard voting and ballot-counting machines that the experts knew could be penetrated even when disconnected from the internet.
But the list was stopped in its tracks. A year later, even as U.S. intelligence agencies warn that Russian operatives have their eyes on 2018 and beyond, America's more than 7,000 election jurisdictions nationwide still do not have access to those guidelines for shielding the voting process.
The recommendations were derailed amid an awkward, often unspoken power struggle between, on one end, federal agencies, which have more resources to combat cyberthreats, and on the other, states and localities, which hold absolute constitutional authority over elections.
The states vigorously defend their territory, though they can be naive about cyber risks. Many have insisted their systems are secure.
For their part, federal officials have hesitated to encroach on that turf with the election just around the corner.
Both sides showed a "lack of seriousness" about voting security issues that spells trouble for protecting the nation's jumble of election machinery against increasingly sophisticated threats, warned Neal McBurnett, a Boulder, Colo., consultant who helped develop the guidelines.
State and federal authorities aren't moving fast enough "in coming up with ways to harden our targets and look at the problem with clear eyes," he said.
Cybersecurity experts often blame state and local officials for the lack of action.
That includes leaders of the National Association of Secretaries of State, or NASS, which has concerns about undermining public confidence in voting systems. The leaders have insisted computer-driven equipment is secure when it's not hooked to the internet — which is wrong. And most NASS members represent states that permit internet voting, mainly by military and overseas voters — another vulnerability.
As for the feds, among the most vital things they can do is share intelligence about cyber threats and provide national cybersecurity expertise that no state can be expected to produce. That's where the U.S. government appears to have failed in 2016.
As the working group met last summer, the FBI had already begun sending out "flash alerts" to election officials nationwide about attempted penetrations of statewide electronic voter registration databases. Homeland Security officials gave similar warnings. In Illinois, they and FBI agents examined the illegal download of records from 200,000 voters.
Attempted intrusions were discovered in Arizona and at least 19 other states.
Federal officials linked the attempted Arizona hack to Russia, and cyber experts publicly blamed the Kremlin for a major hack of the Democratic National Committee that exposed, with the help of transparency site WikiLeaks, embarrassing internal e-mails.
The hacks of voter registration databases had demonstrated that voting jurisdictions, many operating with equipment more than a decade old, had few defenses against these cyber perils. The tiny Election Assistance Commission (EAC), which plays a key role in delivering federal funding and election guidance to state and local agencies, and Homeland Security had responded to those hacks by issuing new guidelines for protecting registration data, as well as systems for reporting vote totals on election night.
But they did little to safeguard the voting equipment itself.
In the online cybersecurity working group, several experts prepared guidelines for a formal committee led by the EAC and the National Institute of Standards and Technology (NIST), which provides cyber expertise to federal agencies.
On Aug. 7, 2016, David Wagner, a University of California, Berkeley computer science professor who had a lead role on the working group, wrote in an e-mail: "I'd like to push to see if we can get out something very soon, to provide as a resource for election officials preparing for elections this November. That means we need to move quickly."
E-mail chains and other records show that, with NIST fully in the loop, the group hurried to prepare the guidelines with the assumption they would be circulated before the election.
But three weeks later, on Aug. 30, NIST pulled the plug. No distribution would be formally considered in 2016 because it was too close to the election, NIST official Andrew Regenscheid told Susan Greenhalgh, a watchdog at the nonprofit Verified Voting who shepherded completion of the recommendations in the working group. Greenhalgh, who said she was stunned, confirmed the decision a couple of days later in a phone call with the head of NIST's voting unit.
"I told them I thought they were making a big mistake," Greenhalgh said.
From that moment until Election Day, Russia completed what one computer security expert privately described as a "cyber Pearl Harbor."
Meanwhile, many states and counties nationwide opted to allow federal reviews of their cyber hookups in the fall of 2016. They revealed widespread vulnerabilities. In South Carolina alone, National Guard cyber specialists found at least "high" risks in all 46 counties evaluated, 20 of which had issues identified as critical, according to public records obtained by University of South Carolina computer scientist Duncan Buell and Frank Heindl, a Charleston activist.
Any number of routes can lead to disaster, including denial.
Last April, Denise Merrill, Connecticut's Democratic Secretary of State and then the president of NASS, testified that the vast majority of U.S. voting systems "are not cyber" because they're not connected to the internet.
"The 2016 cycle demonstrated we're not really cyber at all, except for our voter registration databases, which have nothing to do with the actual tallying of votes," she said.
Other NASS officials have echoed that erroneous belief.
The lead item in the working group's shelved guidelines confronted such claims, seeking to demolish perceptions that a system must be "continually connected to the internet to expose it to (an) online actor or that the system must be connected to the internet at the time the votes are being counted for the attack to be successful."
If a vote-counting system is on an internal network in which any component is hooked to the internet, it creates "an exploitable" situation for hackers, the guidelines say, urging states and counties to map their networks to be sure they are fully offline and to take other precautions to avoid infection.
NASS didn't provide detailed responses to McClatchy's questions. Executive Director Leslie Reynolds said in a statement that the organization has worked closely with the EAC and Homeland Security over the last year "to share security information with the states as soon as it is available."
James Scott, co-founder of the Institute for Critical Infrastructure Technology, and others note that the CIA has been credited with hacking the Iranian nuclear weapons system while it was off the internet in 2010 by circulating malware-tainted thumb drives, at least one of which eventually was plugged into the system. The so-called Stuxnet virus caused centrifuges used to enrich uranium gas to fail, stalling the weapons program.
A chemical engineer's recent account described how part of a large European petrochemical company was crippled by a virus that migrated onto its internal network through coffee machines connected to the internet.
If hackers gain access to a machine, they can plant malware that does its dirty work and then "auto deletes," leaving behind no noticeable trace.
Absent witnesses or a confession, election officials could uncover vote rigging only if the affected jurisdiction required backup paper copies of electronic ballots so that postelection auditors could verify each candidate's totals.
EAC Chairman Matthew Masterson, who acknowledges that attempted cyberattacks on election systems are "persistent and sophisticated," says about 25 percent of the votes cast in the 2016 election lacked paper backups.
A decade ago, when Congress tried to enact the most obvious solution to that problem — a law requiring all electronic voting machines to have a "verifiable paper trail" — state and local officials largely opposed it.
Beyond the voting machines themselves, other dangers lurk: Scott, of the Institute for Critical Infrastructure Technology, said his group warned NASS last year that bad actors were likely to try to infect vote-tallying equipment through vendors.
"We told them and we told them," he said. "We showed them two schematics of exactly where the attacks would come from" months before the election.
Scott said, for example, that hackers could embed malware in a routine software upgrade before it was distributed to client agencies, which could then find its way into a central vote-tallying machine and instruct it to switch votes until one candidate led by a specified margin.
The National Security Agency last year detected a Russian attack on Florida-based VR Systems, which makes software to manage huge state voter registration databases. In VR's files, the hackers found an e-mail list of election agencies across the country and sent out Trojan Horse-type e-mails to try to lure election officials into providing access to their databases.
Just this month, it was disclosed that personal data for 1.8 million Chicago voters stored on a cloud computing server, including some driver's license numbers, was left exposed on the internet by the nation's largest voting systems vendor, Nebraska-based Election Systems & Software. ES&S said it secured the files as soon as it was alerted to the mishandling of Chicago Elections Board data on Aug. 12.
Jon Hendren, Upguard's strategy director, told USA Today that the breach included encrypted passwords for ES&S employee accounts. He said that in the worst case, "they could be completely infiltrated right now."
Officials at the two federal agencies who oversaw the cyber working group say there was nothing surprising or untoward about their holding back the group's recommendations.
The working group's guidelines were meant for consideration for 2018, not 2016, said Mary Brady, head of NIST's voting unit.
"These weren't official recommendations," she said. "(W)e did not try to block any efforts."
Instead, when the EAC-NIST committee met in September 2016, Brady, EAC Chairman Masterson and Wagner advised members it was too late to employ the cyber volunteers' recommendations for an election 53 days away.
Masterson also said in an interview that the group's guidelines were "largely captured in guidance we've provided elsewhere." But best practices posted for state and local officials on EAC's and Homeland Security's websites do not contain the working group's central recommendations.
Some state and local officials wanted the guidelines out. On learning at the September meeting that they wouldn't be circulated, Robert Giles, head of New Jersey's Division of Elections, asked what would be needed "to bring the (Department of) Homeland Security and the FBI and whoever else we need to bring in, in a very short time period?"
Before ending his term as Homeland Security secretary last January, Jeh Johnson sought to give impetus to the new security concern by designating the nation's voting equipment as "Critical Infrastructure," making it a top priority to protect those systems — but only when requested to do so by states and localities.
A month later, NASS took its own formal action. It passed a resolution calling on newly inaugurated President Donald Trump to rescind the declaration, saying that "the U.S. Department of Homeland Security has no authority to interfere with elections, even in the name of national security."