Last year, a malicious strain of code called Mirai recruited millions of security cameras, routers, and other gadgets in a digital assault on servers that act like the internet’s switchboard. The attack crippled websites in large swathes of the United States, a spectacular display of the security holes in the Internet of Things.
After the dust settled, a Chinese electronics firm recalled 4.3 million cameras using its circuit boards, which the so-called botnet had enlisted to carry out the attack. Xiongmai Technologies could not fix the security flaws remotely, so it recalled the offending devices and sent out an update for customers to install in newer models.
The episode is a textbook example of how not to repair security flaws in connected devices, according to security researchers. That response would never work in smart cities, they say, where millions of connected sensors will be embedded in street lights, traffic signals, and even roads to gather data on traffic and even crime.
These devices will never be perfectly secure. But it is vital, security researchers say, to have a way to monitor millions of devices and push out automatic updates, which may be too expensive to install manually. The software must also verify authorized updates to block hackers from dropping in malicious code.
“People want to deploy something like an internet connected trash can and they don’t realize that they’re going to have to update it every month, just like you update your laptop,” said Tom Cross, chief technology officer of security firm Drawbridge Networks and a former manager of IBM's Security X-Force Research unit.
The issue is that most software for connected sensors and other devices cannot be patched, Cross said in a smart cities panel at the recent South by Southwest festival in Austin, Texas. "Something will be out there for 20 years and vulnerabilities will be disclosed for it and there’s no way to get it fixed.”
Even though smart city devices could be built to last for decades, manufacturers are not making software that can receive updates, said Robert Hansen, another panelist and founder of security research firm Outside Intel. With a system for automatic updates, engineers can patch software as vulnerabilities arise and code new capabilities.
The need for automatic updates was made clear in 2014 when the research firm IoActive found that hundreds of thousands of traffic control sensors in New York, Washington, D.C., and other cities could accept unauthorized updates. The sensors were embedded in streets to detect cars at traffic lights and optimize when the lights change.
A series of connected building switches. (Image courtesy of Jens Braune del Angel, Creative Commons).
Cesar Cerrudo, IoActive's chief technology officer, found that hackers could potentially break into the sensors and hijack the lights, causing accidents or severe traffic. In the end, the company that makes the sensors, Sensys Networks, issued a patch to customers later that same year. Luckily, local officials could push out the updates wirelessly, which kept them from digging up the sensors.
"If you have to dig up a city to upgrade something, it’s probably not going to happen,” Hansen said.
So far the drive to patch flaws automatically has been slow. Google said that its Android Things operating system will give developers an “infrastructure” to refresh firmware in household devices when it is released this year. In October, ARM introduced a software platform called mbed Cloud that lets developers manage, monitor, and update firmware inside devices remotely.
Others are thinking along the same lines. A technology start-up called Particle hopes that sales of its microcontrollers will lure developers to its cloud, which offers tools for sending over-the-air updates. Resin, a four-year-old company, has devised a platform using containers to develop and manage embedded code, while Mender released last month an open-source system with similar objectives.
The automotive industry is also considering changes to how it mends software in vehicles. Tesla was the first to send out new versions of its software to patch security flaws and upgrade its driving systems. Now, companies from Harman to Movimento have started offering “reflashing” services for connected cars, which is causing highway officials to rethink how recalls are issued.
But in the view of security researchers, manufacturers are still skimping on security because it is seen as too expensive. Not only that but most companies will not continue updating devices after models are released, said Hansen. Engineers also might not want to spare battery life in remote sensors for large security updates.
“Like all engineering, it’s all about trade-offs,” said Hal Kurkowski, managing director of security at Maxim Integrated, in an interview last year about embedded security. “When you go to the hardware store, there’s not just one lock hanging in the aisle where you buy locks.”
But most industry groups believe that automatic updates can't be sacrificed. The Industrial Internet Consortium, for instance, has advised rolling out very small updates to help save bandwidth and battery life. The group recently published a security framework for devices used in everything from infrastructure to oil processing plants.
The Department of Homeland Security also underlined the pitfalls of manual security updates for smart cities in a 2015 report. The report warned that "the impact of the exploitation of a vulnerability may be understood but the risk and consequence to the infrastructure and its connected components is not."
"The confluence of rapid technology evolution and the unknown trajectory of its adoption create even greater future uncertainty,” the report said.
Key to securing smart cities is not treating infrastructure like smartphones or laptops, whose hardware and software are improved with every generation. The software in these embedded devices must be constantly upgraded, updated, and patched over its entire lifetime, Cross said.
“Generally what companies like to do is ship something and then refine. They say, we will get this thing out there, let people buy it, and then we’ll work on improving the quality of it,” he said. “But really, security testing is a kind of quality assurance testing.”
Update April 3, 2017: An update to this article clarified that Sensys Networks had issued a patch for its road sensors, which had been shown to be vulnerable to hackers. Local officials could install those patches, issued in 2014, automatically over wireless networks. A video showing how workers pry the sensors from the road was also removed for clarity.