When Updates go Bad
In the space of less than a week, software updates caused two major system and network outages in Japan and China, and a less major outage over the weekend.
In Japan, a set of updates to Cisco routers led to a network-wide failure for the NTT East and West networks. Up to 4,000 routers were affected by an update that led to router tables (what tells the network traffic where the next hop in the path to the end destination is) being rewritten on each device. This rewrite caused the routing tables to fail, and ultimately the devices stopped forwarding network traffic (effectively self-DoS'ing). The outage, from Tuesday night until Wednesday morning, left millions of users without Internet access and has led to the network provider considering the use of a heterogenous network structure (i.e. multivendor devices, rather than Cisco-only - most likely to introduce Juniper devices).
It appears that some of the cause for the incident was the use of routers without sufficient spare capacity, which meant that when a handful of devices suffered resource exhaustion, it led to a cascading failure as subsequent routers quickly ran out of capacity while trying to identify new routes for network traffic.
Chinese Windows XP SP2 users who regularly maintain their system by applying the latest patches from Microsoft and running Norton-branded antivirus tools from Symantec (Norton AntiVirus, Norton 360, Norton Internet Security) found that a definitions file delivered by Symantec late last week left their systems unusable. The rogue update misidentified two critical system files as being part of the Haxdoor trojan - a very nasty piece of malware that has been causing significant problems for Windows users globally. The quarantining of the critical files left systems unbootable, even in safe mode, requiring significant effort to rebuild and recover the systems.
According to Chinese reports, several thousand users were affected, mainly corporate users. Symantec quickly released an updated definitions file which no longer quarantined the system files, so the current updates should be safe for ongoing use.
On Saturday, it was then reported that Kaspersky Antivirus was identifying files from Rising Antivirus (a popular Chinese antivirus product) as part of a malicious trojan, thus disabling the product. While it requires both antivirus products to be present and in use, that configuration is a common setup - according to the Chinese Incident Security Response Team.
These events aren't the first time that rogue updates from Antivirus vendors have led to significant system damage or outages, and it is sure to not be the last time, either. Incidents like this lead to users being conditioned against installing the latest system and critical software updates (Windows and OS X updates have also been known to break systems) for fear of having their system rendered inoperable from a rogue updae. This extends the exploitation window that attackers can make use of to compromise vulnerable systems.
21 May 2007
Social bookmark this page at eKstreme.
Alternatively, Bookmark or Share via AddThis
Do you like how we cover Information Security news? How about checking out our company services, delivered the same way our news is.
Let our Free OS X Screen Saver deliver the latest security alerts and commentary to your desktop when you're not at your system.
Comments will soon be available for registered users.
Subscribe to our feed.