From Staff
A latent bug from July took down Outlook. A router swap in Arkansas grounded the country's air traffic for more than four hours. A software fault in Chase's own data centers stalled transactions at banks across the country. An internal update at Apple knocked out retail stores on two continents. A patch brought down the National Security Agency's classified network. And somewhere underneath all of it, attackers spent the better part of three weeks using a pair of critical flaws in the programming language that runs most of the web to walk through firewall gateways and wipe servers.
The instinct to connect all of this into a single event is understandable and, on the available record, wrong. But so is the reflexive announcement that nothing is connected. Those are two different claims, and only one of them is being made by people who have looked.
The trouble began on November 12, when Microsoft 365 services went down globally. The company's own account, published four days later, is worse than the outage. A fix prepared specifically for its networking gateway servers began rolling out in July. It ran worldwide for more than three months with no visible impact. Microsoft's load testing did not catch the flaw because the tests ran on older hardware with fewer processor cores, and the defect gets worse on newer machines with more cores. Fifteen hours before the failure, Teams failed over from the backup gateway to the primary one that also serves Exchange Online, raising the load. Roughly 14 percent of capacity was already out of service from an unrelated problem the day before. The result was a cascading failure that left more than 80 percent of gateway servers unresponsive, and the partitions meant to keep a regional problem from going global did not hold. Teams, OneDrive, SharePoint and the Office apps could route around the damage. Exchange Online could not. That is why Outlook stayed down longest.
On November 13, the Federal Aviation Administration issued a nationwide ground stop at about 3:30 in the afternoon Eastern time and lifted it at 8:15 that evening. Seventeen of the agency's 22 air route traffic control centers went unstable. Data was flooding the network at roughly 1,000 connection attempts every four to five seconds. Two sites were implicated: a radar facility in Amarillo, Texas, and a radio site in El Dorado, Arkansas, where the contractor L3Harris was replacing routers that date to the 2001 to 2006 period with new Juniper equipment. Investigators say a multicast or broadcast feature that is not enabled by default somehow got turned on while the new router was talking to two decade old Cisco gear. The FAA paused further Juniper installations and disabled that functionality on more than 60 other routers. HPE says its equipment was working as configured. The Transportation Secretary blamed the old Cisco routers. The agency's own question and answer document blamed the legacy transmission system. Nobody has explained how the feature was enabled in the first place. The agency spends about a billion dollars a year maintaining a network built in the 1980s, and has installed 400 of 3,648 planned replacement routers, of which 69 are operational.
On November 16, JPMorgan Chase reported a temporary issue tied to a software bug in hardware systems inside its own facilities. The bank said there was no evidence of anything cyber related and no systemic problem, and that other banks were affected only as counterparties to transactions. Zelle payments slowed. Bank of America, Wells Fargo, U.S. Bank, Truist and PNC all reported knock on delays while denying direct impact. Normal operations resumed that evening.
The same day, United Airlines lost the system pilots use to calculate final weight and balance figures. Flights already airborne and regional United Express flights were unaffected. Accounts differ on duration, with the airline describing roughly half an hour and fewer than 150 flights, and the Associated Press reporting more than 90 minutes. United said there was no indication of a cybersecurity problem and never disclosed what actually broke.
On November 17, an internal Apple update took down Apple Pay, iCloud, the App Store, Apple Music, AppleCare, and the business and school management services. It also took down the company's own employee portal, its internal sign on system, remote access, and the internal website that tracks outages. Stores could not complete purchases. Everything was restored by early evening.
That same day, the NSA's classified internal network went down. The agency called it temporary degradation during scheduled routine maintenance and said mission critical capabilities remained available. The Director of National Intelligence, Jay Clayton, told Fox Business something more specific: patching that followed the PHP vulnerability had caused systems not to come back up as expected, and that is what happened at the agency. Staff watched the internal alert level move from code yellow to code green to code orange in a single day before Fort Meade closed to most employees. The New York Times reported the network had been unstable for weeks.
What PHP2Shell actually is. Two memory safety flaws in PHP, disclosed with patches on October 21 and scored 9.2 out of 10 under the CVSS version 4 scale. One corrupts internal state when an allocation fails, leaving an unfinished data structure that the cleanup routine later treats as valid. The other leaves stale state from one request pointing into freed memory, which a later request handled by the same worker can follow. Neither requires a login. A malicious request is processed by the language runtime before any application code executes, which means login pages, web application firewalls and application level authentication do nothing to stop it. Only patching closes it.
Fixed releases are PHP 8.2.35, 8.3.36, 8.4.27 and 8.5.12. Versions 7.4 through 8.1 received minimal source backports with no official release. Versions 5.4 through 7.3 are end of life and will never be fixed. The earliest affected code path dates to PHP 5.4.0.
The flaws were found with the help of frontier AI models used by researchers at Wiz and at Calif, the latter as part of an OpenAI effort to hunt vulnerabilities in critical open source code. Wiz has called them the most severe vulnerabilities ever discovered with AI. That claim deserves more scrutiny than it has received.
Amazon Threat Intelligence observed exploitation attempts against its honeypot network on October 17. That is four days before public disclosure and before patches existed. Amazon has said it is unclear how the attacker obtained exploit information. CISA added both flaws to its Known Exploited Vulnerabilities catalog on October 26 and later warned that attackers were chaining them with a Linux kernel flaw to gain root. Cloudflare recorded millions of attempts to weaponize them after disclosure.
The industry framing is that AI gave defenders a window, that the good guys found these first. The October 17 exploitation undermines that. Either the window was already open or the claim is marketing. The same capability that finds bugs at internet scale also adapts working exploits across PHP versions, configurations and memory layouts in hours, and the labs only advertise one direction.
The destructive attacks documented by Check Point Research are the part of this story that deserves the most attention, and they had nothing to do with the outages above.
Beginning October 30, an attacker scanned and exploited organizations running unpatched Check Point security gateways, entering through Mobile Access portals using PHP2Shell. The first wave targeted Israeli organizations and included destructive activity. A second, broader wave on November 2 hit organizations across multiple countries and sectors with no destructive activity observed.
Once inside, the attackers used a separate flaw in the DynamicCLI interface to move from the portal service account to root. They dropped PHP webshells into the portal templates directory behind a hardcoded key, installed a backdoor that persists under a disguised name, and injected a networking component into legitimate appliance processes so that attacker traffic slid past ordinary monitoring. They ran a scanner to map internal assets. They then reached a VMware ESXi host over SSH using the vpxuser service account that vCenter uses for host management, staged a wiper in the temporary directory, zeroed files before deleting them, and left behind a politically themed note rejecting ransom negotiations.
That last detail matters more than any vendor advisory in circulation. The hypervisor was not compromised through a VMware vulnerability. It was reached with a working service account, from a security appliance that should never have had an interactive session to it.
Attribution is genuinely unclear. The tooling shares lineage and cryptographic material with a malware family tied to a China linked group known as UNC5221, the infrastructure sat on a hosting provider previously associated with Chinese activity, and the scanning hours line up with a Beijing workday. But the destruction, the focus on Israel, and the internal naming do not fit that group's known motives. Check Point leaves open that the actor ran the campaign, served as an access broker, or staged a false flag.
Check Point's remediation is the November Jumbo Hotfix for management servers and gateways, prioritized for any internet facing Mobile Access portal. Signs of compromise include session files containing a distinctive upload progress marker, unexpected daemon processes, unexplained gaps in firewall logs, and SSH activity from a gateway to an internal ESXi host.
The precautionary wave on October 21 and 22 was far larger than most summaries suggest. In the House of Lords, the technology minister Baroness Lloyd of Effra reported 32 government services unavailable and four partially unavailable, attributing the disruption largely to precautionary measures. The Office for Budget Responsibility took its site down a week before the Budget. Councils including Edinburgh, Oxford, Bristol, Lambeth and Barnet went offline. In Scotland, at least seven of 32 councils and six of 14 health boards pulled their sites, along with the Care Inspectorate. Germany, France, Belgium and Finland shut down government services. Finland's state ICT center took down public services including the national land survey. Australia's own cybersecurity agency site went dark. The Wikimedia Foundation took down a number of systems, leaving parts of Wikipedia unreachable, and the BBC's own program pages went offline.
The Netherlands took down Overheid.nl, MijnOverheid.nl and DigiD. DigiD is the national digital identity system used for tax filing, health portals and benefits. A government does not switch off its own front door for routine maintenance. It does it when the risk of leaving it open exceeds the cost of locking citizens out.
The UK's National Cyber Security Centre was briefed by OpenAI the week before disclosure and stood up an operation across departments before the public announcement, then sent 50,000 warnings to organizations.
Nepal's disruption illustrates the cost better than anything else in the record, and one detail in circulation is wrong: the shuttered system was not run by the customs department. It is the Vehicle Consignment Tracking System, operated by the Department of Revenue Investigation to track freight vehicles for revenue enforcement. With it offline, drivers and businesses had to carry physical bills and invoices.
Thousands of government sites went dark at once because most of them run on a single shared platform, which is now being audited centrally. Nepal Rastra Bank suspended its site and web services and told banks, payment operators and remittance companies to tighten monitoring. The Employees Provident Fund and the Social Security Fund went offline too. Nabil, Sanima and Nepal Investment Mega Bank restored service after patching. Mobile and QR payments kept working. Internet banking and anything tied to a web server did not. The national cyber center's director, Raj Kumar Maharjan, said the shutdowns were precautionary and that no unauthorized access or attack had been detected.
The central bank told the public the cause was a technical glitch in PHP software. A remotely exploitable flaw that runs commands before an application loads was described to citizens as a glitch.
Vendor guidance on protecting hypervisors is correct and, as written, internally contradictory. Patch the host through supported workflows. Limit management access and avoid root. Enable Lockdown Mode so administration flows through vCenter. Disable SSH when it is not required. Use Secure Boot and the execInstalledOnly setting to restrict what can run. Use maintenance mode and prechecks rather than making uncontrolled changes across an environment. Do not install conventional antivirus or third party agents on ESXi or the vCenter appliance, because the hypervisor is a purpose built system more like router firmware than a general purpose server, and unsupported modification voids support.
Now read the vendor's own note on monitoring. Security tooling for these hosts typically works agentless, through remote access over SSH, and customers are told to weigh the exposure of enabling SSH against the benefit of the monitoring. The attackers reached ESXi over SSH using a service account. The only way to watch the hypervisor with conventional tools is to open the path that was used to destroy it. Disabling SSH and enabling monitoring are not compatible recommendations, and no vendor advisory resolves that.
Three additions belong on any real checklist. A gateway should never authenticate interactively to a hypervisor as the vCenter service account, so that path should alarm on its own. Lockdown Mode concentrates administration in vCenter, which simply relocates the crown jewel. And anyone exposed should search for a wiper note left in every directory it processed, and for execution from the temporary directory of a host that should never execute anything there.
Patching ESXi closes nothing. The entry point was the gateway.
The record supports three separate conclusions. Several outages resulted from maintenance, software changes or defects, with no established hostile activity and no demonstrated single cause. PHP2Shell is a serious risk that justified urgent patching and deliberate shutdowns, and the NSA disruption was linked to patching by the Director of National Intelligence rather than established as an intrusion. And confirmed exploitation of unpatched gateways led in some cases to destructive attacks on virtualization infrastructure, which is why external entry points and internal administrative systems have to be defended as one problem.
Everything beyond that is being asserted rather than shown. There is no victim list, no count of wiped hosts, no independent forensic review of the agency network, the aviation backbone or the gateway fleet, and no loss estimate anyone can defend. Every explanation in this account was issued by the organization that failed, about its own failure, within hours, by parties with an interest in calm. That does not make the explanations false. It makes them untested.
What is testable is smaller and less comfortable. Six of seven major failures in one week came from change, not from attackers. The systems running aviation, banking, government identity and classified intelligence have grown too complex for the organizations operating them to modify safely. Patching breaks them. Not patching breaks them. A defect in a July fix disabled Outlook in November. A router swap grounded the country. A patch in the middle of an emergency closed Fort Meade.
That is the finding, and it does not require a villain.
Sources:
PHP2Shell technical record and disclosure
Datadog Security Labs, mechanism and reproduction: securitylabs.datadoghq.com
Tenable FAQ, affected versions and remediation: tenable.com
Wiz Research, discovery and AI involvement: wiz.io
Calif Research, original writeup: calif.io
OpenAI, coordinated disclosure and the defender window framing: openai.com
Project page: php2shell.com
Axios on the AI angle and industry skepticism: axios.com
Check Point gateways and the ESXi wiper
Check Point Research, full kill chain, attribution, indicators: research.checkpoint.com
BleepingComputer summary with CISA catalog and chaining detail: bleepingcomputer.com
Microsoft 365 outage, November 12
The Register, defect introduced in July: theregister.com
NDTV Profit, full text of the company statement: ndtvprofit.com
The Hindu BusinessLine, more than 80 percent of servers down: thehindubusinessline.com
Thenxtweb, rollback and failover detail: thenextweb.com
Gulf News: gulfnews.com
FAA ground stop, November 13
FAA's own questions and answers: faa.gov
AP, router installations paused: apnews.com
Reuters, initial account: reuters.com
CRN, Duffy's blame of the old Cisco gear: crn.com
Network World, the multicast feature that is not enabled by default: networkworld.com
JPMorgan Chase, November 16
Politico: politico.com
CNBC: cnbc.com
New York Times: nytimes.com
NBC News, counterparty detail and Zelle: nbcnews.com
Reuters via Investing.com: investing.com
United Airlines, November 16
AP, more than 90 minutes: apnews.com
CNBC: cnbc.com
CNN, airline's own count of flights: us.cnn.com
AirlineGeeks, weight and balance detail: airlinegeeks.com
Fox 4: fox4news.com
Apple, November 17
New York Times: nytimes.com
CBS News, restored and not an attack: cbsnews.com
NBC News, store and status page detail: nbcnews.com
Bloomberg via Business Standard, internal systems and the outage tracker: business-standard.com
Reuters via Investing.com: investing.com
NSA and Jay Clayton
Fox Business, the interview transcript: foxbusiness.com
Computing.net, Fort Meade closure and alert levels: computing.net
Precautionary shutdowns, Europe
The Register, 32 UK services down, House of Lords disclosure, councils: theregister.com
BBC, worldwide takedowns including Wikimedia and BBC sites: bbc.com
BBC, Scottish councils and health boards: bbc.com
Dutch government statement, DigiD and MijnOverheid: rijksoverheid.nl
Yle, Finland's state ICT center: yle.fi
Nepal
Kathmandu Post, thousands of sites, GIWIMS, central bank: kathmandupost.com
Kathmandu Post, banks and funds, "technical glitch" wording: kathmandupost.com
Kathmandu Post, government advisory and emergency team: kathmandupost.com
Nmkhabar, Vehicle Consignment Tracking System and the Department of Revenue Investigation: nmkhabar.com
myRepublica, wider digital exposure: myrepublica.nagariknetwork.com
VMware and Broadcom hardening guidance
Broadcom, third party agents and antivirus on ESXi and the vCenter appliance: knowledge.broadcom.com
Broadcom, general security recommendations for ESXi hosts: techdocs.broadcom.com
No comments:
Post a Comment