Cloud giants Amazon Web Services, Google, and Cloudflare are warning about a novel zero-day vulnerability in the HTTP/2 protocol that allows threat groups to launch massive distributed denial-of-service (DDoS) attacks that dwarf previous record-setting incidents.

All three companies this morning unveiled details of the flaw – tracked as CVE-2023-44487 – they’re calling “Rapid Reset,” which was first detected in August and has since been exploited to launch a “barrage of attacks in recent months, including an attack that three times larger than any previous attack that we’ve observed,” Cloudflare Chief Security Officer Grant Bourzikas wrote in a blog post.

That HTTP attack reached just more than 201 requests-per-second; its previous record was 71 million rps earlier this year. Google researchers said the company saw a two-minute Rapid Reset DDoS attack in August generate 398 million rps, more than seven times the size of the previous largest one it blocked last year, which hit 46 million rps.

AWS said it saw spikes of DDoS Rapid Reset attacks between August 28 and 29, with the peak hitting 155 million rps. Over those two days, the cloud services giant mitigated more than a dozen such attacks.

Sending, Then Cancelling, Requests

The zero-day attack uses HTTP/2’s stream cancellation feature to send a request and then immediately canceling it over and over.

“By automating this trivial ‘request, cancel, request, cancel’ pattern at scale, threat actors are able to create a denial of service and take down any server or application running the standard implementation of HTTP/2,” Bourzikas wrote.

HTTP/2 allows clients to send a RST_STREAM frame, indicating to the server that a previous stream should be cancelled, according to Juho Snellman, staff software engineer, and Daniele Iamartino, staff site reliability engineer, for Google. The client can do this unilaterally; there is no coordination between the client and server.

In addition, the client can also assume the cancellation will happen immediately when the server receives the first RST_STREAM frame, before any other data from that TCP connection is processed.

“This attack is called Rapid Reset because it relies on the ability for an endpoint to send a RST_STREAM frame immediately after sending a request frame, which makes the other endpoint start working and then rapidly resets the request,” Snellman and Iamartino wrote. “The request is canceled, but leaves the HTTP/2 connection open.”

A Simple Attack

It’s a relatively simple attack, they wrote. The client opens a large number of streams at once, which is standard in a HTTP/2 DDoS attack. However, rather than waiting for a response to each request stream from the server or proxy, the client cancels each one immediately.

“The ability to reset streams immediately allows each connection to have an indefinite number of requests in flight,” they wrote. “By explicitly canceling the requests, the attacker never exceeds the limit on the number of concurrent open streams. The number of in-flight requests is no longer dependent on the round-trip time (RTT), but only on the available network bandwidth.”

Most of the Layer 7 DDoS attacks seen by Google since 2021 are based on HTTP/2, both in the number of attacks and the peak request rates, they wrote, adding that “a primary design goal of HTTP/2 was efficiency, and unfortunately the features that make HTTP/2 more efficient for legitimate clients can also be used to make DDoS attacks more efficient.”

A Small Botnet, But Big Problems

Another unusual feature of the attack is that it was launched using a botnet of about 20,000 machines.

“Cloudflare regularly detects botnets that are orders of magnitude larger than this – comprising hundreds of thousands and even millions of machines,” Cloudflare’s Bourzikas wrote. “For a relatively small botnet to output such a large volume of requests, with the potential to incapacitate nearly any server or application supporting HTTP/2, underscores how menacing this vulnerability is for unprotected networks.”

All three cloud players said they noticed a surge in these kinds of attacks starting in late August and said that their respective security services were able to protect organizations against them. What startled them was the size of the attacks.

“Threat actors used botnets in tandem with the HTTP/2 vulnerability to amplify requests at rates we have never seen before,” Bourzikas wrote. “As a result, our team at Cloudflare experienced some intermittent edge instability. While our systems were able to mitigate the overwhelming majority of incoming attacks, the volume overloaded some components in our network, impacting a small number of customers’ performance with intermittent 4xx and 5xx errors — all of which were quickly resolved.”

While mitigating the attack, Cloudflare engineers developed new purpose-built technology to stop the DDoS assaults and mitigate future ones, the CSO wrote.

Another vendor impacted by Rapid Reset, web server company NGINX, said its investigation of the zero-day yielded a method for improving server resiliency under myriad forms of flood attacks that could be possible over HTTP/2. NGINX said it will release a patch for its products October 11.

Rapid Reset Variants Popping Up

Google’s Snellman and Iamartino noted that since the initial round of attacks, they’ve detected variants of Rapid Reset that, while not as efficient as the original one, might be more dangerous than standard HTTP/2 DDoS attacks.

Rather than immediately cancelling the streams, the first variant instead opens a batch of streams at once, waits, and then cancels the streams, only to open another large collection of new streams. Such attacks may bypass mitigations that are based on just the rate of inbound RST-STREAM frames.

“These attacks lose the main advantage of the canceling attacks by not maximizing connection utilization, but still have some implementation efficiencies over standard HTTP/2 DDoS attacks,” they wrote. “But this variant does mean that any mitigation based on rate-limiting stream cancellations should set fairly strict limits to be effective.”

Another variant moves away from cancelling streams entirely. Instead, it “optimistically tries to open more concurrent streams than the server advertised,” they wrote. “The benefit of this approach over the standard HTTP/2 DDoS attack is that the client can keep the request pipeline full at all times, and eliminate client-proxy RTT as a bottleneck. It can also eliminate the proxy-server RTT as a bottleneck if the request is to a resource that the HTTP/2 server responds to immediately.”

Log4j, SolarWinds, Now Rapid Reset

Bourzikas put Rapid Reset in the same category as other significant incidents, including Log4j, the SolarWinds attack, WannaCry and NotPetya early ransomware, and Heartbleed. Such incidents and threats set off a “tremendous explosion that ripples across the world and creates an opportunity to completely disrupt any … organizations,” he wrote.

“While I wish I could say that Rapid Reset may be different this time around, it is not,” Bourzikas wrote. “I am calling all CSOs – no matter if you’ve lived through the decades of security incidents that I have, or this is your first day on the job – this is the time to ensure you are protected and stand up your cyber incident response team.”