In recent years, Distributed Denial of Service (DDoS) events have become an ever-present threat, featuring attack traffic pushing to levels measured in terabits per second (Tbps). One of the key tools in the hands of cybercriminals seeking to increase the bandwidth of their attacks is UDP-based reflection. For example, the 2018 DDoS attack on GitHub made use of an application layer service called Memcached to direct, at peak, 1.35 Tbps of reflected UDP traffic at GitHub’s servers. In 2020, the industry learned about a 2017 DDoS attack that used a bundle of UDP services as reflectors (CLDAP, DNS, and SMTP) to achieve wire rates of up to 2.5 Tbps.
At Black Lotus Labs, we leverage visibility from our global network to identify services potentially being manipulated to launch attacks, such as Memcached instances, CLDAP and DNS, and then work to confirm whether they are open to use as reflectors. Based on our data from the first quarter of 2021, we see each of these services being actively used to launch significant DDoS attacks today.
Memcached, the Big BAF Reflector
Black Lotus Labs’ interest in Memcached is due to its remarkably high Bandwidth Amplification Factor (BAF). A BAF is the ratio of response bytes reflected at the target to request bytes sent to the reflector by the attacker; a BAF of 5 means for each byte sent to the reflector it directs 5 bytes at the target. By day, Memcached is a hardworking, distributed caching service designed to help your web page load faster. Most commonly deployed in support of web server clusters to eliminate unnecessary calls to the database by caching data, there is no reason why Memcached should be exposed to public internet. If you wouldn’t expose your backend database to the public internet, why would you put your backend caching service out there? While not all internet facing Memcached instances are open to use in reflection attacks, we’ve determined that a number of them could be utilized for reflection in a DDoS attack.
The determination of whether a given Memcached instance constitutes an open reflector is simple math: do they respond to protocol commands over UDP? Memcached supports its caching service with a protocol that can be accessed over TCP or UDP, and only UDP can be used for this type of attack. As one might expect from a caching service, this protocol features commands like Set and Get. Set allows you to store arbitrary blocks of data under a given key. Get, inversely, allows you to fetch the data back by key. The key can be quite small while the data stored under the key can be up to 1 MB with out-of-the-box configuration. This means a very small request packet, measured in tens of bytes, can generate a significantly larger response. In the case of Memcached, the BAF is 10,000 to 51,000x. This is high bandwidth amplification to the extreme: a 100 Mbps internet connection could produce an attack volume ranging from 125 Gbps to more than 630 Gbps. A few years ago, after a series of notable Memcached attacks, the internet community made a concerted effort to mitigate the more than 35,000 publicly available Memcached servers. Yet, as the following heat map shows, there are still currently more than 1,500 open Memcached services exposed on the internet, with, geographically, China and the United States hosting the lion’s share.
Global distribution of Memcached services exposed to the internet
How UDP Reflection Works
Though other UDP reflector services do not have the same BAF as Memcached, reflection works the same for all UDP services. Let’s look at the structure of a UDP reflection attack. In the following figure, we have three components: the attack initiator, the reflector service and the target system.
UDP reflection with IP spoofing
As the above figure shows, the reflection depends upon tricking the reflector service into sending the response payload to the target system rather than back to the attack initiator system. This act of reflection depends upon IP spoofing; UDP’s big weakness is that it allows spoofing to occur. To spoof an IP address in UDP communication, you only need to put the target system’s IP address as the source address in the header of the IP packet. With UDP, there is nothing inherent to the protocol preventing this.
Why does UDP allow spoofing while TCP disallows it? Consider this: If the Memcached service is only open to TCP communication, you could still issue the same Get command to fetch the large data bomb, but you wouldn’t be able to spoof the return address. This is because you would have had to successfully complete TCP’s three-way handshake to establish a connection before sending any packets with actual Memcached commands. TCP requires several administrative packets be exchanged to establish the connection prior to any real communication; this is the three-way handshake. When this handshake is complete, both sides of the communication know precisely who is at the other end of the connection. When a TCP packet containing the Get command shows up at the Memcached instance, if its source address doesn’t align with state of the connection, it will not be serviced.
On the other hand, UDP does not even bother tracking connections. It’s kind of like answering the phone back before the era of caller ID; unless you want to risk missing a call from a friend, you were obliged to answer every time it rang, thus leaving you vulnerable to all manner of prank calls and predatory sales calls. Similarly, each UDP packet that arrives must be taken at face value. UDP simply doesn’t have the ability to examine the source address and say, “Hey, I wasn’t talking to this person. Where did this come from?”
Since this shortcoming belongs to UDP itself, any service implemented over it can be used as an open reflector. To confirm a UDP service can be used as a reflector, all that’s required is to test that the service responds to queries, just as we saw with Memcached.
Top UDP Services Launching Attacks
While we observed over 1,500 Memcached servers open to use in reflective attacks, Memcached is not the only reflective protocol prime for abuse. Protocols such as connectionless LDAP (CLDAP), NTP, SSDP, DNS, and many others can be leveraged for reflection. Looking over the past several months of our data, a few of them rank much higher than Memcached for the size of the largest attack observed. This is expected due to the overall trending decline of Memcached reflectors that are available for abuse.
Distribution of largest UDP attacks observed in 1Q21
Connectionless LDAP
Connectionless LDAP, in particular, poses a significant threat given the combination of a 50-70x bandwidth amplification factor and widespread availability due to its associated use in Active Directory. We looked at IP addresses observed receiving a potential reflection request of 67 bytes on UDP port 389 and having responded with data on UDP 389 at some point over the quarter. Using this methodology, Black Lotus Labs discovered over 281,000 unique IP addresses that appear to be responding to CLDAP reflection requests.
In addition, CLDAP has been leveraged in several recent large-scale attacks, such as the 2.3 Tbps attack against Amazon in 2020 or more recently as we’ve discovered the attack on May 4, 2021, against one of the Belgium government ISPs, Belnet.
During the time of the Belnet attack, Black Lotus Labs observed that over 78% of the traffic destined for Belnet transitioned from the expected norm of HTTP/HTTPS to almost entirely CLDAP. While we observed some uptick in other reflective protocols, none appear to have generated nearly as much attack traffic by bandwidth as CLDAP.
Percentage of Belnet traffic by protocol
DNS itself as a reflection vector
In addition to abuse with CLDAP, DNS is another protocol that is often leveraged over UDP that can be potentially abused for reflection attacks. While some open DNS resolvers will rate limit the types of queries that are often abused for reflection, this methodology is not a perfect solution for stopping all types of reflective DNS attacks.
When attackers are looking to launch a reflective attack with DNS, they need a domain and record type that will return a large response. Some of the most common record types that return large values are ANY, SRV and TXT. After removing A/AAAA, which is the largest volume of DNS queries we commonly see, ANY queries are a very sizable portion of the remaining queries we observe on the internet.
Percentage of ANY DNS queries
What More Can We Do?
Cleaning up reflective DDoS sources across the internet is a complicated issue that requires a continual community effort. Organizations that expose services on the internet need to be mindful of how they can be used to participate in reflective DDoS attacks and either not expose them or implement measures to mitigate. Internet service providers, managed security providers and security researchers can also help by informing their customers when they see reflective services exposed and being manipulated. Such efforts have been effective at significantly reducing the number of exposed Memcached instances over the past years.
For our part at Lumen, when we find open reflectors hosted within our customers’ networks that are being abused to cause substantial damage across the internet, we proactively reach out to them to better understand the intent of the service and share guidance on how to remediate the danger.
If you would like to collaborate on similar research, please contact us on Twitter @BlackLotusLabs.
To learn more about DDoS threats and attack trends, check out the Lumen Quarterly DDoS Report for 1Q21.
This information is provided “as is” without any warranty or condition of any kind, either express or implied. Use of this information is at the end user’s own risk.
