It was just an ordinary day at the office — developers developing, managers managing, sysadmins doing nothing. All of a sudden, the eternal peace and harmony was broken with a single email starting with:
“We are the *** *** and we will DDoS your network and take down your website permanently if you do not pay — 50 Bitcoin.”
Okay, well… Challenge accepted! We blew the dust off our keyboards and started looking at what’s going on.
The website in danger was a fresh Magento 2 installation running one of the oldest 2.0.x community editions. AWS setup for it was not planned for large volumes of traffic — pretty much basic setup you’d find in most Magento installations on AWS.
As can be seen from the diagram, the main bottleneck is the single Varnish instance. This infrastructure flaw allowed the attackers to run successful SYN flood DDoS on the single point of entrance:
There is no autoscaling for Varnish nodes. Setting up scalable Varnish cache is a separate topic that requires a set of articles, so I will not cover it here.
The instance is exposed to the world. Even though instance’s security group allows only connections to 80/443 TCP + SSH from whitelisted IP addresses, it is enough for SYN flood DDoS attack
Since SYN flood is aimed to exhaust server resources, the obvious solution was to put more servers between the attackers and the infrastructure. In AWS, two services are capable of providing scalable resources for this purpose: CloudFront and Elastic Load Balancer. Both accept only well-formed TCP connections, making any SYN flood or UDP reflection attacks useless.
From our previous experience in DDoS attack mitigation we knew that this wave will not be the last one and wanted to prepare for the upcoming attacks which were likely to happen on the higher OSI model levels. Therefore, as a way to mitigate this SYN flood attack we chose to configure CloudFront with WAF in “Allow all” mode to keep an eye on the traffic.
In addition to configuring CloudFront and WAF, we have also restricted access to the Varnish instance to whitelisted IPs and CloudFront only, so any future direct attack on the IP address will fail.
Updated infrastructure diagram:
Just as we expected, the second wave of DDoS hit the infrastructure the next morning. This time it was on the Application level — WordPress XML-RPC flood. The botnet generated over 6000 requests per minute to the application, all targeting the home page of the website.
Blocking WordPress pingback flood does not mean that application level is now 100% secure. The attackers still can come back with the third wave, this time trying to mimic legitimate user behaviour. Mitigating such threat would require more advanced WAF configuration in conjunction with some AWS Lambda functions to limit request rate.
Another way to avoid downtime in the future is to remove Varnish bottleneck. More on Magento 2, autoscaling and Varnish caching in the future articles.
Even for a single instance setup, it is always a good idea to have AWS ELB or CloudFront in front of it and restrict connections to the EC2 instances only to certain IP addresses. With the scale and resources, AWS services can absorb any low-level DDoS attack without breaking your budget.
Adding CloudFront to the infrastructure setup not only improved the website security and performance, but also created new room for improvement: after the WordPress installation is fixed to support full-https pages, Varnish server can be removed and completely replaced by caching CloudFront distribution.