In short
A saturated radio uplink left a whole town with severe connection problems. Flow data revealed an outgoing DDoS attack using DNS reflection, and the ISP mitigated it with BGP FlowSpec.
How to spot and stop an outgoing DDoS attack
What showed up in the data in this case:
| Signal | Seen in | In this case |
|---|---|---|
| Uplink saturated at regular intervals | SNMP | Radio uplink to a remote town |
| Outgoing DNS responses drive the upload | Flow data, top source ports | Source port 53 |
| Sudden jump in talkers | Flow data | From 1,000 to 23,000 |
| Exact port combination | Flow data | Source/destination 53/24335 |
| Customer behind the attack | Not visible from flows alone when many customers share one public IPv4 | A customer with DNS open to the public, as the ISP later found |
How the attack was stopped:
| RTBH | BGP FlowSpec | |
|---|---|---|
| What it filters | All traffic for the IP | Traffic by ports, packet length, TCP flags or ICMP codes |
| In this case | Not used | Used on the 53/24335 port combination |
To catch it next time: an alerting rule for unwanted open ports and built-in detection of outgoing amplification attacks via open DNS.
Situation
An alert notified the network administrator that one radio uplink was saturated in the middle of the day. The uplink served a remote town, so all homes and institutions there experienced severe connection problems.
Challenge
SNMP data showed the uplink saturating at regular intervals, but not why. Behind this link, the ISP had no per-customer detail, so the administrator could not tell whether a single endpoint was responsible for the upload.
Solution
The administrator analyzed the anomaly using flow data:
- The ISP exported NetFlow from the perimeter.
- Flows were stored in a central FLOWCUTTER collector.
- FLOWCUTTER allows fast drill-down analysis of NetFlow and other data sources.
- Built-in detection covers various volumetric DDoS attacks.
This revealed the nature of the attack and the information needed to mitigate it.
Results
Plotting top source ports by number of flows showed that DNS responses were driving the uplink saturation. Incoming DNS queries stayed hidden in the overall traffic, but the outgoing DNS responses were clearly visible. Filtering on source port 53 took about two seconds.
Comparing traffic before and during the anomaly showed the number of talkers jump from 1,000 to 23,000, which matched the distributed nature of the attack.
Many customers shared each public IPv4 address, so the administrator could not pinpoint a single end customer from flows alone. He did identify a reflective DDoS using DNS and the exact port combination (source/destination 53/24335). With that, he mitigated the attack using BGP FlowSpec. FlowSpec works similarly to a remotely triggered black hole, but offers finer control over ports, packet length, TCP flags or ICMP codes.
In one subnet, customers reported several outages. The uplink radio connection was saturated. With great help from FLOWCUTTER, I found out that one customer had opened DNS to the public, causing uplink saturation and affecting others in the subnet. After mitigation, I set up an alerting rule to detect unwanted open ports.
R. Beneš, DobruskaNet, ISP in Czechia and Croatia
Resources
- NetFlow analysis in Grafana
- Open ports scan
- SNMP vs flow telemetry
Takeaway
SNMP showed that the uplink was saturated. Flow data showed why: an outgoing DDoS attack using DNS reflection, with enough detail to stop it using BGP FlowSpec. Today, the ISP gets alerted on such anomalies automatically.