Title
Proving public IP to customers is related to a serious risk. If a DNS port is opened to internet, a reflection type of DDoS starts flowing through an ISP, eventually.
Situation
It’s a rainy Sunday evening, and most households are watching TV or Netflix. That’s when internet connection starts to drop from time to, and eventually, all households lose it completely.
The support line desk at the regional ISP was busy that evening, hundreds of customers were angry enough to call the operator and demand that the outage be resolved immediately.
Challange
ISP’s network administrator left the dinner table at home and went working to resolve the situation.
However, after careful analysis of SNMP telemetry in Zabbix, he wasn’t able to find out the root cause of an issue.
The operator thought that if this happens next weekend, we are going to lose customers for sure.
What to do now?
Solution
Network administrator looked into netflow data – traffic telemetry (link).
- ISP had netflow export in place on all CORE routers
- Netflow data streams were continually sent to the central collector with FLOWCUTTER software.
Among other anomalies, FLOWCUTTER focuses on clear analysis of outgoing attacks. One of those is a DNS reflection attack passing through the network.
The administrator was able to find the root cause of an issue – a customer with a public IP address provided to him. At the household, a customer by accident did factory reset of router/modem w/ RouterOS, ending up with no password setup and open service port to the internet.
Later analysis revealed that the attacker took control of the router and included it in the botnet using it for DNS reflection type of DDoS.
The operator blocked the attacking IP, getting rid of the attack. Later called the customer for cleaning up.
Moreover, the network administrator setup daily open ports scan detection in order to be alerted early, when something like this happens again. Next time he will be ready.
Results
Let’s walk us through the netflow analysis of the issue at the time of the incident.
Outgoing attack was clearly visible when filtering out outgoing connections from source port 53. This particular port is reserved for DNS resolving and only resolvers should give DNS responses.
It’s also important to note that the volume of this attack was just 25.Mb/s which is completely invisible in aggregate traffic of an ISP. Hence other tools weren’t able to detect it, moreover find an attacking vector.
On the attacking IP, spoofed DNS requests could be observed from foreign countries.
Drill-down analysis revealed that the anomaly is DNS related.
In addition to netflow data, periodical scan of open ports was set up in FLOWCUTTER. That helps the operator to be pro-active and expose the anomaly next time.
Upcoming feature will among other things allow the detection of outgoing DNS reflection attack detection without any involvement of technical staff of an ISP.
Resources
- Netflow analysis in FLOWCUTTER
- Open ports scan
- SNMP vs Flow telemetry
- Outgoing DNS reflection attack detection
Takeaway
Opened DNS ports even on a customer’s IP usually lead to reflection DNS attacks pouring from the operator’s network. This can wreak havoc on ISP’s network.
-
- This is one of several root causes that cannot be revealed by analyzing SNMP-like telemetry.
- That’s where netflow data comes in handy. It helps by identifying suspicious traffic patterns in DNS responses.
- It’s useful to set up a daily open ports scan to prevent before attack happens.
ISP resolved the issue with ease.
Testimonials
“It is not a pleasant experience to see our network crumbling but at the same time being blind to why it’s happening.
FLOWCUTTER helped us confirm the root cause – opened DNS ports and to identify a customer device where it’s happening. Next time we will be “pro active” and find the issues faster before it influences other customers.”
“One weekend, we experienced degradation of service in periodic interval. We were not able to find root cause from SNMP telemetry. FLOWCUTTER helped us identify reflexive amplification DDoS attack. On monday, my networks worked perfectly again.”