Title
Malware in just one customer’s device almost ruined the whole prefix reputation potentially causing problems to all ISP’s customers.
Situation
Operator provider internet to both enterprise and home customers. Some of the home connections could pay extra for public IP, for example when having a camera security system at home and want to check home safety from the work. One of those home modem/router got infected by malware. Consequently the device was included in the botnet.
In the case of this botnet, the goal of the week was to scan devices around the internet for possible open telnet ports, and then try to infect them with the latest load of possible attacks that take advantage of vulnerabilities.
Challange
Such an attacking device quickly ends up on public blacklists. This influences just the device with 1 IP address. So far so good.
What can easily happen later is for the whole prefix (in this case /22) to be backlisted on IP reputation. Potentially peering partners start to challenge the operator of the AS (Autonomous System) and demand correcting the issue.
At this point a small anomaly on one modem causes a lot of damage. Amount of work to be done week later is enormous in comparison to correcting issue right at the beginning.
So it’s “no brainer”, we have to spot such anomalies, right?
Not so fast. Normally such an anomaly flies under the radar, undetected, if an ISP relies just on SNMP (e.g. Zabbix, Nagios). Administrators usually can’t detect it. Routers aren’t aware of it, as it does not tax hardware or ends up in many bytes and packets travelling around the network.
What to do?
Solution
First of all, an operator should use flow-based traffic analysis, so that he/she can catch this anomaly.
Fortunately, in this case ISP had some measures installed:
- ISP had netflow export from perimeter routers
- Netflow was stored in the central collector with FLOWCUTTER software.
- With FLOWCUTTER, any admin can easily do a fast drill-down analysis of netflow and other data sources.
A quick morning look at the overview (Home dashboard) in FLOWCUTTER with just a few metrics revealed a trend shift in the number of talkers (distinct communication source-destination IP pairs).

Fast drill-down analysis revealed that anomaly is situated on one particular IP (home customer with public IP).

Fast drill-down analysis revealed that anomaly is situated on one particular IP (home customer with public IP).
It took just a few hours to this IP being backlisted on IP reputation lists.

Results
What if admin don’t want to look at FLOWCUTTER every single day?
For that purpose, FLOWCUTTER helps in two ways:
- to set up “out of the box” detection of various network anomalies – including Telnet,
- Enrich Netflow data by IP reputation, checking and alerting on when any of your IPs is blacklisted.
Resources
- Netflow analysis in Grafana
- SNMP vs Flow telemetry
- IP reputation
- Flow-based Anomaly detection
Takeaway
ISP detected a telnet anomaly early, and so was able to prevent cascade of bad outcomes.
- Some misconfigurations and infected endpoints can result in damaging operator’s IP prefix or AS reputation.
- Within flow-based troubleshooting, these anomalies can be spotted and corrected early when they don’t create havoc within the network
For the future, ISP used FLOWCUTTER’s ability to monitor and alert on network anomalies as well as regularly check reputation its IP range. And next time be alerted even faster.