Title
Malware on a single customer's modem started scanning the internet for open Telnet ports. Flow data caught it early, before the ISP's /22 prefix and IP reputation took the hit.
Situation
The operator serves both business and home customers. Home customers can pay extra for a public IP address, for example to check a home security camera from work.
One of these home modems got infected with malware and became part of a botnet. That week, the botnet's task was to scan the internet for open Telnet ports and try to infect the devices it found.
Challenge
A device like this quickly ends up on public blacklists. At first, that affects only its single IP address. Left unchecked, the whole prefix, in this case a /22, can lose its IP reputation. Peering partners may then start challenging the operator of the AS and demand a fix.
A small anomaly on one modem can turn into weeks of work, compared with a quick fix at the start. The catch is that such anomalies usually stay below the radar of SNMP monitoring such as Zabbix or Nagios. They do not load router hardware and do not generate large volumes of bytes or packets.
Solution
The ISP had flow-based traffic analysis in place:
- NetFlow was exported from perimeter routers.
- Flows were stored in a central FLOWCUTTER collector.
- FLOWCUTTER allows fast drill-down analysis of NetFlow and other data sources.
A quick morning look at the Home dashboard in FLOWCUTTER showed a trend shift in the number of talkers (distinct source-destination IP pairs).
Drill-down located the anomaly on one IP address, a home customer with a public IP.
The device was sending millions of Telnet packets (port 23) to the whole internet every few minutes.
Within a few hours, the IP appeared on blacklists.
To avoid relying on daily manual checks, the ISP set up two things in FLOWCUTTER:
- Built-in detection of network anomalies, including Telnet scanning.
- NetFlow enrichment with IP reputation data, with an alert whenever any of the ISP's IP addresses is blacklisted.
Results
The ISP caught the Telnet anomaly early and dealt with the infected modem before the problem spread to the whole /22 prefix. Alerts now flag similar anomalies and blacklisted IP addresses without anyone having to check dashboards every day.
Resources
- NetFlow analysis in Grafana
- SNMP vs flow telemetry
- IP reputation
- Flow-based anomaly detection
Takeaway
One infected endpoint can damage the IP reputation of a whole prefix or AS. Flow-based troubleshooting spots such anomalies while they are still small, and alerts keep the ISP ahead of the next one.