Detecting an OT network attack through ISP infected routers

Detecting an OT network attack through ISP infected routers

Title

How compromised ISP routers can Reveal attacks on OT Networks

Situation

This case study highlights a real-world cybersecurity incident that could occur in contemporary enterprise networks.

A company experienced network outages and performance degradation, which significantly impacted its Operational Technology (OT) manufacturing network. The attack vector originated from an ISP’s infrastructure, allowing the adversary to progressively move closer to industrial control systems and the organization’s central database.

Attack Progression

  • Reconnaissance Phase: The attacker employed a “spray and pray” tactic, scanning multiple targets indiscriminately in search of vulnerabilities. Using automated scanning tools, the attacker probed exposed services on public IP ranges. High-interest targets included remote management interfaces (SSH, Winbox, and Telnet), outdated web applications, and network infrastructure devices.

Picture: MITRE ATT&CK® Matrix: visualization of attack phases



  • ‘’Initial Access: The ISP operated RouterOS-based devices, one of which was compromised due to an outdated firmware vulnerability. The attacker exploited CVE-2022-45315, a vulnerability allowing unauthorized execution of code via specially crafted SNMP packets. The exploit provided a foothold into the ISP’s core network, allowing the attacker to execute commands remotely and establish persistence.

  • Privilege Escalation & Establishing Persistence: Once inside, the attacker elevated privileges by exploiting weak credentials and misconfigured access control rules. They also deployed custom scripts to maintain access even after system reboots.

  • Lateral Movement: The compromised ISP router began brute-force attacks on Telnet, SSH, and Winbox services on other network devices. The attacker attempted to map the ISP’s internal network structure, identifying routers, firewalls, and enterprise edge devices that could be leveraged for further exploitation.

Picture: MITRE ATT&CK® Matrix: visualization of attack phases



  • Enterprise Network Compromise: The attacker successfully took control of the enterprise’s perimeter router, establishing communication with Command & Control (C2) servers. They used DNS tunneling and encrypted HTTP requests to mask malicious activity and avoid detection by standard firewall monitoring.

  • OT Network Intrusion: The attacker attempted to compromise the perimeter of the OT network, initiating brute-force attacks on the enterprise’s database servers and performing slow, targeted scans against OT network segments, probing for vulnerable** devices**. However, the attack was detected at this stage due to volumetric NetFlow analysis, and mitigation efforts were implemented before a full breach occurred. The intrusion detection systems (IDS) regerated abnormal amount of logs and this triggered alert, allowing network administrators to block the attack and temporarily isolate the attacked network before incident was revealed and attack mitigated.

Challange

Challenges cover detection of such incidents

  • Firewall monitoring failed to detect the attack since the adversary leveraged a trusted ISP infrastructure to move laterally.
  • Absence of Endpoint Detection and Response (EDR) across all systems limited forensic reconstruction of the attack sequence.
  • Attack on OT network activity remained nearly undetectable, as it was performed gradually and from trusted devices.

Solution

  • Deployment of NetFlow analysis at the ISP level helped reveal abnormal traffic patterns.
    • Correlation of multiple data sources is the key for early detection: NetFlow data, Vulnerability scan, Intrusion Detection System (IDS) logs, usage of IP reputation and threat feeds, and DNS telemetry.
      • NetFlow data helped to reconstruct the incident and decide on how to improve security posture.

      • Cleanup mainly consisted of a) better isolation (segmentation) of network devices, and **b) updating ** all infected devices.

      • Monitoring of both ISP and enterprise infrastructure, namely regular vulnerability scans and update status of all key devices.

        Results

        By leveraging advanced network analysis, data correlation, and proactive monitoring, security teams were able to detect the attack before critical OT assets were compromised. This case study underscores the importance of collaboration between ISPs and enterprises in fortifying cybersecurity defenses.

          Resources

          • Netflow analysis in Grafana
          • Anomaly detection
          • Flow enrichment with IP reputation to distinguish known scanners and attackers
          • Vulnerability scan
          • Integration of DNS security solution data feed
          • Integration of IDS logs in Grafana

          Takeaway

          Firewall monitoring alone is insufficient—NetFlow data provides deeper insights into traffic flows.

          Proactive security measures at the ISP level can prevent attack propagation.

          Network segmentation and multi-source log correlation enhance detection capabilities.

          Attacks often exploit the weakest link—in this case, vulnerabilities within ISP infrastructure.



          With SNMP only, ISP would lose key customer

          With SNMP only, ISP would lose key customer

          Title

          An anomaly that SNMP monitoring couldn’t spot but flow-based analysis revealed root cause and helped ISP to retain key enterprise customer.

          Situation

          A key enterprise customer called ISP’s technical support complaining about latency issues when using Teams. The network administrator checked the router where the customer is connected together with hundreds of other customers. He analyzed latency data stored in Prometheus.

          Screenshot: latency on 30s intervals on router

          The latency graph revealed a periodicity of an anomaly that took 10 minutes. This repeated every hour.

          As well, packets dropped, and cpu usage revealed a similar trend.

          Challange

          However, based on SNMP telemetry, the administrator wasn’t able to find out the root cause of an issue.

          What to do now?

          Solution

          Network administrator looked into netflow data – traffic telemetry (link).

          • ISP had netflow export in place on all CORE routers
          • Netflow data streams were continually sent to the central collector with FLOWCUTTER software.

          With help of FLOWCUTTER’s ability to easily perform a fast drill-down analysis of flow dataset, the administrator was able to find the root cause of an issue.

          In addition to netflow data, periodical scan of open ports was set up in FLOWCUTTER. That helped to expose the first root cause of the anomaly.

            Results

            On the target router, there was an anomaly – traffic went down while talkers went up.

              Drill-down analysis revealed that the anomaly is DNS related.

              After that, the administrator checked the dashboard with results from the open ports scan from the previous night. It showed that another customer with public IP opened the DNS port to the public. That led to additional stress for the router influencing other customers in the same region.

              There are more examples of what can be revealed within seconds about the customer:

              • Upload/download
              • Ports and protocols related to specific services: ftp, telnet, ssh
              • IP is blacklisted
              • Communication w/ botnet 
              • Open ports and vulnerabilities visible from outside

              Resources

              • Netflow analysis in Grafana
              • Open ports scan
              • SNMP vs Flow telemetry

              Takeaway

              1. There are many root causes that cannot be revealed by analyzing SNMP-like telemetry. That’s where netflow data comes in handy. It helps by providing deeper insight into the source and destination of each traffic flow.
              2. In addition to SNMP and Netflow, it’s useful to correlate with other data sources – in this case open ports scan.

              ISP resolved the issue with ease. 

              The second customer, where the root cause dwelled, was called, pointing to misconfiguration. The port was closed, anomalies stopped.

              For the key enterprise customer, the latency issue was resolved helping ensure a good relationship.

              Testimonials

              “One weekend, we experienced degradation of service in periodic interval. We were not able to find root cause from SNMP telemetry. FLOWCUTTER helped us  identify reflexive amplification DDoS attack. On monday, my networks worked perfectly again.”

              Lukáš Vacek

              Viridium

              How infected modem could quietly block /22 prefix

              How infected modem could quietly block /22 prefix

              Title

              Malware in just one customer’s device almost ruined the whole prefix reputation potentially causing problems to all ISP’s customers.

              Situation

              Operator provider internet to both enterprise and home customers. Some of the home connections could pay extra for public IP, for example when having a camera security system at home and want to check home safety from the work. One of those home modem/router got infected by malware. Consequently the device was included in the botnet.

              In the case of this botnet, the goal of the week was to scan devices around the internet for possible open telnet ports, and then try to infect them with the latest load of possible attacks that take advantage of vulnerabilities.

              Challange

              Such an attacking device quickly ends up on public blacklists. This influences just the device with 1 IP address. So far so good.

              What can easily happen later is for the whole prefix (in this case /22) to be backlisted on IP reputation. Potentially peering partners start to challenge the operator of the AS (Autonomous System) and demand correcting the issue.

              At this point a small anomaly on one modem causes a lot of damage. Amount of work to be done week later is enormous in comparison to correcting issue right at the beginning.

              So it’s “no brainer”, we have to spot such anomalies, right? 

              Not so fast. Normally such an anomaly flies under the radar, undetected, if an ISP relies just on SNMP (e.g. Zabbix, Nagios). Administrators usually can’t detect it. Routers aren’t aware of it, as it does not tax hardware or ends up in many bytes and packets travelling around the network.

              What to do?

              Solution

              First of all, an operator should use flow-based traffic analysis, so that he/she can catch this anomaly. 

              Fortunately, in this case ISP had some measures installed:

              • ISP had netflow export from perimeter routers 
              • Netflow was stored in the central collector with FLOWCUTTER software.
              • With FLOWCUTTER, any admin can easily do a fast drill-down analysis of netflow and other data sources.

              A quick morning look at the overview (Home dashboard) in FLOWCUTTER with just a few metrics revealed a trend shift in the number of talkers (distinct communication source-destination IP pairs).

                Fast drill-down analysis revealed that anomaly is situated on one particular IP (home customer with public IP). 

                Fast drill-down analysis revealed that anomaly is situated on one particular IP (home customer with public IP). 

                It took just a few hours to this IP being backlisted on IP reputation lists.

                Results

                What if admin don’t want to look at FLOWCUTTER every single day?

                For that purpose, FLOWCUTTER helps in two ways: 

                1. to set up “out of the box” detection of various network anomalies – including Telnet,
                2. Enrich Netflow data by IP reputation, checking and alerting on when any of your IPs is blacklisted.

                  Resources

                  • Netflow analysis in Grafana
                  • SNMP vs Flow telemetry
                  • IP reputation
                  • Flow-based Anomaly detection

                  Takeaway

                  ISP detected a telnet anomaly early, and so was able to prevent cascade of bad outcomes.

                  1. Some misconfigurations and infected endpoints can result in damaging operator’s IP prefix or AS reputation.
                  2. Within flow-based troubleshooting, these anomalies can be spotted and corrected early when they don’t create havoc within the network

                  For the future, ISP used FLOWCUTTER’s ability to monitor and alert on network anomalies as well as regularly check reputation its IP range. And next time be alerted even faster.

                    How undetected DDoS cut town from internet

                    How undetected DDoS cut town from internet

                    Title

                    Outgoing DDoS stopped before making havoc by saturating uplink for the whole network segment

                    Situation

                    Network administrator was notified by an alert that one particular radio uplink becomes saturated in the middle of the day. This network segment represented a remote town. Consequently all homes and institutions in there experienced severe connection problems.

                    Challange

                    SNMP data showed the saturation of uplink in regular intervals. But nothing apparent was going on. Behind this link ISP didn’t have detailed information per customer, hence, the network administrator was able to find out whether there is one particular endpoint responsible for the upload.

                    What to do?

                    Solution

                    Network administrator analyzed anomaly using netflow data – traffic telemetry.

                    • ISP had netflow export on the perimeter 
                    • Netflow was stored in the central collector with FLOWCUTTER software.
                    • With FLOWCUTTER, any admin can easily do a fast drill-down analysis of netflow and other data sources.
                    • In addition, FLOWCUTTER allows to set up “out of the box” detection of various volumetric DDoS attacks

                    That helped identify the nature of the DDoS attack and revealed information used to effectively mitigate it.

                      Results

                      The traffic analysis in FLOWCUTTER showed the nature of the anomaly. By plotting Top N source ports by number of flows, the administrator correctly identified that responses to DNS queries were contributing most to the uplink saturation. Incoming DNS packets stayed hidden in the overall volume of traffic, but the outgoing DNS responses were clearly visible.

                        Focusing on just DNS responses meant easy operation (filtering src.port=53 only) that took only 2 seconds. This is where FLOWCUTTER excels above all competitive solutions.

                        The next step was to compare behavior before and during the anomaly. It showed that normally there were 1.000 talkers, but suddenly raised to 23.000 corresponding to the distributed nature of the attack.

                        Even though the administrator wasn’t able to find a particular end customer responsible, because there were many customers hidden behind each public IPv4 address, he identified this reflexive DDoS using DNS. 

                        Moreover, analysis revealed information (combination of Src/Dest. ports = 53/24335) that he could use to mitigate the attack using BGP FlowSpec (RFC 5575). Use of FlowSpec is similar to BGP Remotely trigger black hole, but it provides more granular control (ports, packet length, TCP flags, ICMP code, …) and shape these flows. 

                        Resources

                        • Netflow analysis in Grafana
                        • Open ports scan
                        • SNMP vs Flow telemetry

                        Takeaway

                        ISP detected uplink saturation.

                        1. Within netflow, the administrator find root cause in outgoing reflexive DDoS attack using DNS.
                        2. He was able to mitigate attack using BGP FlowSpec

                         

                        For the future, ISP used FLOWCUTTER’s ability to monitor and alert on traffic anomalies – admin can set up “out of the box” detection of various volumetric DDoS attacks. And next time be alerted even faster.

                          Testimonials

                          “In one subnet, customers reported several outages. Uplink radio connection was saturated. With great help of FLOWCUTTER 

                          I find out that one customer opened DNS to public, causing uplink saturation, and influencing others in the subnet. After mitigation I set up alerting rule to detect unwanted open ports.”

                          Roman Beneš

                          DobruškaNet

                          How to lower a price of Data retention for ISP

                          How to lower a price of Data retention for ISP

                          Title

                          What helped ISP to comply with national data retention policy with low costs within complex network of 130+ sites with NAT translation

                          Situation

                          Most countries oblige its internet service providers to collect forensic data on who is communicating with whom. In essence it usually translates to netflow data. Mandatory data retention policy for operators ranges from 6 months to 5 years depending on the country.

                           

                          What is the challenge for an operator to comply with national policy?

                          Challange

                          There are several challenges, all of them were the case for this operator:

                          • In order to identify the customer responsible for communication, an operator has to consider NAT translation – there can be many private IP addresses behind one public IP address.
                          • Architecture differs a lot among operators – some operator quite diverse network with many pop in many regions. In this case, there were individual isolated 130+ sites, each with its own NAT router.
                          • Routers can be of various vendors and firmware versions.
                          • Amount of traffic for small ISP is about 10K flows per second, up to 1M flows/s for large operators.
                          • Storing such a traffic’s netflow data is demanding.

                          Solution

                          ISP implemented following solution to comply Data retention policy:

                          • Operator setup export of (private traffic) netflow from 130+ routers
                          • In addition FLOWCUTTER probes were installed at perimeter to monitor public traffic
                          • Netflow data streams were continually sent to the central collector with FLOWCUTTER software.
                          • FLOWCUTTER supports all the incoming flow format NF5/8/IPFIX

                            Results

                            The project was successfully delivered and all the expectations were met. Correct sizing of the project made sure ISP would be able to store the data for a necessary period for another 3 to 5 years.

                            In comparison to the competing projects, the solution including FLOWCUTTER results in 75% price tag decrease due to its versatility and broad compatibility because the operator was able to leverage its own resources and not buy the whole complete solution from scratch.



                              Resources

                              • Network probes
                              • Flow formats & compatibility
                              • Hardware appliance
                              • Flows compression ratio

                              Takeaway

                              The project was successfully delivered, expectations were met.

                              In addition, the total price tag was lowered by 75%.