Outgoing DDoS stopped before making havoc by saturating uplink for the whole network segment

Network administrator was notified by an alert that one particular radio uplink becomes saturated in the middle of the day. This network segment represented a remote town. Consequently all homes and institutions in there experienced severe connection problems.

SNMP data showed the saturation of uplink in regular intervals. But nothing apparent was going on. Behind this link ISP didn’t have detailed information per customer, hence, the network administrator was able to find out whether there is one particular endpoint responsible for the upload.

What to do?

Network administrator analyzed anomaly using netflow data – traffic telemetry.

• ISP had netflow export on the perimeter 
• Netflow was stored in the central collector with FLOWCUTTER software.
• With FLOWCUTTER, any admin can easily do a fast drill-down analysis of netflow and other data sources.
• In addition, FLOWCUTTER allows to set up “out of the box” detection of various volumetric DDoS attacks

That helped identify the nature of the DDoS attack and revealed information used to effectively mitigate it.

The traffic analysis in FLOWCUTTER showed the nature of the anomaly. By plotting Top N source ports by number of flows, the administrator correctly identified that responses to DNS queries were contributing most to the uplink saturation. Incoming DNS packets stayed hidden in the overall volume of traffic, but the outgoing DNS responses were clearly visible.

Focusing on just DNS responses meant easy operation (filtering src.port=53 only) that took only 2 seconds. This is where FLOWCUTTER excels above all competitive solutions.

The next step was to compare behavior before and during the anomaly. It showed that normally there were 1.000 talkers, but suddenly raised to 23.000 corresponding to the distributed nature of the attack.

Even though the administrator wasn’t able to find a particular end customer responsible, because there were many customers hidden behind each public IPv4 address, he identified this reflexive DDoS using DNS. 

Moreover, analysis revealed information (combination of Src/Dest. ports = 53/24335) that he could use to mitigate the attack using BGP FlowSpec (RFC 5575). Use of FlowSpec is similar to BGP Remotely trigger black hole, but it provides more granular control (ports, packet length, TCP flags, ICMP code, …) and shape these flows. 

• Netflow analysis in Grafana
• Open ports scan
• SNMP vs Flow telemetry

ISP detected uplink saturation.

1. Within netflow, the administrator find root cause in outgoing reflexive DDoS attack using DNS.
2. He was able to mitigate attack using BGP FlowSpec

For the future, ISP used FLOWCUTTER’s ability to monitor and alert on traffic anomalies – admin can set up “out of the box” detection of various volumetric DDoS attacks. And next time be alerted even faster.

What helped ISP to comply with national data retention policy with low costs within complex network of 130+ sites with NAT translation

Most countries oblige its internet service providers to collect forensic data on who is communicating with whom. In essence it usually translates to netflow data. Mandatory data retention policy for operators ranges from 6 months to 5 years depending on the country.

What is the challenge for an operator to comply with national policy?

There are several challenges, all of them were the case for this operator:

• In order to identify the customer responsible for communication, an operator has to consider NAT translation – there can be many private IP addresses behind one public IP address.
• Architecture differs a lot among operators – some operator quite diverse network with many pop in many regions. In this case, there were individual isolated 130+ sites, each with its own NAT router.
• Routers can be of various vendors and firmware versions.
• Amount of traffic for small ISP is about 10K flows per second, up to 1M flows/s for large operators.
• Storing such a traffic’s netflow data is demanding.

ISP implemented following solution to comply Data retention policy:

• Operator setup export of (private traffic) netflow from 130+ routers
• In addition FLOWCUTTER probes were installed at perimeter to monitor public traffic

• Netflow data streams were continually sent to the central collector with FLOWCUTTER software.
• FLOWCUTTER supports all the incoming flow format NF5/8/IPFIX

The project was successfully delivered and all the expectations were met. Correct sizing of the project made sure ISP would be able to store the data for a necessary period for another 3 to 5 years.

In comparison to the competing projects, the solution including FLOWCUTTER results in 75% price tag decrease due to its versatility and broad compatibility because the operator was able to leverage its own resources and not buy the whole complete solution from scratch.

• Network probes
• Flow formats & compatibility
• Hardware appliance
• Flows compression ratio

The project was successfully delivered, expectations were met.

In addition, the total price tag was lowered by 75%.

Complaining customer handled with ease with context from traffic telemetry in ISP’s fingertips

Most calls to the support line of an operator are easy to deal with (missed payment, etc.). However few calls generate the majority of effort and time of the support team, especially in case of technical support calls.

One of those recurrently complaining customers called his ISP support line. John, as always, complained that the internet doesn’t work and he needed it for work, e.g. online meetings (via Teams/Meet/Zoom).

A technical support person is not always an over-paid network admin. To rule out operator faults when “internet’s not working”, is not trivial and consumes time.

How to speed up dealing with such calls?

The goal is to help technical support personnel to get customer’s traffic context easily and fast?

• ISP had to collect and store netflow – traffic telemetry including NAT IP address translation to see individual customers behavior – a perfect job for  FLOWCUTTER collector.
• With FLOWCUTTER, an administrator can provide user-friendly dashboard to support team
• Upon calling he/she input customer’s IP into dashboard box and within second can see and understand basic behavior of the calling customer

From the dashboard, even less technical guy/gall can determine that an issue is not operator’s but on the customer’s side. For example he/she can give answers such:

1. Not working? I can see a lot of traffic passing down your line from TikTok (AS13869). Maybe someone at home is secretly watching videos instead of doing their homework.

There are more examples of what can be revealed within seconds about the customer:

• Upload/download
• Ports and protocols related to specific services: ftp, telnet, ssh
• IP is blacklisted
• Communication w/ botnet 
• Open ports and vulnerabilities visible from outside

• Netflow analysis in Grafana – “single host IP” dashboard
• SNMP vs Flow telemetry
• IP reputation
• AS and country of traffic origin
• Flows w/ NAT IP address translation to see individual customers behavior 

ISP support line can be overwhelmed by calls regarding technical issues. The first step is to rule out mistakes on the customer’s side, where the operator cannot influence things.

This is where FLOWCUTTER can help technical support personnel by providing customer’s traffic context.

1. 1. Provide user-friendly dashboard to support team
   2. Upon calling operator can see and understand basic behavior of the calling customer