1 

 

 

 

    

Clean Pipes 2.0 

 

  2 

Clean Pipe Solution 2.0  

Executive Summary................................................................................................................................3 Best Current Practices.........................................................................................................................5 Network Infrastructure BCPs......................................................................................................................5 Host Based BCPs ..............................................................................................................................................5 Dedicated DDoS BCPs.....................................................................................................................................6 

Cisco Clean Pipes Solution Overview .................................................................................................6 Evolution of Cisco Clean Pipes Solution ......................................................................................................6 Protection Mechanism of Cisco Clean Pipes Solution ...............................................................................8 

Cisco Clean Pipes 2.0 Components ..................................................................................................10 Cisco Netflow ................................................................................................................................................... 10 Arbor Peakflow SP......................................................................................................................................... 11 Arbor Peakflow SP Threat Management System (TMS) ...................................................................... 12 

DDoS Protection Flow in Clean Pipes 2.0.......................................................................................14 1.  Baseline and Thresholds ....................................................................................................................... 14 2.  Detection ................................................................................................................................................... 14 3.  Diversion................................................................................................................................................... 14 4.  Scrubbing ................................................................................................................................................. 15 5.  Injection .................................................................................................................................................... 15 

Migration to Cisco Clean Pipes 2.0 ..................................................................................................16 Cisco Anomaly Guard and TMS Countermeasure Comparison...........................................17 Clean Pipes 2.0 Deployment Considerations............................................................................19 Netflow considerations for Peakflow SP .............................................................................................. 19 Deployment of Peakflow SP PI ................................................................................................................. 20 Deployment of Peakflow SP Collectors ................................................................................................. 21 Deployment of Peakflow SP TMS ............................................................................................................ 22 Peakflow SP Communication Ports ........................................................................................................ 24 Data collection ...............................................................................................................................................................24 Inter‐appliance communication (all appliances)............................................................................................24 Peakflow SP PI and leader appliances.................................................................................................................25 

Scaling Clean Pipes 2.0 ............................................................................................................................... 25 Conclusion ............................................................................................................................................28 Appendix...............................................................................................................................................29 Appendix­A:  Peakflow SP System­wide Enforced and Guideline Limits................................... 29 Appendix­B:  Peakflow SP PI 5500 Appliance Enforced and Guideline Limits........................ 32 Appendix­C:  Peakflow SP CP 5500­series Appliance Enforced and Guideline Limits.......... 32 Appendix­D:  Peakflow SP TMS Appliance Limits.............................................................................. 34 Appendix E:  Six­Phase Approach to Infrastructure Security........................................................ 35 

 

  3 

Executive Summary Distributed Denial-of-Service (DDoS) attacks are amongst the most prominent attacks targeting network infrastructures or computer services resources. The primary goal of DDoS attacks is to deny legitimate users access to a particular computer or network resources, which results in service degradation, loss of reputation and irretrievable data loss. DDoS attacks are aimed at businesses of any size and type. Businesses with on-line presence are all potential victims of DDoS attacks, including all vertical markets such as financial, retail, media and entertainment, manufacturing, services and the government. Even individuals are being attacked. Many enterprises are migrating to cloud computing models making use of centralized data centers and virtualization to reduce capital and operations expenses. The data centers that house these large virtualized data stores are particularly sensitive targets to DDoS attack as a single attack can produce considerable collateral damage beyond the direct victim. DDoS has evolved from random hacker exploits to organized criminal activities which often involve botnets, which are large groups of compromised host computers controlled by a central, coordinated commander. The size, complexity and sophistication of DDoS attacks are increasing at alarming rates making it more challenging to protect network resources. According to Arbor Networks’ Annual Worldwide Infrastructure Security Report (2008), Internet Service Providers have seen DDoS attacks as large as 40 Gbps – that’s an increase of 67% from 2007 and a 100 fold increase since 2001. An attack of over 80 Gbps was detected in the first half of 2009 showing that the growth trend is not slowing down. To address these inevitable, growing network threats, network operators are urged to employ the best current practices (BCP) for protecting networks. BCPs are pro-active methods that have been adopted in the industry to prepare networks against threats. BCPs include network infrastructure best practices, host best practices and deployment of dedicated DDoS detection and mitigation, such as Cisco Clean Pipes Solution. Cisco Clean Pipes Solution is the purpose-built architecture for dedicated DDoS detection and mitigation. As opposed to traditional DDoS defense techniques, the Cisco Clean Pipes solution can accurately distinguish legitimate traffic from malicious traffic destined for a mission-critical host or application. It precisely blocks the attack traffic while allowing legitimate traffic to pass through, which enables the maximum business and service continuity. The Clean Pipes Solution allows service providers to deliver in-cloud, managed anti-DDoS services to their customers. It also provides enterprise customers with the ability to defeat DDoS attacks on their own premises with surgical DDoS attack detection and protection with finer granularity. The Cisco Clean Pipes solution has been a great success and widely adopted by many large service providers, hosting providers and large enterprise customers. Given the constant evolving nature of DDoS attacks, the Cisco Clean Pipes Solution also evolves rapidly. Starting with Cisco Anomaly Guard appliance which has 1Gbps mitigation capability, it now features Anomaly Guard Module (AGM) for up to 3Gbps mitigation per module. Multiple AGM modules can cluster offering 10+ Gbps protection. Moving forward, in an effort to achieve cost efficiency and high feature velocity, Cisco has decided to stop developing the Anomaly Guard Module and the Anomaly Detector Module, and

 

  4 

to partner with Arbor Networks to continue to provide a comprehensive and tightly integrated Anti-DDoS solution, which will evolve the Cisco Clean Pipes Solution to version 2.0. In Clean Pipes 2.0, Cisco and Arbor will closely collaborate in the integration of Cisco Netflow technology and Arbor DDoS detection and mitigation technology to provide more advanced and higher performance Anti-DDoS protection. Arbor Networks is well known for their security expertise and Peakflow® SP solution which is deployed in many service provider networks where it provides comprehensive DDoS detection, surgical mitigation and reporting. The Arbor Networks and Cisco partnership is not new as Arbor’s Peakflow solutions have leveraged Cisco’s Netflow technology for a number of years. Also, Arbor’s Peakflow SP product has been a supported option for attack detection in Clean Pipes Solution 1.0 and 1.5. For Clean Pipes 2.0, Arbor’s Peakflow SP product will be used for anomaly detection, while the Peakflow SP Threat Management System (TMS) product will be used for surgical mitigation of DDoS attacks. For customers who’re using Clean Pipes solution 1.0 or 1.5 today, Clean Pipes 2.0 is the migration path to achieve higher scalability and new functionalities in the future. Cisco and Arbor will continue the joint effort of creating an ever tighter integrated anti-ddos solution.

 

  5 

Best Current Practices  There are a number of industry best current practices (BCPs) which should be proactively deployed by network operators responsible for Internet-facing infrastructure and properties. These practices were established in accordance with the Cisco-Arbor developed 6 phases of infrastructure security.

Network Infrastructure BCPs  Network  infrastructure  BCPS  are  pro‐active  measures  that  are  implemented  directly  on Cisco router and switching infrastructure along with other network devices.  

• Interface ACLs (iACLs) should be employed at the relevant network edges (peering/transit, customer aggregation edge, etc.) to protect the network infrastructure itself.

• Service-specific ACLs should be used on data center routers to restrict traffic destined for Internet-facing servers to the ports and protocols associated with the services and applications on those servers. 

• Control and management plane protection mechanisms should be deployed per device, protocol and vendor recommendations

•  All network infrastructure devices should be accessible only via designated management hosts, and this access should be facilitated via a dedicated out-of-band (OOB) management network.

• Flow telemetry using Cisco NetFlow should be enabled at all network edges, and exported into a collection/analysis system such as Peakflow SP

• Source-based remotely-triggered black holing (S/RTBH) is a powerful reaction technique that allows tens or even hundreds of thousands of attacking source IPs to be rapidly black holed based upon their source addresses. S/RTBH leverages BGP as a control-plane mechanism to instantaneously signal edge devices to start dropping attack traffic.

• Reverse-proxy caching in front of Internet-facing Web properties allows for scaling of capacity as well as a policy control point which enables filtering of layer-7 application protocol traffic.

Host Based BCPs  Host  based  BCPs  are measures  that  are  applied  directly  on  hosts  that may  come  under attack and provide a degree of initial protection.     

• Pro‐active patching of the host 

 

  6 

• Server  hardening  including  the  shut  down  of  any  unnecessary  services  and  host based  ACLs  restricting  access  to  the  server  to  only  specific  source  hosts  and  on specific ports. 

• Out of band management access to the device. • Service  specific  configuration  hardening  including  shut  down  of  unused  features 

and access mechanisms • IP stack tuning • Employ anti‐virus and anti‐spam mechanisms 

 

Dedicated DDoS BCPs  The use of dedicated DDoS detection and mitigation infrastructure completes the net of a complete  DDoS  protection  infrastructure.          DDoS  detection  and mitigation  devices  are specifically designed to detect DDoS events as they occur, provide traceback and analysis to operators, and intelligently mitigate attacks by dropping malicious traffic while preserving legitimate traffic. Cisco Clean Pipes is the industry BCP for dedicated DDoS and mitigation.  

Cisco Clean Pipes Solution Overview The Cisco Clean Pipes Solution enables service providers to provide DDoS protection services to their customers and simultaneously harden and protect their own networks. Enterprise customers can also deploy Cisco Clean Pipes Solution on their own premises to protect their network infrastructure and server resources from DDoS attacks.

Evolution of Cisco Clean Pipes Solution The essentials of Cisco Clean Pipes Solution are the DDoS attack detection and mitigation devices. Table 1 shows Anti-DDoS devices in Clean Pipes 1.x and 2.0: Table 1. Anti-DDoS Devices Cisco Clean Pipe Solution Detection Device Mitigation Device

Clean Pipes 1.0 & 1.5

Cisco Anomaly Detection Appliance; Cisco Anomaly Detection Module for Cisco Catalyst 6500/ Cisco 7600; Arbor Peakflow SP;

Cisco Anomaly Guard Appliance; Cisco Anomaly Guard Module for Cisco Catalyst 6500/ Cisco 7600

Clean Pipes 2.0 Arbor Peakflow SP Arbor Peakflow SP Threat Mitigation System (TMS) 

 

  7 

In Clean Pipes 1.0 and 1.5, detection can be done by either Cisco Anomaly Detection Appliance/Anomaly Detection Module or Arbor Peakflow SP. Cisco Anomaly Guard Appliance/Anomaly Guard Module are the featured attack mitigation devices. Clean Pipes 2.0 will use Arbor Peakflow SP for detection and Arbor Peakflow Threat Management System for mitigation. Cisco routing and switching devices will provide Arbor Peakflow SP with netflow information which Peakflow SP will use to analyze and establish network traffic profile and detect traffic anomaly. Differences exist among Cisco and Arbor Anti-DDos devices in terms of how they function for DDoS attack detection and mitigation. Cisco Traffic Anomaly Detection vs. Arbor Peakflow SP: Both Cisco Traffic Anomaly Detection Appliance and Cisco Traffic Anomaly Detection Modules are packet-based anomaly detectors. They monitor a mirrored copy of selected inbound traffic flowing toward destinations under protection, building detailed profiles of normal behavior of each protected devices. Any activities deviating from these profiles can be potential attacks. If it senses abnormal or anomalous behavior, the Cisco Traffic Anomaly Detection device dynamically configures a set of dynamic filters to record the events and trigger an alarm to the network staff. It can also signal Cisco Anomaly Guard devices to activate the protection and mitigation if configured to do so. Arbor Peakflow SP is a netflow-based anomaly detector. It receives Netflow telemetry from Cisco routers and switches in the network. It continually models network behavior based on the netflow statistics creating baselines of expected traffic rates. Any events deviating from the established baseline model will be identified as an anomaly and trigger an alert which can lead to further actions, including informing the network staff, and/or activating the DDoS protection function on the mitigation device. Cisco Traffic Anomaly Guard vs. Arbor Threat Management System (TMS): Cisco Traffic Anomaly Guard XT and Cisco Traffic Anomaly Guard Modules are designed as central intelligence devices which are capable of both detecting and mitigating attacks, once they have been activated. When the Cisco Traffic Anomaly Detector Module (or any other Anomaly detection device) identifies a potential attack, it alerts the Cisco Anomaly Guard Module to begin dynamic diversion, which redirects traffic destined for the targeted resources-and only that traffic-for inspection and scrubbing. From point of diversion start, Cisco Guard operates independently from other devices. It applies blocking techniques, based on Cisco's unique multi-layer verification process architecture, which delivers multiple interactive layers of defense to identify and block all types of attacks. Arbor’s Peakflow SP Threat Management System (TMS) is designed to provide centralized cleaning capacity in the network providing the active packet level processing needed to thwart complex attacks. TMS maintains an active communication with the Peakflow SP system for ongoing exchange of mitigation activities, health of the TMS scrubbing capacity, real-time data exchange for mitigation and supplemental application visibility. The TMS features a set of mitigation countermeasures that are designed to isolate and block malicious traffic while passing

 

  8 

desirable traffic. The countermeasures available currently include anti-spoofing, host authentication techniques, packet level threshold, application specific threshold, protocol verification, baseline enforcement, idle discovery, blacklist/whitelist and payload filtering techniques. Countermeasures are continuously added or updated as new threat vectors emerge. In addition, extensive real time and post mitigation reports are available on the Peakflow SP system allowing operators to make more informed decisions on how to adapt defenses during and between attacks. The Peakflow SP system provides a single pane of glass for command, control and reporting for one or more TMS devices. Together, the Peakflow SP system and the TMS devices provide a comprehensive threat management solution for the entire network.

Protection Mechanism of Cisco Clean Pipes Solution DDoS attacks are among the most difficult network threats to defend against. They mimic valid requests, spoof source identification, and use armies of compromised “zombie” hosts to initiate attacks so that illegitimate packets are indistinguishable from legitimate packets. This makes detection more difficult and threat mitigation with business continuity more challenging. Network devices and traditional perimeter security technologies such as firewalls and intrusion detection systems (IDSs) do not by themselves provide comprehensive DDoS protection. To pick up where traditional DDoS defense techniques leave off, the Cisco Clean Pipes solution can accurately distinguish good traffic from bad traffic destined for a mission-critical host or application. It not only detects the presence of an attack, but also filters out only the bad traffic, allowing good traffic to pass through, enabling maximum business and service continuity. This solution offers three major functional elements that work towards protecting a network from DDoS attacks:

Detection

Identify and classify attacks

based on anomaly characteristics

Diversion/Injection

Divert “dirty” traffic to the

cleaning center to be “scrubbed”

inject clean traffic back to the DDoS targeted host

Mitigation

Anti-spoofing,

anomaly recognition and packet inspection

cleaning(scrubbing) of “dirty” traffic

• Detection The fundamental premise of detecting attacks is to build a baseline of normal network traffic levels and then look for anomalies in traffic patterns compared with the baseline. A network traffic anomaly is an event or condition in the network characterized by a statistical abnormality compared to typical traffic patterns gleaned from previously collected profiles and baselines. Any difference in traffic patterns that are above a certain threshold will trigger an alarm.

• Traffic diversion and injection Traffic diversion is the mechanism used to instruct an upstream router in the core network to divert traffic of compromised servers to the mitigation devices for scrubbing the “dirty” traffic. After scrubbing off anomaly packets, the cleaned traffic is injected back to the normal data path to reach the destination in the network. There are multiple mechanisms for traffic diversion and

 

  9 

injection which will be discussed in later sections. • Mitigation Mitigation in the Cisco Clean Pipes solution is the process in which attack traffic is “scrubbed” (i.e., checked via anti-spoofing, anomaly recognition, packet inspection, and cleaned to drop bad traffic and allow legitimate traffic to the same destination). The figure in below shows the typical Cisco Clean Pipes Solution Architecture:

In general, Cisco Clean Pipes Solution provide four specific service deployment models, based on the common Clean Pipes architecture, along with design guidelines tailored for DDoS protection for different parts of the SP infrastructure and customer networks: – Managed Network DDoS Protection — Provides enterprise customers effective protection against DDoS attacks on their last-mile connections to SPs and internal infrastructures by subscribing to the Cisco Clean Pipes service offered by SPs. – Managed Hosting DDoS Protection — Enables hosting providers to protect their web and other hosting services from DDoS attacks. – Peering Edge DDoS Protection — Enables SPs to prevent bandwidth saturation by DDoS attacks against their peering points. – On-premise DDoS Protection — Enables enterprise customers to deploy anti-DDoS detection and protection on their own premises with finer granularity for anomaly detection and protection.

 

 10 

Cisco Clean Pipes 2.0 Components Cisco Netflow Netflow, as a indispensable tool to provide the visibility into network traffic, has become a standard for acquiring IP operational data for many customers. Applications for NetFlow data are constantly being invented. One of them is Anti-DDoS protection. The highly scalable view of network traffic characteristics provided by netflow data makes netflow technology the most widely deployed DDoS identification technology for large scale IP networks. At the same time, the granular flow information enables netflow-based DDoS detection devices, such as Arbor Peakflow SP, to provide a surgical detection of traffic anomalies.  Netflow classifies IP packets into flows and generates flow records which can be exported to a flow collector for further analysis. Each flow is defined by its unique seven-key characteristics:

• Ingress interface • IP protocol type • Type-of-service (ToS) byte • Source IP address • Destination IP address • Source port number • Destination port number

Led by Cisco core routing and switching platforms (e.g CRS-1, Nexus 7000, ASR 1000 and Catalyst 6500), Cisco Netflow technology has been constantly enhanced and refreshed, such as:

• Supports multiple netflow formats (e.g. v5, v7. v8 and v9); • Industry’s premier platform to support v9; • Highly scalable netflow table; • Flexible netflow which allows users to select which key or non-key fields to

define a flow. It grants users more flexibility, aggregation and scalability; • Time-based sampled netflow; • Packet-based sampled netflow; • Netflow for both ingress and egress traffic; • Netflow for MPLS & Multicast traffic; • Netflow for bridged traffic (enables bump-in-the-wire deployments);

When the network is operating under a normal situation, Netflow yields enough data to profile the network traffic and establish a baseline which is used for traffic anomaly detection. In the event of DDoS attacks, the statistic netflow information shows deviations from the traffic baseline which can be the first sign of the attacks. Further analysis of traffic pattern and behavior can be carried out with the detailed flow information. Once a traffic anomaly is identified, corresponding Anti-DDoS countermeasure can be initiated manually by the network operator or automatically by the Anti-DDoS protection system. When Netflow is used in Anti-DDoS protection, NetFlow is usually deployed across the edge of an SP or enterprise network to monitor inbound traffic on edge and peer interfaces, because these are the typical ingress points for most attacks. The router maintains a live NetFlow cache to track the current flows. IP flow information can be exported from the NetFlow cache to an external

 

 11 

collector for further analysis. In Clean Pipes 2.0, Arbor Peakflow SP is the flow collector. Flow data from multiple collectors can be mapped to identify the network nodes under DDoS attack and also to determine the attack characteristics. Cisco will continuously invest in high performance netflow technology and collaborate with Arbor Networks to ensure that the Clean Pipe Solution 2.0 get enhanced with the new Netflow features. This allows the maximum investment protection of Cisco network platforms for Clean Pipe Solution 2.0 customers. For more information about NetFlow, see the following URL: http://www.cisco.com/en/US/tech/tk812/tsd_technology_support_protocol_home.html

Arbor Peakflow SP Arbor Networks Peakflow SP is a scalable platform that provides a comprehensive solution delivering powerful DDoS as well as traffic and routing analysis to service providers and their customers. Peakflow SP Provides three leading solutions to the market place: managed security services enablement, infrastructure security, and traffic and routing visibility and analysis. The Peakflow SP solution scales with its multi-tier detection architecture of collectors:

- Tier 1: Peakflow SP Portal Intelligence (PI), which provides a central point of command and control including event correlation and traceback. PI systems provide the leader and central command function for the deployment.

- Tier 2: Peakflow SP Collector Platform (CP), which collects NetFlow statistics from multiple routers and acts as a correlation engine syncing data sets between all network collectors and the PI system.

- Tier 3: Peakflow SP Flow Sensor (FS), that acts as additional layer of netflow and data collection designed to scale the Peakflow deployment to the largest world-wide networks.  

For Clean Pipes v1.5, the Peakflow SP solution works in conjunction with the Cisco Guard for DDoS protection. Upon receiving an anomaly fingerprint for a zone from a Peakflow SP CP collector, the Peakflow SP PI controller establishes a SSH connection to activate the Cisco Guard, putting the zone under attack in protection mode. For the Clean Pipes 2.0 Solution, Peakflow SP offers a streamlined approach to DDoS attack detection, traceback, and mitigation. Peakflow SP CP systems first build baselines of normal behavior, network-wide, leveraging flow data available from the routers already deployed on their network. In contrast to inline data collection methods, Peakflow SP collects Cisco NetFlow flow-based statistics from Cisco routers, which allows Peakflow SP to scale with the network. Alternatively, Peakflow SP TMS can use packet capture/ SPAN ports on routers in which NetFlow is not available. Neither NetFlow nor packet capture imposes a performance or reliability impact upon the network; the data collection is non-intrusive. The Peakflow SP network-wide anomaly detection identifies attacks using the two most effective methods available: signature analysis and dynamic profiling. Arbor’s Active Threat Feed (ATF), a data feed of traffic signatures that pinpoint potential threats and concerns to network security can be used to match traffic reported via NetFlow. Alerts based on ATF matches can be reliably detected through netflow analysis. Peakflow also actively detects anomalies through misuse identification and dynamic profile detection. Netflow provides Peakflow SP the unique

 

 12 

perspective to run signature analysis pervasively with a high level of accuracy while augmenting that network wide visibility with targeted packet processing analysis of suspect traffic through Arbor’s Peakflow SP TMS. The figure below shows the Clean Pipes 2.0 Solution Architecture:

Arbor Peakflow SP Threat Management System (TMS) Arbor Network’s Peakflow SP Threat Management System (TMS) provides surgical mitigation, service analysis and reporting. TMS provides scrubbing and application specific visibility to the Peakflow SP system. TMS can be deployed in centralized scrubbing locations, regional service POPs and IDCs for infrastructure protection and clean pipes. The TMS can also be deployed as dedicated solution for specific service protection and visibility. Arbor Networks TMS is a separate and purpose built hardware platform for advanced, high-speed traffic scrubbing and analysis. The TMS product family contains systems that provide throughput from 1.5 Gbps up to 10 Gbps throughput. The range of TMS models offered provides the correct performance package for each of the multiple use-cases of the Clean Pipes 2.0 Solution. Each system provides the same feature and functionality at different performance levels to meet the desired use case. Deployed in centralized locations for scrubbing center architecture, the 5 Gbps TMS-3050 and 10 Gbps TMS-3110 models provide very high speed performance with the ability to manage multiple events on the same platform. Multiple deployments of TMS distributed throughout the

 

 13 

network within the network can provide a distributed response to a coordinated DDoS event. Grouping multiple TMS systems into a single logical entity provides network operators the best solution to difficult trade-offs of backhauling attack traffic across the network or requiring each of the systems to be individually configured and managed. Grouping the TMS systems provides the ability to maintain a disturbed attack to a geographically limited set of locations and protects collateral damage of network assets. The figure below provides range of TMS models available in the Clean Pipes 2.0 Solution.

TMS can also be offered as a dedicated solution for specific service protection or customers of a Clean Pipes service. As a dedicated solution, TMS provides specific mitigation actions to a specific customers leveraging the integration with Peakflow SP CP systems and the ability to model customer traffic through Netflow analysis saving these baselines in Managed Objects, each TMS can also employ a customer specific template of mitigation countermeasures to ensure custom handling of the event with respect to sensitive traffic. The TMS interacts with distributed Peakflow SP CP systems in the network to both baseline data

 

 14 

at Layer 7 as well as provide scrubbing statistics and forensic data gleaned from attack events. Advanced visibility within the attack event provides actionable data to the user while the mitigation is ongoing. This near real time interface provides best of breed management of DDoS events augmenting the operations tool kits with the ability to drill into attack packets, correlate data common to the attack traffic streams as well as measure the effects of countermeasure filters or REGEX expressions before they are employed into the configuration ensuring the least amount of negative impact to good traffic during the event.

DDoS Protection Flow in Clean Pipes 2.0 1. Baseline and Thresholds

Collection  of  Netflow  data  from  various  router  locations  and  correlating  this  data  into  a comprehensive model of  the network  is  critical  to development of a  surgical  response  to threats.  The Clean Pipes components provide this functionality natively in the solution and ensure  that  normal  network  traffic  variability  is  accounted  for  in  these  measurements.  Further  developing  granular  models  of  network  assets,  customers,  services,  and infrastructure serves to provide accurate levels of granularity relative to the network scale for pinpoint detection.  The Clean Pipes 2.0 solution provides the ability to accurately build thousands of models, each with relative baselines, thresholds, and traffic pattern reporting to scale to very largest service offerings and global networks. 

2. Detection Data retrieved from Netflow updates provided by Cisco Router infrastructure is correlated to  the baseline and  threshold data held  in Peakflow SP system. Clean Pipes 2.0  identifies threshold  violations  and  provides  actionable  information  to  the  operations  teams  as rapidly as possible.  The system then provides options to the operator who has the options of auto‐mitigation, manual mitigation, use of layered mitigation techniques such as ACL or interface level filters on Cisco infrastructure, or black holing traffic are some choices to be made.  If the attack is to be mitigated through the scrubbing technology such as TMS, then the operator will initiate a diversion event.  

3. Diversion Surgically redirect (off‐ramp) traffic into scrubbing locations using the BGP control plane, remove  traffic  attacking  the  network  and  pass  that  attack  traffic  into  locations  of  the network  with  the  scale  and  the  scrubbing  systems  deployed  to  mitigate  the  attack. Architecture goals of following best practices have greatly enhanced methods for diversion and  distribution  of  diverted  traffic  to  scrubbing  systems.    Multiple  methods  exist  to accomplish this successfully  including BGP Anycast, BGP route‐maps and BGP community use  can  all  ensure  that  the  network  is  resilient  to  the  threat  itself,  maintaining  a  self‐defending network. 

 

 15 

4. Scrubbing Identification of malicious and legitimate traffic through DPI Packet analysis, heuristics and validation methods called countermeasures.  Each countermeasure can provide additional granular  identification of  traffic.   Malicious  traffic  is removed  from the  traffic stream and legitimate traffic is placed back into the network. All actions taken by the TMS are reported in  both  the  real‐time  mitigation  report  and  information  is  included  in  the  after‐action reports for the event. 

5. Injection Post Processed legitimate traffic routed through a unique path to the ultimate destination of  the original  traffic  flow.   Like diversion, architectures exist  to optimize  injection paths and traffic delivery to victims of attacks.  These methods can leverage network capabilities enabled by Cisco routers as well as provide  for additional opportunity  to  the Clean Pipes provider for additional dedicated, clean capacity into the destination network.     The following diagram maps the DDoS Protection flow in Clean Pipes 2.0

 

 16 

Migration to Cisco Clean Pipes 2.0 As the Cisco Guard and Detector Modules approaching the end of their life cycle, the Arbor Peakflow SP Threat Management System (TMS) replaces the scrubbing technology in the clean pipes solution. The migration from the Cisco Guard to the Threat Management System (TMS) can be achieved through a mapping of protected resources configured in the Cisco Guards (Zones) to Peakflow SP (Managed Objects) as well as deployments of TMS in similar architectures as the Cisco Guard. Understanding the terminologies of Cisco Guard and Arbor PS/TMS and knowing how to map them will greatly increased knowledge transfer from the successful deployments of Clean Pipes 1.0/1.5 with Cisco Guard appliance/module for successful migration to Clean Pipes 2.0 with Arbor TMS for mitigation. The table below shows the key concepts mapping between these two technologies. Cisco Guard  Arbor Peakflow SP

TMS Common Definition 

Zone  Managed Object  The basic models that builds baseline, detection, mitigation and reporting. Model definitions are flexible and can be combined with many layers of data to specifically match critical areas of interest on the network. These models are used to monitor customer, peer, service, or profiled relationships. 

-  Boundary  A boundary demarcation point between administrative domains. Peakflow SP immediately builds a topological map of the monitored network using the network definition as the default global boundary. Boundaries are flexible and can be inherited such as a global boundary or specifically configured relative to the Managed Object monitored. The global boundary defines the point traffic enters or exists the monitored network. 

Baseline  Baseline  Collection of traffic behavior profiles building the expected traffic volume and anomaly detection thresholds. 

Protect mode / Zone Protection Enabled 

Mitigation   – Enabled Scrubbing configuration. This can include details about the destination, BGP prefix used to

 

 17 

change traffic path, active and passive filtering rules. 

Filter  Countermeasure  - Rule describing an evaluation of traffic to be scrubbed by the solution  

Template  Mitigation Template  - Preset configuration information used to protect a destination from specific vectors or to use specific filters in protection. 

Diversion  Off-ramp  - BGP Prefix announcement of destination with a change in Next-hop attribute to ensure inbound traffic will pass through the scrubbing solution. 

Re-injection  On-ramp  - Returning the cleaned traffic post processing from the scrubbing system to the network in a loop-free path to the protected destination 

Cisco Anomaly Guard and TMS Countermeasure Comparison  Cisco Anomaly Guard and Arbor Peakflow SP TMS provide for significant mitigation capabilities based on available countermeasures. These countermeasures are designed to deal with the current common DDoS attack types as well as provide a capability to protect a target from a zero-day attack as well. Both solutions provide strong protection from spoofed source attacks, resource consumption attacks at the application layer, session layer or network layer. Both solutions provide solutions to brute force flooding attacks such as packet per second floods, TCP connection floods, UDP and ICMP floods to name just a few examples of common attacks. TMS provides for advanced capabilities to filter web-enabled (HTTP-based) services through authentication, validation, request tracking and limits as well as payload filtering. Each of these countermeasures, or any set of countermeasures can be brought into service across a group of TMS systems through the configuration of a single mitigation. This capability sets the TMS apart from previous mitigation solutions for large, distributed network deployments.

Mitigation Countermeasure Function TMS Cisco Guard

White list / black list filtering √ √

Per source IP rate thresholds √ √

TCP SYN Authentication with reset to Host √ √

TCP SYN Authentication with refresh sent to host √

TCP SYN Authentication with HTTP Authentication √ √

TCP SYN Authentication with safe reset to host √ √

 

 18 

TCP SYN ACK Authentication √ √

TCP other flag authentication √

Basic/Default authentication of other protocols for client based on passed TCP authentication √

Strong Mode TCP Authentication using proxy * √

DNS Authentication through packet drop / re-transmission √

DNS Authentication By Reflexive-Redirection Target Release 5.1 Q4 2009

DNS Request Type Limiting By Source /32 Target Release 5.1 Q4 2009

DNS Cache Poison Defense * Target Release 5.1 Q4 2009

DNS Authentication by converting to TCP Target Release 5.5 Q2 2010 √

Strong Mode DNS Auth using TCP and TTL Target Release 5.5 Q2 2010 √

Regex based filtering √ √

DNS DPI REGEX Filtering X

Further Enhanced Target Release 5.1

Q4 2009

HTTP Header REGEX Filtering √ √

Protocol baseline enforcement √ √

Source /24 based baseline enforcement √

Connection metrics based baseline enforcement √

Rate limiting √ √

Malformed HTTP √

Malformed SIP √

Malformed DNS √ √

SIP Authentication √

SIP source request thresholding √

 

 19 

HTTP source IP rate thresholding √

HTTP source IP object get rate thresholding √

TCP Idle Timeout √ √

TCP multiple bad connection blacklisting √

* Requires the mitigation system to be in-line in both directions Despite the differences in specific filtering options available, each of the mitigation systems have demonstrated an ability  to effectively mitigate  the  types of  threats  that are seen on the Internet today.   Cisco and Arbor performed joint testing on the mitigation capabilities of  the  TMS  system  and  the  tests  concluded  that  the  TMS was  able  to mitigate  the  same types  of  attacks  that  the  Guard  was  capable  of  mitigating.      It  sometimes  used  similar methods as the Guard solution and sometimes used different methods that were proven as effective. 

Clean Pipes 2.0 Deployment Considerations The Clean Pipes 2.0 solution provides a wide variety of protection values.

- Managed DDoS Detection and Protection Services: Arbor Peakflow SP provides a complete solution for providers to offer a turnkey managed DDoS service. Managed objects monitoring each component of the subscribed customer network and services.

- Managed Hosting DDoS Protection: Protection of critical services at the application layer with unique capabilities to further differentiate increasingly sophisticated attacks at the application layer itself. Managed services offerings can be offered granularly to customers per application or critical service that may be protected or protection of the customer site entirely.

- Peering Point Bandwidth Protection: Detection and mitigation capabilities at the peering points can protect against collateral damage from transit of DDoS attacks.

- Network Service Protection: Arbor Networks TMS can provide service specific application reporting as well as dedicate DDoS protection to critical network services.

Netflow considerations for Peakflow SP Netflow telemetry is an intrinsic part of the Peakflow SP solution.   Peakflow SP leverages Netflow data  to  provide  operators  complete  visibility  into  network  traffic  characteristics and  rates,  create  baselines  of  normal  traffic,  detect  deviations  from  these  baselines  that may be due  to  threats,  characterize  the  threats and  trace  them back  to network borders.    In  short,  Netflow  provides  the  basic  building  blocks  for  each  of  the  6  phases  of infrastructure security.  

 

 20 

Pervasive monitoring of the network is a necessary component of the Clean Pipes solution so Netflow  telemetry  should  be  exported  from  all  Cisco  routers where  critical  data may traverse including those in the network peering, core, distribution and data center.    It is recommended that ingress Netflow be enabled on all logical interfaces for each router being monitored.   This provides the Peakflow solution with a full picture of what traffic is going  through  the  router  regardless  of  direction.        Peakflow  has  the  ability  to  focus  on desired traffic within each Netflow data stream so it is not necessary to filter what traffic is sent to the Peakflow SP collectors.    Unsampled netflow provides accurate flow information on the network traffic which can be leveraged by features such as ATF and fingerprints that rely on matching certain behaviors for  more  effective  triggers.  However,  when  the  network  traffic  load  is  high,  unsampled netflow can limit the scalability of anomaly detection due to the excessive processing work load  on  the  flow  reporting  devices  and  the  Peakflow  appliance.  In  this  case,  sampled netflow  can be  implemented  as  a way of  facilitating  a  better  scalability. The  appropriate sampling rate to apply is a function of what type of router it  is, how much traffic  is going through the router and what line cards are available in the router.   Sampling rates can go over 1000:1  in  large hardware accelerated platforms  (CRS‐1, Catalyst 6500).      In general the more traffic going through, the higher the sampling rate that would be applied.   When exporting Netflow from routers that are carrying IPv6 or MPLS traffic, it is necessary to utilize Netflow v9 with IPv6 and MPLS explicitly enabled to get visibility into this traffic.  The following are specific considerations when enabling Netflow on Cisco devices:  

• Netflow versions 5, 7 and 9 are supported on Peakflow devices. • Netflow export should always be set to a 1 minute active flow timeout ensuring real 

time analysis can be done on longer lived connections. • On  Catalyst  switches,  full‐interface  flow  mask  should  be  used  to  ensure  that  all 

Netflow fields available will be populated • Peakflow SP supports a single sampling rate per network device so do not configure 

multiple sampling rates on a single router. 

Deployment of Peakflow SP PI  The Peakflow PI appliances provide users direct access to the data stored and distributed in the collector devices in the network. The PI functions as a leader to coordinate all data reports, alerts and system health, data from the individual collectors and present that data in a unified view. The PI appliance provides for secure access to user, administrators and service customers groups. Each PI deployed provides access to the deployment through the GUI or API. PI appliances support hot/hot active redundancy of up to 10 devices and together act as a unified access point into the deployment. Administrative, DDoS Alert and API access is synced between PI systems

 

 21 

to ensure each system provides equal level of access based on centralized access, accounting and user privileges. For the hardware enforced scalability limits and the guidelines for scaling the Peakflow SP PI deployment, refer to Appendix-B.

Deployment of Peakflow SP Collectors Peakflow SP collectors provide distributed data collection and detection in the network deployment. Detection for specific customer or services on the network can be done through the processing and correlation of netflow information from any point that traffic passes through the network. This provides the Peakflow system the ability to scale to the network cloud for detection and reporting. By intelligently matching traffic to the object definition within the system each configured managed object baseline is developed constantly and reported virtually. Placement of the collector is critical to ensure that detection and reporting visibility are optimized for the protection and reporting desired. Pervasively monitoring peering capacity, external border connectivity and long haul capacity typically provides value in increased transit optimization. Protecting the network from off-net or transiting attack traffic is often the first level of protection successful Managed DDoS services as well as Peering point protection. Deployments of Collector platforms at the aggregation or within the core of the network ensure that on-net traffic does not become a threat customers or network assets from internal attack. This layer of detection can be critical in isolation of internal network attacks, customer-to-customer attacks, capacity issues, and critical network service monitoring. Setting network Netflow settings for sampling rate, export timing, Netflow export locations and pervasive enablement all affect the detection and reporting of data on the network. Peakflow SP can provide for very accurate detection and reporting through sampled Netflow processing. Evaluating the correct level of sampling ratio relies on both the ability of the router platform and the software version its running. Recent versions of IOS and Cisco router infrastructure provide extremely accurate and high performance platforms for Netflow export to Peakflow SP Collectors. Peakflow SP Collectors also manage downstream systems such as TMS. TMS reports health and mitigation statistics as well as additional Netflow data back into the Peakflow SP collector. All inter-device communications are carried out over SSL. Through this secure connection, the collector and TMS have a dedicated link to exchange detection data, baselines, thresholds and mitigation configurations. This link is secure and can be maintained over geographically separate devices. The following diagram illustrates deployment concepts of Peakflow SP CP and TMS

 

 22 

   For the hardware enforced scalability limits and the guidelines for scaling the Peakflow SP CP deployment, refer to Appendix-C.

Deployment of Peakflow SP TMS  Deployment of the mitigation component of the Clean Pipes solution can be optimized to fit your specific network architecture. Considerations of the specific protection values will influence the deployment of the TMS into the network. The TMS can be deployed within the Peering layer of the network or adjacent to the network border to ensure that off-net attacks are mitigated directly at the network edge. This provides for an advantage of a direct solution to peering point threats and the ability to maintain distributed attack traffic separated to ensure that it does not threaten the network through an aggregation of this traffic. Central scrubbing capacity can be deployed in centralized locations to provide a consistent experience to protection services customers. Regional mitigation or scrubbing centers provide dedicated locations where diverted attack traffic can be scrubbed and returned to the ultimate destination of the threat traffic. Providing regionally or geographically based mitigation capacity can provide protection services to regional customers avoiding excessive backhaul of traffic. Dedicated TMS deployments can provide advanced value for specific infrastructure, customers or services. Dedicated systems can be deployed directly in data centers adjacent to the resource

 

 23 

can ensure that application reporting, service specific reports, performance data, change alerts and packet level forensics that can aid in both validation of normal operation but also improve troubleshooting and application visibility. Data center placement of TMS appliances also provides protection for critical resources from sources within the network that may not be protected by peering and central scrubbing locations. Broadband consumers, infected hosts within the network and customers networks present a potential threat to network resources and must be considered when building defenses. The following diagram depicts TMS network deployment locations:

Diversion of traffic into the TMS systems is most often triggered by an IP traffic routing change such as a BGP announcement changing the attack target destination route to a next-hop of the mitigation systems available. BGP route maps, anycast route announcement, community attribute setting can all be used to design a specific solution to the problem of diverting traffic into central or distributed TMS systems within the network for mitigation. TMS provides the ability to group devices into a single event to gain efficiency of central management as well as provide scale to the mitigation capabilities. Policy based routing as well as static routes may also provide local traffic diversion into the system. In some cases diversion can also be accomplished through local techniques at layer 2 to ensure that traffic is passed through the TMS. Policy based routing architectures, static ARP table entries and VLAN mappings all can provide for layer 2 diversions into the TMS systems. Reinjection of traffic post processing through the TMS must avoid the diversion method used to ensure that a routing loop does not occur. This reinjection is typically accomplished through GRE encapsulation from the TMS to the provide edge or CPE device. Another well known method is to configure a MPLS VRF instance to separate the forwarding data from the diversion

 

 24 

segment and the reinjection segment. Last, VLANs can be used to separate the forwarding path if the destination is reachable through a layer 2 domain. Successful reinjection of cleaned traffic into the network establishes an alternative traffic flow of the attack traffic from the normal data path to a clean virtual pipe anywhere within the network. For the hardware enforced scalability limits and the guidelines for scaling the Peakflow SP TMS deployment, refer to Appendix-D.

Peakflow SP Communication Ports  For Peakflow SP to function properly, the appropriate communication ports must be allowed through the network devices and firewalls. For all appliances, the following ports must be allowed:

• NTP • SNMP management (recommended) • Management console < > CP/FS/PI/TMS

Data collection 

The following ports must be allowed through the firewalls so that the Peakflow SP CP, FS, and TMS appliances can collect data:

Port Traffic flow

BGP 179 TCP appliance < > routers

UDP flows any (default)

• router > CP • router > FS

SNMP

• CP > router • FS > router

Inter‐appliance communication (all appliances) 

The following ports must be allowed through the firewalls so that the CP, FS and TMS appliances can communicate with other appliances:

Port Traffic flow

ArborFlow (31373 UDP)

• FS > CP appliance • TMS > SP appliance (if you have ArborFlow enabled)

 

 25 

HTTPS 443 • CP browser > SP leader appliance

SSL 40000-40030 (configurable)

• CP/FS/PI > CP/FS/PI

 

Peakflow SP PI and leader appliances 

The following ports must be allowed through the following firewalls for the PI and leader appliances:

Port Traffic flow

HTTPS 443 CP browser > SP leader appliance

Remote Arbor services

• remote services HTTPS/443 • routeviews (off by default) • ATF • anonymous statistics • fingerprint sharing

Local services

• DNS • NTP • AAA/TACACS

 

Scaling Clean Pipes 2.0 Peakflow SP provides the largest network scale available today. Using a many to one monitoring model made possible by using Netflow, the system can scale to monitor over 2200 routers within the network. As more and more collectors are deployed for visibility each collector increases the total number of routers monitored. As the number of collectors increases another layer of the detection hierarchy can be added to the solution to further increase its ability to provide detection across very large networks of routers. The Flow Sensor provides collector level functions at the aggregation level of the network. This extension of the collector platform further increases the ability of Peakflow SP to provide

 

 26 

pervasive coverage of both the external border of the network as well as the internal aggregation edge for detection. TMS provides a range of capacity in the mitigation portion of the solution ranging from 1.5 Gbps to 10 Gbps throughput. Collectors can manage up to 50 TMS systems within a single deployment. Peakflow SP allows for up to 1000 native managed objects on a system for baseline, threshold and reporting data. This number can be scaled to a total of 10,000 managed objects monitored on a single deployment with the addition of Business Intelligence appliances. This additional component provides scale to the number of managed objects in incremental blocks of 500 to grow the monitoring and detection capabilities to keep pace with network and service growth. Appendix A through D provides the scalability numbers and guidelines for the Peakflow SP system and each individual appliance.  Best Practices in Cisco Clean Pipes 2.0 General deployments of Clean Pipes Solutions follow a set of known best practices to ensure the successful delivery of the service. Operational experience and problems have improved on the best practices employed in dealing with attacks over the course of the last decade the Internet has been experiencing both a growing frequency and severity of the DDoS attacks.

- Operationalize the Six Phases of Security Best Practice: Jointly developed by Arbor Networks and Cisco Systems, the six phases of security is a framework for continuous assessment, action and improvement. Following this process of preparedness and improvement is a critical step to deploying any security solution, including a Clean Pipes solution.

- Maintain a separate and secure management network for the Clean Pipes solution. The communication between detection and mitigation components in the solution is most critical during an attack so a separate, isolated management network is required to ensure that communication is maintained during adverse conditions.

- Out of band management to the devices in the Clean Pipes solution must be employed to

ensure that network operators will have full access to the systems at all time and can perform necessary maintenance without significant outage.

- Build sufficient mitigation capacity: Minimum capacity requirements are typically equal

to the amount of available bandwidth at the network border. Typically providers target enough capacity to protect critical infrastructure and match that capacity to ensure that peering capacity can be cleaned. This ensures that attacks that overwhelm the capacity of a given resource can still be effectively mitigated.

- Ensure comprehensive path detection for the network resources protected: Ensure that

any critical path of traffic to or from critical network resources have a level of detection that will measure traffic threshold violations.

 

 27 

- Minimize traffic backhaul and maintain distribution of the attack traffic: Traffic backhaul and aggregation of the attack traffic may actually result in a portion of the network being overwhelmed. Managing the attack traffic in a distributed manor will limit the possibility that the traffic will overwhelm network resources.

- Manage attacks as near the source of the attack as possible: Trace-back traffic to its

source and limit the network’s exposure to that traffic will limit collateral damage and impact of the event on the network.

- Build redundancy into the mitigation solution such that the network will remain protected

even if one or more mitigation devices become unreachable - Drop known threat traffic as quickly as possible: Bringing operational and historical

knowledge of attack vectors and their sources can aid in the amount of traffic that must be analyzed when protecting a network service such as DNS, VoIP or Web Services. Being able to run a course-grained filter before a more surgical filter is employed will limit the amount of advanced scrubbing capacity required to mitigate attacks.

- Automate response and traffic redirection when ever possible: Automation will increase response time to network events and attacks. Where possible, course filters, traffic diversion and traffic scrubbing can be automated to ensure rapid response to threats.

- Maintain reports and data from the alert through the mitigation for analysis: Reporting

and comprehensive history of mitigation actions provide learning opportunities for future events.

 

 28 

Conclusion Clean  Pipes  solutions  have  been  rolled  out  throughout  the  world  as  an  answer  to established  and  evolving  DDoS  threats.    These  solutions  have  been  a  result  of  the  close relationship  between Arbor  and  Cisco  Systems  to  both  develop  as well  as  leverage  each company’s technology to bring about a better combined solution.  Clean Pipes 2.0 continues to provide the best available solution to emerging DDoS threats.  Clean Pipes 2.0 now focuses technology from each of the providing partners into their core competency and furthers the development of these technologies by focused and expertise of each vendor.  The partnership between Cisco Systems and Arbor Networks continues to provide best of breed solutions to customer problems and evolve existing solutions to the next phase of capabilities.  

 

 29 

Appendix 

Appendix‐A:  Peakflow SP System‐wide Enforced and Guideline Limits The following table includes the system‐wide enforced limits: 

Type  Limit 

CP appliances  35 Important: You must assign a PI appliance as the leader if you have five or more CP, TMS, or BI appliances in your deployment. Up to 30 CP appliances are supported in a single deployment. 

FS appliances  150 (maximum of 5 per CP appliance) 

TMS appliances  50 (maximum of 5 per CP appliance) Note: Up to 3 stacked TMS 2700 appliances count as 1 appliance toward this limit. 

PI appliances  10 

Third‐Party Firewalls  10 (Cisco Guard) 

Monitored routers  2,250 (maximum of 5 per CP appliance or 15 per FS appliance) 

Monitored interfaces  100,000 

Managed objects  10,000 Note:  If you have more than 1,000 managed objects, then you must add Business Intelligence (BI) appliances for additional managed object storage. 

Mitigation templates  1,000 

Fingerprints (2.0)  100 

Applications  500 Note: These are also bound by the managed object limit. 

  

 

 30 

 The following table includes the system‐wide guideline limits: 

Type  Limit 

Concurrent logins per PI appliance on a 10 PI load. 

125 (requires multiple PI appliances) 

Concurrent logins to the leader appliance 

10 

Configured users  500 

SOAP queries per minute, per appliance 

200 

Simultaneously active DoS alerts  1,000 

BGP traps  100 

Active fingerprints  20 

Managed objects with filters  20 

Reports  500 completed or up to 20 GB of disk space 

Report templates  500 

Active mitigation actions  1,000 Note: This includes blackhole offramps, TMS mitigations, Cisco Guard offramps, and Flowspec mitigations. 

CIDR group prefixes  50,000 Note: This limit applies to CIDR entries across all CIDR groups, including duplicates. 

Unique CIDR blocks across all CIDR groups for all managed objects 

2,000 

 

 31 

Number of prefixes per CIDR group entry 

100 Example: If you have a CIDR group called “Datacenter” that has three prefixes (10.0.5.0/24, 10.0.6.0/24, 10.0.8.0/24), then you can add prefixes, but you cannot exceed 100 total. 

Auto‐configuration rules  5,000 

Archived alerts  100,000 

Alert deletions per day  2,000 

Multisite members  15 

Services  50 

CIDRs defined per service  100 

 

 32 

 

Appendix‐B:  Peakflow SP PI 5500 Appliance Enforced and Guideline Limits 

The following table includes the enforced and guideline limits for the PI 5500 appliance:

Type Limit

Enforced • Non-leader PI supports all mitigation configuration except third-party

• Leader PI supports all mitigation configuration, including third-party

Guidelines • Supports software upgrades from CP 5000 but may require new hardware

• 200 automated SOAP queries per minute • 250 managed services systems (system-wide) • 50 simultaneous users per PI appliance • 120 simultaneous users (system-wide) • 100 Web 2.0 API objects

  

Appendix‐C:  Peakflow SP CP 5500‐series Appliance Enforced and Guideline Limits 

The following table includes the enforced limits for the CP 5500 series of appliances:

Type Model Limit

CP 5500-5 5 Note: Up to 3 stacked TMS 2700 appliances count as 1 appliance toward this limit.

Data sources (routers or appliances)

CP 5500-2 2 Note: Up to 3 stacked TMS 2700 appliances count as 1 appliance toward this limit.

Interfaces seen All 20,000

Interfaces monitored All 10,000

Mitigation slots CP 5500-5 2

 

 33 

CP 5500-2 1

OSPF area All 1

Simultaneous DoS alerts

All 300

The following table includes the guideline limits for the CP 5500-series appliances:

Type Limit

Input: flows per second 50,000 (supports peak rates of up to 100,000 fps) Note: Flow throughput depends on how many managed objects match as in/out per flow. The current supported flow limit assumes that no more than five managed objects match per flow.

Input: ArborFlows per second from the TMS and FS appliance

50,000 (supports peak rates of up to 50,000 fps)

Forensic flows  Up to four days Note: This is constrained by the system‐wide limit. 

Steady‐state BGP routes  2 million (300,000 per every router) 

Local managed objects per CP appliance 

500 Note: This limit is constrained by the system limit. 

Ongoing DoS alerts  300 

BGP flaps per second  1,000 

 

 

 

 

 

 34 

Appendix‐D:  Peakflow SP TMS Appliance Limits 

TMS 3100 and 3110 appliance limits

The following table includes the application monitoring and mitigation limits for the TMS 3100 and 3110 appliances:

Type Limit

Offramping or inline traffic 10 Gbps

Offramping or inline traffic 8 Mpps

Ongoing mitigations per appliance 50

Interfaces (physical, logics, sub-interface / VLAN) 1,000

TMS 3050 appliance limits

The following table includes the application monitoring and mitigation limits for the TMS 3050 appliance:

Type Limit

Offramping or inline traffic 5 Gbps

Offramping or inline traffic 3.5 Mpps

Ongoing mitigations per appliance 50

Interfaces (physical, logics, sub-interface / VLAN) 1,000

 

 35 

Appendix E:  Six‐Phase Approach to Infrastructure Security 

Cisco and Arbor advocate a six-phase framework for deploying security systems. The six phases are:

• Preparation  • Detection  • Classification  • Traceback  • Reaction • Post‐mortem  

 

While the six-phase approach was designed primarily to counter DDoS attacks, this framework provides a good overall approach to securing service provider environments.

Preparation Phase 

Preparation is probably the most important of the six phases. This phase includes setting up both technical and nontechnical processes, tools, and organizational structure that constitute the security system. The tasks in the preparation phase include:

• Select, develop, install, and test the security tools and techniques you will use.  • Define and agree upon security policy and incident response procedures.  • Set  up  communications  channels with  service  provider  peers  and  customers,  and 

establish equipment vendor incident response teams.  

Identification Phase  

In the identification phase, you detect unusual activity or behavior and activate appropriate measures after an alert is raised. You can use many tools and data sources to identify these issues, including NetFlow information, SNMP information about the CPU, and interface

 

 36 

utilization data. A customer's report that service is unavailable is often an early indicator of an attack.

Classification Phase  

After an attack has been detected, you'll need to collect comprehensive information about it, including the spoofed or non-spoofed source addresses, destination IP addresses, packet sizes, and Layer 4 information, such as protocol and port numbers. NetFlow telemetry can provide information about the attack based on the data elements tracked in each individual flow.

Traceback Phase  

Assuming you have identified the attack vector in the preceding phase, you now need to identify the ingress points in order to mitigate the attack efficiently. The traceback phase entails tracing the attack flows from the attacked sections of the network toward the network edges. You can take a hop-by-hop approach tracking the sources upstream from the victim toward network edges, or you can directly jump on the network ingress points to check them for the presence of attack flows. You can track flows through the network in various ways: through ACLs (with or without the log-input clause), by deploying NetFlow, or by using backscatter mechanisms.

Reaction Phase  

In this phase, you mitigate the attacking flows using the various mechanisms you identified as appropriate for your network during the preparation phase. These tools and techniques can include ACLs, remotely triggered source-based and destination-based black holing, rate limiting, or traffic scrubbing. For a service provider, it is important that the techniques can be deployed quickly and on a high number of ingress points. A primary concern is deploying techniques that have minimal or no negative impact on the non attacking traffic flows and on the performance of the network.

Post­Mortem Phase  

The post-mortem phase is critical. This is where you review the whole attack-handling process, analyze the experience, and look for ways to improve either organizational or technical aspects of the response. By incorporating post-mortem conclusions into a new preparation phase, you can begin to close what is often referred to as the security wheel to help ensure that the security of the service provider network remains at a high level. The security wheel illustrates that security is a cycle in which security measures are tested and improved and policies are updated so that they reflect changing security needs and drive security enhancement. Because Internet attacks are not a temporary phenomenon and they will only become more sophisticated, it is important to continually review and refine attack-handling tools and procedures.