Implementation Best Practices by David French – Technical Account Manager, Qualys Inc.

Deploying Vulnerability Management and Policy Compliance on a Global ScaleON TIME – ON BUDGET – ON DEMAND

December 2007

Asset groups

Used for three main purposes: To Grant Access

– Typically based on OS and/or Application To Schedule Scans

– Typically based on Network Segments To Facilitate Reporting

– Typically based on combinations of the above that comprise Business Units, Applications, Internal vs. Externally facing systems, etc.

For remote sites, the Asset Groups used to facilitate scanning (by segment) are often used to grant access to the local IT resources as well.

Keep Risk score (Business Risk and CVSS) in mind when configuring Asset Groups– If a system is in more than one Asset Group, the risk score calculated for that particular host will be

an average of the settings for each Asset Group it is in.

Eliminating “Scanning” Asset Groups in Large Environments

In larger environments, it may be necessary to schedule scans (and maps) by blocks of IP addresses instead of scheduling using Asset Groups.

This can help to keep the overall number of Asset Groups in the subscription lower.

Business Risk

Starting point for Microsoft centric environments:– Internet Facing Systems/DMZ’s – Critical– Datacenter Server Segments – High– Workstation Segments – Medium– Can any segments be low?

Factors for determining what Level to assign for a given Asset Group:– Database servers vs. Front-end Web servers– Server segments vs. Workstation segments– Internet Facing systems vs. Internal systems– Operating System types and versions contained in the Asset Group

Business Units

Used when you want to delegate authority. Business Units are not meant to be used solely as containers for Asset Groups– Be careful about delegating authority for remediation.– Be careful about delegating authority for purging.– Be careful about delegating the ability to create Option Profiles– Remember the golden rule of delegation: It is easier to grant

additional rights than to revoke rights that have been already granted.

The decision to utilize Business Units will depend on whether or not your QualysGuard implementation will be centralized or de-centralized.

Mapping

The ability to “diff” map reports or to compare regularly scheduled maps against an approved host list (using the Unknown Device Report) makes it possible to quickly identify change/rogue systems on DMZ and critical server segments.– Most customers start doing this as they become aware of this

capability.– Comparing daily maps for DMZ and critical server segments can

easily be added to daily Security Operations Center (SOC) Analyst procedures as this only takes a few clicks of the mouse.

Unknown Device Report Use

Create a “fake” domain to server as a container for the Internet facing systems or datacenter systems you want to monitor for changes. Be sure to do a whois to make sure it is not a real Domain.

Example: acmeallexternal.net

Run an initial map and examine the results. Via the action bar at the top left, approve all hosts once you have verified their legitamacy.

Run a map daily against acmeallexternal.net

All you have to do to check for rogue devices is to run the Unknown Device Report and select the domain (in this case acmeallexternal.net) and the data set for the previous days map run.

Mapping Strategies

One method is to map geographically. Make a domain for each scanner throughout the enterprise (scanner1.com, scanner2.com, etc.).– This method allows you to map all areas of a global enterprise during the

day so the maximum number of devices will be discovered. (Remember, maps only use 13 TCP ports, and a few random UDP ports for traceroute purposes, by default.)

– Always perform a Whois lookup when you create “fake” Domains to be used for mapping.

Another method is to use the None Domain to enable mapping via the same Asset Groups used for scanning.

Sometimes a combination of the above methods is used. Geographically based mapping for the bulk of an enterprises IP addresses and by Asset Group for DMZs and Critical Server Segments.

Scanning

Scans are typically run by segment, example: 192.168.1.1-192.168.1.254. This makes it so you catch new/rogue devices as they are placed onto the network. Catching these systems via scanning as opposed to mapping makes it so tickets are generated for new systems in a timely fashion.

– In scenarios where only servers have been licensed, scans are run against specific IPs for all segments but datacenter segments. All other segments throughout the enterprise are mapped regularly to discover new servers.

Most customers run full scans to take advantage of QualysGuard’s inference based scanning engine. This also allows making the change from “point” scanning to continuous scanning of your environment. Refresh the data regularly and mine it via Template Based Reporting and the Asset Search Portal.

Most customers are starting to perform their Windows trusted scans using an Administrative account.– As the Windows operating system matures, not all policy/configuration related

checks can be made with a user account.

Trusted vs. Un-Trusted Scanning - Windows Trusted scanning reduces the amount of potential vulnerabilities reported

due to the ability to verify versions, list packages, read the registry etc.

Trusted scanning allows QualysGuard to provide governance for configuration management due to the additional information that can be gathered.

As the Microsoft operating system becomes more secure, some settings (Example: auditing settings) require Administrative access to be enumerated. For this reason, more customers are scanning Windows systems with administrative accounts to facilitate providing governance for configuration management.– The QualysGuard administrator can have a systems administrator type in the

password for an administrative scanning account i.e. the person administering QualysGuard does NOT have to know the password for the account used for scanning.

Metrics on the amount of data gathered from a Windows 2003 system

Comments on previous Slide’s Numbers

Un-trusted scans can only verify vulnerabilities that are exploitable over the wire.

Users with RX to the winreg key can only verify Microsoft Operating System patches.

Backup Operators can see other applications. In this case Cold Fusion is accounting for the deltas in confirmed and potential vulnerabilities.

Administrators can see all registry keys for Cold Fusion and other 3rd party applications QualysGuard has detections/checks for.

In the information gathered section, the large increase in the number of items detected is due to security log access, audit settings, and security settings in the user rights section. (You can view these settings when you run secpol.msc.)

I suspect that by the time you gave an account the rights to the entire file system, assigned all user rights, granted access to keys beyond the winreg key, and allowed access to the security logs, you would essentially have an administrator.

Risk Analysis for using a Domain Admin account for Scanning (1)

Credentials are stored in an encrypted database.

The team responsible for Domain Admin accounts can enter the password into the QualysGuard GUI for the QualysGuard administrators. The QualysGuard administrators do not need to know the password.

QualysGuard submits the password over the wire the same as an administrator’s workstation using an MMC console.

One issue with any credentials used for scanning is that the credentials may be obtained via a sniffer on a host that is being scanned or a sniffer on a mirror port. This is also the case for "normal" administrator activity throughout the network. The answer for mitigating this is to change the password right after scans have run. This can be accomplished using custom scripts, the following free utility: http://jdhitsolutions.com/pwdman/index.htm, or a commercial product like the one found at: http://www.keroonsoftware.com/go/resetpro.htm

Also, if someone sniffs a Domain Admin account the first thing a serious attacker would do is make an account that was NOT in the Domain Admins group but was in the local administrators group on some target systems. It is easy to use log parser and/or homegrown scripts to monitor additions to the Domain Admins group. It is hard to scale this out to hundreds or thousands of servers without very expensive software like TNT.

Risk Analysis for using a Domain Admin account for Scanning (2)

Managing Domain Controllers for scanning or any other maintenance/auditing function without using an account that is a Domain Administrator it not feasible. Since Domain Controllers are typically considered a critical system, auditing their configurations and patches should be a priority that would warrant the use of Domain Admin credentials. As we move into the world of read only Domain Controllers this will change.

Microsoft is fixing their Operating System security more with each new version. That is why a user with RX to the Winreg key can not enumerate auditing setting remotely. Try using auditpol.exe remotely as anything but an administrator. If you were to author an OS, why would you make it so someone who was NOT root could remotely enumerate audit settings etc.?

There is always the possibility that an application with an admin service account may inadvertently damage the environment via a malfunction. Risk analyses should deal with probabilities tied to possibilities...not just possibilities. The nature of what our product does make it highly unlikely we will use any provided administrative authority to damage systems via configuration changes etc. as our engine has no such functionality.

Scanning Frequencies

Most commonly observed frequencies:

Server/Datacenter Segments Weekly

Internet Facing Systems: Monthly at a minimum More customers are scanning their Internet presence's weekly Some customers scan daily

Workstations: Monthly, Twice a Month, or Weekly - depending on the size of the

environment

Scanning – Time of Day

Servers within datacenters are typically scanned off-hours.– If scans are performed during Administrator’s maintenance

windows, systems may be missed due to reboots from patching etc.

Workstation segments are typically scanned during the day so most systems are active.

Remote sites, even though they typically have Domain Controllers and File/Print servers, are typically scanned during the day in order to catch all the workstations.

Scanning Server/Datacenter Segments

You may want to split server segments into Class C (/24) networks for scanning.– If there’s an issue while scanning a host and you have to

pause a scan, most of your servers will still be scanned.– In Incident Response scenarios, it is helpful to have Asset

Groups pre-defined for each server segment for emergency scans and maps

Options Profiles – How Many?

Typically customers will start with the following: Profile for Internet Scans

– Brute forcing typically set to limited– Mapping options set to all hosts and NOT to ignore hosts only discovered via

DNS Profile for Internal Scans

– Brute forcing typically minimal or disabled– Mapping options set to netblock only and to ignore hosts only discovered via

DNS Profile for Internal Scans with Authentication (default)

– Brute forcing typically minimal or disabled– Mapping options set to netblock only and to ignore hosts only discovered via

DNS Profile for Scanning across WAN segments (if needed)

– Same options as your default Internal scan profile with the scan performance settings modified

It is also recommended that customers create a “light scan profile” detailed in the next slide

Light Scan/Inventory Scan Profile

TCP - none - additional

– 21-23, 25, 53, 80, 88, 100-111, 135, 139, 443, 445, 515, 1433

UDP - none - additional

– 53, 111, 135, 137, 161, 500

Custom Vulnerabilities

– Add the following QIDs at a minimum:

QID 6 DNS Host nameQID 45039 Host Names FoundQID 82044 NetBIOS Host NameQID 82062 NetBIOS Workgroup Name Detected (is it offering

connectivity)QID 82023 Open TCP Services ListQID 82004 Open UDP Services ListQID 45017 Operating System DetectedQID 12230 Default Web Page (good for identifying unknown

Wireless Access Points)QID 82040 ICMP Replies Received (good to know if its simply alive)QID 43007 Network Adapter MAC Address QID 105194 Wireless Access Point DetectedQID 78032 Wireless Access Point InformationQID 34011 Firewall DetectedQID 45038 Host Scan Time

Optional QIDs that are good to add:

QID 70004 NetBIOS Bindings Information

Add the following if you are performing trusted scanning:

QID 105015 Windows Authentication FailedQID 105192 SNMP Authentication FailedQID 105053 Unix Authentication FailedQID 105193 Oracle Authentication Failed

Advanced Tab

– Ignore RST– Ignore SYN ACK

Light/Inventory Scan vs. Mapping

In environments where firewalls and other network/host level filtering exists that you may not be aware of, using the Light/Inventor scan can increase the accuracy of your initial inventory because the scanning engine will do more than TCP fingerprinting.

Sample Statistics : Mapping vs. Scanning for OS Identification

Total Devices # of Unknown % Unknown # behind firewalls identified

Inventory Map 780 183 23% unknown

Inventory Scan 708 42 5.4% 57

Scanning Internet Facing Systems

Many customers are performing monthly PCI scans, even if they do not have to comply to the Payment Card Industry standards. Monthly PCI scanning provides a means to show

management that your company is compliant to a rigorous and well known industry security standard.

If you are subject to the PCI standard, waiting until the end of each Quarter to run a scan can cause you to have a very busy week.

Discovering Rogue Networks

Although mapping large network address blocks can find new network segments, it is still best to obtain a copy of the configurations of a WAN router and your core switches to make sure all network segments and VLANs are accounted for within QualysGuard.

It is still advisable to periodically review copies of the configurations of your core (Layer 3) switches and one of your WAN routers to look for new VLAN/Network definitions. “Routing Tables Don’t Lie ”

Broadcast and Network Addresses

QualysGuard will not scan the network address. Example: 192.168.1.0

QualysGuard will quickly (based on numerous replies) determine when it is processing a broadcast address and stop scanning it.

Reporting

Types of reporting Trend Reporting for Executives and Managers Compliance Reporting Server Certification Reports Authorized/Unauthorized Services

Trend Reporting for Executives and Managers Executive reporting is typically not implemented until all segments

have been scanned a few times. The trend in an initial deployment is typically increasing as new segments are added.

The canned Executive Report template is a good place to start.

Executive reports typically contain graphics only and should be downloaded as a PDF.

Most customers automate the generation of Monthly Executive Reports via the API.– When you create a global reader account for generating Executive

reports, change the account’s color scheme to Olive Green. This will make the graphics more vivid.

Compliance Reporting

Maintaining copies of trend and remediation reports will assist in your compliance efforts with regulations like SOX etc. Having historical reports on hand demonstrates the fact that your processes are in place and being followed.

In addition to the above, Qualys will be releasing a policy compliance module to facilitate reporting on systems that do not comply with your organization’s specific host configuration/hardening standards.

Server Certification Reports

Scan reports for new systems are generally stored electronically, in hard copy, or both, depending on how you maintain build folders.

Over time, we are observing less of this type of reporting as most customers scan by segment so they catch systems that are placed onto the network with missing patches or incorrect builds and tickets/reports are immediately generated.

Reporting on Authorized/Unauthorized Services Using the advanced tab while creating a report template,

it is possible to create a report, sorted by vulnerability, that will identify system that are missing required services or that have unauthorized services running.

Running reports of this nature on a regular basis can be easily added to customer’s existing SOC Analyst duties.

To show only Authorized/Unauthorized services, reports can be edited with Nvu Portable.

– (http://portableapps.com/apps/development/nvu_portable)

Report Distribution

Below is a sample of a url that can be used to send an e-mail with a link that, when clicked on, will prompt the recipient for their QualysGuard login and then run the specified report. This can only be used with Automatic Report Templates.

https://qualysguard.qualys.com/fo/report/report_view.php?run=284183&authfirst=true

Alternatively, you could use the URL in conjunction with an Internet Explorer Icon and publish it to the appropriate individuals desktops via Group Policy (If in an Active Directory environment.)

Report Archival

Many customers create a file share that they use to store copies of all regular reports that are run. (Executive Reports, Weekly Tickets per User Reports, etc.)

Reports are typically downloaded and stored in PDF format as they make for a more indelible report.

When auditors are on-site, they can be granted read access to the share so they can verify the metrics etc. you use to track vulnerabilities and the remediation of vulnerabilities in the environment.

Purging effects on Metrics

Purging information from hosts that have not been scanned in X amount of time (done via the Asset Search Portal – X is determined by your scan frequencies) increases accuracy of the metrics for executive and trend reporting.

Issues to keep in mind:

When you purge the data for a host, historical ticketing information is lost. If you have data retention requirements for ticket information, you can automate the downloading of all ticketing information via the API prior to purging.

Purging will also alter you total “fixed” vulnerability count. Keeping historical ticket and executive reports will allow you to have more accurate long term trending numbers.

Data retention settings are on a per user basis for accounts with scanner authority and greater. Ensure all personnel with scanner level authority and above modify their settings as the default is to purge scan results after six months.

Asset Search Portal (ASP)

Since results from ASP searches can be downloaded to a CSV file, the Asset Search Portal can be used to quickly generate actionable lists.

– Example: If you are concerned about rogue ftp servers in your environment, you can run a search using the ftp service category and find all systems running ftp, regardless of the port. (Demo)

Results also have workflow buttons at the bottom. This means that query results can easily be used to refresh the memberships of Asset Groups.

– Example: You could query for all systems that have QID 45022 to obtain a list of all Active Directory Domain controllers. You could then select all the systems and add them to an asset group for Domain Controllers.

Many customers have standard queries they run monthly for the above reasons.

Remediation (Ticketing System)

Using remediation in lieu of sending reports to administrators has the following benefits:

Makes QualysGuard more visible i.e. more people use the application.

Tracking and reporting on remediation is ideal for SOX (and other regulation) compliance. It proves that your Vulnerability Management process is resulting in action.

Remediation is “actionable reporting”. Instead of just raising awareness of an issue, the issue is assigned to someone so it will be resolved. Tickets leverage the QualysGuard Knowledgebase so details on how to fix the vulnerability are provided within the tickets.

Some customers report metrics strictly on remediation tickets. If a remediation policy (rule) does not address a given vulnerability, then the risk associated with the vulnerability has been accepted.

Remediation Policies (Rules) (1)

Remediation policies are processed from Top to Bottom. The first rule to match you ticket generation criteria wins. (Tickets are sent to the user specified in the first rule to match.)

To facilitate administration, you may want to create a convention for rule titles that make it easy to tell non-create rules from ticket creation rules etc.

The above point means that your specific policies should be at the top of the list.– The first rule is generally an ignore rule for cluster IP addresses and

INTERNAL VIPs.– The last rule is generally an “All” rule that assigns tickets to a

QualysGuard Administrator. This rule will make it so you identify new systems and/or systems that were not covered in the existing list of policies.

Remediation Policies (Rules) (2)

Other rules that have been used in the field:

Zero Day Rule – QualysGuard administrators can scan the daily e-mail for new detections for Zero Day vulnerabilities. Since a true Zero Day has no fix, you can ad them to a Zero Day rule (near the top of the rule set) so systems administrators do not receive tickets for items there is no patch or workaround for.

No ticket for X days rule – In environments where systems administrators have x time to patch systems, you may want to create a rule (for Microsoft, one rule each month) that specifies not to create tickets for a set of QIDs. When the grace period is up, delete the rule. This way administrators have tickets for items they are currently accountable for.

NOTE: If you use these types of rules you can not use ticketing metrics for risk reporting, you will need to use template based reporting to show risk as even thought the grace period is in effect the risk is still present in your environment.

A rule at the top of the rule set that states NOT to create tickets for levels of vulnerabilities you do not want to ticket on.

Example: If you only ticket on Level 4 and Level 5 Confirmed Vulnerabilities, create a rule at the top of your policy set that specifies not to create tickets for Level 1 -3 Confirmed Vulnerabilities and Level 1 – 5 Potential Vulnerabilities.

Remediation – Where do I start?

Most organizations try to start by ticketing Critical, Level 5 confirmed vulnerabilities (Red 5’s).– This is not always practical for large enterprises. If necessary,

you may start by ticketing a set of specific vulnerabilities for the first six to twelve months of your QualysGuard deployment.

A good way to determine where to start is to put test remediation policies in place prior to performing baseline scans. This allows you to see how many tickets would be generated for administrators throughout your enterprise.

Remediation Integration with Other Ticketing Systems Even if you plan to integrate QualysGuard’s ticketing with another

system (Ex. Remedy) at a later time, you will still need to create the remediation policies (rules) within QualysGuard.– For this reason, many Enterprise Customers start out using

QualysGuard’s built-in remediation (ticketing) system.

Ticket metrics are still typically gathered from QualysGuard post-integration as QualysGuard tracks closed tickets based on when ticketed vulnerabilities are no longer discovered by a scan.

Working towards using Remediation

If you begin your QualysGuard implementation using only template based reporting, your report templates will be used to define your Remediation (ticketing) rules.

A possible order for implementing remediation: Start with implementing reporting Use reporting to define remediation rules Integrate QualysGuard’s built in Remediation (ticketing)

system with your enterprise-wide ticketing solution.

Showing Value

Executive Reporting – Having executives receive Vulnerability trend reports on a monthly basis serves as a reminder of the fact that QualysGuard is being used.

Having users log in either for remediation, reporting, or to use the Asset Search Portal on a regular basis makes QualysGuard more visible and can make it become a part of Administrators and Application Owners operational tool set. – Administrators can track changes to their systems that may have

slipped through change control.– In scenarios where systems reside in an outsourced environment,

Administrators can use QualysGuard to discover unauthorized changes on their segments.

QualysGuard’s Value Beyond Vulnerability ManagementIn addition to demonstrating your organization's discovery and remediation of vulnerabilities over time, QualysGuard provides the following:

Discovery of Rogue devices/Governance for Change Management

Governance for Patch Management Governance for Configuration Management Verification of agent based technology deployments Compliance Reporting for all the above

Security/Compliance Frameworks

In the help…resources… section of QualysGuard is a PDF titled “CobIT and Sarbanes-Oxley (SOX) Compliances with QualysGuard “. This document lists CobIT 4.0 controls that QualysGuard can be used for.

For companies implementing ISO over NIST QualysGuard is directly relevant to the following NIST 800-53 sections: RA-5 : Vulnerability Scanning CM-4 : Asset Creation

QualysGuard Care and Feeding

Monthly purging AFTER Executive Reports have been run.

Monthly queries looking for IP tracked Workstations (to keep your NetBIOS tracking accurate).

Monthly queries to verify agent based deployments (AV, SMS, etc.)

Monthly reports on Ignored Vulnerabilities

Monthly queries used to ensure OS and Application based Asset Groups are up to date

Monthly queries to verify authentication records are correct