1. What is the difference between manual and automated testing?

Automated testing is good at finding common well-known (published) exploits. Typically 5-10 new exploits are announced each week. A manual audit can additionally find faults unique to your site, caused by application programming and design errors, as well as more complex and subtle configuration errors. A manual test can also interpret and exploit leaked information and give proactive advice on defensive design. Netcraft recommends manual testing to establish a secure baseline and then regular automated testing to maintain security.

2. Is this the same as ethical hacking?

Yes, the Netcraft eCommerce Analysis is commonly referred to as Professional hacking, Penetration testing, White hat hacking or Security Auditing. A hacker or "black hat", is usually looking for just one vulnerability to exploit. Once he succeeds in compromising a system, he has no need to find more. In contrast, Netcraft's task is to find as many vulnerabilities as possible, but in an efficient and safe manner. Our approach is going to be more 'noisy' and more intensive than a real attacker, but will not stop when a means of compromising the system is discovered. We also relay our findings to you via a comprehensive analysis report and to provide recommendations that will help you improve the security of your systems.

3. Are the automated scan reports checked for accuracy?

Yes. Checking is normally a quick process for an experienced analyst, but sometimes testing of a large network can take many hours. There are many reasons for checking results:

i) Automated security reports are notorious for generating false positives (incorrect issues) and false negatives (missed issues). If these issues are not checked, they reduce the quality of the report, undermine confidence in the results and waste precious administration time. We continually improve our tools to try and remove as many false positive issues as possible, and manually check the results before notification to improve the quality of the report.

ii) By checking issues, our experts can individually highlight changes to system administration. System administrators and IT managers like our team to highlight new issues. Our analysts get to know your network and can quickly spot things which look unusual, or highlight the impact of seemingly minor issues, which may have greater impact due to their context.

iii) Sometimes new services and new issues are discovered during testing, which need investigating by our analysts and possibly require new

advisory information. For example, if an unfamiliar service is added to your network, we will check it manually and update our tools and advisory before publishing the report.

In summary, checking results improves the quality of the report and saves our clients considerable time. Very few vulnerability scanning services offer this level of quality.

4. Couldn't I test my own network?

We do recommend that clients test their own networks - and many of our clients have very experienced security teams. However, relying on your own testing can be like marking your own examination paper. By using Netcraft, you get access to people who test many hundreds of networks each year, not just one. The tools Netcraft use and maintain are more up-to-date and comprehensive than public-domain or commercial tools. Other benefits include independence (important for audit), a genuine Internet's eye view of your network from outside of your own firewall, safety, speed, efficiency and accuracy - all based on experience. Security issues are not always black and white, and can develop very quickly from small issues into fast developing incidents. By using Netcraft, you get direct access to a professional second opinion.

5. Will you disrupt my services?

In theory this is possible, but in practice it is unlikely as we take great care to avoid damage to the services we test. We are very experienced at testing business-critical live services. In an ideal world, manual testing is done just prior to live launch, and if a server does stop responding during a manual test we will contact you immediately. Server failures of this kind nearly always indicate a serious security problem, which should be fixed. The load on the test site is typically small and will not disrupt other users. Denial of Service exploits are detected by passive methods only.

6. Can you test just a subset of my servers?

Yes - often on large networks it makes sense to test predetermined "at risk" servers or restrict in-depth testing to new services or infrastructure components. Once a security baseline has been established, incremental security assessments are often more appropriate. Of course, all services can continue to be tested using automated tools and procedures.

7. How much notice do you need?

For manual testing we normally need about 2-3 weeks notice, but there are always changes to test plans by clients, so often we can fit people in a short notice. Automated tests can be set up the same day.

8. Do you do application testing?

Yes - this is a key part of Netcraft's manual penetration test. As knowledge and understanding of general network and system security issues becomes more widespread, more systems are relatively secure "out of the box". However, as these particular security doors are closed, web-based applications are becoming prime targets for attackers. Netcraft's application testing team has been involved in the development and testing of high profile secure Internet applications since the mid-1990s. Netcraft also performs source code reviews, or onsite tests of internal networks.

9. Do I need a full retest once issues have been corrected?

Some issues take only a short time to retest after correction, as Netcraft's analysis reports contain (where possible) example exploits that allow our customers simple 'one click' testing of fixes. If independent confirmation of corrected issues is required, a common strategy is to retest only those issues from the original test report. Once we are satisfied that all the original recommendations have been followed, a follow-up report can be made available for regulatory or management purposes. Obviously the longer the time between test and retest, and the bigger the changes required, the more appropriate it is to have a full retest.

10. How quickly will I get my reports?

This depends on how many servers and services are tested and, in the case of manual testing, the complexity of the applications tested. Normally, the results of manual penetration tests are published within hours of the test finishing (once the reports have been peer reviewed internally). Automated tests will typically run overnight, and the reports are checked and published the following working day.

11. Who sees the results?

The results of a manual test are seen by the analyst, a peer reviewer and the test coordinator. Automated tests are checked by one of our analysts and seen by the test coordinator. Netcraft security staff are all full-time employees and all have been security checked. External access to HTML reports is restricted by username, password and IP address (usually your gateway or other IP addresses specified by you). Audit logs are kept of all attempts to view the report.

12. Do you find vulnerabilities in third-party software, and what do you do if you find them?

Yes, often. If the vulnerability is new and not specific to your servers, and will affect others, then we work with you and the third-party vendor to find a solution before public announcement.

13. Do you use your own tools?

Yes - it is the only way we can be sure that we are up-to-date and it ensures we fully understand all issues. We do not resell other companies products or skills. Netcraft typically runs well over a thousand distinct security scans a year, with the experience and results being continually fed back into the Netcraft tools on a daily basis. This is a very important aspect of providing a comprehensive and safe service.

14. Can you give 100% assurance that all security issues have been found?

No. By definition a testing service can only find vulnerabilities and cannot prove the absence of vulnerabilities. Independent client tests consistently show the quality of our services and tools.

That said, our reports clearly show our methods and test scope, so a person with reasonable security experience can gauge the thoroughness of the tests. Netcraft has the custom of an impressive list of clients, with several well known companies testing their networks with us for more than five years.

15. How often should I run a test?

Reasonable practice is to perform a detailed examination after major application changes, and automated testing on a weekly basis. Very sensitive networks can be tested on a daily basis, and we also offter montly testing as a cheaper option.

16. Can I run the test at a specific time?

Normally manual testing is done during UK office hours. The time that a scheduled automated test runs at each week will vary slightly from week to week as new customers are added and the number of machines responding from other clients networks will fluctuate and correspondingly affect the time other tests start.

17. I have 4 servers behind one external load balanced address. How do you test them and quote for them?

If we can only see one IP address externally, we treat that as one machine. If you want us to test all four, either make them individually visible to us on separate IP addresses, or arrange for us to perform an test on-site. Manual testing of load-balanced services will provide a more complete assessment of any inconsistencies within the load-balanced pool, for example, where some, but not all machines have been patched against a particular vulnerability.

18. Do you do internal testing?

Yes, we use a similar methodology to external testing, but the definition of what you are trying to protect (and who you are protecting it from) needs

more careful consideration. Onsite testing of a DMZ is also useful, as it allows us to test the effects of "outer shell" compromise, and comment on the internal security layers in more detail.

19. What happens if I increase the size of my network during the year and exceed the scope of my automated test subscription?

We will contact you and agree an incremental cost for testing the larger network.