Special Publication 800-77
Guide to IPsec VPNs
Recommendations of the National Institute of Standards and Technology
Sheila Frankel Karen Kent Ryan Lewkowski Angela D. Orebaugh Ronald W. Ritchey Steven R. Sharma
NIST Special Publication 800-77
C O M P U T
Guide to IPsec VPNs Recommendations of the National Institute of Standards and Technology Sheila Frankel Karen Kent Ryan Lewkowski Angela D. Orebaugh Ronald W. Ritchey Steven R. Sharma
E R S E C U R I T Y
Computer Security Division Information Technology Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899-8930 December 2005
U.S. Department of Commerce
Carlos M. Gutierrez, Secretary
Technology Administration
Michelle O'Neill, Acting Under Secretary of Commerce for Technology
National Institute of Standards and Technology
William A. Jeffrey, Director
GUIDE TO IPSEC VPNS
Reports on Computer Systems Technology
The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the nations measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analysis to advance the development and productive use of information technology. ITLs responsibilities include the development of technical, physical, administrative, and management standards and guidelines for the cost-effective security and privacy of sensitive unclassified information in Federal computer systems. This Special Publication 800-series reports on ITLs research, guidance, and outreach efforts in computer security and its collaborative activities with industry, government, and academic organizations.
Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately.
Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose.
National Institute of Standards and Technology Special Publication 800-77 Natl. Inst. Stand. Technol. Spec. Publ. 800-77, 126 pages (December 2005)
ii
GUIDE TO IPSEC VPNS
Acknowledgements
The authors, Sheila Frankel of the National Institute of Standards and Technology (NIST), and Karen Kent, Ryan Lewkowski, Angela D. Orebaugh, Ronald W. Ritchey, and Steven R. Sharma of Booz Allen Hamilton, wish to thank their colleagues who reviewed drafts of this document, including Bill Burr, Tim Grance, Okhee Kim, Peter Mell, and Murugiah Souppaya from NIST. The authors would also like to express their thanks to Darren Hartman and Mark Zimmerman of ICSA Labs; Paul Hoffman of the VPN Consortium; and representatives from the Department of Energy, the Department of State, the Environmental Protection Agency, and the U.S. Nuclear Regulatory Commission for their particularly valuable comments and suggestions.
Trademark Information
Microsoft, Windows, Windows 2000, and Windows XP are either registered trademarks or trademarks of Microsoft Corporation in the United States and other countries.
PGP is a trademark or registered trademark of PGP Corporation in the United States and other countries.
Cisco and Cisco IOS are registered trademarks of Cisco Systems, Inc. in the United States and certain other countries.
Lucent Technologies is a trademark or service mark of Lucent Technologies Inc.
All other names are registered trademarks or trademarks of their respective companies.
iii
GUIDE TO IPSEC VPNS
Table of Contents
Executive Summary............................................................................................................ES-1
1. Introduction ................................................................................................................... 1-1 1.1 Authority................................................................................................................ 1-1 1.2 Purpose and Scope............................................................................................... 1-1 1.3 Audience ............................................................................................................... 1-1 1.4 Document Structure .............................................................................................. 1-1
2. Network Layer Security................................................................................................. 2-1 2.1 The Need for Network Layer Security.................................................................... 2-1 2.2 Virtual Private Networking (VPN)........................................................................... 2-4
2.2.1 Gateway-to-Gateway Architecture.............................................................. 2-5 2.2.2 Host-to-Gateway Architecture .................................................................... 2-6 2.2.3 Host-to-Host Architecture ........................................................................... 2-7 2.2.4 Model Comparison..................................................................................... 2-8
2.3 Summary............................................................................................................... 2-8 3. IPsec Fundamentals...................................................................................................... 3-1
3.1 Authentication Header (AH)................................................................................... 3-1 3.1.1 AH Modes .................................................................................................. 3-1 3.1.2 Integrity Protection Process........................................................................ 3-2 3.1.3 AH Header ................................................................................................. 3-2 3.1.4 How AH Works........................................................................................... 3-3 3.1.5 AH Version 3.............................................................................................. 3-4 3.1.6 AH Summary.............................................................................................. 3-5
3.2 Encapsulating Security Payload (ESP).................................................................. 3-5 3.2.1 ESP Modes................................................................................................ 3-5 3.2.2 Encryption Process .................................................................................... 3-6 3.2.3 ESP Packet Fields ..................................................................................... 3-7 3.2.4 How ESP Works......................................................................................... 3-8 3.2.5 ESP Version 3............................................................................................ 3-9 3.2.6 ESP Summary............................................................................................ 3-9
3.3 Internet Key Exchange (IKE) ............................................................................... 3-10 3.3.1 Phase One Exchange .............................................................................. 3-10 3.3.2 Phase Two Exchange .............................................................................. 3-15 3.3.3 Informational Exchange............................................................................ 3-17 3.3.4 Group Exchange ...................................................................................... 3-17 3.3.5 IKE Version 2 ........................................................................................... 3-18 3.3.6 IKE Summary........................................................................................... 3-18
3.4 IP Payload Compression Protocol (IPComp) ....................................................... 3-19 3.5 Putting It All Together.......................................................................................... 3-20
3.5.1 ESP in a Gateway-to-Gateway Architecture ............................................. 3-20 3.5.2 ESP and IPComp in a Host-to-Gateway Architecture ............................... 3-21 3.5.3 ESP and AH in a Host-to-Host Architecture.............................................. 3-22
3.6 Summary............................................................................................................. 3-23 4. IPsec Planning and Implementation ............................................................................ 4-1
4.1 Identify Needs ....................................................................................................... 4-1
iv
GUIDE TO IPSEC VPNS
4.2 Design the Solution ............................................................................................... 4-2 4.2.1 Architecture................................................................................................ 4-3 4.2.2 Authentication ............................................................................................ 4-8 4.2.3 Cryptography ........................................................................................... 4-10 4.2.4 Packet Filter ............................................................................................. 4-10 4.2.5 Other Design Considerations ................................................................... 4-11 4.2.6 Summary of Design Decisions ......................
