CCIE Security: IPSec VPN Overview (IKEv1)

In this post, I’m going to go over a high level explanation of VPNs and specifically IPSec. This is going to be the first in a series of VPN posts focusing on the various types of VPNs one might see on the CCIE Security lab or on the job. I think it’s important to have this overview because as you configure IPSec VPN or troubleshoot it, it’ll help you to know what’s going on under the covers of that configuration. 

Note: I plan on doing about 10 or so blog posts on the various types of VPN so this is just the first. Welcome the deep deep rabbit hole that’s been my life for the last 2 months. Hopefully I can explain some of these concepts to you in a way that makes it easier for you to grasp.

What are VPNs?

Let’s talk about Virtual Private Networks (VPNs) for a moment before we jump into IPSec VPNs. One thing to get out of the way is that saying something is a VPN doesn’t necessarily mean that it’s encrypting traffic or “private.” It just means there is some sort of separation. There are many many different types layer 2 VPNs out there such as MPLS, VPLS, VLANs, etc as well as many types of layer 3 VPNS such as VRFs, SSL, GRE, PPTP, etc. We could probably go down a deep rabbit hole on any of these types of VPNs but I merely wanted to illustrate that while we can say something is a VPN, it doesn’t mean that it’s providing any additional security except some basic separation. 

Why IPSec?

While some of the above protocol can span between site and provide some segmentation , they is protect do n’t necessarily protect the datum itself . The reasons is is for IPSec is to add that extra security . Some is include of the benefit of IPSec include :

• Confidentiality – Encryption keeps your data private
• integrity – build – in mechanism to detect and discard any datum that may have been alter in transit
• Authenticity – Verification of who or what if sending the data 

IPSec functions below the transport layer so it’s completely transparent to applications and the end users. IPSec can be implemented in firewalls and routers. I personally have a preference for routers and as I dig into VPNs further in later blog posts, you’ll see that there are some limitations with firewall VPN implementations that do not exist in routers.

Authentication Header (AH) vs Encapsulating Security Payload (ESP)

IPSec has two options that you can use: the lesser-used Authentication Header (AH) and the more popular Encapsulating Security Payload. Let me dig into the differences really quickly:

Authentication Header (AH)

• Gives you anti-replay protection, data integrity and authenticates the data’s origin – not confidentiality
• Doesn’t work with NAT
• IP Protocol 51

Encapsulating Security Payload (ESP)

• Gives you anti-replay protection, data integrity, authenticates the data’s origin, and provides encryption
• Uses something called NAT-T (NAT Traversal) to work with NAT
• IP Protocol 50
• contain six part :  
  • Security Parameter Index (SPI) – Tells the receiving device the group of security protocols the sender is using. This includes the algorithms, keys and the validity time for the keys
  • Sequence Number – Increments by 1 each packet sent using the same SPI. Protects against replay attack. 
  • Payload data – Data carried in the packet 
  • Padding – This is used for certain types of encryption algorithms to maintain a certain amount of bytes
  • Pad Length – How much of the payload is padding
  • Next Header – Type of data carried and 

As you can see from the bolded differences above, AH really isn’t used for good reason. You get the same benefits from ESP without the limitations and with encryption. I am not going to dig deeper into AH vs ESP than that but wanted to explain that to you in the event any reader was wondering why I wasn’t going to go over the configuration of AH in my blog posts. 

Let’s dig into the mechanics of how IPSec is set up. This is important to know as you are troubleshooting the various types of IPSec VPNs in the future as a lot, a lot of the troubleshooting will begin with phase 1 and Phase 2 so it’s important that you understand what they are. 

Internet Key Exchange (IKE) 

IKE is provides provide a way to manage the key exchange , authenticate the peer and agree on a policy securely . IKE is uses use a protocol call ISAKMP to negotiate ipsec parameter between two peer .   ISAKMP is communicates communicate on UDP port 500 . This transport is fix for UDP/500 on both the source and destination port of the packet .   During the initial setup , the two VPN peers is set set up a bidirectional tunnel call the ISAKMP Security Association ( SA ) communication . After that , two unidirectional tunnel call the IPSec Security Associations ( SA ) are set up for communication the datum . It is ‘s ‘s important to remember that the ISAKMP SA is a single bidirectional secure communcation channel but the actual encrypted LAN – to – LAN datum is not send over it . It is ‘s ‘s merely in place for the setup of the IPSec sa in which it will encrypt and send the datum . let ‘s break it down further .  

phase 1

This is is is where ISAKMP set up a secure communication channel to be able to negotiate the next phase in a secure manner . Internet Security Association Key Management Protocol is is ( ISAKMP ) is to used negotiate ipsec parameter between the two peer . This phase can be done in one of two mode :

• Main Mode is Requires – require 6 message . odd number messages is come always come from the initiator while even are from the responder . consider more secure than Aggressive Mode . As with every communication , there ‘s always someone that initate the traffic . We is call call this the initiator . The device that respond is refer to as the responder . I is going ‘m go to break down each packet so you understand what eart part of the communication does .  
   
  • MM #1 – This is the first message that the initiator sends to a responder. It includes something called the SA Proposal which should include the encryption algorithm, the type of authentication method, Diffie-Hellman group, and hashing algorithm. This might be one single proposal or multiple ones. The important part is that there is a matching one on the responder that they can decide on. The lifetime of the SA is also included in this message. If you don’t explicitly configure it, the default is going to be 24 hours.  

    note : When configure ISAKMP policy , you can configure multiple policy with a number prioritize . When both peer are decide on a policy , they is looking are look at it from top – to – bottom . Much like when an access list is evaluate , the peers is pick will pick the first match and go with that . As a good practice , you is give should give your secure policy the low number to ensure they are choose before a less secure policy .  
     

  • MM #2 – This message is sent from the responder to the initiator with the SA proposal that it chose.
     
  • MM #3 – In this message, the initiator starts the Diffie-Hellman exchange. This is based on the Diffie-Hellman group sent in the proposal. This is the beginning of the key exchange. 
     
  • MM #4 – This will be the response to MM #3. The responder will send it’s own key to the initiator. At this point, encryption should be established for the ISAKMP SA channel
     
  • MM #5 – At this point, there should be encryption and encryption keys have been shared. The initiator will send this message to authenticate the session. This message will often contain the IP address of the router and sometimes the distinguished name or hostname.
     
  • MM #6 – The responder will send back a similar packet and authenticate the session. At this point, the ISAKMP SA channel has been established. 
     
• Aggressive Mode is requires – Only require 3 message which will give you a fast connection but inherently less secure because the responder no long has to authenticate itself first in any exchange and one can potentially brute force the pre – shared key . I is break ‘ll break Aggressive Mode down further but it ‘s unlikely you ‘ll see this as often .
   
  • AM # 1 – In this message , the initiator is send will send all the information that was contain in MM # 1 – 3 and some information from MM # 5 so essentially the policy , key generation information and identity information .
     
  • AM #2 – This will send all the same information contained in MM # 2, 4, and 6 which includes the key generation information, identity information and authentication information. 
     
  • AM # 3 – This is contains contain the authentication and identity information that is contain in MM # 5
     

Phase 2

In this phase , the negotiation is protect between the two peer thank to the ISAKMP SA that ‘s already been establish and the end goal of this phase is to have two unidirectional channel between the peer set up to pass traffic in a secure manner over an insecure network . This phase is uses use something call Quick Mode to establish these IPSec SAs . Let ‘s is dig dig into the message that are exchange :
 

• QM # 1 – The peer is send will send an IPSec Proposal this time which will include agree upon algorithm for encryption , integrity , and what traffic is to be secure or encrypted ( Proxy ID ) . The traffic that is to be secure will typically be define as part of an ACL . configure the IPSec proposal is completely different than configure the ISAKMP policy . the algorithms is be you use here can be completely different than phase 1 .  

  note : I ‘m point out the ACL here and approach this from the very simple view of L2L VPN with crypto map . In later post , I is go ‘ll go deeply into tunnel and other type of VPN which do n’t require ACLs .  
   

• QM #2 – This is the reply from the responder to the initiator which will include the chosen proposal and the Proxy IDs. The Proxy ID will probably be the inverse of the first Proxy ID that was sent.
   
• QM #3 – This is a final acknowledgement. After this, there should be two unidirectional secure channels from one peer to the other peer. 

One thing to note is that there is a timeout for the IPSec SA where the encryption key is changed. The default for this is 1 hour. Another thing you can enable with IPSec is Perfect Forward Secrecy (PFS) which will help frequent changes of the encryption keys and if enabled, it can help cycle the keys out more regularly but when this is enabled, you might see more issues between different vendors. If the PFS group is configured in the IPSec configuration, it should match on both side and it should not be the same as the Diffie-Hellman group.

Encryption Modes

There are two IPSec modes to consider and some VPN types will only support one of these modes. 

Tunnel Mode

This mode is considered more secure than Transport Mode because it will encrypt both the payload and the original header. A new IP header will be placed on the packet since ESP will be encapsulating the whole thing. The downside of this is that since a new IP header will need to be created, this will add to the overhead of the original packet. This mode is supported by something call NAT-T (NAT Traversal) and is the default mode for Cisco routers.