Always On VPN IKEv2 Load Balancing and NAT

Always On VPN IKEv2 Load Balancing and NATOver the last few weeks, I’ve worked with numerous organizations and individuals troubleshooting connectivity and performance issues associated with Windows 10 Always On VPN, and specifically connections using the Internet Key Exchange version 2 (IKEv2) VPN protocol. An issue that appears with some regularity is when Windows 10 clients fail to connect with error 809. In this scenario, the server will accept connections without issue for a period of time and then suddenly stop accepting requests. When this happens, existing connections continue to work without issue in most cases. Frequently this occurs with Windows Server Routing and Remote Access Service (RRAS) servers configured in a clustered array behind an External Load Balancer (ELB).

Network Address Translation

It is not uncommon to use Network Address Translation (NAT) when configuring Always On VPN. In fact, for most deployments the public IP address for the VPN server resides not on the VPN server, but on an edge firewall or load balancer connected directly to the Internet. The firewall/load balancer is then configured to translate the destination address to the private IP address assigned to the VPN server in the perimeter/DMZ or the internal network. This is known a Destination NAT (DNAT). Using this configuration, the client’s original source IP address is left intact. This configuration presents no issues for Always On VPN.

Source Address Translation

When troubleshooting these issues, the common denominator seems to be the use of Full NAT, which includes translating the source address in addition to the destination. This results in VPN client requests arriving at the VPN server as appearing not to come from the client’s original IP address, but the IP address of the network device (firewall or load balancer) that is translating the request. Full NAT may be explicitly configured by an administrator, or in the case of many load balancers, configured implicitly because the load balancer is effectively proxying the connection.

Known Issues

IKEv2 VPN connections use IPsec for encryption, and by default, Windows limits the number of IPsec Security Associations (SAs) coming from a single IP address. When a NAT device is performing destination/full NAT, the VPN server sees all inbound IKEv2 VPN requests as coming from the same IP address. When this happens, clients connecting using IKEv2 may fail to connect, most commonly when the server is under moderate to heavy load.

Resolution

The way to resolve this issue is to ensure that any load balancers or NAT devices are not translating the source address but are performing destination NAT only. The following is configuration guidance for F5, Citrix ADC (formerly NetScaler), and Kemp load balancers.

F5

On the F5 BIG-IP load balancer, navigate to the Properties > Configuration page of the IKEv2 UDP 500 virtual server and choose None from the Source Address Translation drop-down list. Repeat this step for the IKEv2 UDP 4500 virtual server.

Always On VPN IKEv2 Load Balancing and NAT

Citrix ADC

On the Citrix ADC load balancer, navigate to System > Settings > Configure Modes and check the option to Use Subnet IP.

Always On VPN IKEv2 Load Balancing and NAT

Next, navigate to Traffic Management > Load Balancing > Service Groups and select the IKEv2 UDP 500 service group. In the Settings section click edit and select Use Client IP. Repeat these steps for the IKEv2 UDP 4500 service group.

Always On VPN IKEv2 Load Balancing and NAT

Kemp

On the Kemp LoadMaster load balancer, navigate to Virtual Services > View/Modify Services and click Modify on the IKEv2 UDP 500 virtual service. Expand Standard Options and select Transparency. Repeat this step for the IKEv2 UDP 4500 virtual service.

Always On VPN IKEv2 Load Balancing and NAT

Caveat

Making the changes above may introduce routing issues in your environment. When configuring these settings, it may be necessary to configure the VPN server’s default gateway to use the load balancer to ensure proper routing. If this is not possible, consider implementing the workaround below.

Workaround

To fully resolve this issue the above changes should be made to ensure the VPN server can see the client’s original source IP address. If that’s not possible for any reason, the following registry key can be configured to increase the number of established SAs from a single IP address. Be advised this is only a partial workaround and may not fully eliminate failed IKEv2 connections. There are other settings in Windows that can prevent multiple connections from a single IP address which are not adjustable at this time.

To implement this registry change, open an elevated PowerShell command window on the RRAS server and run the following commands. Repeat these commands on all RRAS servers in the organization.

New-ItemProperty -Path ‘HKLM:SYSTEM\CurrentControlSet\Services\IKEEXT\Parameters\’ -Name IkeNumEstablishedForInitialQuery -PropertyType DWORD -Value 50000 -Force

Restart-Service IKEEXT -Force -PassThru

Additional Information

IPsec Traffic May Be Blocked When A Computer is Behind a Load Balancer

Windows 10 Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Windows 10 Always On VPN IKEv2 Load Balancing with F5 BIG-IP

Windows 10 Always On VPN IKEv2 Load Balancing with Kemp LoadMaster

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Always On VPN SSTP Load Balancing with Citrix NetScaler ADCThe Internet Key Exchange version 2 (IKEv2) VPN protocol is the protocol of choice when the highest level of security is required for Always On VPN connections. It uses IPsec and features configurable security parameters that allow administrators to adjust policies to meet their specific security requirements. IKEv2 is not without some important limitations, but organizations may insist on the use of IKEv2 to provide the greatest protection possible for remote connected clients. Due to complexities of the IKEv2 transport, special configuration on the Citrix ADC is required when load balancing this workload.

Special Note: In December 2019 a serious security vulnerability was discovered on the Citrix ADC that gives an unauthenticated attacker the ability to arbitrarily execute code on the appliance. As of this writing a fix is not available (due end of January 2020) but a temporary workaround can be found here.

Load Balancing IKEv2

When an Always On VPN client establishes a connection using IKEv2, communication begins on UDP port 500, but switches to UDP port 4500 if Network Address Translation (NAT) is detected in the communication path between the client and the server. Because UDP is connectionless, custom configuration is required to ensure that VPN clients maintain connectivity to the same backend VPN server during this transition.

Initial Configuration

Load balancing IKEv2 using the Citrix ADC is similar to other workloads. Below are specific settings and parameters required to load balance IKEv2 using the Citrix ADC.

Note: This article is not a comprehensive configuration guide for the Citrix ADC. It assumes the administrator is familiar with basic load balancing concepts and has experience configuring the Citrix ADC.

Service Settings

The load balancing services for IKEv2 VPN will use UDP ports 500 and 4500. Create the service group and assign group members for UDP 500 as follows.

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Repeat the steps above to create the service group for UDP port 4500.

Virtual Server Settings

Two virtual servers are required, one for UDP port 500 and one for UDP port 4500. Ensure that the service group using UDP port 500 is bound to the virtual server using the same port.

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Repeat the steps above to create the virtual service for UDP port 4500.

Service Monitoring

Since IKEv2 uses the UDP protocol, the only option for service monitoring is to use PING, which is configured by default. Ensure that the firewall on the VPN server allows inbound ICMPv4 and ICMPv6 Echo Request. The default PING monitor on the Citrix ADC will ping the resource every 5 seconds. If a different interval is required, the administrator can edit the PING monitor and bind that to the service or service group as necessary.

Persistency Group

A Persistency Group on the Citrix ADC will be configured to ensure that IKEv2 VPN client requests from the same client are always routed to the same backend server. Follow the steps below to create a Persistency Group and assign it to both IKEv2 virtual servers created previously.

  1. In the Citrix ADC management console expand Traffic Management > Load Balancing > Persistency Groups.
  2. Click Add.
  3. Enter a descriptive name for the Persistency Group.
  4. Select SOURCEIP from the Persistence drop-down list.
  5. Next to the Virtual Server Name section click the Add button.
  6. Add both previously configured IKEv2 virtual servers for UDP 500 and 4500.
  7. Click Create.

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Use Client IP

To ensure reliable connectivity for IKEv2 VPN connections it is necessary for the VPN server to see the client’s original source IP address. Follow the steps below to configure the Service Group to forward the client’s IP address to the VPN server.

  1. In the Citrix ADC management console expand System, click Settings, and then click Configure Modes.
  2. Select Use Subnet IP.
  3. Click Ok.Always On VPN IKEv2 Load Balancing and NAT
  4. Expand Traffic Management, click Load Balancing, and then click Service Groups.
  5. Select the IKEv2 UDP 500 Service Group.
  6. Click Edit in the Settings section.
  7. Select Use Client IP.
  8. Repeat these steps on the IKEv2 UDP 4500 Service Group.Always On VPN IKEv2 Load Balancing and NAT

Note: Making the above changes will require configuring the VPN server to use the Citrix ADC as its default gateway.

Additional Information

Windows 10 Always On VPN IKEv2 Load Balancing and NAT

Windows 10 Always On VPN SSTP Load Balancing with Citrix NetScaler ADC

Windows 10 Always On VPN IKEv2 Features and Limitations

Windows 10 AlWAYS On VPN and IKEv2 Fragmentation

Windows 10 Always On VPN IKEv2 Security Configuration

Windows 10 Always On VPN Certificate Requirements for IKEv2

Always On VPN IKEv2 Policy Mismatch Error

Always On VPN IKEv2 Policy Mismatch ErrorThe Internet Key Exchange version 2 (IKEv2) VPN protocol is the protocol of choice for Windows 10 Always On VPN deployments where the highest levels of security and assurance are required. However, as I’ve written about in the past, often the default IKEv2 security settings are less than desirable. Before using IKEv2 VPN in a production environment the administrator will need to update these security settings accordingly.

Connection Failure

When configuring Windows Server Routing and Remote Access Service (RRAS) or a third-party VPN appliance to support IKEv2 using custom security policies, the administrator may encounter a scenario in which a connection cannot be established due to a policy mismatch error. When the connection attempt fails, an error will be recorded in the Windows Application event log from the RasClient source with Event ID 20227. The error message states the following:

“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 13868.”

Always On VPN IKEv2 Policy Mismatch Error

Error Code 13868

Error code 13868 translates to ERROR_IPSEC_IKE_POLICY_MATCH. Essentially this error indicates that the IKEv2 security policy on the client did not match the configuration on the server.

Server Configuration

To view the current IKEv2 IPsec policy configuration, open an elevated PowerShell command window and run the following command.

Get-VpnServerIPsecConfiguration

Always On VPN IKEv2 Policy Mismatch Error

Client Configuration

To ensure interoperability, the VPN client must be configured to use the same IKEv2 security policy as defined on the sever. To view a VPN client’s currently configured IKEv2 security policy, open an elevated PowerShell command window and run the following command.

Get-VpnConnection -Name [connection name] | Select-Object -ExpandProperty IPsecCustomPolicy

Always On VPN IKEv2 Policy Mismatch Error

Note: If this PowerShell command returns no output, the VPN connection is not using a custom IKEv2 IPsec security policy.

Updating Settings

Guidance for configuring IKEv2 security policies on Windows Server RRAS and Windows 10 can be found here.

Summary

IKEv2 policy mismatch errors can be resolved easily by ensuring both the VPN server and client are configured to use the same IPsec security policies. Use the PowerShell commands in the above referenced above to validate settings and make changes when necessary.

Additional Information

Windows 10 Always On VPN IKEv2 Security Configuration

Windows 10 Always On VPN IKEv2 Features and Limitations

Show-VpnConnectionIPsecConfiguration PowerShell script on Github

Set-IKEv2SecurityBaseline PowerShell script on Github

Always On VPN IKEv2 Features and Limitations

Always On VPN IKEv2 Features and LimitationsThe Internet Key Exchange version 2 (IKEv2) VPN protocol is a popular choice for Windows 10 Always On VPN deployments. IKEv2 is a standards-based IPsec VPN protocol with customizable security parameters that allows administrators to provide the highest level of protection for remote clients. In addition, it provides important interoperability with a variety of VPN devices, including Microsoft Windows Server Routing and Remote Access Service (RRAS) and non-Microsoft platforms such as Cisco, Checkpoint, Palo Alto, and others.

IKEv2 Limitations

IKEv2 is clearly the protocol of choice in terms of security. It supports modern cryptography and is highly resistant to interception. It’s not without some operational challenges, however. Consider the following.

Firewalls

IKEv2 uses UDP ports 500 and 4500 for communication. Unfortunately, these ports are not always open. Often, they are blocked by network administrators to prevent users from bypassing security controls or attackers from exfiltrating data.

Fragmentation

IKEv2 packets can become quite large at times, especially when using client certificate authentication with the Protected Extensible Authentication Protocol (PEAP). This can result in fragmentation occurring at the network layer. Unfortunately, many firewalls and network devices are configured to block IP fragments by default. This can result in failed connection attempts from some locations but not others.

Load Balancing

Load balancing IKEv2 connections is not entirely straightforward. Without special configuration, load balancers can cause intermittent connectivity issues for Always On VPN connections. Guidance for configuring IKEv2 load balancing on the Kemp LoadMaster and the F5 BIG-IP can be found here:

IKEv2 Fragmentation

IKEv2 fragmentation can be enabled to avoid IP fragmentation and restore reliable connectivity. IKEv2 fragmentation is supported in Windows 10 and Windows Server beginning with v1803. Guidance for enabling IKEv2 fragmentation on Windows Server RRAS can be found here. Support for IKEv2 fragmentation on non-Microsoft firewall/VPN devices is vendor-specific. Consult with your device manufacturer for more information.

IKEv2 Security and RRAS

Be advised that the default security settings for IKEv2 on Windows Server RRAS are very poor. The minimum recommended security settings and guidelines for implementing them can be found here.

IKEv2 or TLS?

IKEv2 is recommend for deployments where the highest level of security and protection is required for remote connections. In these scenarios, the sacrifice of ubiquitous availability in favor of ultimate security might be desired.

SSTP or another TLS-based VPN protocol is recommended if reliable operation and connectivity are desired. SSTP and TLS VPNs can be configured to provide very good security by following the security and implementation guidelines found here.

IKEv2 with TLS Fallback

In theory, preferring IKEv2 and falling back to the Secure Socket Tunneling Protocol (SSTP) or another TLS-based VPN protocol when IKEv2 is unavailable would seem like a logical choice. This would ensure the highest level of protection, while still providing reliable connectivity. Unfortunately, the Windows VPN client doesn’t work this way in practice. Details here.

Additional Information

Windows 10 Always On VPN IKEv2 Load Balancing with F5 BIG-IP

Windows 10 Always On VPN IKEv2 Load Balancing with Kemp LoadMaster

Windows 10 Always On VPN IKEv2 Fragmentation

Windows 10 Always On VPN IKEv2 and SSTP Fallback

Windows 10 Always On VPN IKEv2 Security Configuration

Windows 10 Always On VPN Certificate Requirements for IKEv2

Windows 10 Always On VPN Protocol Recommendations for Windows Server RRAS

%d bloggers like this: