Always On VPN IKEv2 Load Balancing and NAT

Always On VPN IKEv2 Load Balancing and NATOver the last few weeks, I’ve worked with numerous organizations and individuals troubleshooting connectivity and performance issues associated with Windows 10 Always On VPN, and specifically connections using the Internet Key Exchange version 2 (IKEv2) VPN protocol. An issue that appears with some regularity is when Windows 10 clients fail to connect with error 809. In this scenario, the server will accept connections without issue for a period of time and then suddenly stop accepting requests. When this happens, existing connections continue to work without issue in most cases. Frequently this occurs with Windows Server Routing and Remote Access Service (RRAS) servers configured in a clustered array behind an External Load Balancer (ELB).

Network Address Translation

It is not uncommon to use Network Address Translation (NAT) when configuring Always On VPN. In fact, for most deployments the public IP address for the VPN server resides not on the VPN server, but on an edge firewall or load balancer connected directly to the Internet. The firewall/load balancer is then configured to translate the destination address to the private IP address assigned to the VPN server in the perimeter/DMZ or the internal network. This is known a Destination NAT (DNAT). Using this configuration, the client’s original source IP address is left intact. This configuration presents no issues for Always On VPN.

Source Address Translation

When troubleshooting these issues, the common denominator seems to be the use of Full NAT, which includes translating the source address in addition to the destination. This results in VPN client requests arriving at the VPN server as appearing not to come from the client’s original IP address, but the IP address of the network device (firewall or load balancer) that is translating the request. Full NAT may be explicitly configured by an administrator, or in the case of many load balancers, configured implicitly because the load balancer is effectively proxying the connection.

Known Issues

IKEv2 VPN connections use IPsec for encryption, and by default, Windows limits the number of IPsec Security Associations (SAs) coming from a single IP address. When a NAT device is performing destination/full NAT, the VPN server sees all inbound IKEv2 VPN requests as coming from the same IP address. When this happens, clients connecting using IKEv2 may fail to connect, most commonly when the server is under moderate to heavy load.

Resolution

The way to resolve this issue is to ensure that any load balancers or NAT devices are not translating the source address but are performing destination NAT only. The following is configuration guidance for F5, Citrix ADC (formerly NetScaler), and Kemp load balancers.

F5

On the F5 BIG-IP load balancer, navigate to the Properties > Configuration page of the IKEv2 UDP 500 virtual server and choose None from the Source Address Translation drop-down list. Repeat this step for the IKEv2 UDP 4500 virtual server.

Always On VPN IKEv2 Load Balancing and NAT

Citrix ADC

On the Citrix ADC load balancer, navigate to System > Settings > Configure Modes and check the option to Use Subnet IP.

Always On VPN IKEv2 Load Balancing and NAT

Next, navigate to Traffic Management > Load Balancing > Service Groups and select the IKEv2 UDP 500 service group. In the Settings section click edit and select Use Client IP. Repeat these steps for the IKEv2 UDP 4500 service group.

Always On VPN IKEv2 Load Balancing and NAT

Kemp

On the Kemp LoadMaster load balancer, navigate to Virtual Services > View/Modify Services and click Modify on the IKEv2 UDP 500 virtual service. Expand Standard Options and select Transparency. Repeat this step for the IKEv2 UDP 4500 virtual service.

Always On VPN IKEv2 Load Balancing and NAT

Caveat

Making the changes above may introduce routing issues in your environment. When configuring these settings, it may be necessary to configure the VPN server’s default gateway to use the load balancer to ensure proper routing. If this is not possible, consider implementing the workaround below.

Workaround

To fully resolve this issue the above changes should be made to ensure the VPN server can see the client’s original source IP address. If that’s not possible for any reason, the following registry key can be configured to increase the number of established SAs from a single IP address. Be advised this is only a partial workaround and may not fully eliminate failed IKEv2 connections. There are other settings in Windows that can prevent multiple connections from a single IP address which are not adjustable at this time.

To implement this registry change, open an elevated PowerShell command window on the RRAS server and run the following commands. Repeat these commands on all RRAS servers in the organization.

New-ItemProperty -Path ‘HKLM:SYSTEM\CurrentControlSet\Services\IKEEXT\Parameters\’ -Name IkeNumEstablishedForInitialQuery -PropertyType DWORD -Value 50000 -Force

Restart-Service IKEEXT -Force -PassThru

Additional Information

IPsec Traffic May Be Blocked When A Computer is Behind a Load Balancer

Windows 10 Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Windows 10 Always On VPN IKEv2 Load Balancing with F5 BIG-IP

Windows 10 Always On VPN IKEv2 Load Balancing with Kemp LoadMaster

Always On VPN SSTP Connects then Disconnects

Always On VPN SSTP Connects then DisconnectsWhen Always On VPN clients are configured to use the Secure Socket Tunneling Protocol (SSTP) with Windows Server Routing and Remote Access Service (RRAS), administrators may encounter a scenario in which a client can establish a VPN connection using SSTP successfully, but is then disconnected immediately. The system event log contains an entry with Event ID 6 from the RasSstp source that includes the following error message.

“The SSTP-based VPN connection to the remote access server was terminated because of a security check failure. Security settings on the remote access server do not match settings on this computer. Contact the system administrator of the remote access server and relay the following information.”

Always On VPN Connect and Disconnect with SSTP

Common Causes

The two most common causes of this issue are when SSTP is configured for SSL offload, and when a VPN client is on a network where SSL inspection is taking place.

SSTP Offload

The most common cause of this issue is when SSL offload is configured for SSTP on an external load balancer or application delivery controller (ADC). To prevent interception from a Man-in-the-Middle attack, the VPN client sends the certificate hash of the SSL certificate used when the VPN connection was established. If this information does not match what is configured on the RRAS server, the connection is assumed to be compromised and the connection is immediately dropped.

SSL Inspection

Another scenario where this issue may occur is when a VPN client is behind a network device configured to perform SSL deep-packet inspection (DPI). SSTP VPN clients will be unable to connect to the VPN server in this scenario.

Resolution

When offloading SSL to another device, the RRAS server must be configured to know which SSL certificate is being presented to remote clients. This information is stored in the following registry key.

HKLM:\SYSTEM\CurrentControlSet\Services\SstpSvc\Parameters\SHA256CertificateHash

However, this registry entry requires a binary value, which makes it a challenge to configure manually. To resolve this problem, it is recommended that the same SSL certificate installed on the load balancer/ADC also be installed on the VPN server (even though SSL will be offloaded). To do this, first import the SSL certificate and private key in to the Local Computer certificate store, then open the RRAS management console and perform the following steps.

  1. Right-click the VPN server and choose Properties.
  2. Select the Security tab.
  3. Uncheck Use HTTP in the SSL Certificate Binding section.
  4. Select the appropriate SSL certificate from the Certificate drop-down list (click View to verify).
  5. Click Apply.

This will add the correct SSL certificate information to the registry. Next, re-enable HTTP for SSL offload by performing the following steps.

  1. Check Use HTTP in the SSL Certificate Binding section.
  2. Click Apply.

PowerShell Configuration

If the SSL certificate cannot be installed on the VPN server, or to automate this configuration across multiple servers remotely, download and run the Enable-SstpOffload PowerShell script from my GitHub repository here and run the following command.

Enable-SSTPOffload -CertificateHash [SHA256 Certificate Hash of Public SSL Certificate] -Restart

For example…

Enable-SSTPOffload -CertificateHash “C3AB8FF13720E8AD9047DD39466B3C8974E592C2FA383D4A3960714CAEF0C4F2” -Restart

Additional Information

Windows 10 Always On VPN Load Balancing and SSL Offload

Windows 10 Always On VPN SSTP Load Balancing with F5 BIG-IP

Windows 10 Always On VPN SSL Certificate Requirements for SSTP

Windows 10 Always On VPN Protocol Recommendations for Windows Server RRAS

 

DirectAccess Selective Tunneling

DirectAccess Selective TunnelingDirectAccess administrators, and network administrators in general, are likely familiar with the terms “split tunneling” and “force tunneling”. They dictate how traffic is handled when a DirectAccess (or VPN) connection is established by a client. Split tunneling routes only traffic destined for the internal network over the DirectAccess connection; all other traffic is routed directly over the Internet. Force tunneling routes all traffic over the DirectAccess connection.

Force Tunneling

DirectAccess uses split tunneling by default. Optionally, it can be configured to use force tunneling if required. Force tunneling is commonly enabled when DirectAccess administrators want to inspect and monitor Internet traffic from field-based clients.

Note: One-time password user authentication is not supported when force tunneling is enabled. Details here.

Drawbacks

Force tunneling is not without its drawbacks. It requires that an on-premises proxy server be used by DirectAccess clients to access the Internet, in most cases. In addition, the user experience is often poor when force tunneling is enabled. This is caused by routing Internet traffic, which is commonly encrypted, over an already encrypted connection. The added protocol overhead caused by double encryption (triple encryption if you are using Windows 7!) along with using a sub-optimal network path increases latency and can degrade performance significantly. Also, location-based services typically fail to work correctly.

Selective Tunneling

“Selective Tunneling” is a term that I commonly use to describe a configuration where only one or a few specific public resources are tunneled over the DirectAccess connection. A common use case is where access to a cloud-based application is restricted to the IP address of a corporate proxy or firewall.

Using the Name Resolution Policy Table (NRPT) and taking advantage of DirectAccess and its requirement for IPv6, DirectAccess administrators can choose to selectively route requests for public hosts or domains over the DirectAccess connection. The process involves defining the public Fully Qualified Domain Name (FQDN) as “internal” in the DirectAccess configuration and then assigning an on-premises proxy server for DirectAccess clients to use to access that namespace.

Enable Selective Tunneling

While some of the selective tunneling configuration can be performed using the Remote Access Management console, some of it can only be done using PowerShell. For this reason, I prefer to do everything in PowerShell to streamline the process.

Run the following PowerShell commands on the DirectAccess server to enable selective tunneling for the “.example.com” domain.

$namespace = “.example.com” # include preceding dot for namespace, omit for individual host
$dnsserver = Get-ItemPropertyValue –Path HKLM:\\SYSTEM\CurrentControlSet\Services\RaMgmtSvc\Config\Parameters -Name DnsServers

Add-DAClientDnsConfiguration -DnsSuffix $namespace -DnsIpAddress $dnsserver -PassThru

$gpo = (Get-RemoteAccess).ClientGpoName
$gpo = $gpo.Split(‘\’)[1]
$proxy = “proxy.corp.example.net:8080” # this is the FQDN and port for the internal proxy server
$rule = (Get-DnsClientNrptRule -GpoName $gpo | Where-Object Namespace -eq $namespace | Select-Object -ExpandProperty “Name”)

Set-DnsClientNrptRule -DAEnable $true -DAProxyServerName $proxy -DAProxyType “UseProxyName” -Name $rule -GpoName $gpo

If Windows 7 client support has been enabled, run the following PowerShell commands on the DirectAccess server. If multisite is enabled, run these commands on one DirectAccess server in each entry point.

$downlevelgpo = (Get-RemoteAccess).DownlevelGpoName
$downlevelgpo = $downlevelgpo.Split(‘\’)[1]
$proxy = “proxy.corp.example.net:8080” # this is the FQDN and port for the internal proxy server
$downlevelrule = (Get-DnsClientNrptRule -GpoName $downlevelgpo | Where-Object Namespace -eq $namespace | Select-Object -ExpandProperty “Name”)

Set-DnsClientNrptRule -DAEnable $true -DAProxyServerName $proxy -DAProxyType “UseProxyName” -Name $downlevelrule -GpoName $downlevelgpo

To remove a namespace from the NRPT, run the following PowerShell command.

Remove-DAClientDnsConfiguration -DnsSuffix $namespace

Caveats

While selective tunneling works well for the most part, the real drawback is that only Microsoft browsers (Internet Explorer and Edge) are supported. Web sites configured for selective tunneling will not be reachable when using Chrome, Firefox, or any other third-party web browser. In addition, many web sites deliver content using more than one FQDN, which may cause some web pages to load improperly.

Additional Resources

DirectAccess Force Tunneling and Proxy Server Configuration

NetMotion Mobility for DirectAccess Administrators – Split vs. Force Tunneling