Always On VPN IKEv2 Load Balancing and NAT

Always On VPN IKEv2 Load Balancing and NATOver the last few weeks, I’ve worked with numerous organizations and individuals troubleshooting connectivity and performance issues associated with Windows 10 Always On VPN, and specifically connections using the Internet Key Exchange version 2 (IKEv2) VPN protocol. An issue that appears with some regularity is when Windows 10 clients fail to connect with error 809. In this scenario, the server will accept connections without issue for a period of time and then suddenly stop accepting requests. When this happens, existing connections continue to work without issue in most cases. Frequently this occurs with Windows Server Routing and Remote Access Service (RRAS) servers configured in a clustered array behind an External Load Balancer (ELB).

Network Address Translation

It is not uncommon to use Network Address Translation (NAT) when configuring Always On VPN. In fact, for most deployments the public IP address for the VPN server resides not on the VPN server, but on an edge firewall or load balancer connected directly to the Internet. The firewall/load balancer is then configured to translate the destination address to the private IP address assigned to the VPN server in the perimeter/DMZ or the internal network. This is known a Destination NAT (DNAT). Using this configuration, the client’s original source IP address is left intact. This configuration presents no issues for Always On VPN.

Source Address Translation

When troubleshooting these issues, the common denominator seems to be the use of Full NAT, which includes translating the source address in addition to the destination. This results in VPN client requests arriving at the VPN server as appearing not to come from the client’s original IP address, but the IP address of the network device (firewall or load balancer) that is translating the request. Full NAT may be explicitly configured by an administrator, or in the case of many load balancers, configured implicitly because the load balancer is effectively proxying the connection.

Known Issues

IKEv2 VPN connections use IPsec for encryption, and by default, Windows limits the number of IPsec Security Associations (SAs) coming from a single IP address. When a NAT device is performing destination/full NAT, the VPN server sees all inbound IKEv2 VPN requests as coming from the same IP address. When this happens, clients connecting using IKEv2 may fail to connect, most commonly when the server is under moderate to heavy load.

Resolution

The way to resolve this issue is to ensure that any load balancers or NAT devices are not translating the source address but are performing destination NAT only. The following is configuration guidance for F5, Citrix ADC (formerly NetScaler), and Kemp load balancers.

F5

On the F5 BIG-IP load balancer, navigate to the Properties > Configuration page of the IKEv2 UDP 500 virtual server and choose None from the Source Address Translation drop-down list. Repeat this step for the IKEv2 UDP 4500 virtual server.

Always On VPN IKEv2 Load Balancing and NAT

Citrix ADC

On the Citrix ADC load balancer, navigate to System > Settings > Configure Modes and check the option to Use Subnet IP.

Always On VPN IKEv2 Load Balancing and NAT

Next, navigate to Traffic Management > Load Balancing > Service Groups and select the IKEv2 UDP 500 service group. In the Settings section click edit and select Use Client IP. Repeat these steps for the IKEv2 UDP 4500 service group.

Always On VPN IKEv2 Load Balancing and NAT

Kemp

On the Kemp LoadMaster load balancer, navigate to Virtual Services > View/Modify Services and click Modify on the IKEv2 UDP 500 virtual service. Expand Standard Options and select Transparency. Repeat this step for the IKEv2 UDP 4500 virtual service.

Always On VPN IKEv2 Load Balancing and NAT

Caveat

Making the changes above may introduce routing issues in your environment. When configuring these settings, it may be necessary to configure the VPN server’s default gateway to use the load balancer to ensure proper routing. If this is not possible, consider implementing the workaround below.

Workaround

To fully resolve this issue the above changes should be made to ensure the VPN server can see the client’s original source IP address. If that’s not possible for any reason, the following registry key can be configured to increase the number of established SAs from a single IP address. Be advised this is only a partial workaround and may not fully eliminate failed IKEv2 connections. There are other settings in Windows that can prevent multiple connections from a single IP address which are not adjustable at this time.

To implement this registry change, open an elevated PowerShell command window on the RRAS server and run the following commands. Repeat these commands on all RRAS servers in the organization.

New-ItemProperty -Path ‘HKLM:SYSTEM\CurrentControlSet\Services\IKEEXT\Parameters\’ -Name IkeNumEstablishedForInitialQuery -PropertyType DWORD -Value 50000 -Force

Restart-Service IKEEXT -Force -PassThru

Additional Information

IPsec Traffic May Be Blocked When A Computer is Behind a Load Balancer

Windows 10 Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Windows 10 Always On VPN IKEv2 Load Balancing with F5 BIG-IP

Windows 10 Always On VPN IKEv2 Load Balancing with Kemp LoadMaster

Always On VPN DNS Registration Update Available

Always On VPN DNS Registration Update AvailableWhen configuring Always On VPN, administrators have the option to enable DNS registration for VPN clients. When this option is set, VPN clients will register the IP address assigned to their VPN interface in the internal DNS. This allows client devices to be managed using their hostname from the internal network whenever they are connected remotely.

DNS Registration

DNS registration is enabled in one of two ways, depending on how Always On VPN client devices are managed.

Intune

When using the native Microsoft Intune UI to manage Always On VPN profiles, DNS registration can be configured by selecting Enabled next to Register IP addresses with internal DNS in the Base VPN settings section.

Always On VPN DNS Registration Update Available

ProfileXML

When using custom ProfileXML with PowerShell, SCCM, or Intune, the administrator will define the RegisterDNS element to enable DNS registration.

Always On VPN DNS Registration Update Available

Known Issues

Some users have reported unexpected behavior when DNS registration is enabled. Specifically, under some circumstances the VPN client will register the IP address of the VPN network interface along with the IP address of its public network interface (Wi-Fi, Ethernet, etc.). However, the VPN client can only be managed using the VPN interface. If the VPN client’s hostname resolves to its public IP address, manage out will fail.

This appears to happen only when Name Resolution Policy Table (NRPT) rules are defined in Intune DNS settings, or if the DomainNameInformation element is defined in ProfileXML.

Always On VPN DNS Registration Update AvailableAlways On VPN DNS Registration Update Available

Resolution

Microsoft recently released fixes for this DNS registration issue for Windows 10. The fix for this issue is included in the following updates.

Windows 10 1803 – KB4507466
Windows 10 1809 – KB4505658
Windows 10 1903 – KB4505903

Additional Configuration

After installing the update, the following registry entry must be defined on each VPN client.

HKLM\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters\DisableNRPTForAdapterRegistration DWORD = 1

To enable this setting, open an elevated PowerShell window and run the following command.

New-ItemProperty -Path ‘HKLM:SYSTEM\CurrentControlSet\Services\Dnscache\Parameters\’ -Name DisableNRPTForAdapterRegistration -PropertyType DWORD -Value 1 -Force

Once complete, restart the client device for the changes to take effect. After validation testing is complete, the registry entry can be deployed to Always On VPN clients using Active Directory group policy preferences or Intune.

Additional Information

Deploying Windows 10 Always On VPN with Intune using Custom ProfileXML

Windows 10 Always On VPN Updates to Improve Connection Reliability

Windows 10 Always On VPN Device Tunnel Configuration using Microsoft Intune

Windows 10 Always On VPN Hands-On Training Classes

Always On VPN RasMan Errors in Windows 10 1903

Always On VPN RasMan Errors in Windows 10 1903After deploying or upgrading to Windows 10 1903, administrators may find that Windows 10 Always On VPN connections fail to establish successfully. Always On VPN connections continue to work for Windows 10 1809 and earlier clients, however.

Important Note: The issue described in this article has been addressed in KB4505903 (build 18362.267) released July 26, 2019.

RasMan Event Log Errors

When this occurs, the application event log contains an error with Event ID 1000 that includes the following information.

“Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005”

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Root Cause

RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix.

Workaround

As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell.

Group Policy

Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps.

1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds
2. Double-click Allow Telemetry.
3. Select Enabled.
4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security).
5. Click Ok.

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Registry

Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting.

HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Note: The AllowTelemetry value can be removed entirely, if desired.

PowerShell

PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting.

New-ItemProperty -Path ‘HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\’ -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force

Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely.

Remove-ItemProperty -Path ‘HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\’ -Name AllowTelemetry

Service Restart Required

Once these changes have been made, restart the Remote Access Connection Manager service (RasMan) using the Services mnagement console (services.msc) or by running the following PowerShell command.

Restart-Service RasMan -PassThru

Optionally, the client can be rebooted to apply these changes.

Additional Information

Windows 10 1903 Known Issues