Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMasterA recent update to the Kemp LoadMaster load balancer may cause failed connections for Always On VPN connections using IKEv2. SSTP VPN connections are unaffected.

Load Balancing IKEv2

When using the Kemp LoadMaster load balancer to load balance IKEv2, custom configuration is required to ensure proper operation. Specifically, the virtual service must be configured to use “port following” to ensure both the initial request on UDP port 500 and the subsequent request on UDP port 4500 are sent to the same real server. This requires the virtual service to be configured to operate at layer 7. Detailed configuration guidance for load balancing IKEv2 on the Kemp LoadMaster load balancer can be found here.

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Issues with LMOS 7.2.48.0

A recent release of the Load Master Operating System (LMOS) v7.2.48.0 introduced a bug that affects UDP services configured to operate at layer 7, which includes IKEv2. This bug breaks Always On VPN connections using IKEv2, resulting in failed connections. When this occurs, the administrator may encounter an error 809 message for device tunnel or user tunnel.

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Update Available

Administrators who use the Kemp LoadMaster load balancer to load balance Always On VPN IKEv2 connections and have updated to LMOS 7.2.48.0 are encouraged to update to LMOS 7.2.48.1 immediately. This latest update includes a fix that resolves broken IKEv2 load balancing for Always On VPN. Once the LoadMaster has been updated to 7.2.48.1, Always On VPN connections using IKEv2 should complete successfully.

Additional Information

Windows 10 Always On VPN IKEv2 Load Balancing and NAT

Windows 10 Always On VPN IKEv2 Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN SSTP Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN Load Balancing with Kemp LoadMaster in Azure

Windows 10 Always On VPN Load Balancing Deployment Guide for Kemp Load Balancers

Always On VPN RasMan Errors in Windows 10 1903

Always On VPN RasMan Errors in Windows 10 1903After deploying or upgrading to Windows 10 1903, administrators may find that Windows 10 Always On VPN connections fail to establish successfully. Always On VPN connections continue to work for Windows 10 1809 and earlier clients, however.

Important Note: The issue described in this article has been addressed in KB4505903 (build 18362.267) released July 26, 2019.

RasMan Event Log Errors

When this occurs, the application event log contains an error with Event ID 1000 that includes the following information.

“Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005”

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Root Cause

RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix.

Workaround

As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell.

Group Policy

Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps.

1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds
2. Double-click Allow Telemetry.
3. Select Enabled.
4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security).
5. Click Ok.

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Registry

Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting.

HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Note: The AllowTelemetry value can be removed entirely, if desired.

PowerShell

PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting.

New-ItemProperty -Path ‘HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\’ -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force

Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely.

Remove-ItemProperty -Path ‘HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\’ -Name AllowTelemetry

Service Restart Required

Once these changes have been made, restart the Remote Access Connection Manager service (RasMan) using the Services mnagement console (services.msc) or by running the following PowerShell command.

Restart-Service RasMan -PassThru

Optionally, the client can be rebooted to apply these changes.

Additional Information

Windows 10 1903 Known Issues

 

Always On VPN Device Tunnel and Certificate Revocation

Always On VPN Device Tunnel and Certificate RevocationRecently I wrote about denying access to Windows 10 Always On VPN users or computers. In that post I provided specific guidance for denying access to computers configured with the device tunnel. To summarize, the process involved exporting the device certificate from the issuing Certification Authority (CA) server and placing it in the Untrusted Certificates certificate store on each VPN server. In theory, simply revoking the device certificate should be all that’s required to prevent device tunnel connections.

Revocation Check Failure

As it turns out, a bug in Windows Server Routing and Remote Access prevents this from working as expected. Windows Server 2012 R2, 2016, and 2019 all fail to check the Certificate Revocation List (CRL) for IKEv2 VPN connections using machine certificate authentication (for example an Always On VPN device tunnel).

Updates for Windows Server

Microsoft has released fixes to support device tunnel certificate revocation for the following operating systems.

Windows Server 2019 – KB4505658 (build 17763.652)

Windows Server 2016 – KB4503294 (build 14393.3053)

Windows Server 2012/R2 – Will not be updated.

Enable Revocation Check

Additional configuration is required to enable support for CRL checking. Microsoft published guidance for configuring CRL revocation checks for IKEv2 VPN connections using machine certificate authentication here. Specifically, administrators must enable the RootCertificateNameToAccept parameter and set a registry key to enable this functionality.

Open an elevated PowerShell window and run the following commands to enable CRL checking for IKEv2 VPN connections using machine certificate authentication.

$Thumbprint = ‘Root CA Certificate Thumbprint’
$RootCACert = (Get-ChildItem -Path cert:\LocalMachine\root | Where-Object {$_.Thumbprint -eq $Thumbprint})
Set-VpnAuthProtocol -RootCertificateNameToAccept $RootCACert -PassThru

New-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Services\RemoteAccess\Parameters\Ikev2\’ -Name CertAuthFlags -PropertyTYpe DWORD -Value ‘4’ -Force

Restart-Service RemoteAccess -PassThru

Always On VPN Device Tunnel and Certificate Revocation

A PowerShell script to update the RootCertificateNameToAccept parameter on multiple VPN servers can be found here.

Revoking Certificates

To prevent a Windows 10 Always On VPN device tunnel connection, the administrator must first revoke the certificate on the issuing CA. Next, open an elevated command window an enter the following commands. Repeat these steps on each VPN server in the enterprise.

certutil -urlcache * delete
certutil -setreg chain\ChainCacheResyncFiletime @now

Additional Information

Denying Access to Windows 10 Always On VPN Users or Computers

Blocking VPN Clients that use Revoked Certificates

PowerShell Script to Configure RootCertificateNameToAccept on GitHub

 

 

%d bloggers like this: