Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMasterA recent update to the Kemp LoadMaster load balancer may cause failed connections for Always On VPN connections using IKEv2. SSTP VPN connections are unaffected.

Load Balancing IKEv2

When using the Kemp LoadMaster load balancer to load balance IKEv2, custom configuration is required to ensure proper operation. Specifically, the virtual service must be configured to use “port following” to ensure both the initial request on UDP port 500 and the subsequent request on UDP port 4500 are sent to the same real server. This requires the virtual service to be configured to operate at layer 7. Detailed configuration guidance for load balancing IKEv2 on the Kemp LoadMaster load balancer can be found here.

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Issues with LMOS 7.2.48.0

A recent release of the Load Master Operating System (LMOS) v7.2.48.0 introduced a bug that affects UDP services configured to operate at layer 7, which includes IKEv2. This bug breaks Always On VPN connections using IKEv2, resulting in failed connections. When this occurs, the administrator may encounter an error 809 message for device tunnel or user tunnel.

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Update Available

Administrators who use the Kemp LoadMaster load balancer to load balance Always On VPN IKEv2 connections and have updated to LMOS 7.2.48.0 are encouraged to update to LMOS 7.2.48.1 immediately. This latest update includes a fix that resolves broken IKEv2 load balancing for Always On VPN. Once the LoadMaster has been updated to 7.2.48.1, Always On VPN connections using IKEv2 should complete successfully.

Additional Information

Windows 10 Always On VPN IKEv2 Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN SSTP Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN Load Balancing with Kemp LoadMaster in Azure

Windows 10 Always On VPN Load Balancing Deployment Guide for Kemp Load Balancers

Always On VPN RasMan Errors in Windows 10 1903

Always On VPN RasMan Errors in Windows 10 1903After deploying or upgrading to Windows 10 1903, administrators may find that Windows 10 Always On VPN connections fail to establish successfully. Always On VPN connections continue to work for Windows 10 1809 and earlier clients, however.

Important Note: The issue described in this article has been addressed in KB4505903 (build 18362.267) released July 26, 2019.

RasMan Event Log Errors

When this occurs, the application event log contains an error with Event ID 1000 that includes the following information.

“Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005”

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Root Cause

RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix.

Workaround

As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell.

Group Policy

Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps.

1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds
2. Double-click Allow Telemetry.
3. Select Enabled.
4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security).
5. Click Ok.

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Registry

Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting.

HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Note: The AllowTelemetry value can be removed entirely, if desired.

PowerShell

PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting.

New-ItemProperty -Path ‘HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\’ -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force

Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely.

Remove-ItemProperty -Path ‘HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\’ -Name AllowTelemetry

Service Restart Required

Once these changes have been made, restart the Remote Access Connection Manager service (RasMan) using the Services mnagement console (services.msc) or by running the following PowerShell command.

Restart-Service RasMan -PassThru

Optionally, the client can be rebooted to apply these changes.

Additional Information

Windows 10 1903 Known Issues

 

Always On VPN Device Tunnel and Certificate Revocation

Always On VPN Device Tunnel and Certificate RevocationRecently I wrote about denying access to Windows 10 Always On VPN users or computers. In that post I provided specific guidance for denying access to computers configured with the device tunnel. To summarize, the process involved exporting the device certificate from the issuing Certification Authority (CA) server and placing it in the Untrusted Certificates certificate store on each VPN server. In theory, simply revoking the device certificate should be all that’s required to prevent device tunnel connections.

Revocation Check Failure

As it turns out, a bug in Windows Server Routing and Remote Access prevents this from working as expected. Windows Server 2012 R2, 2016, and 2019 all fail to check the Certificate Revocation List (CRL) for IKEv2 VPN connections using machine certificate authentication (for example an Always On VPN device tunnel).

Updates for Windows Server

Microsoft has released fixes to support device tunnel certificate revocation for the following operating systems.

Windows Server 2019 – KB4505658 (build 17763.652)

Windows Server 2016 – KB4503294 (build 14393.3053)

Windows Server 2012/R2 – Will not be updated.

Enable Revocation Check

Additional configuration is required to enable support for CRL checking. Microsoft published guidance for configuring CRL revocation checks for IKEv2 VPN connections using machine certificate authentication here. Specifically, administrators must enable the RootCertificateNameToAccept parameter and set a registry key to enable this functionality.

Open an elevated PowerShell window and run the following commands to enable CRL checking for IKEv2 VPN connections using machine certificate authentication.

$Thumbprint = ‘Root CA Certificate Thumbprint’
$RootCACert = (Get-ChildItem -Path cert:\LocalMachine\root | Where-Object {$_.Thumbprint -eq $Thumbprint})
Set-VpnAuthProtocol -RootCertificateNameToAccept $RootCACert -PassThru

New-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Services\RemoteAccess\Parameters\Ikev2\’ -Name CertAuthFlags -PropertyTYpe DWORD -Value ‘4’ -Force

Restart-Service RemoteAccess -PassThru

Always On VPN Device Tunnel and Certificate Revocation

A PowerShell script to update the RootCertificateNameToAccept parameter on multiple VPN servers can be found here.

Revoking Certificates

To prevent a Windows 10 Always On VPN device tunnel connection, the administrator must first revoke the certificate on the issuing CA. Next, open an elevated command window an enter the following commands. Repeat these steps on each VPN server in the enterprise.

certutil -urlcache * delete
certutil -setreg chain\ChainCacheResyncFiletime @now

Additional Information

Denying Access to Windows 10 Always On VPN Users or Computers

Blocking VPN Clients that use Revoked Certificates

PowerShell Script to Configure RootCertificateNameToAccept on GitHub

 

 

Always On VPN Updates to Improve Connection Reliability

Always On VPN Updates to Improve Connection ReliabilityA longstanding issue with Windows 10 Always On VPN is that of VPN tunnel connectivity reliability and device tunnel/user tunnel interoperability. Many administrators have reported that Always On VPN connections fail to establish automatically at times, that only one tunnel comes up at a time (user tunnel or device tunnel, but not both), or that VPN tunnels fail to establish when coming out of sleep or hibernate modes. Have a look at the comments on this post and you’ll get a good understanding of the issues with Always On VPN.

Recent Updates

The good news is that most of these issues have been resolved with recent updates to Windows 10 1803 and 1809. Specifically, the February 19, 2019 update for Windows 10 1803 (KB4487029) and the March 1, 2019 update for Windows 10 1809 (KB4482887) include fixes to address these known issues. Administrators are encouraged to deploy Windows 10 1803 with the latest updates applied when implementing Always On VPN. Windows 10 1809 with the latest updates applied is preferred though.

Persistent Issues

Although initial reports are favorable for these updates and based on my experience the effectiveness and reliability of Windows 10 Always On VPN is greatly improved, there have still been some reports of intermittent VPN tunnel establishment failures.

Possible Causes

During my testing, after applying the updates referenced earlier both device tunnel and user tunnel connections are established much more consistently than before the updates were applied. I did encounter some issues, however. Specifically, when coming out of sleep or hibernate, VPN connections would fail to establish. Occasionally VPN connections would fail after a complete restart.

NCSI

After further investigation it was determined that the connectivity failure was caused by the Network Connectivity Status Indicator (NCSI) probe failing, causing Windows to report “No Internet access”.

Always On VPN Updates to Improve Connection Reliability

Cisco Umbrella Roaming Client

In this instance the NCSI probe failure was caused by the Cisco Umbrella Roaming Client installed and running on the device. The Umbrella Roaming Client is security software that provides client protection by monitoring and filtering DNS queries. It operates by configuring a DNS listener on the loopback address. NCSI probes are known to fail when the DNS server is running on a different interface than is being tested.

Resolution

Microsoft released a fix for this issue in Windows 10 1709. The fix involves changing a group policy setting to disable interface binding when perform DNS lookups by the NCSI. You can enable this setting via Active Directory group policy by navigating to Computer Configuration > Administrative Templates > Network > Network Connectivity Status Indicator > Specify global DNS. Select Enabled and check the option to Use global DNS, as shown here.

Always On VPN Updates to Improve Connection Reliability

For testing purposes this setting can be enabled individual using the following PowerShell command.

New-ItemProperty -Path “HKLM:\SOFTWARE\Policies\Microsoft\Windows\NetworkConnectivityStatusIndicator\” -Name UseGlobalDNS -PropertyType DWORD -Value 1 -Force

Third-Party Software

As Always On VPN connectivity can be affected by NCSI, any third-party firewall or antivirus/antimalware solution could potentially introduce VPN connection instability. Observe NCSI operation closely when troubleshooting unreliable connections with Always On VPN.

Additional Information

Windows 10 1803 Update KB4487029

Windows 10 1809 Update KB4482887

Cisco Umbrella Roaming Client Limited Network Connectivity Warning

Network Connectivity Status Indicator (NCSI) Operation Explained

Hotfix Available for DirectAccess OTP Configuration Issues

If you’ve ever tried configuring DirectAccess to use One-Time Password (OTP) authentication, you’ve no doubt discovered that the native Microsoft Remote Access Management console would return the following error when trying to detect and locate Certificate Authority (CA) servers.

No CA servers can be detected, and OTP cannot be configured. Ensure that
servers added to the list are available on each domain controller in the
corporate network.

Configure DirectAccess with OTP Authentication

The workaround for this issue required dropping to the command line and executing PowerShell commands to complete this configuration as I outlined here.

Thankfully Microsoft has made available a hotfix to address this issue, returning full GUI functionality for configuring DirectAccess and OTP authentication. For additional details about this hotfix and to request the update itself, click here.

Critical Update MS15-034 and DirectAccess

Microsoft Security Bulletin MS15-034 Vulnerability in HTTP.sys affects DirectAccessThe April 2015 monthly security update release from Microsoft includes a fix for a serious vulnerability in HTTP.sys. On an unpatched server, an attacker who sends a specially crafted HTTP request will be able to execute code remotely in the context of the local system account. DirectAccess leverages HTTP.sys for the IP-HTTPS IPv6 transition protocol and is critically exposed. Organizations who have deployed DirectAccess are urged to update their systems immediately.

More information can be found on MS15-034 here.

Unable to Install DirectAccess Hotfix KB2953212 to Disable NRPT

Last year I wrote about Microsoft hotfix KB2953212 that that allowed users to disable the Name Resolution Policy Table (NRPT) on a DirectAccess client. This hotfix addressed a specific scenario where a DirectAccess client on the internal corporate network could not connect to local resources due to Network Location Server (NLS) unreachability.

When installing this update, you many encounter the following error message:

Windows Update Standalone Installer
The update is not applicable to your computer

Unable to Install DirectAccess Hotfix KB2953212 to Disable NRPT

This occurs because the KB2953212 hotfix was included in KB3000850, the November 2014 update rollup for Windows 8.1 and Windows Server 2012 R2. You can verify this by opening the Control Panel and selecting Programs and then clicking View installed updates under Programs and Features.

Unable to Install DirectAccess Hotfix KB2953212 to Disable NRPT

If you have the November 2014 update rollup installed there is no need to install KB2953212, as that hotfix is already included in the rollup.

Hotfix Available to Disable NRPT on Windows 8.x DirectAccess Clients

Updated April 9, 2015: The hotfix referred to in this article is now included in the November 2014 update rollup for Windows 8.1 and Windows Server 2012 R2. You will receive an error message when installing this update on Windows 8.x clients with the update rollup installed. More details here.

The Network Location Server (NLS) is a critical infrastructure component for DirectAccess deployments. The NLS is used by DirectAccess clients to determine if the client is located inside or outside of the corporate network. If the NLS becomes unavailable, DirectAccess clients that are already outside the corporate network are unaffected. However, DirectAccess clients that are inside the corporate network will mistakenly believe that they are outside and the Name Resolution Policy Table (NRPT) will be enabled, forcing name resolution requests for hosts in the internal namespace to be sent to the DNS64 service running on the DirectAccess server. If the DirectAccess server is unreachable from the internal network (a common scenario for a variety of reasons), DirectAccess clients inside the corporate network will be unable to connect to any local network resources by name until the NLS is once again reachable.

Configuring the Network Connectivity Assistant to Allow DirectAccess clients to use local name resolution does not resolve this issue. Although it sounds intuitive, it doesn’t resolve this specific issue where the NLS is unreachable.

Hotfix Available to Disable NRPT on Windows 8.x DirectAccess Clients

When the option to Allow DirectAccess clients to use local name resolution is enabled, the client can only choose to disconnect (use local name resolution) after it has successfully established a connection to the DirectAccess server. If the DirectAccess connection shows that it is still connecting, the option to disconnect is not available.

Hotfix Available to Disable NRPT on Windows 8.x DirectAccess Clients

To address this issue, Microsoft has released update KB2953212 for Windows 8.x clients that allows the disabling of the NRPT regardless if the client has successfully established a DirectAccess connection. With this update, if a DirectAccess client is located on the corporate network and is unable to reach the NLS, the user will be able to disable the NRPT (effectively disconnect DirectAccess) and once again connect to resources on the corporate network.
Hotfix Available to Disable NRPT on Windows 8.x DirectAccess Clients

This update is certainly no excuse not to deploy your NLS in a highly-available configuration using Windows Network Load Balancing (NLB) or a third-party external load balancer (hardware or software), but it can be a life-saver if your NLS becomes unavailable for any reason. I’d recommend deploying this update to all of your Windows 8.x DirectAccess clients soon.

For more information and to download the hotfix, click here.

Error 0x80040001 When Using OTP on Windows 7 SP1 DirectAccess Clients

Microsoft recently released a hotfix to resolve an issue where Windows 7 SP1 DirectAccess clients fail to connect to a DirectAccess server with the IP-HTTPS IPv6 transition protocol and using One-Time Password (OTP) authentication via the DirectAccess Connectivity Assistant (DCA) 2.0. In this scenario you may receive an HTTP 403 error from the DirectAccess server in response to the certificate signing requests and a 0x80040001 error after entering the OTP.

You can learn more about the hotfix for DCA 2.0 on Windows 7 SP1 and download the associated hotfix here.

Hotfix Available for Windows Server 2012 R2 DirectAccess Configuration Issue

A while back I wrote about an issue that I encountered when attempting to configure DirectAccess in Windows Server 2012 R2 using a dedicated Network Location Server (NLS). In this deployment scenario, the Remote Access Setup Wizard would fail and return the following error message:

The configuration was rolled back successfully. The URL specified for the network location server cannot be resolved to an IP address.

Windows Server 2012 R2 DirectAccess Name Resolution Issue

Upon further investigation, the NLS server name does indeed resolve correctly, and clicking validate when defining the NLS works without issue. Originally I proposed a workaround that involved changing a registry setting. However, after working with Microsoft to identify the issue they have released a hotfix to resolve this issue correctly. You can download the hotfix here.

%d bloggers like this: