Always On VPN Fails with Windows 10 2004 Build 610

Updated 11/10/2020: Microsoft update KB4586781 has resolved the connectivity issues described in this post. If you had previously installed update KB4580364, please update to KB4586781 immediately.

A recent preview update for Windows 10 2004 has broken Always On VPN. Specifically, after installing the latest Preview update for Windows 10 2004 (KB4580364), Always On VPN connections will fail to connect automatically. They can be established manually, however.

Affected Builds

This issue affects Windows 10 2004 with build 19041.610 and 19042.610.

Always On VPN Fails with Windows 10 2004

Workaround

The only workaround currently is to remove this update.

Caveat

Although this is a “preview” update and an optional installation, it is important to know that preview updates are released in the next “patch Tuesday” release. Administrators are advised to carefully consider delaying the implementation until additional testing has been completed.

Additional Information

October 29, 2020 – KB4580364 (OS Builds 19041.610 and 19042.610) Preview Update

Windows 10 Always On VPN Updates for Windows 10 2004

Always On VPN Updates for Windows 10 2004

Always On VPN Updates for Windows 10 2004Microsoft recently made available an update for Windows 10 2004 that includes many important fixes for outstanding issues with Windows 10 Always On VPN. KB4571744 (build 19041.488) addresses many challenges faced by Always On VPN administrators today, including the following.

TPM

This update addresses an issue that prevents hash signing from working correctly using the Microsoft Platform Crypto Provider for Trusted Platform Module (TPM). This issue can occur when administrators configure Always On VPN to use Protected Extensible Authentication Protocol (PEAP) with client certificate authentication using a FortiGate security device.

Sleep/Hibernate

This update also addresses issues with Windows 10 Always On VPN failing to automatically reconnect when resuming from sleep or hibernate. I’ve written about issues with Always On VPN and sleep/hibernate in the past. This is an issue that has plagued Always On VPN since its introduction, so let’s hope this finally provides some meaningful relief from this persistent problem.

Certificate Authentication

When both the Always On VPN device tunnel and user tunnel are provisioned to a Windows 10 clients, user tunnel connections may be authenticated using the machine certificate and not EAP/PEAP. This can result in connections that are not validated as intended, and allowing a user to bypass configured NPS policies, MFA requirements, or conditional access rules. This update includes a fix for this issue, restoring proper authentication for the user tunnel when the device tunnel is also provisioned.

Device and User Tunnel Coexistence

A bug that first appeared when Windows 10 2004 was introduced prevented a device tunnel and user tunnel Always On VPN connection from being established to the same VPN server if the user tunnel used Internet Key Exchange Version 2 (IKEv2). This update restores full functionality under those conditions.

Update KB4571744

To resolve these issues with Windows 10 Always On VPN as well as others, download and install update KB4571744 today. If you are experiencing any of these issues with releases of Windows 10 prior to 2004, look for updates for those build to come later this year.

Additional Information

September 3, 2020 – KB4571744 (OS Build 19041.488) Preview

Windows 10 Always On VPN Connection Issues after Sleep or Hibernate

Windows 10 Always On VPN Bug in Windows 10 2004

Always On VPN Device Tunnel Operation and Best Practices

Always On VPN Device Tunnel Operation and Best PracticesUnlike DirectAccess, Windows 10 Always On VPN settings are deployed to the individual user, not the device. As such, there is no support for logging on without cached credentials using the default configuration. To address this limitation, and to provide feature parity with DirectAccess, Microsoft later introduced the device tunnel option in Windows 10 1709.

Device Tunnel Use Cases

The device tunnel is designed to allow the client device to establish an Always On VPN connection before the user logs on. This enables important scenarios such as logging on without cached credentials. This feature is crucial for organizations who expect users to log on to devices the first time remotely. The device tunnel can also be helpful for remote support, allowing administrators to manage remotely connected Always On VPN clients without having a user logged on. In addition, the device tunnel can alleviate some of the pain caused by administrators resetting remote worker’s passwords, or by users initiating a Self-Service Password Reset (SSPR).

Device Tunnel Requirements

The device tunnel requires Windows 10 Enterprise edition 1709 or later, and the client device must be joined to the domain. The device tunnel must be provisioned in the context of the local system account. Guidance for configuring and deploying a Windows 10 Always On VPN device tunnel can be found here.

Device Tunnel Authentication

The device tunnel is authenticated using a certificate issued to the client device, much the same as DirectAccess does. Authentication takes place on the Routing and Remote Access Service (RRAS) VPN server. It does not require a Network Policy Server (NPS) to perform authentication for the device tunnel.

Always On VPN Device Tunnel Operation and Best Practices

CRL Checking

Eventually an administrator may need to deny access to a device configured with an Always On VPN device tunnel connection. In theory, revoking the client device’s certificate and terminating their IPsec Security Associations (SAs) on the VPN server would accomplish this. However, Windows Server RRAS does not perform certificate revocation checking for Windows 10 Always On VPN device tunnel connections by default. Thankfully an update is available to enable this functionality. See Always On VPN Device Tunnel and Certificate Revocation for more details.

Configuration Best Practices

As the device tunnel is designed only to support domain authentication for remote clients, it should be configured with limited access to the on-premises infrastructure. Below is a list of required and optional infrastructure services that should be reachable over the device tunnel connection.

Required

  • All domain controllers
  • Enterprise DNS servers (if DNS is running on servers other than domain controllers)

Optional

  • All issuing certification authority (CA) servers
  • All certificate services online HTTP responders
  • All certificate services Online Certificate Status Protocol (OCSP) servers
  • System Center Configuration Manager (SCCM) distribution point servers
  • Windows Server Update Services (WSUS) servers
  • Management workstations

Limiting Access

Limiting access over the Always On VPN device tunnel can be accomplished in one of the following two ways.

Traffic Filters

The administrator can configure traffic filters on the device tunnel to restrict access only to those IP addresses required. However, be advised that when a traffic filter is enabled on the device tunnel, all inbound access will be blocked. This effectively prevents any remote management of the device from an on-premises system over the device tunnel.

Host Routes

An alternative to using traffic filters to limit access over the device tunnel is using host routes. Host routes are configured with a /32 prefix size and define a route to a specific individual host. The following is an example of host route configuration in ProfileXML.

Always On VPN Device Tunnel Operation and Best Practices

Note: A PowerShell script that enumerates all enterprise domain controllers and outputs their IP addresses in XML format for use in ProfileXML can be found here.

Caveats

Some organizations may have hundreds or even thousands of domain controllers, so creating individual host route entries for all domain controllers in profileXML may not be practical. In this scenario it is recommended to add host routes only for the domain controllers that belong to the Active Directory site where the VPN server resides.

Supportability

Do not use the <DomainNameInformation> element in ProfileXML or enable force tunneling for the device tunnel. Neither of these configurations are supported.

Tunnel Coexistence

The device tunnel can be safely deployed in conjunction with the user tunnel whenever its functionality is required.

DNS Registration

If the device tunnel and user tunnel are both deployed, it is recommended that only one of the tunnels be configured to register in DNS. If the device tunnel is configured to register its IP address in DNS, be advised that only those devices with routes configured in the device tunnel VPN profile will be able to connect remotely to Always On VPN clients.

Additional Information

Windows 10 Always On VPN Device Tunnel with Azure VPN Gateway

Windows 10 Always On VPN Device Tunnel and Certificate Revocation

Windows 10 Always On VPN Device Tunnel Configuration with Microsoft Intune

Windows 10 Always On VPN Device Tunnel Does Not Connect Automatically

Windows 10 Always On VPN Device Tunnel Missing in Windows 10 UI

Deleting a Windows 10 Always On VPN Device Tunnel

Windows 10 Always On VPN Device Tunnel Configuration using PowerShell

Always On VPN Client Connections Fail with Status Connecting

Administrators who have deployed Windows 10 Always On VPN may encounter a scenario in which an Always On VPN connection fails, yet the connectivity status indicator perpetually reports a “Connecting” status.

Always On VPN Client Connections Fail with Status Connecting

Affected Clients

This is a known issue for which Microsoft has recently released updates to address. Affected clients include Windows 10 1909, 1903, and 1809.

Updates Available

The following Windows updates include a fix to resolve this problem.

KB4541335 – Windows 10 1909 and 1903

KB4541331 – Windows 10 1809

Additional Information

Always On VPN Hands-On Training

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMasterA recent update to the Kemp LoadMaster load balancer may cause failed connections for Always On VPN connections using IKEv2. SSTP VPN connections are unaffected.

Load Balancing IKEv2

When using the Kemp LoadMaster load balancer to load balance IKEv2, custom configuration is required to ensure proper operation. Specifically, the virtual service must be configured to use “port following” to ensure both the initial request on UDP port 500 and the subsequent request on UDP port 4500 are sent to the same real server. This requires the virtual service to be configured to operate at layer 7. Detailed configuration guidance for load balancing IKEv2 on the Kemp LoadMaster load balancer can be found here.

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Issues with LMOS 7.2.48.0

A recent release of the Load Master Operating System (LMOS) v7.2.48.0 introduced a bug that affects UDP services configured to operate at layer 7, which includes IKEv2. This bug breaks Always On VPN connections using IKEv2, resulting in failed connections. When this occurs, the administrator may encounter an error 809 message for device tunnel or user tunnel.

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Update Available

Administrators who use the Kemp LoadMaster load balancer to load balance Always On VPN IKEv2 connections and have updated to LMOS 7.2.48.0 are encouraged to update to LMOS 7.2.48.1 immediately. This latest update includes a fix that resolves broken IKEv2 load balancing for Always On VPN. Once the LoadMaster has been updated to 7.2.48.1, Always On VPN connections using IKEv2 should complete successfully.

Additional Information

Windows 10 Always On VPN IKEv2 Load Balancing and NAT

Windows 10 Always On VPN IKEv2 Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN SSTP Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN Load Balancing with Kemp LoadMaster in Azure

Windows 10 Always On VPN Load Balancing Deployment Guide for Kemp Load Balancers

Always On VPN DNS Registration Update Available

Always On VPN DNS Registration Update AvailableWhen configuring Always On VPN, administrators have the option to enable DNS registration for VPN clients. When this option is set, VPN clients will register the IP address assigned to their VPN interface in the internal DNS. This allows client devices to be managed using their hostname from the internal network whenever they are connected remotely.

DNS Registration

DNS registration is enabled in one of two ways, depending on how Always On VPN client devices are managed.

Intune

When using the native Microsoft Intune UI to manage Always On VPN profiles, DNS registration can be configured by selecting Enabled next to Register IP addresses with internal DNS in the Base VPN settings section.

Always On VPN DNS Registration Update Available

ProfileXML

When using custom ProfileXML with PowerShell, SCCM, or Intune, the administrator will define the RegisterDNS element to enable DNS registration.

Always On VPN DNS Registration Update Available

Known Issues

Some users have reported unexpected behavior when DNS registration is enabled. Specifically, under some circumstances the VPN client will register the IP address of the VPN network interface along with the IP address of its public network interface (Wi-Fi, Ethernet, etc.). However, the VPN client can only be managed using the VPN interface. If the VPN client’s hostname resolves to its public IP address, manage out will fail.

This appears to happen only when Name Resolution Policy Table (NRPT) rules are defined in Intune DNS settings, or if the DomainNameInformation element is defined in ProfileXML.

Always On VPN DNS Registration Update AvailableAlways On VPN DNS Registration Update Available

Resolution

Microsoft recently released fixes for this DNS registration issue for Windows 10. The fix for this issue is included in the following updates.

Windows 10 1803 – KB4507466
Windows 10 1809 – KB4505658
Windows 10 1903 – KB4505903

Additional Configuration

After installing the update, the following registry entry must be defined on each VPN client.

HKLM\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters\DisableNRPTForAdapterRegistration DWORD = 1

To enable this setting, open an elevated PowerShell window and run the following command.

New-ItemProperty -Path ‘HKLM:SYSTEM\CurrentControlSet\Services\Dnscache\Parameters\’ -Name DisableNRPTForAdapterRegistration -PropertyType DWORD -Value 1 -Force

Once complete, restart the client device for the changes to take effect. After validation testing is complete, the registry entry can be deployed to Always On VPN clients using Active Directory group policy preferences or Intune.

Additional Information

Deploying Windows 10 Always On VPN with Intune using Custom ProfileXML

Windows 10 Always On VPN Updates to Improve Connection Reliability

Windows 10 Always On VPN Device Tunnel Configuration using Microsoft Intune

Windows 10 Always On VPN Hands-On Training Classes

Always On VPN Device Tunnel and Certificate Revocation

Always On VPN Device Tunnel and Certificate RevocationRecently I wrote about denying access to Windows 10 Always On VPN users or computers. In that post I provided specific guidance for denying access to computers configured with the device tunnel. To summarize, the process involved exporting the device certificate from the issuing Certification Authority (CA) server and placing it in the Untrusted Certificates certificate store on each VPN server. In theory, simply revoking the device certificate should be all that’s required to prevent device tunnel connections.

Revocation Check Failure

As it turns out, a bug in Windows Server Routing and Remote Access prevents this from working as expected. Windows Server 2012 R2, 2016, and 2019 all fail to check the Certificate Revocation List (CRL) for IKEv2 VPN connections using machine certificate authentication (for example an Always On VPN device tunnel).

Updates for Windows Server

Microsoft has released fixes to support device tunnel certificate revocation for the following operating systems.

Windows Server 2019 – KB4505658 (build 17763.652)

Windows Server 2016 – KB4503294 (build 14393.3053)

Windows Server 2012/R2 – Will not be updated.

Enable Revocation Check

Additional configuration is required to enable support for CRL checking. Microsoft published guidance for configuring CRL revocation checks for IKEv2 VPN connections using machine certificate authentication here. Specifically, administrators must enable the RootCertificateNameToAccept parameter and set a registry key to enable this functionality.

Open an elevated PowerShell window and run the following commands to enable CRL checking for IKEv2 VPN connections using machine certificate authentication.

$Thumbprint = ‘Root CA Certificate Thumbprint’
$RootCACert = (Get-ChildItem -Path cert:\LocalMachine\root | Where-Object {$_.Thumbprint -eq $Thumbprint})
Set-VpnAuthProtocol -RootCertificateNameToAccept $RootCACert -PassThru

New-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Services\RemoteAccess\Parameters\Ikev2\’ -Name CertAuthFlags -PropertyTYpe DWORD -Value ‘4’ -Force

Restart-Service RemoteAccess -PassThru

Always On VPN Device Tunnel and Certificate Revocation

A PowerShell script to update the RootCertificateNameToAccept parameter on multiple VPN servers can be found here.

Revoking Certificates

To prevent a Windows 10 Always On VPN device tunnel connection, the administrator must first revoke the certificate on the issuing CA. Next, open an elevated command window an enter the following commands. Repeat these steps on each VPN server in the enterprise.

certutil -urlcache * delete
certutil -setreg chain\ChainCacheResyncFiletime @now

Additional Information

Denying Access to Windows 10 Always On VPN Users or Computers

Blocking VPN Clients that use Revoked Certificates

PowerShell Script to Configure RootCertificateNameToAccept on GitHub

 

 

Always On VPN Updates to Improve Connection Reliability

Always On VPN Updates to Improve Connection ReliabilityA longstanding issue with Windows 10 Always On VPN is that of VPN tunnel connectivity reliability and device tunnel/user tunnel interoperability. Many administrators have reported that Always On VPN connections fail to establish automatically at times, that only one tunnel comes up at a time (user tunnel or device tunnel, but not both), or that VPN tunnels fail to establish when coming out of sleep or hibernate modes. Have a look at the comments on this post and you’ll get a good understanding of the issues with Always On VPN.

Recent Updates

The good news is that most of these issues have been resolved with recent updates to Windows 10 1803 and 1809. Specifically, the February 19, 2019 update for Windows 10 1803 (KB4487029) and the March 1, 2019 update for Windows 10 1809 (KB4482887) include fixes to address these known issues. Administrators are encouraged to deploy Windows 10 1803 with the latest updates applied when implementing Always On VPN. Windows 10 1809 with the latest updates applied is preferred though.

Persistent Issues

Although initial reports are favorable for these updates and based on my experience the effectiveness and reliability of Windows 10 Always On VPN is greatly improved, there have still been some reports of intermittent VPN tunnel establishment failures.

Possible Causes

During my testing, after applying the updates referenced earlier both device tunnel and user tunnel connections are established much more consistently than before the updates were applied. I did encounter some issues, however. Specifically, when coming out of sleep or hibernate, VPN connections would fail to establish. Occasionally VPN connections would fail after a complete restart.

NCSI

After further investigation it was determined that the connectivity failure was caused by the Network Connectivity Status Indicator (NCSI) probe failing, causing Windows to report “No Internet access”.

Always On VPN Updates to Improve Connection Reliability

Cisco Umbrella Roaming Client

In this instance the NCSI probe failure was caused by the Cisco Umbrella Roaming Client installed and running on the device. The Umbrella Roaming Client is security software that provides client protection by monitoring and filtering DNS queries. It operates by configuring a DNS listener on the loopback address. NCSI probes are known to fail when the DNS server is running on a different interface than is being tested.

Resolution

Microsoft released a fix for this issue in Windows 10 1709. The fix involves changing a group policy setting to disable interface binding when perform DNS lookups by the NCSI. You can enable this setting via Active Directory group policy by navigating to Computer Configuration > Administrative Templates > Network > Network Connectivity Status Indicator > Specify global DNS. Select Enabled and check the option to Use global DNS, as shown here.

Always On VPN Updates to Improve Connection Reliability

For testing purposes this setting can be enabled individual using the following PowerShell command.

New-ItemProperty -Path “HKLM:\SOFTWARE\Policies\Microsoft\Windows\NetworkConnectivityStatusIndicator\” -Name UseGlobalDNS -PropertyType DWORD -Value 1 -Force

Third-Party Software

As Always On VPN connectivity can be affected by NCSI, any third-party firewall or antivirus/antimalware solution could potentially introduce VPN connection instability. Observe NCSI operation closely when troubleshooting unreliable connections with Always On VPN.

Additional Information

Windows 10 1803 Update KB4487029

Windows 10 1809 Update KB4482887

Cisco Umbrella Roaming Client Limited Network Connectivity Warning

Network Connectivity Status Indicator (NCSI) Operation Explained

Always On VPN Device Tunnel Missing in Windows 10 UI

Always On VPN Device Tunnel Missing in Windows 10 UIUnlike DirectAccess, Always On VPN connections are provisioned to the user, not the machine. Beginning with Windows 10 release 1709 Microsoft introduced the device tunnel option to provide feature parity with DirectAccess. The device tunnel provides pre-logon network connectivity to support important deployment scenarios such as logging on without cached credentials and unattended remote systems management.

Device Tunnel Configuration

Guidance for creating and deploying a device tunnel connection can be found here. It’s important to note that the device tunnel is always on by default. Also, there can only be a single device tunnel configured per device. You must remove an existing device tunnel before configuring a new one.

Known Issues

After configuring a Windows 10 Always On VPN device tunnel the administrator may notice two anomalies. First, the device tunnel is missing in the Windows UI after it is created. Second, viewing the status of the device tunnel connection using PowerShell indicates the connection is “disconnected” even though it is connected.

Device Tunnel Missing

As you can see below, event though both a device and user tunnel have been provisioned, the Windows UI reports only a single Always On VPN connection, that being the user connection.

Always On VPN Device Tunnel Missing in Windows 10 UI

However, the device tunnel does appear in the Network Connections control panel applet (ncpa.cpl), as shown here.

Always On VPN Device Tunnel Missing in Windows 10 UI

This is expected and by design. The device tunnel is not displayed to the user in the Windows UI as it is provisioned to the machine, not the user. It appears on the Control Panel because the applet is capable of enumerating both user and system connections.

Device Tunnel Disconnected

The status of the Windows 10 Always On VPN device tunnel connection can be viewed by running the Get-VpnConnection -AllUserConnection PowerShell command. However, at the time of this writing, PowerShell always reports the connection status as “Disconnected”. This appears to be a bug; one which Microsoft is hopefully working to address.

Always On VPN Device Tunnel Missing in Windows 10 UI

Summary

The Windows 10 Always On VPN device tunnel option allows administrators to enable scenarios previously supported with DirectAccess, including logging on without cached credentials and unattended remote support. Not all deployments require a device tunnel, but it is an important option available to administrators to address specific use cases.

Additional Information

Windows 10 Always On VPN Device Tunnel Configuration using PowerShell

Windows 10 Always On VPN RasMan Device Tunnel Failure

Deleting a Windows 10 Always On VPN Device Tunnel

 

DirectAccess Inbox Accounting Database Optimization

DirectAccess Inbox Accounting Database OptimizationRecently I wrote about an issue with DirectAccess servers exhibiting high SQL Server CPU usage. In that article I demonstrated a way to resolve the issue by adding a crucial index to a table in the remote access inbox accounting database. The process was a bit involved and required downloading third-party tools to make configuration changes on the DirectAccess server.

Going forward, making these changes will now be much easier. Microsoft has published guidance for optimizing the remote access inbox accounting database using PowerShell. They’ve also provided scripts to back up the database and to confirm that optimization has been implemented.

For more information and to download the remote access inbox accounting database optimization PowerShell scripts, click here.

%d bloggers like this: