Always On VPN IPsec Root Certificate Configuration Issue

Always On VPN Device Tunnel Status IndicatorWhen configuring a Windows Routing and Remote Access Service (RRAS) server to support Internet Key Exchange version 2 (IKEv2) VPN connections, it is essential for the administrator to define the root certification authority for which to accept IPsec security associations (SAs). Without defining this setting, the VPN server will accept a device certificate issued by any root certification authority defined in the Trusted Root Certification Authorities store. Details about configuring IKEv2 security and defining the root certification authority can be found here.

Multiple Root Certificates

Administrators may find that when they try to define a specific root certification authority, the setting may not be implemented as expected. This commonly occurs when there is more than one root certificate in the Trusted Root Certification Authorities store for the same PKI.

Always On VPN IPsec Root Certificate Configuration Issue

Certificate Selection

When running the PowerShell command Set-VpnAuthProtocol to define the root certification authority, PowerShell may ignore the administrator-defined certificate and choose a different one, as shown here. This will result in failed IPsec VPN connections from Windows 10 Always On VPN clients using IKEv2.

Always On VPN IPsec Root Certificate Configuration Issue

Certificate Publishing

This issue can occur when root certification authority certificates are published using Active Directory group policy. It appears that Windows prefers Active Directory group policy published certificates over those published directly in the Certification Authorities Container in Active Directory. To resolve this issue, remove any group policy objects that are publishing root certification authority certificates and ensure those root certificates are published in the Certification Authorities container in Active Directory.

PowerShell Script

A PowerShell script to configure this setting that can be found in my Always On VPN GitHub repository here. I have updated this script to validate the defined root certification authority certificate and warn the user if it does not match.

Additional Information

Set-Ikev2VpnRootCertificate.ps1 PowerShell script on GitHub

Windows 10 Always On VPN IKEv2 Security Configuration

Windows 10 Always On VPN IKEv2 Load Balancing and NAT

Windows 10 Always On VPN IKEv2 Features and Limitations

Windows 10 Always On VPN IKEv2 Fragmentation

Windows 10 Always On VPN IKEv2 Certificate Requirements

Always On VPN Updates for Windows 10 2004

Always On VPN Updates for Windows 10 2004Microsoft recently made available an update for Windows 10 2004 that includes many important fixes for outstanding issues with Windows 10 Always On VPN. KB4571744 (build 19041.488) addresses many challenges faced by Always On VPN administrators today, including the following.

TPM

This update addresses an issue that prevents hash signing from working correctly using the Microsoft Platform Crypto Provider for Trusted Platform Module (TPM). This issue can occur when administrators configure Always On VPN to use Protected Extensible Authentication Protocol (PEAP) with client certificate authentication using a FortiGate security device.

Sleep/Hibernate

This update also addresses issues with Windows 10 Always On VPN failing to automatically reconnect when resuming from sleep or hibernate. I’ve written about issues with Always On VPN and sleep/hibernate in the past. This is an issue that has plagued Always On VPN since its introduction, so let’s hope this finally provides some meaningful relief from this persistent problem.

Certificate Authentication

When both the Always On VPN device tunnel and user tunnel are provisioned to a Windows 10 clients, user tunnel connections may be authenticated using the machine certificate and not EAP/PEAP. This can result in connections that are not validated as intended, and allowing a user to bypass configured NPS policies, MFA requirements, or conditional access rules. This update includes a fix for this issue, restoring proper authentication for the user tunnel when the device tunnel is also provisioned.

Device and User Tunnel Coexistence

A bug that first appeared when Windows 10 2004 was introduced prevented a device tunnel and user tunnel Always On VPN connection from being established to the same VPN server if the user tunnel used Internet Key Exchange Version 2 (IKEv2). This update restores full functionality under those conditions.

Update KB4571744

To resolve these issues with Windows 10 Always On VPN as well as others, download and install update KB4571744 today. If you are experiencing any of these issues with releases of Windows 10 prior to 2004, look for updates for those build to come later this year.

Additional Information

September 3, 2020 – KB4571744 (OS Build 19041.488) Preview

Windows 10 Always On VPN Connection Issues after Sleep or Hibernate

Windows 10 Always On VPN Bug in Windows 10 2004

Always On VPN SSTP Certificate Binding Error

Always On VPN SSTP Certificate Binding ErrorWhen configuring a Windows Server with the Routing and Remote Access Service (RRAS) role to support Windows 10 Always On VPN connections, the administrator may encounter the following error message when installing or updating the TLS certificate used for Secure Socket Tunneling Protocol (SSTP) connections.

“The thumbprint (cert hash) of the certificate used for Secure Socket Tunneling Protocol (SSTP) is different than the certificate bound to the Web listener (HTTP.sys). Configure SSTP to use the default certificate or the certificate bound to SSL. You can configure web server applications to use the same certificate used by SSTP.”

Always On VPN SSTP Certificate Binding Error

IIS Binding

Most commonly this error can occur if an administrator mistakenly binds a TLS certificate directly in IIS. To resolve this problem, open the IIS management console (inetmgr.exe), navigate to the Default Web Site and click Bindings in the Actions section. Highlight the HTTPS binding and click Remove. Once complete, open an elevated command window and run the iisreset.exe command.

Always On VPN SSTP Certificate Binding Error

Netsh

In some instances, the administrator may find no certificate bindings in the IIS management console. However, a certificate binding may still be present. To confirm, open an elevated command window and run the following command.

netsh.exe http show sslcert

Always On VPN SSTP Certificate Binding Error

Remove existing certificate binding by running the following commands.

netsh.exe http delete sslcert ipport=0.0.0.0:443
netsh.exe http delete sslcert ipport=[::]:443

SSTP Configuration

When configuring SSTP in RRAS for Always On VPN, certificate assignment should always be performed using the Routing and Remote Access management console (rrasmgmt.msc). No changes are required to be made in the IIS management console for SSTP.

Additional Information

Windows 10 Always On VPN SSL Certificate Requirements for SSTP

Windows 10 Always On VPN SSTP Load Balancing with Citrix NetScaler ADC Load Balancer

Windows 10 Always On VPN SSTP Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN SSTP Load Balancing with F5 BIG-IP Load Balancer

Always On VPN Device Tunnel Status Indicator

Always On VPN Device Tunnel Status IndicatorI’ve written many articles about the Windows 10 Always On VPN device tunnel over the years. If you are not familiar with the device tunnel, it is an optional configuration that provides pre-logon connectivity for domain-joined, Enterprise edition Windows 10 clients. Although the device tunnel was designed to supplement the user tunnel connection, some administrators have deployed the device tunnel exclusively and use it for general on-premises network access. While I do not typically recommend this configuration for a variety of reasons, there are some use cases for which using the device tunnel might be acceptable.

Device Tunnel Status

For those administrators who have decided to deploy the device tunnel exclusively, a common complaint is that the device tunnel connection status does not appear in the Windows 10 notification area like other network or user tunnel connections.

Always On VPN Device Tunnel Status Indicator

However, the device tunnel does appear in the classic Network Connections control panel applet (ncpa.cpl).

Always On VPN Device Tunnel Status Indicator

Enable Device Tunnel Status Indicator

Fortunately, there is a simple workaround that allows for the device tunnel connection status to appear in the Windows 10 notification area. This can be done by setting the following registry value.

HKLM\SOFTWARE\Microsoft\Flyout\VPN\ShowDeviceTunnelInUI DWORD = 1

You can set this registry value using Active Directory group policy preferences or locally by running the following PowerShell command.

New-Item -Path ‘HKLM:\SOFTWARE\Microsoft\Flyout\VPN’ -Force
New-ItemProperty -Path ‘HKLM:\Software\Microsoft\Flyout\VPN\’ -Name ‘ShowDeviceTunnelInUI’ -PropertyType DWORD -Value 1 -Force

Once this registry value is set, the Always On VPN device tunnel will appear in the notification area long with other network connections.

Caveat

Although the UI will now display the connectivity status of the Always On VPN device tunnel, clicking Disconnect has no effect. This is expected and by design, as the device tunnel is deployed in the context of the system, not the user. Disconnecting the device tunnel must be performed by an administrator using the GUI tool rasphone.exe or the command line tool rasdial.exe.

Always On VPN Device Tunnel Status Indicator

Blog Post Comments

For the record, several readers of this blog had posted this workaround in the comments of this post. In the past. I declined to approve those comments because initially I did not want to encourage people to deploy the device tunnel standalone. However, recently I have had a change of heart, and decided to publish this information for those administrators who want to use the device tunnel exclusively, and would also benefit from a visual connectivity status indicator for the Windows 10 Always On VPN device tunnel. Although I still do not recommend using the device tunnel alone, I understand that it may be acceptable for others, so I have decided to release that information here.

Additional Information

Windows 10 Always On VPN Device Tunnel Only Deployment Considerations

Windows 10 Always On VPN Device Tunnel Operation and Best Practices

Windows 10 Always On VPN Device Tunnel with Azure VPN Gateway

Windows 10 Always On VPN Device Tunnel and Certificate Revocation

Windows 10 Always On VPN Device Tunnel Configuration using Microsoft Intune

Windows 10 Always On VPN Device Tunnel Does Not Connect Automatically

Windows 10 Always On VPN Device Tunnel Missing in Windows 10 UI

Removing Always On VPN Connections

Removing Always On VPN ConnectionsMuch has been written about provisioning Windows 10 Always On VPN client connections over the past few years. While the preferred method for deploying Always On VPN is Microsoft Intune, using PowerShell is often helpful for initial testing, and required for production deployment with System Center Configuration Manager (SCCM) or Microsoft Endpoint Manager (MEM). That said, there will invariably come a time when an administrator has to remove an Always On VPN connection. It is not as simple as you might think.

PowerShell

There are a variety of ways to remove an existing Always On VPN connection, with the quickest and simplest being PowerShell and the Remove-VpnConnection cmdlet.

Get-VpnConnection -Name ‘Always On VPN’ | Remove-VpnConnection -Force

There are several limitations to this method, however.

Active Connections

Administrators will quickly realize that PowerShell fails to remove a VPN connection that is currently connected. As shown here, attempting to remove an active VPN connection will return the following error message.

“The VPN connection [connection name] cannot be removed from the local user connections. Cannot delete a connection while it is connected.”

Removing Always On VPN Connections

Registry Artifacts

Removing Always On VPN connections using PowerShell commonly leaves behind registry artifacts that can potentially cause problems. For example, there are several Always On VPN-related registry entries in several locations including the HKLM\SOFTWARE\Microsoft\EnterpriseResourceManager\Tracked hive that may not be deleted when removing an Always On VPN connection. When provisioning a new Always On VPN connection after deleting one with the same name previously, the administrator may encounter the following error message.

“Unable to create [connection name] profile: A general error occurred that is not covered by a more specific error code.”

Removing Always On VPN Connections

Note: This error can also be caused by improperly formatted XML configuration files. More details here.

Remove-AovpnConnection Script

Veteran Always On VPN administrators are likely familiar with PowerShell scripts I’ve created called New-AovpnConneciton.ps1 and New-AovpnDeviceConnection.ps1, which are hosted on my GitHub. These scripts are adapted from code samples published by Microsoft to which I have included additional functionality. To address the limitations highlighted in this article I have published a new PowerShell script called Remove-AovpnConnection.ps1. It will remove any Always On VPN connection, even those that are currently active. It also includes logic to remove known registry artifacts common to Always On VPN. Download the script from GitHub and use the following syntax to remove an Always On VPN connection, established or not.

.\Remove-AovpnConnection.ps1 -ProfileName [connection name]

Running this PowerShell command will forcibly remove an Always On VPN connection. Use the -DeviceTunnel switch when removing a device tunnel connection (requires running in the system context). I have also included a -CleanUpOnly switch to remove registry artifacts when the VPN connection was previously removed using another method.

Updated Installation Scripts

I have also updated New-AovpnConnection.ps1 to include these registry clean up steps. This will prevent future errors when provisioning an Always On VPN client where a connection of the same name was removed previously.

Note: New-AovpnConnection.ps1 has also been updated to support device tunnel deployments. As such, I have deprecated New-AovpnDeviceConnection.ps1. Simply use New-AovpnConnection.ps1 with the -DeviceTunnel switch to deploy an Always On VPN device tunnel.

Additional Information

Windows 10 Always On VPN Device Tunnel Configuration using PowerShell

Troubleshooting Always On VPN Unable to Create Profile General Error

 

Always On VPN Connection Issues After Sleep or Hibernate

Always On VPN Connection Issues After Sleep or HibernateLikely the single most common complaint about Windows 10 Always On VPN is that device tunnel or user tunnel VPN connections fail to reconnect automatically after a laptop computer wakes from sleep or hibernate. You will find many complaining about this issue and discussing various attempts at resolution on the Microsoft forums. And while Microsoft has released many fixes the last few years to improve connection reliability for Always On VPN, this one seems to continue to plague them. This issue is also prevalent with DirectAccess deployments.

Fix or Workaround?

Unfortunately, I do not have a specific fix or workaround to share that will magically resolve this ongoing issue. However, there are a few group policy settings that may prove effective in some cases.

Connected Standby Settings

To help address issues with Always On VPN connections failing after sleep or hibernate, open the group policy management console and navigate to Computer Configuration > Administrative Templates > System > Power Management > Sleep Settings and enable the following settings.

  • Allow network connectivity during connected-standby (plugged in)
  • Allow network connectivity during connected-standby (on battery)

Always On VPN Connection Issues After Sleep or Hibernate

Always On VPN Connection Issues After Sleep or Hibernate

Additional Information

Are you experiencing issues with Always On VPN reconnecting automatically after sleep or hibernate? Have you found an effective workaround? Share your experience in the comments below!

Troubleshooting Always On VPN Error 691 and 812 – Part 3

Troubleshooting Always On VPN Error 691 and 812 – Part 2When implementing Windows 10 Always On VPN, administrators may encounter errors 691 or 812 when establishing a VPN connection. There are several different configuration issues that will result in these errors. For example they may occur when TLS 1.0 has been disabled on the RRAS server when installed on servers prior to Windows Server 2016. It can also happen if a user’s Active Directory account is configured to deny dial-in access and the NPS server is not configured to ignore user account dial-in properties. Another scenario that can result in 691/812 errors is when the Active Directory security groups are configured as conditions on the Network Policy Server (NPS) Network Policy. See below for more details.

SSTP and Error 691

When attempting to establish an Always On VPN connection using the Secure Socket Tunneling Protocol (SSTP), administrators may encounter the following error message.

“The remote connection was denied because the user name and password combination you provided is not recognized, or the selected authentication protocol is not permitted on the remote access server.”

Troubleshooting Always On VPN Error 691 and 812 – Part 2

In addition, an error 691 with event ID 20227 from the RasClient source can be found in the Application event log on the client.

“The user <domain\user> dialed a connection named which has failed. The error code returned on failure is 691.”

Troubleshooting Always On VPN Error 691 and 812 – Part 2

IKEv2 and Error 812

When attempting to establish an Always On VPN connection using Internet Key Exchange version 2 (IKEv2), administrators may encounter the following error message.

“The connection as prevented because of a policy configured on your RAS/VPN server. Specifically, the authentication method used by the server to verify your username and password may not match the authentication method configured in your connection profile. Please contact the Administrator of the RAS server and notify them of this error.”

Troubleshooting Always On VPN Error 691 and 812 – Part 2

In addition, an error 812 with event ID 20227 from the RasClient source can be found in the Application event log on the client.

Troubleshooting Always On VPN Error 691 and 812 – Part 2

NPS Event Log

On the NPS server the administrator will find an entry in the application event log with event ID 6273 from the Microsoft Windows security auditing source and the Network Policy Server task category indicating the network policy server denied access to the user. Looking closely at this event log message shows Reason Code 48 and the following reason.

“The connection request did not match any configured network policy.”

Troubleshooting Always On VPN Error 691 and 812 – Part 2Group Membership

As stated earlier, another scenario in which administrators will encounter errors 691 and/or 812 is when the Network Policy on the NPS server is configured incorrectly. Specifically, and administrator may wish to grant access to more than one group but intend for access to be granted to users who are a member of any of them. Conversely, they may wish to require access in all specified groups to gain access to the VPN. Configuring each of these conditions is subtly different, however.

Open the NPS management console on the NPS server and follow the steps below to configure user group conditions correctly for the following scenarios.

Any Group

1. Right-click the Always On VPN network policy and choose Properties.
2. Click on the Conditions tab.
3. Click the Add button.
4. Click User Groups.
5. Click Add.
6. Click Add Groups.
7. Enter the name of the group you want to grant access to.
8. Click Ok.
9. Repeat the steps 6-8 above to specify additional groups.

Troubleshooting Always On VPN Errors 691 and 812

All Groups

1. Right-click the Always On VPN network policy and choose Properties.
2. Click on the Conditions tab.
3. Click the Add button.
4. Click User Groups.
5. Click Add.
6. Click Add Groups.
7. Enter the name of the group you want to grant access to.
8. Click Ok.
9. Repeat steps 3-8 above to specify additional groups (you must go back to the Add button on step 3!).

Troubleshooting Always On VPN Errors 691 and 812

Additional Information

Troubleshooting Always On VPN Error 691 and 812 – Part 1

Troubleshooting Always On VPN Error 691 and 812 – Part 2

Always On VPN Bug in Windows 10 2004

Always On VPN Bug in Windows 10 2004While performing Always On VPN evaluation testing with the latest release of Windows 10 (2004), a bug was discovered that may result in failed VPN connections, but only under certain conditions. Specifically, the failure occurs when both the device tunnel and user tunnel are configured on the same client, and the user tunnel is configured to use IKEv2 exclusively.

Error 829

After upgrading to Windows 10 2004, and when the device tunnel and user tunnel are both deployed and the user tunnel is configured to use IKEv2, the administrator will notice that if the device tunnel connection is established, the user tunnel connects successfully but is then terminated abruptly with error code 829.

Always On VPN Bug in Windows 10 2004

Note: This can happen in reverse if the user tunnel is established before the device tunnel for some reason. In this scenario the user tunnel would be connected but attempts to establish the device tunnel would result in failure.

Error 619

If the user tunnel connection is initiated using rasdial.exe or rasphone.exe, the error code returned is 619.

Always On VPN Bug in Windows 10 2004

Always On VPN Bug in Windows 10 2004

Workaround

The workaround for this issue is to either use a single tunnel, or if both user tunnel and device tunnel are required, configure the user tunnel to use the SSTP VPN protocol instead of IKEv2.

Additional Information

Windows 10 Always On VPN Device Tunnel Only Deployment Considerations

Enterprise Networking Magazine Top 10 VPN Consulting Services 2020

It is with great pleasure that I announce Richard M. Hicks Consulting, Inc. has been included in Enterprise Networking Magazine’s Top 10 VPN consulting services for 2020! Enterprise Networking Magazine is a leading magazine and web site dedicated to the enterprise networking industry and its professionals.

Enterprise Networking Magazine Top 10 VPN Consulting Services 2020

You can read the full write-up on Richard M. Hicks Consulting, Inc. at Enterprise Networking’s web site here.

Always On VPN Split vs. Force Tunneling

Always On VPN Split vs. Force TunnelingDuring the planning phase of a Windows 10 Always On VPN implementation the administrator must decide between two tunneling options for VPN client traffic – split tunneling or force tunneling. When split tunneling is configured, only traffic for the on-premises network is routed over the VPN tunnel. Everything else is sent directly to the Internet. With force tunneling, all client traffic, including Internet traffic, is routed over the VPN tunnel. There’s been much discussion recently on this topic, and this article serves to outline the advantages and disadvantages for both tunneling methods.

Force Tunneling

Force tunneling is typically enabled to meet the following requirements.

Visibility and Control

By routing all the client’s Internet traffic over the VPN tunnel, administrators can inspect, filter, and log Internet traffic using existing on-premises security solutions such as web proxies, content filters, or Next Generation Firewalls (NGFW).

Privacy

Enabling force tunneling ensures privacy and protection of all Internet communication. By routing all Internet traffic over the VPN, administrators can be certain that all communication from the Always On VPN client is encrypted, even when clients access unencrypted web sites or use untrusted or insecure wireless networks.

Force Tunneling Drawbacks

While configuring force tunneling for Always On VPN has some advantages, it comes with some serious limitations as well.

Poor User Experience

User experience is often degraded when all Internet traffic is routed over the VPN. These suboptimal network paths increase latency, and VPN encapsulation and encryption overhead increase fragmentation, leading to reduced throughput. Most Internet traffic is already encrypted in some form, and encrypting traffic that is already encrypted makes the problem even worse. In addition, force tunneling short-circuits geographic-based Content Delivery Networks (CDNs) further reducing Internet performance. Further, location-based services are often broken which can lead to improper default language selection or inaccurate web search results.

Increased Resource Consumption

Additional resources may need to be provisioned to support force tunneling. With corporate and Internet traffic coming over the VPN, more CPU, memory, and network resources may be required. Deploying additional VPN servers and higher throughput load balancers to support the increase in network traffic may also be necessary. Force tunneling also places higher demands on Internet Service Provider (ISP) links to the corporate datacenter.

Split Tunneling

The alternative to force tunneling is “split tunneling”. With split tunneling configured, only traffic destined for the internal corporate network is routed over the VPN. All other traffic is sent directly to the Internet. Administrators define IP networks that should be routed over the VPN, and those networks are added to the routing table on the VPN client.

Security Enforcement

The challenge of providing visibility and control of Internet traffic with split tunneling enabled can be met using a variety of third-party security solutions. Microsoft Defender ATP recently introduced support for web content filtering. Also, there are numerous cloud-based security offerings from many vendors that allow administrators to monitor and control client-based Internet traffic. Zscaler and Cisco Umbrella are two popular solutions, and no doubt there are many more to choose from.

Recommendations

The general guidance I provide customers is to use split tunneling whenever possible, as it provides the best user experience and reduces demands on existing on-premises infrastructure. Enabling split or force tunneling is ultimately a design decision that must be made during the planning phase of an Always On VPN implementation project. Both configurations are supported, and they each have their merits.

In today’s world, with many applications accessible via public interfaces, force tunneling is an antiquated method for providing visibility and control for managed devices in the field. If required, investigate the use of Microsoft or other third-party solutions that enforce security policy in place without the requirement to backhaul client Internet traffic to the datacenter over VPN for inspection, logging, and filtering.

Additional Information

Whitepaper: Enhancing VPN Performance at Microsoft

Whitepaper: How Microsoft Is Keeping Its Remote Workforce Connected

Microsoft Defender ATP Web Content Filtering

%d bloggers like this: