Implementing Microsoft Windows Always On VPN, my latest video training course on Pluralsight is now available! This course provides comprehensive training for designing, implementing, and supporting a Windows Always On VPN solution. The course is nearly seven hours long (!) and covers all Always On VPN configuration aspects. The video training course covers the following Always On VPN topics.
Microsoft introduced important changes affecting certificate-based authentication on Windows domain controllers as part of the May 10, 2022 update KB5014754 that may affect Always On VPN deployments. The update addresses privilege escalation vulnerabilities when a domain controller is processing a certificate-based authentication request. The recommendation from Microsoft is that the update be applied to all Windows domain controllers and Active Directory Certificate Services (AD CS) servers as soon as possible.
After applying the update to certification authority (CA) servers, a non-critical extension with Object Identifier (OID) 1.3.6.1.4.1.311.25.2 is added to all issued certificates with the user or device security identifier (SID) included. Domain controllers with the update installed will use this information to validate the certificate used for authentication and ensure that it matches the information in Active Directory.
Domain Controllers
The update operates in Compatibility Mode, by default, when applied to domain controllers. Windows monitors authentication requests and records audit events for certificates presented for authentication under the following conditions.
No strong mapping (event ID 39) – The certificate has not been mapped explicitly to a domain account, and the certificate did not include the new SID extension.
Certificate predates account (event ID 40) – A certificate was issued before the user existed in Active Directory, and no explicit mapping could be found.
User’s SID does not match certificate (event ID 41) – A certificate contains the new SID extension, but it does not match the SID of the corresponding user account.
Certificate Mapping
Administrators can map certificates explicitly to accounts in Active Directory, but this results in a significant administrative burden in most environments. A better option is to reissue user and device authentication certificates after applying the KB5014754 update to all issuing CA servers.
Reenroll Certificates
Administrators should reissue user and device authentication certificates after applying the KB5014754 update. Open the Certificate Templates management console (certtmpl.msc), identify the user or device authentication certificate template, then right-click on the template and choose Reenroll All Certificate Holders.
Enforcement Mode
After applying update KB5014754, administrators should monitor domain controller event logs for event IDs 39, 40, and 41. Once all certificates have been updated, and none of these events have been recorded for 30 days, administrators can switch to Full Enforcement Mode by enabling it in the registry on all domain controllers.
Updated 12/8/2022: Microsoft has pushed back the original enforcement date of May 9, 2023, to November 14, 2023 “or later”. Stay tuned!
Updated 11/29/2022: Microsoft now states that the full enforcmenet mode date is now February 11, 2025 “or later”. Once again, stay tuned!
Known Issues
There have been some reports of authentication issues after installing the KB5014754 update. Early indications are that device authentication certificates missing a Subject Alternative Name (SAN) entry are to blame. Administrators are encouraged to update their device certificates to include the SAN entry. Optionally, but not recommended, administrators can place the update in disabled mode by editing the registry.
Note: An out-of-band update for these authentication issues is now available. See the reference links at the top of this article for more information.
Caveat
It’s important to understand that this new OID is added only to online templates. Online templates are those that build the subject information from Active Directory. Unfortunately, this new OID is NOT applied to offline templates (templates where the subject name is supplied in the request), such as those used for delivering certificates with Microsoft Endpoint Manager/Intune using PKCS or SCEP. It is impossible to move to enforcement mode when issuing user or device authentication certificates with Microsoft Endpoint Manager or Intune today. Microsoft is aware of this limitation and is working to address this issue as we speak. I expect a fix to be available sometime before the May 2023 deadline when Microsoft permanently switches on enforcement mode.
The two most common VPN protocols used with Always On VPN are Internet Key Exchange version 2 (IKEv2) and Secure Socket Tunneling Protocol (SSTP). Each protocol has its advantages and disadvantages. For example, IKEv2 has better security options, but SSTP is more firewall-friendly and reliable.
IKEv2 with SSTP Fallback
Instead of selecting one protocol over the other, some administrators may choose to configure Always On VPN to prefer IKEv2, then fall back to SSTP if IKEv2 is unavailable. Unfortunately, there is no way to configure this using Intune, XML, or PowerShell. To change this setting, the administrator must update the VPN configuration file (rasphone.pbk) and change the value of VpnStrategy to 14. While editing a text file is easy, doing it at scale is more complicated. The setting can be changed using Intune proactive remediation or a PowerShell script. However, it’s even easier using Always On VPN DPC. Simply enable the VPN Protocol advanced setting in group policy and choose IKEv2 First, SSTP Fallback.
Interface Metric
Another common problem Always On VPN administrators encounter is name resolution, specifically when the endpoint uses a wired local connection. Here, name resolution queries may fail or return incorrect IP addresses. This happens because the wired connection has a lower network interface metric than the VPN tunnel adapter. Once again, there is no option for changing this setting using Intune or XML. Administrators can update the interface metrics using PowerShell, but it is not persistent. To fully resolve this, the administrator must edit the rasphone.pbk file. With Always On VPN DPC, enable the VPN Tunnel Metric group policy setting and enter a value lower than the wired connection to solve this problem.
IKE Mobility
The Windows VPN client includes support for IKE Mobility, which allows an IKEv2 VPN connection to reconnect automatically after a loss of network connectivity. IKE Mobility is enabled by default, and the network outage time is set to 30 minutes. However, this setting can have negative side effects, especially when VPN servers are behind a load balancer. Reducing the network outage time or disabling it completely can improve failover if a VPN server goes offline. Here again, this setting cannot be changed using Intune, XML, or PowerShell; it is only configurable in rasphone.pbk. With Always On VPN DPC, enable the Network Outage Time advanced setting in group policy and choose a value that meets your requirements.
Exclusion Routes for Office 365
Force tunneling ensures that all network traffic on the client is routed over the VPN tunnel, including Internet traffic. However, Always On VPN supports exclusion routes which allow administrators to exempt selected traffic from the VPN tunnel when force tunneling is enabled. Commonly this is configured for trusted cloud applications like Microsoft Office 365. Defining exclusion routes for cloud services is more complicated than it sounds. Many cloud services, including Microsoft Office 365, have multiple IP addresses that are constantly changing. This makes keeping Always On VPN clients updated with the correct list of IP address exclusions quite challenging. With Always On VPN DPC, administrators can enable the Exclude Office 365 from VPN group policy setting, allowing the endpoint to automatically configure the necessary exclusion routes for Office 365 IP addresses. Importantly, Always On VPN DPC periodically monitors this list of IP addresses and ensures that endpoints are continually updated with Office 365 exclusion routes as they change to ensure reliable connectivity.
IP Routing
Always On VPN administrators must define which IP addresses and networks are routed over the VPN tunnel when split tunneling is enabled. However, Intune has a known issue that may pose a challenge in some environments.
IPv6
Although IPv4 routes can be configured using the Intune UI, IPv6 routes cannot. This is because the Intune UI does not correctly validate the default IPv6 prefix length, insisting that the administrator use a value between 1 and 32. 🤦♂️
However, the Always On VPN DPC Allowed Routes group policy setting happily accepts the proper IPv6 prefix.
Route Metrics
In addition, there is no option to define the metric values for routes configured using Intune. Assigning non-default route metrics is required to resolve routing conflicts in some scenarios. Defining route metrics requires custom XML. The Always On VPN DPC Route Metric group policy settings allow administrators to define route metrics as required.
Are you interested in learning more about PowerON Platforms Always On VPN DPC? Fill out the form below, and I’ll contact you with more information. In addition, you can visit aovpndpc.com to register for an evaluation license.
Recently I wrote about Always On VPN Dynamic Profile Configurator (DPC). This software solution enables administrators to natively provision and manage Always On VPN client configuration settings using Active Directory and group policy. In that post, I provided some high-level details about the product, along with a brief overview of its advanced features.
Demonstration Video
I have recorded a video demonstrating how to install and configure Always On VPN DPC and use its basic features. You will find that demonstration video here.
Advanced Features
Soon I will share more details about Always On VPN DPC and using its advanced capabilities to solve some common challenges faced by Always On VPN administrators. Stay tuned!
Learn More
Are you interested in learning more about PowerON Platforms Always On VPN DPC? Fill out the form below, and I’ll contact you with more information. In addition, you can visit aovpndpc.com to register for an evaluation license.
Always On VPN Administrators may encounter a scenario where Always On VPN connections suddenly stop working for all clients using the Secure Socket Tunneling Protocol (SSTP) VPN protocol. IKEv2 VPN connections continue to work, however.
Event Log
Reviewing the event log on a client machine reveals an error event ID 20227 from the RasClient source. The error message states the following.
“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is -2146762495.”
Error -2146762495?
Always On VPN administrators will be familiar with error codes such as 809, 691 and 812, 853, 858, and even 13801, 13806, and 13868. However, this error code seems to be formatted much differently. As it turns out, this message is in decimal format. Thankfully it’s pretty easy to convert it to something more meaningful, like hexadecimal. To do this, open the Windows calculator (calc.exe) and switch to programmer mode. Highlight DEC and enter -2146762495. The hexadecimal value will be displayed in the HEX field, as shown here.
Error 0x800B0101
After converting the error message from decimal to hex, use the Microsoft Error Lookup tool (err.exe) to translate the hex value of this error. As shown here, 0x800B0101 translates to CERT_E_EXPIRED.
Expired TLS Certificate
Once again, an expired certificate is to blame! In this case, the TLS certificate installed on the VPN server has expired and is no longer valid.
Resolution
The problem is simple enough to resolve, of course. Obtain a new TLS certificate from your certification authority (CA) of choice and update your VPN server configuration. You can find detailed guidance for updating the RRAS VPN server’s TLS certificate here. You will also find a video demonstration of the RRAS SSL/TLS certificate renewal process here.
On Wednesday, February 9, from 9:00 AM to 10:30 AM CST, I will be delivering a 90-minute mini-course on Always On VPN infrastructure requirements and deployment options as part of the ViaMonstra Online Academy training series. The Always On VPN mini-course is a free training session and is open to everyone. Be sure to register even if you can’t join in person. A session recording will be available to all registrants after the broadcast.
Course Outline
The live online training course will include but is not limited to the following topics.
Always On VPN infrastructure requirements
On-premises and cloud-based deployment options
Implementation and security best practices
Avoiding common mistakes when deploying Always On VPN
Provisioning and managing Always On VPN profiles using Microsoft Endpoint Manager/Intune
Understanding current limitations and workarounds
Register Today
There will also be plenty of time for questions and answers. Register now, and don’t miss out on this opportunity to learn about Always On VPN!
The January 2022 security updates for Microsoft Windows include several important updates that will affect Always On VPN deployments. Specifically, CVE-2022-21849 addresses a Remote Code Execution (RCE) vulnerability that should be addressed immediately. The January 2022 security update also includes updates for several IKE Denial-of-Service (DoS) vulnerabilities, in addition to privilege escalation vulnerabilities in the Remote Access Connection Manager.
Update – January 17, 2022: Microsoft has released out-of-band updates to address the issues with IPsec (IKEv2 and L2TP) when using non-Microsoft VPN devices. Updates can be found here.
Update – January 13, 2022: There have been numerous reports of this update breaking VPN functionality when using non-Microsoft VPN devices. If you are using Windows Server and RRAS you can safely update. If you are using a third-party device, you may encounter problems. In addition, there have been reports of issues with domain controllers and Hyper-V servers after installing this update. Please proceed carefully and be sure to have a backup before updating!
Vulnerable Systems
These vulnerabilities are present on both Windows Server and Client operating systems. Essentially, any Windows server or client using IPsec is vulnerable and potentially exploitable.
Vulnerabilities
The following is a list of security updates related to Always On VPN deployments.
Windows IKE Extension Remote Code Execution (RCE) Vulnerability
As a follow-up to my last post regarding Always On VPN error 13801, this post will cover a similar and related error administrators may encounter, the 13806 error. As mentioned previously, certificate configuration is crucial for Always On VPN deployments. I described some specific certificates requirements for IKEv2 in this earlier post. Following this guidance, administrators should have no issues with IKEv2 Always On VPN connections. However, it is always possible to encounter an error if any of these certificates are missing or misconfigured.
Error 13806
Much like the error 13801 described previously, 13806 is also common. When an Always On VPN connection using IKEv2 fails, the Windows Application event log will record an event ID 20227 from the RasClient source. The error message states the following:
“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 13806”.
IKE Failed To Find Valid Machine Certificate
Error 13806 translates to ERROR_IPSEC_IKE_NO_CERT, indicating IKE failed to find a valid machine certificate. The problem can be on the device, the VPN server, or an issue with the VPN server configuration.
Device Certificate
For the device tunnel, the most obvious cause of this error is a missing device authentication certificate on the client itself. Ensure the endpoint has a valid certificate issued by the organization’s internal PKI that includes Client Authentication EKU (OID 1.3.6.1.5.5.7.3.2). The certificate must have a subject name matching the device’s FQDN. It must also be valid (not expired), trusted, and not revoked.
Certificate Chain
A 13806 error will occur if the device certificate installed on the client is not trusted or if the client does not trust the certificate installed on the VPN server. Ensure the client has all the necessary root and intermediate certification authority (CA) certificates installed in their respective certificate stores.
VPN Server Certificate
A 13806 error can also occur if the VPN server does not have a properly configured server certificate. Ensure the VPN server has a valid certificate issued by the organization’s internal PKI that includes both the Server Authentication (OID 1.3.6.1.5.5.7.3.1) and IP security IKE intermediate (OID 1.3.6.1.5.5.8.2.2) EKUs. The subject name must match the public fully qualified domain name (FQDN) used by VPN clients to connect to the VPN server (not the server’s NetBIOS name). Again, ensure the certificate is valid (not expired), trusted, not revoked, and all necessary root and intermediate CA certificates are installed in their respective certificate stores.
Certificate Revocation
An expired Certificate Revocation List (CRL) can also result in a 13806 error. Open the Enterprise PKI console (pkiview.msc) on an issuing CA and review the status of all CRLs. If any are expired, resolve any issues preventing the CRL from publishing successfully, then issue a new CRL by running certutil.exe -crl on the issuing CA server.
RRAS Configuration
Another cause of the 13806 error for the user tunnel is a misconfigured Routing and Remote Access Service (RRAS) VPN server. An error 13806 can happen if the administrator incorrectly defines a trusted root CA using Set-VpnAuthProtocol. Ensure that the root certificate thumbprint matches exactly the root CA server’s thumbprint used to issue certificates to VPN devices and the VPN server.
Get-VpnAuthProtocol
Root CA Certificate Thumbprint
Resolution
Ensure that devices and VPN servers have correctly configured certificates installed. If the root CA certificate is assigned incorrectly on the VPN server, follow the guidelines detailed here to update the configuration.
Certificate configuration is crucial for Always On VPN deployments. I described some specific certificates requirements for IKEv2 in this previous post. Following this guidance, administrators should have no issues with IKEv2 Always On VPN connections. However, it is always possible to encounter an error if any of these certificates are missing or misconfigured.
Error 13801
One of the most common errors related to IKEv2 and certificates is 13801. When an Always On VPN connection using IKEv2 fails, the Windows Application event log will record an event ID 20227 from the RasClient source. The error message states the following:
“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 13801”.
IKE Authentication Credentials are Unacceptable
Error 13801 translates to ERROR_IPSEC_IKE_AUTH_FAIL, indicating an authentication failure related to IPsec. The problem can be on the device, the VPN server, or an issue with the VPN server configuration.
Authentication Methods
To begin, ensure that the appropriate authentication method is enabled on the Routing and Remote Access (RRAS) server. Right-click the VPN server in the RRAS management console (rrasmgmt.msc) and choose Properties. Next, click on the Security tab and then click on the Authentication Methods button.
Select the ‘Extensible authentication protocol (EAP)’ to support IKEv2 user tunnel connections. In addition, select ‘Allow machine certificate authentication for IKEv2’ to support Always On VPN device tunnel connections.
Certificate Chain
A 13801 error will occur if the client does not trust the certificate installed on the VPN server. Ensure the client has all the necessary root and intermediate certification authority (CA) certificates installed in their respective certificate stores.
VPN Server Certificate
A 13801 error can also occur if the VPN server does not have a properly configured server certificate. Ensure the VPN server has a valid certificate issued by the organization’s internal PKI that includes both the Server Authentication (OID 1.3.6.1.5.5.7.3.1) and IP security IKE intermediate (OID 1.3.6.1.5.5.8.2.2) EKUs. The subject name must match the public fully qualified domain name (FQDN) used by VPN clients to connect to the VPN server (not the server’s NetBIOS name). Again, ensure the certificate is valid (not expired), trusted, not revoked, and all necessary root and intermediate CA certificates are installed in their respective certificate stores.
Certificate Revocation
An expired Certificate Revocation List (CRL) can also result in a 13801 error. Open the Enterprise PKI console (pkiview.msc) on an issuing CA and review the status of all CRLs. If any are expired, resolve any issues preventing the CRL from publishing successfully, then issue a new CRL by running certutil.exe -crl on the issuing CA server.
RRAS Configuration
Another cause of the 13801 error for the device tunnel is a misconfigured Routing and Remote Access Service (RRAS) VPN server. An error 13801 can happen if the administrator incorrectly defines a trusted root CA using Set-VpnAuthProtocol. Ensure that the root certificate thumbprint matches exactly the root CA server’s thumbprint used to issue certificates to VPN devices and the VPN server.
Get-VpnAuthProtocol
Root CA Certificate Thumbprint
Resolution
Ensure that devices and VPN servers have correctly configured certificates installed. If the root CA certificate is assigned incorrectly on the VPN server, follow the guidelines detailed here to update the configuration.
When configuring and deploying Windows Always On VPN using Microsoft Endpoint Manager (MEM)/Intune, administrators may find that some settings are not exposed in the MEM UI. In some cases, deploying the configuration profile using custom XML is the workaround. However, many crucial Always On VPN settings are not exposed using either method. Here, administrators must resort to editing settings in the VPN configuration file on the client after provisioning the VPN profile.
Phonebook
A file called rasphone.pbk stores all Windows VPN settings on the endpoint. It includes name/value pairs that correspond to many settings administrators change manually in the GUI. Other settings can be changed using PowerShell. Depending on the connection type, the file can be found in one of two locations.
User Tunnel: $env:AppData\Microsoft\Network\Connections\Pbk\rasphone.pbk
Documentation for Windows VPN client phonebook entry settings can be found here.
Limitations
Unfortunately, editing the rasphone.pbk file isn’t always convenient. Making the changes is technically easy. Administrators can write a simple PowerShell script to update the text file as required. However, automating this at scale is challenging. Thankfully, Intune Proactive Remediations can help.
Proactive Remediations
With Intune Proactive Remediations, administrators can create and deploy script packages to monitor and optionally update specific configuration settings. The package includes two scripts, a detection script, and a remediation script. The detection script looks at the current value of a particular setting and reports on its compliance. The remediation script is triggered to update the setting if the value is incorrect.
Requirements
Intune Proactive Remediations has some specific licensing requirements. Administrators must also enroll devices into Endpoint analytics and provision a Windows Health Monitoring configuration profile. There are also limitations on the size and type of scripts that administrators can use. More information on prerequisites can be found here.
Script Packages
Administrators can create detection and remediation PowerShell scripts to update settings in rasphone.pbk, or optionally, they can download sample scripts from my GitHub repository here. This repository contains user and device tunnel detection and remediation scripts for many popular settings in rasphone.pbk. Examples include updating the VPN Strategy, changing VPN interface metrics, disabling class-based default routes, and many more.
Note: The scripts in my GitHub repository are examples only. While they can be used in production environments, they are basic and may not work as expected in all scenarios. For example, the scripts as written today assume only a single VPN profile provisioned. Unexpected results may occur if more than one VPN profile exists. Please use them at your own risk.
Deployment
In this example, we’ll deploy a Proactive Remediation to disable IKE mobility for user tunnel VPN connections. To configure an Intune Proactive Remediation, open the Microsoft Endpoint Manager portal (https://intune.microsoft.com/) and navigate to Devices > Remediations. After creating or downloading the detection and remediation scripts, perform the following steps to create and deploy a Proactive Remediation script package.
Click Create script package.
Enter a name for the package in the Name field.
Enter a description for the package in the Description field (optional).
Click Next.
Click the blue folder icon next to the Detection script file field and upload the detection script.
Click the blue folder icon next to the Remediation script file field and upload the associated remediation script.
For user tunnel connections, click Yes next to Run this script using the logged-on credentials. For device tunnel connections, click No.
Click Next.
Define scope tags as required.
Click Next.
Assign a group and choose a schedule. The detection script can be run once, daily, hourly, or on an hourly interval.
Click Next, then click Create.
Click Refresh to update the UI to display the newly created script package.
Caveats
Be advised that timing issues could lead to delays in functionality. For example, if there’s a change to an Always On VPN profile after a Proactive Remediation detection script runs, the changes will not be detected until the detection script runs again. Also, changes made while the VPN is active will not take effect until after restarting the connection.
Special Thanks
Special thanks to Tom Klaver at Inspark for turning me on to this feature. It has been an absolute lifesaver for sure!
