The January 2022 security updates for Microsoft Windows include several important updates that will affect Always On VPN deployments. Specifically, CVE-2022-21849 addresses a Remote Code Execution (RCE) vulnerability that should be addressed immediately. The January 2022 security update also includes updates for several IKE Denial-of-Service (DoS) vulnerabilities, in addition to privilege escalation vulnerabilities in the Remote Access Connection Manager.
Update – January 13, 2022: There have been numerous reports of this update breaking VPN functionality when using non-Microsoft VPN devices. If you are using Windows Server and RRAS you can safely update. If you are using a third-party device, you may encounter problems. In addition, there have been reports of issues with domain controllers and Hyper-V servers after installing this update. Please proceed carefully and be sure to have a backup before updating!
Vulnerable Systems
These vulnerabilities are present on both Windows Server and Client operating systems. Essentially, any Windows server or client using IPsec is vulnerable and potentially exploitable.
Vulnerabilities
The following is a list of security updates related to Always On VPN deployments.
Windows IKE Extension Remote Code Execution (RCE) Vulnerability
Certificate configuration is crucial for Always On VPN deployments. I described some specific certificates requirements for IKEv2 in this previous post. Following this guidance, administrators should have no issues with IKEv2 Always On VPN connections. However, it is always possible to encounter an error if any of these certificates are missing or misconfigured.
Error 13801
One of the most common errors related to IKEv2 and certificates is 13801. When an Always On VPN connection using IKEv2 fails, the Windows Application event log will record an event ID 20227 from the RasClient source. The error message states the following:
“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 13801”.
IKE Authentication Credentials are Unacceptable
Error 13801 translates to ERROR_IPSEC_IKE_AUTH_FAIL, indicating an authentication failure related to IPsec. The problem can be on the device, the VPN server, or an issue with the VPN server configuration.
Certificate Chain
A 13801 error will occur if the client does not trust the certificate installed on the VPN server. Ensure the client has all the necessary root and intermediate certification authority (CA) certificates installed in their respective certificate stores.
VPN Server Certificate
A 13801 error can also occur if the VPN server does not have a properly configured server certificate. Ensure the VPN server has a valid certificate issued by the organization’s internal PKI that includes both the Server Authentication (OID 1.3.6.1.5.5.7.3.1) and IP security IKE intermediate (OID 1.3.6.1.5.5.8.2.2) EKUs. The subject name must match the public fully qualified domain name (FQDN) used by VPN clients to connect to the VPN server (not the server’s NetBIOS name). Again, ensure the certificate is valid (not expired), trusted, not revoked, and all necessary root and intermediate CA certificates are installed in their respective certificate stores.
RRAS Configuration
Another cause of the 13801 error for the device tunnel is a misconfigured Routing and Remote Access Service (RRAS) VPN server. An error 13801 can happen if the administrator incorrectly defines a trusted root CA using Set-VpnAuthProtocol. Ensure that the root certificate thumbprint matches exactly the root CA server’s thumbprint used to issue certificates to VPN devices and the VPN server.
Get-VpnAuthProtocol
Root CA Certificate Thumbprint
Resolution
Ensure that devices and VPN servers have correctly configured certificates installed. If the root CA certificate is assigned incorrectly on the VPN server, follow the guidelines detailed here to update the configuration.
Additional Information
Microsoft Windows Always On VPN Error 13806 (Coming soon)
When configuring and deploying Windows Always On VPN using Microsoft Endpoint Manager (MEM)/Intune, administrators may find that some settings are not exposed in the MEM UI. In some cases, deploying the configuration profile using custom XML is the workaround. However, many crucial Always On VPN settings are not exposed using either method. Here, administrators must resort to editing settings in the VPN configuration file on the client after provisioning the VPN profile.
Phonebook
A file called rasphone.pbk stores all Windows VPN settings on the endpoint. It includes name/value pairs that correspond to many settings administrators change manually in the GUI. Other settings can be changed using PowerShell. Depending on the connection type, the file can be found in one of two locations.
User Tunnel: $env:AppData\Microsoft\Network\Connections\Pbk\rasphone.pbk
Documentation for Windows VPN client phonebook entry settings can be found here.
Limitations
Unfortunately, editing the rasphone.pbk file isn’t always convenient. Making the changes is technically easy. Administrators can write a simple PowerShell script to update the text file as required. However, automating this at scale is challenging. Thankfully, Intune Proactive Remediations can help.
Proactive Remediations
With Intune Proactive Remediations, administrators can create and deploy script packages to monitor and optionally update specific configuration settings. The package includes two scripts, a detection script, and a remediation script. The detection script looks at the current value of a particular setting and reports on its compliance. The remediation script is triggered to update the setting if the value is incorrect.
Requirements
Intune Proactive Remediations has some specific licensing requirements. Administrators must also enroll devices into Endpoint analytics and provision a Windows Health Monitoring configuration profile. There are also limitations on the size and type of scripts that administrators can use. More information on prerequisites can be found here.
Script Packages
Administrators can create detection and remediation PowerShell scripts to update settings in rasphone.pbk, or optionally, they can download sample scripts from my GitHub repository here. This repository contains user and device tunnel detection and remediation scripts for many popular settings in rasphone.pbk. Examples include updating the VPN Strategy, changing VPN interface metrics, disabling class-based default routes, and many more.
Note: The scripts in my GitHub repository are examples only. While they can be used in production environments, they are basic and may not work as expected in all scenarios. For example, the scripts as written today assume only a single VPN profile provisioned. Unexpected results may occur if more than one VPN profile exists. Please use them at your own risk.
Deployment
In this example, we’ll deploy a Proactive Remediation to disable IKE mobility for user tunnel VPN connections. To configure an Intune Proactive Remediation, open the Microsoft Endpoint Manager portal (https://endpoint.microsoft.com/) and navigate to Reports > Endpoint analytics > Proactive remediations. After creating or downloading the detection and remediation scripts, perform the following steps to create and deploy a Proactive Remediation script package.
Click Create script package.
Enter a name for the package in the Name field.
Enter a description for the package in the Description field (optional).
Click Next.
Click the blue folder icon next to the Detection script file field and upload the detection script.
Click the blue folder icon next to the Remediation script file field and upload the associated remediation script.
For user tunnel connections, click Yes next to Run this script using the logged-on credentials. For device tunnel connections, click No.
Click Next.
Assign scope tags and group assignments as necessary, then click Create. Click Refresh to update the UI to display the newly created script package.
Caveats
Be advised that Proactive Remediation script packages run immediately after the first device sync and then every 24 hours after that. Timing issues could lead to delays in functionality. For example, if an Always On VPN profile is provisioned after a Proactive Remediation script runs, the changes made by the remediation script won’t be available until much later. Also, changes made while the VPN is active will not take effect until after restarting the connection.
Special Thanks
Special thanks to Tom Klaver at Inspark for turning me on to this feature. It has been an absolute lifesaver for sure!
Deploying user or device authentication certificates to support Always On VPN requires installing the Certificate Connector for Microsoft Intune. The same connector can link Intune to on-premises public key infrastructure (PKI) using PKCS or SCEP certificates. The connector can be configured to run in the SYSTEM context or a domain service account.
Configuration Failure
Administrators may encounter the following error message when installing the certificate connector and selecting the option to use a domain service account.
“Configuration failed. Configuring Microsoft Intune Certificate Connector failed. No changes were made to Feature or Proxy settings. Please try again.”
Root Cause
This error occurs because the service account does not have the correct permissions assigned on the server where the connector is being installed. Specifically, the service account must have the Logon as a service right assigned. To do this, open the local group policy editor (gpedit.msc) and perform the following steps.
Expand Computer Configuration > Windows Settings > Security Settings > Local Policies > User Rights Assignment.
Double-click Log on as a service.
Click Add User or Group.
Add the service account.
Click OK.
Once complete, remove the Certificate Connector for Intune and re-run the installation again.
Since the introduction of Windows 11, there have been numerous reports of issues with Always On VPN when deployed using Microsoft Endpoint Manager/Intune. Specifically, administrators have been reporting that Always On VPN profiles are being deleted, then later reappearing. Obviously, this is highly disruptive to users in the field.
Causes
According to Microsoft, there are several causes for deleted VPN profiles.
Changes to an Existing Profile
Missing Always On VPN profiles commonly occurs when updating settings for an existing VPN profile applied to Windows 11 endpoints. In this scenario, the VPN profile is deleted but not immediately replaced. Synchronize the device with Microsoft Endpoint Manager/Intune once more to return the VPN profile.
Multiple Profiles
Issues with Always On VPN profiles may also occur if two new VPN profiles are applied to the endpoint simultaneously.
Remove and Replace
Removing and replacing an Always On VPN profile at the same time will also result in connectivity issues.
There is no known workaround for these issues at this time. Microsoft is aware of the problem and is working on a fix, and until then, rolling out Windows 11 with Always On VPN should be avoided.
Additional Issues
There have been reports of other known issues with Windows 11 and Always On VPN. For instance, my PowerShell script that removes an Always On VPN connection doesn’t work with Windows 11. I’m working to resolve that issue as we speak.
Are you experiencing any issues with Always On VPN on Windows 11? Please share them in the comments below!
Great news! My new book, Implementing Always On VPN, is now available for pre-order on Amazon.com. This new book, scheduled for release in late 2021, is a comprehensive implementation guide for Windows 10 Always On VPN. Drawing on many years of experience deploying Always On VPN for organizations worldwide, it covers all aspects of an Always On VPN deployment, including planning and design, prerequisite gathering, infrastructure preparation, and client deployment.
In addition, it contains detailed, prescriptive guidance for advanced configuration options such as application and traffic filtering and proxy server configuration. Cloud deployments using Azure VPN gateway and Virtual WAN are covered, and it includes guidance for configuring Azure MFA and Conditional Access.
Also, the book includes thorough guidance for provisioning certificates using Microsoft Endpoint Manager/Intune using both PKCS and SCEP. It outlines options for high availability for VPN and authentication infrastructure and provides details for ongoing system maintenance and operational support.
Finally, the book has an entire chapter dedicated to troubleshooting and resolving common (and not so common!) issues encountered with Windows 10 Always On VPN.
Using Microsoft Endpoint Manager (Intune), administrators can provision Always On VPN to devices that are Azure AD joined only. Users accessing on-premises resources from these devices can still use seamless single sign-on, making this deployment option popular for organizations moving to the cloud.
Short Names
After deploying Always On VPN to Windows 10 devices that are Azure AD joined only and configured to use client certificate authentication, administrators may find that users cannot access on-premises resources by their short name, such as \\app1. The connection fails and returns the following error message.
“Windows can’t find <servername/sharename>. Check the spelling and try again.”
FQDN
Interestingly, on-premises resources are accessible using their fully qualified domain name (FQDN), such as \\app1.corp.example.net.
Troubleshooting
Testing name resolution using the short name works as expected, and the resource is reachable at the network layer, as shown here.
Workaround
This issue is related to how Windows performs authentication when connected via VPN. To resolve this issue, edit the rasphone.pbk file and change the value of UseRasCredentials to 0. Rasphone.pbk can be found in the $env:AppData\Microsoft\Network\Connections\Pbk folder.
After updating this setting, restart the VPN connection for the change to take effect.
Proactive Remediations
While helpful for testing, editing rasphone.pbk manually obviously does not scale well. To address this, consider using Intune Proactive Remediations. Intune Proactive Remediations allows administrators to deploy detection and remediation PowerShell scripts to monitor specific settings and update them if or when they change. Proactive Remediations will ensure the setting is applied consistently across all managed endpoints.
GitHub Repository
I have created a new GitHub repository dedicated to PowerShell scripts for Endpoint Manager Proactive Remediations for Always On VPN. There you will find detection and remediation scripts for the UseRasCredentials settings change described in this article.
Using Windows Server Network Policy Server (NPS) servers is a common choice for authenticating Microsoft Windows 10 Always On VPN user tunnel connections. The NPS server is joined to the domain and configured with a Network Policy that defines the authentication scheme used by clients for authentication when establishing an Always On VPN connection. Protected Extensible Authentication Protocol (PEAP) using client authentication certificates recommended for most Always On VPN deployment scenarios.
Can’t Connect
Users establishing an Always On VPN user tunnel connection using PEAP and client authentication certificates may encounter a scenario in which a VPN connection attempt fails with the following error message.
“The remote access connection completed, but authentication failed because the certificate that authenticates the client to the server is not valid. Ensure that the certificate used for authentication is valid.”
Error 853
In addition, the Application event log records an event ID 20227 from the RasClient source that includes the following error message.
“The user <username> dialed a connection named <connection name> which has failed. The error code is 853.”
Missing NTAuth Certificate
Error code 853 is commonly caused by a missing issuing Certification Authority (CA) certificate in the NTAuth store on the NPS server. The NPS server must have the issuing CA certificate included in this store to perform authentication using client certificates. You can see the contents of the NTAuth certificate store by opening an elevated command window on the NPS server and running the following command.
certutil.exe -enterprise -viewstore NTAuth
Install Certificate
To install the issuing CA server’s certificate into the NTAuth store, copy the CA certificate to the NPS server, open an elevated command window, then run the following command.
certutil.exe -enterprise -addstore NTAuth <issuing CA certificate>
Once complete, view the store again, and you’ll see the issuing CA certificate listed in the NTAuth certificate store.
Using Traffic Filters with Always On VPN provides administrators the option to configure a true Zero Trust Network Access (ZTNA) solution for their field-based users and devices. By enabling traffic filtering, network access over the Always On VPN connection can be controlled using fine-grained policies. Traffic Filter rules can be configured to restrict access based source and destination IP addresses, protocols, and source and destination ports. Administrators can further restrict access based on the application generating the traffic.
IPv6
While testing these features recently, I learned that the Microsoft Endpoint Manager (formerly Intune) user interface does not appear to support IPv6 when configuring traffic filter rules. As you can see here, the UI explicitly asks for an IPv4 address and complains when entering an IPv6 address in the address field, as shown here.
Interestingly, it is possible to add IPv6 addresses in XML, as follows.
Unfortunately, after loading the XML on a test client, the Always On VPN connection fails with the following error message.
“Can’t connect to <ConnectionName>. Catastrophic failure.”
In addition, the Application event log records an event ID 20227 from the RasClient source with the following error.
“The user <UserName> dialed a connection name <ConnectionName> which has failed. The error code returned on failure is -2147418113.”
Workaround
At this time, the only known workaround is to update the configuration on the RRAS server to use IPv4 addressing for VPN clients.
Summary
Unfortunately, IPv6 is still a second-class citizen when it comes to Always On VPN. Although enabling IPv6 works well in most common deployment scenarios, the Microsoft Endpoint Manager management console often fails to accept IPv6 entries in IP address fields. In addition, some advanced features such as traffic filtering are incompatible with IPv6.
Web proxy servers are not as common today as they once were, but a few organizations still leverage them to provide secure Internet access for their employees. Commonly they are used to inspect and control Internet traffic and to enforce acceptable use policies. Some organizations may wish to extend this protection to Always On VPN clients in the field by enabling force tunneling. Administrators can define a web proxy server for Always On VPN connections globally for web traffic or individual websites or domains.
VPN Proxy
A VPN web proxy server can be defined when the Always On VPN user tunnel connection uses force tunneling. Although you can still configure a VPN web proxy server with split tunneling enabled, it will not work. It is only functional when force tunneling is in use.
Administrators can configure a VPN web proxy server using the Microsoft Endpoint Manager UI or custom XML deployed with Endpoint Manager or PowerShell. Administrators can define a VPN web proxy server explicitly, or a proxy automatic configuration (PAC) file can be specified.
Note: VPN proxy server settings only work when force tunneling is enabled. Force tunneling is an unsupported configuration for the device tunnel, making the global proxy server setting for the device tunnel unsupported.
Proxy Autoconfiguration
Perform the following steps to configure a VPN web proxy server with a PAC file in Endpoint Manager.
Expand the Proxy section in the Configuration settings of the Always On VPN configuration profile.
Enter the URL for the PAC file in the Automatic configuration script field. Be sure to include the port number in the URL when using a non-standard port.
Leave the Address and Port number fields blank.
Choose Enable or Disable from the Bypass proxy for local addresses drop-down list.
To configure a VPN web proxy server with a PAC file using custom XML, include the following code between the <VPNProfile> and </VPNProfile> tags in the Always On VPN XML configuration file.
Explicit Proxy
Perform the following steps to configure an explicit VPN web proxy server in Endpoint Manager.
Enter the IP address, hostname, or fully qualified domain name (recommended) in the Address field.
Enter the port number in the Port number field.
Choose Enable or Disable from the Bypass proxy for local addresses drop-down list.
To configure an explicit VPN web proxy server using custom XML, include the following code between the <VPNProfile> and </VPNProfile> tags in the Always On VPN XML configuration file.
Namespace Proxy
Administrators can also define VPN web proxy servers on a per-namespace or per-hostname basis. Namespace VPN proxy servers can be helpful for scenarios where routing public websites over the Always On VPN connection is required. Most commonly, this is necessary because the public website restricts access to the IP address of the on-premises Internet gateway.
A namespace VPN proxy server is implemented using a Name Resolution Policy Table (NRPT) rule. At the time of this writing, a bug in Microsoft Endpoint Manager prevents administrators from deploying this option using the UI.
As you can see here, administrators can specify a proxy server as part of an NRPT rule in the Endpoint Manager UI. Notice this section of the UI validates the proxy FQDN correctly.
However, when you try to save the configuration profile, Endpoint Manager returns the following error.
“Unable to save due to invalid data. Update your data then try again: ProxyServerUri must be a valid URL or be empty.”
Interestingly, when entering a URL such as http://proxy.lab.richardhicks.net:8080/ in the Proxy field, the Endpoint Manager UI accepts it and successfully validates. But according to the VPNv2 Configuration Service Provider (CSP) reference, the value must be entered as an IP address. A hostname or FQDN also works based on my testing. Entering a URL as shown in the example above will not work at all.
With that, the only way to implement a namespace VPN web proxy server is to use custom XML. To do this, include the following code between the <VPNProfile> and </VPNProfile> tags in the Always On VPN XML configuration file.
Include the leading “.” to specify the entire domain, as shown above. Omit the leading “.” to specify an individual host (for example, app.richardhicks.com). Repeat this section for each additional host or domain, as required.
Caveat
Unfortunately, the Microsoft Internet Explorer web browser is the only browser that functions with the namespace VPN web proxy server. All modern web browsers, including Microsoft Edge, ignore the namespace proxy setting entirely, which seriously limits this feature’s usefulness in most organizations today.
Workaround
If routing a public website over the Always On VPN tunnel is required, adding its IP address(es) to the Always On VPN connection’s routing table is needed. However, doing this presents some unique challenges, as public websites frequently have many IP addresses, which are often dynamically changing. Also, it is common for public websites to pull content from many different domains or use Content Delivery Networks (CDNs), making the problem of identifying which IP addresses to add to the Always On VPN connection’s routing table even more challenging. Further, administrators must update the client configuration each a public website’s IP address changes, adding significant management overhead.
Summary
Routing client Internet traffic through an on-premises web proxy server for Always On VPN clients works well when force tunneling is enabled. Administrators can explicitly define a web proxy server or use a proxy automatic configuration (PAC) file. All web browsers work without issue in this scenario. Using a namespace proxy is only effective when browsing with Microsoft Internet Explorer. All modern web browsers, including Microsoft Edge, ignore namespace proxy settings.
