Always On VPN IKEv2 Security Vulnerabilities – January 2022

The January 2022 security updates for Microsoft Windows include several important updates that will affect Always On VPN deployments. Specifically, CVE-2022-21849 addresses a Remote Code Execution (RCE) vulnerability that should be addressed immediately. The January 2022 security update also includes updates for several IKE Denial-of-Service (DoS) vulnerabilities, in addition to privilege escalation vulnerabilities in the Remote Access Connection Manager.

Update – January 13, 2022: There have been numerous reports of this update breaking VPN functionality when using non-Microsoft VPN devices. If you are using Windows Server and RRAS you can safely update. If you are using a third-party device, you may encounter problems. In addition, there have been reports of issues with domain controllers and Hyper-V servers after installing this update. Please proceed carefully and be sure to have a backup before updating!

Vulnerable Systems

These vulnerabilities are present on both Windows Server and Client operating systems. Essentially, any Windows server or client using IPsec is vulnerable and potentially exploitable.

Vulnerabilities

The following is a list of security updates related to Always On VPN deployments.

Windows IKE Extension Remote Code Execution (RCE) Vulnerability

Windows IKE Extension Denial of Service Vulnerabilities

Windows Remote Access Connection Manager Elevation of Privilege Vulnerability

Additional Information

A list of all fixes in the January 2022 security update, along with links to the updates themselves, can be found here.

Always On VPN Error 13806

Troubleshooting Always On VPN Error 691 and 812 – Part 2

As a follow-up to my last post regarding Always On VPN error 13801, this post will cover a similar and related error administrators may encounter, the 13806 error. As mentioned previously, certificate configuration is crucial for Always On VPN deployments. I described some specific certificates requirements for IKEv2 in this earlier post. Following this guidance, administrators should have no issues with IKEv2 Always On VPN connections. However, it is always possible to encounter an error if any of these certificates are missing or misconfigured.

Error 13806

Much like the error 13801 described previously, 13806 is also common. When an Always On VPN connection using IKEv2 fails, the Windows Application event log will record an event ID 20227 from the RasClient source. The error message states the following:

“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 13806”.

IKE Failed To Find Valid Machine Certificate

Error 13806 translates to ERROR_IPSEC_IKE_NO_CERT, indicating IKE failed to find a valid machine certificate. The problem can be on the device, the VPN server, or an issue with the VPN server configuration.

Device Certificate

For the device tunnel, the most obvious cause of this error is a missing device authentication certificate on the client itself. Ensure the endpoint has a valid certificate issued by the organization’s internal PKI that includes Client Authentication EKU (OID 1.3.6.1.5.5.7.3.2). The certificate must have a subject name matching the device’s FQDN. It must also be valid (not expired), trusted, and not revoked.

Certificate Chain

A 13806 error will occur if the device certificate installed on the client is not trusted or if the client does not trust the certificate installed on the VPN server. Ensure the client has all the necessary root and intermediate certification authority (CA) certificates installed in their respective certificate stores.

VPN Server Certificate

A 13806 error can also occur if the VPN server does not have a properly configured server certificate. Ensure the VPN server has a valid certificate issued by the organization’s internal PKI that includes both the Server Authentication (OID 1.3.6.1.5.5.7.3.1) and IP security IKE intermediate (OID 1.3.6.1.5.5.8.2.2) EKUs. The subject name must match the public fully qualified domain name (FQDN) used by VPN clients to connect to the VPN server (not the server’s NetBIOS name). Again, ensure the certificate is valid (not expired), trusted, not revoked, and all necessary root and intermediate CA certificates are installed in their respective certificate stores.

RRAS Configuration

Another cause of the 13806 error for the user tunnel is a misconfigured Routing and Remote Access Service (RRAS) VPN server. An error 13806 can happen if the administrator incorrectly defines a trusted root CA using Set-VpnAuthProtocol. Ensure that the root certificate thumbprint matches exactly the root CA server’s thumbprint used to issue certificates to VPN devices and the VPN server.

Get-VpnAuthProtocol

Root CA Certificate Thumbprint

Resolution

Ensure that devices and VPN servers have correctly configured certificates installed. If the root CA certificate is assigned incorrectly on the VPN server, follow the guidelines detailed here to update the configuration.

Additional Information

Microsoft Windows Always On VPN Error 13801

Microsoft Windows Always On VPN Certificate Requirements for IKEv2

Microsoft Windows Always On VPN IPsec Root Certificate Configuration Issue

Microsoft Windows Always On VPN IKEv2 Policy Mismatch Error

Microsoft Windows Always On VPN IKEv2 Security Configuration

Microsoft Windows Always On VPN IKEv2 Fragmentation

Microsoft Windows Always On VPN IKEv2 Load Balancing and NAT

Microsoft Windows Always On VPN IKEv2 Features and Limitations

Always On VPN Error 13801

Troubleshooting Always On VPN Error 691 and 812 – Part 2

Certificate configuration is crucial for Always On VPN deployments. I described some specific certificates requirements for IKEv2 in this previous post. Following this guidance, administrators should have no issues with IKEv2 Always On VPN connections. However, it is always possible to encounter an error if any of these certificates are missing or misconfigured.

Error 13801

One of the most common errors related to IKEv2 and certificates is 13801. When an Always On VPN connection using IKEv2 fails, the Windows Application event log will record an event ID 20227 from the RasClient source. The error message states the following:

“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 13801”.

IKE Authentication Credentials are Unacceptable

Error 13801 translates to ERROR_IPSEC_IKE_AUTH_FAIL, indicating an authentication failure related to IPsec. The problem can be on the device, the VPN server, or an issue with the VPN server configuration.

Certificate Chain

A 13801 error will occur if the client does not trust the certificate installed on the VPN server. Ensure the client has all the necessary root and intermediate certification authority (CA) certificates installed in their respective certificate stores.

VPN Server Certificate

A 13801 error can also occur if the VPN server does not have a properly configured server certificate. Ensure the VPN server has a valid certificate issued by the organization’s internal PKI that includes both the Server Authentication (OID 1.3.6.1.5.5.7.3.1) and IP security IKE intermediate (OID 1.3.6.1.5.5.8.2.2) EKUs. The subject name must match the public fully qualified domain name (FQDN) used by VPN clients to connect to the VPN server (not the server’s NetBIOS name). Again, ensure the certificate is valid (not expired), trusted, not revoked, and all necessary root and intermediate CA certificates are installed in their respective certificate stores.

RRAS Configuration

Another cause of the 13801 error for the device tunnel is a misconfigured Routing and Remote Access Service (RRAS) VPN server. An error 13801 can happen if the administrator incorrectly defines a trusted root CA using Set-VpnAuthProtocol. Ensure that the root certificate thumbprint matches exactly the root CA server’s thumbprint used to issue certificates to VPN devices and the VPN server.

Get-VpnAuthProtocol

Root CA Certificate Thumbprint

Resolution

Ensure that devices and VPN servers have correctly configured certificates installed. If the root CA certificate is assigned incorrectly on the VPN server, follow the guidelines detailed here to update the configuration.

Additional Information

Microsoft Windows Always On VPN Error 13806 (Coming soon)

Microsoft Windows Always On VPN Certificate Requirements for IKEv2

Microsoft Windows Always On VPN IPsec Root Certificate Configuration Issue

Microsoft Windows Always On VPN IKEv2 Policy Mismatch Error

Microsoft Windows Always On VPN IKEv2 Security Configuration

Microsoft Windows Always On VPN IKEv2 Fragmentation

Microsoft Windows Always On VPN IKEv2 Load Balancing and NAT

Microsoft Windows Always On VPN IKEv2 Features and Limitations

Always On VPN Authentication Failure with Azure Conditional Access

Always On VPN Clients Prompted for Authentication when Accessing Internal Resources

Integrating Microsoft Azure Conditional Access with Windows 10 Always On VPN has several important benefits. The most important is that it allows administrators to improve their security posture by enforcing access polices that can be dynamically applied. For example, requiring multifactor authentication (MFA) for privileged users (e.g., administrators) or sign-ins that appear to be risky, the type of device they are connecting with, the health of the endpoint, and much more.

Authentication Failure

When configuring Always On VPN to support Azure Conditional Access, administrators may expeirence a failed authentication during preliminary testing. Specifically, an event ID 20227 from the RasClient source may be encountered with the following error message.

“The user <username> dialed a connection named <connection name> which has failed. The error code returned on failure is 812.”

Looking at the event logs on the Network Policy Server (NPS) server reveals an event ID 6273 from the Microsoft Windows security auditing source with Reason Code 258 and the following Reason.

“The revocation function was unable to check revocation for the certificate.”

Root Cause

When Azure Conditional Access is configured for Always On VPN, a short-lived certificate (1 hour lifetime) is provisioned by Azure. This certificate does not include revocation information because, by design, a short-lived certificate does not need to be revoked. However, by default NPS always checks revocation when client authentication certificates are used for authentication. Since the certificate does not include this information, certificate revocation fails.

Resolution

The way to resolve this issue is to disable certificate revocation checking for Protected Extensible Authentication Protocol (PEAP) authentication requests. To do this, open an elevated PowerShell window on the NPS server and run the following command.

New-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Services\RasMan\PPP\EAP\13\’ -Name IgnoreNoRevocationCheck -PropertyType DWORD -Value 1 -Force

Once complete, restart the NPS server for the changes to take effect.

Additional Information

Windows 10 Always On VPN Network Policy Server (NPS) Load Balancing

Windows 10 Always On VPN Network Policy Server (NPS) Server 2019 Bug

Always On VPN Updates for RRAS and IKEv2

Always On VPN Updates for RRAS and IKEv2

Many users have reported connection stability issues using Windows Server 2019 Routing and Remote Access Service (RRAS) and the IKEv2 VPN protocol. Specifically, there have been reports of random disconnects for which the connection cannot be re-established for an extended period. At the same time, other VPN connections may work without issue.

KB5003703

Microsoft has identified an issue in RRAS where the RemoteAccess service enters DoS protection mode, limiting incoming IKEv2 connection attempts. They released an update on June 15 (OS Build 17763.2028) that addresses this issue. Previously, the only workaround was to restart the IKEEXT service, which was highly disruptive if performed during peak hours.

No More Files

In addition, this update includes another Always On VPN-related fix for Windows 10 1809 clients. An Always On VPN user tunnel connection may fail, with an error message stating, “There are no more files.” The problem can occur after an existing user’s certificate is automatically renewed.

Additional Information

Microsoft Update June 15, 2021 KB5003703 (OS Build 17763.2028)

Always On VPN SSTP Security Configuration

Always On VPN SSTP Security Configuration

When using Windows Server Routing and Remote Access Service (RRAS) to terminate Always On VPN client connections, administrators can leverage the Secure Socket Tunneling Protocol (SSTP) VPN protocol for client-based VPN connections. SSTP is a Microsoft proprietary VPN protocol that uses Transport Layer Security (TLS) to secure connections between the client and the VPN gateway. SSTP provides some crucial advantages over IKEv2 in terms of operational reliability. It uses the TCP port 443, the standard HTTPS port, which is universally available and ensures Always On VPN connectivity even behind highly restrictive firewalls.

TLS Certificate

When configuring SSTP, the first thing to consider is the certificate installed on the server. A certificate with an RSA key is most common, but for SSTP, provisioning a certificate with an ECDSA key is recommended for optimal security and performance. See the following two articles regarding SSTP certificate requirements and ECDSA Certificate Signing Request (CSR) creation.

Always On VPN SSL Certificate Requirements for SSTP

Always On VPN ECDSA SSL Certificate Request for SSTP

TLS Configuration

Much like IKEv2, the default TLS security settings for SSTP are less than optimal. However, SSTP can provide excellent security with some additional configuration.

TLS Protocols

There are several deprecated TLS protocols enabled by default in Windows Server. These include SSLv3.0, TLS 1.0, and TLS 1.1. They should be disabled to improve security for TLS. To do this, open an elevated PowerShell window on the VPN server and run the following commands.

New-Item -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server\’ -Force

New-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server\’ -Name Enabled -PropertyType DWORD -Value ‘0’

New-Item -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server\’ -Force

New-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server\’ -Name Enabled -PropertyType DWORD -Value ‘0’

New-Item -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Server\’ -Force

New-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Server\’ -Name Enabled -PropertyType DWORD -Value ‘0’

Cipher Suites

Many weak TLS cipher suites and enabled by default in Windows Server. To further enhance security and performance, they can be optimized using a tool such as IIS Crypto. For example, consider prioritizing cipher suites that use ECDHE and GCM with ECDSA to improve security. Also, remove ciphers that use AES-256 to enhance scalability and performance.

Note: AES-256 does not provide any additional practical security over AES-128. Details here.

PowerShell Script

I have published a PowerShell script on GitHub that performs security hardening and TLS cipher suite optimization to streamline the configuration TLS on Windows Server RRAS servers. You can download the script here.

Validation Testing

After running the script and restarting the server, visit the SSL Labs Server Test site to validate the configuration. You should receive an “A” rating, as shown here.

Note: An “A” rating is not achievable on Windows Server 2012 or Windows Server 2012 R2 when using an RSA TLS certificate. A TLS certificate using ECDSA is required to receive an “A” rating on these platforms.

Additional Information

Always On VPN SSL/TLS Certificate Requirements for SSTP

Always On VPN ECDSA SSL Certificate Request for SSTP

Qualys SSL Labs Server Test Site

Always On VPN Protocol Recommendations for Windows Server RRAS

Microsoft SSTP Specification on MSDN

Always On VPN Windows Server RRAS Service Does Not Start

Always On VPN Windows Server RRAS Service Does Not StartAdministrators configuring a Windows Server Routing and Remote Access Service (RRAS) server to support Windows 10 Always On VPN connections may encounter an issue where the RemoteAccess service fails to start. Attempts to start the service might seem to work at first, but the service immediately stops again.

Troubleshooting

On the RRAS server, the Services management console (services.msc) or PowerShell Get-Service command shows the RemoteAccess service as being stopped. Attempts to start the service result in failure.

Always On VPN Windows Server RRAS Service Does Not Start

Event Log

Looking at the System event log on the RRAS server shows an error with event ID 7024 from the Service Control Manager source indicating “The Routing and Remote Access service terminated with the following service-specific error: A device attached to the system is not functioning.

Always On VPN Windows Server RRAS Service Does Not Start

Resolution

This issue is commonly caused when IPv6 is disabled on the server via the registry. To verify, open the registry editor on the RRAS server and navigate to the following location.

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters

If the DisabledComponents value is present and set to anything other than 0, set it to 0 or simply delete the DisabledComponents value completely and reboot the server.

Always On VPN Windows Server RRAS Service Does Not Start

The following PowerShell command can be used to remove the DisabledComponents value.

Remove-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters -Name DisabledComponents

Additional Information

IPv6 Recommended Reading for Always On VPN and DirectAccess Administrators

Guidance for Configuring IPv6 in Windows for Advanced Users (Microsoft)

Always On VPN IPsec Root Certificate Configuration Issue

Always On VPN Device Tunnel Status IndicatorWhen configuring a Windows Routing and Remote Access Service (RRAS) server to support Internet Key Exchange version 2 (IKEv2) VPN connections, it is essential for the administrator to define the root certification authority for which to accept IPsec security associations (SAs). Without defining this setting, the VPN server will accept a device certificate issued by any root certification authority defined in the Trusted Root Certification Authorities store. Details about configuring IKEv2 security and defining the root certification authority can be found here.

Multiple Root Certificates

Administrators may find that when they try to define a specific root certification authority, the setting may not be implemented as expected. This commonly occurs when there is more than one root certificate in the Trusted Root Certification Authorities store for the same PKI.

Always On VPN IPsec Root Certificate Configuration Issue

Certificate Selection

When running the PowerShell command Set-VpnAuthProtocol to define the root certification authority, PowerShell may ignore the administrator-defined certificate and choose a different one, as shown here. This will result in failed IPsec VPN connections from Windows 10 Always On VPN clients using IKEv2.

Always On VPN IPsec Root Certificate Configuration Issue

Certificate Publishing

This issue can occur when root certification authority certificates are published using Active Directory group policy. It appears that Windows prefers Active Directory group policy published certificates over those published directly in the Certification Authorities Container in Active Directory. To resolve this issue, remove any group policy objects that are publishing root certification authority certificates and ensure those root certificates are published in the Certification Authorities container in Active Directory.

PowerShell Script

A PowerShell script to configure this setting that can be found in my Always On VPN GitHub repository here. I have updated this script to validate the defined root certification authority certificate and warn the user if it does not match.

Additional Information

Set-Ikev2VpnRootCertificate.ps1 PowerShell script on GitHub

Windows 10 Always On VPN IKEv2 Security Configuration

Windows 10 Always On VPN IKEv2 Load Balancing and NAT

Windows 10 Always On VPN IKEv2 Features and Limitations

Windows 10 Always On VPN IKEv2 Fragmentation

Windows 10 Always On VPN IKEv2 Certificate Requirements

Always On VPN SSTP Certificate Binding Error

Always On VPN SSTP Certificate Binding ErrorWhen configuring a Windows Server with the Routing and Remote Access Service (RRAS) role to support Windows 10 Always On VPN connections, the administrator may encounter the following error message when installing or updating the TLS certificate used for Secure Socket Tunneling Protocol (SSTP) connections.

“The thumbprint (cert hash) of the certificate used for Secure Socket Tunneling Protocol (SSTP) is different than the certificate bound to the Web listener (HTTP.sys). Configure SSTP to use the default certificate or the certificate bound to SSL. You can configure web server applications to use the same certificate used by SSTP.”

Always On VPN SSTP Certificate Binding Error

IIS Binding

Most commonly this error can occur if an administrator mistakenly binds a TLS certificate directly in IIS. To resolve this problem, open the IIS management console (inetmgr.exe), navigate to the Default Web Site and click Bindings in the Actions section. Highlight the HTTPS binding and click Remove. Once complete, open an elevated command window and run the iisreset.exe command.

Always On VPN SSTP Certificate Binding Error

Netsh

In some instances, the administrator may find no certificate bindings in the IIS management console. However, a certificate binding may still be present. To confirm, open an elevated command window and run the following command.

netsh.exe http show sslcert

Always On VPN SSTP Certificate Binding Error

Remove existing certificate binding by running the following commands.

netsh.exe http delete sslcert ipport=0.0.0.0:443
netsh.exe http delete sslcert ipport=[::]:443

SSTP Configuration

When configuring SSTP in RRAS for Always On VPN, certificate assignment should always be performed using the Routing and Remote Access management console (rrasmgmt.msc). No changes are required to be made in the IIS management console for SSTP.

Additional Information

Windows 10 Always On VPN SSL Certificate Requirements for SSTP

Windows 10 Always On VPN SSTP Load Balancing with Citrix NetScaler ADC Load Balancer

Windows 10 Always On VPN SSTP Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN SSTP Load Balancing with F5 BIG-IP Load Balancer

Always On VPN IKEv2 Load Balancing and NAT

Always On VPN IKEv2 Load Balancing and NATOver the last few weeks, I’ve worked with numerous organizations and individuals troubleshooting connectivity and performance issues associated with Windows 10 Always On VPN, and specifically connections using the Internet Key Exchange version 2 (IKEv2) VPN protocol. An issue that appears with some regularity is when Windows 10 clients fail to connect with error 809. In this scenario, the server will accept connections without issue for a period of time and then suddenly stop accepting requests. When this happens, existing connections continue to work without issue in most cases. Frequently this occurs with Windows Server Routing and Remote Access Service (RRAS) servers configured in a clustered array behind an External Load Balancer (ELB).

Network Address Translation

It is not uncommon to use Network Address Translation (NAT) when configuring Always On VPN. In fact, for most deployments the public IP address for the VPN server resides not on the VPN server, but on an edge firewall or load balancer connected directly to the Internet. The firewall/load balancer is then configured to translate the destination address to the private IP address assigned to the VPN server in the perimeter/DMZ or the internal network. This is known a Destination NAT (DNAT). Using this configuration, the client’s original source IP address is left intact. This configuration presents no issues for Always On VPN.

Source Address Translation

When troubleshooting these issues, the common denominator seems to be the use of Full NAT, which includes translating the source address in addition to the destination. This results in VPN client requests arriving at the VPN server as appearing not to come from the client’s original IP address, but the IP address of the network device (firewall or load balancer) that is translating the request. Full NAT may be explicitly configured by an administrator, or in the case of many load balancers, configured implicitly because the load balancer is effectively proxying the connection.

Known Issues

IKEv2 VPN connections use IPsec for encryption, and by default, Windows limits the number of IPsec Security Associations (SAs) coming from a single IP address. When a NAT device is performing destination/full NAT, the VPN server sees all inbound IKEv2 VPN requests as coming from the same IP address. When this happens, clients connecting using IKEv2 may fail to connect, most commonly when the server is under moderate to heavy load.

Resolution

The way to resolve this issue is to ensure that any load balancers or NAT devices are not translating the source address but are performing destination NAT only. The following is configuration guidance for F5, Citrix ADC (formerly NetScaler), and Kemp load balancers.

F5

On the F5 BIG-IP load balancer, navigate to the Properties > Configuration page of the IKEv2 UDP 500 virtual server and choose None from the Source Address Translation drop-down list. Repeat this step for the IKEv2 UDP 4500 virtual server.

Always On VPN IKEv2 Load Balancing and NAT

Citrix ADC

On the Citrix ADC load balancer, navigate to System > Settings > Configure Modes and check the option to Use Subnet IP.

Always On VPN IKEv2 Load Balancing and NAT

Next, navigate to Traffic Management > Load Balancing > Service Groups and select the IKEv2 UDP 500 service group. In the Settings section click edit and select Use Client IP. Repeat these steps for the IKEv2 UDP 4500 service group.

Always On VPN IKEv2 Load Balancing and NAT

Kemp

On the Kemp LoadMaster load balancer, navigate to Virtual Services > View/Modify Services and click Modify on the IKEv2 UDP 500 virtual service. Expand Standard Options and select Transparency. Repeat this step for the IKEv2 UDP 4500 virtual service.

Always On VPN IKEv2 Load Balancing and NAT

Caveat

Making the changes above may introduce routing issues in your environment. When configuring these settings, it may be necessary to configure the VPN server’s default gateway to use the load balancer to ensure proper routing. If this is not possible, consider implementing the workaround below.

Workaround

To fully resolve this issue the above changes should be made to ensure the VPN server can see the client’s original source IP address. If that’s not possible for any reason, the following registry key can be configured to increase the number of established SAs from a single IP address. Be advised this is only a partial workaround and may not fully eliminate failed IKEv2 connections. There are other settings in Windows that can prevent multiple connections from a single IP address which are not adjustable at this time.

To implement this registry change, open an elevated PowerShell command window on the RRAS server and run the following commands. Repeat these commands on all RRAS servers in the organization.

New-ItemProperty -Path ‘HKLM:SYSTEM\CurrentControlSet\Services\IKEEXT\Parameters\’ -Name IkeNumEstablishedForInitialQuery -PropertyType DWORD -Value 50000 -Force

Restart-Service IKEEXT -Force -PassThru

Additional Information

IPsec Traffic May Be Blocked When A Computer is Behind a Load Balancer

Windows 10 Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Windows 10 Always On VPN IKEv2 Load Balancing with F5 BIG-IP

Windows 10 Always On VPN IKEv2 Load Balancing with Kemp LoadMaster

%d bloggers like this: