Always On VPN administrators may be familiar with an issue that affects Windows Server Routing and Remote Access Service (RRAS) servers, where many stale VPN connections appear in the list of active connections. The issue is most prevalent when using IKEv2, either for the Always On VPN device tunnel or the user tunnel. Typically, this does not cause problems, but some administrators have reported issues related to port exhaustion or failed IKEv2 connections when many stale connections are present. Stale connections happen so frequently that I created a PowerShell script to clean them up on the RRAS server. Restarting the RemoteAccess service or rebooting the server also clears stale connections.
Microsoft Fix
Thankfully, Microsoft has addressed these issues in Windows Server 2019 and Windows Server 2022 this month. An update is now available in the March 2023 security update that resolves this problem.
You can find more information about the updates here.
The update was not made available for Windows Server 2016, however. Organizations are encouraged to upgrade to Windows Server 2019 or later to address this problem.
Windows Server Routing and Remote Access Service (RRAS) is a popular choice for administrators deploying Always On VPN. It is easy to configure and scales out easily. Most commonly, RRAS servers are configured to use RADIUS authentication to provide user authentication for Always On VPN client connections. The RADIUS server can be Microsoft Network Policy and Access Server (NPAS, or simply NPS) or a third-party RADIUS solution. It is best to have the RADIUS service running on a server separate from the RRAS server.
RADIUS Authentication
Administrators can configure RADIUS authentication by opening the Routing and Remote Access management console (rrasmgmt.msc), right-clicking the VPN server, then choosing Properties > Security. Click the Configure button to configure authentication and accounting providers accordingly.
RADIUS Options Missing
In some cases, the administrator will notice that the option to configure RADIUS authentication and accounting servers is missing. In its place is the following error message.
“Because Network Policy Server (NPS) is installed, you must use it to configure authentication and accounting providers. To configure authentication and accounting providers, create or modify connection request policies.”
Configuration Options
It might be tempting for administrators to follow this guidance when presented with this message by opening the Network Policy management console (nps.msc) to configure it. However, that is not recommended or necessary. This message results from a common configuration error that should be corrected.
NPS Role Installed
The error message above occurs when an administrator mistakenly installs the NPAS role on the RRAS server itself. Again, this is not recommended or required. To resolve this issue, uninstall the NPS role by opening an elevated PowerShell command window and running the following command.
Uninstall-WindowsFeature NPAS
Configuration Corrupted
After removing the NPAS role from the RRAS server, administrators may encounter the following error message when configuring RADIUS authentication and accounting servers in RRAS.
“The connection request policy used for authentication and accounting configuration is corrupted. Either install Network Policy Server (NPS) and restore the connection request policy manually, or click Repair Settings to restore the connection request policy by using Windows Accounting and Windows Authentication.”
Repair Settings
To resolve this issue, click the Repair Settings button. Once complete, RADIUS authentication and accounting configuration should work as expected.
HTTP Strict Transport Security (HSTS) is a feature commonly used by websites to protect against protocol downgrade attacks, where an attacker forces the use of insecure HTTP instead of HTTPS. If successful, the attacker can intercept unencrypted communication between the client and the web server. This is undesirable for obvious reasons. As such, web server administrators implement an HTTP response header named Strict-Transport-Security with some additional settings that instruct the user agent, in this case, a web browser, to only use secure HTTPS when communicating with the web server. Attempts to use HTTP will not work.
VPN and SSTP
As security is always a top concern when building an Always On VPN infrastructure, careful attention must be paid to VPN protocol configuration to ensure optimal security. Secure Socket Tunneling Protocol (SSTP) is a popular VPN protocol for Always On VPN user tunnel connections. SSTP uses Transport Layer Security (TLS) for encryption, so administrators are encouraged to implement recommended security configurations, such as disabling insecure protocols like TLS 1.0 and TLS 1.1 and optimizing TLS cipher suites as described here.
SSTP with HSTS
It would seem that enabling HSTS on a Windows RRAS VPN server would be ideal for improving SSTP security. However, that’s not the case. HSTS prevents protocol downgrade attacks from HTTPS to HTTP, but SSTP already uses HTTPS exclusively, making the use of HSTS irrelevant. If an attacker attempted a protocol downgrade attack on an SSTP VPN connection, it would fail because the service does not support HTTP between the client and the VPN gateway. Additionally, even if it were possible to configure RRAS to send an HSTS response header, it would be ignored by the client because the user agent is not a web browser.
Always On VPN administrators troubleshooting connectivity issues may find the Internal network interface in the Routing and Remote Access management console (rrasmgmt.msc) administrative status indicates ‘Unknown’. They will also notice the Operational Status shows Non-operational.
Internal Interface
For clarification, the ‘Internal’ network interface in the Routing and Remote Access management console, as shown above, is not a physical network adapter on the server. Instead, it is a virtual network interface used only for incoming VPN connections.
Non-Operational
The Internal virtual network interface will not be created until the VPN server accepts its first VPN connection. Because of this, the Internal interface will have an operational status of non-operational until the first client attempts to connect. When this occurs, RRAS creates the interface, then assigns it the first IP address from the static IPv4 address pool. Alternatively, if DHCP is configured, it will assign the first IP address returned by the DHCP server.
Interface Names
While discussing network interfaces, I typically recommend renaming them in Windows to identify their function, especially when using two NIC configurations. However, be careful not to name the server’s internal network adapter ‘Internal’, as this can be confusing in the future. In my example above, I use the name ‘LAN’ to identify the internal adapter to distinguish it from the server’s ‘Internal’ virtual interface.
The Routing and Remote Access Service (RRAS) role in Windows Server is a popular VPN server choice for administrators deploying Windows Always On VPN. It is easy to configure, scales well, and is cost-effective. After installing RRAS, administrators can optionally enable inbox accounting to log historical data and generate user access and activity reports as described in Always On VPN RRAS Monitoring and Reporting.
Inbox Accounting Database
A Windows Internal Database (WID) is automatically installed and configured for data storage when inbox accounting is enabled.
WID is nothing more than a basic instance of Microsoft SQL Server. As such, the database will require periodic maintenance to perform optimally.
Inbox Accounting Database Management Scripts
I have created a series of PowerShell scripts to address the inbox accounting database management requirements for organizations using Windows Server RRAS. Scripts are available to perform the following inbox accounting database management tasks.
Optimize the inbox accounting database.
View the size of the inbox accounting database files.
Compress the size of the inbox accounting database.
Back up the inbox accounting database to a file on disk.
Restore the inbox accounting database from a backup file.
Move the inbox accounting database file to a different location.
Remove the inbox accounting database.
Optimize Database
A known issue with the inbox accounting database can result in high CPU and memory utilization for very busy RRAS VPN servers. Specifically, a crucial index is missing from one of the tables in the logging database. This issue persists in Windows Server 2022. To correct this issue, download and run the following PowerShell script on each RRAS VPN server in the organization.
The database can grow rapidly depending on how busy the RRAS server is. Administrators can view the current database file sizes by downloading and running the following PowerShell script on the RRAS server.
Over time, the database can become fragmented, decreasing performance. Compressing the database can improve performance and result in significant recovery of disk space. To compress the inbox accounting database, download and run the following PowerShell script on each RRAS server in the organization.
In this example, compressing the database reduced its size by more than 8MB, resulting in a nearly 70% reduction in disk space usage.
Backup Database
Administrators may wish to back up the inbox accounting database before purging older records from the inbox accounting database. Also, backing up the database preservers access records when migrating to a new server. To back up the inbox accounting database, download and run the following PowerShell script on each RRAS server in the organization.
Restoring a database from backup will erase all records in the current database. It does not append. Proceed with caution!
Move Database Files
Inbox accounting database and log files are located in C:\Windows\DirectAccess\Db by default.
However, storing database and log files on the system drive is not ideal. A better alternative is to place the inbox accounting database and log files on a separate disk for optimum performance. To move the inbox accounting database, download and run the following PowerShell script on each VPN server in the organization.
Moving inbox accounting files may not be formally supported by Microsoft. Use caution when making this change.
Remove Database
Occasionally an inbox accounting database becomes corrupt and can no longer be managed. If this happens, completely removing the database is required. It is essential to know that simply disabling and re-enabling inbox accounting on the VPN server does not delete the database. To delete the database completely, download and run the following PowerShell script.
To simplify things, the PowerShell scripts described in this article are available in a PowerShell module that can be installed from the PowerShell gallery using the following command.
Always On VPN Administrators may encounter a scenario where Always On VPN connections suddenly stop working for all clients using the Secure Socket Tunneling Protocol (SSTP) VPN protocol. IKEv2 VPN connections continue to work, however.
Event Log
Reviewing the event log on a client machine reveals an error event ID 20227 from the RasClient source. The error message states the following.
“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is -2146762495.”
Error -2146762495?
Always On VPN administrators will be familiar with error codes such as 809, 691 and 812, 853, 858, and even 13801, 13806, and 13868. However, this error code seems to be formatted much differently. As it turns out, this message is in decimal format. Thankfully it’s pretty easy to convert it to something more meaningful, like hexadecimal. To do this, open the Windows calculator (calc.exe) and switch to programmer mode. Highlight DEC and enter -2146762495. The hexadecimal value will be displayed in the HEX field, as shown here.
Error 0x800B0101
After converting the error message from decimal to hex, use the Microsoft Error Lookup tool (err.exe) to translate the hex value of this error. As shown here, 0x800B0101 translates to CERT_E_EXPIRED.
Expired TLS Certificate
Once again, an expired certificate is to blame! In this case, the TLS certificate installed on the VPN server has expired and is no longer valid.
Resolution
The problem is simple enough to resolve, of course. Obtain a new TLS certificate from your certification authority (CA) of choice and update your VPN server configuration. You can find detailed guidance for updating the RRAS VPN server’s TLS certificate here. You will also find a video demonstration of the RRAS SSL/TLS certificate renewal process here.
Windows Server Routing and Remote Access Service (RRAS) is popular for Always On VPN deployments because it supports the Secure Socket Tunneling Protocol (SSTP). The SSTP VPN protocol is recommended for use with the Always On VPN user tunnel because it is firewall friendly. Installing a TLS certificate on the VPN server is necessary to support SSTP VPN connections. Administrators should use a TLS certificate signed by a public certification authority (CA) for optimal reliability and performance.
Click here to view a video demonstration of the procedures outlined in this article.
Certificate Expiration
Of course, all certificates expire, and the TLS certificate used for SSTP is no exception. When using a public TLS certificate, the certificate lifetime is typically no more than one year, which means Always On VPN administrators will be renewing this certificate regularly.
Certificate Renewal
The process of “renewing” an SSTP TLS certificate is essentially the same as installing a new one, as it is best to create a new public/private key pair when renewing a certificate. The following outlines the steps required to generate a Certificate Signing Request (CSR), import the certificate, then assign the certificate to the SSTP listener on the VPN server.
Note: The guidance provided here assumes using an ECC certificate, which is best for optimal security and performance. More details here.
Certificate Request
Open the local computer certificate store (certlm.msc) on the VPN server and perform the following steps to generate a new CSR.
Expand Certificates – Local Computer > Personal.
Right-click the Certificates folder and choose All Tasks > Advanced Operations > Create Custom Request.
Click Next.
Highlight Proceed without enrollment policy.
Click Next.
Select (No template) CNG key from the Template drop-down list.
Select PKCS #10 in the Request format section.
Click Next.
Click on the down arrow next to Details.
Click on the Properties button.
Select the General tab.
Enter the public hostname for the certificate in the Friendly name field.
Select the Subject tab.
Select Common name from the Type drop-down list in the Subject name section.
Enter the public hostname for the certificate in the Value field.
Click Add.
In the Alternative name section, select DNS from the Type drop-down list.
Enter the public hostname for the certificate in the Value field.
Click Add.
Select the Extensions tab.
Expand the Extended Key Usage section.
Select Server Authentication from the Available options section.
Click Add.
Select the Private Key tab.
Expand the Cryptographic Service Provider section.
Uncheck the box next to RSA,Microsoft Software Key Storage Provider.
Check the box next to ECDSA_P256,Microsoft Software Key Storage Provider.
Expand the Key options section.
Check the box next to Make private key exportable.
Click Ok.
Click Next.
Enter a name for the file in the File Name field.
Select Base 64 in the File format section.
Click Finish.
Import Certificate
Once complete, submit the file created to a public CA for signing. When the CA returns the signed certificate, perform the following steps to import it to the local compute certificate store.
Right-click the Certificates folder and choose All Tasks > Import.
Click Next.
Enter the name of the certificate file returned by the public CA in the File name field.
Click Next.
Select Place all certificates in the following store and ensure that Personal is listed in the Certificate store field.
Click Next.
Click Finish.
Click Ok.
Assign Certificate
After importing the new TLS certificate in the local computer’s certificate store, open the Routing and Remote Access management console (rrasmgmt.msc) and perform the following steps to assign the TLS certificate to the SSTP listener.
Right-click the VPN server and choose Properties.
Select the Security tab.
Select the new TLS certificate from the Certificate drop-down list in the SSL Certificate Binding section. When replacing an existing certificate, you may see a certificate with the same name more than once. Click the View button and ensure the new certificate is selected.
Click Ok.
Click Yes to restart the RemoteAccess service.
Demonstration Video
A recorded video demonstration of this process can be found here. The video recording also includes guidance for making these changes on Windows Server Core servers.
The January 2022 security updates for Microsoft Windows include several important updates that will affect Always On VPN deployments. Specifically, CVE-2022-21849 addresses a Remote Code Execution (RCE) vulnerability that should be addressed immediately. The January 2022 security update also includes updates for several IKE Denial-of-Service (DoS) vulnerabilities, in addition to privilege escalation vulnerabilities in the Remote Access Connection Manager.
Update – January 17, 2022: Microsoft has released out-of-band updates to address the issues with IPsec (IKEv2 and L2TP) when using non-Microsoft VPN devices. Updates can be found here.
Update – January 13, 2022: There have been numerous reports of this update breaking VPN functionality when using non-Microsoft VPN devices. If you are using Windows Server and RRAS you can safely update. If you are using a third-party device, you may encounter problems. In addition, there have been reports of issues with domain controllers and Hyper-V servers after installing this update. Please proceed carefully and be sure to have a backup before updating!
Vulnerable Systems
These vulnerabilities are present on both Windows Server and Client operating systems. Essentially, any Windows server or client using IPsec is vulnerable and potentially exploitable.
Vulnerabilities
The following is a list of security updates related to Always On VPN deployments.
Windows IKE Extension Remote Code Execution (RCE) Vulnerability
As a follow-up to my last post regarding Always On VPN error 13801, this post will cover a similar and related error administrators may encounter, the 13806 error. As mentioned previously, certificate configuration is crucial for Always On VPN deployments. I described some specific certificates requirements for IKEv2 in this earlier post. Following this guidance, administrators should have no issues with IKEv2 Always On VPN connections. However, it is always possible to encounter an error if any of these certificates are missing or misconfigured.
Error 13806
Much like the error 13801 described previously, 13806 is also common. When an Always On VPN connection using IKEv2 fails, the Windows Application event log will record an event ID 20227 from the RasClient source. The error message states the following:
“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 13806”.
IKE Failed To Find Valid Machine Certificate
Error 13806 translates to ERROR_IPSEC_IKE_NO_CERT, indicating IKE failed to find a valid machine certificate. The problem can be on the device, the VPN server, or an issue with the VPN server configuration.
Device Certificate
For the device tunnel, the most obvious cause of this error is a missing device authentication certificate on the client itself. Ensure the endpoint has a valid certificate issued by the organization’s internal PKI that includes Client Authentication EKU (OID 1.3.6.1.5.5.7.3.2). The certificate must have a subject name matching the device’s FQDN. It must also be valid (not expired), trusted, and not revoked.
Certificate Chain
A 13806 error will occur if the device certificate installed on the client is not trusted or if the client does not trust the certificate installed on the VPN server. Ensure the client has all the necessary root and intermediate certification authority (CA) certificates installed in their respective certificate stores.
VPN Server Certificate
A 13806 error can also occur if the VPN server does not have a properly configured server certificate. Ensure the VPN server has a valid certificate issued by the organization’s internal PKI that includes both the Server Authentication (OID 1.3.6.1.5.5.7.3.1) and IP security IKE intermediate (OID 1.3.6.1.5.5.8.2.2) EKUs. The subject name must match the public fully qualified domain name (FQDN) used by VPN clients to connect to the VPN server (not the server’s NetBIOS name). Again, ensure the certificate is valid (not expired), trusted, not revoked, and all necessary root and intermediate CA certificates are installed in their respective certificate stores.
Certificate Revocation
An expired Certificate Revocation List (CRL) can also result in a 13806 error. Open the Enterprise PKI console (pkiview.msc) on an issuing CA and review the status of all CRLs. If any are expired, resolve any issues preventing the CRL from publishing successfully, then issue a new CRL by running certutil.exe -crl on the issuing CA server.
RRAS Configuration
Another cause of the 13806 error for the user tunnel is a misconfigured Routing and Remote Access Service (RRAS) VPN server. An error 13806 can happen if the administrator incorrectly defines a trusted root CA using Set-VpnAuthProtocol. Ensure that the root certificate thumbprint matches exactly the root CA server’s thumbprint used to issue certificates to VPN devices and the VPN server.
Get-VpnAuthProtocol
Root CA Certificate Thumbprint
Resolution
Ensure that devices and VPN servers have correctly configured certificates installed. If the root CA certificate is assigned incorrectly on the VPN server, follow the guidelines detailed here to update the configuration.
Certificate configuration is crucial for Always On VPN deployments. I described some specific certificates requirements for IKEv2 in this previous post. Following this guidance, administrators should have no issues with IKEv2 Always On VPN connections. However, it is always possible to encounter an error if any of these certificates are missing or misconfigured.
Error 13801
One of the most common errors related to IKEv2 and certificates is 13801. When an Always On VPN connection using IKEv2 fails, the Windows Application event log will record an event ID 20227 from the RasClient source. The error message states the following:
“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 13801”.
IKE Authentication Credentials are Unacceptable
Error 13801 translates to ERROR_IPSEC_IKE_AUTH_FAIL, indicating an authentication failure related to IPsec. The problem can be on the device, the VPN server, or an issue with the VPN server configuration.
Authentication Methods
To begin, ensure that the appropriate authentication method is enabled on the Routing and Remote Access (RRAS) server. Right-click the VPN server in the RRAS management console (rrasmgmt.msc) and choose Properties. Next, click on the Security tab and then click on the Authentication Methods button.
Select the ‘Extensible authentication protocol (EAP)’ to support IKEv2 user tunnel connections. In addition, select ‘Allow machine certificate authentication for IKEv2’ to support Always On VPN device tunnel connections.
Certificate Chain
A 13801 error will occur if the client does not trust the certificate installed on the VPN server. Ensure the client has all the necessary root and intermediate certification authority (CA) certificates installed in their respective certificate stores.
VPN Server Certificate
A 13801 error can also occur if the VPN server does not have a properly configured server certificate. Ensure the VPN server has a valid certificate issued by the organization’s internal PKI that includes both the Server Authentication (OID 1.3.6.1.5.5.7.3.1) and IP security IKE intermediate (OID 1.3.6.1.5.5.8.2.2) EKUs. The subject name must match the public fully qualified domain name (FQDN) used by VPN clients to connect to the VPN server (not the server’s NetBIOS name). Again, ensure the certificate is valid (not expired), trusted, not revoked, and all necessary root and intermediate CA certificates are installed in their respective certificate stores.
Certificate Revocation
An expired Certificate Revocation List (CRL) can also result in a 13801 error. Open the Enterprise PKI console (pkiview.msc) on an issuing CA and review the status of all CRLs. If any are expired, resolve any issues preventing the CRL from publishing successfully, then issue a new CRL by running certutil.exe -crl on the issuing CA server.
RRAS Configuration
Another cause of the 13801 error for the device tunnel is a misconfigured Routing and Remote Access Service (RRAS) VPN server. An error 13801 can happen if the administrator incorrectly defines a trusted root CA using Set-VpnAuthProtocol. Ensure that the root certificate thumbprint matches exactly the root CA server’s thumbprint used to issue certificates to VPN devices and the VPN server.
Get-VpnAuthProtocol
Root CA Certificate Thumbprint
Resolution
Ensure that devices and VPN servers have correctly configured certificates installed. If the root CA certificate is assigned incorrectly on the VPN server, follow the guidelines detailed here to update the configuration.
