Always On VPN Updates to Improve Connection Reliability

Always On VPN Updates to Improve Connection ReliabilityA longstanding issue with Windows 10 Always On VPN is that of VPN tunnel connectivity reliability and device tunnel/user tunnel interoperability. Many administrators have reported that Always On VPN connections fail to establish automatically at times, that only one tunnel comes up at a time (user tunnel or device tunnel, but not both), or that VPN tunnels fail to establish when coming out of sleep or hibernate modes. Have a look at the comments on this post and you’ll get a good understanding of the issues with Always On VPN.

Recent Updates

The good news is that most of these issues have been resolved with recent updates to Windows 10 1803 and 1809. Specifically, the February 19, 2019 update for Windows 10 1803 (KB4487029) and the March 1, 2019 update for Windows 10 1809 (KB4482887) include fixes to address these known issues. Administrators are encouraged to deploy Windows 10 1803 with the latest updates applied when implementing Always On VPN. Windows 10 1809 with the latest updates applied is preferred though.

Persistent Issues

Although initial reports are favorable for these updates and based on my experience the effectiveness and reliability of Windows 10 Always On VPN is greatly improved, there have still been some reports of intermittent VPN tunnel establishment failures.

Possible Causes

During my testing, after applying the updates referenced earlier both device tunnel and user tunnel connections are established much more consistently than before the updates were applied. I did encounter some issues, however. Specifically, when coming out of sleep or hibernate, VPN connections would fail to establish. Occasionally VPN connections would fail after a complete restart.

NCSI

After further investigation it was determined that the connectivity failure was caused by the Network Connectivity Status Indicator (NCSI) probe failing, causing Windows to report “No Internet access”.

Always On VPN Updates to Improve Connection Reliability

Cisco Umbrella Roaming Client

In this instance the NCSI probe failure was caused by the Cisco Umbrella Roaming Client installed and running on the device. The Umbrella Roaming Client is security software that provides client protection by monitoring and filtering DNS queries. It operates by configuring a DNS listener on the loopback address. NCSI probes are known to fail when the DNS server is running on a different interface than is being tested.

Resolution

Microsoft released a fix for this issue in Windows 10 1709. The fix involves changing a group policy setting to disable interface binding when perform DNS lookups by the NCSI. You can enable this setting via Active Directory group policy by navigating to Computer Configuration > Administrative Templates > Network > Network Connectivity Status Indicator > Specify global DNS. Select Enabled and check the option to Use global DNS, as shown here.

Always On VPN Updates to Improve Connection Reliability

For testing purposes this setting can be enabled individual using the following PowerShell command.

New-ItemProperty -Path “HKLM:\SOFTWARE\Policies\Microsoft\Windows\NetworkConnectivityStatusIndicator\” -Name UseGlobalDNS -PropertyType DWORD -Value 1 -Force

Third-Party Software

As Always On VPN connectivity can be affected by NCSI, any third-party firewall or antivirus/antimalware solution could potentially introduce VPN connection instability. Observe NCSI operation closely when troubleshooting unreliable connections with Always On VPN.

Additional Information

Windows 10 1803 Update KB4487029

Windows 10 1809 Update KB4482887

Cisco Umbrella Roaming Client Limited Network Connectivity Warning

Network Connectivity Status Indicator (NCSI) Operation Explained

Always On VPN Device Tunnel Does Not Connect Automatically

When configuring a Windows 10 Always On VPN device tunnel, the administrator may encounter a scenario in which the device tunnel does not connect automatically. This can occur even when ProfileXML is configured with the AlwaysOn element set to “true”.

Always On VPN Device Tunnel Does Not Connect Automatically

Manual Connection

An administrator can establish a device tunnel connection manually using rasdial.exe however, indicating no issues with connectivity or authentication that would prevent a successful automatic connection.

Always On VPN Device Tunnel Does Not Connect Automatically

Root Cause

This scenario will occur when the device tunnel configuration is applied to a Windows 10 Professional edition client.

Always On VPN Device Tunnel Does Not Connect Automatically

Device Tunnel Support

The Windows 10 Always On VPN device tunnel is supported only on Windows 10 1709 or later Enterprise edition clients that are domain-joined. To ensure the device tunnel connects automatically, upgrade to Windows 10 Enterprise 1709 or later and join it to a domain.

Always On VPN Device Tunnel Does Not Connect Automatically

Source: https://docs.microsoft.com/en-us/windows-server/remote/remote-access/vpn/vpn-device-tunnel-config#device-tunnel-requirements-and-features

Additional Information

Windows 10 Always On VPN Device Tunnel Configuration using PowerShell

Windows 10 Always On VPN Device Tunnel Missing in the Windows UI

Deleting a Windows 10 Always On VPN Device Tunnel

Always On VPN SSTP Connects then Disconnects

Always On VPN SSTP Connects then DisconnectsWhen Always On VPN clients are configured to use the Secure Socket Tunneling Protocol (SSTP) with Windows Server Routing and Remote Access Service (RRAS), administrators may encounter a scenario in which a client can establish a VPN connection using SSTP successfully, but is then disconnected immediately. The system event log contains an entry with Event ID 6 from the RasSstp source that includes the following error message.

“The SSTP-based VPN connection to the remote access server was terminated because of a security check failure. Security settings on the remote access server do not match settings on this computer. Contact the system administrator of the remote access server and relay the following information.”

Always On VPN Connect and Disconnect with SSTP

Common Causes

The two most common causes of this issue are when SSTP is configured for SSL offload, and when a VPN client is on a network where SSL inspection is taking place.

SSTP Offload

The most common cause of this issue is when SSL offload is configured for SSTP on an external load balancer or application delivery controller (ADC). To prevent interception from a Man-in-the-Middle attack, the VPN client sends the certificate hash of the SSL certificate used when the VPN connection was established. If this information does not match what is configured on the RRAS server, the connection is assumed to be compromised and the connection is immediately dropped.

SSL Inspection

Another scenario where this issue may occur is when a VPN client is behind a network device configured to perform SSL deep-packet inspection (DPI). SSTP VPN clients will be unable to connect to the VPN server in this scenario.

Resolution

When offloading SSL to another device, the RRAS server must be configured to know which SSL certificate is being presented to remote clients. This information is stored in the following registry key.

HKLM:\SYSTEM\CurrentControlSet\Services\SstpSvc\Parameters\SHA256CertificateHash

However, this registry entry requires a binary value, which makes it a challenge to configure manually. To resolve this problem, it is recommended that the same SSL certificate installed on the load balancer/ADC also be installed on the VPN server (even though SSL will be offloaded). To do this, first import the SSL certificate and private key in to the Local Computer certificate store, then open the RRAS management console and perform the following steps.

  1. Right-click the VPN server and choose Properties.
  2. Select the Security tab.
  3. Uncheck Use HTTP in the SSL Certificate Binding section.
  4. Select the appropriate SSL certificate from the Certificate drop-down list (click View to verify).
  5. Click Apply.

This will add the correct SSL certificate information to the registry. Next, re-enable HTTP for SSL offload by performing the following steps.

  1. Check Use HTTP in the SSL Certificate Binding section.
  2. Click Apply.

PowerShell Configuration

If the SSL certificate cannot be installed on the VPN server, or to automate this configuration across multiple servers remotely, download and run the Enable-SstpOffload PowerShell script from my GitHub repository here and run the following command.

Enable-SSTPOffload -CertificateHash [SHA256 Certificate Hash of Public SSL Certificate] -Restart

For example…

Enable-SSTPOffload -CertificateHash “C3AB8FF13720E8AD9047DD39466B3C8974E592C2FA383D4A3960714CAEF0C4F2” -Restart

Additional Information

Windows 10 Always On VPN Load Balancing and SSL Offload

Windows 10 Always On VPN SSL Certificate Requirements for SSTP

Windows 10 Always On VPN Protocol Recommendations for Windows Server RRAS

Troubleshooting Always On VPN Error Code 809

When testing an Always On VPN connection, the administrator may encounter a scenario where the VPN client fails to connect to the VPN server. On the Windows 10 client the error message states the following.

“Can’t connect to [connection name]. The network connection between your computer and the VPN server could not be established because the remote server is not responding. This could be because one of the network devices (e.g. firewalls, NAT, routers, etc.) between your computer and the remote server is not configured to allow VPN connections. Please contact your Administrator or your service provider to determine which device may be causing the problem.”

Always On VPN and IKEv2 Fragmentation

In addition, the Application event log records an error message with Event ID 20227 from the RasClient source. The error message states the following.

“The User [username] dialed a connection named [connection name] with has failed. The error code returned on failure is 809.”

Troubleshooting Always On VPN Error Code 809

Connection Timeout

The error code 809 indicates a VPN timeout, meaning the VPN server failed to respond. Often this is related directly to network connectivity, but sometimes other factors can come in to play.

Troubleshooting VPN Error Code 809

When troubleshooting VPN error code 809 the following items should be carefully checked.

  • Name Resolution – Ensure the VPN server’s public hostname resolves to the correct IP address.
  • Firewall Configuration – Confirm the edge firewall is configured properly. Inbound TCP port 443 is required for the Secure Socket Tunneling Protocol (SSTP) and inbound UDP ports 500 and 4500 are required for the Internet Key Exchange version 2 (IKEv2) protocol. Make sure that any NAT rules are forwarding traffic to the correct server.
  • Load Balancer Configuration – If VPN servers are located behind a load balancer, make certain that virtual IP address and ports are configured correctly and that health checks are passing. For IKEv2 specifically, it is crucial that UDP ports 500 and 4500 be delivered to the same backend server. This commonly requires custom configuration. For example, on the KEMP LoadMaster the administrator will configure “port following”. On the F5 BIG-IP a  custom “persistence profile” must be configured. On the Citrix NetScaler a “persistency group” must be defined.

IKEv2 Fragmentation

VPN error code 809 can also be caused by IKE fragmentation when using the IKEv2 VPN protocol. During IKEv2 connection establishment, payload sizes may exceed the IP Maximum Transmission Unit (MTU) for the network path between the client and server. This causes the IP packets to be fragmented. However, it is not uncommon for intermediary devices (routers, NAT devices, or firewalls) to block IP fragments. When this occurs, a VPN connection cannot be established. However, looking at a network trace of the connection attempt, the administrator will see that the connection begins but subsequently fails.

Troubleshooting Always On VPN Error Code 809

Enable IKEv2 Fragmentation Support

The IKEv2 protocol includes support for fragmenting packets at the IKE layer. This eliminates the need for fragmenting packets at the IP layer. IKEv2 fragmentation must be configured on both the client and server.

Client

IKEv2 fragmentation was introduced in Windows 10 1803 and is enabled by default. No client-side configuration is required.

Server

IKEv2 is commonly supported on many firewall and VPN devices. Consult the vendor’s documentation for configuration guidance. For Windows Server Routing and Remote Access (RRAS) servers, IKEv2 fragmentation was introduced in Windows Server 1803 and is also supported in Windows Server 2019. It is enabled via a registry key. The following PowerShell command can be used to enable IKEv2 fragmentation on supported servers.

New-ItemProperty -Path “HKLM:\SYSTEM\CurrentControlSet\Services\RemoteAccess\Parameters\Ikev2\” -Name EnableServerFragmentation -PropertyType DWORD -Value 1 -Force

Validation

Once IKEv2 fragmentation is configured on the VPN server, a network capture will reveal the IKE_SA_INIT packet now includes the IKEV2_FRAGMENTATION_SUPPORTED notification message.

Always On VPN and IKEv2 Fragmentation

Additional Information

Windows 10 Always On VPN and IKEv2 Fragmentation

Windows 10 Always On VPN IKEv2 Security Configuration

Windows 10 Always On VPN Hands-On Training Classes

Always On VPN and IKEv2 Fragmentation

The IKEv2 protocol is a popular choice when designing an Always On VPN solution. When configured correctly it provides the best security compared to other protocols. The protocol is not without some unique challenges, however. IKEv2 is often blocked by firewalls, which can prevent connectivity. Another lesser know issue with IKEv2 is that of fragmentation. This can result in failed connectivity that can be difficult to troubleshoot.

IP Fragmentation

IKEv2 uses UDP for transport, and typically most packets are relatively small. The exception to this is when authentication takes place, especially when using client certificate authentication. The problem is further complicated by long certificate chains and by RSA keys, especially those that are greater than 2048 bit. If the payload exceeds 1500 bytes, the IP packet will have to be broken in to smaller fragments to be sent over the network. If an intermediary device in the path is configured to use a smaller Maximum Transmission Unit (MTU), that device may fragment the IP packets.

IP Fragmentation and Firewalls

Many routers and firewalls are configured to drop IP fragments by default. When this happens, IKEv2 communication may begin initially, but subsequently fail. This typically results in an error code 809 with a message stating the following.

“Can’t connect to [connection name]. The network connection between your computer and the VPN server could not be established because the remote server is not responding. This could be because one of the network devices (e.g. firewalls, NAT, routers, etc.) between your computer and the remote server is not configured to allow VPN connections. Please contact your Administrator or your service provider to determine which device may be causing the problem.”

Always On VPN and IKEv2 Fragmentation

Troubleshooting

When troubleshooting potential IKEv2 fragmentation-related connection failures, a network trace should be taken of the connection attempt on the client. Observe the packet sizes during the conversation, especially IKE_AUTH packets. Packet sizes exceeding the path MTU will have to be fragmented, as shown here.

Always On VPN and IKEv2 Fragmentation

Measuring Path MTU

Measuring the path MTU between the client and server can be helpful when troubleshooting fragmentation related issues. The mtupath.exe utility is an excellent and easy to use tool for this task. The tool can be downloaded here.

Always On VPN and IKEv2 Fragmentation

IKEv2 Fragmentation

To address the challenges with IP fragmentation and potential connectivity issues associated with network devices dropping fragmented packets, the IKEv2 protocol itself can be configured to perform fragmentation at the IKE layer. This eliminates the need for IP layer fragmentation, resulting in better reliability for IKEv2 VPN connections.

Both the server and the client must support IKEv2 fragmentation for this to occur. Many firewall and VPN vendors include support for IKEv2 fragmentation. Consult the vendor’s documentation for configuration guidance. For Windows Server Routing and Remote Access (RRAS) servers, the feature was first introduced in Windows Server 1803 and is supported in Windows Server 2019. Windows 10 clients support IKEv2 fragmentation beginning with Windows 10 1803.

Enabling IKEv2 Fragmentation

Windows 10 clients support IKEv2 fragmentation by default. However, it must be enabled on the server via the registry. The following PowerShell command will enable IKEv2 fragmentation support on Windows Server 1803 and later.

New-ItemProperty -Path “HKLM:\SYSTEM\CurrentControlSet\Services\RemoteAccess\Parameters\Ikev2\” -Name EnableServerFragmentation -PropertyType DWORD -Value 1 -Force

Validation Testing

Once IKEv2 fragmentation is configured on the VPN server, a network capture will reveal the IKE_SA_INIT packet now includes the IKEV2_FRAGMENTATION_SUPPORTED notification message.

Always On VPN and IKEv2 Fragmentation

Additional Information

Windows 10 Always On VPN IKEv2 Security Configuration

RFC 7383 – IKEv2 Message Fragmentation

IEA Software MTU Path Scan Utility

Windows 10 Always On VPN Hands-On Training Classes

Always On VPN IKEv2 Connection Failure Error Code 800

Always On VPN administrators may encounter a scenario in which Windows 10 clients are unable to establish an IKEv2 VPN connection to a Windows Server Routing and Remote Access Service (RRAS) server or a third-party VPN device under the following conditions.

  1. The VPN connection is configured using ProfileXML.
  2. ProfileXML includes the <CryptographySuite> element.
  3. The VPN server is configured to use a custom IPsec policy.
  4. The VPN server supports only IKEv2.
  5. The <NativeProtocolType> in ProfileXML is set to Automatic.

When these specific conditions are met, the client will be unable to connect to the VPN server using IKEv2. The error message states:

The remote connection was not made because the attempted VPN tunnels failed. The VPN server might be unreachable. If this connection is attempting to use an L2TP/IPsec tunnel, the security parameters required for IPsec negotiation might not be configured properly.

Always On VPN IKEv2 VPN Connection Failure Error Code 800

In addition, the event log will include an error message from the RasClient source with event ID 20227 that includes the following error message.

The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 800.

Always On VPN IKEv2 VPN Connection Failure Error Code 800

A manually configured VPN connection using IKEv2 will connect successfully under these same conditions, however.

IKEv2 Error Code 800

Error code 800 translates to ERROR_AUTOMATIC_VPN_FAILED, which is somewhat ambiguous. The error description is:

Unable to establish the VPN connection. The VPN server may be unreachable, or security parameters may not be configured properly for this connection.

Digging Deeper

A network trace of the IKEv2 VPN connection reveals the true source of the problem, which is a failure of the client and server to successfully negotiate an IKEv2 security association (SA). During the SA initiation process, the parameters offered by the client are unacceptable to the server, resulting in a NO_PROPOSAL_CHOSEN notification being returned by the server.

Always On VPN IKEv2 VPN Connection Failure Error Code 800

Custom Cryptography Settings Ignored

It appears that the Always On VPN connection ignores the custom cryptography settings defined in the CryptographySuite element in ProfileXML. However, this only occurs when the NativeProtocolType is set to Automatic. Presumably, this is a bug. 🙂

Workaround

As a workaround, set the NativeProtocolType to IKEv2. When NativeProtocolType is set to IKEv2, the VPN connection recognizes the IKEv2 parameters defined in the CryptographySuite element and the VPN connection will be established successfully.

Additional Information

Always On VPN IKEv2 Security Configuration

Always On VPN Certificate Requirements for IKEv2

Always On VPN IKEv2 Load Balancing with the KEMP LoadMaster Load Balancer

DirectAccess IP-HTTPS Not Working Properly in Windows Server 2019

After installing and configuring DirectAccess in Windows Server 2019 you may encounter an error message indicating that IP-HTTPS is not working properly. Looking at the Operations Status overview in the Dashboard of the Remote Access Management console shows that the IP-HTTPS interface is in error.

DirectAccess IP-HTTPS Not Working Properly in Windows Server 2019

IP-HTTPS Route Error

Viewing the detailed Operations Status shows the following error message.

Error: The IP-HTTPS route does not have published property enabled.

DirectAccess IP-HTTPS Not Working Properly in Windows Server 2019

Missing Route

Looking at the routing table on the DirectAccess server reveals that a route to the client IPv6 prefix is indeed missing.

DirectAccess IP-HTTPS Not Working Properly in Windows Server 2019

Resolution

To resolve this error message, add the client IPv6 route to the DirectAccess server’s routing table and publish it. This is accomplished by running the following PowerShell commands on the DirectAccess server.

$IPv6prefix = (Get-RemoteAccess).ClientIPv6Prefix
New-NetRoute -AddressFamily IPv6 -DestinationPrefix $IPv6prefix -InterfaceAlias “Microsoft IP-HTTPS Platform Interface” -Publish Yes

Next, restart the Remote Access Management service (RaMgmtSvc) using the following PowerShell command.

Restart-Service RaMgmtSvc -PassThru

DirectAccess IP-HTTPS Not Working Properly in Windows Server 2019

Once complete, refresh the management console and the IP-HTTPS error message should be resolved and the operations status should state that it is now working properly.

DirectAccess IP-HTTPS Not Working Properly in Windows Server 2019

 

Additional Information

SSL Certificate Conisderations for DirectAccess IP-HTTPS

DirectAccess Expire IP-HTTPS Certificate and Error 0x800b0101

DirectAccess Get-NetIPHttpsState Fails on Windows 10 1803

DirectAccess Get-NetIPHttpsState Fails on Windows 10 1803PowerShell is an essential tool for Windows administrators for configuration, task automation, monitoring, reporting, and problem resolution. When troubleshooting DirectAccess connectivity using the IP-HTTPS IPv6 transition technology, the Get-NetIPHttpsConfiguration and Get-NetIPHttpsState PowerShell commands are important for assessing the configuration and current state of the IP-HTTPS connection. When DirectAccess connectivity fails, these are some of the first commands an administrator will use to identify and resolve the issue.

Get-NetIPHttpsState

Get-NetIPHttpsState is especially helpful when IP-HTTPS connectivity fails because it returns an error code and interface status information that can provide clues as to why the connection was not completed successfully.

DirectAccess Get-NetIPHttpsState Fails on Windows 10 1803

No Output in 1803

Beginning with Windows 10 1803, the DirectAccess administrator will notice that Get-NetIPHttpsState returns no data. The output of Get-NetIPHttpsState is blank.

DirectAccess Get-NetIPHttpsState Fails on Windows 10 1803

Changes in 1803

As it turns out, this is a bug first introduced in Windows 10 1803 that is the result of a fundamental change in the way in which the IP-HTTPS interface is implemented in Windows. As of this writing, the bug has not been addressed in Windows 10 1803 or 1809.

Workaround

The good news is that there’s an easy workaround for this. Instead of using Get-NetIPHttpsState, the administrator can retrieve essential information about the IP-HTTPS interface using the following netsh command.

netsh interface httpstunnel show interface

DirectAccess Get-NetIPHttpsState Fails on Windows 10 1803

Additional Information

SSL Certificate Considerations for DirectAccess IP-HTTPS 

Troubleshooting DirectAccess IP-HTTPS Error Code 0x800b0109

Troubleshooting DirectAccess IP-HTTPS Error Code 0x80090326

Troubleshooting DirectAccess IP-HTTPS Error Code 0x90320

Troubleshooting DirectAccess IP-HTTPS Error Code 0x2af9

Troubleshooting DirectAccess IP-HTTPS Error Code 0x800b0101

Troubleshooting Always On VPN Error Code 0x80092013

Troubleshooting Always On VPN Error Code 0x80092013Windows Server Routing and Remote Access Service (RRAS) is commonly used for Windows 10 Always On VPN deployments because it is easy to configure and manage and it includes Microsoft’s proprietary Secure Socket Tunneling Protocol (SSTP). SSTP is a Transport Layer Security (TLS) VPN protocol that is firewall-friendly and ubiquitously available. However, a common configuration mistake can lead to failed connections.

Error 0x80092013

A Windows 10 Always On VPN client may fail to establish a VPN connection to an RRAS VPN server when using SSTP. The VPN client will return the following error message.

“Can’t connect to Always On VPN. The revocation function was unable to check revocation because the revocation server was offline.”

Troubleshooting Always On VPN Error Code 0x80092013

The event log will also include RasClient event ID 20227 with the following error.

“The user [domain\user] dialed a connection named [connection name] which has failed. The error code returned on failure is -2146885613.”

Troubleshooting Always On VPN Error Code 0x80092013

The Win32 error code –2146885613 converts to hexadecimal 0x80092013, which translates to CRYPT_E_REVOCATION_OFFLINE, indicating that the client was unable to successfully perform a check of the VPN server’s SSL certificate.

Revocation Checking

When the VPN client attempts to establish an SSTP connection to the Windows RRAS VPN, it will check the Certification Revocation List (CRL) using the information provided in the SSL certificate. If the CRL is unreachable for any reason, the client will not complete the connection

Common Cause of Error 0x80092013

Certificate revocation failures for Windows 10 Always On VPN SSTP connections commonly occur when the RRAS VPN server is configured with an SSL certificate issued by an internal certification authority (CA) and the CRL is not publicly available.

Resolving Error 0x80092013

Making the internal CA’s CRL available publicly will of course resolve this error. However, best practice recommendations for the SSTP SSL certificate call for the use of a certificate issued by a public CA. For detailed information about SSL certificate requirements and recommendations, please see Always On VPN SSL Certificate Requirements for SSTP.

Additional Information

Always On VPN SSL Certificate Requirements for SSTP

Always On VPN ECDSA SSL Certificate Request for SSTP

Always On VPN Protocol Recommendations for Windows RRAS

Always On VPN RasMan Device Tunnel Failure

Always On VPN RasMan Device Tunnel FailureAn Always On VPN device tunnel is an optional configuration for Windows 10 Enterprise edition clients designed to provide machine-level remote network connectivity. This capability provides feature parity with DirectAccess for domain-joined clients to support scenarios such as logging on without cached credentials and unattended remote support, among others.

Device Tunnel Failure

When configuring a Windows 10 client to use an Always On VPN device tunnel, you may find that the device tunnel works without issue after initial deployment but fails to connect after the computer restarts. In addition, the Windows event log will include an Event ID: 1000 application error with the following error message:

Faulting application name: svchost.exe_RasMan

Always On VPN RasMan Device Tunnel Failure

Known Issue

This can occur when a Windows 10 machine is configured with a device tunnel only (no user tunnel). This is a known issue with Windows 10 v1709. It has been resolved in Windows 10 v1803 (RS4).

Additional Information

Windows 10 Always On VPN Device Tunnel Step-by-Step Configuration using Powershell

Deleting an Always On VPN Device Tunnel

%d bloggers like this: