Always On VPN RasMan Errors in Windows 10 1903

Always On VPN RasMan Errors in Windows 10 1903After deploying or upgrading to Windows 10 1903, administrators may find that Windows 10 Always On VPN connections fail to establish successfully. Always On VPN connections continue to work for Windows 10 1809 and earlier clients, however.

Important Note: The issue described in this article has been addressed in KB4505903 (build 18362.267) released July 26, 2019.

RasMan Event Log Errors

When this occurs, the application event log contains an error with Event ID 1000 that includes the following information.

“Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005”

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Root Cause

RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix.

Workaround

As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell.

Group Policy

Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps.

1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds
2. Double-click Allow Telemetry.
3. Select Enabled.
4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security).
5. Click Ok.

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Registry

Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting.

HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1

Always On VPN RasMan Errors in Windows 10 1903 Administrators may find that Windows 10 Always On VPN connections fail after deploying or upgrading to Windows 10 1903. Always On VPN connections continue to work for Windows 10 1809 and earlier clients. RasMan Event Log Errors When this occurs, the application event log contains an error with Event ID 1000 that includes the following information. “Faulting application name: svchost.exe_RasMan…”, “Faulting module name: rasmans.dll”, and “Exception code: 0xc0000005” Root Cause RasMan failures can occur in Windows 10 1903 clients when telemetry is disabled via group policy or the registry. Microsoft has identified the issue and is currently working on a fix. Workaround As a temporary workaround to restore Always On VPN connectivity, enable telemetry on Windows 10 1903 using Active Directory or local group policy, the local registry, or PowerShell. Group Policy Create a new GPO or edit an existing one by opening the group policy management console (gpmc.msc) and performing the following steps. 1. Expand Computer Configuration > Administrative Templates > Windows Components > Data Collection and Preview Builds 2. Double-Click Allow Telemetry. 3. Select Enabled. 4. Choose 1-Basic, 2-Enhanced, or 3-Full (do not select 0-Security). 5. Click Ok. Registry Telemetry can also be enabled locally by opening the registry editor (regedit.exe) and modifying the following registry setting. HKLM\SOFTWARE\Policies\Microsoft\Windows\DataCollection\AllowTelemetry DWORD = 1 Note: The AllowTelemetry value can be removed entirely, if desired. PowerShell PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting. New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely. Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\' -Name AllowTelemetry Restart Required Once these changes have been made, restart the client and test the Always On VPN connection. Additional Information asdf

Note: The AllowTelemetry value can be removed entirely, if desired.

PowerShell

PowerShell can also be used modify or remove the AllowTelemetry value on Windows 10 1903 clients. Run the following PowerShell command to update the AllowTelemetry setting.

New-ItemProperty -Path ‘HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\’ -Name AllowTelemetry -PropertyType DWORD -Value 1 -Force

Optionally, run the following PowerShell command to remove the AllowTelemetry setting entirely.

Remove-ItemProperty -Path ‘HKLM:\SOFTWARE\Policies\Microsoft\Windows\DataCollection\’ -Name AllowTelemetry

Service Restart Required

Once these changes have been made, restart the Remote Access Connection Manager service (RasMan) using the Services mnagement console (services.msc) or by running the following PowerShell command.

Restart-Service RasMan -PassThru

Optionally, the client can be rebooted to apply these changes.

Additional Information

Windows 10 1903 Known Issues

 

Always On VPN Device Tunnel and Certificate Revocation

Always On VPN Device Tunnel and Certificate RevocationRecently I wrote about denying access to Windows 10 Always On VPN users or computers. In that post I provided specific guidance for denying access to computers configured with the device tunnel. To summarize, the process involved exporting the device certificate from the issuing Certification Authority (CA) server and placing it in the Untrusted Certificates certificate store on each VPN server. In theory, simply revoking the device certificate should be all that’s required to prevent device tunnel connections.

Revocation Check Failure

As it turns out, a bug in Windows Server Routing and Remote Access prevents this from working as expected. Windows Server 2012 R2, 2016, and 2019 all fail to check the Certificate Revocation List (CRL) for IKEv2 VPN connections using machine certificate authentication (for example an Always On VPN device tunnel).

Updates for Windows Server

Microsoft has released fixes to support device tunnel certificate revocation for the following operating systems.

Windows Server 2019 – KB4505658 (build 17763.652)

Windows Server 2016 – KB4503294 (build 14393.3053)

Windows Server 2012/R2 – Will not be updated.

Enable Revocation Check

Additional configuration is required to enable support for CRL checking. Microsoft published guidance for configuring CRL revocation checks for IKEv2 VPN connections using machine certificate authentication here. Specifically, administrators must enable the RootCertificateNameToAccept parameter and set a registry key to enable this functionality.

Open an elevated PowerShell window and run the following commands to enable CRL checking for IKEv2 VPN connections using machine certificate authentication.

$Thumbprint = ‘Root CA Certificate Thumbprint’
$RootCACert = (Get-ChildItem -Path cert:\LocalMachine\root | Where-Object {$_.Thumbprint -eq $Thumbprint})
Set-VpnAuthProtocol -RootCertificateNameToAccept $RootCACert -PassThru

New-ItemProperty -Path ‘HKLM:\SYSTEM\CurrentControlSet\Services\RemoteAccess\Parameters\Ikev2\’ -Name CertAuthFlags -PropertyTYpe DWORD -Value ‘4’ -Force

Restart-Service RemoteAccess -PassThru

Always On VPN Device Tunnel and Certificate Revocation

A PowerShell script to update the RootCertificateNameToAccept parameter on multiple VPN servers can be found here.

Revoking Certificates

To prevent a Windows 10 Always On VPN device tunnel connection, the administrator must first revoke the certificate on the issuing CA. Next, open an elevated command window an enter the following commands. Repeat these steps on each VPN server in the enterprise.

certutil -urlcache * delete
certutil -setreg chain\ChainCacheResyncFiletime @now

Additional Information

Denying Access to Windows 10 Always On VPN Users or Computers

Blocking VPN Clients that use Revoked Certificates

PowerShell Script to Configure RootCertificateNameToAccept on GitHub

 

 

Always On VPN Updates to Improve Connection Reliability

Always On VPN Updates to Improve Connection ReliabilityA longstanding issue with Windows 10 Always On VPN is that of VPN tunnel connectivity reliability and device tunnel/user tunnel interoperability. Many administrators have reported that Always On VPN connections fail to establish automatically at times, that only one tunnel comes up at a time (user tunnel or device tunnel, but not both), or that VPN tunnels fail to establish when coming out of sleep or hibernate modes. Have a look at the comments on this post and you’ll get a good understanding of the issues with Always On VPN.

Recent Updates

The good news is that most of these issues have been resolved with recent updates to Windows 10 1803 and 1809. Specifically, the February 19, 2019 update for Windows 10 1803 (KB4487029) and the March 1, 2019 update for Windows 10 1809 (KB4482887) include fixes to address these known issues. Administrators are encouraged to deploy Windows 10 1803 with the latest updates applied when implementing Always On VPN. Windows 10 1809 with the latest updates applied is preferred though.

Persistent Issues

Although initial reports are favorable for these updates and based on my experience the effectiveness and reliability of Windows 10 Always On VPN is greatly improved, there have still been some reports of intermittent VPN tunnel establishment failures.

Possible Causes

During my testing, after applying the updates referenced earlier both device tunnel and user tunnel connections are established much more consistently than before the updates were applied. I did encounter some issues, however. Specifically, when coming out of sleep or hibernate, VPN connections would fail to establish. Occasionally VPN connections would fail after a complete restart.

NCSI

After further investigation it was determined that the connectivity failure was caused by the Network Connectivity Status Indicator (NCSI) probe failing, causing Windows to report “No Internet access”.

Always On VPN Updates to Improve Connection Reliability

Cisco Umbrella Roaming Client

In this instance the NCSI probe failure was caused by the Cisco Umbrella Roaming Client installed and running on the device. The Umbrella Roaming Client is security software that provides client protection by monitoring and filtering DNS queries. It operates by configuring a DNS listener on the loopback address. NCSI probes are known to fail when the DNS server is running on a different interface than is being tested.

Resolution

Microsoft released a fix for this issue in Windows 10 1709. The fix involves changing a group policy setting to disable interface binding when perform DNS lookups by the NCSI. You can enable this setting via Active Directory group policy by navigating to Computer Configuration > Administrative Templates > Network > Network Connectivity Status Indicator > Specify global DNS. Select Enabled and check the option to Use global DNS, as shown here.

Always On VPN Updates to Improve Connection Reliability

For testing purposes this setting can be enabled individual using the following PowerShell command.

New-ItemProperty -Path “HKLM:\SOFTWARE\Policies\Microsoft\Windows\NetworkConnectivityStatusIndicator\” -Name UseGlobalDNS -PropertyType DWORD -Value 1 -Force

Third-Party Software

As Always On VPN connectivity can be affected by NCSI, any third-party firewall or antivirus/antimalware solution could potentially introduce VPN connection instability. Observe NCSI operation closely when troubleshooting unreliable connections with Always On VPN.

Additional Information

Windows 10 1803 Update KB4487029

Windows 10 1809 Update KB4482887

Cisco Umbrella Roaming Client Limited Network Connectivity Warning

Network Connectivity Status Indicator (NCSI) Operation Explained