Always On VPN Client Routes Missing

Choosing an Enterprise VPN

When configuring Always On VPN for Windows 10 and Windows 11 clients, administrators may encounter a scenario where an IPv4 route defined in Microsoft Endpoint Manager/Intune or custom XML is not reachable over an established Always On VPN connection. Further investigation indicates the route is added to the configuration on the endpoint but does not appear in the routing table when the connection is active.

Routing Configuration

When split tunneling is enabled, administrators must define routes to IP networks that are reachable over the Always On VPN connection. The method of defining these routes depends on the client configuration deployment method.

Endpoint Manager

Using Microsoft Endpoint Manager, administrators define IP routes in the Split Tunneling section of the configuration settings for the Always On VPN device configuration profile. Routes are defined by entering the destination prefix and prefix size. In this example, the 10.0.0.0/8 and 172.21.12.0/21 IPv4 networks are defined for routing over the Always On VPN tunnel.

Custom XML

Using custom XML deployed using Microsoft Endpoint Manager, System Center Configuration Manager (SCCM), or PowerShell, routes are defined in the XML file using the following syntax.

Client Configuration

Validate the routing configuration has been implemented on the endpoint successfully by running the following PowerShell command.

Get-VpnConnection -Name <Connection Name> | Select-Object -ExpandProperty Routes

As you can see here, the IPv4 routes 10.0.0.0/8 and 172.21.12.0/21 are included in the client’s Always On VPN configuration, as shown below.

Missing Route

However, after establishing an Always On VPN connection, the 172.21.12.0/21 network is not reachable. To continue troubleshooting, run the following PowerShell command to view the active routing table.

Get-NetRoute -AddressFamily IPv4

As you can see above, the only IPv4 route in the VPN configuration added to the routing table is the 10.0.0.0/8 network. The 172.21.12.0/21 IPv4 route is missing.

Network Prefix Definition

IPv4 routes missing from the Always On VPN client’s routing table result from incorrect network prefix definition. Specifically, the IPv4 route 172.21.12.0/21 used in the example here is not a valid network address. Rather, it is a host address in the 172.21.8.0/21 network, as shown below.

The Get-Subnet PowerShell cmdlet is part of the Subnet PowerShell module. To install this module, run the following PowerShell command.

Install-Module Subnet

Resolution

Using the example above, enabling access to the 172.21.12.0/21 subnet would require defining the IPv4 prefix in the routing configuration as 172.21.8.0/21. The moral of this story is always validate routing prefixes to ensure they are, in fact, network addresses and not host addresses.

Additional Information

Always On VPN Routing Configuration

Always On VPN Default Class-based Route and Microsoft Endpoint Manager/Intune

Always On VPN Windows 11 Issues with Intune

Always On VPN RasMan Errors in Windows 10 1903

Since the introduction of Windows 11, there have been numerous reports of issues with Always On VPN when deployed using Microsoft Endpoint Manager/Intune. Specifically, administrators have been reporting that Always On VPN profiles are being deleted, then later reappearing. Obviously, this is highly disruptive to users in the field.

Causes

According to Microsoft, there are several causes for deleted VPN profiles.

Changes to an Existing Profile

Missing Always On VPN profiles commonly occurs when updating settings for an existing VPN profile applied to Windows 11 endpoints. In this scenario, the VPN profile is deleted but not immediately replaced. Synchronize the device with Microsoft Endpoint Manager/Intune once more to return the VPN profile.

Multiple Profiles

Issues with Always On VPN profiles may also occur if two new VPN profiles are applied to the endpoint simultaneously.

Remove and Replace

Removing and replacing an Always On VPN profile at the same time will also result in connectivity issues.

Reference: https://docs.microsoft.com/en-us/mem/intune/configuration/vpn-settings-configure

Workaround

There is no known workaround for these issues at this time. Microsoft is aware of the problem and is working on a fix, and until then, rolling out Windows 11 with Always On VPN should be avoided.

Additional Issues

There have been reports of other known issues with Windows 11 and Always On VPN. For instance, my PowerShell script that removes an Always On VPN connection doesn’t work with Windows 11. I’m working to resolve that issue as we speak.

Are you experiencing any issues with Always On VPN on Windows 11? Please share them in the comments below!

Always On VPN Book Available for Pre-Order

Great news! My new book, Implementing Always On VPN, is now available for pre-order on Amazon.com. This new book, scheduled for release in late 2021, is a comprehensive implementation guide for Windows 10 Always On VPN. Drawing on many years of experience deploying Always On VPN for organizations worldwide, it covers all aspects of an Always On VPN deployment, including planning and design, prerequisite gathering, infrastructure preparation, and client deployment.

In addition, it contains detailed, prescriptive guidance for advanced configuration options such as application and traffic filtering and proxy server configuration. Cloud deployments using Azure VPN gateway and Virtual WAN are covered, and it includes guidance for configuring Azure MFA and Conditional Access.

Also, the book includes thorough guidance for provisioning certificates using Microsoft Endpoint Manager/Intune using both PKCS and SCEP. It outlines options for high availability for VPN and authentication infrastructure and provides details for ongoing system maintenance and operational support.

Finally, the book has an entire chapter dedicated to troubleshooting and resolving common (and not so common!) issues encountered with Windows 10 Always On VPN.

Reserve your copy today. Pre-order Implementing Always On VPN now!

Chapter List

  1. Always On VPN Overview
  2. Plan an Always On VPN Deployment
  3. Prepare the Infrastructure
  4. Configure Windows Server for Always On VPN
  5. Provision Always On VPN clients
  6. Advanced Configuration
  7. Cloud Deployments
  8. Deploy Certificates with Intune
  9. Integrating Azure MFA
  10. High Availability
  11. Monitor and Report
  12. Troubleshooting

Always On VPN Short Name Access Failure

Using Microsoft Endpoint Manager (Intune), administrators can provision Always On VPN to devices that are Azure AD joined only. Users accessing on-premises resources from these devices can still use seamless single sign-on, making this deployment option popular for organizations moving to the cloud.

Short Names

After deploying Always On VPN to Windows 10 devices that are Azure AD joined only and configured to use client certificate authentication, administrators may find that users cannot access on-premises resources by their short name, such as \\app1. The connection fails and returns the following error message.

“Windows can’t find <servername/sharename>. Check the spelling and try again.”

FQDN

Interestingly, on-premises resources are accessible using their fully qualified domain name (FQDN), such as \\app1.corp.example.net.

Troubleshooting

Testing name resolution using the short name works as expected, and the resource is reachable at the network layer, as shown here.

Workaround

This issue is related to how Windows performs authentication when connected via VPN. To resolve this issue, edit the rasphone.pbk file and change the value of UseRasCredentials to 0. Rasphone.pbk can be found in the $env:AppData\Microsoft\Network\Connections\Pbk folder.

After updating this setting, restart the VPN connection for the change to take effect.

Proactive Remediations

While helpful for testing, editing rasphone.pbk manually obviously does not scale well. To address this, consider using Intune Proactive Remediations. Intune Proactive Remediations allows administrators to deploy detection and remediation PowerShell scripts to monitor specific settings and update them if or when they change. Proactive Remediations will ensure the setting is applied consistently across all managed endpoints.

GitHub Repository

I have created a new GitHub repository dedicated to PowerShell scripts for Endpoint Manager Proactive Remediations for Always On VPN. There you will find detection and remediation scripts for the UseRasCredentials settings change described in this article.

Additional Information

Always On VPN Endpoint Manager Proactive Remediation Scripts on GitHub

Endpoint Manager Proactive Remediations Tutorial

Always On VPN and Zero Trust Network Access (ZTNA)

Always On VPN and Zero Trust Network Access (ZTNA)

Zero Trust Network Access (ZTNA) is a term that administrators are likely familiar with, as it is one of the hottest marketing buzzwords in circulation today. ZTNA can mean different things depending on the deployment scenario. ZTNA is fundamentally about enforcing the principle of least privilege for endpoints connecting remotely to the corporate network when it comes to enterprise mobility and remote access.

Trusted Access

Historically, VPNs and even DirectAccess granted full, unrestricted network access to authenticated devices and users. Once the endpoint has an IP address, and in the absence of other controls (routing limitations, firewall access controls, etc.), the user could access any resource on the internal network. The rationale was that authenticated devices and users should be considered “trusted”.

Limitations

The Trusted Access model has some significant limitations. It assumes that all traffic from authorized users and devices is legitimate. However, if an endpoint is compromised, an attacker has broad access to the internal network, which is not ideal from a security perspective.

Zero Trust

Zero Trust Network Access is a concept where administrators define explicitly the minimum level of access required to support remote workers. Instead of granting full network access to the endpoint, controlling access using fine-grained policies is enforced on the VPN connection. Configuring limited network access for Always On VPN clients dramatically reduces exposure of the internal network to compromised endpoints.

ZTNA Management

There is a significant management burden associated with this approach, however. Administrators must identify each application requiring VPN access and determine all associated protocols and ports to be allowed, and internal resources to which they will communicate. Although this task isn’t difficult if clients require access to a small subset of internal resources, it can be a substantial undertaking if clients require access to many internal resources from numerous client applications.

Moving Targets

Making things more challenging is that application and network infrastructure often change constantly, requiring administrators to manage network access continually to ensure application availability. When adding new applications or changing the internal infrastructure, updating the configuration on all remote endpoints will be required.

Updating Always On VPN configuration for devices managed with Microsoft Endpoint Manager (formerly Intune) isn’t difficult. However, it can be more challenging when using PowerShell with System Center Configuration Manager (SCCM) or another endpoint management platform.

Traffic Filters

ZTNA can be configured with Always On VPN using Traffic Filters. With Traffic Filters, administrators can apply fine-grained access control for VPN traffic based on a combination of the following.

  • Source IP address (IP address, address range, or subnet)
  • Destination IP address (IP address, address range, or subnet)
  • Protocol (TCP, UDP, IP, etc.)
  • Source Port
  • Destination Port

Endpoint Manager Configuration

Configuring Traffic Filters for Always On VPN connections can be performed using Microsoft Endpoint Manager. Open the Endpoint Manager management console (https://endpoint.microsoft.com), navigate to the Always On VPN device configuration profile, then perform the following steps.

  1. Expand App and Traffic Rules.
  2. Click Add next to Network traffic rules for this VPN connection.
  1. Enter a descriptive name in the Name field.
  2. Select Split tunnel from the Rule type drop-down list.
  3. Enter “6” in the Protocol field.
  4. Enter “3389” in the Lower port and Upper port fields in the Remote port ranges section.
  5. Enter an IPv4 address in the Lower IPv4 address field.
  6. Enter an IPv4 address in the Upper IPv4 address field. Enter the same IPv4 address as the lower address to specify a single host.
  7. Click Save.

The example above shows a traffic filter restricting access to TCP port 3389 (Remote Desktop Protocol) from all VPN clients to the 172.16.0.0/24 network.

Note: Repeat these steps to create as many traffic filters as required for any processes or applications that must communicate over the Always On VPN connection.

XML Configuration

Traffic Filters can also be configured using custom XML. To implement the same Traffic Filter described previously, add the following code between the <VPNProfile> and </VPNProfile> tags in your XML configuration file.

<TrafficFilter>
   <Protocol>6</Protocol>
   <RemotePortRanges>3389</LocalPortRanges>
   <RemoteAddressRanges>172.16.0.0/24</RemoteAddressRanges>
</TrafficFilter>

Note: Address ranges used in Traffic Filters can be defined using CIDR notation in XML, but they are not supported using Microsoft Endpoint Manager today.

Default Deny

When configuring a Traffic Filter for an Always On VPN profile, an implicit “deny all” rule is automatically enabled. Any traffic not explicitly defined in a Traffic Filter will be denied, including unsolicited inbound traffic, which has crucial implications for the device tunnel because it is used commonly for system management of remote devices.

Direction

Traffic Filters are enabled for the Outbound direction only, by default. Beginning with Windows 10 2004, Microsoft introduced support for Inbound traffic filters. Before Windows 10 2004, configuring a Traffic Filter on the device tunnel would break manage-out scenarios by denying all unsolicited inbound network access.

As of this writing, configuring inbound Traffic Filters using Microsoft Endpoint Manager is not supported. They are only configurable using custom XML.

To implement a Traffic Filter to allow inbound RDP access from the internal network over the device tunnel, add the following code between the <VPNProfile> and </VPNProfile> tags in your XML configuration file.

<TrafficFilter>
   <Protocol>6</Protocol>
   <LocalPortRanges>3389</LocalPortRanges>
   <RemoteAddressRanges>172.16.0.0/16</RemoteAddressRanges>
   <Direction>Inbound</Direction>
</TrafficFilter>

Note: When configuring inbound Traffic Filters, specify the port of the listening process or application using the LocalPortRanges field.

Application Filters

Administrators can combine Application Filters with Traffic Filters to control network access over the Always On VPN connection even more granularly. Applications can be defined by the following.

  • Package Family Name (PFN) – This is the unique name of a Microsoft Store application. Use the Get-AppxPackage PowerShell command to find the PFN for an application.
  • File Path – This is the full path to any executable on the file system. For example, c:\Windows\System32\mstsc.exe.
  • SYSTEM – This allows Windows kernel-mode drivers (such as ping.exe and net.exe) to send traffic over the Always On VPN connection.

As of this writing, configuring Application Filters using Microsoft Endpoint Manager is not supported. They are only configurable using custom XML.

Application Filter Examples

Below are three examples showing different Application Filters based on file path, Package Family Name, and SYSTEM.

File Path

This example shows a Traffic Filter configured to allow RDP access to an internal subnet using the native Windows Remote Desktop client (mstsc.exe).

<TrafficFilter>
   <App>
      <Id>C:\Windows\System32\mstsc.exe</Id>
   </App>
   <Protocol>6</Protocol>
   <RemotePortRanges>3389</RemotePortRanges>
   <RemoteAddressRanges>172.16.0.0/24</RemoteAddressRanges>
</TrafficFilter>

Package Family Name

This example shows a Traffic Filter configured to allow RDP access to an internal subnet using the Microsoft Windows Store Remote Desktop client.

<TrafficFilter>
   <App>
      <Id>Microsoft.RemoteDesktop_8wekyb3d8bbwe</Id>
   </App>
   <Protocol>6</Protocol>
   <RemotePortRanges>3389</RemotePortRanges>
   <RemoteAddressRanges>172.16.0.0/24</RemoteAddressRanges>
</TrafficFilter>

SYSTEM

This example shows a Traffic Filter configured to allow the netsh.exe process access to an internal subnet.

<TrafficFilter>
   <App>
      <Id>SYSTEM</Id>
   </App>
   <Protocol>6</Protocol>
   <RemotePortRanges>445</RemotePortRanges>
   <RemoteAddressRanges>172.16.0.0/24</RemoteAddressRanges>
</TrafficFilter>

This example shows a Traffic Filter configured to allow the ping.exe process access to an internal subnet.

<TrafficFilter>
   <App>
      <Id>SYSTEM</Id>
   </App>
   <Protocol>1</Protocol>
   <RemoteAddressRanges>172.16.0.0/24</RemoteAddressRanges>
</TrafficFilter>

Note: Ping uses ICMP (IP protocol 1), which is a network layer protocol. As such, defining ports for the filter is not required.

IPv6 Compatibility

Sadly, the filtering techniques described in this article do not work when also configuring IPv6 on the Always On VPN connection. As of this writing, enabling Traffic Filters when an IPv6 address is assigned to the VPN interface is not supported. More details can be found here.

Always On VPN Traffic Filters and IPv6

Summary

Configuring Zero Trust Network Access (ZTNA) with Windows 10 Always On VPN is not trivial. Still, with attention to detail, it can be a highly effective tool to enforce fine-grained network access policies and reduce exposure of the internal network to compromised endpoints. Combining Traffic Filters with Application Filters allows administrators to tightly control Always On VPN access and ensure the principle of least privilege is applied.

Additional Information

Windows 10 Always On VPN Traffic Filters and IPv6

Windows 10 Always On VPN User Tunnel XML Configuration Reference File

Windows 10 Always On VPN Device Tunnel XML Configuration Reference File

Windows 10 Always On VPN VPNv2 CSP Reference

IP Protocol Numbers

Always On VPN Traffic Filters and IPv6

Always On VPN Windows Server RRAS Service Does Not Start

Using Traffic Filters with Always On VPN provides administrators the option to configure a true Zero Trust Network Access (ZTNA) solution for their field-based users and devices. By enabling traffic filtering, network access over the Always On VPN connection can be controlled using fine-grained policies. Traffic Filter rules can be configured to restrict access based source and destination IP addresses, protocols, and source and destination ports. Administrators can further restrict access based on the application generating the traffic.

IPv6

While testing these features recently, I learned that the Microsoft Endpoint Manager (formerly Intune) user interface does not appear to support IPv6 when configuring traffic filter rules. As you can see here, the UI explicitly asks for an IPv4 address and complains when entering an IPv6 address in the address field, as shown here.

Interestingly, it is possible to add IPv6 addresses in XML, as follows.

<TrafficFilter>
   <App>
      <Id>Microsoft.RemoteDesktop_8wekyb3d8bbwe</Id>
   </App>
   <Protocol>6</Protocol>
   <RemotePortRanges>3389</RemotePortRanges>
   <RemoteAddressRanges>2001:470:f109::/48</RemoteAddressRanges>
</TrafficFilter>

Connection Failure

Unfortunately, after loading the XML on a test client, the Always On VPN connection fails with the following error message.

“Can’t connect to <ConnectionName>. Catastrophic failure.”

In addition, the Application event log records an event ID 20227 from the RasClient source with the following error.

“The user <UserName> dialed a connection name <ConnectionName> which has failed. The error code returned on failure is -2147418113.”

Workaround

At this time, the only known workaround is to update the configuration on the RRAS server to use IPv4 addressing for VPN clients.

Summary

Unfortunately, IPv6 is still a second-class citizen when it comes to Always On VPN. Although enabling IPv6 works well in most common deployment scenarios, the Microsoft Endpoint Manager management console often fails to accept IPv6 entries in IP address fields. In addition, some advanced features such as traffic filtering are incompatible with IPv6.

Additional Information

Windows 10 Always On VPN and Zero Trust Network Access (ZTNA)

Windows 10 Always On VPN Windows Server RRAS Service Does Not Start

Always On VPN Updates for RRAS and IKEv2

Always On VPN Updates for RRAS and IKEv2

Many users have reported connection stability issues using Windows Server 2019 Routing and Remote Access Service (RRAS) and the IKEv2 VPN protocol. Specifically, there have been reports of random disconnects for which the connection cannot be re-established for an extended period. At the same time, other VPN connections may work without issue.

KB5003703

Microsoft has identified an issue in RRAS where the RemoteAccess service enters DoS protection mode, limiting incoming IKEv2 connection attempts. They released an update on June 15 (OS Build 17763.2028) that addresses this issue. Previously, the only workaround was to restart the IKEEXT service, which was highly disruptive if performed during peak hours.

No More Files

In addition, this update includes another Always On VPN-related fix for Windows 10 1809 clients. An Always On VPN user tunnel connection may fail, with an error message stating, “There are no more files.” The problem can occur after an existing user’s certificate is automatically renewed.

Additional Information

Microsoft Update June 15, 2021 KB5003703 (OS Build 17763.2028)

Always On VPN Class-Based Default Route and Intune

`Always On VPN Class-Based Default Route and IntuneIn a recent post, I described how to configure routing for Windows 10 Always On VPN clients. In that article, I shared guidance for disabling the class-based default route in favor of defining specific routes for the VPN client. While this is easy enough to do when you use custom XML (deployed via PowerShell, SCCM, or Intune), there is a known limitation when using the native Intune UI that could present some challenges.

Intune VPN Profile Configuration

Defining specific routes is easy to do in Intune using the native VPN configuration profile. In the Configuration settings expand Split Tunneling and click Enable. The administrator can then add routes by entering their Destination prefix and Prefix size, as shown here.

Always On VPN Class-Based Default Route and Intune

Class-Based Default Route

The limitation with using Intune to configure routes is that there is currently no option to disable the class-based default route as there is with custom XML. This means the routes shown in the example above will be added to the client, but the class-based route will also be added automatically, as shown here (class-based default route highlighted with the arrow).

Always On VPN Class-Based Default Route and Intune

Considerations

In most cases, the inclusion of the class-based default route along with the administrator-defined routes will not be a problem. However, in some scenarios, it could yield unexpected results. Specifically, Always On VPN clients may have unintended access to some networks over the VPN tunnel. This is most significant for the Always On VPN device tunnel, where it is common to limit access to only specific resources using individual host routes.

Workaround

Today there is no option to disable the class-based default route using the native Intune UI. Your only option is to deploy the Always On VPN profile using custom XML, as described here.

Additional Information

Deploying Windows 10 Always On VPN with Intune and Custom XML

Deploying Windows 10 Always On VPN Device Tunnel with Intune and Custom XML

Windows 10 Always On VPN Routing Configuration

Windows 10 Always On VPN Device Tunnel Operation and Best Practices

Always On VPN Windows Server RRAS Service Does Not Start

Always On VPN Windows Server RRAS Service Does Not StartAdministrators configuring a Windows Server Routing and Remote Access Service (RRAS) server to support Windows 10 Always On VPN connections may encounter an issue where the RemoteAccess service fails to start. Attempts to start the service might seem to work at first, but the service immediately stops again.

Troubleshooting

On the RRAS server, the Services management console (services.msc) or PowerShell Get-Service command shows the RemoteAccess service as being stopped. Attempts to start the service result in failure.

Always On VPN Windows Server RRAS Service Does Not Start

Event Log

Looking at the System event log on the RRAS server shows an error with event ID 7024 from the Service Control Manager source indicating “The Routing and Remote Access service terminated with the following service-specific error: A device attached to the system is not functioning.

Always On VPN Windows Server RRAS Service Does Not Start

Resolution

This issue is commonly caused when IPv6 is disabled on the server via the registry. To verify, open the registry editor on the RRAS server and navigate to the following location.

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters

If the DisabledComponents value is present and set to anything other than 0, set it to 0 or simply delete the DisabledComponents value completely and reboot the server.

Always On VPN Windows Server RRAS Service Does Not Start

The following PowerShell command can be used to remove the DisabledComponents value.

Remove-ItemProperty HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters -Name DisabledComponents

Additional Information

IPv6 Recommended Reading for Always On VPN and DirectAccess Administrators

Guidance for Configuring IPv6 in Windows for Advanced Users (Microsoft)

Always On VPN Fails with Windows 10 2004 Build 610

Updated 11/10/2020: Microsoft update KB4586781 has resolved the connectivity issues described in this post. If you had previously installed update KB4580364, please update to KB4586781 immediately.

A recent preview update for Windows 10 2004 has broken Always On VPN. Specifically, after installing the latest Preview update for Windows 10 2004 (KB4580364), Always On VPN connections will fail to connect automatically. They can be established manually, however.

Affected Builds

This issue affects Windows 10 2004 with build 19041.610 and 19042.610.

Always On VPN Fails with Windows 10 2004

Workaround

The only workaround currently is to remove this update.

Caveat

Although this is a “preview” update and an optional installation, it is important to know that preview updates are released in the next “patch Tuesday” release. Administrators are advised to carefully consider delaying the implementation until additional testing has been completed.

Additional Information

October 29, 2020 – KB4580364 (OS Builds 19041.610 and 19042.610) Preview Update

Windows 10 Always On VPN Updates for Windows 10 2004

%d bloggers like this: