Preventing Port Exhaustion on Entra Private Network Connector Servers

Microsoft Entra Private Access is a powerful zero-trust network access solution that is remarkably simple to install and configure. Administrators can quickly install the Global Secure Access (GSA) agent on their endpoints, then install the Entra Private Network Connector to enable secure remote access to private, internal resources. However, the ease with which Entra Private Access can be configured can potentially lead to connectivity issues in some scenarios. This post demonstrates how to diagnose port exhaustion issues and expand the available port range to address them.

Entra Private Network Connector

The Entra Private Network Connector is a key component of the Entra Private Access solution. The Private Network Connector is essentially the old Azure Application Proxy, enhanced to support TCP and UDP applications in addition to HTTP-based web applications. It is installed on an on-premises Windows server to provide GSA clients with access to internal data and applications.

Network Connectivity

The GSA client is not a virtual network adapter like most traditional VPN clients. Instead, the GSA client installed on the client operates as a filter driver in the network stack, selectively intercepting traffic and tunneling it over the GSA tunnel based on configured policy. As such, it does not appear as a network adapter in the operating system and does not have its own IP address.

Translation

When traffic from the GSA client is routed over the Entra Private Network Connector, the traffic egressing from the connector server to the internal network is effectively translated. That is, the source IP address of traffic destined for an internal resource is the connector server’s IP address, not the client’s original source IP address.

Port Exhaustion

The ephemeral port range on Windows servers spans from 49152 to 65535, leaving only 16,384 ports available. This can easily be exhausted when many clients are connected to a single Entra Private Network Connector server. This pool can also be depleted by poorly written or badly behaving applications that needlessly open many socket connections to internal resources.

Troubleshooting

Administrators can view the ephemeral port configuration for both TCP and UDP by running the following commands.

netsh.exe interface ipv4 show dynamicportrange protocol=tcp

netsh.exe interface ipv4 show dynamicportrange protocol=udp

To determine if port exhaustion is an issue, open an elevated PowerShell command window and run the following command.

Get-NetTcpConnection | Where-Object State -match ‘established’ | Measure-Object

Next, run the following PowerShell command to identify the number of ports consumed exclusively by the Entra Private Network Connector.

$ProcessId = Get-CimInstance -ClassName win32_service | Where-Object Name -eq ‘WAPCSvc’ | Select-Object -ExpandProperty ProcessID

Get-NetTCPConnection | Where-Object { $_.State -match ‘established’ -and $_.OwningProcess -eq $ProcessId } | Measure-Object

If the number of ports consumed by the Entra Private Network Connector approaches the upper limit of available ports, administrators should increase the ephemeral port range to ensure the connector server operates reliably.

Note: Use the Get-NetUdpEndpoint PowerShell command to monitor UDP port consumption on Entra Private Network Connector servers.

Resolution

To increase the ephemeral port range on the Entra Private Network Connector server, open an elevated command window and run the following commands.

netsh.exe interface ipv4 set dynamicportrange protocol=tcp startport=10000 numberofports=55535
netsh.exe interface ipv4 set dynamicportrange protocol=udp startport=10000 numberofports=55535
netsh.exe interface ipv6 set dynamicportrange protocol=tcp startport=10000 numberofports=55535
netsh.exe interface ipv6 set dynamicportrange protocol=udp startport=10000 numberofports=55535

Running these commands will increase the number of available ephemeral ports on the server to more than 50,000, well above the default. In most cases, this should be sufficient to handle many GSA client connections. However, administrators are cautioned to monitor port usage on the Entra Private Network Connector servers to ensure continued reliable operation. It may be necessary to deploy additional connector servers to process the existing workload.

Summary

Entra Private Network Connectors can exhaust the default 16,384-port ephemeral range when many GSA clients access internal TCP/UDP resources. Administrators can diagnose the issue by filtering Get-NetTCPConnection results by the WAPCSvc process, then expanding the range to over 50,000 ports using netsh.exe, as shown above. Monitor usage continuously in high-load environments to ensure consistent and stable access. And if you find you need more than 50,000 ports per server, it’s probably time to deploy additional connector servers. 😊

Additional Information

Microsoft Entra Private Access

Entra Private Access Channels are Unreachable

Microsoft Entra private network connectors

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Always On VPN SSTP Load Balancing with Citrix NetScaler ADCThe Internet Key Exchange version 2 (IKEv2) VPN protocol is the protocol of choice when the highest level of security is required for Always On VPN connections. It uses IPsec and features configurable security parameters that allow administrators to adjust policies to meet their specific security requirements. IKEv2 is not without some important limitations, but organizations may insist on the use of IKEv2 to provide the greatest protection possible for remote connected clients. Due to complexities of the IKEv2 transport, special configuration on the Citrix ADC (formerly NetScaler) is required when load balancing this workload.

Special Note: In December 2019 a serious security vulnerability was discovered on the Citrix ADC that gives an unauthenticated attacker the ability to arbitrarily execute code on the appliance. As of this writing a fix is not available (due end of January 2020) but a temporary workaround can be found here.

Load Balancing IKEv2

When an Always On VPN client establishes a connection using IKEv2, communication begins on UDP port 500, but switches to UDP port 4500 if Network Address Translation (NAT) is detected in the communication path between the client and the server. Because UDP is connectionless, custom configuration is required to ensure that VPN clients maintain connectivity to the same backend VPN server during this transition.

Initial Configuration

Load balancing IKEv2 using the Citrix ADC is similar to other workloads. Below are specific settings and parameters required to load balance IKEv2 using the Citrix ADC.

Note: This article is not a comprehensive configuration guide for the Citrix ADC. It assumes the administrator is familiar with basic load balancing concepts and has experience configuring the Citrix ADC.

Service Settings

The load balancing services for IKEv2 VPN will use UDP ports 500 and 4500. Create the service group and assign group members for UDP 500 as follows.

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Repeat the steps above to create the service group for UDP port 4500.

Virtual Server Settings

Two virtual servers are required, one for UDP port 500 and one for UDP port 4500. Ensure that the service group using UDP port 500 is bound to the virtual server using the same port.

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Repeat the steps above to create the virtual service for UDP port 4500.

Service Monitoring

Since IKEv2 uses the UDP protocol, the only option for service monitoring is to use PING, which is configured by default. Ensure that the firewall on the VPN server allows inbound ICMPv4 and ICMPv6 Echo Request. The default PING monitor on the Citrix ADC will ping the resource every 5 seconds. If a different interval is required, the administrator can edit the PING monitor and bind that to the service or service group as necessary.

Persistency Group

A Persistency Group on the Citrix ADC will be configured to ensure that IKEv2 VPN client requests from the same client are always routed to the same backend server. Follow the steps below to create a Persistency Group and assign it to both IKEv2 virtual servers created previously.

  1. In the Citrix ADC management console expand Traffic Management > Load Balancing > Persistency Groups.
  2. Click Add.
  3. Enter a descriptive name for the Persistency Group.
  4. Select SOURCEIP from the Persistence drop-down list.
  5. Next to the Virtual Server Name section click the Add button.
  6. Add both previously configured IKEv2 virtual servers for UDP 500 and 4500.
  7. Click Create.

Always On VPN IKEv2 Load Balancing with Citrix NetScaler ADC

Use Client IP

To ensure reliable connectivity for IKEv2 VPN connections it is necessary for the VPN server to see the client’s original source IP address. Follow the steps below to configure the Service Group to forward the client’s IP address to the VPN server.

  1. In the Citrix ADC management console expand System, click Settings, and then click Configure Modes.
  2. Select Use Subnet IP.
  3. Click Ok.Always On VPN IKEv2 Load Balancing and NAT
  4. Expand Traffic Management, click Load Balancing, and then click Service Groups.
  5. Select the IKEv2 UDP 500 Service Group.
  6. Click Edit in the Settings section.
  7. Select Use Client IP.
  8. Repeat these steps on the IKEv2 UDP 4500 Service Group.Always On VPN IKEv2 Load Balancing and NAT

Note: Making the above changes will require configuring the VPN server to use the Citrix ADC as its default gateway.

Additional Information

Windows 10 Always On VPN IKEv2 Load Balancing and NAT

Windows 10 Always On VPN SSTP Load Balancing with Citrix NetScaler ADC

Windows 10 Always On VPN IKEv2 Features and Limitations

Windows 10 AlWAYS On VPN and IKEv2 Fragmentation

Windows 10 Always On VPN IKEv2 Security Configuration

Windows 10 Always On VPN Certificate Requirements for IKEv2

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMasterA recent update to the Kemp LoadMaster load balancer may cause failed connections for Always On VPN connections using IKEv2. SSTP VPN connections are unaffected.

Load Balancing IKEv2

When using the Kemp LoadMaster load balancer to load balance IKEv2, custom configuration is required to ensure proper operation. Specifically, the virtual service must be configured to use “port following” to ensure both the initial request on UDP port 500 and the subsequent request on UDP port 4500 are sent to the same real server. This requires the virtual service to be configured to operate at layer 7. Detailed configuration guidance for load balancing IKEv2 on the Kemp LoadMaster load balancer can be found here.

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Issues with LMOS 7.2.48.0

A recent release of the Load Master Operating System (LMOS) v7.2.48.0 introduced a bug that affects UDP services configured to operate at layer 7, which includes IKEv2. This bug breaks Always On VPN connections using IKEv2, resulting in failed connections. When this occurs, the administrator may encounter an error 809 message for device tunnel or user tunnel.

Always On VPN IKEv2 Load Balancing Issue with Kemp LoadMaster

Update Available

Administrators who use the Kemp LoadMaster load balancer to load balance Always On VPN IKEv2 connections and have updated to LMOS 7.2.48.0 are encouraged to update to LMOS 7.2.48.1 immediately. This latest update includes a fix that resolves broken IKEv2 load balancing for Always On VPN. Once the LoadMaster has been updated to 7.2.48.1, Always On VPN connections using IKEv2 should complete successfully.

Additional Information

Windows 10 Always On VPN IKEv2 Load Balancing and NAT

Windows 10 Always On VPN IKEv2 Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN SSTP Load Balancing with Kemp LoadMaster Load Balancer

Windows 10 Always On VPN Load Balancing with Kemp LoadMaster in Azure

Windows 10 Always On VPN Load Balancing Deployment Guide for Kemp Load Balancers