Always On VPN Troubleshooting with Windows Packet Monitor PktMon.exe

When troubleshooting Always On VPN, taking a network packet capture or network trace is sometimes required to identify the root cause of a problem. After all, Packets Don’t Lie™. There are numerous ways to capture packets. Many administrators will install Wireshark for this purpose. However, Windows has a native packet capture tool called PktMon.exe that offers many advantages over Wireshark.

Wireshark

Many Always On VPN administrators will be familiar with Wireshark. Wireshark is a popular open-source network protocol analyzer that enables the capture and analysis of network traffic for troubleshooting. A packet capture driver must first be installed to capture network traffic with Wireshark. Typically, administrators will install Npcap, which is part of the default installation of Wireshark. Installing a capture driver poses a potential problem, as the administrator must install software on the target device before capturing traffic. Installing software may not always be feasible or possible. Fortunately, there’s an alternative.

PktMon.exe

The Windows Packet Monitor (PktMon.exe) is a built-in command-line tool first introduced in Windows 10 1809 and Windows Server 2019. It is designed to capture network traffic on Windows servers and client systems. This native lightweight tool is ideal for collecting network traces for offline analysis.

Capture All Interfaces

The most common scenario for PktMon.exe is to capture data for offline analysis. Use the following command to capture all network traffic on all active network interfaces.

PktMon.exe start –capture –file c:\capture.etl –pkt-size 0 –comp nics –flags 0x10

The command breaks down as follows:

–capture – captures network traffic

–file – the path of the file to save the data to

–pkt-size 0 – captures the full packet (not truncated)

–comp nics – captures traffic on all active network interfaces

–flags 0x10 – captures the raw packet

After reproducing the issue, you can stop the trace by running the following command.

PktMon.exe stop

Capture Specific Interface

Administrators may wish to capture traffic on a specific network interface instead of all active network interfaces. In this example, I have a multi-homed VPN server and want to capture traffic on only the DMZ interface. To do this, use PktMon.exe to enumerate all interfaces using the following command.

PktMon.exe list

Note: The output of PktMon.exe filter list does not include information that easily maps to existing network interfaces. I suggest also running the Get-NetAdapter PowerShell command to view detailed information about network interfaces. You can use this information to select the correct Network ID for PktMon.exe filtering.

Next, change the value of –comp nics in the command referenced above to –comp <Network ID>. Here’s an example.

PktMon.exe start –capture –file c:\capture.etl –pkt-size 0 –comp 62 –flags 0x10

Filtering

It’s also possible to use PktMon.exe to capture network traffic selectively. Filtering allows you to narrow the capture to relevant traffic, making analysis easier and faster. Add a filter, then start a trace to restrict data capture to traffic that matches the defined filters. You can add one or more filters to apply to the capture. Here are a few examples.

Protocols and Ports

Let’s say you are troubleshooting a device tunnel connection and want to see only IKEv2 traffic. The following filter will restrict the network capture to only the IKEv2-related protocols and ports.

PktMon.exe filter add IKEv2 -t UDP -p 500
PktMon.exe filter add IKEv2 -t UDP -p 4500

IP Address

The following filter will capture data that includes the specified IP address in the source or destination address field.

PktMon.exe filter add VPN1 -i 172.21.12.50

You can also specify IP address subnets using their CIDR notation.

PktMon.exe filter add Subnet1 -i 172.16.0.0/16

View and Clear Filters

You can view configured filters using the following command.

PktMon.exe filter list

You can remove configured filters using the following command. Use with caution, as this removes ALL filters!

PktMon.exe filter remove

Reference

You’ll find a complete list of PktMon.exe filters here.

Analysis

PktMon.exe outputs captured data in ETL format. Administrators can convert captured data to the standard PCAP format by running the following command.

PktMon.exe etl2pcap <path of trace file>

This command converts the file from ETL to PCAPNG format. Administrators can then open the capture in Wireshark for further detailed analysis.

Display Only

PktMon.exe can be configured to display network traffic in the console for quick troubleshooting. Console traffic display can be helpful for those scenarios where a quick check to validate traffic is reaching a particular destination is required. Here’s an example.

PktMon.exe start –capture –pkt-size 0 –comp nics –flags 0x10 -m real-time

Note: In the example above, I applied a traffic filter to limit the capture to only SSTP traffic (TCP 443).

Limitations

One crucial limitation of PktMon.exe is that it doesn’t support persistent network captures that survive a reboot. Persistent captures can be helpful when troubleshooting a device tunnel connection or slow logons. In this scenario, you must use netsh.exe.

netsh.exe trace start capture=yes tracefile=c:\tracefile.etl persistent=yes

<reboot>

netsh.exe trace stop

Although PktMon.exe supports the ‘etl2pcap’ switch, it does NOT work for converting .etl files generated with netsh.exe. To convert captures created with netsh.exe, use the open-source etl2pcapng tool.

Learn More

PktMon.exe has many different uses. This post barely scratches the surface of what PktMon.exe can do. PktMon.exe comes with robust help, accessible by adding the ‘help’ switch to commands. Here are some examples.

PktMon.exe start help
PktMon.exe filter add help

Be sure to view the online help to explore various options for capturing and logging to meet your specific needs.

Summary

PktMon.exe is a native command-line utility in Windows that provides a lightweight solution for capturing network traffic, making it particularly useful for Always On VPN troubleshooting. Key functionalities include full-packet captures, selective filtering by protocol, port, or IP address, and conversion of ETL files to PCAPNG format for analysis in tools like Wireshark. Real-time traffic displays are also supported for quick diagnostics. While effective for many scenarios, PktMon.exe lacks support for persistent captures across reboots, for which netsh.exe is recommended. The techniques outlined above offer administrators a practical, software-free approach to deep packet inspection for troubleshooting Always On VPN issues.

Have you used PktMon.exe for network troubleshooting? Feel free to share tips and tricks in the comments section below!

Additional Information

Getting Started with Windows Packet Monitor (PktMon.exe)

PktMon.exe Filter Reference

Open-source Etl2pcap for netsh.exe captures

Always On VPN Training at TechMentor Redmond 2019

TechMentor Redmond 2019I’m pleased to announce that I’ll be delivering two technical training sessions at this year’s TechMentor Redmond event. This event takes place on the Microsoft campus in Redmond, WA August 5-9, 2019. I’ll be presenting two sessions on Thursday, August 8. They are:

TH16 – Wireshark Essentials: Your First Day with Wireshark

During this session you will learn essential techniques for optimizing packet analysis using Wireshark. Topics will include filter and display tips, workspace oganization, using shortcuts for common tools, and configuring Wireshark profiles. I’ll also touch upon some advanced techniques such as graphing and geography database integration.

TH20 – Always On VPN: The Good, the Bad, and the Ugly!

During this session you will gain a full understanding of Always On VPN including and how it compares with its predecessor, DirectAccess. I’ll share detailed information about this new technology, and how it best fits in to your organizations mobility strategy. Always On VPN has some important advantages over DirectAccess, and some challenging drawbacks. I’ll explain everything good, bad, and even the ugly.

TechMentor Redmond 2019

 

Don’t miss out on this fantastic event. Register now to take advantage of early bird savings, which end June 7. Hope to see you there!

Always On VPN and Network Policy Server (NPS) Load Balancing

Always On VPN and Network Policy Server (NPS) Load BalancingLoad balancing Windows Server Network Policy Servers (NPS) is straightforward in most deployment scenarios. Most VPN servers, including Windows Server Routing and Remote Access Service (RRAS) servers allow the administrator to configure multiple NPS servers for redundancy and scalability. In addition, most solutions support weighted distribution, allowing administrators to distribute requests evenly between multiple NPS servers (round robin load balancing) or to distribute them in order of priority (active/passive failover).

The Case for NPS Load Balancing

Placing NPS servers behind a dedicated network load balancing appliance is not typically required. However, there are some deployment scenarios where doing so can provide important advantages.

Deployment Flexibility

Having NPS servers fronted by a network load balancer allows the administrator to configure a single, virtual IP address and hostname for the NPS service. This provides deployment flexibility by allowing administrators to add or remove NPS servers without having to reconfigure VPN servers, network firewalls, or VPN clients. This can be beneficial when deploying Windows updates, migrating NPS servers to different subnets, adding more NPS servers to increase capacity, or performing rolling upgrades of NPS servers.

Traffic Shaping

Dedicated network load balancers allow for more granular control and of NPS traffic. For example, NPS routing decisions can be based on real server availability, ensuring that authentication requests are never sent to an NPS server that is offline or unavailable for any reason. In addition, NPS traffic can be distributed based on server load, ensuring the most efficient use of NPS resources. Finally, most load balancers also support fixed or weighted distribution, enabling active/passive failover scenarios if required.

Traffic Visibility

Using a network load balancer for NPS also provides better visibility for NPS authentication traffic. Most load balancers feature robust graphical displays of network utilization for the virtual server/service as well as backend servers. This information can be used to ensure enough capacity is provided and to monitor and plan for additional resources when network traffic increases.

Configuration

Before placing NPS servers behind a network load balancer, the NPS server certificate must be specially prepared to support this unique deployment scenario. Specifically, the NPS server certificate must be configured with the Subject name of the cluster, and the Subject Alternative Name field must include both the cluster name and the individual server’s hostname.

Always On VPN and Network Policy Server (NPS) Load Balancing

Always On VPN and Network Policy Server (NPS) Load Balancing

Create Certificate Template

Perform the following steps to create a certificate template in AD CS to support NPS load balancing.

  1. Open the Certificate Templates management console (certtmpl.msc) on the certification authority (CA) server or a management workstation with the remote administration tools installed.
  2. Right-click the RAS and IAS Servers default certificate template and choose Duplicate.
  3. Select the Compatibility tab.
    1. Select Windows Server 2008 or a later version from the Certification Authority drop-down list.
    2. Select Windows Vista/Server 2008 or a later version from the Certificate recipient drop-down list.
  4. Select the General tab.
    1. Enter a descriptive name in the Template display name field.
    2. Choose an appropriate Validity period and Renewal period.
    3. Do NOT select the option to Publish certificate in Active Directory.
  5. Select the Cryptography tab.
    1. Choose Key Storage Provider from the Provider Category drop-down list.
    2. Enter 2048 in the Minimum key size field.
    3. Select SHA256 from the Request hash drop-down list.
  6. Select the Subject Name tab.
    1. Select the option to Supply in the request.
  7. Select the Security tab.
    1. Highlight RAS and IAS Servers and click Remove.
    2. Click Add.
    3. Enter the security group name containing all NPS servers.
    4. Check the Read and Enroll boxes in the Allow column in the Permissions for [group name] field.
  8. Click Ok.

Perform the steps below to publish the new certificate template in AD CS.

  1. Open the Certification Authority management console (certsrv.msc) on the certification authority (CA) server or a management workstation with the remote administration tools installed.
  2. Expand Certification Authority (hostname).
  3. Right-click Certificate Templates and choose New and Certificate Template to Issue.
  4. Select the certificate template created previously.
  5. Click Ok.

Request Certificate on NPS Server

Perform the following steps to request a certificate for the NPS server.

  1. Open the Certificates management console (certlm.msc) on the NPS server.
  2. Expand the Personal folder.
  3. Right-click Certificates and choose All Tasks and Request New Certificate.
  4. Click Next.
  5. Click Next.
  6. Select the NPS server certificate template and click More information is required to enroll for this certificate link.
  7. Select the Subject tab.
    1.  Select Common name from the Type drop-down list in the Subject name section.
    2. Enter the cluster fully-qualified hostname (FQDN) in the Value field.
    3. Click Add.
    4. Select DNS from the Type drop-down list in the Alternative name section.
    5. Enter the cluster FQDN in the Value field.
    6. Click Add.
    7. Enter the NPS server’s FQDN in the Value field.
    8. Click Add.
      Always On VPN and Network Policy Server (NPS) Load Balancing
  8. Select the General tab.
    1. Enter a descriptive name in the Friendly name field.
  9. Click Ok.
  10. Click Enroll.

Load Balancer Configuration

Configure the load balancer to load balance UDP ports 1812 (authentication) and 1813 (accounting). Optionally, to ensure that authentication and accounting requests go to the same NPS server, enable source IP persistence according to the vendor’s guidance. For the KEMP LoadMaster load balancer, the feature is called “port following”. On the F5 BIG-IP it is called a “persistence profile”, and on the Citrix NetScaler it is called a “persistency group”.

Additional Information

Always On VPN IKEv2 Load Balancing with KEMP LoadMaster

Always On VPN Hands-On Training Classes in U.S. and Europe