CRLcheck.exe Certificate Revocation List Check Tool to verify all CRL and OCSP on Windows client

CRLcheck.exe Certificate Revocation List Check Tool to automatic verify CRL and OCSP internet reachability of all your EXE files that your client runs.

Over the past 20 years, I have personally witnessed how Certificate Revocation on Windows systems is often underestimated, even within large enterprises. This issue significantly affects both client and server systems if not managed correctly. It can manifest in various ways, from start-up delays of applications to failed upgrades on servers, to inexplicable occurrences that happen sporadically. Worst of all, these issues can be challenging to reproduce consistently. Problems like these often persist for weeks when left unaddressed. By the time somebody delves into the problem in detail, the issue may have disappeared, only to resurface days later.

In numerous instances, workplaces experienced start up delays, with security software like Antivirus wrongly accused of causing these delays.

This is how this happens and we have seen several times from SBS to enterprise:

Moreover, during OS deployments such as Windows 7, 10, and 11, we have encountered several large-scale deployments gone awry due to delays in driver installations, particularly for storage or NIC drivers, leading to clients being unable to reach the Certificate Revocation List (CRL) Fully Qualified Domain Name (FQDN) through the WAN.

Those involved in patching server systems, such as Windows Server or Exchange Server, often observe varying patching times or overall slowness. These issues may be directly linked to the endpoint’s inability to reach specific Certificate Revocation lists on the internet.

While there are command-line tools like certutil.exe available to analyse these problems, we found that they were not sufficient for our needs. Consequently, we made the decision to develop our own tool to address these challenges effectively.

What crlcheck.exe does?

• Find all EXE files in a common directory, processes running as users, or a directory specified by the user
• Check if the executable is digitally signed and has CRL + OCSP listed
• Verify if it can download the CRL + OCSP from the Internet
• Export all .CER and .CRL files, along with decrypted information found in each CRL list
• Check the cached CRL + OSCP your client often uses and do the same

What crlcheck.exe generates at the end / What is the Target:

You have a list of FQDN Fully qualified Domain Names that your endpoint references regarding digital signing and you can hand over the list to your Security or
network–engineer to verify those URL and make sure they are opn on firewall.

CRLcheck.exe is a tool designed to collect executables (*.exe) from well-known paths on your client.

It checks whether they are digitally signed with a certificate and then retrieves the Certificate Revocation Lists (CRL) and OCSP for each signed file it finds.

It then verifies if the client can successfully request and download that file from the Internet.

 What we talk about shown in a certificate itself:

Why is it essential to check CRL and why this should work?

1. Security reason
   
2. Performance reason
   

For software suppliers, it is a best practice to code-sign their executables with commercial certificates. If those certificates are compromised or stolen, the operating system provides an option to check for such incidents through the Certificate Revocation List (CRL) over HTTP (Port 80).

Because it is not possible to recall a certificate, the certificate revocation list details certificates signed by valid CA that should no longer be trusted. Certificates may be revoked for many reasons, such as if the certificate was issued erroneously or if the private key of a valid certificate has been compromised.

This process functions only when the client is connected to the internet and, in an enterprise setting, is not behind a proxy or firewall that blocks these requests.

If you have an isolated environment with no internet access, you may even need to supply those CRL files on an internal web server so that some software works flawlessly.

Why this is important in Enterprise or where you have Proxy and Web filters

Often, browser traffic is directed to a Web Security Proxy, while the client (system) traffic goes through another route, potentially being blocked. In such cases, the CRL process may lead to significant delays when starting an executable. In most Windows systems, you will observe a blue circle beside the mouse pointer and the application you just initiated when this delay occurs.

Please see sample from the field at end of this post for Microsoft Server, Fortinet Forticlient, Citrix, Exchange etc.

This does NOT affect me because we have a category on our firewall with all the CRL+OCSP in a realtime list:

Yes, but that list is not always ajour/uptodate. We had a case where it affected OS deployment with INTEL drivers for storage.

https://www.butsch.ch/post/missing-entry-in-fortigate-application-filter-rootcertificateurl-and-ocsp-source-of-w10-setup-failing/

Why this happens

Firewall, Proxy, or security engineers sometimes overlook or may not be aware of this process, resulting in potential performance issues on client machines. They isolate clients in VLANs or behind a Proxy and forget about the Certificate Revocation List (CRL) process. The controlled traffic is routed to the proxy, but some applications or drivers hit the WAN gateway directly and are blocked. This is good and wanted, but some traffic must be allowed. It is critical that in such cases, people know which CRL/OCSP address an application that is signed uses. This tool will help you find those addresses.

Why this is always and issue which lands at security engineer’s desk instead of PKI-engineer which does certificates?

When encountering performance issues, it is common for IT professionals to mistakenly attribute blame to security tools like Antivirus. However, in many cases, these allegations are misdirected.

Modern security solutions, including Antivirus and Intrusion Prevention Systems (IPS), rigorously inspect certificates to assess file risk. This comprehensive examination often triggers multiple Certificate Revocation List (CRL) and Online Certificate Status Protocol (OCSP) lookups.

Consider the scenario of launching an application such as myapp.exe. In this situation, a sophisticated AV solution not only scrutinizes the application’s code signature but also investigates auxiliary files like DLLs and other executables, along with their associated chains. This meticulous process can initiate a series of CRL lookups, inadvertently causing delays during application start up or even resulting in application hang-ups.

Through our experiences, we have observed that security endpoints are frequently accused of causing performance slowdowns, when in reality; the root cause often lies in the resource-intensive nature of Certificate Revocation checking. Disabling or uninstalling the antivirus may appear to resolve the issue, but it merely halts the CRL lookup process, leading to inaccurate attribution of the problem.

Furthermore, disabling a third party Antivirus or shutting it down often results in a fallback to Windows Defender. This transition, however, does not necessarily resolve the underlying issue, as both Windows Defender and the third-party AV solution may cease certificate checking operations, falsely indicating that the problem has been resolved.

The tool running:

Download:

 Crlcheck.exe should run on foollwing os and endpoints:

 File information:

Security AV check done for file crlcheck.exe

I spent half of my life with endpoint security products and Threat Sandboxing. Mainly those two RED flags in virustotal are based on the fact that our own EXE code is not digital signed 😉

The files has been verified in ATD/TIS Trellix Sandboxes and also on Fortisandbox.

 Download version 1.6.5.0 from https://www.butsch.ch/wp-content/uploads/tools/crlcheck/1.6.5.0/crlcheck.exe

Check on TIS/ATD Sandbox of file “7c215e15dd2f007bb0cbf86598122798” crlcheck.exe

Do further search or open found CRL or CER files in Windows tools

Check the cache of the user like with certutil -urlcache

Checking if your client can reach the found CRL + OCSP URL’s

Finding DENIED or INVALID CRL + OCSP URL:

In this sample, it was an old Adobe Elements software, which is out of support:

Automatic decrypt / decode CRL raw files into readable textfiles:

Check if your client uses REVOKED certificates which would be a security concern

Easy browse through certificates information of executables on your client

Generates CSV/SSV exports for your security/firewall/proxy teams

You can also investigate another common source of network problems, such as the Maximum Transmission Unit (MTU)/Maximum Segment Size (MSS) size in your network or Proxy settings and conduct a download speed test.

Check MTU size per interface

Check PROXY settings, which is also relevant for CRL checks:

Because MS tries to explain:

Sample usage cases from the daily field:

Sample case Fortinet Forticlient VPN fail,
stuck at 40% percent Status

 Forticlient VPN reaches 40% and is unable to connect when a user blocks CRL on private firewall. Fortinet does what other should they warn the user if that happens. Sadly this happens in the background of the VPN client (hidden) and you

Endless try to find out the reason.

https://community.fortinet.com/t5/Support-Forum/SOLVED-Forticlient-VPN-SSL-Stops-at-40/td-p/249616/page/2

https://community.fortinet.com/t5/Support-Forum/VPN-Client-stuck-at-40-with-certificate-error/m-p/222766

With crlcheck.exe of the client you would have seen that the VPN client in that environment is unable to read the request OCSP adress from sectigo.com

Sample case Fortinet Fortigate enterprise environment

Customer 1000+ enterprise had sporadic delay on all kind of applications. Whenever external for the applications analysed the issue it was non-present just to come back.

The effect was discovered after the network team migrated to a newer Fortigate model and integrated an internal firewall between different VLANs, implementing a serial hook-up and a McAfee Web Gateway cluster behind the Fortigate Web/IPS filter.

Had the person in charge had our tool at hand, this issue may have been avoided. The main focus of the network team was on directing attention towards the client management team and client security team. They even considered replacing the AV solution.

The team relied heavily on built-in service/protocol lists from Fortigate, which, at that time, did not include all the necessary Certificate Revocation List (CRL) and Online Certificate Status Protocol (OCSP) addresses.

This oversight was based on too much trust to built-in groups on Fortinet, which were expected to cover those Fully Qualified Domain Names (FQDNs).

Service list built in and updated realtime by FortiGuard

Result:

We solved this with making our own rules:

 You can see also the same effect here:

https://www.butsch.ch/post/missing-entry-in-fortigate-application-filter-rootcertificateurl-and-ocsp-source-of-w10-setup-failing/

Sample case Microsoft Windows Server

The server is unable to reach a certain OCSP FQDN because heavy isolated. Connection fails from or to a another endpoint via TLS.

There is an effect on the CA 2022 itself that has nothing to do with this.

Sample Microsoft Exchange Server

https://www.butsch.ch/post/the-certificate-is-invalid-for-exchange-server-usage-exchange-2010-sanuc/

On one of my favorite Exchange blogs i used the last 10 years 😉

https://practical365.com/exchange-2010-certificate-revocation-checks-and-proxy-settings/

Sample case Citrix

ICA launch delay issue after upgrading to Workspace app 2203 CU2 (citrix.com)

https://support.citrix.com/article/CTX492837/ica-launch-delay-issue-after-upgrading-to-workspace-app-2203-cu2

After upgrading to Workspace app 2203 CU2, customer may experience ICA launch delay issues

Problem Cause

From Workspace app version 2203 CU2, cryptnet.dll will be loaded for checking Certificate Revocation List.

If CryptoAPI is NOT able to access any internet CRL server due to customer network security requirements,

it will take more time for CRL checking. It leads delayed launch of wfica32.exe

Sample case Microsoft M365 Intunes

https://www.butsch.ch/post/mastering-firewalls-for-intunes-and-autopilot-success-fqdn-ip-crl-to-get-intunes-running/

People think with M365 and Intunes Autopilot all problems would be easy? Well kind of until you run into a problem with CRL and want to do an Update of a driver via Intunes.

Sample case Ivanti, modern workplace, Workspace control