VMWARE Workspace One, iPhone, iOS 17, Enterprise Wipe after Update. Issue known since 07.07.2023

by butsch 27. September 2023 17:50




VMWARE Workspace One, iPhone, iOS 17, Enterprise WIPI after Update. Issue known since 07.07.2023 to 27.09.2023 and still affects endusers

Well; looks like some things change too fast in the modern world workplace.

We wish to inform you of a current concern pertaining to Workspace ONE UEM-enrolled iOS devices that have undergone an upgrade to iOS 17. It has come to our attention that these devices are occasionally being incorrectly identified as compromised following the upgrade. The VMware development team is actively engaged in addressing this matter.


Upon the upgrade to iOS 17, Workspace ONE UEM-enrolled iOS, mobile devices are intermittently detected as compromised by the Workspace ONE mobile SDK, leading to the execution of an enterprise wipe. This process results in the removal of all enterprise data from these devices and their disconnection from the system.


Our product team is diligently working towards resolving this issue. For devices operating on iOS versions lower than iOS 17, the fix is anticipated to be automatically applied and does not necessitate an SDK/Productivity Apps/UEM upgrade. We estimate that the issue will be resolved by the 29th of September 2023.

Possible Workaround:

To mitigate this problem, re-enrolling the affected devices should provide a solution. As a temporary measure, administrators can disable the 'Compromise Protection' in the SDK profiles within the UEM console. Please be aware that Compromise Protection is a vital security feature and should be re-enabled once the Workspace ONE team issues the fix for this issue.

Note: Enterprise Wipe and/or Device Wipe commands do not land on the device unless the device is unlocked or connected to a network.

Often, it is recommended to use enterprise wipe rather than deleting the device. When the Delete Device command has been completely processed, all commands associated with that DeviceID are purged from the AirWatch DB, including the Enterprise Wipe Command tied to that device's DeviceID.

Therefore, if the Delete Device operation is performed on a device that is turned off/disconnected from the network, it will not receive the Enterprise Wipe Command issued by the console immediately upon the Delete Device command. This is why it is recommended to make use of the Enterprise Wipe Command as this preserves the device records associated with this device's DeviceID.













VMWare | M365/AZURE

Powershell, Retrieve DisplayName,DisplayVersion,Publisher from Registry or export.REG

by butsch 5. September 2023 18:43


You need and easy way to get an inventory list of software installed on a machine via Powershell. What we need for a list is:


We have two version of the scripts:

Version 1

Use regedit and export the HIVES into .REG Files. Copy to your admin machine an then parse the two files with the script. Use this where you are not allowded to run PS on Servers because of compliance (Signed/external source etc.)

Version 2

Directly access the Registry 32/64BIT Hive Uninstall info on local machine and generate an output.txt file.


Here is how to easily extract the most important information from a .REG export from the UNINSTALL Registry HIVE. We don't want to run the PS directly on the server or via the server because of compliance. So, you can export the Registry Hives from a server as a .REG file, transport them through valid methods to the management machine, and then generate a semicolon-separated list for Excel import.

At the bottom, you will find a version we made if you want to retrieve the info directly from the local machine and directly from the Registry Hives (Without the way over the .REG export) (32 and 64-bit Hives):



Version 1



# www.butsch.ch

# Version 1 Import from .REG File which was exported on machine/server

# V1.0, 03.09.2023, First Release



(Get-Content -Path 'C:\edv\00_modern_workplace\software.reg' -Raw) -split '\r?\n\r?\n' | ForEach-Object {

$UninstallInfo = $_ -split '\r?\n' | ForEach-Object {

$Line = $_ -split '='

if ($Line.Count -eq 2) {


Key = $Line[0].Trim()

Value = $Line[1].Trim()




$DisplayName = $UninstallInfo | Where-Object { $_.Key -eq '"DisplayName"' }

$DisplayVersion = $UninstallInfo | Where-Object { $_.Key -eq '"DisplayVersion"' }

$Publisher = $UninstallInfo | Where-Object { $_.Key -eq '"Publisher"' }

if ($DisplayName -and $DisplayVersion -and $Publisher) {

($DisplayName.Value), ($DisplayVersion.Value), ($Publisher.Value) -join ';'





Windows Registry Editor Version 5.00




[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\Microsoft Edge Update]

"DisplayName"="Microsoft Edge Update"







Here is the version which get the information direct from the local machine:


# www.butsch.ch

# Version 2 direct access from Registry Hives

# V1.0, 03.09.2023, First Release

# This PS will retrieve all information he can find about installed Software 32/64BIT Hives and write the data into a file output.txt

# -----------------------------------------------------------------------------------------------------------------------------------


# Define the Registry paths for both 32-bit and 64-bit programs

$registryPaths = @(





# Initialize an empty array to store the results

$results = @()


# Iterate through the Registry paths


foreach ($path in $registryPaths) {

# Get all subkeys (uninstall entries) under the Registry path

$uninstallKeys = Get-ChildItem -Path $path | Where-Object { $_.PSChildName -match '^{[A-F0-9-]+}' }


# Iterate through each uninstall entry

foreach ($key in $uninstallKeys) {

$properties = Get-ItemProperty -Path "$path\$($key.PSChildName)" -ErrorAction SilentlyContinue


if ($properties -ne $null) {

$entry = $properties.DisplayName

if ($properties.DisplayVersion) {

$entry += ";$($properties.DisplayVersion)"


if ($properties.Publisher) {

$entry += ";$($properties.Publisher)"


$results += $entry





# Output the results to a file

$results | Out-File -FilePath "output.txt"


# Display the results on the console (optional)








Scripting | Client Management | Scripting

Eventviewer, eventvwr.exe commandline filter XML query buildingm (Call and pre filter view with one line)

by butsch 28. July 2023 21:46


Event logs provide valuable insights into system operations, allowing IT professionals to monitor and troubleshoot potential issues. When dealing with Windows event logs, PowerShell is a powerful tool that enables event filtering, but it may not be everyone's preferred choice due to complexity and perceived security concerns. In this blog post, we will explore an alternative method - using the command-line filter of Event Viewer - to easily access Critical, Error, and Warning events from the Application log.

In this command, we construct an XML query that filters events based on severity levels:

Level=1 corresponds to Critical,

Level=2 corresponds to Error, and

Level=3 corresponds to Warning.

The command will open the Event Viewer and present the filtered results, saving valuable time for IT operations and support personnel.

eventvwr.exe /f:"<QueryList><Query Id='0' Path='Application'><Select Path='Application'>*[System[(Level=1 or Level=2 or Level=3)]]</Select></Query></QueryList>\"


Advantages of the Command-Line Filter:

Familiarity: Many seasoned IT professionals prefer using a simple command-line approach they are familiar with, making it easier to find and work with event logs efficiently.

Accessibility: The command-line filter is available on all modern Windows operating systems, eliminating compatibility concerns.

Trustworthy: Since the command-line tool is a built-in Windows feature, there are no worries about third-party dependencies or security issues.


PowerShell Approach:

Before diving into the command-line filter, let's quickly review the PowerShell method. By leveraging the Get-EventLog cmdlet, one can filter events based on severity levels and display them in PowerShell console. The script would look like this:

Get-EventLog -LogName "Application" -EntryType Error, Warning, Information | Where-Object { $_.EntryType -eq "Error" -or $_.EntryType -eq "Warning" -or $_.EntryType -eq "Information" }

But we want it commandline style because most people with long term expierence are used to that tool/console to find and work.

eventvwr.exe /h

What? ;-)


Ok let's try:

Go to XML tab

Cut and paste into notepad


<Query Id="0" Path="Application">

<Select Path="Application">*[System[(Level=1 or Level=2 or Level=3)]]</Select>





If you see this than all fine:


If you see this than you did not read 100% and checked all " > ' and the \ at the end or you try to

call together with /c:application (Channel) and Query (/f:)


Wrong sample:


Some Microsoft Links (One sample is wrong there and I did not understand what he tells at once and the eventvwr.exe /h is really nerdy? ;-)









Client Management | Deployment | Scripting

Azure Application Proxy | Die Eierlegende kostenlose Woll-Milch-Sau um on-premises Server extern erreichbar zu machen

by butsch 13. July 2023 23:00



Azure Application Proxy (Die Eierlegende kostenlose Woll-Milch-Sau um on-premises Server extern erreichbar zu machen)


Es gibt die Kafeemaschine für CH 29.- mit 30 gratis Kapseln, aber eine weitere Kapsel für CHF 3.-.

Es gibt einen Foto Tintenstrahldrucker fuer CHF 19.- aber die Patronen kosten CHF 99.- pro Farbe.

Genau davon reden wir hier nur merkt es keiner.


Was hört man an M365/Azure Schulungen Schönes?


Wieso macht ihr alles so kompliziert? Publiziert doch Eure in-house Server ganz einfach und schnell über einen Azure Application Proxy? Dann braucht ihr all die teure Hardware nicht mehr. Es ist kostenlos bei den meisten MS licence Suites dabei. Zwei, drei Klicks und der interne Intranet-Server steht sicher im Web. Keine DMZ Vmware Cluster, kein DMZ Veeam Server, Keine Segmentation FW keine teuren RSA FMS Lizenzen bei uns ist alles dabei!


Zudem nach dem Fortinet Fortigate SSL VPN Exploit im Juli 2023 erkundigen sich einige wieder, was andere wie machen. Jahrelange sichere und stabile Konzepte sowie Strategien werden plötzlich in Frage gestellt. Dann gab es noch die Citrix Access Gateway Lücken im November 2022 und Juli 2023. Der einte oder andere fängt an zu Grübeln. Das gab es alles schon früher nur redete keiner davon und meist wussten auch niemand davon.


Das liest man über Azure Application Proxy:


  • Einfach zu bedienen. Benutzer können auf die On-Premises-Anwendungen auf die gleiche Weise zugreifen, wie sie Microsoft 365 und andere SaaS-Anwendungen, die mit Azure AD integriert sind, nutzen.


  • Sicher. On-Premises-Anwendungen können Azure's Autorisierungssteuerungen und Sicherheitsanalysen nutzen, einschließlich bedingtem Zugriff und MFA (Multi-Faktor-Authentifizierung). Außerdem erfordert Application Proxy nicht, dass Sie eingehende Verbindungen über Ihre Firewall oder VPN öffnen.


  • Kosteneffizient. Um Application Proxy zu nutzen, müssen Sie die Netzwerkinfrastruktur nicht ändern oder zusätzliche Geräte in Ihrer On-Premises-Umgebung installieren.


Man muss nur folgendes beachten:


Best Practices für Azure Application Proxy:

1. Installieren Sie den Connector-Server physisch in der Nähe der Anwendungsserver, um die Leistung zwischen dem Connector und der Anwendung zu optimieren.

2. Der Connector-Server und die Webanwendungsserver sollten sich im selben Active Directory-Domäne befinden oder sich über vertrauenswürdige Domänen erstrecken.

3. Installieren Sie mehrere Connector-Server lokal, um einen einzelnen Ausfallpunkt zu vermeiden.

4. Konfigurieren Sie FQDN (vollqualifizierter Domänenname) und DNS-Einträge intern und verwenden Sie keine IP-Adressen.

5. Stellen Sie sicher, dass Ihre Connector-Server uneingeschränkten Internetzugang haben und das Microsoft-Netzwerk erreichen können.

6. Verwenden Sie bedingten Zugriff, um den Zugriff auf Ihre Umgebung abzusichern.

7. Konfigurieren Sie SSO (Single Sign-On), um die Benutzererfahrung zu verbessern.



MS Links:






Alles Cloudy? Alles Ersetzen?



 Was ich persönlich davon halte:


  • Für gelegentlichen Zugriff auf KMU-On-Premises-Server, die ich ins Web stellen möchte, ohne komplexe DMZ-Architektur intern: OK Jederzeit (Ticket, Zugang zur Abacus Zeiterfassung, Service Portal, Intranet, irgendetwas Kleines: Super)
  • Für Unternehmen mit über 1'000+ MA, die Hauptanwendungen nutzen, landen wir dort, wo grosse Unternehmen seit Jahren mit WAN zwischen Standorten zu kämpfen haben: LATENZ / SESSION TTL Einstellungen usw. In der Regel setzte man dann im Enterprise Bereich WAN-Optimizer ein auf beiden Seiten welche jeweils den Preis eines Autos hatten.

    https://www.gartner.com/reviews/market/wan-optimization (Fortinet/Cisco/Riverbed...)



  • Azure Application Proxy > JA, man kann mehrere Connector lokal installieren und sie bieten internes Load Balancing. Failover funktioniert, aber wenn einer ausfällt und man ihn nicht abmeldet, dann erhöht sich die Latenz++, z.B. bei RDP weil die Logik dort immer wieder versucht zu connecten. (Man lese im Web von verzweifelten Endkunden welche Citrix Accessgateway ersetzen wollten un den Effekt hatten [Wir bleiben fair: Letzte grössere Lücke Citrix AG NOV 2022 und JUL 2023])
  • Das Erste, was man dann bei verschiedenen Softwareherstellern zu diesem Thema findet, ist, dass man die Timeout-Zeit erhöhen muss (Appliance, Windows Server oder in der App, sofern möglich). Das ist denen noch so recht wenn es eine kostenlose Option gibt um Ihre Lösung WAN tauglich zu machen. Nur der neueste Release ist dann auch für das optimiert?
  • Ist doch nichts kostenlos, oder? Ist doch keiner so naiv und glaubt, dass es bei Microsoft etwas umsonst gibt, ausser Systernals von Mark? Das Ziel ist, dass man alles zu Microsoft bringt und dann feststellt, dass es nicht kostenlos bleibt und Latenz hat? Aber dafür gibt es eine weitere Lösung und die heissst Express Route Direct
  • Latenz? Kein Problem, sagen die Microsoft-Cloud-Sales-Evangelisten. Dafür gibt es ja den allseits bekannten Express Route Direct, für den man bis zu CHF 44'867.-/MONAT zahlen kann. Das ist ein "goldenes" Kabel zu den MS Rechenzentren mit SLA.


  • Für KMU hoffen wir, dass es besser wird, wenn man es CH > CH (Standort Cloud Schweiz) routet? Für Grosskunden mit mehreren internationalen Standorten bleibt dann nur eine Lösung. MS wird nicht blöde sein und sich das Express Route Milliarden-Business kaputt machen. (Peering Traffic muss irgendjemand zahlen, lernt man bei den ISPs?)
  • Aus über 25 Jahren Vergangenheit: Nicht davon ausgehen, dass jeder On-Premises-APP-Server für WAN-Zugriff gebaut ist. (Timeout/Session)


Preise Express Route, Switzerland, 13.07.2023




Das meinen andere dazu. Wunderbarer Kommentar von Michael Loftus aus den Blog eines MS Mitarbeiters der uns erzählt wir brauchen kein VPN mehr.

Source Linkedin 12.07.2023

Eine Schnellsuche zum Thema Azure Application Proxy und Probleme bringt folgendes:



Jemand intergiert in Enterprise Umgebung, probiert, Produktiv und am Schluss eine Frage im neuen FAQ/Learning bei MS.

Oder war es der Endkunde selber im M365 Admin Portal und die IT-Service Firma kann dann helfen?


Red Teams use Azure Application Proxy to publish their C2 Server

Nice writeup by Adam Chester. So we can asume that it's "Safe" when Hacker use it. Right? Sudan agrees?



Why do they mention ExpressRoute so many times?

Screenshot Auszug MS KB:




02.07.2023, CITRIX 0-DAY, Pre Authentication XSS in Citrix Gateway (CVE-2023-24488)

by butsch 2. July 2023 23:42


Attacker is able to change the redirection of the LOGOUT page. To date we are unsure if this is only if you you use SAML as in the NOV 2022 Exploit.

GET /oauth/idp/logout?post_logout_redirect_uri=%0d%0a%0d%0a%3Cscript%3Ealert(document.cookie)%3C/script%3E HTTP/1.1

Pre Authentication XSS in Citrix Gateway (CVE-2023-24488)

Die Abfrageparameter für URL werden nicht ausreichend gesäubert, bevor sie in den HTTP Location-Header eingefügt werden. Ein Angreifer kann dies ausnutzen, um einen Link zu erstellen, der beim Klicken das Opfer zu einer beliebigen Stelle umleitet. Alternativ kann der Angreifer Zeilenumbruchzeichen in den Location-Header einfügen, um vorzeitig die HTTP-Header zu beenden und eine XSS-Payload in den Antworttext einzufügen.

Ein Angreifer kann bösartige Links erstellen, die entweder das Opfer zu einer von ihm kontrollierten Website umleiten oder JavaScript im Browser des Opfers ausführen, wenn sie geklickt werden.

Betroffene Software
Die folgenden Versionen sind von dieser Schwachstelle betroffen:

Citrix ADC und Citrix Gateway 13.1 vor 13.1-45.61
Citrix ADC und Citrix Gateway 13.0 vor 13.0-90.11
Citrix ADC und Citrix Gateway 12.1 vor 12.1-65.35
Citrix ADC 12.1-FIPS vor 12.1-55.296
Citrix ADC 12.1-NDcPP vor 12.1-55.296

Citrix Gateway ist eine Netzwerk-Appliance, die verschiedene Funktionen, einschließlich Remote-Zugriffs-VPN-Diensten, bereitstellt.

Aktualisieren Sie auf die neueste Version von Citrix Gateway.


Citrix ADC and Citrix Gateway Security Bulletin for CVE-2023-24487, CVE-2023-24488


Translated from English:

Anwendbare Produkte
Citrix ADC Citrix Gateway
Beschreibung des Problems
Es wurden Sicherheitslücken in den unten aufgeführten Citrix ADC- und Citrix Gateway-Versionen entdeckt, die bei Ausnutzung zu folgenden Sicherheitsproblemen führen könnten:

Betroffene Produkte, Versionen und Komponenten
Die folgenden unterstützten Versionen von Citrix ADC und Citrix Gateway sind von dieser Sicherheitslücke betroffen:

Citrix ADC und Citrix Gateway 13.1 vor 13.1-45.61
Citrix ADC und Citrix Gateway 13.0 vor 13.0-90.11
Citrix ADC und Citrix Gateway 12.1 vor 12.1-65.35
Citrix ADC 12.1-FIPS vor 12.1-55.296
Citrix ADC 12.1-NDcPP vor 12.1-55.296

Dieses Bulletin betrifft nur von Kunden verwaltete Citrix ADC- und Citrix Gateway-Installationen. Kunden, die Citrix-gemanagte Cloud-Services oder Citrix-gemanagte Adaptive Authentication nutzen, müssen keine Maßnahmen ergreifen.

CVE ID Beschreibung Voraussetzungen CWE CVSS
CVE-2023-24488 Cross-Site Scripting Appliance muss als Gateway konfiguriert sein (SSL VPN, ICA Proxy, CVPN, RDP Proxy) oder AAA-Virtual Server CWE-79 6,1
CVE-2023-24487 Willkürliches Lesen von Dateien Zugriff auf NSIP oder SNIP mit Zugriff auf das Management-Interface CWE-253 6,3

Was Kunden tun sollten
Betroffene Kunden von Citrix ADC und Citrix Gateway sollten so schnell wie möglich die entsprechenden aktualisierten Versionen von Citrix ADC oder Citrix Gateway installieren:

Citrix ADC und Citrix Gateway 13.1-45.61 und neuere Versionen
Citrix ADC und Citrix Gateway 13.0-90.11 und neuere Versionen von 13.0
Citrix ADC und Citrix Gateway 12.1-65.35 und neuere Versionen von 12.1
Citrix ADC 12.1-FIPS 12.1-55.296 und neuere Versionen von 12.1-FIPS
Citrix ADC 13.1-FIPS 13.1-37.150 und neuere Versionen von 13.1-FIPS
Citrix ADC 12.1-NDcPP 12.1-55.296 und neuere Versionen von 12.1-NDcPP

Citrix dankt Petr Juhanak von Accenture, Dylan Pindur von Assetnote und Wisdomtree vom Ant Group Digital Financial Security Team für die Zusammenarbeit zum Schutz der Citrix-Kunden.

Was Citrix unternimmt
Citrix benachrichtigt Kunden und Vertriebspartner über dieses potenzielle Sicherheitsproblem durch die Veröffentlichung dieses Sicherheitsbulletins im Citrix Knowledge Center unter https://support.citrix.com/securitybulletins.

Unterstützung zu diesem Thema erhalten
Wenn Sie technische Unterstützung zu diesem Thema benötigen, wenden Sie sich bitte an den Citrix Technical Support. Die Kontaktdaten für den Citrix Technical Support finden Sie unter https://www.citrix.com/support/open-a-support-case.

Die offizielle Sicherheitsmeldung von Citrix finden Sie hier.


IT | Hotfixes / Updates | SECURITY

M365/CLOUD, watch Microsoft status LIVE | See who has high volume traffic at homeoffice

by butsch 27. June 2023 23:00



Microsoft M36X has been scaling up its information system to address downtime, which is unsurprising considering the recent outages experienced in the past months.

Checking the Twitter Status M365 (https://twitter.com/MSFT365Status/status) in the morning is a good way to anticipate what to expect before opening your IT Support Ticket system. This allows you to make informed decisions, such as staying at home or securing the IT support door. You may also think in long term about studying art or other jobs beyond relying solely on the Cloud-O-Matic-Manager (COMM) duty. Maybe you find time to check SLA you have with your customers or with the company who runs your cloud.

Microsoft offers another tool that traces VPN-traffic or aM365 access worldwide, which can be used to identify existing cloud customers. Therefore, the next time your neighbour displays unusual Behavior, like throwing his BYOD laptop out of the window or being excessively loud in their home office, you can interactively investigate the reason using this tool


So this enables you too see high traffic points. But there are several green points at private only location. Only private person rentals block as example.

You can also utilize a map provided by Microsoft, highlighting locations where a high volume of traffic is received from customers connected to Microsoft Teams, Exchange, SharePoint, and other services. This map serves as a useful resource to crosscheck whether an address corresponds to a private residence or a company complex. Furthermore, if a specific point on the map exhibits a large data volume, you can assume that the person or company:

  1. Has implemented a split/non-split (FULL) VPN policy and may be downloading movies/warez from onedrive? He does not understand that everything he does private on his business laptop or his BYOD-enterprise-integrated-somehow may route all traffic to his employer (firm) and in this case straight through Microsoft Services somehow? On-premise you you took him down or took away his laptop toy. In the cloud you expect a really fat Express Route bill at the end of the month. You can even realtime see the Cloud cost in CLI/PS Microsoft told this week. That is a real neat tool you can see your Azure cost exploding in real time for your developers who profit from scalability as they always said.

b) Is using plain Microsoft Windows Defender ATP and lacks robust commercial enterprise endpoint protection such as McAfee/Trellix. In such cases, there is a possibility that the endpoint is currently infected and vulnerable due to the high traffic flowing through M365 services. While this level of paranoia might not be common, it's essential to ensure proper endpoint protection.

In the past, on-premise enterprise IT teams effectively managed similar challenges with great success. These aspects can be effectively addressed in on-premise enterprise IT environments, especially when supported by dedicated IT teams.

That guy at Ochsengartenweg in Switzerland, i know him. I am sure he does not know we can see when he works or not? Please note that respecting privacy and confidentiality is crucial when using tools and maps to gather information about individuals or companies.

Microsoft 365 network health status

This map shows the network health status of locations where Microsoft receives a high volume of traffic from customers connected to Microsoft Teams, Exchange, SharePoint, and other services.

A degraded or unhealthy status can be caused by various issues across the entire customer connection path, including issues with Microsoft’s network and availability, the customer’s network infrastructure, internet service provider availability, and third-party intermediary devices. Go to Service health in the Microsoft 365 admin center for more details on any issues we’ve detected, including the scope of impact.


Region Basel

Not too many M365 users where "Borat" lives? Does BORAT use Google or TUX? Does he have an Amiga?

Some actual alerts these days: 

Screenshot MS customer Portal 28.06.2023


22-28.06 from Twitter Microsoft 365 Status


VPN differences:

Full VPN, as opposed to split VPN, can often be a challenge when it comes to bandwidth. Here's why:


1. Routing all traffic through the VPN: With a full VPN configuration, all network traffic from the user's device is typically routed through the VPN tunnel. This means that even non-business-related internet traffic, such as browsing social media or streaming videos, will be redirected through the VPN server. As a result, it increases the overall bandwidth consumption.


2. Increased latency: The additional distance that data has to travel through the VPN server before reaching its destination can introduce latency or delay. This latency can impact real-time applications, such as video conferencing or online gaming, where responsiveness is crucial. The increased round-trip time can affect the user experience and degrade the overall performance.


3. VPN encryption overhead: VPNs use encryption to secure the data transmitted through the tunnel, which adds an additional layer of processing. This encryption process can increase the size of data packets, leading to larger bandwidth usage. The overhead caused by encryption can become more pronounced when dealing with high-volume data transfers or bandwidth-intensive activities.


4. Bottlenecks at the VPN server: Full VPN setups often involve funneling all traffic through a central VPN server or a limited number of servers. This concentration of traffic can lead to congestion and potential bottlenecks at the server level. As more users connect to the VPN and utilize the available bandwidth, it can impact the performance for all users connected to that server.


5. Limited VPN server capacity: VPN servers have finite resources, including processing power, memory, and network bandwidth. When multiple users connect to a VPN server simultaneously, especially during peak usage periods, the server's capacity may be strained. This can result in reduced bandwidth availability for each user, affecting overall performance.


It's important to note that while full VPNs can present challenges with bandwidth, they also provide enhanced security and privacy by encrypting all internet traffic. Organizations and individuals should consider the trade-offs between security and bandwidth requirements when choosing between full VPN and split VPN configurations.



Links - Important | M365/AZURE

M365/Azure, Full Hybrid Mode, M365 user unable to see free/Busy Room/meeting or book on-premise

by butsch 22. June 2023 17:06

PROBLEM: The M365 (cloud) users are unable to access free/busy info from the on-premise room mailbox and part two are unable to book it


In hybrid environment that combine on-premises and Microsoft 365 (M365) accounts, organizations may encounter difficulties when attempting to add conference rooms to meetings. This issue affects not only Outlook Web App (OWA) but also the Outlook desktop client (outlook.exe).

Problem Overview


The problem arises when attempting to book conference rooms from the room list in Outlook. In both OWA and Outlook desktop client, users may find that conference rooms are missing or unavailable for selection during the meeting scheduling process. This issue is particularly prevalent in hybrid environments where room mailboxes are synchronized between on-premises Exchange Server and M365.

Root Cause


The root cause of this problem lies in the configuration of the RoomList distribution group, which is responsible for organizing and displaying room mailboxes. In hybrid environments, the RoomList distribution group needs to be properly synchronized and maintained between on-premises and M365.

Inaccurate recipient TypeDetails


One potential cause is the incorrect recipientTypeDetails property associated with room mailboxes during synchronization. When on-premises room mailboxes are synchronized to M365, the recipientTypeDetails property may be set as "MailUser" instead of the expected value, "RoomMailbox". This discrepancy prevents the room mailboxes from being recognized as valid conference rooms in M365.


What is wrong again in detail:

In a Full hybrid Exchange <> M365 Setup with Room resources on on-premise side you see following effect.

Error #1:

The test.M365 users are unable to access free/busy info from the on-premise room mailbox as example Audi_A3_OST_179602@butschlab.ch.


The test.M365 users can see the calender and free/busy but they are unable to BOOK the room effective (No permission changed and users was in on-premise side before migration)


We have following users:

User on-premise

User M365




room all on-premise Exchange

room M365 (Make one new M365 if you dont have for solution)




This is also valid for:

In a hybrid deployment of on-premises Microsoft Exchange Server and Exchange Online, assume that Exchange Online users select add room in a meeting request in Outlook Web App. In this situation, on-premises room mailboxes aren't displayed in the list as expected. Therefore, Exchange Online users can't add an on-premises conference room to a meeting request.


This is kind of unsupported solution as some people say at MS in Forum. On the other hand there is a KB which clearly explains part of the solution.

But only parts of it? Here is the full working solution:


Error #1:

The M365 users are unable to access free/busy info from the on-premise room mailbox.

  1. You have to generate a ROOMLIST or lets say NEW Distribution List with the option –roomlist
  2. In that room list you make all existing on-premise rooms member (Their object)
  3. You already have OR newly generate a ROOM on the M365 Side m365.room@butschlab.ch (Make a DUMMY room on Cloud side)
  4. You make the new m365.room ROOM also member of that Room List (Distribution group) < Most important STEP



The M365 users can see the calender and free/busy but they are unable to BOOK the room effective (No permission changed and users was in on-premise side before migration)

Here is how to see which rooms are affected:

Get-Mailbox -RecipientTypeDetails RoomMailbox | Get-CalendarProcessing | ?{!$_.ProcessExternalMeetingMessages}

You have to enable the flag -ProcessExternalMeetingMessages $true for the on-premise accounts. We never touched this in on-premise ONLY because it was rather dangerous if you did it wrong. If you did not follow compliance and enabled some things like INLCUDE/SHOW subject or BODY from meetings and ACTIVATE that wrong external people could you see conf. Data from inide the org.


Powershell Exchange on-premise

# V1.0, 23.06.2023, Mike Butsch, Solution to see on-premise ROOM Resources from M365 side

 # Solve Error #1:

 # Generate a Distribution Group with paramter –roomlist in your favorite OU.

# The OU has to be synced to Azure/M365. So include the OU in your Azure Connector Setup.

# ------------------------------------------------------------

New-DistributionGroup -Name Autos -roomlist -OrganizationalUnit "OU=Azure_SYNC,OU=Gruppen_AG,DC=butsch,DC=local"

New-DistributionGroup -Name Raeume -roomlist -OrganizationalUnit "OU=Azure_SYNC,OU=Gruppen_AG,DC=butsch,DC=local"

# ------------------------------------------------------------

Add-DistributionGroupMember Autos -member Audi_A3_OST_179602

Add-DistributionGroupMember Autos -member Peugeot_508_OST_162117

Add-DistributionGroupMember Raeume -member m365.room

# ABOVE most important step you NEED to add a M365 ROOM or DUMMY Resource as member

# ------------------------------------------------------------

Add-DistributionGroupMember Raeume -member OST_Sitzungszimmer_Vogebank

Add-DistributionGroupMember Raeume -member Sitzungszimmer_EG654

Add-DistributionGroupMember Raeume -member Sitzungszimmer_121_OG

Add-DistributionGroupMember Raeume -member FAT_Raum_2_OG

Add-DistributionGroupMember Raeume -member Konferenzraum_3_OGE

Add-DistributionGroupMember Raeume -member Sitzungszimmer_3_OGE

Add-DistributionGroupMember Raeume -member m365.room

# ABOVE most important step you NEED to add a M365 ROOM or DUMMY Resource as member


 # Solve Error#2.


# ------------------------------------------------------------

set-CalendarProcessing OST_Sitzungszimmer_Vogebank -ProcessExternalMeetingMessages $true

set-CalendarProcessing Sitzungszimmer_EG654 -ProcessExternalMeetingMessages $true

set-CalendarProcessing Sitzungszimmer_121_OG -ProcessExternalMeetingMessages $true

set-CalendarProcessing FAT_Raum_2_OG -ProcessExternalMeetingMessages $true

set-CalendarProcessing Konferenzraum_3_OGE -ProcessExternalMeetingMessages $true

set-CalendarProcessing Sitzungszimmer_3_OGE -ProcessExternalMeetingMessages $true

set-CalendarProcessing Audi_A3_OST_179602 -ProcessExternalMeetingMessages $true

set-CalendarProcessing Peugeot_508_OST_162117 -ProcessExternalMeetingMessages $true



GUI, notive that the user accounts of ROOM resources are always disabled. Keep them seperated so nobody (Azure Admin)

Does delete them when he has his cleaning day and want's to be a good employee.




Since you are in Hybrid Mode also check shortly both sides on-premise and M365 if you have the same settings there:

Set-OrganizationRelationship | fl



The KB was written for the OWA / WEAPP but we proof it's also valid for outlook.exe

Please notice that the KB does NOT contain the full solution as we provide in our Butsch KB.



Maybe it's the TOKENS and AUTHENTICATION also so Check ANYTHING related to 4001 if you have OAUTH oder DAUTH:



EventID which may be related if you have OAUTH active

Event ID 1003: This event indicates successful OAuth authentication requests. It confirms that the authentication process between Microsoft 365 and on-premises Exchange was successful.

Event ID 1004: This event indicates failed OAuth authentication requests. It can provide details about the reason for the authentication failure, such as incorrect credentials or authorization issues.

Event ID 1022: This event is logged when an OAuth token is issued to a client application. It confirms that the token was successfully issued for the authentication process.

Please note that these specific event IDs are not guaranteed to be available in all versions of Exchange Server or in every configuration. The availability of events and their IDs can vary based on your Exchange Server version and the specific logging settings enabled.

Additionally, it's important to understand that the Application log on the Exchange server may not provide comprehensive logging specifically for OAuth access from Microsoft 365 to on-premises Exchange. As mentioned earlier, monitoring the client-side logs in Outlook and the logs in Microsoft 365 (such as audit logs and trace logs) will provide more detailed information for troubleshooting OAuth-related issues.

If you encounter specific error messages or need more specific guidance on troubleshooting OAuth access, I recommend referring to Microsoft's official documentation, reaching out to Microsoft Support, or consulting with an Exchange Server specialist for further assistance.




M365/AZURE | Microsoft Exchange | Exchange 2016 | Exchange 2019

M365/Exchange Hybrid OAuth Testing command, OAuth-Cert out-of-sync 4001, IIS VDIR OAuth wrong

by butsch 21. June 2023 19:00


Resolve and find OAuth problem in Exchange Hybrid Setup Enviroments


Short Understanding OAuth:

OAuth (Open Authorization) is an industry-standard protocol that enables secure authorization for third-party applications without the need to disclose user credentials. It allows users to grant limited access to their resources on one site to another site, without sharing their credentials. In the context of Exchange and Hybrid setups, OAuth provides a streamlined and secure way for users to access their emails, calendars, and other resources.



Since end 2022 every M365 TENANT has by default cloud side OAuth enabled. There is DAUTH (Older MS own made) and there is

an OPEN Source Authentication method called OAuth.



You may land here if:

a) You just want to test if OAuth works on both sides between Exchange<>M365

b) Or your IIS Virtualdirectory do not accept OAuth beside NTLM etc.

c) If you have done a Self Signed OAuth Cert renew (The one with 2H+ UTC/CET Local time offset delay bug)

We explain how to fix and how to check if both sides have the latest certificate UTC time corrected.

So this KB is on how to TEST OAuth Exchange onpremise<>m365 but also why you came here because maybe it does not work and reports FAILURE not SUCCESS.


First let us check if your Microsoft 365 side has OAUTH enabled?

M365 side: Get-OrganizationConfig | FL oau*


Is Oauth working incoming and outgoing?


Following are detailed steps on how to test OAuth from from both sides in your Exchange <> M365 Hybrid Setup.

We have following users:

User on-premise 

User M365 




Short without any help or explanation:

Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.office365.com/ews/exchange.asmx -Mailbox test.onpremise@schweiz.ch -Verbose | fl

Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc -Mailbox test.onpremise@schweiz.ch -Verbose | fl

Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.schweiz.ch/ews/metadata/json/1 -Mailbox test.m365@schweiz.ch -Verbose | fl

Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover.schweiz.ch/autodiscover/autodiscover.svc -Mailbox test.m365@schweiz.ch -Verbose | fl


Reference for command Test-OAuthConnectivity:


Default value for all TEST outgoing to M365: https://outlook.office365.com/ews/exchange.asmx

Default value for all TEST outgoing to M365: https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc


Test on On-premise > M365 Exchange outgoing direction:

Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.office365.com/ews/exchange.asmx -Mailbox test.onpremise@schweiz.ch -Verbose | fl

Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc -Mailbox test.onpremise@schweiz.ch -Verbose | fl


If succesfull:


Test M365 > on-premise incoming direction:

Yellow change to what you have in Exchange Virtual Directory:

Retrieve those from your on-premise:

On-premise: Get-WebServicesVirtualDirectory | fl Extern*

Example: https://outlook.schweiz.ch

On-premise: Get-ClientAccessServer | fl autodis*

Example: https://autodiscover.schweiz.ch


Test https://autodiscover.schweiz.ch from external:

Open UP in a web browser from EXTERNAL (Mobily other network) and make sure you can LOGON to the Autodiscover URL with the on-premise account.



Test on M365 side:

Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.schweiz.ch/ews/metadata/json/1 -Mailbox test.m365@schweiz.ch -Verbose | fl

Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover.schweiz.ch/autodiscover/autodiscover.svc -Mailbox test.m365@schweiz.ch -Verbose | fl


If succesfull:

Interpreting the Test Results:

After executing the Test-OAuthConnectivity command for both local and Microsoft 365 users, you will receive the test results. Here are a few possible outcomes:



If the test is successful, it means that the OAuth connectivity for the specified user is functioning correctly, and they can access Exchange resources seamlessly.


If the test fails, it indicates an issue with the OAuth connectivity. Check for any misconfigurations in the OAuth configuration, network connectivity problems, or any other relevant errors. Troubleshooting these issues will be crucial to ensure uninterrupted access for the user.


Failure may be because of......


OAuth not Set on VirtualDirectory EWS and AUTODISCOVER on Exchange on-premise


OAuth Set, but experiencing a PULL/REDO issue > FLIP-FLOP effect on IIS VirtualDirectory Settings for EWS and AUTODISCOVER on Exchange on-premise (That what MS says is possible....)


You have renewed the OAuth Certificate on the Exchange on-premise but

Failure due to OAuth Certificate Renewal without completing the final step in the KB



CHECK1: OAuth not Set on VirtualDirectory EWS and AUTODISCOVER on Exchange on-premise


When troubleshooting OAuth-related issues, the first step is to ensure that OAuth is correctly set on the VirtualDirectory configurations for EWS (Exchange Web Services) and Autodiscover in the on-premises Exchange environment. In some cases, OAuth might not be properly configured or enabled on these VirtualDirectories, resulting in authentication failures and connectivity issues.

To address this, it is essential to verify and configure OAuth settings correctly on the EWS and Autodiscover VirtualDirectories, ensuring that the necessary authentication methods are enabled and OAuth is correctly configured.


Check VirtualDirectory on-premise:

Save and run "vdir.sp1" relevant for Exchange<>M365 OAuth

$VirtualDirectories = @()

 $VirtualDirectories += Get-WebServicesVirtualDirectory | Select-Object Name, InternalAuthenticationMethods, ExternalAuthenticationMethods

$VirtualDirectories += Get-AutodiscoverVirtualDirectory | Select-Object Name, InternalAuthenticationMethods, ExternalAuthenticationMethods


foreach ($VirtualDirectory in $VirtualDirectories) {

Write-Host "Virtual Directory Name: $($VirtualDirectory.Name)"

Write-Host "Internal Authentication Methods: $($VirtualDirectory.InternalAuthenticationMethods)"

Write-Host "External Authentication Methods: $($VirtualDirectory.ExternalAuthenticationMethods)"


# Check if OAuth is present in internal authentication methods

if ($VirtualDirectory.InternalAuthenticationMethods -like "*OAuth*") {

$highlightedAuthMethods = $VirtualDirectory.InternalAuthenticationMethods -replace "OAuth", "`e[92mOAuth`e[0m"

Write-Host "OAuth authentication is enabled." -BackgroundColor Green



Does it show OAuth right side?

Or yourself:

Get-AutodiscoverVirtualDirectory | fl *auth*

Get-WebServicesVirtualDirectory | fl *auth*



CHECK2: OAuth Set, but does not PULL > FLIP-FLOP effect IIS VirtualDirectory Settings for EWS and AUTODISCOVER on Exchange on-premise


IF FAILURE: FLIP-FLOP the IIS Virtualdirectory Authentication Settings to be fireproof.

Even if ALL seems fine sometimes their is a FLIP FLOP effect which evn MS can't explain. So turn it OFF and TURN it ON.

Commands to move out the the FLIP/FLOP effect

ONLY READS VALUES | Does not change anything

Get-AutodiscoverVirtualDirectory -Server MYSERVERNAME |Set-AutodiscoverVirtualDirectory -OAuthAuthentication $false

 Get-AutodiscoverVirtualDirectory -Server MYSERVERNAME |Set-AutodiscoverVirtualDirectory -OAuthAuthentication $true

 Get-WebServicesVirtualDirectory -Server MYSERVERNAME |Set-WebServicesVirtualDirectory -OAuthAuthentication $false

 Get-WebServicesVirtualDirectory -Server MYSERVERNAME |Set-WebServicesVirtualDirectory -OAuthAuthentication $true

Cmd.exe elevated


and if you like you can Recycle in IIS the APP Pools with:

Restart-WebAppPool MSExchangeAutodiscoverAppPool
Restart-WebAppPool MSExchangeServicesAppPool


IF you use DAUTH the OLD authentication then the parameter would be:


Question: Can i see OAuth in IIS GUI?

 In the IIS Manager GUI, you cannot directly see the specific OAuth configuration settings for Exchange virtual directories. The GUI does not provide a dedicated section or flag to indicate whether OAuth is enabled for a virtual directory.

 OAuth configuration settings are managed at a higher level in Exchange Server, typically through PowerShell cmdlets or other management tools specific to Exchange. These settings control the OAuth functionality and its integration with Microsoft 365.

 While you can view general authentication settings in the IIS Manager GUI, it does not provide granular visibility into OAuth settings. To get detailed information about OAuth configuration in Exchange Server 2013, it is recommended to use PowerShell cmdlets or consult the Exchange management documentation for accurate and up-to-date information.



CHECK3: You have renewed the OAuth Certificate on the Exchange on-premise but

FAILURE: You did not do the last step in the KB on how to renew OAuth cert, Event 4001 Exchange on-premise

If you have recently renewed the OAuth certificate on your Exchange on-premises server but encounter authentication failures or other issues afterward, it is crucial to ensure that you have completed the final step mentioned in the corresponding knowledge base (KB) article.

Renewing the OAuth certificate involves multiple steps, and failure to follow the complete process can result in incomplete certificate installation or incorrect configuration. By carefully reviewing the KB article and verifying that all required steps, including post-renewal configuration, have been executed, you can avoid common pitfalls and ensure a successful OAuth certificate renewal.



Find out / ICO if you are affected

Event 4001 on Exchange on-premise

The request failed with HTTP status 401: Unauthorized in Event

Missing signing certificate if you do the Test-OAuthConnectivity 





 Autodiscover failed for email address <alani.berseddi@schweiz.mail.onmicrosoft.com>

SMTP:alani.berseddi@schweiz.mail.onmicrosoft.com with error System.Net.WebException:

The request failed with HTTP status 401: Unauthorized. at

System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.SoapWebClient.AutoDiscover.DefaultBinding_Autodiscover.EndGetUserSettings(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.<>c__DisplayClass4.<EndInvoke>b__3() at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.ExecuteAndHandleException(ExecuteAndHandleExceptionDelegate operation). ---> System.Net.WebException: The request failed with HTTP status 401: Unauthorized. at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.SoapWebClient.AutoDiscover.DefaultBinding_Autodiscover.EndGetUserSettings(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.<>c__DisplayClass4.<EndInvoke>b__3() at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.ExecuteAndHandleException(ExecuteAndHandleExceptionDelegate operation) --- End of inner exception stack trace --- . Name of the server where exception originated: exchange2013R17


Cloud to On-premise OAuth certificate missing:

1) Check certificate in AuthConfig in On-Premises Exchange Management Shell:


Test-OAuthConnectivity -Service AutoD -TargetUri <OnPremises Autodiscover.svc
endpoint -https://mail.domain.com/autodiscover/autodiscover.svc> -Mailbox
<Exchange Online Mailbox> -Verbose | fl

it will give you this exception:

Microsoft.Exchange.Security.OAuth.OAuthTokenRequestFailedException: Missing signing certificate. 




Reason OAuth is not working anymore from on-premise > M365 side because OAuth Cert renew done on on-premise

DONE: You have renewed the OAuth Certificate on the Exchange on-premise but the CLOUD side still has the OLD version (You did not do the last step in the KB on how to renew OAuth cert)



Thumbprint Ending: ***62



(Get-AuthConfig).CurrentCertificateThumbprint | Get-ExchangeCertificate | Format-List


 All seems fine, you also waited the 2H+ because of the UTC/CET bug and /owa did not work for that time.

 You done all steps but did not do the purple box? Azure needs to KNOW the new cert now that's the part missing. Not the Exchange Online M365 the Azure Cloud.



When you deploy a new OAuth certificate in a hybrid Exchange environment, it is essential to rerun the Hybrid Configuration Wizard (HCW) to ensure that the cloud side of the environment is aware of the new certificate. The HCW is a tool provided by Microsoft to simplify the configuration of a hybrid deployment between on-premises Exchange and Exchange Online.

 By rerunning the HCW, you allow the wizard to update the necessary settings and configurations, including the OAuth certificate, on both the on-premises Exchange servers and the Exchange Online environment. This ensures seamless communication and authentication between the two environments, enabling features such as cross-premises free/busy lookup and mailbox moves.

 It is important to note that when deploying a new OAuth certificate, running the HCW is a crucial step to ensure proper functionality and synchronization between the on-premises and cloud components of the hybrid Exchange environment.

Solution 1:

 RE-RUN The HCW (Hybrid Configuration Wizard) wizard on Exchange with existing settings as quick fix.


Solution 2: (Very technical)

 This is the step you did read but it's only for 2007/2010 and mix with newer Exchange and M365 so you did not read.



So normal you ONLY need to do this IF you have 2007/2010 in MIXED mode with 2013/2016 and M365 or if you have DONE the OAuth Cert renew on a 2013/2016/2019 on-premise.

 Follow the manual steps as you when your create an Organisation Trust between two companys Exchange related.


For the OAuth Cert renew Steps 3 and 4 are relevant. This will upload the inhouse OAuth Certificate to Azure. As per 21.06.2023 those are:

 Step 3: Export the on-premises authorization certificate

 Step 4: Upload the on-premises authorization certificate to Azure Active Directory Access Control Service (ACS) 



Check IF Azure HAS your RE-done(latest) OAuth Exchange Certificate active


Azure side:

Connect to Azure from the machine where have acccess to Azure or the Exchange with the HCW Wizard.

Connect-MsolService (Logon with your Tenant or Admin account)

Get-MsolServicePrincipalCredential -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 -ReturnKeyValues $true


Note the StartDate and Enddate from the latest you made.


Get-ExchangeCertificate (Get-AuthConfig).CurrentCertificateThumbprint |fl Not*

Or CHECK below Script to get the time in UTC direct on-premise

 Attention Local Time Zone and Azure Time Zone are different. Remember when you did renew the OAuth cert

It took 2 or 4 hours to get active (OWA and PS was not wokting if so...)

On-premise Local Time Swiss

M365 side

NotAfter : 12.06.2028 13:39:54

NotBefore : 12.06.2023 13:39:54

StartDate : 12.06.2023 11:39:54

EndDate : 12.06.2028 11:39:54

On-premise Convertes to UTC USA Azure 

NotAfter : 12.06.2028 11:39:54

NotBefore : 12.06.2023 11:39:54


Now hopefully, when doing the renew of the OAuth cert onpremise, you did not start 10 times.


YOU = Local Time Zone +/- 2-4hrs or whatever

Use below script to get the time as it looks with azure


Get the TIMES from local OPAUTH Cert in all time zones relevant

ONLY READS VALUE | Does not change anything

# V1.0, 18.06.2023, www.butsch.ch

# Get the TIMES from local OPAUTH Cert in all time zones relevant


$certThumbprint = (Get-AuthConfig).CurrentCertificateThumbprint

$cert = Get-ExchangeCertificate -Thumbprint $certThumbprint


$localTimeZone = [System.TimeZoneInfo]::Local

$utcTimeZone = [System.TimeZoneInfo]::Utc


$cert | Select-Object -Property NotBefore, NotAfter, `

@{Name="LocalNotBefore"; Expression={$_.NotBefore.ToLocalTime().ToString()}}, `

@{Name="LocalNotAfter"; Expression={$_.NotAfter.ToLocalTime().ToString()}}, `

@{Name="UTCNotBefore"; Expression={$_.NotBefore.ToUniversalTime().ToString()}}, `

@{Name="UTCNotAfter"; Expression={$_.NotAfter.ToUniversalTime().ToString()}}, `

@{Name="LocalTimeZone"; Expression={$localTimeZone.DisplayName}}, `

@{Name="UTCTimeZone"; Expression={$utcTimeZone.DisplayName}} 


Re-check all as explained in:




FAILURE, 401 Access Denied: You don't not have all internal URL from your Exchange listed with AZURE as SPN

(He will accept only those address which the Cloud has SPN listed to easy explain)



On-premise: Check the OnPremisesDiscoveryEndPoint and OnPremisesWebServiceEndPoint from:

Get-IntraOrganizationConfiguration | Fl

On-premise: Should be the same as per:

Get-WebServicesVirtualDirectory | FL ExternalUrl


Check Azure side:

Connect to Azure from the machine where have acccess to Azure or the Exchange with the HCW Wizard.

Connect-MsolService (Logon with your Tenant or Admin account)

(Get-MsolServicePrincipal -ServicePrincipalName "00000002-0000-0ff1-ce00-000000000000").ServicePrincipalNames

Check that ALL URL which you have on-premise are listed also there.






And any further DOMAINS you may have on your on-premise Exchange Domains .EU .DE .COM don't forget those.


External Links mentioned:




Microsoft Azure Active Directory OAuth 2.0 Fundamentals:



Deprecation of Basic Authentication in Exchange Online:



Test-OAuthConnectivity PowerShell cmdlet:



Troubleshooting OAuth Expiration in Exchange:



Configuring OAuth Authentication in Exchange 2013:




M365/AZURE | Microsoft Exchange | Exchange 2013 | Exchange 2016 | Exchange 2019

Exchange 2013, 2016, 2019, Event 3028, KILLBIT error, you can safely ignore since 2014

by butsch 20. June 2023 13:25

We just stumbeld again over the KILLBIT event 3028 warning on several Exchange Versions dated 06/2023.

It still does not sound good from the text and you would think that some third party plugin or som health check blocks the files.

Some say this is normal, some say you can safely ignore this. MS does not seem to know.

Event 3028, MSExchangeApplicationLogic

Scenario: ProcessKillBit. Failed to read killbit list file because of exception System.IO.IOException: The process cannot access the file 'D:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\owa\prem\15.0.1497.48\ext\killbit\killbit.xml' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)

at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)

at Microsoft.Exchange.Data.ApplicationLogic.Extension.KillBitHelper.TryReadKillBitFile(Int32& refreshRate, DateTime& lastModifiedTime)





On this machine we found following versions of the file. Deletion of the FILE and then IISRESET or Recycle APP Pools does not solve it.


Solution: NONE, just wait and do not handle on SIEM J





Exchange 2019 | Exchange 2016 | M365/AZURE

M365 | on-premise, Outlook.exe DEBUG logging for troubleshooting complete guide

by butsch 16. June 2023 07:00

Enhancing Outlook Debug Logging for Troubleshooting

Mike Butsch, www.butsch.ch

What we want to do and why

Outlook debug logging is a valuable tool for diagnosing and resolving issues within Microsoft Outlook. By enabling advanced logging, you gain deeper insights into the application's behavior, allowing for more effective troubleshooting. In this blog post, we will explore the process of enabling global and advanced logging for Outlook, along with additional steps to enhance the logging capabilities.

In conclusion, by enabling and leveraging Outlook debug logging, you can gain valuable insights into the behavior of Microsoft Outlook and efficiently troubleshoot any issues that may arise. Remember to exercise caution when modifying the Windows Registry and follow the necessary steps outlined in this blog post.

Some key points in this paper:

* Attention: Do not make user you analyzer Local Adminstrator

* How to make the KEY under another user Registry hive

* Your are admin or user how to generate the Registry Key

* How to find something or for what to search for in the Debug Logfiles you made:

* Analyze WPA and ETL files

* Outlook 2010 Debug Logfiles

* Outlook 2013 and Outlook 2016 Debug Logfiles

* What if you want to enable this for some computers automatic in your Windows Domain on-premise with GPO?



in addition to the steps mentioned in the documentation, there is an additional parameter that can be enabled to gather more detailed information. This parameter involves modifying a specific subkey in the Windows Registry. Please note that the following steps require careful attention to avoid unintended issues:

1. Launch the Windows Registry Editor by opening "regedit.exe".

2. Navigate to the following subkey path: "HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\xx.0\Outlook\Options\Shutdown".

3. In the specified subkey, create a new "DWORD (32-bit) Value" named "FastShutdownBehavior".

4. Double-click on the newly created "FastShutdownBehavior" value and set its value data to 2.


By following these steps, you will successfully create the DWORD (32-bit) value "FastShutdownBehavior" with a value of 2 in the Windows 10 Registry.


Attention: Do not make user you analyser Local Adminstrator, this will produce probleme related to Exchange, ActiveSync GPO and Azure sync (Infos not Synced back to local Domain from Azure for that user)



Note: If the user experiencing the issue does not have sufficient permissions to create the key, it is important not to grant them local admin privileges without addressing the "ADMINSholder" flag attribute. Granting local admin privileges without correcting the "ADMINSholder" flag may lead to further complications. Refer to these resources for more information on the "ADMINSholder" flag:

Attention: https://www.butsch.ch/post/Activesync-with-Exchange-2013-does-not-work-ADMINSHOLDER-Flag-(an-old-bad-friend) or https://www.butsch.ch/post/Exchange-Activesync-1053-Event-4003-Error-2007201020132016-Adminsholder

Again a sample why you should not make the user Local Admin to explain: Also be aware that if you are in M365 Azure Hybrid Mode those account with ADMINS Holder Flag Set wan't sync back or to Azure.


Target what we need for the USER who has the OUTLOOK.EXE Problem. That may be a different user than the LOGGED on s non domain joined machine.

How to set key for another user?


It's important to note that the user encountering the "OUTLOOK.EXE" problem may not be the same as the one logged on to the non-domain joined machine. Therefore, we need to set the key for the relevant user. Here is a brief explanation of how to set the key for another user:

1. Create a CMD shortcut on the desktop.

2. Right-click on the shortcut while holding down the SHIFT key and choose "Run as different user."

3. A command prompt (CMD.exe) will open.

4. Launch the Registry Editor (regedit.exe) from the command prompt.

5. Locate the appropriate registry hive for the user you want to debug in Outlook.


INFO: Why not make him local Admin again because most don't understand impact if not corrected afterwards! Please read carefull above IF you have that idea.


How to make the KEY under another user Registry hive Explanation #1


Solution you need to run regedit.exe under a seperate LOCAL ADMIN or Service account to have the permission to CREATE that missing key structure per user. Also keep in MIND that existing POLICY/GPO will overwrite the settings within 15-X minutes if you may have those set in Domain enviroment.

How to make a REGISTRY KEY under ANOTHER user hive

If the user experiencing the issue does not have sufficient permissions to create the registry key, it is important to take the following precautions:


1. Avoid granting the user local admin privileges without addressing the "ADMINSholder" flag attribute.

2. Granting local admin privileges without correcting the "ADMINSholder" flag can lead to additional complications and should be avoided.


To proceed with modifying the registry for the affected user:


1. Request access to the user's registry hive or have the affected user provide it to you.

2. Open the Windows Registry Editor ("regedit.exe").

3. In the Registry Editor, click on "HKEY_USERS" and then go to "File" -> "Load Hive."

4. Browse to the user's registry hive file (NTUSER.DAT) located in their profile folder (e.g., C:\Users\Username).

5. Enter a name (e.g., "UserHive") for the loaded hive.

6. Navigate to "UserHive\Software\Policies\Microsoft\Office\xx.0\Outlook\Options\Shutdown" in the loaded hive.

7. Create a new "DWORD (32-bit) Value" named "FastShutdownBehavior" and set its value to 2.

8. Unload the loaded hive by selecting the loaded hive ("UserHive") and clicking on "File" -> "Unload Hive."


By following these steps, you can modify the registry using the hive of the affected user, without granting them local admin privileges. Remember to exercise caution and address the "ADMINSholder" flag issue if encountered, as granting local admin privileges without correcting it can lead to further complications.


Find the office version "SHORTY" the user has as exmaple "15.0", "16.0" etc.


Your are admin or user and how to generate the Registry Key Explanation #2


When troubleshooting Outlook issues for a specific user, there are two options to access the user's registry hive.

By following either of these options, you can access the user's registry hive and make necessary

modifications to troubleshoot the Outlook.exe problem. Remember to exercise caution and ensure you

have the appropriate permissions and authority to access the user's registry hive.



Option 1: User Logged On to the System


If the user experiencing the problem is currently logged on to the system, you can utilize the following method:


1. Open the Command Prompt (cmd.exe) with administrative privileges.

2. Execute the "regedit.exe" command using the "RUNAS" trick. This allows you to run the Registry Editor as a different user.

3. Skip to the next step if you can already see the user's registry hive in the Registry Editor. This occurs when the user is logged on to the system and you are accessing the Registry Editor using the "RUNAS" trick or a remote session, such as Dameware or another remote support tool.

Option 2: Administrator/IT Logged On


If you are logged on as an administrator or IT personnel and know the username of the user experiencing the Outlook.exe problem, follow these steps:


1. Launch the Registry Editor ("regedit.exe") with administrative privileges.

2. Click on "HKEY_USERS" in the Registry Editor.

3. Go to "File" -> "Load Hive" in the menu.

4. Browse to the user's registry hive file (NTUSER.DAT) located in their profile folder (e.g., C:\Users\Username).

5. Provide a name (e.g., "UserHive") for the loaded hive. This name will be used to identify the user's registry hive in the Registry Editor.




Once you have made the necessary changes to the registry, log off and log back on to apply the settings. Launch Outlook, and you should now see the logging in action. This allows you to identify any problematic areas or errors that may be occurring.

Check the key again under the HKCU > OK

Start Outlook, yes loggin active and you see warning

Check the things that don't work to produce logs. Click around and open things and other Calendar from OAB adressbook as example.

Check the logs generates



To gather relevant log files for troubleshooting purposes, you can focus on specific folders or actions that are not functioning as expected. These log files can be shared with Microsoft Support to assist in resolving the issue. To convert the log files (ETL format) into a more readable format, you can use the PowerShell command `tracerpt.exe FILENAME.ETL -lr`. You can then preview the converted files or open them using the Event Viewer (`eventvwr.exe`) to examine the events more closely.

Convert in Powershell to some TEXT/XML.

ETL > Event Viewer ;-) ???

You can then preview the converted files or open them using the Event Viewer (`eventvwr.exe`) to examine the events more closely.

Filter may help if there are any error?

How to find something or for what to search for in the Debug Logfiles you made:


When analyzing ETL or XML files for debug information related to free/busy functionality in Outlook, you can look for specific events and data related to free/busy operations. Here are some key points to consider:


1. Event ID: Look for events with specific event IDs that pertain to free/busy operations. These event IDs can vary depending on the version of Outlook and the specific scenario. Typically, you may find events with IDs such as 2000, 2007, or 2016, which are commonly associated with free/busy processing.

2. Provider Names: Look for events associated with providers or services related to free/busy functionality. These providers can include Autodiscover, Exchange Web Services (EWS), Availability Service, or any other components involved in free/busy data retrieval and processing.

3. Timestamps: Pay attention to the timestamps of the events and their sequencing. This can help you understand the flow and timing of free/busy operations. Look for patterns, delays, or any inconsistencies that might indicate potential issues.

4. Error Codes and Messages: Take note of any error codes or error messages associated with the free/busy events. These can provide valuable information about the nature of the problem or any specific errors encountered during free/busy processing.

5. Data Fields: Look for specific data fields or attributes that contain information related to free/busy functionality. These can include user identifiers, calendar data, availability status, or any other relevant data that helps in understanding the free/busy operations.

To locate this debug information within the ETL or XML file, search for the relevant event IDs, provider names, error codes, or specific data fields mentioned above. Use the search functionality in your text editor or XML viewer to locate and analyze the corresponding events. Additionally, you can filter the events based on the time range or other criteria to focus on the relevant information.

By carefully examining these events and data within the ETL or XML file, you can gain insights into the free/busy operations, identify any errors or delays, and troubleshoot any issues affecting the free/busy functionality in Outlook.


Analyze WPA and ETL files


Also read following KB on how to process the files further with WPA



If you are specifically working with ETL files generated from Outlook.exe debug mode the files are binary files containing event traces, and parsing them requires specific tools and techniques.

To analyze ETL files generated from Outlook.exe debug mode, you can use Microsoft Message Analyzer (MMA) or Windows Performance Analyzer (WPA), which are powerful tools for analyzing event traces. These tools allow you to load the ETL file and apply filters to extract specific information or errors related to Outlook.exe.

Here's a high-level overview of the steps involved in using Microsoft Message Analyzer to analyze Outlook.exe ETL files:

  1. Download and install Microsoft Message Analyzer from the Microsoft Download Center.


2. Launch Microsoft Message Analyzer.

3. Go to the "File" menu and select "New Session" to create a new session.

4. In the "New Session" window, click on the "Live Trace" tab.

5. Select the "Microsoft-Windows-Diagnostics-Performance" provider from the list.

6. Click on the "Capture" button to start capturing events.

7. Reproduce the issue or scenario for which you want to capture the event trace.

8. Once the desired events have been captured, click on the "Stop" button to stop capturing.

9. Go to the "File" menu and select "Save As" to save the captured events as an ETL file.

10. Open the saved ETL file in Microsoft Message Analyzer.

11. Use the filtering and analysis capabilities of Microsoft Message Analyzer to locate the specific errors or information related to free/busy or any other aspect of Outlook.exe.


Please note that analyzing ETL files can be a complex task, and familiarity with Microsoft Message Analyzer or similar tools is recommended for effective analysis. The steps provided above are a general guideline, and the exact steps may vary based on the version of Microsoft Message Analyzer or other tools you are using.

Remember to always exercise caution when analyzing ETL files and ensure you have the necessary permissions to access and analyze the files.


What if you want to enable this for some computers automatic in your Windows Domain on-premise with GPO?


Be carefull with the GPO. The Debug Mode of Outlook will generate 50MB+ Files maybe per minute. That is

why we want to target that Policy to a ADS-Group with Computer accounts. If you are unsure hot wo do that leave it and do it manual.

In addition to enabling Outlook debug logging using Group Policy Objects (GPO), you can further refine the application of the GPO by creating an Active Directory security group and assigning the GPO only to computers that are members of that group. This allows you to control which computers have the debug logging enabled. Here's how you can do it:

Step 2: Add Computers to the Security Group:

1. Locate the computers that you want to enable Outlook debug logging for.

2. Select the desired computers, right-click, and choose "Add to a group" or "Add to a security group" (depending on your Active Directory administrative tools).

3. Search for and select the "Outlook Debug Users" security group you created in Step 1.

4. Click "OK" to add the selected computers to the security group.

Step 3: Assigning the GPO to the Security Group:

1. Return to the Group Policy Management Console.

2. Right-click on the GPO that configures the Outlook debug logging settings and select "Properties".

3. In the "Properties" window, navigate to the "Security" tab.

4. Click the "Add" button to add a new permission entry.

5. In the "Select User, Computer, or Group" dialog box, enter the name of the security group ("Outlook Debug Users") and click "OK".

6. Back in the "Properties" window, select the added security group from the permission entries list.

7. Under the "Permissions for [Group Name]" section, check the "Read" and "Apply Group Policy" checkboxes.

8. Click "OK" to save the changes.

By following these steps, you have created an Active Directory security group, added the desired computers to that group, and assigned the GPO to the security group. This ensures that the Outlook debug logging settings will only be applied to computers that are members of the specified security group, allowing you to control the scope of the debug logging feature.


Some links regarding this:

Outlook Debug Logging


MS Learn:



Logfiles reference for different Outlook version:

Note The log files are created in multiple folders. These folders vary, depending on the version of Outlook that you're running.

Outlook 2010 Debug Logfiles

Log files in the %temp% folder

File name

Outlook RPC log


AutoDiscover log


Outlook/SharePoint synchronization logs

.htm and .xml files


Log files in the %temp%\OlkAS folder

File name

Availability Service, OOF, and Meeting Suggestion log files

date-time -AS.log

Protection Rules log files

date-time -PB4S.log

Unified messaging log files

date-time -UM.log

Unified Messaging configuration log files

date-time .UMCFG.log


Log files in the %temp%\OlkCalLogs folder

File name

Outlook calendar log files


Log files in the folder

%temp%\Outlook Logging

File name

Outlook advanced ETW log


MailTips log


OOF log


Transport log file


Outlook profile logs


SMTP log files


POP3 log files


IMAP log files


HTTP DAV log files


Outlook Hotmail Connector log files


Outlook Sharing Engine log files

SharingEngine date.log

Outlook-Windows Desktop Search indexing log files

data file name.log

Outlook first-run process log file


Outlook 2013 and Outlook 2016 Debug Logfiles

Log files in the %temp% folder

File name

Outlook/SharePoint synchronization logs

.htm and .xml files

Log files in the %temp%\EASLogFiles

File name

EAS logs for Hotmail accounts

.bin and .xml folders

Log files in the %temp%\OlkCalLogs folder

File name

Outlook calendar log files

OlkCalLog_date_time .etl

Log files in the folder

%temp%\Outlook Logging

File name

Advanced ETW log


Transport log file


Outlook profile logs



SMTP log files 

Note The log files are only logged in Outlook 2016 and earlier versions.


POP3 log files

Note The log files are only logged in Outlook 2016 and earlier versions.


IMAP log files

Note The log files are only logged in Outlook 2016 and earlier versions.


Outlook Sharing Engine log files

SharingEngine date.log

Outlook-Windows Desktop Search indexing log files

data file name.log

Outlook first-run process log file


Note You can sort by Date modified to find the files that were created most recently.



Client Management | Microsoft Exchange | Exchange 2016 | Exchange 2019 | M365/AZURE

Werbung von Drittfirmen (Nicht Butsch Informatik):

Werbung von Drittfirmen via Google Adsense: