PROBLEM: The M365 (cloud) users are unable to access free/busy info from the on-premise room mailbox and part two are unable to book it

In hybrid environment that combine on-premises and Microsoft 365 (M365) accounts, organizations may encounter difficulties when attempting to add conference rooms to meetings. This issue affects not only Outlook Web App (OWA) but also the Outlook desktop client (outlook.exe).

Problem Overview

The problem arises when attempting to book conference rooms from the room list in Outlook. In both OWA and Outlook desktop client, users may find that conference rooms are missing or unavailable for selection during the meeting scheduling process. This issue is particularly prevalent in hybrid environments where room mailboxes are synchronized between on-premises Exchange Server and M365.

Root Cause

The root cause of this problem lies in the configuration of the RoomList distribution group, which is responsible for organizing and displaying room mailboxes. In hybrid environments, the RoomList distribution group needs to be properly synchronized and maintained between on-premises and M365.

Inaccurate recipient TypeDetails

One potential cause is the incorrect recipientTypeDetails property associated with room mailboxes during synchronization. When on-premises room mailboxes are synchronized to M365, the recipientTypeDetails property may be set as "MailUser" instead of the expected value, "RoomMailbox". This discrepancy prevents the room mailboxes from being recognized as valid conference rooms in M365.

What is wrong again in detail:

In a Full hybrid Exchange <> M365 Setup with Room resources on on-premise side you see following effect.

Error #1:

The test.M365 users are unable to access free/busy info from the on-premise room mailbox as example Audi_A3_OST_179602@butschlab.ch.

Error#2.

The test.M365 users can see the calender and free/busy but they are unable to BOOK the room effective (No permission changed and users was in on-premise side before migration)

We have following users:

This is also valid for:

In a hybrid deployment of on-premises Microsoft Exchange Server and Exchange Online, assume that Exchange Online users select add room in a meeting request in Outlook Web App. In this situation, on-premises room mailboxes aren't displayed in the list as expected. Therefore, Exchange Online users can't add an on-premises conference room to a meeting request.

This is kind of unsupported solution as some people say at MS in Forum. On the other hand there is a KB which clearly explains part of the solution.

But only parts of it? Here is the full working solution:

Error #1:

The M365 users are unable to access free/busy info from the on-premise room mailbox.

1. You have to generate a ROOMLIST or lets say NEW Distribution List with the option –roomlist
2. In that room list you make all existing on-premise rooms member (Their object)
3. You already have OR newly generate a ROOM on the M365 Side m365.room@butschlab.ch (Make a DUMMY room on Cloud side)
4. You make the new m365.room ROOM also member of that Room List (Distribution group) < Most important STEP

Error#2.

The M365 users can see the calender and free/busy but they are unable to BOOK the room effective (No permission changed and users was in on-premise side before migration)

Here is how to see which rooms are affected:

Get-Mailbox -RecipientTypeDetails RoomMailbox | Get-CalendarProcessing | ?{!$_.ProcessExternalMeetingMessages}

You have to enable the flag -ProcessExternalMeetingMessages $true for the on-premise accounts. We never touched this in on-premise ONLY because it was rather dangerous if you did it wrong. If you did not follow compliance and enabled some things like INLCUDE/SHOW subject or BODY from meetings and ACTIVATE that wrong external people could you see conf. Data from inide the org.

GUI, notive that the user accounts of ROOM resources are always disabled. Keep them seperated so nobody (Azure Admin)

Does delete them when he has his cleaning day and want's to be a good employee.

Since you are in Hybrid Mode also check shortly both sides on-premise and M365 if you have the same settings there:

Set-OrganizationRelationship | fl

The KB was written for the OWA / WEAPP but we proof it's also valid for outlook.exe

Please notice that the KB does NOT contain the full solution as we provide in our Butsch KB.

https://learn.microsoft.com/en-US/exchange/troubleshoot/calendars/cannot-add-conference-rooms-to-meeting-in-owa#more-information

Maybe it's the TOKENS and AUTHENTICATION also so Check ANYTHING related to 4001 if you have OAUTH oder DAUTH:

http://www.butsch.ch/post/M365Exchange-Hybrid-OAuth-Testing-command-OAuth-Cert-out-of-sync-4001-IIS-VDIR-OAuth-wrong

EventID which may be related if you have OAUTH active

Event ID 1003: This event indicates successful OAuth authentication requests. It confirms that the authentication process between Microsoft 365 and on-premises Exchange was successful.

Event ID 1004: This event indicates failed OAuth authentication requests. It can provide details about the reason for the authentication failure, such as incorrect credentials or authorization issues.

Event ID 1022: This event is logged when an OAuth token is issued to a client application. It confirms that the token was successfully issued for the authentication process.

Please note that these specific event IDs are not guaranteed to be available in all versions of Exchange Server or in every configuration. The availability of events and their IDs can vary based on your Exchange Server version and the specific logging settings enabled.

Additionally, it's important to understand that the Application log on the Exchange server may not provide comprehensive logging specifically for OAuth access from Microsoft 365 to on-premises Exchange. As mentioned earlier, monitoring the client-side logs in Outlook and the logs in Microsoft 365 (such as audit logs and trace logs) will provide more detailed information for troubleshooting OAuth-related issues.

If you encounter specific error messages or need more specific guidance on troubleshooting OAuth access, I recommend referring to Microsoft's official documentation, reaching out to Microsoft Support, or consulting with an Exchange Server specialist for further assistance.

www.butsch.ch

Resolve and find OAuth problem in Exchange Hybrid Setup Enviroments

Short Understanding OAuth:

OAuth (Open Authorization) is an industry-standard protocol that enables secure authorization for third-party applications without the need to disclose user credentials. It allows users to grant limited access to their resources on one site to another site, without sharing their credentials. In the context of Exchange and Hybrid setups, OAuth provides a streamlined and secure way for users to access their emails, calendars, and other resources.

https://datatracker.ietf.org/doc/html/rfc6749

https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/auth-OAuth2

Since end 2022 every M365 TENANT has by default cloud side OAuth enabled. There is DAUTH (Older MS own made) and there is

an OPEN Source Authentication method called OAuth.

https://learn.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/deprecation-of-basic-authentication-exchange-online

You may land here if:

a) You just want to test if OAuth works on both sides between Exchange<>M365

b) Or your IIS Virtualdirectory do not accept OAuth beside NTLM etc.

c) If you have done a Self Signed OAuth Cert renew (The one with 2H+ UTC/CET Local time offset delay bug)

We explain how to fix and how to check if both sides have the latest certificate UTC time corrected.

So this KB is on how to TEST OAuth Exchange onpremise<>m365 but also why you came here because maybe it does not work and reports FAILURE not SUCCESS.

First let us check if your Microsoft 365 side has OAUTH enabled?

M365 side: Get-OrganizationConfig | FL oau*

Is Oauth working incoming and outgoing?

Following are detailed steps on how to test OAuth from from both sides in your Exchange <> M365 Hybrid Setup.

We have following users:

Short without any help or explanation:

Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.office365.com/ews/exchange.asmx -Mailbox test.onpremise@schweiz.ch -Verbose | fl

Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc -Mailbox test.onpremise@schweiz.ch -Verbose | fl

Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.schweiz.ch/ews/metadata/json/1 -Mailbox test.m365@schweiz.ch -Verbose | fl

Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover.schweiz.ch/autodiscover/autodiscover.svc -Mailbox test.m365@schweiz.ch -Verbose | fl

Reference for command Test-OAuthConnectivity:

https://learn.microsoft.com/en-us/powershell/module/exchange/test-OAuthconnectivity?view=exchange-ps

Default value for all TEST outgoing to M365: https://outlook.office365.com/ews/exchange.asmx

Default value for all TEST outgoing to M365: https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc

Test on On-premise > M365 Exchange outgoing direction:

If succesfull:

Test M365 > on-premise incoming direction:

Yellow change to what you have in Exchange Virtual Directory:

Retrieve those from your on-premise:

On-premise: Get-WebServicesVirtualDirectory | fl Extern*

Example: https://outlook.schweiz.ch

On-premise: Get-ClientAccessServer | fl autodis*

Example: https://autodiscover.schweiz.ch

Test https://autodiscover.schweiz.ch from external:

Open UP in a web browser from EXTERNAL (Mobily other network) and make sure you can LOGON to the Autodiscover URL with the on-premise account.

https://autodiscover.schweiz.ch/autodiscover/autodiscover.svc

Test on M365 side:

If succesfull:

Interpreting the Test Results:

After executing the Test-OAuthConnectivity command for both local and Microsoft 365 users, you will receive the test results. Here are a few possible outcomes:

Success:

If the test is successful, it means that the OAuth connectivity for the specified user is functioning correctly, and they can access Exchange resources seamlessly.

Failure:

If the test fails, it indicates an issue with the OAuth connectivity. Check for any misconfigurations in the OAuth configuration, network connectivity problems, or any other relevant errors. Troubleshooting these issues will be crucial to ensure uninterrupted access for the user.

Failure may be because of......

Failure:

CHECK1: OAuth not Set on VirtualDirectory EWS and AUTODISCOVER on Exchange on-premise

When troubleshooting OAuth-related issues, the first step is to ensure that OAuth is correctly set on the VirtualDirectory configurations for EWS (Exchange Web Services) and Autodiscover in the on-premises Exchange environment. In some cases, OAuth might not be properly configured or enabled on these VirtualDirectories, resulting in authentication failures and connectivity issues.

To address this, it is essential to verify and configure OAuth settings correctly on the EWS and Autodiscover VirtualDirectories, ensuring that the necessary authentication methods are enabled and OAuth is correctly configured.

Check VirtualDirectory on-premise:

Does it show OAuth right side?

Or yourself:

Get-AutodiscoverVirtualDirectory | fl *auth*

Get-WebServicesVirtualDirectory | fl *auth*

Failure:

CHECK2: OAuth Set, but does not PULL > FLIP-FLOP effect IIS VirtualDirectory Settings for EWS and AUTODISCOVER on Exchange on-premise

IF FAILURE: FLIP-FLOP the IIS Virtualdirectory Authentication Settings to be fireproof.

Even if ALL seems fine sometimes their is a FLIP FLOP effect which evn MS can't explain. So turn it OFF and TURN it ON.

Failure:

CHECK3: You have renewed the OAuth Certificate on the Exchange on-premise but

FAILURE: You did not do the last step in the KB on how to renew OAuth cert, Event 4001 Exchange on-premise

If you have recently renewed the OAuth certificate on your Exchange on-premises server but encounter authentication failures or other issues afterward, it is crucial to ensure that you have completed the final step mentioned in the corresponding knowledge base (KB) article.

Renewing the OAuth certificate involves multiple steps, and failure to follow the complete process can result in incomplete certificate installation or incorrect configuration. By carefully reviewing the KB article and verifying that all required steps, including post-renewal configuration, have been executed, you can avoid common pitfalls and ensure a successful OAuth certificate renewal.

https://learn.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-OAuth-expired

Failure:

Check IF Azure HAS your RE-done(latest) OAuth Exchange Certificate active

Azure side:

Connect to Azure from the machine where have acccess to Azure or the Exchange with the HCW Wizard.

Connect-MsolService (Logon with your Tenant or Admin account)

Get-MsolServicePrincipalCredential -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 -ReturnKeyValues $true

Note the StartDate and Enddate from the latest you made.

On-premise:

Get-ExchangeCertificate (Get-AuthConfig).CurrentCertificateThumbprint |fl Not*

Or CHECK below Script to get the time in UTC direct on-premise

 Attention Local Time Zone and Azure Time Zone are different. Remember when you did renew the OAuth cert

It took 2 or 4 hours to get active (OWA and PS was not wokting if so...)

Now hopefully, when doing the renew of the OAuth cert onpremise, you did not start 10 times.

AZURE = UTC

YOU = Local Time Zone +/- 2-4hrs or whatever

Use below script to get the time as it looks with azure

Re-check all as explained in:

https://learn.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-OAuth-expired

Failure:

FAILURE, 401 Access Denied: You don't not have all internal URL from your Exchange listed with AZURE as SPN

(He will accept only those address which the Cloud has SPN listed to easy explain)

https://learn.microsoft.com/en-us/exchange/troubleshoot/administration/401-access-denied-error-when-running-test-OAuthconnectivity

On-premise: Check the OnPremisesDiscoveryEndPoint and OnPremisesWebServiceEndPoint from:

Get-IntraOrganizationConfiguration | Fl

On-premise: Should be the same as per:

Get-WebServicesVirtualDirectory | FL ExternalUrl

Check Azure side:

Connect to Azure from the machine where have acccess to Azure or the Exchange with the HCW Wizard.

Connect-MsolService (Logon with your Tenant or Admin account)

(Get-MsolServicePrincipal -ServicePrincipalName "00000002-0000-0ff1-ce00-000000000000").ServicePrincipalNames

Check that ALL URL which you have on-premise are listed also there.

https://outlook.schweiz.ch

https://autodiscover.schweiz.ch

THUS

00000002-0000-0ff1-ce00-000000000000/outlook.schweiz.ch

00000002-0000-0ff1-ce00-000000000000/autodiscover.schweiz.ch

And any further DOMAINS you may have on your on-premise Exchange Domains .EU .DE .COM don't forget those.

External Links mentioned:

RFC6749:

https://datatracker.ietf.org/doc/html/rfc6749

Microsoft Azure Active Directory OAuth 2.0 Fundamentals:

https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/auth-OAuth2

Deprecation of Basic Authentication in Exchange Online:

https://learn.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/deprecation-of-basic-authentication-exchange-online

Test-OAuthConnectivity PowerShell cmdlet:

https://learn.microsoft.com/en-us/powershell/module/exchange/test-OAuthconnectivity?view=exchange-ps

Troubleshooting OAuth Expiration in Exchange:

https://learn.microsoft.com/en-us/exchange/troubleshoot/administration/cannot-access-owa-or-ecp-if-OAuth-expired

Configuring OAuth Authentication in Exchange 2013:

https://learn.microsoft.com/en-us/exchange/configure-OAuth-authentication-between-exchange-and-exchange-online-organizations-exchange

Enhancing Outlook Debug Logging for Troubleshooting

Mike Butsch, www.butsch.ch

What we want to do and why

Outlook debug logging is a valuable tool for diagnosing and resolving issues within Microsoft Outlook. By enabling advanced logging, you gain deeper insights into the application's behavior, allowing for more effective troubleshooting. In this blog post, we will explore the process of enabling global and advanced logging for Outlook, along with additional steps to enhance the logging capabilities.

In conclusion, by enabling and leveraging Outlook debug logging, you can gain valuable insights into the behavior of Microsoft Outlook and efficiently troubleshoot any issues that may arise. Remember to exercise caution when modifying the Windows Registry and follow the necessary steps outlined in this blog post.

Some key points in this paper:

* Attention: Do not make user you analyzer Local Adminstrator

* How to make the KEY under another user Registry hive

* Your are admin or user how to generate the Registry Key

* How to find something or for what to search for in the Debug Logfiles you made:

* Analyze WPA and ETL files

* Outlook 2010 Debug Logfiles

* Outlook 2013 and Outlook 2016 Debug Logfiles

* What if you want to enable this for some computers automatic in your Windows Domain on-premise with GPO?

in addition to the steps mentioned in the documentation, there is an additional parameter that can be enabled to gather more detailed information. This parameter involves modifying a specific subkey in the Windows Registry. Please note that the following steps require careful attention to avoid unintended issues:

Attention: Do not make user you analyser Local Adminstrator, this will produce probleme related to Exchange, ActiveSync GPO and Azure sync (Infos not Synced back to local Domain from Azure for that user)

http://www.butsch.ch/post/Active-Directory-accounts-with-ADMINSholderadminCount-flag-%7C-No-syncback-from-Azure-ms-ds-consistencyGuid

Note: If the user experiencing the issue does not have sufficient permissions to create the key, it is important not to grant them local admin privileges without addressing the "ADMINSholder" flag attribute. Granting local admin privileges without correcting the "ADMINSholder" flag may lead to further complications. Refer to these resources for more information on the "ADMINSholder" flag:

Attention: https://www.butsch.ch/post/Activesync-with-Exchange-2013-does-not-work-ADMINSHOLDER-Flag-(an-old-bad-friend) or https://www.butsch.ch/post/Exchange-Activesync-1053-Event-4003-Error-2007201020132016-Adminsholder

Again a sample why you should not make the user Local Admin to explain: Also be aware that if you are in M365 Azure Hybrid Mode those account with ADMINS Holder Flag Set wan't sync back or to Azure.

Target what we need for the USER who has the OUTLOOK.EXE Problem. That may be a different user than the LOGGED on s non domain joined machine.

How to set key for another user?

It's important to note that the user encountering the "OUTLOOK.EXE" problem may not be the same as the one logged on to the non-domain joined machine. Therefore, we need to set the key for the relevant user. Here is a brief explanation of how to set the key for another user:

1. Create a CMD shortcut on the desktop.

2. Right-click on the shortcut while holding down the SHIFT key and choose "Run as different user."

3. A command prompt (CMD.exe) will open.

4. Launch the Registry Editor (regedit.exe) from the command prompt.

5. Locate the appropriate registry hive for the user you want to debug in Outlook.

INFO: Why not make him local Admin again because most don't understand impact if not corrected afterwards! Please read carefull above IF you have that idea.

How to make the KEY under another user Registry hive Explanation #1

Solution you need to run regedit.exe under a seperate LOCAL ADMIN or Service account to have the permission to CREATE that missing key structure per user. Also keep in MIND that existing POLICY/GPO will overwrite the settings within 15-X minutes if you may have those set in Domain enviroment.

Find the office version "SHORTY" the user has as exmaple "15.0", "16.0" etc.

Your are admin or user and how to generate the Registry Key Explanation #2

When troubleshooting Outlook issues for a specific user, there are two options to access the user's registry hive.

By following either of these options, you can access the user's registry hive and make necessary

modifications to troubleshoot the Outlook.exe problem. Remember to exercise caution and ensure you

have the appropriate permissions and authority to access the user's registry hive.

FastShutdownBehavior

Once you have made the necessary changes to the registry, log off and log back on to apply the settings. Launch Outlook, and you should now see the logging in action. This allows you to identify any problematic areas or errors that may be occurring.

Check the key again under the HKCU > OK

Start Outlook, yes loggin active and you see warning

Check the things that don't work to produce logs. Click around and open things and other Calendar from OAB adressbook as example.

Check the logs generates

Folders:

To gather relevant log files for troubleshooting purposes, you can focus on specific folders or actions that are not functioning as expected. These log files can be shared with Microsoft Support to assist in resolving the issue. To convert the log files (ETL format) into a more readable format, you can use the PowerShell command `tracerpt.exe FILENAME.ETL -lr`. You can then preview the converted files or open them using the Event Viewer (`eventvwr.exe`) to examine the events more closely.

Convert in Powershell to some TEXT/XML.

ETL > Event Viewer ;-) ???

You can then preview the converted files or open them using the Event Viewer (`eventvwr.exe`) to examine the events more closely.

Filter may help if there are any error?

How to find something or for what to search for in the Debug Logfiles you made:

When analyzing ETL or XML files for debug information related to free/busy functionality in Outlook, you can look for specific events and data related to free/busy operations. Here are some key points to consider:

1. Event ID: Look for events with specific event IDs that pertain to free/busy operations. These event IDs can vary depending on the version of Outlook and the specific scenario. Typically, you may find events with IDs such as 2000, 2007, or 2016, which are commonly associated with free/busy processing.

2. Provider Names: Look for events associated with providers or services related to free/busy functionality. These providers can include Autodiscover, Exchange Web Services (EWS), Availability Service, or any other components involved in free/busy data retrieval and processing.

3. Timestamps: Pay attention to the timestamps of the events and their sequencing. This can help you understand the flow and timing of free/busy operations. Look for patterns, delays, or any inconsistencies that might indicate potential issues.

4. Error Codes and Messages: Take note of any error codes or error messages associated with the free/busy events. These can provide valuable information about the nature of the problem or any specific errors encountered during free/busy processing.

5. Data Fields: Look for specific data fields or attributes that contain information related to free/busy functionality. These can include user identifiers, calendar data, availability status, or any other relevant data that helps in understanding the free/busy operations.

To locate this debug information within the ETL or XML file, search for the relevant event IDs, provider names, error codes, or specific data fields mentioned above. Use the search functionality in your text editor or XML viewer to locate and analyze the corresponding events. Additionally, you can filter the events based on the time range or other criteria to focus on the relevant information.

By carefully examining these events and data within the ETL or XML file, you can gain insights into the free/busy operations, identify any errors or delays, and troubleshoot any issues affecting the free/busy functionality in Outlook.

Analyze WPA and ETL files

Also read following KB on how to process the files further with WPA

https://learn.microsoft.com/en-us/windows-hardware/test/wpt/opening-and-analyzing-etl-files-in-wpa

https://learn.microsoft.com/en-us/message-analyzer/message-analyzer-tutorial

If you are specifically working with ETL files generated from Outlook.exe debug mode the files are binary files containing event traces, and parsing them requires specific tools and techniques.

To analyze ETL files generated from Outlook.exe debug mode, you can use Microsoft Message Analyzer (MMA) or Windows Performance Analyzer (WPA), which are powerful tools for analyzing event traces. These tools allow you to load the ETL file and apply filters to extract specific information or errors related to Outlook.exe.

Here's a high-level overview of the steps involved in using Microsoft Message Analyzer to analyze Outlook.exe ETL files:

1. Download and install Microsoft Message Analyzer from the Microsoft Download Center.

   https://learn.microsoft.com/en-us/message-analyzer/installing-and-upgrading-message-analyzer

2. Launch Microsoft Message Analyzer.

3. Go to the "File" menu and select "New Session" to create a new session.

4. In the "New Session" window, click on the "Live Trace" tab.

5. Select the "Microsoft-Windows-Diagnostics-Performance" provider from the list.

6. Click on the "Capture" button to start capturing events.

7. Reproduce the issue or scenario for which you want to capture the event trace.

8. Once the desired events have been captured, click on the "Stop" button to stop capturing.

9. Go to the "File" menu and select "Save As" to save the captured events as an ETL file.

10. Open the saved ETL file in Microsoft Message Analyzer.

11. Use the filtering and analysis capabilities of Microsoft Message Analyzer to locate the specific errors or information related to free/busy or any other aspect of Outlook.exe.

Please note that analyzing ETL files can be a complex task, and familiarity with Microsoft Message Analyzer or similar tools is recommended for effective analysis. The steps provided above are a general guideline, and the exact steps may vary based on the version of Microsoft Message Analyzer or other tools you are using.

Remember to always exercise caution when analyzing ETL files and ensure you have the necessary permissions to access and analyze the files.

What if you want to enable this for some computers automatic in your Windows Domain on-premise with GPO?

Be carefull with the GPO. The Debug Mode of Outlook will generate 50MB+ Files maybe per minute. That is

why we want to target that Policy to a ADS-Group with Computer accounts. If you are unsure hot wo do that leave it and do it manual.

Some links regarding this:

Outlook Debug Logging

https://support.microsoft.com/en-gb/topic/how-to-enable-global-and-advanced-logging-for-microsoft-outlook-15c74560-2aaa-befd-c256-7c8823b1aefa

MS Learn:

https://learn.microsoft.com/en-us/outlook/troubleshoot/performance/enable-and-collect-logs-for-profile-creation-issues

Logfiles reference for different Outlook version:

Note The log files are created in multiple folders. These folders vary, depending on the version of Outlook that you're running.

Outlook 2010 Debug Logfiles

Log files in the folder

Outlook 2013 and Outlook 2016 Debug Logfiles

Log files in the folder

Note You can sort by Date modified to find the files that were created most recently.

What is this about keywords: M365, ADMINSholder/adminCount, Synchronization by Azure AD Connect, ImmutableId, WP;ms-ds-consistencyGuid

Event: ADSYNC, 6100

Error in Synchronisation Service Manager: Insufficient access rights to perform the operation, 8344, permission-issue

Problem:

Synchronisation Service Manager (Microsoft Azure AD Connect Synchronization Services) does not sync the attribute WP;ms-ds-consistencyGuid Back from Azure to Local Active Directory.

This is because a local normal employee user has the ADMINSholder/adminCount flags set to 1/true. (Which is not good in Domain Enviroments)

This is by purpose to protect sensitive Windows account of been written wrong by error.

When a user has been temporarily part of the "Local Admin group" or a member of specific ADS groups like Print Operator or Backup Operators, they receive a special FLAG (ADMINSholder/adminCount=1). The objective is to remove this flag if it is no longer necessary or remove the user from those groups altogether. If not done these accounts may encounter limitations when it comes to functionality with ActiveSync, GPO, and Azure/M365.

If you would like to gain a better understanding of what ADMINSholder/adminCount entails, we recommend reading our informative blog posts. They provide insights into the impact of this flag, especially concerning ActiveSync and GPO. Here are the steps to remove the flag:

https://www.butsch.ch/post/Activesync-with-Exchange-2013-does-not-work-ADMINSHOLDER-Flag-(an-old-bad-friend)

https://www.butsch.ch/post/Exchange-Activesync-1053-Event-4003-Error-2007201020132016-Adminsholder

Please be aware that ADMINSholder Attribute is here to PROTECT sensitive accounts like ADMINISTRATOR, Service, IIS-accounts and resets or overwrites them every 15 minutes to prevent catastrophic impact of wrong GPO/Policies or Powershell commands who manipulate such accounts.

Here is Powershell sample to find the users under your ROOT ADS with excluding certain OU Llike deactivated or service:

Get-ADUser -Filter {admincount -gt 0} -Properties adminCount -ResultSetSize $null -SearchBase "DC=yourdomain,DC=local" | Where-Object { $_.DistinguishedName -notlike "*OU=Benutzer_deaktiviert*" -and $_.DistinguishedName -notlike "*OU=SERVICE_users_IT*" -and $_.DistinguishedName -notlike "*CN=Users,DC=yourdomain,DC=local" } | Format-List DistinguishedName, Enabled, SamAccountName

Change the yellow field to your enviroment or what your OU-folders contain.

This will provide you with a list of on-premises users who have the ADMINSholder FLAG set and will not synchronize back from Azure/M365 to your local Active Directory.

This is because Microsoft wants to protect those special accounts. (AdminSholder refers to a special account.)

Synchronisation Service Manager, Microsoft Azure AD Connect Synchronization Services

Event, ADSYNC, 6100,

The error (Or not because it helps you to find error on-premise and protects you from mistakes) as you see it

There is a user named CLAUDIO whose ms-ds-consistencyGuid value does not sync back from Azure to the Local Active Directory.

This synchronization failure is due to the Sync tool recognizing that the local user has the ADMINSHolder/Admincount FLAG set to 1 (true). As a result, the synchronization process does not update the value in the Local Active Directory.

Links:

https://techcommunity.microsoft.com/t5/core-infrastructure-and-security/hybrid-identity-getting-users-aligned/ba-p/2274690

Primary goal would be to FIX those accounts who are regular users and have the ADMINSolder/admount=1 flag set. You can do this manual:

https://www.butsch.ch/post/Activesync-with-Exchange-2013-does-not-work-ADMINSHOLDER-Flag-(an-old-bad-friend)

https://www.butsch.ch/post/Exchange-Activesync-1053-Event-4003-Error-2007201020132016-Adminsholder

Or you use a scripted solution. Be carefull you don't touch any service accounts my mistake.

C7Solutions has an excellent write up of the issue and also scripts with which you can modify the permission.

https://c7solutions.com/2015/07/configuring-writeback-permissions-in-active-directory-for-azure-active-directory-sync

Also Microsoft has some good Blogs posts:

https://techcommunity.microsoft.com/t5/core-infrastructure-and-security/hybrid-identity-getting-users-aligned/ba-p/2274690

How we found this info beside Technet:

Error: Outlook 2016 verlangt neues Update (Aktuell: 16.0.4266.1001 / Erforderlich: 16.0.4600.1000)

There is a link in the warning which leads to the rather delicate info abour EOL of Office 2016/2019 with M365.

Fact:

If you don't want to update your Office 2016/2019, keep your Exchange on-premise DAG with KEMP and all is fine ;-)

If you run CITRIX and use Microsoft Office, you have a problem that you will have to solve by 2025.

News or not? Since 2017 in a blog published?

End of support for Office 2016 and 2019 for customers who have their email accounts in Microsoft Office M365 is set for October 2023.

We often warn customers that by moving certain things to the Microsoft Cloud, they open themselves up to new dependencies or problems of another kind. These can be plugins from third-party suppliers that don't support the latest Office version or simply that their users are not educated on the latest release.

Here is one issue that we often see customers falling into.

If your email Exchange account is not on-premises and you have an M365 account, you are FORCED to use a newer Microsoft Office version by October 2023. This was communicated in an April 2017 blog post from Microsoft. "We won't take any active measures to block older Office versions from connecting to Microsoft 365 services if they're in extended support and are kept up to date"

https://learn.microsoft.com/en-us/lifecycle/faq/extended-security-updates

Just like with Windows 7 or 2008 R2, enterprise customers had the option to BUY Post Service Support through ESU (meaning patching and getting updates for old OS). This is what is meant by extended support for the connection to M365.

On a full updated W10 with Outlook.exe 2016 VL:

We haven seen this on a TEST Eval machine we use for testing things at a customer site.

W10, Outlook 2016 VL. While adding an add M365 Outlook account to the existing

on-premise Exchange account we suddenly had a warning in Outlook we never did

see before in Domain Enviroment. All our productive and client machines are allways updated

7+ days past patchday so we never did see the link or warning i guess.

We downloaded last patch we could find with that number 16.0.4600.1000 info

If you want to receive the Outlook Update and are not DOMAIN JOINED and internal WSUS you

Should activate this option on W10 to gt Office 2016 Updates (Just a side note....). We don't

See that often because we all have WSUS for customers.

In the Outlook itself you will see this warning and LINK to this URL.

https://learn.microsoft.com/en-gb/deployoffice/endofsupport/microsoft-365-services-connectivity

Older Office versions not supported for connecting to Microsoft 365 services

Older Office versions might still be able to connect to Microsoft 365 services, but that connectivity isn't supported.

In practical terms, what this means is that these older Office versions might not be able to use all the latest functionality and features of Microsoft 365 services. In addition, over time, these older versions might encounter other unexpected performance or reliability issues while using Microsoft 365 services. That's because as we make improvements to Microsoft 365 services, we're not taking into account or testing with these older Office versions.

We won't take any active measures to block older Office versions from connecting to Microsoft 365 services if they're in extended support and are kept up to date. This includes Office 2019 and Office 2016 after October 10, 2023. Both of these versions are in extended support until October 14, 2025.

Therefore, to provide the best experience with using Microsoft 365 services, we strongly recommend that you move off older Office versions to versions supported for connecting to Microsoft 365 services.

TIP: Cleanup everything LOCAL before you even think of moving anything to M365 or Azure or even starting the Connector

PRO TIP: Full manual list of Objects/attribute to check on your local ADS in this blog.

This blog entry is mainly about those two steps of the MS Technet: https://learn.microsoft.com/en-us/microsoft-365/enterprise/prepare-for-directory-synchronization?view=o365-worldwide

1. Directory Clean-up Tasks
2. Directory object and attribute preparation

which are often just skipped because people think all is fine because they logon to their client an IT runs. Normally the things we talk about here are checked and cleaned with any serious IT-Service company who migrated something in direction of LDAP/Windows Domain Controller/Active Directory or Exchange inhouse.

But time may have passed since then so it's time to check again like the OnPremise migrations and one of the reasons why it's was a little more expensive inhouse. However, daily business shows that it's the same related to cloud. The difference is just that people who never really done such complex things inhouse now think all the problems are gone.

This is a collection of things we noticed during some test and inspection of logs and dumps from existing migrations.

As with Exchange 2 Exchange Onpremise or with Active Directory Domain Controller Migrations. It is recommended to clean-up the source that means your existing environment as clean as possible.

You do not want to run into errors during the sync or setup. Prevent it, think ahead, and do not be too naive. Problems want go away just because you move everything to the cloud that were there before.

All the things that are wrong come up with a migration even if things worked before and you lived with it.

Warning

MS describes it in a clear warning on most of their pages somewhere. In addition, as always on Technet if something is mentioned with a "!" in that form read it.

Like if, you remove last Hybrid in Onpremise and you remove the Exchange ORG. You do not ready no E-Mail next day for the cleaners who wanted to make all proper and nothing in the server room!

But maybe someone who migrated you last on-premise to on-premise fixed those for you already? Mainly most of these did happen in an environment, which have grown over the years from NT/2000/2003/2008/2016 etc.

So if someone cleaned already you may have an easy ride and even do not understand what those Exchange Experts done all the time because it was so easy now for you as cloud Manager. Forums are full with those it happens in Azure/M365 too ;-)

https://www.butsch.ch/post/Exchange-Error-when-you-want-to-change-a-Receive-Connector-TLS-with-a-Cert-with-no-Common-Name

 Some things we did see: and to check:

• Things like special Chars somewhere where they not should be like "\ < > ( ) ; , [ ] [ \ " | , / : < > + = ; ? * '] > IN: sAMAccountName or userPrincipalName
• Too many groups in other groups or user in too many groups (Everything is possible in Enterprise with Academic naming convention and strategy)

https://www.butsch.ch/post/Exchange-20132016-and-2010-Proxy-back-(400)-Bad-Request-ADS-user-in-too-many-ADS-groups-member

• User get's it done to get a special Category Color in Public Folders events? Yes possible and failure did happen.

https://www.butsch.ch/post/Exchange-Public-Folder-Migration-Fixing-malformed-Category-VBA-Script-Swiss-version

Then there are points which may have been changed after the last person did a serious Exchange<>Exchange inhouse migration otherwise they would not be there.

PROXYADDRESSES > those get built by Exchange itself and are not relevant inhouse except for routing in ENT until mig or now

For their DFB-11 DE, EMEA Region ;-)

Stop using ö, ä, ü in IT identifiers we tell you since 30 years. Let them Mary and change the name it's cheaper! Or just don't use them.

Every 5 years there is a technology, which does not swallow UMLAUT it and that fails because of it.

Your Smartcard to enter the last ship, which will fly to Mars, will fail because you have müller on it instead of mueller

All other steps please reference the MS KB. We just wanted to highlight something, which should NOT be there. If you find ANY

Of those mentioned above, we recommend checking all other points VERY carefully. Find as much info as you can about the pre checks.

No Easy Solution?

• There are many scripts around from ADS/DC and Exchange Migrations to check these things
• CLOUD MANAGER: Also below see the IDFIX which may help with some errors.
• PRO: Check the list below we made "Manual Reference Butsch what to check on on-premise enviroment before you start any Hybrid M365/Azure project, collected from different sources 02.02.2023"

Be careful with script that also Change things in same step. Best is a script which checks and reports and then has an option to change and also logs.

Also do not be Junior System Powershell-just-back-from-education and just run it first thing you find on google without a backup before?

1. Check the regular Backup before you touch (Really do it open the console view the job or let the Backup Admin show you): Acronis, NetB, BEXEC, veeeam, and for Windows Domain just a regaultr Windows Backup so Tier3 MS will support it!
2. Make an LDAP Export of ADS just as second backup (Example: ldifde -f C:\edv\export.ldf -d "CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=ch" -p subtree)

https://www.butsch.ch/post/ADSIEDIT-replacement-and-tools

Believe us:

https://redmondmag.com/articles/2017/08/22/edit-the-active-directory-using-adsi-edit.aspx

MS Links in general to that:

https://learn.microsoft.com/en-us/microsoft-365/enterprise/prepare-for-directory-synchronization?view=o365-worldwide

AD DS Preparation

To help ensure a seamless transition to Microsoft 365 by using synchronization, you must prepare your AD DS forest before you begin your Microsoft 365 directory synchronization deployment.

Your directory preparation should focus on the following tasks:

• Remove duplicate proxyAddress and userPrincipalName attributes
• Update blank and invalid userPrincipalName attributes with valid userPrincipalName attributes

Remove invalid and questionable characters in the givenName, surname ( sn ), sAMAccountName, displayName, mail, proxyAddresses, mailNickname, and userPrincipalName attributes. For details about preparing attributes, see List of attributes that are synced by the Azure Active Directory Sync Tool.

 PITFALLS: The TOP or First Level Domain of the FQDN of your Active Directory has to Routable in Internet

 https://learn.microsoft.com/en-us/windows-server/identity/ad-fs/operations/configuring-alternate-login-id#applications-and-user-experience-after-the-additional-configuration

If you Windows Active Directory Domain is like it was learned correct in 1999 then i will end as .local as best practice.

What is what if you did not work for an ISP and have other things to do in life...

Secondleveldomain    firstleveldomain (top domain)

mycompany        local

mycompany.local < This is a non routable DNS Domain

ads.meinfirma.local < This is a non routable DNS Domain

ads.meinefirma.ch < This is a routable DNS Domain

If your internal Windows Domain is something green above > Good

If you have something that is like red above > Not good but can fix here is how:

You may have to adapt a second Domain name as you maybe already own from your Internet Web Site or if you have an SMTP MX from your E-Mail system. (Check you Internet Provider/Domains/DNS/MX). You will have to prove to Azure that you own the Domain with a special DNS entry.

Maybe you noticed the time where you could not get a Security Certificate for your .local domain. Maybe you already have a second UPN per user?

Microsoft explains it here maybe you understand maybe not. Clear up with Exchange People and security before you do this on Active Directory. It may have a lot if impact if you have a third party software, which enumerates these, fields wrong [Yes seen....] (Breaking things on other side).

https://learn.microsoft.com/en-us/microsoft-365/enterprise/prepare-a-non-routable-domain-for-directory-synchronization?view=o365-worldwide

You can do this DOMAIN Wide is MS Explains this will be for further users.

As with Exchange Domains this is not adapted at once to existing users in some enviroments.

Ir you can do this just for one user maybe for testing or the ones who you will have Hybrid etc.

For sure you can do this with Powershell but as always TEST on a VM DC lab and limit the change to an OU if possible so you don't HIT service accounts, IT accounts etc.

$LocalUsers = Get-ADUser -Filter "UserPrincipalName -like '*contoso.local'" -Properties userPrincipalName -ResultSetSize $null

$LocalUsers | foreach {$newUpn = $_.UserPrincipalName.Replace("@contoso.local","@contoso.com"); $_ | Set-ADUser -UserPrincipalName $newUpn}

External Links to blogs: 

https://blogs.perficient.com/2015/07/07/office-365-why-your-upn-should-match-your-primary-smtp-address/

https://www.essential.co.uk/blog/articles/office-365-upn-smtp-mismatch-fix/

PITFALL 3: Fixing UPN on-premise by Markus Vilcinskas (Scripts moved to EXE solution called IDFIX)

As example, somehow, people get it done that on both sides on-premise and in Azure you have objects with the same UPN name.

There where once some Powershell script sets posted under the links below. However, you are unable to download those mentioned.

MS replaced and consolidated on a .NET EXE tool so the Cloud Managers can also run it. The Script seemed to have a too large impact when running

Wrong by people who suddenly run and unerstood PS and never used ADSIedit or DCDIAG.exe before ;-)

https://redmondmag.com/articles/2017/08/22/edit-the-active-directory-using-adsi-edit.aspx

Here is the tool which will report all errors on your ADS on-premise:

idFix is used to perform discovery and remediation of identity objects and their attributes in an on-premises Active Directory environment in preparation for migration to Azure Active Directory. IdFix is intended for the Active Directory administrators responsible for directory synchronization with Azure Active Directory.

The purpose of IdFix is to reduce the time involved in remediating the Active Directory errors reported by Azure AD Connect. Our focus is on enabling the customer to accomplish the task in a simple expedient fashion without relying upon subject matter experts.

The Microsoft Office 365 IdFix tool provides the customer with the ability to identify and remediate object errors in their Active Directory in preparation for deployment to Azure Active Directory or Office 365. They will then be able to successfully synchronize users, contacts, and groups from the on-premises Active Directory into Azure Active Directory.

Info:

https://microsoft.github.io/idfix/

Download:

https://raw.githubusercontent.com/Microsoft/idfix/master/publish/setup.exe

Microsoft is working to reduce the time required to remediate identity issues when onboarding to Microsoft 365. A portion of this effort is intended to address the time involved in remediating the Windows Server Active Directory (Windows Server AD) errors reported by the directory synchronization tools such as Azure AD Connect and Azure AD Connect cloud sync. The focus of IdFix is to enable you to accomplish this task in a simple, expedient fashion.

The IdFix tool provides you the ability to query, identify, and remediate the majority of object synchronization errors in your Window's Server AD forests in preparation for deployment to Microsoft 365. The utility does not fix all errors, but it does find and fix the majority. This remediation will then allow you to successfully synchronize users, contacts, and groups from on-premises Active Directory into Microsoft 365. Note: IdFix might identify errors beyond those that emerge during synchronization. The most common example is compliance with rfc 2822 for smtp addresses. Although invalid attribute values can be synchronized to the cloud, the product group recommends that these errors be corrected.

This was sourced from those scripts before:

OLD LINK: https://social.technet.microsoft.com/wiki/contents/articles/16692.how-to-use-powershell-to-fix-duplicate-user-principal-name-for-on-premises-active-directory-users.aspx

https://learn.microsoft.com/en-GB/microsoft-365/troubleshoot/active-directory/duplicate-attributes-prevent-dirsync

FIXID Proxyaddress Error:

For the PROXY Error we assume this will be changed by the latest version of the Hybris Wizard because MS describes that RUS on-premise will be changed. 

We assumed AS long as the Primary (FAT) SMTP: address is the correct this would work. But we also see that MS changes it on-premise with the wizard (new or since ever)

https://learn.microsoft.com/en-us/exchange/troubleshoot/administration/primary-smtp-address-is-overwritten-by-targetaddress

https://learn.microsoft.com/en-us/exchange/hybrid-configuration-wizard

Primary SMTP proxy address is replaced by targetAddress value in a hybrid environment

You find following code which is run through the wizard on-premise:

Set-EmailAddressPolicy -Identity "Default Policy" -ForceUpgrade "True" -EnabledEmailAddressTemplates ("SMTP: @domain.com", "smtp:%m@domain.mail.onmicrosoft.com", + "SMTP: @domain.com" + "smtp:%m@domain.mail.onmicrosoft.com")

Update-EmailAddressPolicy -Identity "Default Policy" -UpdateSecondaryAddressesOnly "True" -DomainController "GlobalCatalog.domain.com"

If you want to make a script which changes that PLEASE note:

The field "email" (Under Street/address) should be in sync with PROXYaddress but often it's not.

The field E-Mail main focus is for CRM/ERP reference or for vcards.

The PROXYaddress primary SMTP is for routing that's the one you should select to fetch the E-Mail address for the Hybrid Wizard selection.

That's also what ID FIX checks. 

As per MS Reference the field email is Synced FROM Proxyaddress but depending on the WAY you handle the ADS user not anytime. (See posts below link)

https://techcommunity.microsoft.com/t5/exchange-team-blog/fun-with-changing-e-mail-addresses/ba-p/609781