Click on the Category button to get more articles regarding that product.
www.butsch.ch
Resolve and find OAuth problem in Exchange Hybrid Setup Environment
Short Understanding OAuth:
OAuth (Open Authorization) is an industry-standard protocol that enables secure authorization for third-party applications without the need to disclose user credentials. It allows users to grant limited access to their resources on one site to another site, without sharing their credentials. In the context of Exchange and Hybrid setups, OAuth provides a streamlined and secure way for users to access their emails, calendars, and other resources.
https://datatracker.ietf.org/doc/html/rfc6749
https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/auth-OAuth2
Since end 2022 every M365 TENANT has by default cloud side OAuth enabled. There is DAUTH (Older MS own made) and there is an OPEN Source Authentication method called OAuth.
You may land here if:
a) You just want to test if OAuth works on both sides between Exchange<>M365
b) Or your IIS Virtualdirectory do not accept OAuth beside NTLM etc.
c) If you have done a Self Signed OAuth Cert renew (The one with 2H+ UTC/CET Local time offset delay bug)
d) You did a RENEW of the OAUTH Cert like under:
https://learn.microsoft.com/en-us/exchange/plan-and-deploy/integration-with-sharepoint-and-skype/maintain-oauth-certificate?view=exchserver-2019#how-to-rotate-the-exchange-server-auth-certificate
We explain how to fix and how to check if both sides have the latest certificate UTC time corrected.
So this KB is on how to TEST OAuth Exchange on-premises<>m365 but also why you came here because maybe it does not work and reports FAILURE not SUCCESS.
First let us check if your Microsoft 365 side has OAUTH enabled?
M365 side: Get-OrganizationConfig | FL oau*
Is Oauth working incoming and outgoing?
Following are detailed steps on how to test OAuth from from both sides in your Exchange <> M365 Hybrid Setup.
We have following users:
| User on-premise | User M365 |
| test.onpremise@schweiz.ch | test.m365@schweiz.ch |
Short without any help or explanation:
Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.office365.com/ews/exchange.asmx -Mailbox test.onpremise@schweiz.ch -Verbose | fl
Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc -Mailbox test.onpremise@schweiz.ch -Verbose | fl
Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.schweiz.ch/ews/metadata/json/1 -Mailbox test.m365@schweiz.ch -Verbose | fl
Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover.schweiz.ch/autodiscover/autodiscover.svc -Mailbox test.m365@schweiz.ch -Verbose | fl
Reference for command Test-OAuthConnectivity:
https://learn.microsoft.com/en-us/powershell/module/exchange/test-OAuthconnectivity?view=exchange-ps
Default value for all TEST outgoing to M365: https://outlook.office365.com/ews/exchange.asmx
Default value for all TEST outgoing to M365: https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc
Test on On-premise > M365 Exchange outgoing direction:
| Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.office365.com/ews/exchange.asmx -Mailbox test.onpremise@schweiz.ch -Verbose | fl |
| Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover-s.outlook.com/autodiscover/autodiscover.svc -Mailbox test.onpremise@schweiz.ch -Verbose | fl |
If succesfull:
Test M365 > on-premise incoming direction:
Yellow change to what you have in Exchange Virtual Directory:
Retrieve those from your on-premise:
On-premise: Get-WebServicesVirtualDirectory | fl Extern*
Example: https://outlook.schweiz.ch
On-premise: Get-ClientAccessServer | fl autodis*
Example: https://autodiscover.schweiz.ch
Test https://autodiscover.schweiz.ch from external:
Open UP in a web browser from EXTERNAL (Mobily other network) and make sure you can LOGON to the Autodiscover URL with the on-premise account.
https://autodiscover.schweiz.ch/autodiscover/autodiscover.svc
Test on M365 side:
| Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.schweiz.ch/ews/metadata/json/1 -Mailbox test.m365@schweiz.ch -Verbose | fl |
| Test-OAuthConnectivity -Service AutoD -TargetUri https://autodiscover.schweiz.ch/autodiscover/autodiscover.svc -Mailbox test.m365@schweiz.ch -Verbose | fl |
If succesfull:
Interpreting the Test Results:
After executing the Test-OAuthConnectivity command for both local and Microsoft 365 users, you will receive the test results. Here are a few possible outcomes:
Success:
If the test is successful, it means that the OAuth connectivity for the specified user is functioning correctly, and they can access Exchange resources seamlessly.
Failure:
If the test fails, it indicates an issue with the OAuth connectivity. Check for any misconfigurations in the OAuth configuration, network connectivity problems, or any other relevant errors. Troubleshooting these issues will be crucial to ensure uninterrupted access for the user.
Failure may be because of……
| CHECK1:OAuth not Set on VirtualDirectory EWS and AUTODISCOVER on Exchange on-premise |
| CHECK2:OAuth Set, but experiencing a PULL/REDO issue > FLIP-FLOP effect on IIS VirtualDirectory Settings for EWS and AUTODISCOVER on Exchange on-premise (That what MS says is possible….) |
| CHECK3:You have renewed the OAuth Certificate on the Exchange on-premise butFailure due to OAuth Certificate Renewal without completing the final step in the KB |
Failure:
CHECK1: OAuth not Set on VirtualDirectory EWS and AUTODISCOVER on Exchange on-premise
When troubleshooting OAuth-related issues, the first step is to ensure that OAuth is correctly set on the VirtualDirectory configurations for EWS (Exchange Web Services) and Autodiscover in the on-premises Exchange environment. In some cases, OAuth might not be properly configured or enabled on these VirtualDirectories, resulting in authentication failures and connectivity issues.
To address this, it is essential to verify and configure OAuth settings correctly on the EWS and Autodiscover VirtualDirectories, ensuring that the necessary authentication methods are enabled and OAuth is correctly configured.
Check VirtualDirectory on-premise:
| Save and run “vdir.sp1” relevant for Exchange<>M365 OAuth |
| $VirtualDirectories = @()$VirtualDirectories += Get-WebServicesVirtualDirectory | Select-Object Name, InternalAuthenticationMethods, ExternalAuthenticationMethods$VirtualDirectories += Get-AutodiscoverVirtualDirectory | Select-Object Name, InternalAuthenticationMethods, ExternalAuthenticationMethodsforeach ($VirtualDirectory in $VirtualDirectories) {Write-Host “Virtual Directory Name: $($VirtualDirectory.Name)”Write-Host “Internal Authentication Methods: $($VirtualDirectory.InternalAuthenticationMethods)”Write-Host “External Authentication Methods: $($VirtualDirectory.ExternalAuthenticationMethods)”
# Check if OAuth is present in internal authentication methods if ($VirtualDirectory.InternalAuthenticationMethods -like “*OAuth*”) { $highlightedAuthMethods = $VirtualDirectory.InternalAuthenticationMethods -replace “OAuth”, “`e[92mOAuth`e[0m” Write-Host “OAuth authentication is enabled.” -BackgroundColor Green } } |
Does it show OAuth right side?
Or yourself:
Get-AutodiscoverVirtualDirectory | fl *auth*
Get-WebServicesVirtualDirectory | fl *auth*
Failure:
CHECK2: OAuth Set, but does not PULL > FLIP-FLOP effect IIS VirtualDirectory Settings for EWS and AUTODISCOVER on Exchange on-premise
IF FAILURE: FLIP-FLOP the IIS Virtualdirectory Authentication Settings to be fireproof.
Even if ALL seems fine sometimes their is a FLIP FLOP effect which evn MS can’t explain. So turn it OFF and TURN it ON.
| Commands to move out the the FLIP/FLOP effectONLY READS VALUES | Does not change anything |
| Get-AutodiscoverVirtualDirectory -Server MYSERVERNAME |Set-AutodiscoverVirtualDirectory -OAuthAuthentication $falseGet-AutodiscoverVirtualDirectory -Server MYSERVERNAME |Set-AutodiscoverVirtualDirectory -OAuthAuthentication $trueGet-WebServicesVirtualDirectory -Server MYSERVERNAME |Set-WebServicesVirtualDirectory -OAuthAuthentication $falseGet-WebServicesVirtualDirectory -Server MYSERVERNAME |Set-WebServicesVirtualDirectory -OAuthAuthentication $true |
| Cmd.exe elevatedIISresetand if you like you can Recycle in IIS the APP Pools with:Restart-WebAppPool MSExchangeAutodiscoverAppPool Restart-WebAppPool MSExchangeServicesAppPool |
| IF you use DAUTH the OLD authentication then the parameter would be:-WSSecurityAuthentication:$FalseQuestion: Can i see OAuth in IIS GUI?In the IIS Manager GUI, you cannot directly see the specific OAuth configuration settings for Exchange virtual directories. The GUI does not provide a dedicated section or flag to indicate whether OAuth is enabled for a virtual directory.OAuth configuration settings are managed at a higher level in Exchange Server, typically through PowerShell cmdlets or other management tools specific to Exchange. These settings control the OAuth functionality and its integration with Microsoft 365.While you can view general authentication settings in the IIS Manager GUI, it does not provide granular visibility into OAuth settings. To get detailed information about OAuth configuration in Exchange Server 2013, it is recommended to use PowerShell cmdlets or consult the Exchange management documentation for accurate and up-to-date information. |
Failure:
CHECK3: You have renewed the OAuth Certificate on the Exchange on-premise but
FAILURE: You did not do the last step in the KB on how to renew OAuth cert, Event 4001 Exchange on-premise
If you have recently renewed the OAuth certificate on your Exchange on-premises server but encounter authentication failures or other issues afterward, it is crucial to ensure that you have completed the final step mentioned in the corresponding knowledge base (KB) article.
Renewing the OAuth certificate involves multiple steps, and failure to follow the complete process can result in incomplete certificate installation or incorrect configuration. By carefully reviewing the KB article and verifying that all required steps, including post-renewal configuration, have been executed, you can avoid common pitfalls and ensure a successful OAuth certificate renewal.
| Find out / ICO if you are affectedEvent 4001 on Exchange on-premiseThe request failed with HTTP status 401: Unauthorized in EventMissing signing certificate if you do the Test-OAuthConnectivity |
| Microsoft.Exchange.InfoWorker.Common.Delayed`1[System.String]<alani.berseddi@schweiz.mail.onmicrosoft.com>SMTP:alani.berseddi@schweiz.mail.onmicrosoft.comMailtipsMicrosoft.Exchange.InfoWorker.Common.Availability.AutoDiscoverFailedException:Autodiscover failed for email address <alani.berseddi@schweiz.mail.onmicrosoft.com>SMTP:alani.berseddi@schweiz.mail.onmicrosoft.com with error System.Net.WebException:The request failed with HTTP status 401: Unauthorized. at
System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.SoapWebClient.AutoDiscover.DefaultBinding_Autodiscover.EndGetUserSettings(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.<>c__DisplayClass4.<EndInvoke>b__3() at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.ExecuteAndHandleException(ExecuteAndHandleExceptionDelegate operation). —> System.Net.WebException: The request failed with HTTP status 401: Unauthorized. at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.EndInvoke(IAsyncResult asyncResult) at Microsoft.Exchange.SoapWebClient.AutoDiscover.DefaultBinding_Autodiscover.EndGetUserSettings(IAsyncResult asyncResult) at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.<>c__DisplayClass4.<EndInvoke>b__3() at Microsoft.Exchange.InfoWorker.Common.Availability.SoapAutoDiscoverRequest.ExecuteAndHandleException(ExecuteAndHandleExceptionDelegate operation) — End of inner exception stack trace — . Name of the server where exception originated: exchange2013R17 |
Cloud to On-premise OAuth certificate missing: 1) Check certificate in AuthConfig in On-Premises Exchange Management Shell: (Get-AuthConfig).CurrentCertificateThumbprint Test-OAuthConnectivity -Service AutoD -TargetUri <OnPremises Autodiscover.svc it will give you this exception: Microsoft.Exchange.Security.OAuth.OAuthTokenRequestFailedException: Missing signing certificate. |
| Reason OAuth is not working anymore from on-premise > M365 side because OAuth Cert renew done on on-premise |
DONE: You have renewed the OAuth Certificate on the Exchange on-premise but the CLOUD side still has the OLD version (You did not do the last step in the KB on how to renew OAuth cert) Thumbprint Ending: ***62 (Get-AuthConfig).CurrentCertificateThumbprint | Get-ExchangeCertificate | Format-List All seems fine, you also waited the 2H+ because of the UTC/CET bug and /owa did not work for that time.
You done all steps but did not do the purple box? Azure needs to KNOW the new cert now that’s the part missing. Not the Exchange Online M365 the Azure Cloud.
|
| When you deploy a new OAuth certificate in a hybrid Exchange environment, it is essential to rerun the Hybrid Configuration Wizard (HCW) to ensure that the cloud side of the environment is aware of the new certificate. The HCW is a tool provided by Microsoft to simplify the configuration of a hybrid deployment between on-premises Exchange and Exchange Online.By rerunning the HCW, you allow the wizard to update the necessary settings and configurations, including the OAuth certificate, on both the on-premises Exchange servers and the Exchange Online environment. This ensures seamless communication and authentication between the two environments, enabling features such as cross-premises free/busy lookup and mailbox moves.It is important to note that when deploying a new OAuth certificate, running the HCW is a crucial step to ensure proper functionality and synchronization between the on-premises and cloud components of the hybrid Exchange environment. |
Solution 1:RE-RUN The HCW (Hybrid Configuration Wizard) wizard on Exchange with existing settings as . |
Solution 2: (Very technical)This is the step you did read but it’s only for 2007/2010 and mix with newer Exchange and M365 so you did not read. So normal you ONLY need to do this IF you have 2007/2010 in MIXED mode with 2013/2016 and M365 or if you have DONE the OAuth Cert renew on a 2013/2016/2019 on-premise.Follow the manual steps as you when your create an Organisation Trust between two companys Exchange related.https://learn.microsoft.com/en-us/exchange/configure-OAuth-authentication-between-exchange-and-exchange-online-organizations-exchange-2013-help?redirectedfrom=MSDNFor the OAuth Cert renew Steps 3 and 4 are relevant. This will upload the inhouse OAuth Certificate to Azure. As per 21.06.2023 those are:
Step 3: Export the on-premises authorization certificate Step 4: Upload the on-premises authorization certificate to Azure Active Directory Access Control Service (ACS) |
Failure:
Check IF Azure HAS your RE-done(latest) OAuth Exchange Certificate active
Azure side:
Connect to Azure from the machine where have acccess to Azure or the Exchange with the HCW Wizard.
Connect-MsolService (Logon with your Tenant or Admin account)
Get-MsolServicePrincipalCredential -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 -ReturnKeyValues $true
Note the StartDate and Enddate from the latest you made.
On-premise:
Get-ExchangeCertificate (Get-AuthConfig).CurrentCertificateThumbprint |fl Not*
Or CHECK below Script to get the time in UTC direct on-premise
Attention Local Time Zone and Azure Time Zone are different. Remember when you did renew the OAuth cert
It took 2 or 4 hours to get active (OWA and PS was not wokting if so…)
| On-premise Local Time Swiss | M365 side |
| NotAfter : 12.06.2028 13:39:54NotBefore : 12.06.2023 13:39:54 | StartDate : 12.06.2023 11:39:54EndDate : 12.06.2028 11:39:54 |
| On-premise Convertes to UTC USA Azure | |
| NotAfter : 12.06.2028 11:39:54NotBefore : 12.06.2023 11:39:54 |
Now hopefully, when doing the renew of the OAuth cert onpremise, you did not start 10 times.
AZURE = UTC
YOU = Local Time Zone +/- 2-4hrs or whatever
Use below script to get the time as it looks with azure
| cert_local_time.ps1 |
| Get the TIMES from local OPAUTH Cert in all time zones relevantONLY READS VALUE | Does not change anything |
| # V1.0, 18.06.2023, www.butsch.ch# Get the TIMES from local OPAUTH Cert in all time zones relevant#$certThumbprint = (Get-AuthConfig).CurrentCertificateThumbprint$cert = Get-ExchangeCertificate -Thumbprint $certThumbprint$localTimeZone = [System.TimeZoneInfo]::Local$utcTimeZone = [System.TimeZoneInfo]::Utc
$cert | Select-Object -Property NotBefore, NotAfter, ` @{Name=”LocalNotBefore”; Expression={$_.NotBefore.ToLocalTime().ToString()}}, ` @{Name=”LocalNotAfter”; Expression={$_.NotAfter.ToLocalTime().ToString()}}, ` @{Name=”UTCNotBefore”; Expression={$_.NotBefore.ToUniversalTime().ToString()}}, ` @{Name=”UTCNotAfter”; Expression={$_.NotAfter.ToUniversalTime().ToString()}}, ` @{Name=”LocalTimeZone”; Expression={$localTimeZone.DisplayName}}, ` @{Name=”UTCTimeZone”; Expression={$utcTimeZone.DisplayName}} |
Re-check all as explained in:
Failure:
FAILURE, 401 Access Denied: You don’t not have all internal URL from your Exchange listed with AZURE as SPN
(He will accept only those address which the Cloud has SPN listed to easy explain)
On-premise: Check the OnPremisesDiscoveryEndPoint and OnPremisesWebServiceEndPoint from:
Get-IntraOrganizationConfiguration | Fl
On-premise: Should be the same as per:
Get-WebServicesVirtualDirectory | FL ExternalUrl
Check Azure side:
Connect to Azure from the machine where have acccess to Azure or the Exchange with the HCW Wizard.
Connect-MsolService (Logon with your Tenant or Admin account)
(Get-MsolServicePrincipal -ServicePrincipalName “00000002-0000-0ff1-ce00-000000000000”).ServicePrincipalNames
Check that ALL URL which you have on-premise are listed also there.
https://autodiscover.schweiz.ch
THUS
00000002-0000-0ff1-ce00-000000000000/outlook.schweiz.ch
00000002-0000-0ff1-ce00-000000000000/autodiscover.schweiz.ch
And any further DOMAINS you may have on your on-premise Exchange Domains .EU .DE .COM don’t forget those.
External Links mentioned:
RFC6749:
https://datatracker.ietf.org/doc/html/rfc6749
Microsoft Azure Active Directory OAuth 2.0 Fundamentals:
https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/auth-OAuth2
Deprecation of Basic Authentication in Exchange Online:
Test-OAuthConnectivity PowerShell cmdlet:
https://learn.microsoft.com/en-us/powershell/module/exchange/test-OAuthconnectivity?view=exchange-ps
Troubleshooting OAuth Expiration in Exchange:
Configuring OAuth Authentication in Exchange 2013: