Exchange 2016 CU20 Schema Update setup.exe /preparead fail because of case sensitivity of OWA APP Policy

ISO/PATCH: ExchangeServer2016-x64-cu20

Cumulative Update 20 for Exchange Server 2016 (microsoft.com)

 

Problem:

Exchange 2016 CU20 Setup.exe /preparead (Version 15.1.2242.4 Fails) on Server 2016 (1607)

Step Configuring Microsoft Exchange Server Organization Preparation results FAILED

Exchange 2016 CU 20 need and fails to update Active Directory Schema to newer Version (setup.exe /prepareschema works setup.exe /Preparead fails) if you have renames Outlook Web App Policy Default/default/DEFAULT.

We had a case in a Mother / Child Domain setup where we had to update Active Directory of the Mother domain of the company with commandline to a new Schema Version. This was related to the second Exchange 2016 Breach/Hotfix and we wanted to uplift Exchange 2016 from CUMU 19 to 20 urgently.

Prepareschema worked but the second command preparead failed.

 

 

Schema Versions

 

 

ERROR you see during the setup.exe /preparead

 

Error from Powershell

The following error was generated when "$error.Clear();

$policyDefault =

Get-OwaMailboxPolicy -DomainController $RoleDomainController | where

{$_.Identity -eq "Default"};

 

if($policyDefault -eq $null)

{

 

New-OwaMailboxPolicy -Name "Default" -DomainController $RoleDomainController

 

}

" was run:

"Microsoft.Exchange.Data.Directory.ADObjectAlreadyExistsException: Active

Directory operation failed on NOVCHVOLDCW1.novartis.com. The object

'CN=Default,CN=OWA Mailbox Policies,CN=migros,CN=Microsoft

Exchange,CN=Services,CN=Configuration,DC=migros,DC=net' already exists. --->

System.DirectoryServices.Protocols.DirectoryOperationException: The object

exists.

at

System.DirectoryServices.Protocols.LdapConnection.ConstructResponse(Int32

messageId, LdapOperation operation, ResultAll resultType, TimeSpan

 

 

Source of problem:

 

 

You can see the OWA APP Policy you have with following:

Get-owamailboxpolicy –Domaincontroller Butschdcw1 | Fl identity

Notice the case Sensitivity of the IDENTITY "Default/default/DEFAULT"

 

Error full:

Workaround:

Change the identity name of Outlook Web app Policy back to Default

  1. Go into Exchange 2016 GUI (Exchange Administrative Center)
  2. Permission / Berechtigung
  3. Outlook Web App-Policy/Outlook Web App-Richtlinien
  4. Mark the "Default/default/DEFAULT" and click the PENCIL/EDIT
  5. Change the name to Default (D large rest small chars)
  6. On DOS replicate the DC's with repadmin.exe /syncall

After that you can run setup.exe /preparead and update the Schema for Exchange 2016 CU

 

 

 

 

Check the Schema after replication with repadmin.exe /syncall

CHECK OBJECTVERSION:

$RootDSE= ([ADSI]"").distinguishedName

([ADSI]"LDAP://cn=swiss,cn=Microsoft Exchange,cn=Services,cn=Configuration,$RootDSE").objectVersion

CHECK RANGEUPPER:

$RootDSE= ([ADSI]"").distinguishedName

([ADSI]"LDAP://CN=ms-Exch-Schema-Version-Pt,CN=Schema,CN=Configuration,$RootDSE").rangeUpper

 

16220 > OBJECTVERSION

15333 > RANGEUPPER

 

 

Some further reading why this could have happened

https://devblogs.microsoft.com/scripting/weekend-scripter-unexpected-case-sensitivity-in-powershell/

https://superuser.com/questions/720037/powershell-if-statement-case-insensitive

SYNTAX ERROR: Fun with Powershell commands copied from Blogs or KB-solutions

SYNTAX ERROR: Fun with Powershell commands copied from Blogs or KB-solutions

 

We often see that the "-symbol or the minus-symbol are malformed and it looks normal in notepad.exe or the PS-console. The "-symbol effect can be devastation because you may have other objects you handle with identical short names in complex commands.

Worst case: "room 140 left wing Barcelona" and he targets room then etc.

Several times we have seen such effects with Powershell if we copy PS commands direct and don't use the copy-code function that good blog platforms or even KB-platforms on Intranet should support.

 

Here is a sample:

You just see red. First you think the command is not there anymore in this version whatever. Then you think maybe the wrong shell 32/64? Or elevated. The you type it manual and it works ;-)

If you copy the two commands to NOTEPAD.EXE as it opens all looks fine.

You can see in WinWord already that there MUST be a difference (They are not exact the same horizontal length) ;-)

TO really see: If you open the two commands in Notepadd+ or any advanced editor.

U+2013

0x96

â€"

%E2 %80 %93

U+2014

0x97

â€"

%E2 %80 %94

 

Euro? Germany? So someone from European Union made the Blog (Not UK not Swiss/Switzerland/Suisse) no? ;-) A double minus or triple minus?

;-)

 

 

 

 

 

SRV 2016, Windows Patch, BSOD, STOP CODE, 0xc000021a or CRITICAL SERVICE FAILED, ROLLBACK, Blue screen how to

Hello,

This is a collection of some technical things we used to recover a SRV 2016 with blue screens. We assume the first crash was related to a too early SRV 2016 VL Release ISO and ESX 6.5 (From 2018) and a combination of a Windows Defender Update.

This may help you to recover a server 2016 in general. It's so rare with 2012R2/2016 that this may help because it's not daily and people tend to go back A Snapshot or restore from Veeam these days. Still we have seen this happen.

 

SRV 2016, Windows Patch, BSOD, STOP CODE, 0xc000021a or CRITICAL SERVICE FAILED, ROLLBACK, Blue screen how to

Windows Patch, BSOD, STOP CODE, 0xc000021a, ROLLBACK, Blue screen how to (Notice the PAGE FILE Partition where memory DUMP was done)

BSOD, Bluescreen on Server 2016

STOP CODE, 0xc000021a

STOP CODE, CRTITICAL SERVICE FAILED

 

Server 2016 problems patches 02/2021: KB4601318 fails to update, fails at 24% Windows Server 2016 - Microsoft Q&A

Customer did run following VMware setup for the cluster (Because they wanted to test the newer release for some days)

VMware ESX Versions:

6.5 Version 9298722

ESXi 6.5 U2C

ESXi650-201808001

8/14/2018

9298722

NA

6.5 Version 13635690

ESXi 6.5 EP 14

ESXi650-201905001

05/14/2019

13635690

N/A

 

Build numbers and versions of VMware ESXi/ESX (2143832)

 

Rollback of Updates that caused the Bluescreen if you installed Windows Update before.

 

  • Choose blue recovery console
  • Choose troubleshoot
  • Choose cmd.exe
  • Change KEYBOARD layout so you type the Local Admin password correct
  • Logon with Local Admin password

Rollback the last updates with: dism /image:D:\ /cleanup-image /revertpendingactions (D: is the drive where your Windows Server install is thus where c:\porgram files and c:\windows are (Search that Partition from C: to Z:))

 

 

Enter password (Hopefully)

 

Change KEYBOARD layout so you type the Local Admin password correct

Logon with Local Admin password

Search the windows Partition

 

Check with:

Sfc /scannow

 

 

Run: checkdisk if you think there is damage to file system or disk:

Chkdsk G: /f /r /x

 

Check the pending operations he should do or has done during the crash:

 

Remove C:\windows\winsxs\pending.xml.

Cd g:\windows\winsxs\

Rename pending.xml pending.old

 

Rollback the last updates with:

dism /image:D:\ /cleanup-image /revertpendingactions

D: is the drive where your Windows Server install is thus where c:\porgram files and c:\windows are (Search that Partition from C: to Z:)

 

 

Error: 0x800f082f

BAD: (Looks more worse now….)

 

GOOD: (Looks good until you try to reinstall the same patch in 1 week again…)

 

Weputil.exe reboot

OR

Type: EXIT

OR NOT

If you have "CRITICAL SERVICE FAILED" this could be related to a UNSIGNED DRIVER or something is wrong with Certificates (CODE SIGNING). Maybe ask

Firewall TEAM if they block CERTIFICATE REVOCATION and if they don't know what you talk about check yourself. Read more on blog.

Try these BOOT Option with F8

If the Server comes UP with "Disbale Driver Signature Enforcement" and you don't need that (Because it's not a high secure server you could disable it permanent). We do not recommend this on an Exchange Server as example or anything security related.

If it comes up run:

Sfc /scannow

 

Manually disable certificate signature validation:

open CMD.exe as an Administrator

bcdedit -set loadoptions DDISABLE_INTEGRITY_CHECKS

 

Re-enable the certificate validation

open CMD.exe as an Administrator

bcdedit -set loadoptions DENABLE_INTEGRITY_CHECKS

 

    

FAQ: How to remove Remove failed packages in Windows PE

 

Looking why the Server crashed with NIRSOFT tool Bluescreenview

 

 

Microsoft recommends the PAGE FILE Partition to be RAM + SOME XXXMB. This is what happens IF the Server ever has a blue screen so don't do it that way.

I am unsure if the Server would have picked the Partition if no more space was there (Like they recommend). Never like that Rekommandation.

Die Größe der Auslagerungsdatei sollte die Größe des physikalischen RAMs im System überschreiten (SharePoint Server) - SharePoint Server | Microsoft Docs

Ursache: Eine bewährte Methode für Windows ist es, die Größe der Auslagerungsdatei auf einen Wert größer als oder gleich der Gesamtmenge des verfügbaren physischen Speichers festzulegen. Für die automatische Wiederherstellung von Heapspeicher funktioniert die Speicherbereinigung in der Regel effektiver, wenn die Größe des verwalteten Heapspeichers sich der Auslagerungsgröße nähert. Unterschreitet die Größe der Auslagerungsdatei die RAM-Größe, werden neue Zuordnungen von verwaltetem Speicher erteilt, wodurch die Speicherbereinigung aufwändiger wird und die CPU-Beanspruchung steigt.

 

 

 

McAfee free tool GETSUSP.EXE (Cloud scanner for URL and files)

 

Hallo,

 

Es gibt einen neuen Release eines Tools mit welchen man Clients scannen kann und alles was es nicht kennt (spanisch vorkommt) vollautomatisch zu Mcafee GTI sendet. Man kann damit unbekannte Files an McAfee einsenden zur Analyse.

Falls man eine E-Mail Adresse angibt bekommt man am Schluss den Report nach der Analyse. Die Files welche integriert sind kennt Mcafee GTI-CLOUD und alle Produkte «handeln» diese dann als sicherer und effektiver.

 

Das Tool macht 20% der 100% Feature vom grossen ENS und zeigt dann auch wie schnell ENS wäre wenn man nur Muster suchen würde.

 

Einziger Nachteil es sollte jeweils aktuell sein. Also wenn man es braucht dann bitte neu downloaden. Dafür ist es ein Single EXE und man kann damit URL, Office/PDF oder CUSTOM Directory scannen.

 

https://www.mcafee.com/enterprise/en-us/downloads/free-tools/terms-of-use.html?url=https://downloadcenter.mcafee.com/products/mcafee-avert/getsusp/getsusp64.exe

 

  • Was Mcafee GTI nicht kennt frägt er am Schluss und macht automatisch (Ohne Mcafee NAI Vertrag) ein Upload zu Mcafee.
  • Falls man es im Enterprise Bereich braucht bitte unter Preference den Proxy eintragen.

 

Falls die Files in Ordnung sind kennen dann Mcafee und alle Security Alliance Partner das File (Trend, Symantec). Ebenso die Firmen welche von den drei grossen Echtzeit Patterns einkaufen und tauschen. Mcafee VSE ENS kennt dann die Files

und stuft diese sicherer ein.

 

 

How to Use GetSusp | McAfee Free Tools

https://www.mcafee.com/enterprise/en-in/downloads/free-tools/how-to-use-getsusp.html

 

 

 

 

Proxy und wenn Ihr Infos wollt WANN McAfee die Files analysiert hat….

 

03.03.2021 Exchange 2010, 2013, 2013, 2016, 2019 Patch KB5000871 how to Update correct with Links

HAFNIUM targeting Exchange Servers with 0-day exploits

Important Exchange Update 03.03.2021 for all Microsoft Exchange Versions

Affected Exchange Version: 2010, 2013, 2013, 2016, 2019

     

12.03.2021

We have seen so many installations fail on certain blogs and forums. Please make sure understand the update process of patching a special Server like Exchange or SQL-Server. You never patch such system core server application without prior reading a few things.

There is absolute no business justification for an outage in the E-Mail system because someone wanted to patch a leak within a few minutes. Microsoft does its best to put some logic in the patch files. However they are just windows MSP files and the main logic for that is in the Windows installer Module which is on every server or client from MS.

   

10.03.2021

MS has to build/redo/re-tune their Exchange Repair TechNet/MSDN stuff due the high outage of Exchange on premise. Mostly related to non Exchange Groupware Engineer patching complex Exchange Servers in the emergency.

Repair failed installations of Exchange Cumulative and Security updates - Exchange | Microsoft Docs

https://docs.microsoft.com/en-us/exchange/troubleshoot/client-connectivity/exchange-security-update-issues

 

 

REMINDER: Install any Exchange ROLLUP/ROLLUP with elevated Administrator rights!

  1. Select Start, and then type cmd.
  2. Right-click Command Prompt from the search results, and then select Run as administrator.
  3. If the User Account Control window appears, select the option to open an elevated Command Prompt window, and then select Continue. If the UAC window doesn't appear, continue to the next step.
  4. Type the full path of the .msp file for the security update, and then press Enter.
  5. After the update installs, restart the server.

 

     

REMINDER: Make sure you have SCHEMA ADMIN rights for certain updates (Do not update with least privilege account like admin.butsch or 1stlevelsupportuser1 and if you do pre-heck it has all rights it needs

REMINDER: Most of the modern ROLLUP for Exchange does a) Saves the config to many XML files b) does a complete uninstall of the full EXCHANGE d) Re-installs the full exchange with the settings from XML File (Do no interrupt, Wait until it's done!) If you fail you will see Service deactivated and Exchange missing!

     

in larger Enterprise and PROXY environments

 Always make sure that you:

 *That the Exchange Server computer has access too all CRT Cert Revocation List address and also Network protocol related to CERTIFICATES (The computer when locked down MUST have access to WAN for these files)

* You DISABLE Cert revocation in Internet Explorer 10/11 or EDGE on the Server (The computer account uses those settings for PROXY or CERT Revocation settings.

* DO disable Temporary cmd.exe > iexplore.exe > Advanced "Check for Server Certi*" < uncheck both and restart Exchange Setup for the ROLLUP/CU

CHECK: https://www.butsch.ch/post/The-certificate-is-invalid-for-exchange-server-usage-Exchange-2010-SANUC.aspx 

CHECK: http://www.computerladen.ch/post/Exchange-2010-SP3-RU282930-ended-prematurely-(Management-Framework-30-on-Server).aspx

   

Check CERT revocation from Exchange to WAN: If it's not open HIT the Security Engineer in your company who closed internet access for Server and did not understand Cert Revocation. Never lock down Servers if you don't understand what they do fully.

     

     

 Microsoft explained with a graph now also since 25% did not understand at first glance. Including everyone I know.

 https://techcommunity.microsoft.com/t5/exchange-team-blog/march-2021-exchange-server-security-updates-for-older-cumulative/ba-p/2192020

 

      

Tech Links from 09.03.2011:

https://www.reddit.com/r/sysadmin/comments/m0d98h/exchange_nuked_and_reinstalled_what_can_i_and/

https://blueteamblog.com/microsoft-exchange-zero-days-mitigations-and-detections

https://www.volexity.com/blog/2021/03/02/active-exploitation-of-microsoft-exchange-zero-day-vulnerabilities/

      

News coverage done wrong:

Looks like the mainstream security magazines got something wrong here ;-) The patch was releases almost same time as the one from 2013/2016/2019. 2010 was supported and patched last week with a one shot patch already .

CVE we talk about:

CVE-2021-26854

Microsoft Exchange Server Remote Code Execution Vulnerability This CVE ID is unique from CVE-2021-26412, CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, CVE-2021-27065, CVE-2021-27078.

 CVE-2021-26855

Iis a server-side request forgery (SSRF) vulnerability in Exchange which allowed the attacker to send arbitrary HTTP requests and authenticate as the Exchange server.

CVE-2021-26857

Is an insecure deserialization vulnerability in the Unified Messaging service. Insecure deserialization is where untrusted user-controllable data is deserialized by a program. Exploiting this vulnerability gave HAFNIUM the ability to run code as SYSTEM on the Exchange server. This requires administrator permission or another vulnerability to exploit.

CVE-2021-26858

Is a post-authentication arbitrary file write vulnerability in Exchange. If HAFNIUM could authenticate with the Exchange server then they could use this vulnerability to write a file to any path on the server. They could authenticate by exploiting the CVE-2021-26855 SSRF vulnerability or by compromising a legitimate admin's credentials.

CVE-2021-27065

Is a post-authentication arbitrary file write vulnerability in Exchange. If HAFNIUM could authenticate with the Exchange server then they could use this vulnerability to write a file to any path on the server. They could authenticate by exploiting the CVE-2021-26855 SSRF vulnerability or by compromising a legitimate admin's credentials.

      

Here is what we talk about:

https://nvd.nist.gov/vuln/detail/CVE-2021-26854

https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/

https://www.swissinfo.ch/ger/alle-news-in-kuerze/microsoft-schliesst-sicherheitsluecken-bei-exchange-software/46415548

https://www.forbes.com/sites/daveywinder/2021/03/03/microsoft-issues-critical-update-warning-as-chinese-hackers-attack-exchange-servers/?sh=5c92a6f17912

     

Microsoft Made it a little bit complicated today. Here is maybe some help. They adapted their Blogs and documents during the day. First answer to the early links where asking where the patches are they mentioned.

So hopefully this information will help you to get your Exchange on premise safe soon.

 https://support.microsoft.com/en-us/topic/description-of-the-security-update-for-microsoft-exchange-server-2019-2016-and-2013-march-2-2021-kb5000871-9800a6bb-0a21-4ee7-b9da-fa85b3e1d23b

Description of the security update for Microsoft Exchange Server 2019, 2016, and 2013: March 2, 2021 (KB5000871)

     

To see what version you have use this Powershell commands:

Get-ExchangeServer | Format-List Name,Edition,AdminDisplayVersion

Invoke-Command -ScriptBlock {Get-Command Exsetup.exe | ForEach-Object {$_.FileversionInfo}}

Exchange Version overview:

Exchange Server – Buildnummern und Veröffentlichungstermine | Microsoft Docs

https://docs.microsoft.com/de-de/Exchange/new-features/build-numbers-and-release-dates?view=exchserver-2019

You then see following infos:

ProductVersion FileVersion FileName

14.03.0513.000 14.03.0513.000 D:\Program Files\Microsoft\Exchange Server\V14\bin\ExSetup.exe

This changed AFTER the patch KB5000871 which makes certain Exchange version safe. See the15.1.2106.13. That is the Exchange 2016 CU 18 we patched with the KB5000871.

Screenshot: Sample shows a customer having Exchange 2016 RU 18 (Not already on 19 [Not needed to be safe]) 15.01.2106.002 and we made him safe with the Patch .13. (Lower Line)

IMPORTANT and CONFUSING:

You have to be on a certain Patch level (Not the latest) before you can apply the Hot fix which makes you safe.

  • On 2016 and the 2019 the last two regular versions
  • On 2010 and 2013 the last version which was released

     

These versions you can patch with the Hotfix/Security Update for Exchange Server KB5000871

      

Exchange 2019 you minimum need 2019 CU7 (If you are not on CU7 first update to min CU7).

You don't have to update to CU8 to be safe! Just install the KB5000871 for your correct version.

     

Exchange Server 2019 CU8    December 15, 2020    15.2.792.3        15.02.0792.003

https://www.microsoft.com/en-us/download/details.aspx?id=102770 (KB5000871 Download link)

     

Exchange Server 2019 CU7    September 15, 2020    15.2.721.2        15.02.0721.002

https://www.microsoft.com/en-us/download/details.aspx?id=102771 (KB5000871 Download link)

     

If your are not CU7 or CU8 then install one of those first and then the KB5000871 afterwards to be safe.

Check if you need Enterprise Admin/Schema Admin to install CU7 and CU8 (Don't install with delegated Admin accounts!)

Reminder: Most of the modern ROLLUP for Exchange does a) Saves the config to many XML files b) does a complete uninstall of the full EXCHANGE d) Re-installs the full exchange with the settings from XML File (Do no interrupt, Wait until it's done!) If you fail you will see Service deactivated and Exchange missing!

     

     

Exchange 2016 you Minimum need 2016 CU18 (If you are not on CU18 first update to min CU18).

You don't have to update to CU19 to be safe! Just install the KB5000871 for your correct version.

     

     

Exchange Server 2016 CU19 15. Dezember 2020 15.1.2176.2     15.01.2176.002

https://www.microsoft.com/en-us/download/details.aspx?id=102772 (KB5000871 Download link)

     

Exchange Server 2016 CU18 15. September 2020 15.1.2106.2     15.01.2106.002

https://www.microsoft.com/en-us/download/details.aspx?id=102773 (KB5000871 Download link)

     

If your are not CU18 or CU19 then install one of those first and then the KB5000871 afterwards to be safe.

Both CU19 may make Active Directory Schema Updates so need be Domain Admin and Enterprise/Schema Admin! The patch can be installed with regular permission.

Reminder: Most of the modern ROLLUP for Exchange does a) Saves the config to many XML files b) does a complete uninstall of the full EXCHANGE d) Re-installs the full exchange with the settings from XML File (Do no interrupt, Wait until it's done!) If you fail you will see Service deactivated and Exchange missing!

     

     

     

Exchange 2013 There is only one hot fix for the Patch which came out a year ago.

     

Exchange Server 2013 CU23    June 18, 2019        15.0.1497.2        15.00.1497.002

https://www.microsoft.com/en-us/download/details.aspx?id=102775 (KB5000871 Download link)

     

If you are not on 23 install the CU23 first and then the KB5000871 afterwards to be safe.

Reminder: Most of the modern ROLLUP for Exchange does a) Saves the config to many XML files b) does a complete uninstall of the d) Re-installs the full exchange with the settings from XML File (Do no interrupt, Wait until it's done!)

     

      

Exchange 2010 There is new Rollup 32 free to all customers. Install the RU32 and you are good. That came out today and includes all you need.

 Exchange Server 2010 SP3 Rollup 32 (Release today 03.03.2021 you can install without ESU)

https://support.microsoft.com/en-us/topic/description-of-the-security-update-for-microsoft-exchange-server-2010-service-pack-3-march-2-2021-kb5000978-894f27bf-281e-44f8-b9ba-dad705534459

 https://www.microsoft.com/en-us/download/details.aspx?id=102774 (KB5000871 Download link)

     

     

Install the MSP patch and the CU/ROLLUP the right way so you see more info.

* Check Execution Policy Powrshell angepasst? get-executionpolicy –list

* Check UAC OFF

* Run it elevated or it will not work

* Stop Schedule things like Backup, Snapshot from Netapp, Veeam Tasks maybe Antivirus Solution

* Make a link to cmd.exe on your Desktop and run that Elevated

* From that cmd.exe navigate to the Patch and run as example Exchange2016-KB5000871-x64-de.msp /lvx C:\KB5000871_InstallationLogFile.log

     

     

Or with a batch. Same thing it's important that you install the Patch only elevated with run as Administrator.

:: ExchangeServer2016-x64-CU18_HOTFIX

:: V1.0, 03.03.2021, M. Butsch, First Release

:: -----------------------------------------------------------------------------------------------------------------------

cls

@echo off

echo ACHTUNG auf GPO's fuer Powershell oder UAC ACHTUNG

echo -----------------------------------------------------------------------

echo - Execution Policy Powrshell angepasst? get-executionpolicy -list

echo - revoke Cert IE angepasst?

echo - UAC abgestellt

echo -----------------------------------------------------------------------

pause

C:\edv\ExchangeServer2016-x64-CU18_HOTFIX\Exchange2016-KB5000871-x64-de.msp /lvx C:\edv\ExchangeServer2016-x64-CU18_HOTFIX\KB5000871_InstallationLogFile.log

pause