Exchange 2013/2016 and 2010 Proxy back (400) Bad Request, ADS-user in too many ADS-groups member

Exchange 2010/2013/2016 Migration, problem after DNS-pointing to 2016 structure with some users Outlook.exe

When you thought Kerberos Bloating is way back 2012 it returns. And after some research it is still all over the place. It does affect on premise Solutions as well as cloud solution like ADFS, AZURE etc.

 

Error:

This error (HTTP 400 Bad Request) means that Internet Explorer was able to connect to the web server, but the webpage could not be found because of a problem with the address.

Reason:

Active Directory User being in member of too many Active Directory groups. Kerberos Ticket Bloating.

Solution:

You can modify a Parameter on the 2010 CAS to allow larger Kerberos Packets to be used for Authentication to Webservices. This may be also valid for other problems where you Authenticator to a Web server solution with Kerberos (Active Directory) as sample: Ticket Systems, Intranet Solutions, SharePoint, Security Appliance etc.

Pro Keyfacts:

You can test the effects by opening the Autodiscover URL in a web browser. Don't handle too much outlook.exe not opening (Because Autodiscover does not work at that moment you want be able)

MS says on the CAS 2010 only. HPE Services once had a KB which said DC and CAS. (Maybe older DC's that time)

Problem:

We have read/heard with number like user ADS-User-Object in 200+ ADS-groups. So at that point we dropped further research and did think this does not affect the customer because he had max 130 Groups a user was in.

But one employee was affected where one user was in only 83 groups and the second user was in 127 groups. There seem to be other Kerberos info which adds to that and hits some limits when the Kerberos packet is proxy back from Exchange 2016 to 2010 and then to the Domain Controller.

You can count the memberships with a 3-liner in PS:

$user=get-aduser m.butsch

$token=(get-aduser $user -Properties tokengroups).tokengroups

$token.count

 

At the Point where most is setup and you move the Autodiscover SCP DNS to the Exchange 2016 some people are:

The key fact is that you can Test and DEBUG this with by just opening the URL in a web browser. So you don't have to handle around with outlook.exe /rpcdiag. Because you can't open Outlook.exe you are also unable to test Autodiscover with Right on Outlook symbol.

This may be a Pitfall if you had Kerberos Authentication in place and because of that reason FOCUS too much in that direction. If you want to take over Kerberos Authentication from 2010/2016 you may have to build back and then on 2016 build it up again.

 

This is how the effect shows up on a Client with Outlook 2016.

"Die systemresourcen sind sehr niedrig. Schliessen Sie einige Fenster"

 

This is how the Outlook Profiles Look after some debug sessions if the effect is there:

 

To test if the effect is there:

As user > If you open the Autodiscover URL in different Browser you get Error 400

 

Google Chrome:

Internet Explorer 11 (Because mostly people set PROXY and EXCEPTIONS there and then other Browser import it from that) > which has to be working for Exchange 2013/2016 with Web Proxy active. (you have to exclude all Exchange 2010/2013/2016/2019 FQDN from proxy)

 

On the Exchange 2016 Server Logfiles you see following error:

"C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Autodiscover\*.log"

Check the SIZE of the logfiles growing as soon as you hand over the DNS for Auto discover from 2010 to 2016 Servers.

 

Search for text:

 

The remote server returned an error: (400) Bad Request.

2021-06-22T10:28:44.962Z,f2c25044-6223-41d6-9737-da6f010f1ffe,15,1,2242,10,{49184D7A-04D7-47BD-977C-A0DE7BC9AA8B},Autodiscover,autodiscover.fda.ch,/Autodiscover/Autodiscover.xml,,Negotiate,true,fda\u1234,fda.ch,Smtp~Linsi.vonn@fda.ch,Microsoft Office/16.0 (Windows NT 10.0; Microsoft Outlook 16.0.5149; Pro),172.30.46.211,fdaEXC7,400,400,,POST,Proxy,fdacas2.fda.ch,14.03.0123.000,IntraForest,AnchorMailboxHeader-SMTP,,,,353,346,,,9,1,,0,1;,1,,0,1,,0,14,0,1,0,0,0,0,1,0,0,0,5,0,1,3,3,12,14,,,,BeginRequest=2021-06-22T10:28:44.947Z;CorrelationID=<empty>;

ProxyState-Run=None;AccountForestGuard_fda.ch=1;DownLevelTargetRandomHashing=0/3;ClientAccessServer=fdaCAS2.fda.ch;ResolveCasLatency=0;FEAuth=BEVersion-1937997947;ProxyToDownLevel=True;RoutingEntry=DatabaseGuid:81fdd93e-6b0a-49f0-ae6b-c41619e3ebad%40fda.ch%40fda.ch Server:fdaEXC4.fda.ch+1937997947@0;BeginGetRequestStream=2021-06-22T10:28:44.960Z;OnRequestStreamReady=2021-06-22T10:28:44.960Z;BeginGetResponse=2021-06-22T10:28:44.961Z;OnResponseReady=2021-06-22T10:28:44.961Z;EndGetResponse=2021-06-22T10:28:44.961Z;ProxyState-Complete=ProxyResponseData;SharedCacheGuard=0;EndRequest=2021-06-22T10:28:44.962Z;I32:ADS.C[fdaDCW2]=1;F:ADS.AL[fdaDCW2]=0.9151757;I32:ATE.C[fdaDCW2.fda.ch]=1;F:ATE.AL[fdaDCW2.fda.ch]=0,WebExceptionStatus=ProtocolError;ResponseStatusCode=400;WebException=System.Net.WebException:

The remote server returned an error: (400) Bad Request. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<>c__DisplayClass197_0.<OnResponseReady>b__0();,,|RoutingDB:81fdd93e-6b0a-49f0-ae6b-c41619e3ebad,,,CafeV1

 

 

On Domain Controllers you may see following just at that time you open the Autodiscover URL in the browser of the client:

You may see following Error from the Exchange 2010 CAS Server on one of your Domain Controller. Check Events under Security for "Event 4769, Audit Failure"

A Kerberos service ticket was requested.

 

Account Information:

    Account Name:        FDACAS2$@FDA.CH

    Account Domain:        FDA.CH

    Logon GUID:        {00000000-0000-0000-0000-000000000000}

 

Service Information:

    Service Name:        FDAcas2$@FDA.CH

    Service ID:        NULL SID

 

Network Information:

    Client Address:        ::ffff:172.30.46.134

    Client Port:        54554

 

Additional Information:

    Ticket Options:        0x40810000

    Ticket Encryption Type:    0xFFFFFFFF

    Failure Code:        0x12

    Transited Services:    -

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

<System>

<Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />

<EventID>4769</EventID>

<Version>0</Version>

<Level>0</Level>

<Task>14337</Task>

<Opcode>0</Opcode>

<Keywords>0x8010000000000000</Keywords>

<TimeCreated SystemTime="2021-06-22T09:43:13.941534700Z" />

<EventRecordID>1163245364</EventRecordID>

<Correlation />

<Execution ProcessID="484" ThreadID="1188" />

<Channel>Security</Channel>

<Computer>fdaDCW2.fda.ch</Computer>

<Security />

</System>

<EventData>

<Data Name="TargetUserName">fdaCAS2$@fda.CH</Data>

<Data Name="TargetDomainName">fda.CH</Data>

<Data Name="ServiceName">fdacas2$@fda.CH</Data>

<Data Name="ServiceSid">S-1-0-0</Data>

<Data Name="TicketOptions">0x40810000</Data>

<Data Name="TicketEncryptionType">0xffffffff</Data>

<Data Name="IpAddress">::ffff:172.30.46.134</Data>

<Data Name="IpPort">19080</Data>

<Data Name="Status">0x12</Data>

<Data Name="LogonGuid">{00000000-0000-0000-0000-000000000000}</Data>

<Data Name="TransmittedServices">-</Data>

</EventData>

</Event>

 

 

Solution:

Depending on the source where this was offered you may have to adapt that on the:

Exchange 2010 CAS (IIS)

But I found some articles from HPE-IT-Services where it says also on the DC. DO not change it on the DC if the change on the CAS works.

Remember if you have as example several CAS behind a load balancer that the effect is backwards from the Exchange 2016 to the 2010. There is only A little of that process which will go over the front of the Load Balancer (Like KEMP or F5). So you have to patch all CAS.

If am not aware if CAS Server you EXCLUDED from CAS-Serving Service are also affected by this or not.

 

https://docs.microsoft.com/en-US/exchange/troubleshoot/client-connectivity/400-bad-request

 

Microsoft says:

On every Exchange 2010 CAS, locate the following subkey:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters

Under this subkey, increase the MaxFieldLength and MaxRequestBytes entries by using the values in the following table.

 

Value name    Value type    Value data    Value base

MaxFieldLength    DWORD    65536    Decimal

MaxRequestBytes    DWORD    65536    Decimal

 

To check if it's working:

Open up https://autodiscover.yourdomain.ch/autodiscover/autodiscover.xml

A Credentials POPUP is fine if not also.

But you have to see the XML File and Error 600 then all is fine.

 

Find Autodiscover endpoints by using SCP lookup in Exchange | Microsoft Docs

Powershell to check the group membership of all ADS-user to be run on your DC.

Makes a Text Lofile: user_groups.txt

# V1.0, 22.06.2021, M. Butsch, www.butsch.ch

start-transcript -path user_groups.txt

 

$Users = Get-ADUser -Filter * -Properties Name, GivenName, SurName, SamAccountName, UserPrincipalName, MemberOf, Enabled -ResultSetSize $Null

Foreach($User in $users)

{

$MA=get-aduser $User

 

$token=(get-aduser $MA -Properties tokengroups).tokengroups

$MATOKEN=$token.count

Write-Host $MA.SamAccountName';'$MA.name';'$MATOKEN

}

stop-transcript

 

Exchange 2016 CU20 Schema Update setup.exe /preparead fail because of case sensitivity of OWA APP Policy

ISO/PATCH: ExchangeServer2016-x64-cu20

Cumulative Update 20 for Exchange Server 2016 (microsoft.com)

 Problem:

Exchange 2016 CU20 Setup.exe /preparead (Version 15.1.2242.4 Fails) on Server 2016 (1607)

Step Configuring Microsoft Exchange Server Organization Preparation results FAILED

Exchange 2016 CU 20 need and fails to update Active Directory Schema to newer Version (setup.exe /prepareschema works setup.exe /Preparead fails) if you have renames Outlook Web App Policy Default/default/DEFAULT.

We had a case in a Mother / Child Domain setup where we had to update Active Directory of the Mother domain of the company with commandline to a new Schema Version. This was related to the second Exchange 2016 Breach/Hotfix and we wanted to uplift Exchange 2016 from CUMU 19 to 20 urgently.

Prepareschema worked but the second command preparead failed.

 

 Schema Versions

 

 ERROR you see during the setup.exe /preparead

 Error from Powershell

The following error was generated when "$error.Clear();

$policyDefault = Get-OwaMailboxPolicy -DomainController $RoleDomainController | where

{$_.Identity -eq "Default"};

 if($policyDefault -eq $null)

{

New-OwaMailboxPolicy -Name "Default" -DomainController $RoleDomainController

}

" was run:

"Microsoft.Exchange.Data.Directory.ADObjectAlreadyExistsException: Active

Directory operation failed on NOVCHVOLDCW1.novartis.com. The object

'CN=Default,CN=OWA Mailbox Policies,CN=migros,CN=Microsoft

Exchange,CN=Services,CN=Configuration,DC=migros,DC=net' already exists. --->

System.DirectoryServices.Protocols.DirectoryOperationException: The object exists.

at System.DirectoryServices.Protocols.LdapConnection.ConstructResponse(Int32

messageId, LdapOperation operation, ResultAll resultType, TimeSpan

   

   

Source of problem:

   

   

You can see the OWA APP Policy you have with following:

Get-owamailboxpolicy –Domaincontroller Butschdcw1 | Fl identity

Notice the case Sensitivity of the IDENTITY "Default/default/DEFAULT"

   

Error full:

Workaround:

Change the identity name of Outlook Web app Policy back to Default

  1. Go into Exchange 2016 GUI (Exchange Administrative Center)
  2. Permission / Berechtigung
  3. Outlook Web App-Policy/Outlook Web App-Richtlinien
  4. Mark the "Default/default/DEFAULT" and click the PENCIL/EDIT
  5. Change the name to Default (D large rest small chars)
  6. On DOS replicate the DC's with repadmin.exe /syncall

After that you can run setup.exe /preparead and update the Schema for Exchange 2016 CU

   

   

   

   

Check the Schema after replication with repadmin.exe /syncall

CHECK OBJECTVERSION:

$RootDSE= ([ADSI]"").distinguishedName

([ADSI]"LDAP://cn=swiss,cn=Microsoft Exchange,cn=Services,cn=Configuration,$RootDSE").objectVersion

CHECK RANGEUPPER:

$RootDSE= ([ADSI]"").distinguishedName

([ADSI]"LDAP://CN=ms-Exch-Schema-Version-Pt,CN=Schema,CN=Configuration,$RootDSE").rangeUpper

   

16220 > OBJECTVERSION

15333 > RANGEUPPER

   

   

Some further reading why this could have happened

https://devblogs.microsoft.com/scripting/weekend-scripter-unexpected-case-sensitivity-in-powershell/

https://superuser.com/questions/720037/powershell-if-statement-case-insensitive

 

Final note on this issue:

We have seen several other such related issues with 2016/2019 Exchange. Something does not update or install simply because something is case sensitive or some argument is missing or there where it should not be. Mainly in long history customer which where over 15 years on Exchange in several version.

We know how to fix but always say "And then? Next Update or when it runs same? Does it run?" And sometimes Tier 3 from Microsoft does nothing else. They compare what's different with the customer to their reference and then change the Attribute with ADSIEDIT and close the case. That's it, no explanation.

Still the above mentioned gives me some bad feeling. The patch was released ASAP and it was the second patch. If the tested the patch to death someone else would have come again and said why do they keep the patch back so long? (For IT > It was because they had to discuss so long with NSA on how to turn things back).

If you read the story about the FBI who could change your Exchange settings by court you know what happened if you are not a naive IT-world geek.
Cloud Office 365 was not affected because their NSA backdoor works in another way (Read more on Google or search MSDN TechNet

 

 

 

 

SYNTAX ERROR: Fun with Powershell commands copied from Blogs or KB-solutions

SYNTAX ERROR: Fun with Powershell commands copied from Blogs or KB-solutions

 

We often see that the "-symbol or the minus-symbol are malformed and it looks normal in notepad.exe or the PS-console. The "-symbol effect can be devastation because you may have other objects you handle with identical short names in complex commands.

Worst case: "room 140 left wing Barcelona" and he targets room then etc.

Several times we have seen such effects with Powershell if we copy PS commands direct and don't use the copy-code function that good blog platforms or even KB-platforms on Intranet should support.

 

Here is a sample:

You just see red. First you think the command is not there anymore in this version whatever. Then you think maybe the wrong shell 32/64? Or elevated. The you type it manual and it works ;-)

If you copy the two commands to NOTEPAD.EXE as it opens all looks fine.

You can see in WinWord already that there MUST be a difference (They are not exact the same horizontal length) ;-)

TO really see: If you open the two commands in Notepadd+ or any advanced editor.

U+2013

0x96

â€"

%E2 %80 %93

U+2014

0x97

â€"

%E2 %80 %94

 

Euro? Germany? So someone from European Union made the Blog (Not UK not Swiss/Switzerland/Suisse) no? ;-) A double minus or triple minus?

;-)

 

 

 

 

 

SRV 2016, Windows Patch, BSOD, STOP CODE, 0xc000021a or CRITICAL SERVICE FAILED, ROLLBACK, Blue screen how to

Hello,

This is a collection of some technical things we used to recover a SRV 2016 with blue screens. We assume the first crash was related to a too early SRV 2016 VL Release ISO and ESX 6.5 (From 2018) and a combination of a Windows Defender Update.

This may help you to recover a server 2016 in general. It's so rare with 2012R2/2016 that this may help because it's not daily and people tend to go back A Snapshot or restore from Veeam these days. Still we have seen this happen.

 

SRV 2016, Windows Patch, BSOD, STOP CODE, 0xc000021a or CRITICAL SERVICE FAILED, ROLLBACK, Blue screen how to

Windows Patch, BSOD, STOP CODE, 0xc000021a, ROLLBACK, Blue screen how to (Notice the PAGE FILE Partition where memory DUMP was done)

BSOD, Bluescreen on Server 2016

STOP CODE, 0xc000021a

STOP CODE, CRTITICAL SERVICE FAILED

 

Server 2016 problems patches 02/2021: KB4601318 fails to update, fails at 24% Windows Server 2016 - Microsoft Q&A

Customer did run following VMware setup for the cluster (Because they wanted to test the newer release for some days)

VMware ESX Versions:

6.5 Version 9298722

ESXi 6.5 U2C

ESXi650-201808001

8/14/2018

9298722

NA

6.5 Version 13635690

ESXi 6.5 EP 14

ESXi650-201905001

05/14/2019

13635690

N/A

 

Build numbers and versions of VMware ESXi/ESX (2143832)

 

Rollback of Updates that caused the Bluescreen if you installed Windows Update before.

 

  • Choose blue recovery console
  • Choose troubleshoot
  • Choose cmd.exe
  • Change KEYBOARD layout so you type the Local Admin password correct
  • Logon with Local Admin password

Rollback the last updates with: dism /image:D:\ /cleanup-image /revertpendingactions (D: is the drive where your Windows Server install is thus where c:\porgram files and c:\windows are (Search that Partition from C: to Z:))

 

 

Enter password (Hopefully)

 

Change KEYBOARD layout so you type the Local Admin password correct

Logon with Local Admin password

Search the windows Partition

 

Check with:

Sfc /scannow

 

 

Run: checkdisk if you think there is damage to file system or disk:

Chkdsk G: /f /r /x

 

Check the pending operations he should do or has done during the crash:

 

Remove C:\windows\winsxs\pending.xml.

Cd g:\windows\winsxs\

Rename pending.xml pending.old

 

Rollback the last updates with:

dism /image:D:\ /cleanup-image /revertpendingactions

D: is the drive where your Windows Server install is thus where c:\porgram files and c:\windows are (Search that Partition from C: to Z:)

 

 

Error: 0x800f082f

BAD: (Looks more worse now….)

 

GOOD: (Looks good until you try to reinstall the same patch in 1 week again…)

 

Weputil.exe reboot

OR

Type: EXIT

OR NOT

If you have "CRITICAL SERVICE FAILED" this could be related to a UNSIGNED DRIVER or something is wrong with Certificates (CODE SIGNING). Maybe ask

Firewall TEAM if they block CERTIFICATE REVOCATION and if they don't know what you talk about check yourself. Read more on blog.

Try these BOOT Option with F8

If the Server comes UP with "Disbale Driver Signature Enforcement" and you don't need that (Because it's not a high secure server you could disable it permanent). We do not recommend this on an Exchange Server as example or anything security related.

If it comes up run:

Sfc /scannow

 

Manually disable certificate signature validation:

open CMD.exe as an Administrator

bcdedit -set loadoptions DDISABLE_INTEGRITY_CHECKS

 

Re-enable the certificate validation

open CMD.exe as an Administrator

bcdedit -set loadoptions DENABLE_INTEGRITY_CHECKS

 

    

FAQ: How to remove Remove failed packages in Windows PE

 

Looking why the Server crashed with NIRSOFT tool Bluescreenview

 

 

Microsoft recommends the PAGE FILE Partition to be RAM + SOME XXXMB. This is what happens IF the Server ever has a blue screen so don't do it that way.

I am unsure if the Server would have picked the Partition if no more space was there (Like they recommend). Never like that Rekommandation.

Die Größe der Auslagerungsdatei sollte die Größe des physikalischen RAMs im System überschreiten (SharePoint Server) - SharePoint Server | Microsoft Docs

Ursache: Eine bewährte Methode für Windows ist es, die Größe der Auslagerungsdatei auf einen Wert größer als oder gleich der Gesamtmenge des verfügbaren physischen Speichers festzulegen. Für die automatische Wiederherstellung von Heapspeicher funktioniert die Speicherbereinigung in der Regel effektiver, wenn die Größe des verwalteten Heapspeichers sich der Auslagerungsgröße nähert. Unterschreitet die Größe der Auslagerungsdatei die RAM-Größe, werden neue Zuordnungen von verwaltetem Speicher erteilt, wodurch die Speicherbereinigung aufwändiger wird und die CPU-Beanspruchung steigt.

 

 

 

McAfee free tool GETSUSP.EXE (Cloud scanner for URL and files)

 

Hallo,

 

Es gibt einen neuen Release eines Tools mit welchen man Clients scannen kann und alles was es nicht kennt (spanisch vorkommt) vollautomatisch zu Mcafee GTI sendet. Man kann damit unbekannte Files an McAfee einsenden zur Analyse.

Falls man eine E-Mail Adresse angibt bekommt man am Schluss den Report nach der Analyse. Die Files welche integriert sind kennt Mcafee GTI-CLOUD und alle Produkte «handeln» diese dann als sicherer und effektiver.

 

Das Tool macht 20% der 100% Feature vom grossen ENS und zeigt dann auch wie schnell ENS wäre wenn man nur Muster suchen würde.

 

Einziger Nachteil es sollte jeweils aktuell sein. Also wenn man es braucht dann bitte neu downloaden. Dafür ist es ein Single EXE und man kann damit URL, Office/PDF oder CUSTOM Directory scannen.

 

https://www.mcafee.com/enterprise/en-us/downloads/free-tools/terms-of-use.html?url=https://downloadcenter.mcafee.com/products/mcafee-avert/getsusp/getsusp64.exe

 

  • Was Mcafee GTI nicht kennt frägt er am Schluss und macht automatisch (Ohne Mcafee NAI Vertrag) ein Upload zu Mcafee.
  • Falls man es im Enterprise Bereich braucht bitte unter Preference den Proxy eintragen.

 

Falls die Files in Ordnung sind kennen dann Mcafee und alle Security Alliance Partner das File (Trend, Symantec). Ebenso die Firmen welche von den drei grossen Echtzeit Patterns einkaufen und tauschen. Mcafee VSE ENS kennt dann die Files

und stuft diese sicherer ein.

 

 

How to Use GetSusp | McAfee Free Tools

https://www.mcafee.com/enterprise/en-in/downloads/free-tools/how-to-use-getsusp.html

 

 

 

 

Proxy und wenn Ihr Infos wollt WANN McAfee die Files analysiert hat….