Microsoft Exchange Hafnium breach is turning into one of the ugliest security incidents ever, really fast. Here I will try to explain my steps in the process, and what my stages of investigation were (so far). If you already know about problem, I will be happy to share some new info and also learn something new from you.
Updated 10 March 2021 – with new info about scripts and link to website check if you were breached (at the bottom of the post).
Updated 11 March 2021 – Looks like CompareExchangeHashes.ps1 script works ok now.
Updated 11 March 2021 – I see a lot of skepticism howt to proceed further with this – here I can offer my observations/opinions – https://www.informaticar.net/what-to-do-if-you-were-breached-by-hafnium-exchange-breach/
Quick intro
On March 02/03 2021 Microsoft released urgent critical security updates for Microsoft Exchange 2013/16/19 that are known to exploit extremely dangerous vulnerabilities which (to put it plain and simple) can access your Microsoft Exchange with highest privileges and probably traverse your AD since Exchange and AD are tightly integrated. So this is potential disaster for your local network also.
Since every Exchange is internet oriented in part, you are exposed to this to some extent for sure.
Exploits in question are Zero Day vulnerabilities and are used in the wild at least since 06.January 2021 by unknown adversaries.
So, situation is more or less dramatic with this one, assuming it can overtake you email and internal domain.
There are four CVEs for this incident – CVE-2021-26855 CVE-2021-26857 CVE-2021-26858 and CVE-2021-27065.
These are all chainable exploits and according to Volexity, the firm who discovered attacks here is what they do
CVE 26855 is a server-side request forgery (SSRF) vulnerability in Exchange that is used to steal mailbox content.
CVE 26857 is used to run code under the System account.
CVE 26858 and 27065 are allowing an attacker to write a file to any part of the server.
It should be noted that DEVCORE Research Team found these exploits on December 10 2020 and reported them to Microsoft in January 5 2021.
This is in short and best to my knowledge as of now.
First step you need to do – patch
if you haven’t done already – patch your Exchange Server and your Windows Server systems RIGHT NOW!!
I’m up to date in my environments, so it took me 15 minutes (per server) to patch my servers. I did it early in the morning on March 3 2021 with notice to users that there is a possibility of service disruption due to emergency. Since I have DAG and HA in place, there really was no interruption in email service.
If you are reading this on 07/08 March 2021 or at later date, there is high potential that you are breached. You should isolate your Exchange servers and internal network as soon as possible and start with the steps below.
CVEs are – CVE-2021-26855 CVE-2021-26857 CVE-2021-26858 and CVE-2021-27065.
There are patches for MS Exchange 2013/2016/2019 and MS EXCHANGE 2010 although Ex 2010 is no more supported.
KB you are looking for is KB 500871, it should appear in your Windows Updates or you can look it up manually. This KB 500871 will only be available to install after you are on the latest patches. KB name for Exchange 2010 is named differently, I really haven’t bother to explore more about Exchange 2010 since it is not supported anymore.
If you are applying this KB manually you should run it as Administrator. There is a possibility that your Exchange services will stop or break, there are more details on the links I’ll post below.
One more important thing to notice is that you should be at the CU23 for Ex 2013, CU 18 for Ex 2016 and CU 8 for Ex 2019 for this KB 500871 to work.
As of 08 March 2021 you don’t need to be on latest CU to apply this vulnerability!!! You can save youserlf some time and apply security update immediately and then upgrade CUs later.
Here is Microsoft post so you can help yourself…
And here is the presentation done by Microsoft
I see a lot of confusion about this in community – you don’t have to apply CU one by one – CU1, CU2, CU3… CU23 – updates are cumulative so you need to download only last one – also, if you need this info, you should probably seek assistance with Exchange management.
There are also prerequisites that have to be done before deploying CU and you probably need to do forest/domain prep.
This is just a heads up, so you know what to look for before you start upgrading.
Here are links you need before you start patching:
https://msrc-blog.microsoft.com/2021/03/02/multiple-security-updates-released-for-exchange-server/
https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/
Be sure to check the posted links and make sure you revisit them for “fresh” information.
When you are done patching, your worries are not over and you are not secure!!
Especially if you are reading this on 07/08 March 2021 or later.
Even if you patched on 03.March 2021 and later you need to investigate this further – this is far from over.
I’ve patched, now what?
Let me reiterate – with patching, your job on this one is not done. You could be breached in period from 06.January 2021 to 03.March 2021 and even if you patched immediately, you are potentially breached.
Unknown adversary (or adversaries) ramped up their exploit effort since Microsoft made this Zero Day known on 03.March 2021.
So, assume you have been breached and start auditing your systems.
Microsoft calls this incident -> Hafnium.
Also, please be aware that this is not ultimate and final guide, a lot is still now known about Hafnium and we learn as we go. Community is pretty proactive and helpful on this one, and you should actively check various resources to stay up to date and inform yourself.
I will try to summarize as much as I can as I’m actively watching this from a beginning and see a lot of great info, but also many questions…
First step – scan your logs
Microsoft picked ball on this one really good and are trying to shine some light on how you can investigate and mitigate this issue
https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/
Long story short – Main scripts you should run are
TestProxyLogon.ps1, CompareExchangeHashes.ps1, BackendCookieMitigation.ps1 and http-vuln-cve2021-26855.nse
You can look further in the text for details on what I done, below is the source for scripts https://github.com/microsoft/CSS-Exchange/tree/main/Security
Test-ProxyLogon.ps1
Test-ProxyLogon.ps1 script is a good place to start and it can be found here
https://github.com/microsoft/CSS-Exchange/tree/main/Security
Download the script and run it on your Exchange Servers.Be sure to check the links above from time to time, since things are constantly changing with this incident and new things are being discovered about attack.
Test-ProxyLogon.ps1 should check your logs for potential problems and also report suspicious 7zip/zip files on your system.
If you don’t have any errors, congrats, so far so good (you should not however give up at this point, you should explore further)
I got “Suspicious activity found in Http Proxy log” and I also got Suspicious files found: 26
26 suspicious zip files were false alarm in my case.
But file that script saved in my \desktop\log\xxxx-Cve-2021-26855.csv had two entries in it:
DateTime RequestId ClientIpAddress UrlHost UrlStem RoutingHint UserAgent AnchorMailbox HttpStatus
2021-03-03T05:00:14.816Z 245cb23a-3c1d-491a-a871-f32b0b345v1 86.105.18.116 MY PUBLIC IP /ecp/y.js X-BEResource-Cookie ExchangeServicesClient/0.0.0.0 ServerInfo~a]@localmail.local:444/autodiscover/autodiscover.xml?# 200
2021-03-03T07:18:53.760Z 8fdavgf0-8b22-4767-wedc-3b45637d 86.105.18.116 MY PUBLIC IP /ecp/y.js X-BEResource-Cookie ExchangeServicesClient/0.0.0.0 ServerInfo~a]@localmail.local:444/autodiscover/autodiscover.xml?# 200
If there more of these entries in report and some references to files, you have definitely been hit.
So far, logs above are reference for further investigation and log exploration.
Here is an example how breached system logs look like – https://www.reddit.com/r/sysadmin/comments/lz1jp4/youve_been_hit_by_youve_been_struck_by_an/
User YellowOnline also found file discovery.aspx which is a dropped webshell.
Credit for this to fellow Reddit user YellowOnline – I hope he is doing good.
Here are some more great resources that will help you further investigate this incident, be sure to check them for fresh information about this
Microsoft Security Response Center
Multiple Security Updates Released for Exchange Server – updated March 8, 2021
Microsoft Exchange Team (be sure to go through comments)
Huntress
Fireeye
Crowdstrike
https://www.crowdstrike.com/blog/falcon-complete-stops-microsoft-exchange-server-zero-day-exploits/
Volexility
Praetorian – has very nice writeup on this exploit with great explanations.
https://www.praetorian.com/blog/reproducing-proxylogon-exploit/
Further Log Exploration
All great resources for new information and help with this incident.
Before we go further, I will further explore my logs. The script Test-ProxyLogon.ps1 is here just to point you in the right direction and give you clues…
First I will check logs in C:\Program Files\Microsoft\Exchange Server\V15\Logging\Autodiscover since above events point to Autodiscover.
This is one of the entries I found, both were the same in content, just different times (5:00 and 7:18)
2021-03-03T05:00:14.832Z,245cb23a-3c1d-491a-a871-f32b0b345v1,15,0,1497,10,,Negotiate,true,NT AUTHORITY\SYSTEM,,ExchangeServicesClient/0.0.0.0,86.105.18.116,LOCALEXCHANGE.LOCALDOMAIN,POX,200,500,0,0,1,,,,,GlobalThrottlingPolicy_245cb23a-3c1d-491a-a871-f32b0b345v1,,,1,3,0,3,3,,20,ADSessionSettingsFromAddress=0;ADRecipientSessionFindBySid=0;Caller=null;ResolveMethod=Unknown;RequestedRecipient=null;RequestedUser=administrator@MAILDOMAIN.com;S:ServiceCommonMetadata.RequestSize=344;S:WLM.Bal=300000;S:WLM.BT=Ews;S:BudgetMetadata.MaxConn=27;S:BudgetMetadata.MaxBurst=300000;S:BudgetMetadata.BeginBalance=300000;S:BudgetMetadata.Cutoff=3000000;S:BudgetMetadata.RechargeRate=900000;S:BudgetMetadata.IsServiceAct=False;S:BudgetMetadata.LiveTime=00:00:00;S:BudgetMetadata.EndBalance=300000;Dbl:WLM.TS=20;...,,message=The email address can't be found.;,Although this attack bypasses authentication, for some reason it searched for Administrator account, and in my case “The email address can’t be found.”
Next, inside /inetpub/logs/logfiles/w3svc1 log I found following GET /rpc and POST /ecp/y.js
2021-03-03 05:00:14 192.168.1.10 GET /rpc/ &CorrelationID=<empty>;&ClientId=BDOWHSJDHYTUNHDTJQ&RequestId=245cb23a-3c1d-491a-a871-f32b0b345v1&cafeReqId=db8dfe26-fea2-4cf3-99ae-1a979f531921; 443 - 86.105.18.116 python-requests/2.18.4 - 401
2021-03-03 05:00:14 192.168.1.10 POST /ecp/y.js &CorrelationID=<empty>;&ClientId=KDFLFJKIYRGDJFKPDTNG&cafeReqId=8fdavgf0-8b22-4767-wedc-3b45637d; 443 - 86.105.18.116 ExchangeServicesClient/0.0.0.0 - 200 0 0 109GET /rpc ended with status 401 while POST /ecp/y.js ended with status 200
Not good, but looks like system was scanned and probed, without any further impact.
On Microsoft Exchange Team Blog I linked above, Nino Bilic from Microsoft commented that if
"DATETIME","ServerInfo~a]@SERVERNAME.company.com:444/autodiscover/autodiscover.xml?#"is found after running Test-ProxyLogon.ps1 it should not mean that you are breached, just that you were definitely probed. You should definitely search for Indicators of Breach (IOC) to confirm that you were compromized.
Ok, you should definitely search more through your logs if you have further indicators after you ran Test-ProxyLogon.ps1 script.
Recommendation would be to run through Windows Logs, Exchange Logs and IIS logs.
These are some of the IP addresses used to scan, so you can check your logs for any of these…
185[.]125.231.175
86[.]105.18.116
185[.]224.83.137
107[.]173.83.123
201[.]162.109.184
68[.]2.82.62
182[.]215.181.200
45[.]15.9.45
182[.]18.152.105
141[.]164.40.193
172[.]105.87.139CompareExchangeHashes.ps1
Can be also found here as of 08 March 2021 https://github.com/microsoft/CSS-Exchange/tree/main/Security
Per Microsoft Github this is what this script does…
This script provides a mechanism for malicious file detection on Exchange servers running E13, E16 or E19 versions. For more information please go to https://aka.ms/exchangevulns
The script currently only validates files in exchange virtual directories only, it does not check any files in the IIS root (as of 11 March 2021 I think it does, because in some cases it shows c:\inetpub…). This script needs to be run as administrator
The script determines the version of exchange installed on the server and then downloads the hashes for known exchange files from the published known good hashes of exchange files
Script needs to be run as Administrator
.\CompareExchangeHashes.ps1In the beginning script was really buggy, but eventually it got fixed, you can still get few inconclusive results, but is far better than in the beginning. On MS Github site there is also a link on which you can submit your result for analysis.
My scan went great on all machines, and besides falls positives for other services, everyhing came out clean. I confirmed that files that are matched as positive are legit files.
My solution to check hashes (although a bit pedestrian, not fully automated)
Idea of checking file hashes is great, but unfortunately I still cannot verify the script CompareExchangeHashes.ps1 is working properly, so what I did is following.
I installed in my internal LAB 2x fresh Windows Server 2012 R2 machines. One is AD and another one is Exchange Server 2013 CU23.
No internet/no patches – nothing. Installations are from my ISO installation archives…
From what I can understand about this breach \FrontEnd\HttpProxy\* subdirectories are problematic and prone to web shell dropping. So, we would like to confirm that all the files in every subdirectory inside HttpProxy are legit, not modified.
So, first, I started my fresh LAB version of Exchange 2013 CU23 and opened Powershell and navigated to C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa. Once inside that directory I executed
(c:\temp\HttpProxy.csv is a file in which hashes will be logged – define your own log directory here)
ls | Get-FileHash -Algorithm SHA256 | Export-CSV
c:\temp\owamain.csv | Select-Object FullName | Format-Table -AutoSize
This was output
"SHA256","98FC3AA73175D6D55C4EE6AE8DAA0ACC721D792B14DF26B0CD39BBB13CB23D95","C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\global.asax"
"SHA256","1FC8766F6CC2CBCE93329FD0755DF747DA0AADFD3C07FEEEED9BD872BDF4C1AD","C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\web.config"
"SHA256","2818651C18C89392BD7CB1508F9F17AAA1B5FAEA719811D5B1E8F608F45D92A3","C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\web.config.bak"Ok, so again same procedure, but this time on the Ex 2013 CU23 which is production, potentially “compromised” server. And this is the result
In my case middle one of the three results is different. But also, I know that I modified that file, and I know when I modified it, so that is not a big deal.
You can use this method to check all the files that are bothering you. Only thing you need to do is – isntall another copy of Exchange 201x with the same CU you have.
My owa / ecp and other subdirectories came out clean.
AD Users and Groups Check
You should also check if your AD Users and Groups got created/modified recently…
You can run following commands on AD in Powershell…
Get-ADUser -Filter * -Property whenCreated | where {$_.whenCreated -gt $days} | ft Name, whenCreated
Get-ADUser -Filter * -Property whenChanged | where {$_.whenChanged -gt $days} | ft Name, whenChanged
Get-ADGroup -Filter * -Property whenCreated | where {$_.whenCreated -gt $days} | ft Name, whenCreated
Get-ADGroup -Filter * -Property whenChanged | where {$_.whenChanged -gt $days} | ft Name, whenChangedI don’t have anything suspicious also in those. Users that are created are real ones, there is no change on groups…
Scan if your Exchange Servers are vulnerable after patching
On the link below there is nmap script http-vuln-cve2021-26855.nse which should check if your system is vulnerable/patched. According to the internet people had hit and miss results with it (script since got updated).
https://github.com/microsoft/CSS-Exchange/tree/main/Security
You should install nmap, put the script above to scripts folder and run
nmap -Pn -p T:443 --script http-vuln-cve2021-26855 EXCHANGE IPHunting for files/malicious processes
Definite proof that you are compromised would be existence of .aspx, .js and some zip files in specific locations.
After you confirmed in logs that your systems were probed/accessed, we are now looking for web shells deployed on the system.
In short web shells are web-based shell like interfaces that enable remote access and code execution on a web server.
WEB Shell locations
This attack places various .aspx files in following directories and subdirectories within those directories
C:\inetpub\wwwroot\aspnet_client\
C:\inetpub\wwwroot\aspnet_client\system_web\
%PROGRAMFILES%\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\
...\FrontEnd\HttpProxy\owa\auth\
...\FrontEnd\HttpProxy\ecp\auth\
...\FrontEnd\HttpProxy\owa\auth\Current
Temporary ASP.NET files directory
Here is feed of IOCs observed by Microsoft (09.03.2021)
2020-03-09,2021-03-04T08:05:00.5878895Z,511df0e2df9bfa5521b588cc4bb5f8c5a321801b803394ebc493db1ef3c78fa1,sha256,White
2020-03-09,2021-01-06T18:38:17.8341434Z,b75f163ca9b9240bf4b37ad92bc7556b40a17e27c2b8ed5c8991385fe07d17d0,sha256,White
2020-03-09,2021-02-09T00:33:52.5232083Z,4edc7770464a14f54d17f36dc9d0fe854f68b346b27b35a6f5839adf1f13f8ea,sha256,White
2020-03-09,2021-02-23T09:14:05.8243534Z,811157f9c7003ba8d17b45eb3cf09bef2cecd2701cedb675274949296a6a183d,sha256,White
2020-03-09,2021-01-24T12:59:40.6969216Z,65149e036fff06026d80ac9ad4d156332822dc93142cf1a122b1841ec8de34b5,sha256,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\8Lw7tAhF9i1pJnRo.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\OutlookZH.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\authhead.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\bob.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\current\one1.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\errorPage.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\errorPages.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\fatal-erro.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\log.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\logg.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\logout.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\one.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\one1.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\shel.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\shel2.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\shel90.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\a.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\default.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\shell.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\Server.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\aspnet_client.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\aspnet_iisstart.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\aspnet_pages.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\aspnet_www.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\default1.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\errorcheck.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\iispage.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\s.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\session.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\shell.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\system_web\log.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\xclkmcfldfi948398430fdjkfdkj.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\xx.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\Server.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\discover.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\HttpProxy.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\OutlookEN.aspx,filepath,White
2021-03-09,,C:\inetpub\wwwroot\aspnet_client\supp0rt.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\OAB\log.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\log.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\logg.aspx,filepath,White
2021-03-09,,C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\owa\auth\logout.aspx,filepath,WhiteCheck if your inetpub is maybe redirected by checking “PathWWWRoot” value in the “HKLM\SOFTWARE\Microsoft\InetStp” registry key.
Here is a good example of webshell locations/names
https://gist.github.com/JohnHammond/0b4a45cad4f4ed3324939d72dc599883
I especially went through \ecp since there should be y.js file reported but I found nothing, even through entire system sweep.
Suspicious zip data can also be found in
C:\windows\temp\
C:\root\
Be careful to distinguish legitimate files from rogue ones. Rogue files are owned by SYSTEM user and you can also look at the date aspect when looking for files.
LSASS Process
You should also look out for LSASS process, after intruders successfully deployed web shells they will try to dump LSASS process memory using Procdump, after that is done intruders will try to exfiltrate data/emails from the system. More details can be found on the link below.
https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/
According to the link above these are all steps you can expect:
Using Procdump to dump the LSASS process memory
Using 7-Zip to compress stolen data into ZIP files for exfiltration
Adding and using Exchange PowerShell snap-ins to export mailbox data
Using the Nishang Invoke-PowerShellTcpOneLine reverse shell
Downloading PowerCat from GitHub, then using it to open a connection to a remote server
Downloaded the Exchange offline address book from compromised systems.
W3WP Process
According to FireEye you should also be on lookout for following
Child processes of C:\Windows\System32\inetsrv\w3wp.exe on Exchange Servers, particularly cmd.exe.Files written to the system by w3wp.exe or UMWorkerProcess.exe.
Scheduled Tasks
Check for rogue scheduled tasks on your systems. Winnet and VSPerMon are mentioned on various sources.
Try to scan deleted for files on your system to see if it was breached
This may not be in real order here, but if you suspect you were breached, you can also use appropriate tool to retrieve (hard) deleted files and see if you were breached and tracks removed. Be very careful with this, especially if you need to do some auditing/forensic tasks.
DNS – DKIM/DMARC
If you have those in place, check them out and confirm validity. Check your DNS in general.
Certficates
Check certificates inside IIS and Exchange…
In case I missed something in this writeup, be sure to check all the links I posted above and go through that info.
I cannot show you my examples since I haven’t found anything in my systems. All of my Exchange servers are clean. Besides those two log entries I haven’t found anything suspicious in logs or on file system.
My AD is also completely clean and without changes.
Scan your Exchange Servers with specially prepared Microsoft Tool
This morning (07.03.2021) Microsoft decided to make easier for everybody with this Hafnium exploit. Information about all of this is so overwhelming and something new is learned every hour.
So, Microsoft prepared and updated Microsoft Support Emergency Response Tool (MSERT) you can download and scan your system.
Scroll to the bootom of this link https://msrc-blog.microsoft.com/2021/03/05/microsoft-exchange-server-vulnerabilities-mitigations-march-2021/
This should be download link https://docs.microsoft.com/en-us/windows/security/threat-protection/intelligence/safety-scanner-download
Here is a description from Microsoft link above
“Microsoft Defender has included security intelligence updates to the latest version of the Microsoft Safety Scanner (MSERT.EXE) to detect and remediate the latest threats known to abuse the Exchange Server vulnerabilities disclosed on March 2, 2021. Administrators can use this tool for servers not protected by Microsoft Defender for Endpoint or where exclusions are configured for the recommended folders below.”
So, yes, this should help scan and mitigate threats disclosed on March 2 2021
Microsoft (and I) recommends FULL system scan on all Exchange machines. In my case full scan took about 1.5hour per machine.
I of course downloaded and run tools whole day today, through my Exchange Server environment. Results are in.
By looking at these results it looks like my servers were scanned but not compromised.
It looks like that security/antivirus vendors are starting to catch up with this attack, since I see reports from colleagues that anti-viruses start popping up warnings about these threats.
Check If you have been hacked
Please, before you go further and just click on link to check if you were breached – establish that you believe source and that you wish to proceed.
My credible source which reported this website is https://krebsonsecurity.com/2021/03/warning-the-world-of-a-ticking-time-bomb/
Now, that we got disclaimer out of the way – thanks to Unit 221B for their effort and time on this – this is the link on which you can do check – https://checkmyowa.unit221b.com/
If you visit that link from the public IP on which is your exchange server, you will get pop-up from the website if you have been breached. If you are clean – you will not get anything. Important thing is you visit from public IPs on which your Exchange is on (MX record IP/ OWA public IP if it is easier to understand that way. )
Other method is to scroll down the site and enter your email address (it should be on a domain you suspect is breached) – you will get email – I got my report in SPAM, but I got it.
First method, by doing it with IP address and visiting website is better, because mostly there are breached IPs on the list.
I done both and my results are clean.
Here is the email I got from the check (it ended up in my spam folder)
After I got that confirmed I went to the network that is used by my Exchange Servers and check every IP from the range that Exchange is using (MX, OWA…)
I checked with Firefox and Chrome…
All clean…
This is how your browser screen would look like if you were breached
According to Allison Nixon from Unit 221B there should be 86.000 IPs on that list, so if you were breached in first wave, there are good chances that you are on that list.
Thanks again to Unit 221B for this great tool.
What should I do if I’m compromised?
It is a tough question and this is my personal opinion. Full extent of this attack is still not known and situation is still developing.
If you are in situation like I’m – watch your systems closely, change your administrative passwords, reset users passwords, read and watch Microsoft posts, security blogs, news, be proactive and act according to your assessment of situation.
If you confirmed web shells and access to your systems – there is not much you can do. Depending on the industry you work in and where you live (GDPR) report incident as soon as possible, call in security experts, don’t tamper with affected systems so that they can be assessed properly by professionals.
Assume your whole internal network is compromised – Exchange is usually deployed in Active Directory which is also used for local services, so assess the damage/impact of the breach and rebuild your system. Except if you did extensive forensics and are 101% sure your systems are clean – but even then I would not be sure.
I’m not sure pulling backups is wise, especially if you don’t know when you are breached. Remember, this is active more than two months in the wild.
If this was total disaster for your environment, you don’t have a knowledge and skills to manage or upgrade Microsoft Exchange – this is a time to switch to cloud – I’m not preaching cloud, but I’m reading a lot of disaster stories these days, and I think for a lot of people cloud is good solution.
Lessons Learned
We are still learning about this attack, but some things obviously need to improve, I read a lot through these days, and there is a lot that you can improve in your environment.
Patch timely – many people are far behind with CUs for Exchange – in this situation where time was of essence, that meant that they were losing precious hours and in the end if the machines where online during patching – you got breached. If you were in order with CUs, this was 15 minutes patch job and that time made difference between being breached and closing the door.
Upgrade software/migrate on time – number of people with Ex 2010 or older is astonishing. Upgrade or use cloud services, there is a choice.
Backup offsite/cold backup – in situation where somebody can gain access to your entire infrastructure, like this exploit can, offsite/cold storage that is not connected to your network can be life savior for your company and your mind.
Follow security blogs/news/reddit often – also because of the timing this was extremely important in this breach.
Don’t use Administrator account – this really didn’t matter that much, nor is 2FA implemented on your OWA, but it looks like scripts get confused when there is no Administrator sccount in the system.
Last one is going to be bombshell for many of long time Windows/Exchange admins. I know, because I got strange looks every time I mention how I handle Exchange.
Wait for it…
I run my Exchange environments in separate domain, physically disconnected from my “production” domain. So, users in my domains have one login for their PCs/laptop and all their files, data, DBs.. and separate one for email services and they access mail like it is somewhere in the cloud, not on premise.
I often got strange looks and comments like – “but that is not practical, people need to manage with different accounts/passwords, what about single sign-on, it is much more convenient…”
It is convenient, but becomes horror story in situation like this. Even if I have to “nuke” my Exchange server, it will be just that – Exchange. Rest of my production stays intact.