An insight into a hacked Exchange server

Matthieu Faou just wrote a whitepaper at ESET detailing the process where the sophisticated spy network Turla quietly exploited a backdoor in Microsoft Exchange servers that gave attackers unprecedented access to the emails of at least three targets over several years! The fascinating whitepaper is located here: ESET Lightneuron Whitepaper

Emails arrive on mobile device but not Outlook client

In a single AD Domain with an Exchange 2016 environment that was hosting multiple email domains, there was a power user that has several mailboxes with different email suffixes that would sporadically stop receiving inbound emails to his fully patched, Outlook 2016 client. (The 2013 client behaved exactly the same.)

The Exchange server system is a simple 2 server setup, the databases are replicated in a DAG array, with several different databases split out by company/department.

Exchange DB1

 

As you see in the figure, User1 has four different user accounts with four different mailboxes with different suffixes hosted on the same database, as he is from Company1, but needs to receive separated email to different mailboxes (reply with those unique email addresses), and authenticate separately.

After several hours of combing through the environment, and Microsoft support services unable to find anything amiss, one of the tests were creating a new Outlook profile, adding just one user account, and testing, well what do you know, it works! When a second mailbox is added to the profile, inbound mail stops to the client though. (Again, a mobile device receives the inbound mail immediately, but nothing occurs for the desktop Outlook client)

A hint on how to fix it came when I looked at User2. In this case, User2 also was opening up multiple mailboxes with the same clients, but there were no issues at all. As is evident, even though the mailboxes open from the same Exchange environment, the back end databases are separate.

After creating a new database for “@Othersuffix.com”, and migrating the User1 mailbox over to it, when that additional mailbox was opened in Outlook, mail flow continued!

The Exchange environment pictured has a lot more complexity, to end users it is completely separate, seemingly different Auth Domains, DNS, URLs, etc., but in reality is all the same back end infrastructure for ease of maintenance, (hint, KEMP is used to do a bunch of backward and forward URL rewriting) so adding some additional mailbox databases in the back end didn’t really complicate efforts too much.

Exchange database contains one or more mailboxes…

What do you do when you have what appears to be an empty mailbox base, but you get the dreaded: “This mailbox database contains one or more mailboxes, mailbox plans….” message?

remove-db error

The following are some commands to run:

Get-MailboxStatistics -Database DatabaseToRemove | ForEach { Update-StoreMailboxState -Database $_.Database -Identity $_.MailboxGuid -Confirm:$false }

Get-MailboxStatistics -Database DatabaseToRemove | where {$_.DisconnectReason -eq "SoftDeleted"} | foreach {Remove-StoreMailbox -Database $_.database -Identity $_.mailboxguid -MailboxState SoftDeleted}
Get-Mailbox -Database DatabaseToRemove -Archive
Get-Mailbox -Database DatabaseToRemove -PublicFolder
Get-Mailbox -Database DatabaseToRemove -Arbitration
Get-Mailbox -Database DatabaseToRemove -AuditLog

If after all those back empty you still have the issue, try to remove the database with the -Verbose parameter, as that parameter will show you what mailboxes (if any) still reside on the database.

Remove-MailboxDatabase DatabaseToRemove -Verbose

If the removal process still fails, a possibility is that the database in question is an Archive Database for a mailbox residing on a different mailbox database.

The following command helps you list mailboxes using a specific database as Archive Database:

Get-Mailbox | where {$_.ArchiveDatabase -eq DatabaseToRemove}

You can migrate just the archive mailbox to another database like so:

New-MoveRequest username -ArchiveOnly

The database can now be removed after the move is completed!