The FIP-FS “Microsoft” Scan Engine failed to load. PID: 25212, Error Code: 0x80004005. Error Description: Can’t convert “2201010009” to long.

Happy New Year from Microsoft to Exchange on-premise users!

Due to a pretty bad MSExchange Antimailware update that was sent out, As a result of bad code, the MSExchange Antimalware engine broke, and Exchange will not process email.

The following errors are found in the event viewer of the affected systems:

Event 5801:

The anti-malware agent encountered an error while scanning. MessageId: <guid@domain.net> Message sent: 1/1/2022 5:05:21 PM From: <> Size: 12503 Bytes Error: Microsoft.Filtering.ScanAbortedException: Exception of type 'Microsoft.Filtering.ScanAbortedException' was thrown.
   at Microsoft.Filtering.InteropUtils.ThrowPostScanErrorAsFilteringException(WSM_ReturnCode code, String message)
   at Microsoft.Filtering.FilteringService.EndScan(IAsyncResult ar)
   at Microsoft.Exchange.Transport.Agent.Malware.MalwareAgent.OnScanCompleted(IAsyncResult ar)
Event 5300:

The FIP-FS "Microsoft" Scan Engine failed to load. PID: 19120, Error Code: 0x80004005. Error Description: Can't convert "2201010009" to long.
Event 1106:

The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error

The current fix is to disable the anti-malware scan module with the following Exchange system script:

Disable-AntimailwareScanning.ps1

https://docs.microsoft.com/en-us/Exchange/antispam-and-antimalware/antimalware-protection/antimalware-protection?view=exchserver-2019

1/3/2022 Update:
After several days the issue came back. Apparently, disabling the anti-mailware scanning does not actually resolve it, possibly the mail processing still goes through the bad engine! Microsoft has now released an article with the corrected procedure of resetting the engine with the following article: https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-exchange-on-premises-transport-queues/ba-p/3049447

To reset the engine, run the following script: https://aka.ms/ResetScanEngineVersion

If unavailable, here is the code:

[CmdletBinding()]
param (
    [switch]$Force
)

begin {
    #region Remoting Scriptblock
    $scriptBlock = {
        #region Functions
        function Get-ExchangeInstallPath {
            return (Get-ItemProperty -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Setup -ErrorAction SilentlyContinue).MsiInstallPath
        }

        function StopServicesAndProcesses {
            Write-Host "$($env:COMPUTERNAME) Stopping MSExchangeTransport, FMS, and updateservice..."
            Stop-Service FMS -Force
            $updateservice = Get-Process updateservice -ErrorAction SilentlyContinue
            if ($null -ne $updateservice) {
                $updateservice | Stop-Process -Force
                Start-Sleep -Seconds 2
                $updateservice = Get-Process updateservice -ErrorAction SilentlyContinue
                if ($null -ne $updateservice) {
                    Write-Warning "$($env:COMPUTERNAME) Could not end process updateservice.exe. Please end this process and rerun the script."
                    return $false
                }
            }

            return $true
        }

        function RemoveMicrosoftFolder {
            Write-Host "$($env:COMPUTERNAME) Removing Microsoft engine folder..."
            $installPath = Get-ExchangeInstallPath
            if ($null -ne $installPath) {
                $microsoftFolder = Join-Path $installPath "FIP-FS\Data\Engines\amd64\Microsoft"
                Remove-Item -Recurse -Force $microsoftFolder
            }
        }

        function EmptyMetadataFolder {
            Write-Host "$($env:COMPUTERNAME) Emptying metadata folder..."
            $installPath = Get-ExchangeInstallPath
            if ($null -ne $installPath) {
                $metadataFolder = Join-Path $installPath "FIP-FS\Data\Engines\metadata"
                Get-ChildItem $metadataFolder | Remove-Item -Recurse -Force
            }
        }

        function StartServices {
            Write-Host "$($env:COMPUTERNAME) Starting services..."
            Start-Service FMS
            Start-Service MSExchangeTransport
        }

        function StartEngineUpdate {
            Write-Host "$($env:COMPUTERNAME) Starting engine update..."
            $installPath = Get-ExchangeInstallPath
            $updateScriptPath = Join-Path $installPath "Scripts\Update-MalwareFilteringServer.ps1"
            $fqdn = [System.Net.Dns]::GetHostEntry([string]"localhost").HostName
            & $updateScriptPath $fqdn
        }

        function WaitForDownload {
            $percentComplete = 0
            do {
                Start-Sleep -Seconds 1
                $transfer = Get-BitsTransfer -AllUsers | Where-Object { $_.DisplayName -like "Forefront_FPS*" }
                if ($null -ne $transfer) {
                    if ($null -ne $transfer.BytesTotal -and
                        $null -ne $transfer.BytesTransferred -and
                        $transfer.BytesTotal.GetType() -eq [Int64] -and
                        $transfer.BytesTransferred.GetType() -eq [Int64] -and
                        $transfer.BytesTotal -gt 0) {
                        $percentComplete = ($transfer.BytesTransferred * 100 / $transfer.BytesTotal)
                    }

                    Write-Progress -Activity "$($env:COMPUTERNAME) Downloading scan engines" -Status "$($transfer.BytesTransferred) / $($transfer.BytesTotal)" -PercentComplete $percentComplete
                }
            } while ($null -ne $transfer)
        }
        #endregion Functions

        Add-PSSnapin -Name Microsoft.Exchange.Management.Powershell.E2010
        $hasMailboxRole = (Get-ExchangeServer ($env:COMPUTERNAME)).ServerRole -like "*Mailbox*"
        if ((-not $Force) -and (-not $hasMailboxRole)) {
            Write-Host "$($env:COMPUTERNAME) This server does not have the Mailbox role. Add -Force to proceed anyway."
            return
        }

        Add-PSSnapin -Name Microsoft.Forefront.Filtering.Management.PowerShell
        $engineInfo = Get-EngineUpdateInformation
        Write-Host "$($env:COMPUTERNAME) UpdateVersion: $($engineInfo.UpdateVersion)"
        $isImpacted = $engineInfo.UpdateVersion -like "22*"
        if ((-not $Force) -and (-not $isImpacted)) {
            Write-Host "$($env:COMPUTERNAME) This server is not impacted. Add -Force to proceed anyway."
            return
        }

        $succeeded = StopServicesAndProcesses
        if (-not $succeeded) {
            return
        }

        RemoveMicrosoftFolder
        EmptyMetadataFolder
        StartServices
        StartEngineUpdate
        WaitForDownload
    }
    #endregion Remoting Scriptblock
}
process {
    Invoke-Command -ScriptBlock $scriptBlock
}

Advanced 365 Mailbox management with MS Graph, Powershell, and JSON

Create Office 365 mailbox folders and advanced mailbox rules with Powershell and MS Graph

Recently, a request came through to create some email messaging processing for various Office 365 users, which involved creating some folder structure in mailboxes of a list of users, and creating a message rule that had some different criteria based on email message attributes.

The needs were:

  • Create a root folder
  • Create a folder inside that root folder
  • Create a rule:
    • Check the message Header for From address
    • Check for custom Header information
    • Move message to previously created subfolder
  • Finally, make sure there was no second mailbox rule created if already existed.

The New-MailboxFolder Powershell command works perfectly if the folder needed is for your own mailbox, if you want to run it against others, there is no current Powershell commandlet, so custom code must be created. While there are some basic examples out there, there was no comprehensive script anyone has published as of yet, so here is one I came up with.

For brevity purposes, I won’t go into detail the process that’s required to authenticate in order to run scripts against your environment, as there are quite a few resources available easily found by your favorite search engine, so I will skip over that process, and explain the “how to figure out what you need to accomplish with Powershell using MS Graph.”

$Mailboxes = @("userEmail.domain.com")
$Folders = @("RootFolder","OtherRootFolder")
$SubFolders = @("SubFolder")
$MailbRule = "RuleForSubFolder"

$AppId = "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx"
$AppSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$Scope = "https://graph.microsoft.com/.default"
$TenantName = "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx"

$Url = "https://login.microsoftonline.com/$TenantName/oauth2/v2.0/token"

Add-Type -AssemblyName System.Web

$Body = @{
    client_id = $AppId
	client_secret = $AppSecret
	scope = $Scope
	grant_type = 'client_credentials'
}

$PostSplat = @{
    ContentType = 'application/x-www-form-urlencoded'
    Method = 'POST'
    # Create string by joining bodylist with '&'
    Body = $Body
    Uri = $Url
}

$Request = Invoke-RestMethod @PostSplat

$Header = @{
    Authorization = "$($Request.token_type) $($Request.access_token)"
}

foreach($Mailbox in $Mailboxes) {

    $Uri = "https://graph.microsoft.com/v1.0/users/$Mailbox/mailFolders"
 
    $Mailboxfolders = Invoke-RestMethod -Uri $Uri -Headers $Header -Method Get -ContentType "application/json"
    $MailboxfoldersList = $Mailboxfolders.value.displayName
    $NextPage = $Mailboxfolders.'@Odata.NextLink'
   
    While($null -ne $NextPage) {
        $Mailboxfolders = Invoke-RestMethod -Uri $NextPage -Headers $Header -Method Get -ContentType "application/json"
        $MailboxfoldersList += $Mailboxfolders.value.displayName
        $NextPage = $Mailboxfolders.'@Odata.NextLink'
    }

    foreach($Folder in $Folders) {
        $Body = @"
        {
            "displayName": "$Folder"
        }
"@
        Write-Host "Mailbox: $Mailbox`nMailboxfolders: $($MailboxfoldersList)`nFolder wanted: $Folder"

        if($($MailboxfoldersList) -contains $Folder) {
            Write-Host "$Folder folder already found at mailbox $Mailbox, creating subfolder.`n"
            $UriParent = "https://graph.microsoft.com/v1.0/users/$Mailbox/mailFolders/?`$filter=displayname eq '$Folder'"
            $ParentFolder = Invoke-RestMethod -Uri $UriParent -Headers $Header -Method Get -ContentType "application/json"
            $UriSub = "https://graph.microsoft.com/v1.0/users/$Mailbox/mailFolders/$($ParentFolder.value.id)/childFolders"
        }

        else {
            $ParentFolder = Invoke-RestMethod -Uri $Uri -Headers $Header -Method Post -Body $Body -ContentType "application/json"
            Write-Host "Created new folder: $($ParentFolder.displayName) to mailbox $Mailbox!`n"
            $UriSub = "https://graph.microsoft.com/v1.0/users/$Mailbox/mailFolders/$($ParentFolder.value.id)/childFolders"
        }
    }
    
    $MailboxSubfolders = Invoke-RestMethod -Uri $UriSub -Headers $Header -Method Get -ContentType "application/json"
    $MailboxSubfoldersList = $MailboxSubfolders.value.displayName
    
    foreach($SubFolder in $SubFolders) {
        $Body2 = @"
        {
            "displayName": "$SubFolder"
        }
"@
        if($($MailboxSubfoldersList) -contains $Subfolder) {
            Write-Host "$Subfolder folder already found at mailbox $Mailbox.`n"
            $UriGetSub = "https://graph.microsoft.com/v1.0/users/$Mailbox/mailFolders/$($ParentFolder.value.id)/childFolders/?`$filter=displayname eq '$Subfolder'"
            $SubId = Invoke-RestMethod -Uri $UriGetSub -Headers $Header -Method Get -ContentType "application/json"
            
            $UriGetRules = "https://graph.microsoft.com/v1.0/users/$Mailbox/mailFolders/inbox/messageRules"

            $MailboxRules = Invoke-RestMethod -Uri $UriGetRules -Headers $Header -Method Get -ContentType "application/json"
            Write-Host "The rules are: $($MailboxRules.value.displayName)"
            $MailboxRulesList = $MailboxRules.value.displayName

            if($($MailboxRulesList) -contains "$MailbRule") {
                Write-Host "The mailbox rule $MailbRule already found at mailbox $Mailbox.`n"
                break
            }
            else {

                ## For syntax: https://developer.microsoft.com/en-us/graph/graph-explorer
                $RuleBody = @"
                {
                    "displayName": "$MailbRule",
                    "sequence": 2,
                    "isEnabled": true,
                    "conditions": {
                        "headerContains": [
                            "X-SomeCompany-tag: customTag"
                        ]
                    },
                    "actions": {
                        "moveToFolder": "$($SubId.id)",
                        "stopProcessingRules": true
                    },
                    "exceptions": {
                        "headerContains": [
                            "X-SomeCompany-Spam-Reason: eusafe",
                            "Reply-To: noreply@somedomain.com"
                        ]
                    }
                }
"@
                $RuleUri = "https://graph.microsoft.com/v1.0/users/$Mailbox/mailFolders/inbox/messageRules"
                $NewRule = Invoke-RestMethod -Uri $RuleUri -Headers $Header -Method Post -Body $RuleBody -ContentType "application/json"
                Write-Host "Created new Rule: $MailbRule in mailbox $Mailbox!`n"
            }
        }
        else {
            
            $NewSubfolder = Invoke-RestMethod -Uri $UriSub -Headers $Header -Method Post -Body $Body2 -ContentType "application/json"
            Write-Host "Created new subfolder: $($NewSubfolder.displayName) in $Folder to mailbox $Mailbox!`n"

            $RuleBody = @"
            {
                "displayName": "$MailbRule",
                "sequence": 2,
                "isEnabled": true,
                "conditions": {
                    "headerContains": [
                        "X-SomeCompany-tag: customTag"
                    ]
                },
                "actions": {
                    "moveToFolder": "$($NewSubfolder.id)",
                    "stopProcessingRules": true
                },
                "exceptions": {
                    "headerContains": [
                        "X-SomeCompany-Spam-Reason: eusafe",
                        "Reply-To: noreply@somedomain.com"
                    ]
                }
            }
"@
            $RuleUri = "https://graph.microsoft.com/v1.0/users/$Mailbox/mailFolders/inbox/messageRules"

            $NewRule = Invoke-RestMethod -Uri $RuleUri -Headers $Header -Method Post -Body $RuleBody -ContentType "application/json"
            Write-Host "Created new Rule: $MailbRule in mailbox $Mailbox!`n"
        }
    }
}

The key part of this article is not to show how fancy of a script I can write (disclaimer: the fancy spacing is from Visual Studio Code, use it!), but rather, how to get at the MS Graph API and syntax required to do the tremendous amount of capabilities that it’s got access to. I figured by throwing up a script that does quite a few different things that were previously only available if you ran several different scripts one after another (and hoped nothing broke), here’s an example of doing several different things easily using Powershell against the MS Graph API.

To see what JSON I needed, I used https://developer.microsoft.com/en-us/graph/graph-explorer extensively to see the fields to use, and looked up the REST API documentation to see what properties are required in the request body. (For example, for the Message Rule, I went to: https://docs.microsoft.com/en-us/graph/api/mailfolder-post-messagerules)

Logon failure with Azure AD DS based services

Fix a madding “Invalid password” error when trying to use an Azure service that uses Azure AD DS as it’s authentication

I hope this helps someone fix a madding “Invalid password” error when trying to use an Azure service that uses Azure AD DS as it’s authentication with a synced account from on-premise AD.

With a recent implementation of Windows Virtual Desktop, and interesting failure occurred with a set of users that were synchronized to Azure AD from an on-premise AD environment.

The environment was one where there was no longer an Azure AD Connect configuration in place, in fact, the on-premise AD environment was no longer available. All users were using Office 365 services without any issues, and Azure AD Domain Services was implemented for new Azure services, one of them being WVD.

All WVD services were tested with the admin account that created the resources, and some test users created in Azure AD, however, when the group that needed to use the service tried to use it with their accounts, around half the necessary users could not log into the VM image.

After verifying all permissions were correctly assigned, and checking to see if there were any relevant differences between the accounts that were able to log on vs. the ones that were not, I noticed that all accounts were able to log on to the web URL, however after the initial logon to the service, the originally synced accounts were failing with an “invalid password” error, whereas the ones that were directly created in Azure succeeded. – Aha! This pointed me to the fact that the accounts seemed to be having some sort of Azure ADDS failure, as 365 services were not dependent on that.

Quite a few articles were read all over the place, with none being any help, so I went back to the basics, and went over the Azure AD to Azure AD DS synchronization guide much more methodical than I previously had.

I’ll cut to the chase, in the middle of that guide, the following statement is made: When a user is created in Azure AD, they’re not synchronized to Azure AD DS until they change their password in Azure AD.

WVD – RDAgentBootLoader – Object reference not set to an instance of an object

This article is just a reminder to read event log errors carefully, as they tremendously help troubleshoot undocumented errors (which this blog is all about)

A Windows Virtual Desktop implementation kept having it’s Session host VMs randomly go to an Unavailable status, and after going through the full plethora of troubleshooting articles, I decided to take a closer look at the event error 3389:

Unable to retrieve DefaultAgent from registry: System.NullReferenceException: Object reference not set to an instance of an object.
   at RDAgentBootLoader.BootLoaderSettings.get_DefaultAgentPath() in S:\src\RDAgent\src\RDAgentBootloader\BootLoaderSettings.cs:line 82

The above error doesn’t say much, and is pretty cryptic, but when we look at the detail pane, we get some better information:

Log Name:      Application
Source:        RDAgentBootLoader
Date:          12/11/2020 10:21:05 AM
Event ID:      3389
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      lis-wvd-0.xxxxxxx.com
Description:
Unable to retrieve DefaultAgent from registry: System.NullReferenceException: Object reference not set to an instance of an object.
   at RDAgentBootLoader.BootLoaderSettings.get_DefaultAgentPath() in S:\src\RDAgent\src\RDAgentBootloader\BootLoaderSettings.cs:line 82
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="RDAgentBootLoader" />
    <EventID Qualifiers="0">3389</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2020-12-11T15:21:05.2207218Z" />
    <EventRecordID>10136</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>lis-wvd-0.xxxxxxxx.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Unable to retrieve DefaultAgent from registry: System.NullReferenceException: Object reference not set to an instance of an object.
   at RDAgentBootLoader.BootLoaderSettings.get_DefaultAgentPath() in S:\src\RDAgent\src\RDAgentBootloader\BootLoaderSettings.cs:line 82</Data>
  </EventData>
</Event>

The first thing that stood out to me was “Unable to retrieve DefaultAgent from registry“, and after a quick search to see where that registry key was, I found it at: \HKLM\SOFTWARE\Microsoft\RDAgentBootLoader

Looking at my settings, I saw the “DefaultAgent” is pointing to an entry version that is not in my registry:

Hmm…. if it’s pointing to a version that there’s no key, let me check if the binaries are there on the machine. Yes, they were! So let me try creating the missing key by checking to see what was in the existing key:

Ok, let’s create the missing key in the same syntax as the existing one:

After a restart, the session hosts are now Available and stable!