Category Archives: Microsoft Exchange

Everything related to Exchange

Brightmail does not deliver email to Distribution group members

The Story

Note: This article assumes you have Brightmail Gateway

When you try to send an email to a particular Exchange Distribution group Group@domain.com the result is either users don’t exist or you might get the following error if you test with Microsoft Test connectivity online tool.

Error:

The server returned status code 554 – Transaction failed. The server response was: 5.7.1 Delivery not authorized

Other related errors

‘554 5.7.1: You are not allowed to connect’

clip_image001

Cause:

Because the group has been cached in the Symantec gateway with its old members, The result could be an error that users don’t exist or delivery is not authorized.

Solution:

To solve this problem, You need to go to Brightmail gateway Administration > Directory Integration and click on your AD Directory > Advanced and hit on Clear Cache.

This would cleared the cached group and take the most recently updated group and its members.

This should resolve the problem.

clip_image002

How to clear the DDS cache in Messaging Gateway

https://knowledge.broadcom.com/external/article?legacyId=tech132131

Skype for Business IM integration with Exchange 2016 OWA–Part 2

This article is a completion of Part 1, Click here to go to Part 1

Configuration Steps – Part 2

7. On Exchange: Enable OWA VD Instant Messaging
8. On Exchange: Enable Messaging on OWA Policy
9. On Exchange: Create Enterprise Application for Skype Pool.
10. On Exchange: Create new SettingOverride for Skype for Business.
11- Generate a new Certificate for Exchange IM
12. Assign the newly imported certificate to IIS Exchange Back End site
13. On Exchange: Restart the WebAppPool
14. Log out and sign back in to OWA to Check
15. Troubleshooting methods

    7- On Exchange Server: Enable OWA VD Instant Messaging

    Part of enabling IM integration between Exchange and SfB is to enable OWA Virtual Directory to allow this. The below cmdlet does the job for you on all your Exchange Servers

    From Exchange, Launch Exchange Management and run the following cmdlet

    Get-OwaVirtualDirectory | Set-OwaVirtualDirectory -InstantMessagingEnabled $true -InstantMessagingType Ocs

    clip_image001[6]_thumb

    8- On Exchange: Enable Messaging on OWA Policy

    Run the following to enable Messaging for Owa Policy

    Get-OwaMailboxPolicy | Set-OwaMailboxPolicy -InstantMessagingEnabled $true -InstantMessagingType Ocs

    clip_image001[6]_thumb[1]

    9- On Exchange: Create Enterprise Application for Skype Pool.

      From Exchange Management shell Run the following cmdlet

      Cd $exscripts

      .\Configure-EnterprisePartnerApplication.ps1 -AuthMetadataUrl “https://sbg-pool01.domain.com/metadata/json/1” -ApplicationType Lync

      The AuthMetadataUrl is going to be your local Skype for Business Pool URL. This URL should work in your Exchange server without any Certificate error. Meaning that the certificate assigned to your Skype for Business pool should already be imported to Exchange Servers to trust this URL.

      image_thumb[14]

        If your previous configuration is correct then you should see the “The Configuration has Succeeded” Message.

          10- On Exchange: Create new SettingOverride for Skype for Business.

          Notes:

          • To configure the same settings on all Exchange 2016 and Exchange 2019 servers in the Active Directory forest, don’t use the Server parameter.

          New-SettingOverride -Name “<UniqueOverrideName>” -Component OwaServer -Section IMSettings -Parameters @(“IMServerName=<Skype server/pool  name>”,”IMCertificateThumbprint=<Certificate Thumbprint>”) -Reason “<DescriptiveReason>” [-Server <ServerName>]

          The Thumbprint you use here will define if whether IM will work or not as this what secures the communication between Exchange and Skype. If you use the wrong certificate your Integration will fail and users wont be able to login to IM through OWA.

          11- Generate a new Certificate for Exchange IM

          IMPORTANT NOTE:

          In order for IM in OWA to work the certificate you will generate must have its common name set as mail.domain.com to match the configuration.

          Using Digicert tool on Exchange Server I will generate the CSR of the new certificate

          Click on Create CSR

          image_thumb[15]

          Choose SSL certificate type and make sure you choose Mail.domain.com as CN

          In the SANs type all of the involved servers (Skype for Business Frontends, Mailbox servers in FQDN and in Hostnames as in the screenshot below). and click on Generate

          image_thumb[16]

          • Go to your CA Server’s CertSRV URL and copy the CSR code there to generate the new certificate.
          • Import the new certificate to the current server, then export it in PFX format and import it to all the Exchange Servers you’re planning to use.

          image_thumb[18]

          • After importing the certificate I will verify that I can see the private key

          image_thumb[19]

          Click on the Details and copy the Thumbprint or from MMC right click the certificate > Properties give it a friendly name e.g. (IM) and then from Exchange Management shell you can copy the Thumbprint directly.

          Get-ExchangeCertificate | select thumbprint,friendlyName

          image_thumb[20]

          Now use the previous script to create the setting Override for OwaServer.

          Things you can change are in bold “Name, IM Servername Value, and the Thumbprint value”.

          New-SettingOverride -Name “IM Override” -Component OwaServer -Section IMSettings -Parameters @(“IMServerName=SBG-Pool01.domain.com“,”IMCertificateThumbprint= 28E4B1BA0F2FCB1535AF199F02A64EFC78367F2D“) -Reason “Configure IM”

          image_thumb[21]

          If you enter the server parameter to use a single server you can change that by using. Note that you must not use FQDN but rather only the server’s hostname.

          Get-SettingOverride | Set-SettingOverride -Server sbg-mx01,sbg-mx02

          image_thumb[22]

          This should generate an event ID 112 on Exchange servers involved in the deployment.

          clip_image001[9]_thumb

            12. Assign the newly imported certificate to IIS Exchange Back End site

            Once the certificate is in the server store, You will be able to easily find in from IIS and bind it to the Exchange Back End site.

            This is the most crucial step to get IM to work in OWA. Don’t worry about breaking up Exchange Sites or Powershell. If you have added Exchange Servers Hostnames and FQDNs in this certificate then you should be good.

            • Now Launch IIS
            • Click on Exchange Back End
            • Select Binding
            • Click on the 444 port and edit
            • Select the newly generated certificate that has the mail.domain.com as CN. (This certificate must also have all Exchange Servers hostnames and FQDNs set as SANs)

            image_thumb[23]

            image_thumb[24]

            Make sure you change the backend cert to the new on all the involved Exchange Servers.

            13. On Exchange: Restart the WebAppPool

            Restart-WebAppPool MSExchangeOWAAppPool

            image_thumb[25]

              14. Log out and sign back in to OWA to Check

              Log out of OWA and back in and check if you are able to Login to IM . It should normally sign you in automatically but in case of an error then you should see it.

              image_thumb[29]

              In case of an error you should see the following.

              image_thumb[27]

              If it works then you should see the presence

              image_thumb[28]

              15. Troubleshooting Methods

              If you follow the above steps correctly then it should work especially when applying the right certificate for your Exchange Back End IIS part however if you face an error then you should do the following steps to troubleshoot the error

              • Set the Eventlog for Instant Messaging on Exchange from Low to High

              Set-EventLogLevel -Identity “sbg-mx01\MSExchange OWA\InstantMessage” -Level High

              image_thumb[30]

              • Look in the following path for errors

              C:\Program Files\Microsoft\Exchange Server\V15\Logging\OWA\InstantMessaging

              • Check the Healthset of the OWA Instant Messaging.

              Get-ServerHealth -HealthSet OWA.Protocol.Dep -Server sbg-mx01 | Format-Table Name, AlertValue –Auto

              image_thumb[31]

              Get-MonitoringItemIdentity -Server sbg-mx01 -Identity OWA.Protocol.Dep | Format-Table Identity,ItemType,Name -Auto

                image_thumb[32]

                Ref

                https://docs.microsoft.com/en-us/exchange/plan-and-deploy/post-installation-tasks/configure-im-integration-with-owa?view=exchserver-2019

                https://docs.microsoft.com/en-us/exchange/high-availability/managed-availability/health-sets?view=exchserver-2019

                Skype for Business IM integration with Exchange 2016 OWA–Part 1

                The Story

                A good and detailed documentation is everything we need to implement any kind of project especially if it’s an integration between two different servers that perform different roles.

                And with PKI involved the complications multiply thus a good article write up is what we need.

                Previously I have tried a test lab with Skype for Business 2015/2019 IM Integration with Exchange 2016/2019 and the result was a complete failure and endless search for what’s missing to get IM to work from OWA?

                image

                ERROR

                Upon completion of the steps mentioned in Microsoft’s Official documentation and after restarting Exchange IIS or OWAAppPool you will see this when you try to login to OWA with your user

                There’s a problem with instant messaging. Please try again later.

                image

                MS Official Documentation

                In their Official documentation Microsoft says that the certificate in question must be trusted by all the servers involved meaning Skype for Business Frontend and Mailbox Servers.

                Meanwhile this is true, it still would not get the IM to login/work although it might drop the initialize event ID 112 in the event log.

                clip_image001

                Here is what MS says about the certificate.

                Exchange and Skype for Business integration requires server certificates that are trusted by all of the servers involved. The procedures in this topic assume that you already have the required certificates. For more information, see Plan to integrate Skype for Business Server 2015 and Exchange. The required IM certificate thumbprint refers to the Exchange Server certificate assigned to the IIS service.

                REF URL: https://docs.microsoft.com/en-us/exchange/plan-and-deploy/post-installation-tasks/configure-im-integration-with-owa?view=exchserver-2019#what-do-you-need-to-know-before-you-begin

                image

                Step by Step Deployment

                To do things the way that should get this to work, I will detail steps one by one so we can be sure to get the positive results we are all waiting for when dealing with Exchange and Skype for Business.

                Exchange IM URL 1: mail.domain.com

                Skype for Business Pool FQDN: SBG-Pool01.domain.com

                Autodiscover URL : Autodiscover.Domain.com

                Prerequisites

                1. For Default and Web Service Internal, Your Skype for Business Frontend Server/Pool must use a certificate that is generated from an internal CA which you can use later to generate Exchange’s IM Certificate.
                2. UCMA must be installed (Doesn’t matter if version 4 or 5) both are supposed to work with Exchange 2016.
                3. Local Certification Authority must already be deployed in the domain.

                Configuration Steps – Part 1

                1. On SfB: Set CsAuthConfiguration Autodiscover URL for Skype server to find Exchange Autodiscover
                2. On SfB: Get-CsSite to see what is the current site ID.
                3. On Exchange: Check AutodiscoverServiceInternalURI
                4. On SfB: Create new Partner
                5. On SfB: Create new Trusted Application Pool
                6. On SfB: Create new Trusted Application ID

                Configuration Steps – Part 2

                7. On Exchange: Enable OWA VD Instant Messaging
                8. On Exchange: Enable Messaging on OWA Policy
                9. On Exchange: Create Enterprise Application for Skype Pool.
                10. On Exchange: Create new SettingOverride for Skype for Business.
                11- Generate a new Certificate for Exchange IM
                12. Assign the newly imported certificate to IIS Exchange Back End site
                13. On Exchange: Restart the WebAppPool
                14. Log out and sign back in to OWA to Check
                15. Troubleshooting methods

                Prerequisites

                1- Update or Create Server Default and Web Service Internal Certificate for SfB Pool servers

                The certificate installed on the Skype for Business Pool Frontend servers must be generated from a local Certification Authority which can be trusted by Exchange Server services.

                The Certificate generated for Skype for Business pool as in the below screenshot is generated from my CA and includes the names of the servers:

                • Skype for Business Pool
                • Skype for Business Frontend FQDNs
                • Exchange Servers
                • Autodiscover FQDN
                • Lyncdiscover.domains.com
                • Lyncdiscoverinternal.domains.com
                • sip.domains.com
                • meet.domains.com
                • dialin.domain.com
                • External.domain.com

                image

                image

                2- UCMA must be installed

                On both Exchange and Skype for Business servers I already have UCMA 4.0 version installed, but if you don’t have it or have an older version then you can’t continue without it.

                image

                3- Make sure you have a Local Certification Authority deployed in your domain.

                Configuration Steps – Part 1

                1- On SfB: Set CsAuthConfiguration Autodiscover URL for Skype server to find Exchange Autodiscover

                For Skype for Business Server to find Exchange Autodiscover Service point and to be able to authenticate servers we’ll be using the below cmdlet

                This enables both servers to authenticate and share information when needed and without user’s interference.

                Set-CsOauthConfiguration -ExchangeAutodiscoverUrl https://autodiscover.domain.com/autodiscover/autodiscover.svc

                image

                image

                Ref:

                https://docs.microsoft.com/en-us/powershell/module/skype/set-csoauthconfiguration?view=skype-ps

                2- On SfB: Get-CsSite to see what is the current site ID.

                Getting a site ID will be useful for later use to setup the Trusted Application Pool.

                On Skype for Business Management shell. Type the following

                Get-CsSite

                So the Site ID is 1. I will keep this for later use

                image

                3- On Exchange: Check AutodiscoverServiceInternalURI

                Specify the AutodiscoverServiceInternalURI for internal Autodiscover service. Make sure it points to your public URL and certificate not the internal one otherwise your users will get a certificate error through Outlook and might cause IM chat not to work.

                The Cmdlet would be

                Get-ClientAccessService | Set-ClientAccessService –AutoDiscoverServiceInternalUri https://autodiscover.domain.com/autodiscover/autodiscover.xml

                image

                4- On SfB: Create new Partner Application

                On Skype for Business Server, Launch Management Shell and use this cmdlet to add Exchange as a trusted Application to the SfB topology.

                New-CsPartnerApplication -Identity Exchange -ApplicationTrustLevel Full -MetadataUrl “https://autodiscover.domain.com/autodiscover/metadata/json/1

                image

                5- On SfB: Create new Trusted Application Pool

                New-CsTrustedApplicationPool -Identity mail.domain.com -Registrar sbg-pool01.domain.com -Site 1 -RequiresReplication $false

                image

                6- On SfB: Create new Trusted Application ID

                From SfB Management Shell run the following cmdlet .

                New-CsTrustedApplication -ApplicationId OutlookWebAccess -TrustedApplicationPoolFqdn mail.domain.com -Port 5199

                image

                Finally

                clip_image001[4]

                Click on the link below for Part 2

                Skype for Business IM integration with Exchange 2016 OWA–Part 2

                an Exchange mailbox was mistakenly migrated over another user’s object used by another user

                The Story

                If you ever used Prepare moverequest command to migrate a user and forgot to use ADMT to rewrite user’s properties with the old attributes. You might have used ADMT again to rewrite the properties.

                If you use ADMT you will need to exclude all Exchange Attributes from the source since its already copied using Prepare-move request script however, in some cases some people do make mistakes and you might have came through the same mistake my colleague  have done during one of these extremely complicated Cross forest Migrations where you’d prepare a CSV files through PowerShell and names wouldn’t match Sam accounts.

                Don’t Panic

                If however, you forgot again to exclude the Exchange attributes while using ADMT then you most likely wont see the user in the Target forest which will cause to panic thinking the user is gone .. But no the user is not gone don’t panic.

                When you look for the user’s mailbox on the target forest after the move request is completed you’ll get an error reporting the user can’t be found

                image

                Solution

                To fix the problem you’ll need to change to attributes only for this migrated user. (In the target forest after user mailbox move is completed).

                The attributes are

                msExchRecipientDisplayType    1073741824
                msExchRecipientTypeDetails    128

                The wrong Attributes are as following.

                image

                You will need to fix them to look like the following

                image

                Once you apply the change you’ll need to wait for a minute or few depending on your AD replication speed.
                The problem will be then solved

                image

                Microsoft Exchange 2010 SP3 Link HACKED

                Update: Microsoft replied to me and fixed the link. see screenshot below

                WATCH Microsoft Exchange URL Hacked

                If you have Exchange 2010 SP3 and planning to download the latest Rollup , Google will take you to the following link

                https://www.microsoft.com/en-us/download/details.aspx?id=100910

                Once you click on that link to download the RollUp update, You might want to check the system requirements links and that would list two main links

                image

                The Exchange 2010 Prerequisites link will first redirect you to this URL which has an expired certificate.

                http://www.microsoftpinpoint.com/

                And that will then redirect you to this link (Seems to be a Chinese website)

                http://123.wo80.com/

                Luckily the antivirus managed to catch and block this page however, on any server that’s not running any antivirus this would certainly infect the server.

                Phishing Alert!

                image

                image

                Video here

                Contacting Microsoft

                After I got in contact with Microsoft about the issue. Microsoft replied stating they have informed their security team and fixed the issue.

                Microsoft Exchange Vulnerability affects all Exchange versions

                image

                CVE-2020-0688 | Microsoft Exchange Validation Key Remote Code Execution Vulnerability

                Security Vulnerability

                Date of Publishing: February/11/2020

                Microsoft has announced a vulnerability has been found in all Exchange Server 2010 through 2019 versions, The vulnerability allows an attack to send a specially crafted request to the affected server in order to exploit it.

                When could this happen?

                A remote code execution vulnerability exists in Microsoft Exchange Server when the server fails to properly create unique keys at install time.

                Knowledge of a the validation key allows an authenticated user with a mailbox to pass arbitrary objects to be deserialized by the web application, which runs as SYSTEM.

                The security update addresses the vulnerability by correcting how Microsoft Exchange creates the keys during install.

                Affected Versions:

                • Microsoft Exchange Server 2010 Service Pack 3 Update Rollup 30
                • Microsoft Exchange Server 2013 Cumulative Update 23   
                • Microsoft Exchange Server 2016 Cumulative Update 14   
                • Microsoft Exchange Server 2016 Cumulative Update 15   
                • Microsoft Exchange Server 2019 Cumulative Update 3   
                • Microsoft Exchange Server 2019 Cumulative Update 4

                image

                Solution:

                Until now Microsoft has not provided any solution or work around to cover this vulnerability.

                Mitigations

                Microsoft has not identified any mitigating factors for this vulnerability.

                Workarounds

                Microsoft has not identified any workarounds for this vulnerability.

                NOTE:

                Keep an eye on the below link for any change

                https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0688

                Upgrading Exchange Online PowerShell to V2 Module

                Managing Exchange Online

                If you have Exchange Online and your users are MFA enabled then you most likely will be using Exchange Online’ s ECP (Exchange Control Panel or Admin Center) to connect to Exchange Online PowerShell through the Hybrid Windows since this is the only supported way with MFA.

                image

                Clicking on Configure would install the PowerShell Module of Exchange Online which looks like the below screenshot.

                image

                New PowerShell with MFA support

                If you have launched Exchange Online PowerShell today then you most likely have noticed there’s a red line stating the possibility to try the new (Preview Version) of Exchange PowerShell V2 .

                Microsoft has recently released a new version of Exchange Online PowerShell Module which supports MFA and can be run directly from your computer without the need to login to Exchange Online Admin Center and download any files from there.  Check details in this link

                As stated in the article, the Module is just in preview so it has some known and maybe unknown bugs as well.

                How to Install it?

                The installation process is pretty straightforward, Launch Windows PowerShel as an Administrator (It’s required for the installation).

                Run these 4 cmdlets

                Set-ExecutionPolicy RemoteSigned
                Install-Module PowershellGet –Force
                Update-Module PowershellGet
                Install-Module -Name ExchangeOnlineManagement

                image

                You might get a warning that the Module you’re about to install is from an Untrusted Repository, Accept it by typing Y and hit enter

                Type the following cmdlet to ensure that Exchange Online Management module is installed

                Import-Module ExchangeOnlineManagement; Get-Module ExchangeOnlineManagement

                image

                Connecting to Exchange Online

                To connect to Exchange Online, Run the following cmdlet along with the new parameter –EnableErrorReporting which gives the ability to record all the cmdlets that you have run along with errors generated as well.

                Connect-ExchangeOnline -EnableErrorReporting -LogDirectoryPath e:\ExchOnlineLogs.txt -LogLevel All

                image

                image

                After connecting, I am going to try and run two commands the Old Cmdlets and New Cmdlet and see the difference between them:

                Get-CASMailbox -ResultSize 10
                Get-EXOCasMailbox -ResultSize 10
                

                image

                The new Cmdlet has much more details, although it says that it runs faster but it took few seconds more than the old one to run (Probably first time).

                image

                After you run those two Cmdlets, There will be two files generated in the log directory which we have pointed the parameter to save files to.

                The CSV files have details about the two cmdlets and the HTTP Method they are utilizing in order to connect along the Request and response latency.

                imageimage

                This new version seems to be extremely useful esp in environments where such deep details are needed for troubleshooting issues.

                Stay tuned for more

                Reference:

                https://docs.microsoft.com/en-us/powershell/exchange/exchange-online/exchange-online-powershell-v2/exchange-online-powershell-v2?view=exchange-ps

                How to Sync Cloud User to On-premises AD ?

                The Story:

                I have got this client who constantly keeps on making the mistake of create user from Cloud and provision them with a license in an Exchange Hybrid environment.

                Although this is not difficult to fix but it’s not the recommended approach when creating a new user especially in a Hybrid environment since Exchange on-premises won’t recognize this user and most likely will consider any incoming emails from it as spoof or spam.

                How to Create a Cloud user from Exchange On-premises?

                From Exchange on-premises ECP Admin panel you have the option to directly create user on-cloud which will also create a user object on on-premises AD.

                image

                Second option – Using Powershell

                It’s not that much different than the Web UI option but it’s just for people who prefer using PowerShell than GUI

                Enable-RemoteMailbox –Identity User –RemoteRoutingAddress user@yourTenant.mail.onmicrosoft.com

                The reason to follow those two methods is due to the need of Exchange on-premises being aware of each of those users so mail flow between Exchange on-premises and Online would not get affected and route this users mail to the wrong place or flag it as spammed or spoof …etc.

                The Real Question now is: How to Sync Cloud User to On-premises AD ?

                If by mistake we created a user on Cloud (Office 365) and we forgot to create an AD User for this account, that user might already have started using his account on Office 365 (Sharepoint, Exchange, Teams) etc.

                There also might be the intention of moving users from Cloud to On-premises Exchange in case the company wanted to decrease their spending on cloud users and in this case when Migrating a cloud user to on-premises you will get the following errors:

                image

                test3@domain.com

                Status: Failed

                test3@domain.com Skipped item details

                User status

                Data migrated:

                Migration rate:

                Last successful sync date:

                Error: MigrationPermanentException: Cannot find a recipient that has mailbox GUID ‎’03c9764e-8b8e-4f33-94d1-ef098c4de656‎’. –> Cannot find a recipient that has mailbox GUID ‎’03c9764e-8b8e-4f33-94d1-ef098c4de656‎’.

                So how do we overcome this situation since syncing a user might require you to delete the cloud user and recreate it on AD?

                Solution:

                To sync the user from the Cloud to on-premises you will need to follow these steps :

                1- Create an on-premises Mailbox where the following attributes would be matching the cloud user

                • UserPrincipalname
                • ProxyAddresses
                • SamAccountName
                • Alias

                2- The Location of the OU where the On-premises user is going to be created must be provisioned by ADConnect (Azure AD Connect)

                You can look which of these OU are provisioned by Starting AD Connect Sync Manager

                image

                By verifying the user you created in the AD is in the right OU, You can now start AD Sync from PowerShell to speed up the process.

                image

                Below, You can see the user has been successfully synchronized to the cloud without any issue.

                image

                Now we’ll see it from the portal to confirm the user is synced with AD

                image

                Depending on the Source anchor being used in ADConnect there might be a GUID conflict or not, You will get an error similar to when trying to migrate the user in the beginning however you can solve this by replacing the cloud user’s GUID (ImmutableID) with the on-premises user which will force the user to merge with the On-prem user.

                Let’s confirm in our case if the user on-cloud has a matching GUID with the one on-premises.

                From CMD or Powershell you can use the following command to get the user’s ImmutableID (ObjectGUID) .

                ldifde -f c:\Test.txt -d “cn=Test3,DC=Domain,DC=com”

                image

                Checking the notepad we just exported you can see the Immutable ID on AD for the User test3 is IkTni9mw7Ee4YefeGpz7IA==

                image

                To be able to see the user on Office 365, We need to logon to MSOL through Exchange Online powershell

                Connect to Exchange Online’s powershell using your Online ECP.

                image

                Once you click on Configure this should download an executable file that will launch PowerShell Online which allows you to use the Modern Authentication (MFA) to use PowerShell safely.

                image

                Connect-Msoluser will connect you to Office 365 and you’ll be able to get the user’s properties and see if the Immutable ID is matching to the user’s GUID.

                Once you’re connect you can use the following cmdlet to get the user’s properties.

                Get-MsolUser -UserPrincipalName test3@domain.com |fl DisplayName,ImmutableID

                image

                You can see they are matching each other, In case there’s a conflict then you can simply set the online user’s Immutable ID to match the on-premises user’s ImmutableID.

                Once done, Go and force ADConnect to sync the user and you’ll see if the problem has been resolved. The command for changing the Immutableid is as follows:

                Set-MsolUser -UserPrincipleName test3@domain.com -ImmutableID IkTni9mw7Ee4YefeGpz7IA==

                Ref:

                https://support.microsoft.com/en-us/help/2956029/migrationpermanentexception-cannot-find-a-recipient-that-has-mailbox-g

                https://docs.microsoft.com/en-us/exchange/hybrid-deployment/create-cloud-based-archive

                Move Request on Exchange 2019 (During failover) will warn you that it postponed due the move of DB

                The Issue:

                So while working on a new Exchange Migration project, I have encountered a weird issue where I could see users migration batch status complaining about being stalled due to (Big Funnel).

                The error is showing as in the below screenshot and it doesn’t occur instantly after you start the migration of the user but right after it starts.

                StalledDueToTarget_BigFunnel 68.47 MB (71,795,512 bytes) 20

                User StalledDueToTarget_BigFunnel 37.2 MB (39,003,538 bytes) 20

                User2 StalledDueToTarget_BigFunnel 14.71 MB (15,421,154 bytes) 20

                User3 StalledDueToTarget_BigFunnel 44.2 MB (46,345,009 bytes) 20

                User4 StalledDueToTarget_BigFunnel 4.647 MB (4,872,404 bytes) 20

                User5 StalledDueToTarget_BigFunnel 14.47 MB (15,169,768 bytes) 20

                User6  StalledDueToTarget_BigFunnel 171 MB (179,280,335 bytes) 20

                User7 StalledDueToTarget_BigFunnel 753.4 MB (789,980,880 bytes) 20

                User8 StalledDueToTarget_BigFunnel 18.35 MB (19,236,680 bytes) 20

                User9 StalledDueToTarget_BigFunnel 205.9 MB (215,951,208 bytes) 20

                User10 StalledDueToTarget_BigFunnel 166.2 MB (174,243,238 bytes) 20

                User11 StalledDueToTarget_BigFunnel 13.81 MB (14,481,739 bytes) 20

                User12 StalledDueToTarget_BigFunnel

                image

                Error Message

                Request ‘domain.com/CompanyUSER/Region1/User1’ (b5dbf3ff-21a1-4ec1-a29c-15b794a17386) failed.

                Error code: -2146233088

                Connection to the Content Transformation Service has failed.

                Context:

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                Operation: IMapiFxProxy.ProcessRequest

                OperationSide: Target

                b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMailbox.ExportMessages

                Operation: IMailbox.ExportMessages

                OperationSide: Source

                b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

                Flags: SkipItemValidation

                PropTags: (null)

                ——–

                >>>> Scheduled WorkItems: EnumerateFolderMessages(P:29792,R:0,S:0,C:14); EnumerateFolderMessages(P:29807,R:0,S:0,C:24,Cnt=3); WriteFolderMessages(P:0,R:0,S:0,C:686); EnumerateFolderMessages(P:30554,R:0,S:2,C:55); EnumerateFolderMessages(P:30612,R:0,S:0,C:36,Cnt=2); WriteFolderMessages(P:3,R:0,S:0,C:301); EnumerateFolderMessages(P:30975,R:0,S:1,C:21); WriteFolderMessages(P:2,R:0,S:0,C:97); EnumerateFolderMessages(P:31094,R:0,S:0,C:18,Cnt=6); EnumerateFolderMessages(P:31279,R:0,S:0,C:19)

                ————–

                The Microsoft Exchange Mailbox Replication service was unable to save changes to request.

                Request: ‘9a444721-80e2-4cf8-8c81-8a3afe3dc775’ (bbc2c66e-857e-4ba6-8462-9d66da73d400)

                Database: DB01

                Error:

                The request has been temporarily postponed because a database has failed over. The Microsoft Exchange Mailbox Replication service will attempt to continue processing the request when capacity becomes available on the new server hosting the database.

                image

                Looking at the event ID number 1114 it mentions there seems to be an issue with the request seems there might be an issue with the mailbox being moved.

                To dig deeper I am going to search some of the users reporting the same error by using their GUID

                image

                The property “DisplayName” with value “User LastName” is invalid. The value can’t contain leading or trailing whitespace.

                Solution: (For a single user)

                To resolve the problem, I am going to remove the trailing space in the end of the display name. You can safely use the below Powershell script to solve this problem however, if you don’t trust yourself or you’re not familiar much with Powershell, You can try it on a lab or a single test user for instance.

                Get-Mailbox -Identity USER | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

                image

                Solution: (For all users)

                Get-Mailbox | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

                clip_image001

                Some relevant errors you might encounter as you’re moving users to Exchange 2019

                Error code: -2146233088

                Connection to the Content Transformation Service has failed.

                Context:

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                Operation: IMapiFxProxy.ProcessRequest

                OperationSide: Target

                eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMailbox.ExportMessages

                Operation: IMailbox.ExportMessages

                OperationSide: Source

                eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

                Flags: SkipItemValidation

                PropTags: (null)

                ——–

                >>>> Scheduled WorkItems: EnumerateFolderMessages(P:14014,R:0,S:0,C:13); EnumerateFolderMessages(P:14029,R:0,S:0,C:15,Cnt=2); WriteFolderMessages(P:1,R:0,S:0,C:132); EnumerateFolderMessages(P:14192,R:0,S:0,C:17); WriteFolderMessages(P:1,R:0,S:0,C:48); EnumerateFolderMessages(P:14259,R:0,S:0,C:12,Cnt=4); EnumerateFolderMessages(P:14320,R:0,S:1,C:15); EnumerateFolderMessages(P:14337,R:0,S:0,C:20); WriteFolderMessages(P:2,R:0,S:0,C:126); EnumerateFolderMessages(P:14485,R:0,S:0,C:30)

                Search and Delete certain Items/Folders from a Mailbox

                The Story

                During a project of Hybrid migration from Exchange on-premises to Exchange online, I was almost about to finalize the project by moving the last remaining users mailboxes however had an interesting issue to deal with where a user was failing with the following error:

                The Error after migration:

                Error: MigrationPermanentException: Mailbox dumpster size 50.87 GB (54,620,074,576 bytes) exceeds target quota 30 GB –> Mailbox dumpster size 50.87 GB exceeds target quota.

                image

                After some research it turned out that you can clean the dumpster using search-mailbox PowerShell cmdlet, Sync the user’s object with ADConnect and then continue the migration from the last failure.

                To solve the issue, Go on your Exchange on-premises and launch Exchange Management shell

                Solution applied:

                First, Let’s see the user’s dumpster and recoverable items

                Get-MailboxFolderStatistics -Identity “User” -FolderScope RecoverableItems | Format-Table Name,FolderPath,ItemsInFolder,FolderAndSubfolderSize

                image

                To Delete the dumpster only use this

                Delete dumpster only

                Search-mailbox -identity User -SearchDumpsterOnly –DeleteContent

                To delete a certain email with certain subject in the dumpster use the following:

                Get-mailbox “user”| search-mailbox –searchquery “Subject:’*'” –DeleteContent –SearchDumpsterOnly

                image

                The cmdlet will search and delete

                clip_image001

                image

                Reference:

                https://docs.microsoft.com/en-us/microsoft-365/compliance/search-for-and-delete-messagesadmin-help