Category Archives: Exchange 2016

Brightmail does not deliver email to Distribution group members

The Story

Note: This article assumes you have Brightmail Gateway

When you try to send an email to a particular Exchange Distribution group Group@domain.com the result is either users don’t exist or you might get the following error if you test with Microsoft Test connectivity online tool.

Error:

The server returned status code 554 – Transaction failed. The server response was: 5.7.1 Delivery not authorized

Other related errors

‘554 5.7.1: You are not allowed to connect’

clip_image001

Cause:

Because the group has been cached in the Symantec gateway with its old members, The result could be an error that users don’t exist or delivery is not authorized.

Solution:

To solve this problem, You need to go to Brightmail gateway Administration > Directory Integration and click on your AD Directory > Advanced and hit on Clear Cache.

This would cleared the cached group and take the most recently updated group and its members.

This should resolve the problem.

clip_image002

How to clear the DDS cache in Messaging Gateway

https://knowledge.broadcom.com/external/article?legacyId=tech132131

Skype for Business IM integration with Exchange 2016 OWA–Part 2

This article is a completion of Part 1, Click here to go to Part 1

Configuration Steps – Part 2

7. On Exchange: Enable OWA VD Instant Messaging
8. On Exchange: Enable Messaging on OWA Policy
9. On Exchange: Create Enterprise Application for Skype Pool.
10. On Exchange: Create new SettingOverride for Skype for Business.
11- Generate a new Certificate for Exchange IM
12. Assign the newly imported certificate to IIS Exchange Back End site
13. On Exchange: Restart the WebAppPool
14. Log out and sign back in to OWA to Check
15. Troubleshooting methods

    7- On Exchange Server: Enable OWA VD Instant Messaging

    Part of enabling IM integration between Exchange and SfB is to enable OWA Virtual Directory to allow this. The below cmdlet does the job for you on all your Exchange Servers

    From Exchange, Launch Exchange Management and run the following cmdlet

    Get-OwaVirtualDirectory | Set-OwaVirtualDirectory -InstantMessagingEnabled $true -InstantMessagingType Ocs

    clip_image001[6]_thumb

    8- On Exchange: Enable Messaging on OWA Policy

    Run the following to enable Messaging for Owa Policy

    Get-OwaMailboxPolicy | Set-OwaMailboxPolicy -InstantMessagingEnabled $true -InstantMessagingType Ocs

    clip_image001[6]_thumb[1]

    9- On Exchange: Create Enterprise Application for Skype Pool.

      From Exchange Management shell Run the following cmdlet

      Cd $exscripts

      .\Configure-EnterprisePartnerApplication.ps1 -AuthMetadataUrl “https://sbg-pool01.domain.com/metadata/json/1” -ApplicationType Lync

      The AuthMetadataUrl is going to be your local Skype for Business Pool URL. This URL should work in your Exchange server without any Certificate error. Meaning that the certificate assigned to your Skype for Business pool should already be imported to Exchange Servers to trust this URL.

      image_thumb[14]

        If your previous configuration is correct then you should see the “The Configuration has Succeeded” Message.

          10- On Exchange: Create new SettingOverride for Skype for Business.

          Notes:

          • To configure the same settings on all Exchange 2016 and Exchange 2019 servers in the Active Directory forest, don’t use the Server parameter.

          New-SettingOverride -Name “<UniqueOverrideName>” -Component OwaServer -Section IMSettings -Parameters @(“IMServerName=<Skype server/pool  name>”,”IMCertificateThumbprint=<Certificate Thumbprint>”) -Reason “<DescriptiveReason>” [-Server <ServerName>]

          The Thumbprint you use here will define if whether IM will work or not as this what secures the communication between Exchange and Skype. If you use the wrong certificate your Integration will fail and users wont be able to login to IM through OWA.

          11- Generate a new Certificate for Exchange IM

          IMPORTANT NOTE:

          In order for IM in OWA to work the certificate you will generate must have its common name set as mail.domain.com to match the configuration.

          Using Digicert tool on Exchange Server I will generate the CSR of the new certificate

          Click on Create CSR

          image_thumb[15]

          Choose SSL certificate type and make sure you choose Mail.domain.com as CN

          In the SANs type all of the involved servers (Skype for Business Frontends, Mailbox servers in FQDN and in Hostnames as in the screenshot below). and click on Generate

          image_thumb[16]

          • Go to your CA Server’s CertSRV URL and copy the CSR code there to generate the new certificate.
          • Import the new certificate to the current server, then export it in PFX format and import it to all the Exchange Servers you’re planning to use.

          image_thumb[18]

          • After importing the certificate I will verify that I can see the private key

          image_thumb[19]

          Click on the Details and copy the Thumbprint or from MMC right click the certificate > Properties give it a friendly name e.g. (IM) and then from Exchange Management shell you can copy the Thumbprint directly.

          Get-ExchangeCertificate | select thumbprint,friendlyName

          image_thumb[20]

          Now use the previous script to create the setting Override for OwaServer.

          Things you can change are in bold “Name, IM Servername Value, and the Thumbprint value”.

          New-SettingOverride -Name “IM Override” -Component OwaServer -Section IMSettings -Parameters @(“IMServerName=SBG-Pool01.domain.com“,”IMCertificateThumbprint= 28E4B1BA0F2FCB1535AF199F02A64EFC78367F2D“) -Reason “Configure IM”

          image_thumb[21]

          If you enter the server parameter to use a single server you can change that by using. Note that you must not use FQDN but rather only the server’s hostname.

          Get-SettingOverride | Set-SettingOverride -Server sbg-mx01,sbg-mx02

          image_thumb[22]

          This should generate an event ID 112 on Exchange servers involved in the deployment.

          clip_image001[9]_thumb

            12. Assign the newly imported certificate to IIS Exchange Back End site

            Once the certificate is in the server store, You will be able to easily find in from IIS and bind it to the Exchange Back End site.

            This is the most crucial step to get IM to work in OWA. Don’t worry about breaking up Exchange Sites or Powershell. If you have added Exchange Servers Hostnames and FQDNs in this certificate then you should be good.

            • Now Launch IIS
            • Click on Exchange Back End
            • Select Binding
            • Click on the 444 port and edit
            • Select the newly generated certificate that has the mail.domain.com as CN. (This certificate must also have all Exchange Servers hostnames and FQDNs set as SANs)

            image_thumb[23]

            image_thumb[24]

            Make sure you change the backend cert to the new on all the involved Exchange Servers.

            13. On Exchange: Restart the WebAppPool

            Restart-WebAppPool MSExchangeOWAAppPool

            image_thumb[25]

              14. Log out and sign back in to OWA to Check

              Log out of OWA and back in and check if you are able to Login to IM . It should normally sign you in automatically but in case of an error then you should see it.

              image_thumb[29]

              In case of an error you should see the following.

              image_thumb[27]

              If it works then you should see the presence

              image_thumb[28]

              15. Troubleshooting Methods

              If you follow the above steps correctly then it should work especially when applying the right certificate for your Exchange Back End IIS part however if you face an error then you should do the following steps to troubleshoot the error

              • Set the Eventlog for Instant Messaging on Exchange from Low to High

              Set-EventLogLevel -Identity “sbg-mx01\MSExchange OWA\InstantMessage” -Level High

              image_thumb[30]

              • Look in the following path for errors

              C:\Program Files\Microsoft\Exchange Server\V15\Logging\OWA\InstantMessaging

              • Check the Healthset of the OWA Instant Messaging.

              Get-ServerHealth -HealthSet OWA.Protocol.Dep -Server sbg-mx01 | Format-Table Name, AlertValue –Auto

              image_thumb[31]

              Get-MonitoringItemIdentity -Server sbg-mx01 -Identity OWA.Protocol.Dep | Format-Table Identity,ItemType,Name -Auto

                image_thumb[32]

                Ref

                https://docs.microsoft.com/en-us/exchange/plan-and-deploy/post-installation-tasks/configure-im-integration-with-owa?view=exchserver-2019

                https://docs.microsoft.com/en-us/exchange/high-availability/managed-availability/health-sets?view=exchserver-2019

                Skype for Business IM integration with Exchange 2016 OWA–Part 1

                The Story

                A good and detailed documentation is everything we need to implement any kind of project especially if it’s an integration between two different servers that perform different roles.

                And with PKI involved the complications multiply thus a good article write up is what we need.

                Previously I have tried a test lab with Skype for Business 2015/2019 IM Integration with Exchange 2016/2019 and the result was a complete failure and endless search for what’s missing to get IM to work from OWA?

                image

                ERROR

                Upon completion of the steps mentioned in Microsoft’s Official documentation and after restarting Exchange IIS or OWAAppPool you will see this when you try to login to OWA with your user

                There’s a problem with instant messaging. Please try again later.

                image

                MS Official Documentation

                In their Official documentation Microsoft says that the certificate in question must be trusted by all the servers involved meaning Skype for Business Frontend and Mailbox Servers.

                Meanwhile this is true, it still would not get the IM to login/work although it might drop the initialize event ID 112 in the event log.

                clip_image001

                Here is what MS says about the certificate.

                Exchange and Skype for Business integration requires server certificates that are trusted by all of the servers involved. The procedures in this topic assume that you already have the required certificates. For more information, see Plan to integrate Skype for Business Server 2015 and Exchange. The required IM certificate thumbprint refers to the Exchange Server certificate assigned to the IIS service.

                REF URL: https://docs.microsoft.com/en-us/exchange/plan-and-deploy/post-installation-tasks/configure-im-integration-with-owa?view=exchserver-2019#what-do-you-need-to-know-before-you-begin

                image

                Step by Step Deployment

                To do things the way that should get this to work, I will detail steps one by one so we can be sure to get the positive results we are all waiting for when dealing with Exchange and Skype for Business.

                Exchange IM URL 1: mail.domain.com

                Skype for Business Pool FQDN: SBG-Pool01.domain.com

                Autodiscover URL : Autodiscover.Domain.com

                Prerequisites

                1. For Default and Web Service Internal, Your Skype for Business Frontend Server/Pool must use a certificate that is generated from an internal CA which you can use later to generate Exchange’s IM Certificate.
                2. UCMA must be installed (Doesn’t matter if version 4 or 5) both are supposed to work with Exchange 2016.
                3. Local Certification Authority must already be deployed in the domain.

                Configuration Steps – Part 1

                1. On SfB: Set CsAuthConfiguration Autodiscover URL for Skype server to find Exchange Autodiscover
                2. On SfB: Get-CsSite to see what is the current site ID.
                3. On Exchange: Check AutodiscoverServiceInternalURI
                4. On SfB: Create new Partner
                5. On SfB: Create new Trusted Application Pool
                6. On SfB: Create new Trusted Application ID

                Configuration Steps – Part 2

                7. On Exchange: Enable OWA VD Instant Messaging
                8. On Exchange: Enable Messaging on OWA Policy
                9. On Exchange: Create Enterprise Application for Skype Pool.
                10. On Exchange: Create new SettingOverride for Skype for Business.
                11- Generate a new Certificate for Exchange IM
                12. Assign the newly imported certificate to IIS Exchange Back End site
                13. On Exchange: Restart the WebAppPool
                14. Log out and sign back in to OWA to Check
                15. Troubleshooting methods

                Prerequisites

                1- Update or Create Server Default and Web Service Internal Certificate for SfB Pool servers

                The certificate installed on the Skype for Business Pool Frontend servers must be generated from a local Certification Authority which can be trusted by Exchange Server services.

                The Certificate generated for Skype for Business pool as in the below screenshot is generated from my CA and includes the names of the servers:

                • Skype for Business Pool
                • Skype for Business Frontend FQDNs
                • Exchange Servers
                • Autodiscover FQDN
                • Lyncdiscover.domains.com
                • Lyncdiscoverinternal.domains.com
                • sip.domains.com
                • meet.domains.com
                • dialin.domain.com
                • External.domain.com

                image

                image

                2- UCMA must be installed

                On both Exchange and Skype for Business servers I already have UCMA 4.0 version installed, but if you don’t have it or have an older version then you can’t continue without it.

                image

                3- Make sure you have a Local Certification Authority deployed in your domain.

                Configuration Steps – Part 1

                1- On SfB: Set CsAuthConfiguration Autodiscover URL for Skype server to find Exchange Autodiscover

                For Skype for Business Server to find Exchange Autodiscover Service point and to be able to authenticate servers we’ll be using the below cmdlet

                This enables both servers to authenticate and share information when needed and without user’s interference.

                Set-CsOauthConfiguration -ExchangeAutodiscoverUrl https://autodiscover.domain.com/autodiscover/autodiscover.svc

                image

                image

                Ref:

                https://docs.microsoft.com/en-us/powershell/module/skype/set-csoauthconfiguration?view=skype-ps

                2- On SfB: Get-CsSite to see what is the current site ID.

                Getting a site ID will be useful for later use to setup the Trusted Application Pool.

                On Skype for Business Management shell. Type the following

                Get-CsSite

                So the Site ID is 1. I will keep this for later use

                image

                3- On Exchange: Check AutodiscoverServiceInternalURI

                Specify the AutodiscoverServiceInternalURI for internal Autodiscover service. Make sure it points to your public URL and certificate not the internal one otherwise your users will get a certificate error through Outlook and might cause IM chat not to work.

                The Cmdlet would be

                Get-ClientAccessService | Set-ClientAccessService –AutoDiscoverServiceInternalUri https://autodiscover.domain.com/autodiscover/autodiscover.xml

                image

                4- On SfB: Create new Partner Application

                On Skype for Business Server, Launch Management Shell and use this cmdlet to add Exchange as a trusted Application to the SfB topology.

                New-CsPartnerApplication -Identity Exchange -ApplicationTrustLevel Full -MetadataUrl “https://autodiscover.domain.com/autodiscover/metadata/json/1

                image

                5- On SfB: Create new Trusted Application Pool

                New-CsTrustedApplicationPool -Identity mail.domain.com -Registrar sbg-pool01.domain.com -Site 1 -RequiresReplication $false

                image

                6- On SfB: Create new Trusted Application ID

                From SfB Management Shell run the following cmdlet .

                New-CsTrustedApplication -ApplicationId OutlookWebAccess -TrustedApplicationPoolFqdn mail.domain.com -Port 5199

                image

                Finally

                clip_image001[4]

                Click on the link below for Part 2

                Skype for Business IM integration with Exchange 2016 OWA–Part 2

                an Exchange mailbox was mistakenly migrated over another user’s object used by another user

                The Story

                If you ever used Prepare moverequest command to migrate a user and forgot to use ADMT to rewrite user’s properties with the old attributes. You might have used ADMT again to rewrite the properties.

                If you use ADMT you will need to exclude all Exchange Attributes from the source since its already copied using Prepare-move request script however, in some cases some people do make mistakes and you might have came through the same mistake my colleague  have done during one of these extremely complicated Cross forest Migrations where you’d prepare a CSV files through PowerShell and names wouldn’t match Sam accounts.

                Don’t Panic

                If however, you forgot again to exclude the Exchange attributes while using ADMT then you most likely wont see the user in the Target forest which will cause to panic thinking the user is gone .. But no the user is not gone don’t panic.

                When you look for the user’s mailbox on the target forest after the move request is completed you’ll get an error reporting the user can’t be found

                image

                Solution

                To fix the problem you’ll need to change to attributes only for this migrated user. (In the target forest after user mailbox move is completed).

                The attributes are

                msExchRecipientDisplayType    1073741824
                msExchRecipientTypeDetails    128

                The wrong Attributes are as following.

                image

                You will need to fix them to look like the following

                image

                Once you apply the change you’ll need to wait for a minute or few depending on your AD replication speed.
                The problem will be then solved

                image

                Microsoft Exchange Vulnerability affects all Exchange versions

                image

                CVE-2020-0688 | Microsoft Exchange Validation Key Remote Code Execution Vulnerability

                Security Vulnerability

                Date of Publishing: February/11/2020

                Microsoft has announced a vulnerability has been found in all Exchange Server 2010 through 2019 versions, The vulnerability allows an attack to send a specially crafted request to the affected server in order to exploit it.

                When could this happen?

                A remote code execution vulnerability exists in Microsoft Exchange Server when the server fails to properly create unique keys at install time.

                Knowledge of a the validation key allows an authenticated user with a mailbox to pass arbitrary objects to be deserialized by the web application, which runs as SYSTEM.

                The security update addresses the vulnerability by correcting how Microsoft Exchange creates the keys during install.

                Affected Versions:

                • Microsoft Exchange Server 2010 Service Pack 3 Update Rollup 30
                • Microsoft Exchange Server 2013 Cumulative Update 23   
                • Microsoft Exchange Server 2016 Cumulative Update 14   
                • Microsoft Exchange Server 2016 Cumulative Update 15   
                • Microsoft Exchange Server 2019 Cumulative Update 3   
                • Microsoft Exchange Server 2019 Cumulative Update 4

                image

                Solution:

                Until now Microsoft has not provided any solution or work around to cover this vulnerability.

                Mitigations

                Microsoft has not identified any mitigating factors for this vulnerability.

                Workarounds

                Microsoft has not identified any workarounds for this vulnerability.

                NOTE:

                Keep an eye on the below link for any change

                https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0688

                How to Sync Cloud User to On-premises AD ?

                The Story:

                I have got this client who constantly keeps on making the mistake of create user from Cloud and provision them with a license in an Exchange Hybrid environment.

                Although this is not difficult to fix but it’s not the recommended approach when creating a new user especially in a Hybrid environment since Exchange on-premises won’t recognize this user and most likely will consider any incoming emails from it as spoof or spam.

                How to Create a Cloud user from Exchange On-premises?

                From Exchange on-premises ECP Admin panel you have the option to directly create user on-cloud which will also create a user object on on-premises AD.

                image

                Second option – Using Powershell

                It’s not that much different than the Web UI option but it’s just for people who prefer using PowerShell than GUI

                Enable-RemoteMailbox –Identity User –RemoteRoutingAddress user@yourTenant.mail.onmicrosoft.com

                The reason to follow those two methods is due to the need of Exchange on-premises being aware of each of those users so mail flow between Exchange on-premises and Online would not get affected and route this users mail to the wrong place or flag it as spammed or spoof …etc.

                The Real Question now is: How to Sync Cloud User to On-premises AD ?

                If by mistake we created a user on Cloud (Office 365) and we forgot to create an AD User for this account, that user might already have started using his account on Office 365 (Sharepoint, Exchange, Teams) etc.

                There also might be the intention of moving users from Cloud to On-premises Exchange in case the company wanted to decrease their spending on cloud users and in this case when Migrating a cloud user to on-premises you will get the following errors:

                image

                test3@domain.com

                Status: Failed

                test3@domain.com Skipped item details

                User status

                Data migrated:

                Migration rate:

                Last successful sync date:

                Error: MigrationPermanentException: Cannot find a recipient that has mailbox GUID ‎’03c9764e-8b8e-4f33-94d1-ef098c4de656‎’. –> Cannot find a recipient that has mailbox GUID ‎’03c9764e-8b8e-4f33-94d1-ef098c4de656‎’.

                So how do we overcome this situation since syncing a user might require you to delete the cloud user and recreate it on AD?

                Solution:

                To sync the user from the Cloud to on-premises you will need to follow these steps :

                1- Create an on-premises Mailbox where the following attributes would be matching the cloud user

                • UserPrincipalname
                • ProxyAddresses
                • SamAccountName
                • Alias

                2- The Location of the OU where the On-premises user is going to be created must be provisioned by ADConnect (Azure AD Connect)

                You can look which of these OU are provisioned by Starting AD Connect Sync Manager

                image

                By verifying the user you created in the AD is in the right OU, You can now start AD Sync from PowerShell to speed up the process.

                image

                Below, You can see the user has been successfully synchronized to the cloud without any issue.

                image

                Now we’ll see it from the portal to confirm the user is synced with AD

                image

                Depending on the Source anchor being used in ADConnect there might be a GUID conflict or not, You will get an error similar to when trying to migrate the user in the beginning however you can solve this by replacing the cloud user’s GUID (ImmutableID) with the on-premises user which will force the user to merge with the On-prem user.

                Let’s confirm in our case if the user on-cloud has a matching GUID with the one on-premises.

                From CMD or Powershell you can use the following command to get the user’s ImmutableID (ObjectGUID) .

                ldifde -f c:\Test.txt -d “cn=Test3,DC=Domain,DC=com”

                image

                Checking the notepad we just exported you can see the Immutable ID on AD for the User test3 is IkTni9mw7Ee4YefeGpz7IA==

                image

                To be able to see the user on Office 365, We need to logon to MSOL through Exchange Online powershell

                Connect to Exchange Online’s powershell using your Online ECP.

                image

                Once you click on Configure this should download an executable file that will launch PowerShell Online which allows you to use the Modern Authentication (MFA) to use PowerShell safely.

                image

                Connect-Msoluser will connect you to Office 365 and you’ll be able to get the user’s properties and see if the Immutable ID is matching to the user’s GUID.

                Once you’re connect you can use the following cmdlet to get the user’s properties.

                Get-MsolUser -UserPrincipalName test3@domain.com |fl DisplayName,ImmutableID

                image

                You can see they are matching each other, In case there’s a conflict then you can simply set the online user’s Immutable ID to match the on-premises user’s ImmutableID.

                Once done, Go and force ADConnect to sync the user and you’ll see if the problem has been resolved. The command for changing the Immutableid is as follows:

                Set-MsolUser -UserPrincipleName test3@domain.com -ImmutableID IkTni9mw7Ee4YefeGpz7IA==

                Ref:

                https://support.microsoft.com/en-us/help/2956029/migrationpermanentexception-cannot-find-a-recipient-that-has-mailbox-g

                https://docs.microsoft.com/en-us/exchange/hybrid-deployment/create-cloud-based-archive

                Move Request on Exchange 2019 (During failover) will warn you that it postponed due the move of DB

                The Issue:

                So while working on a new Exchange Migration project, I have encountered a weird issue where I could see users migration batch status complaining about being stalled due to (Big Funnel).

                The error is showing as in the below screenshot and it doesn’t occur instantly after you start the migration of the user but right after it starts.

                StalledDueToTarget_BigFunnel 68.47 MB (71,795,512 bytes) 20

                User StalledDueToTarget_BigFunnel 37.2 MB (39,003,538 bytes) 20

                User2 StalledDueToTarget_BigFunnel 14.71 MB (15,421,154 bytes) 20

                User3 StalledDueToTarget_BigFunnel 44.2 MB (46,345,009 bytes) 20

                User4 StalledDueToTarget_BigFunnel 4.647 MB (4,872,404 bytes) 20

                User5 StalledDueToTarget_BigFunnel 14.47 MB (15,169,768 bytes) 20

                User6  StalledDueToTarget_BigFunnel 171 MB (179,280,335 bytes) 20

                User7 StalledDueToTarget_BigFunnel 753.4 MB (789,980,880 bytes) 20

                User8 StalledDueToTarget_BigFunnel 18.35 MB (19,236,680 bytes) 20

                User9 StalledDueToTarget_BigFunnel 205.9 MB (215,951,208 bytes) 20

                User10 StalledDueToTarget_BigFunnel 166.2 MB (174,243,238 bytes) 20

                User11 StalledDueToTarget_BigFunnel 13.81 MB (14,481,739 bytes) 20

                User12 StalledDueToTarget_BigFunnel

                image

                Error Message

                Request ‘domain.com/CompanyUSER/Region1/User1’ (b5dbf3ff-21a1-4ec1-a29c-15b794a17386) failed.

                Error code: -2146233088

                Connection to the Content Transformation Service has failed.

                Context:

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                Operation: IMapiFxProxy.ProcessRequest

                OperationSide: Target

                b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMailbox.ExportMessages

                Operation: IMailbox.ExportMessages

                OperationSide: Source

                b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

                Flags: SkipItemValidation

                PropTags: (null)

                ——–

                >>>> Scheduled WorkItems: EnumerateFolderMessages(P:29792,R:0,S:0,C:14); EnumerateFolderMessages(P:29807,R:0,S:0,C:24,Cnt=3); WriteFolderMessages(P:0,R:0,S:0,C:686); EnumerateFolderMessages(P:30554,R:0,S:2,C:55); EnumerateFolderMessages(P:30612,R:0,S:0,C:36,Cnt=2); WriteFolderMessages(P:3,R:0,S:0,C:301); EnumerateFolderMessages(P:30975,R:0,S:1,C:21); WriteFolderMessages(P:2,R:0,S:0,C:97); EnumerateFolderMessages(P:31094,R:0,S:0,C:18,Cnt=6); EnumerateFolderMessages(P:31279,R:0,S:0,C:19)

                ————–

                The Microsoft Exchange Mailbox Replication service was unable to save changes to request.

                Request: ‘9a444721-80e2-4cf8-8c81-8a3afe3dc775’ (bbc2c66e-857e-4ba6-8462-9d66da73d400)

                Database: DB01

                Error:

                The request has been temporarily postponed because a database has failed over. The Microsoft Exchange Mailbox Replication service will attempt to continue processing the request when capacity becomes available on the new server hosting the database.

                image

                Looking at the event ID number 1114 it mentions there seems to be an issue with the request seems there might be an issue with the mailbox being moved.

                To dig deeper I am going to search some of the users reporting the same error by using their GUID

                image

                The property “DisplayName” with value “User LastName” is invalid. The value can’t contain leading or trailing whitespace.

                Solution: (For a single user)

                To resolve the problem, I am going to remove the trailing space in the end of the display name. You can safely use the below Powershell script to solve this problem however, if you don’t trust yourself or you’re not familiar much with Powershell, You can try it on a lab or a single test user for instance.

                Get-Mailbox -Identity USER | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

                image

                Solution: (For all users)

                Get-Mailbox | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

                clip_image001

                Some relevant errors you might encounter as you’re moving users to Exchange 2019

                Error code: -2146233088

                Connection to the Content Transformation Service has failed.

                Context:

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMapiFxProxy.ProcessRequest

                Operation: IMapiFxProxy.ProcessRequest

                OperationSide: Target

                eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

                OpCode: TransferBuffer

                DataLength: 31680

                ——–

                Operation: IMailbox.ExportMessages

                Operation: IMailbox.ExportMessages

                OperationSide: Source

                eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

                Flags: SkipItemValidation

                PropTags: (null)

                ——–

                >>>> Scheduled WorkItems: EnumerateFolderMessages(P:14014,R:0,S:0,C:13); EnumerateFolderMessages(P:14029,R:0,S:0,C:15,Cnt=2); WriteFolderMessages(P:1,R:0,S:0,C:132); EnumerateFolderMessages(P:14192,R:0,S:0,C:17); WriteFolderMessages(P:1,R:0,S:0,C:48); EnumerateFolderMessages(P:14259,R:0,S:0,C:12,Cnt=4); EnumerateFolderMessages(P:14320,R:0,S:1,C:15); EnumerateFolderMessages(P:14337,R:0,S:0,C:20); WriteFolderMessages(P:2,R:0,S:0,C:126); EnumerateFolderMessages(P:14485,R:0,S:0,C:30)

                Search and Delete certain Items/Folders from a Mailbox

                The Story

                During a project of Hybrid migration from Exchange on-premises to Exchange online, I was almost about to finalize the project by moving the last remaining users mailboxes however had an interesting issue to deal with where a user was failing with the following error:

                The Error after migration:

                Error: MigrationPermanentException: Mailbox dumpster size 50.87 GB (54,620,074,576 bytes) exceeds target quota 30 GB –> Mailbox dumpster size 50.87 GB exceeds target quota.

                image

                After some research it turned out that you can clean the dumpster using search-mailbox PowerShell cmdlet, Sync the user’s object with ADConnect and then continue the migration from the last failure.

                To solve the issue, Go on your Exchange on-premises and launch Exchange Management shell

                Solution applied:

                First, Let’s see the user’s dumpster and recoverable items

                Get-MailboxFolderStatistics -Identity “User” -FolderScope RecoverableItems | Format-Table Name,FolderPath,ItemsInFolder,FolderAndSubfolderSize

                image

                To Delete the dumpster only use this

                Delete dumpster only

                Search-mailbox -identity User -SearchDumpsterOnly –DeleteContent

                To delete a certain email with certain subject in the dumpster use the following:

                Get-mailbox “user”| search-mailbox –searchquery “Subject:’*'” –DeleteContent –SearchDumpsterOnly

                image

                The cmdlet will search and delete

                clip_image001

                image

                Reference:

                https://docs.microsoft.com/en-us/microsoft-365/compliance/search-for-and-delete-messagesadmin-help

                Slow Migration – Office 365

                The story:

                In office 365 when you’re working on Exchange 2010,2013, 2016 or 2019 in a hybrid environment things might look easy but in a big enterprises where Internet security is something being taken into account very seriously. It might cause many issues that you don’t expect at all.

                One of my clients whom I was doing Exchange Migration for had an issue with the Migration. The error was as follows:

                Error occurs after Office 365 Exchange online connects to Exchange on-premises 2010 mailbox server

                Error in Office 365

                         : 20.

                                                           27.04.2016 08:03:17 [DB3PR05MB0778] Transient error DataExportTransientExcep

                                                           tion has occurred. The system will retry (2/1280).

                                                           27.04.2016 08:04:53 [DB3PR05MB0778] The Microsoft Exchange Mailbox Replicati

                                                           on service ‘DB3PR05MB0778.eurprd05.prod.outlook.com’ (15.1.466.25 caps:03FFF

                                                           F) is examining the request.

                                                           27.04.2016 08:04:55 [DB3PR05MB0778] Connected to target mailbox ‘lcwonline.o

                                                           nmicrosoft.com\ec96e315-1059-4710-b358-1c4b42f3edeb (Primary)’, database ‘EU

                                                           RPR05DG049-db131′, Mailbox server ‘DB3PR05MB0778.eurprd05.prod.outlook.com’

                                                           Version 15.1 (Build 466.0).RequestExpiryTimestamp                   : 03.04.2116 07:42:38

                ObjectState                              : New

                Troubleshooting:

                To troubleshoot issues, You need to put so many things into account! The architecture of the infrastructure of where you are doing the project is very important and the need of knowing how things are working matters.

                Things that could always come in mind and handy are what you will need to start your troubleshooting:

                – Bandwidth Limitations or Performance:

                https://technet.microsoft.com/en-us/library/dn592150(v=exchg.150).aspx

                https://technet.microsoft.com/en-us/library/jj200581(v=exchg.150).aspx

                – Exchange Configuration (MRS)

                To troubleshoot the MRs, You need to know what kind of error you’re getting and to see this you can use the following powershell after you connect to Office 365 powershell.

                Get-MoveRequest {email} | Get-MoveRequestStatistics -Diagnostic -IncludeReport | Export-Clixml c:\logfile.xml

                The resultant report will reveal the error and shows you where is the exact culprit.

                – Disk Latency
                – Firewall Configuration (IPS/IDS)

                From Exchange 2016 to 2019 or 2013 to 2016 The transient error might be related to MRSProxy or at least this is the case with me 90% of the time. To resolve this issue you will need to change the MRSProxy values on the target server and depending on the error might also be the Source server too.

                SOLUTION:

                ===========

                1. Some instability was detected in communications as well as saturation by the size of the link.
                2. The procedure to increase the timeout for the service through the file MRSProxy

                File: MsExchangeMailboxReplication.exe.config

                Object / line: DataImportTimeout.

                New Value: 00:10:00

                clip_image001[4]

                New Configuration

                clip_image001[6]

                3- Unified messaging Integration between Exchange 2016 and Skype for Business

                Setting up UM

                To setup UM between Exchange and Skype for business server, the most important step is how you configure the Certificates between both servers in order for them to trust each other.

                For that you don’t have to use a public Certificate but rather an internal CA certificate that has its root certificate installed on all of the server where you intend to deploy the UM. (Exchange, S4B Servers..etc.).

                To claim this certificate, the easiest step would be to get the CSR from Skype for Business’s Deployment Wizard

                Run Deployment Wizard and click on the “Install or Update skype for business Server system”

                clip_image001

                Then click on step 3 (request, install or assign Cert)

                clip_image002

                I already have certificate deployed for S4B service but I’ll request CSR again to get one trusted certificate for both Exchange and S4B.

                I will tick only the services that matters as in the below screenshot (Server default and Web services internal) later also will be used for OWA integration with UM.

                clip_image003

                Click on Request

                clip_image004

                Click on Advanced

                clip_image005

                Next

                clip_image006

                I’ll continue next until I’ve got to the important part which is “Name and Security settings” I’ll need to tick the “Mark the certificate’s private key as exportable” since we’ll export the certificate to Exchange servers

                clip_image007

                Next I’ll add Exchange servers’s FQDNs.

                clip_image008

                clip_image009

                Click Next

                clip_image010

                clip_image011

                Here is the CMDLET

                Request-CSCertificate -New -Type Default,WebServicesInternal -CA “DC2016.moh10ly.com\moh10ly-DC2016-CA” -Country “TR” -FriendlyName “Skype for Business Server 2015 Default certificate 3/18/2016” -KeySize 2048 -PrivateKeyExportable $True -Organization “moh10ly” -OU “moh10ly” -DomainName “sip.moh10ly.com,ex2016.moh10ly.com,ex2016-2.moh10ly.com,ex2010.moh10ly.com” -AllSipDomain -Verbose -Report “C:\Users\administrator.MOH10LY\AppData\Local\Temp\2\Request-CSCertificate-[2016_03_18][11_16_35].html”

                Click Next again and mark the thumbprint for the new Cert as we’ll need to see it later to make sure it’s properly configured for the UM on Exchange.

                8BA9A2C4CD926B01C029F6B9A76D75BBEFDDE069

                clip_image012

                Click next to assign the Cert

                clip_image013

                clip_image014

                Successfully, the certificate has been assigned to the Services

                clip_image015

                The CMDLET that was applied

                Set-CSCertificate -Type Default,WebServicesInternal -Thumbprint 8BA9A2C4CD926B01C029F6B9A76D75BBEFDDE069 -Confirm:$false -Report “C:\Users\administrator.MOH10LY\AppData\Local\Temp\2\Set-CSCertificate-[2016_03_18][11_19_06].html”

                Now it’s time to export this certificate and import it to Exchange servers

                clip_image016

                I’ll find the certificate that I have created today by looking at the expiration date which is 2 years from now with the same day.

                clip_image017

                Now I’ll right click on the certificate and export it with the private key.

                clip_image018

                I’ll open Exchange EMC and import the certificate

                clip_image019

                I’ll have to put the exported cert in a shared folder and provide the path and the password for it

                clip_image020

                I’ll add the two servers below

                clip_image021

                clip_image022

                I’ll double click on the imported certificate and assign the UM services to it on each of the servers

                clip_image023

                clip_image024

                I have got the below error due to not configuring the service to use TLS instead of TCP on both servers.

                clip_image025

                To fix this I’ll go on Exchange Management shell and run the following CMDLET

                Get-UMService | Set-UMService -UMStartupMode TLS

                clip_image026

                clip_image027

                Now I’ll try to save again

                clip_image028

                clip_image029

                I’ll proceed with YES and continue to do the same to the other Server and restart the UM service on both servers

                clip_image030

                Now it’s time to create a UM Dial plan

                clip_image031

                I’ll configure the UM Dial plan according to my Skype for Business settings for users enabled for EV

                clip_image032

                To use powershell, you can use the following cmdlet

                New-UMDialPlan –Name DialplanName –UriType SIPURI –NumberOfDigitsInExtension 4 –VoIPSecurity Secured –CountryOrRegionCode 1 –AccessTelephoneNumber +9012345678

                Next, adding a gateway to the UM (NOTE: If configured incorrect, will cause the service not to start and errors with event ID (1057, 4999,1430, 1038) will appear.

                Time to configure Gateway

                clip_image033

                In the gateway I’ll add my PBX (AsteriskNow) and place my already configured UMDP

                clip_image034

                clip_image035

                When you create the dial plan, Exchange automatically creates a new UM mail policy along with it and it also generates a name that’s related to the Dial plan

                In order to see this policy, you will have to double click on the new dial plan to view it and you can also change the policy in it .. Which I’m going to apply for the length of the policy to make it shorter

                clip_image036

                Double click on the Mailbox policy and navigate to Pin Polices and change it to the length you want to allow

                clip_image037

                Configure Auto Attendant

                clip_image038

                Set the AA as how you want it to be configured and make sure you add the full E.164 format as it won’t accept otherwise.

                clip_image039

                Click Save to continue

                Now time to configure OVA (Outlook voice access)

                Subscriber Access

                If you want to configure Outlook Voice Access (OVA) , sometimes also referred to as Subscriber Access, click on the Configure button. Select Outlook Voice Access in the left hand menu and enter the telephone number you want to use to access OVA. This must be in the E.164 notation.

                clip_image040

                To do so click on Configure

                clip_image041

                To assign the new dial plan to the UM services, both on the Client Access Server (UM Call Router) as well as on the Mailbox server. In an Exchange Management Shell windows enter the following commands:

                1

                2

                Set-UMCallRouterSettings -DialPlans “Exchangelabs Dial Plan” –Server 2012E15FE04

                Get-UMService | Set-UMService -DialPlans “Exchangelabs Dial Plan”

                clip_image042

                clip_image043

                Now I’ll also change the UM call router to TLS and assign Certificate to the service then restart it

                clip_image044

                clip_image045

                Restart the services of the Call router, then associate the service with the dialplan you created.

                Set-UMCallRouterSettings -DialPlans “UMDP1” –Server EX2016

                Set-UMCallRouterSettings -DialPlans “UMDP1” –Server EX2016-2

                Configure Skype for Business Server

                To configure the UM Service to be used with Skype for Business Server. Microsoft has a script that will create and configure all necessary components. This scripts is located in the scripts directory C:\Program Files\Microsoft\Exchange Server\V15\Scripts.

                Run the following CMDLET

                CD $ExScripts

                .\ExchUCUtil.ps1

                clip_image046

                The first time you setup this script it’ll detect the Dial plan and set it up with Skype for Business Server

                clip_image047

                It will show that no setting has changed but the fact that the dial plan is showing here Not found means that there something has changed .. You’ll notice that if you run the same script again.

                clip_image048

                Let’s try it again

                Here you can see that the dial plan has been assigned to the S4B Front end server.

                clip_image049

                This script performs the following:

                • Grants Skype for Business Server permission to read Exchange UM Active Directory components, specifically, the SIP URI dial plan that was created in the first step;
                • Creates a UM IP gateway for each Skype for business Server pool that hosts users who will be enabled for Enterprise Voice;
                • Create an Exchange UM hunt group for each UM IP gateway. The hunt group pilot identifier will be the name of the dial plan associated with the corresponding UM IP gateway. The hunt group must specify the UM SIP dial plan used with the UM IP gateway.

                When the script has run you’ll see a new UM IP Gateway appear in the EAC. Since this script not only creates the UM IP Gateway but also sets the necessary permissions the UM IP Gateway was not created manually in the first step.

                clip_image050

                Next we’ll go to Skype for Business FE server and then run the OcsUmUtil.exe tool which creates the contact objects for Outlook Voice Access and for the auto attendants. This tool can be found in C:\Program Files\Common Files\Skype for Business Server 2015\Support

                clip_image051

                I’ll right click the file to run it as administrator

                clip_image052

                Click on Load Data

                clip_image053

                clip_image054

                Select the SIP dial plan and click ADD

                clip_image055

                Click OK

                Right after configuring this your Voice mail should be enabled once you enable your user for it

                After I enable user for UM and assign a valid dialplan .. Now I can see the user has got his Voice Mail option available.

                clip_image056

                Hope this was useful

                clip_image057

                —-

                UM gateway

                clip_image058

                clip_image059

                clip_image060