Category Archives: Exchange 2019

Move Request on Exchange 2019 (During failover) will warn you that it postponed due the move of DB

The Issue:

So while working on a new Exchange Migration project, I have encountered a weird issue where I could see users migration batch status complaining about being stalled due to (Big Funnel).

The error is showing as in the below screenshot and it doesn’t occur instantly after you start the migration of the user but right after it starts.

StalledDueToTarget_BigFunnel 68.47 MB (71,795,512 bytes) 20

User StalledDueToTarget_BigFunnel 37.2 MB (39,003,538 bytes) 20

User2 StalledDueToTarget_BigFunnel 14.71 MB (15,421,154 bytes) 20

User3 StalledDueToTarget_BigFunnel 44.2 MB (46,345,009 bytes) 20

User4 StalledDueToTarget_BigFunnel 4.647 MB (4,872,404 bytes) 20

User5 StalledDueToTarget_BigFunnel 14.47 MB (15,169,768 bytes) 20

User6  StalledDueToTarget_BigFunnel 171 MB (179,280,335 bytes) 20

User7 StalledDueToTarget_BigFunnel 753.4 MB (789,980,880 bytes) 20

User8 StalledDueToTarget_BigFunnel 18.35 MB (19,236,680 bytes) 20

User9 StalledDueToTarget_BigFunnel 205.9 MB (215,951,208 bytes) 20

User10 StalledDueToTarget_BigFunnel 166.2 MB (174,243,238 bytes) 20

User11 StalledDueToTarget_BigFunnel 13.81 MB (14,481,739 bytes) 20

User12 StalledDueToTarget_BigFunnel

image

Error Message

Request ‘domain.com/CompanyUSER/Region1/User1’ (b5dbf3ff-21a1-4ec1-a29c-15b794a17386) failed.

Error code: -2146233088

Connection to the Content Transformation Service has failed.

Context:

——–

Operation: IMapiFxProxy.ProcessRequest

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMapiFxProxy.ProcessRequest

Operation: IMapiFxProxy.ProcessRequest

OperationSide: Target

b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMailbox.ExportMessages

Operation: IMailbox.ExportMessages

OperationSide: Source

b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

Flags: SkipItemValidation

PropTags: (null)

——–

>>>> Scheduled WorkItems: EnumerateFolderMessages(P:29792,R:0,S:0,C:14); EnumerateFolderMessages(P:29807,R:0,S:0,C:24,Cnt=3); WriteFolderMessages(P:0,R:0,S:0,C:686); EnumerateFolderMessages(P:30554,R:0,S:2,C:55); EnumerateFolderMessages(P:30612,R:0,S:0,C:36,Cnt=2); WriteFolderMessages(P:3,R:0,S:0,C:301); EnumerateFolderMessages(P:30975,R:0,S:1,C:21); WriteFolderMessages(P:2,R:0,S:0,C:97); EnumerateFolderMessages(P:31094,R:0,S:0,C:18,Cnt=6); EnumerateFolderMessages(P:31279,R:0,S:0,C:19)

————–

The Microsoft Exchange Mailbox Replication service was unable to save changes to request.

Request: ‘9a444721-80e2-4cf8-8c81-8a3afe3dc775’ (bbc2c66e-857e-4ba6-8462-9d66da73d400)

Database: DB01

Error:

The request has been temporarily postponed because a database has failed over. The Microsoft Exchange Mailbox Replication service will attempt to continue processing the request when capacity becomes available on the new server hosting the database.

image

Looking at the event ID number 1114 it mentions there seems to be an issue with the request seems there might be an issue with the mailbox being moved.

To dig deeper I am going to search some of the users reporting the same error by using their GUID

image

The property “DisplayName” with value “User LastName” is invalid. The value can’t contain leading or trailing whitespace.

Solution: (For a single user)

To resolve the problem, I am going to remove the trailing space in the end of the display name. You can safely use the below Powershell script to solve this problem however, if you don’t trust yourself or you’re not familiar much with Powershell, You can try it on a lab or a single test user for instance.

Get-Mailbox -Identity USER | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

image

Solution: (For all users)

Get-Mailbox | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

clip_image001

Some relevant errors you might encounter as you’re moving users to Exchange 2019

Error code: -2146233088

Connection to the Content Transformation Service has failed.

Context:

——–

Operation: IMapiFxProxy.ProcessRequest

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMapiFxProxy.ProcessRequest

Operation: IMapiFxProxy.ProcessRequest

OperationSide: Target

eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMailbox.ExportMessages

Operation: IMailbox.ExportMessages

OperationSide: Source

eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

Flags: SkipItemValidation

PropTags: (null)

——–

>>>> Scheduled WorkItems: EnumerateFolderMessages(P:14014,R:0,S:0,C:13); EnumerateFolderMessages(P:14029,R:0,S:0,C:15,Cnt=2); WriteFolderMessages(P:1,R:0,S:0,C:132); EnumerateFolderMessages(P:14192,R:0,S:0,C:17); WriteFolderMessages(P:1,R:0,S:0,C:48); EnumerateFolderMessages(P:14259,R:0,S:0,C:12,Cnt=4); EnumerateFolderMessages(P:14320,R:0,S:1,C:15); EnumerateFolderMessages(P:14337,R:0,S:0,C:20); WriteFolderMessages(P:2,R:0,S:0,C:126); EnumerateFolderMessages(P:14485,R:0,S:0,C:30)

Search and Delete certain Items/Folders from a Mailbox

The Story

During a project of Hybrid migration from Exchange on-premises to Exchange online, I was almost about to finalize the project by moving the last remaining users mailboxes however had an interesting issue to deal with where a user was failing with the following error:

The Error after migration:

Error: MigrationPermanentException: Mailbox dumpster size 50.87 GB (54,620,074,576 bytes) exceeds target quota 30 GB –> Mailbox dumpster size 50.87 GB exceeds target quota.

image

After some research it turned out that you can clean the dumpster using search-mailbox PowerShell cmdlet, Sync the user’s object with ADConnect and then continue the migration from the last failure.

To solve the issue, Go on your Exchange on-premises and launch Exchange Management shell

Solution applied:

First, Let’s see the user’s dumpster and recoverable items

Get-MailboxFolderStatistics -Identity “User” -FolderScope RecoverableItems | Format-Table Name,FolderPath,ItemsInFolder,FolderAndSubfolderSize

image

To Delete the dumpster only use this

Delete dumpster only

Search-mailbox -identity User -SearchDumpsterOnly –DeleteContent

To delete a certain email with certain subject in the dumpster use the following:

Get-mailbox “user”| search-mailbox –searchquery “Subject:’*'” –DeleteContent –SearchDumpsterOnly

image

The cmdlet will search and delete

clip_image001

image

Reference:

https://docs.microsoft.com/en-us/microsoft-365/compliance/search-for-and-delete-messagesadmin-help

In Exchange MRSPROXY.SVC FAILED BECAUSE NO SERVICE WAS LISTENING ON THE SPECIFIED ENDPOINT.

Symptoms

In Exchange MRSPROXY.SVC FAILED BECAUSE NO SERVICE WAS LISTENING ON THE SPECIFIED ENDPOINT. THE REMOTE SERVER RETURNED AN ERROR: (404) NOT FOUND

Exchange 2010 / 2013

You get an error when you’re trying to setup Hybrid configuration between your Exchange On-premises or Online.

After I had one issue like this I did some research and used Fiddler / Wireshark to check for traffic I noticed that the traffic on the server is not encrypted and testing the Migration Server Availability was reporting that the MRS service was not listening on the supposed port which is 443.

CAUSE


This problem may occur if the ExchangeGUID property of the Exchange Online MailUser object does not match the ExchangeGUID property of the on-premises mailbox. To successfully move a mailbox, the value of the ExchangeGUID property in the Exchange Online mailbox and in the associated on-premises remote mailbox must match.

image

In this case the solution was pretty easy, but still you’ll have to make a hard choice of choosing to place Exchange behind a load balancing with SSL Offloading on or not.

In my case I had to turn off the SSL Offloading on the Load balancer and that alone was enough to get this working.

Resolution:

Make sure that SSL Offloading is disabled on OWA/OA and Load balancer if there’s one.

Other resolutions:

https://support.microsoft.com/en-us/kb/3065754

Slow Migration – Office 365

The story:

In office 365 when you’re working on Exchange 2010,2013, 2016 or 2019 in a hybrid environment things might look easy but in a big enterprises where Internet security is something being taken into account very seriously. It might cause many issues that you don’t expect at all.

One of my clients whom I was doing Exchange Migration for had an issue with the Migration. The error was as follows:

Error occurs after Office 365 Exchange online connects to Exchange on-premises 2010 mailbox server

Error in Office 365

         : 20.

                                           27.04.2016 08:03:17 [DB3PR05MB0778] Transient error DataExportTransientExcep

                                           tion has occurred. The system will retry (2/1280).

                                           27.04.2016 08:04:53 [DB3PR05MB0778] The Microsoft Exchange Mailbox Replicati

                                           on service ‘DB3PR05MB0778.eurprd05.prod.outlook.com’ (15.1.466.25 caps:03FFF

                                           F) is examining the request.

                                           27.04.2016 08:04:55 [DB3PR05MB0778] Connected to target mailbox ‘lcwonline.o

                                           nmicrosoft.com\ec96e315-1059-4710-b358-1c4b42f3edeb (Primary)’, database ‘EU

                                           RPR05DG049-db131′, Mailbox server ‘DB3PR05MB0778.eurprd05.prod.outlook.com’

                                           Version 15.1 (Build 466.0).RequestExpiryTimestamp                   : 03.04.2116 07:42:38

ObjectState                              : New

Troubleshooting:

To troubleshoot issues, You need to put so many things into account! The architecture of the infrastructure of where you are doing the project is very important and the need of knowing how things are working matters.

Things that could always come in mind and handy are what you will need to start your troubleshooting:

– Bandwidth Limitations or Performance:

https://technet.microsoft.com/en-us/library/dn592150(v=exchg.150).aspx

https://technet.microsoft.com/en-us/library/jj200581(v=exchg.150).aspx

– Exchange Configuration (MRS)

To troubleshoot the MRs, You need to know what kind of error you’re getting and to see this you can use the following powershell after you connect to Office 365 powershell.

Get-MoveRequest {email} | Get-MoveRequestStatistics -Diagnostic -IncludeReport | Export-Clixml c:\logfile.xml

The resultant report will reveal the error and shows you where is the exact culprit.

– Disk Latency
– Firewall Configuration (IPS/IDS)

From Exchange 2016 to 2019 or 2013 to 2016 The transient error might be related to MRSProxy or at least this is the case with me 90% of the time. To resolve this issue you will need to change the MRSProxy values on the target server and depending on the error might also be the Source server too.

SOLUTION:

===========

1. Some instability was detected in communications as well as saturation by the size of the link.
2. The procedure to increase the timeout for the service through the file MRSProxy

File: MsExchangeMailboxReplication.exe.config

Object / line: DataImportTimeout.

New Value: 00:10:00

clip_image001[4]

New Configuration

clip_image001[6]