Category Archives: Exchange 2019

Upgrading Exchange Online PowerShell to V2 Module

Managing Exchange Online

If you have Exchange Online and your users are MFA enabled then you most likely will be using Exchange Online’ s ECP (Exchange Control Panel or Admin Center) to connect to Exchange Online PowerShell through the Hybrid Windows since this is the only supported way with MFA.

image

Clicking on Configure would install the PowerShell Module of Exchange Online which looks like the below screenshot.

image

New PowerShell with MFA support

If you have launched Exchange Online PowerShell today then you most likely have noticed there’s a red line stating the possibility to try the new (Preview Version) of Exchange PowerShell V2 .

Microsoft has recently released a new version of Exchange Online PowerShell Module which supports MFA and can be run directly from your computer without the need to login to Exchange Online Admin Center and download any files from there.  Check details in this link

As stated in the article, the Module is just in preview so it has some known and maybe unknown bugs as well.

How to Install it?

The installation process is pretty straightforward, Launch Windows PowerShel as an Administrator (It’s required for the installation).

Run these 4 cmdlets

Set-ExecutionPolicy RemoteSigned

Install-Module PowershellGet –Force

Update-Module PowershellGet

Install-Module -Name ExchangeOnlineManagement

image

You might get a warning that the Module you’re about to install is from an Untrusted Repository, Accept it by typing Y and hit enter

Type the following cmdlet to ensure that Exchange Online Management module is installed

Import-Module ExchangeOnlineManagement; Get-Module ExchangeOnlineManagement

image

Connecting to Exchange Online

To connect to Exchange Online, Run the following cmdlet along with the new parameter –EnableErrorReporting which gives the ability to record all the cmdlets that you have run along with errors generated as well.

Connect-ExchangeOnline -EnableErrorReporting -LogDirectoryPath e:\ExchOnlineLogs.txt -LogLevel All

image

image

After connecting, I am going to try and run two commands the Old Cmdlets and New Cmdlet and see the difference between them:

  1. Get-CASMailbox -ResultSize 10
  2. Get-EXOCasMailbox -ResultSize 10

image

The new Cmdlet has much more details, although it says that it runs faster but it took few seconds more than the old one to run (Probably first time).

image

After you run those two Cmdlets, There will be two files generated in the log directory which we have pointed the parameter to save files to.

The CSV files have details about the two cmdlets and the HTTP Method they are utilizing in order to connect along the Request and response latency.

imageimage

This new version seems to be extremely useful esp in environments where such deep details are needed for troubleshooting issues.

Stay tuned for more

Reference:

https://docs.microsoft.com/en-us/powershell/exchange/exchange-online/exchange-online-powershell-v2/exchange-online-powershell-v2?view=exchange-ps

How to Sync Cloud User to On-premises AD ?

The Story:

I have got this client who constantly keeps on making the mistake of create user from Cloud and provision them with a license in an Exchange Hybrid environment.

Although this is not difficult to fix but it’s not the recommended approach when creating a new user especially in a Hybrid environment since Exchange on-premises won’t recognize this user and most likely will consider any incoming emails from it as spoof or spam.

How to Create a Cloud user from Exchange On-premises?

From Exchange on-premises ECP Admin panel you have the option to directly create user on-cloud which will also create a user object on on-premises AD.

image

Second option – Using Powershell

It’s not that much different than the Web UI option but it’s just for people who prefer using PowerShell than GUI

Enable-RemoteMailbox –Identity User –RemoteRoutingAddress user@yourTenant.mail.onmicrosoft.com

The reason to follow those two methods is due to the need of Exchange on-premises being aware of each of those users so mail flow between Exchange on-premises and Online would not get affected and route this users mail to the wrong place or flag it as spammed or spoof …etc.

The Real Question now is: How to Sync Cloud User to On-premises AD ?

If by mistake we created a user on Cloud (Office 365) and we forgot to create an AD User for this account, that user might already have started using his account on Office 365 (Sharepoint, Exchange, Teams) etc.

There also might be the intention of moving users from Cloud to On-premises Exchange in case the company wanted to decrease their spending on cloud users and in this case when Migrating a cloud user to on-premises you will get the following errors:

image

test3@domain.com

Status: Failed

test3@domain.com Skipped item details

User status

Data migrated:

Migration rate:

Last successful sync date:

Error: MigrationPermanentException: Cannot find a recipient that has mailbox GUID ‎’03c9764e-8b8e-4f33-94d1-ef098c4de656‎’. –> Cannot find a recipient that has mailbox GUID ‎’03c9764e-8b8e-4f33-94d1-ef098c4de656‎’.

So how do we overcome this situation since syncing a user might require you to delete the cloud user and recreate it on AD?

Solution:

To sync the user from the Cloud to on-premises you will need to follow these steps :

1- Create an on-premises Mailbox where the following attributes would be matching the cloud user

  • UserPrincipalname
  • ProxyAddresses
  • SamAccountName
  • Alias

2- The Location of the OU where the On-premises user is going to be created must be provisioned by ADConnect (Azure AD Connect)

You can look which of these OU are provisioned by Starting AD Connect Sync Manager

image

By verifying the user you created in the AD is in the right OU, You can now start AD Sync from PowerShell to speed up the process.

image

Below, You can see the user has been successfully synchronized to the cloud without any issue.

image

Now we’ll see it from the portal to confirm the user is synced with AD

image

Depending on the Source anchor being used in ADConnect there might be a GUID conflict or not, You will get an error similar to when trying to migrate the user in the beginning however you can solve this by replacing the cloud user’s GUID (ImmutableID) with the on-premises user which will force the user to merge with the On-prem user.

Let’s confirm in our case if the user on-cloud has a matching GUID with the one on-premises.

From CMD or Powershell you can use the following command to get the user’s ImmutableID (ObjectGUID) .

ldifde -f c:\Test.txt -d “cn=Test3,DC=Domain,DC=com”

image

Checking the notepad we just exported you can see the Immutable ID on AD for the User test3 is IkTni9mw7Ee4YefeGpz7IA==

image

To be able to see the user on Office 365, We need to logon to MSOL through Exchange Online powershell

Connect to Exchange Online’s powershell using your Online ECP.

image

Once you click on Configure this should download an executable file that will launch PowerShell Online which allows you to use the Modern Authentication (MFA) to use PowerShell safely.

image

Connect-Msoluser will connect you to Office 365 and you’ll be able to get the user’s properties and see if the Immutable ID is matching to the user’s GUID.

Once you’re connect you can use the following cmdlet to get the user’s properties.

Get-MsolUser -UserPrincipalName test3@domain.com |fl DisplayName,ImmutableID

image

You can see they are matching each other, In case there’s a conflict then you can simply set the online user’s Immutable ID to the on-premises user and that should solve the problem.

Ref:

https://support.microsoft.com/en-us/help/2956029/migrationpermanentexception-cannot-find-a-recipient-that-has-mailbox-g

https://docs.microsoft.com/en-us/exchange/hybrid-deployment/create-cloud-based-archive

Move Request on Exchange 2019 (During failover) will warn you that it postponed due the move of DB

The Issue:

So while working on a new Exchange Migration project, I have encountered a weird issue where I could see users migration batch status complaining about being stalled due to (Big Funnel).

The error is showing as in the below screenshot and it doesn’t occur instantly after you start the migration of the user but right after it starts.

StalledDueToTarget_BigFunnel 68.47 MB (71,795,512 bytes) 20

User StalledDueToTarget_BigFunnel 37.2 MB (39,003,538 bytes) 20

User2 StalledDueToTarget_BigFunnel 14.71 MB (15,421,154 bytes) 20

User3 StalledDueToTarget_BigFunnel 44.2 MB (46,345,009 bytes) 20

User4 StalledDueToTarget_BigFunnel 4.647 MB (4,872,404 bytes) 20

User5 StalledDueToTarget_BigFunnel 14.47 MB (15,169,768 bytes) 20

User6  StalledDueToTarget_BigFunnel 171 MB (179,280,335 bytes) 20

User7 StalledDueToTarget_BigFunnel 753.4 MB (789,980,880 bytes) 20

User8 StalledDueToTarget_BigFunnel 18.35 MB (19,236,680 bytes) 20

User9 StalledDueToTarget_BigFunnel 205.9 MB (215,951,208 bytes) 20

User10 StalledDueToTarget_BigFunnel 166.2 MB (174,243,238 bytes) 20

User11 StalledDueToTarget_BigFunnel 13.81 MB (14,481,739 bytes) 20

User12 StalledDueToTarget_BigFunnel

image

Error Message

Request ‘domain.com/CompanyUSER/Region1/User1’ (b5dbf3ff-21a1-4ec1-a29c-15b794a17386) failed.

Error code: -2146233088

Connection to the Content Transformation Service has failed.

Context:

——–

Operation: IMapiFxProxy.ProcessRequest

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMapiFxProxy.ProcessRequest

Operation: IMapiFxProxy.ProcessRequest

OperationSide: Target

b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMailbox.ExportMessages

Operation: IMailbox.ExportMessages

OperationSide: Source

b5dbf3ff-21a1-4ec1-a29c-15b794a17386 (Primary)

Flags: SkipItemValidation

PropTags: (null)

——–

>>>> Scheduled WorkItems: EnumerateFolderMessages(P:29792,R:0,S:0,C:14); EnumerateFolderMessages(P:29807,R:0,S:0,C:24,Cnt=3); WriteFolderMessages(P:0,R:0,S:0,C:686); EnumerateFolderMessages(P:30554,R:0,S:2,C:55); EnumerateFolderMessages(P:30612,R:0,S:0,C:36,Cnt=2); WriteFolderMessages(P:3,R:0,S:0,C:301); EnumerateFolderMessages(P:30975,R:0,S:1,C:21); WriteFolderMessages(P:2,R:0,S:0,C:97); EnumerateFolderMessages(P:31094,R:0,S:0,C:18,Cnt=6); EnumerateFolderMessages(P:31279,R:0,S:0,C:19)

————–

The Microsoft Exchange Mailbox Replication service was unable to save changes to request.

Request: ‘9a444721-80e2-4cf8-8c81-8a3afe3dc775’ (bbc2c66e-857e-4ba6-8462-9d66da73d400)

Database: DB01

Error:

The request has been temporarily postponed because a database has failed over. The Microsoft Exchange Mailbox Replication service will attempt to continue processing the request when capacity becomes available on the new server hosting the database.

image

Looking at the event ID number 1114 it mentions there seems to be an issue with the request seems there might be an issue with the mailbox being moved.

To dig deeper I am going to search some of the users reporting the same error by using their GUID

image

The property “DisplayName” with value “User LastName” is invalid. The value can’t contain leading or trailing whitespace.

Solution: (For a single user)

To resolve the problem, I am going to remove the trailing space in the end of the display name. You can safely use the below Powershell script to solve this problem however, if you don’t trust yourself or you’re not familiar much with Powershell, You can try it on a lab or a single test user for instance.

Get-Mailbox -Identity USER | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

image

Solution: (For all users)

Get-Mailbox | Foreach { Set-Mailbox -Identity $_.Identity -DisplayName $_.DisplayName.Trim() }

clip_image001

Some relevant errors you might encounter as you’re moving users to Exchange 2019

Error code: -2146233088

Connection to the Content Transformation Service has failed.

Context:

——–

Operation: IMapiFxProxy.ProcessRequest

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMapiFxProxy.ProcessRequest

Operation: IMapiFxProxy.ProcessRequest

OperationSide: Target

eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

OpCode: TransferBuffer

DataLength: 31680

——–

Operation: IMailbox.ExportMessages

Operation: IMailbox.ExportMessages

OperationSide: Source

eecb073e-e694-4bbc-8652-54dc05a351ea (Primary)

Flags: SkipItemValidation

PropTags: (null)

——–

>>>> Scheduled WorkItems: EnumerateFolderMessages(P:14014,R:0,S:0,C:13); EnumerateFolderMessages(P:14029,R:0,S:0,C:15,Cnt=2); WriteFolderMessages(P:1,R:0,S:0,C:132); EnumerateFolderMessages(P:14192,R:0,S:0,C:17); WriteFolderMessages(P:1,R:0,S:0,C:48); EnumerateFolderMessages(P:14259,R:0,S:0,C:12,Cnt=4); EnumerateFolderMessages(P:14320,R:0,S:1,C:15); EnumerateFolderMessages(P:14337,R:0,S:0,C:20); WriteFolderMessages(P:2,R:0,S:0,C:126); EnumerateFolderMessages(P:14485,R:0,S:0,C:30)

Search and Delete certain Items/Folders from a Mailbox

The Story

During a project of Hybrid migration from Exchange on-premises to Exchange online, I was almost about to finalize the project by moving the last remaining users mailboxes however had an interesting issue to deal with where a user was failing with the following error:

The Error after migration:

Error: MigrationPermanentException: Mailbox dumpster size 50.87 GB (54,620,074,576 bytes) exceeds target quota 30 GB –> Mailbox dumpster size 50.87 GB exceeds target quota.

image

After some research it turned out that you can clean the dumpster using search-mailbox PowerShell cmdlet, Sync the user’s object with ADConnect and then continue the migration from the last failure.

To solve the issue, Go on your Exchange on-premises and launch Exchange Management shell

Solution applied:

First, Let’s see the user’s dumpster and recoverable items

Get-MailboxFolderStatistics -Identity “User” -FolderScope RecoverableItems | Format-Table Name,FolderPath,ItemsInFolder,FolderAndSubfolderSize

image

To Delete the dumpster only use this

Delete dumpster only

Search-mailbox -identity User -SearchDumpsterOnly –DeleteContent

To delete a certain email with certain subject in the dumpster use the following:

Get-mailbox “user”| search-mailbox –searchquery “Subject:’*'” –DeleteContent –SearchDumpsterOnly

image

The cmdlet will search and delete

clip_image001

image

Reference:

https://docs.microsoft.com/en-us/microsoft-365/compliance/search-for-and-delete-messagesadmin-help

In Exchange MRSPROXY.SVC FAILED BECAUSE NO SERVICE WAS LISTENING ON THE SPECIFIED ENDPOINT.

Symptoms

In Exchange MRSPROXY.SVC FAILED BECAUSE NO SERVICE WAS LISTENING ON THE SPECIFIED ENDPOINT. THE REMOTE SERVER RETURNED AN ERROR: (404) NOT FOUND

Exchange 2010 / 2013

You get an error when you’re trying to setup Hybrid configuration between your Exchange On-premises or Online.

After I had one issue like this I did some research and used Fiddler / Wireshark to check for traffic I noticed that the traffic on the server is not encrypted and testing the Migration Server Availability was reporting that the MRS service was not listening on the supposed port which is 443.

CAUSE


This problem may occur if the ExchangeGUID property of the Exchange Online MailUser object does not match the ExchangeGUID property of the on-premises mailbox. To successfully move a mailbox, the value of the ExchangeGUID property in the Exchange Online mailbox and in the associated on-premises remote mailbox must match.

image

In this case the solution was pretty easy, but still you’ll have to make a hard choice of choosing to place Exchange behind a load balancing with SSL Offloading on or not.

In my case I had to turn off the SSL Offloading on the Load balancer and that alone was enough to get this working.

Resolution:

Make sure that SSL Offloading is disabled on OWA/OA and Load balancer if there’s one.

Other resolutions:

https://support.microsoft.com/en-us/kb/3065754

Slow Migration – Office 365

The story:

In office 365 when you’re working on Exchange 2010,2013, 2016 or 2019 in a hybrid environment things might look easy but in a big enterprises where Internet security is something being taken into account very seriously. It might cause many issues that you don’t expect at all.

One of my clients whom I was doing Exchange Migration for had an issue with the Migration. The error was as follows:

Error occurs after Office 365 Exchange online connects to Exchange on-premises 2010 mailbox server

Error in Office 365

         : 20.

                                           27.04.2016 08:03:17 [DB3PR05MB0778] Transient error DataExportTransientExcep

                                           tion has occurred. The system will retry (2/1280).

                                           27.04.2016 08:04:53 [DB3PR05MB0778] The Microsoft Exchange Mailbox Replicati

                                           on service ‘DB3PR05MB0778.eurprd05.prod.outlook.com’ (15.1.466.25 caps:03FFF

                                           F) is examining the request.

                                           27.04.2016 08:04:55 [DB3PR05MB0778] Connected to target mailbox ‘lcwonline.o

                                           nmicrosoft.com\ec96e315-1059-4710-b358-1c4b42f3edeb (Primary)’, database ‘EU

                                           RPR05DG049-db131′, Mailbox server ‘DB3PR05MB0778.eurprd05.prod.outlook.com’

                                           Version 15.1 (Build 466.0).RequestExpiryTimestamp                   : 03.04.2116 07:42:38

ObjectState                              : New

Troubleshooting:

To troubleshoot issues, You need to put so many things into account! The architecture of the infrastructure of where you are doing the project is very important and the need of knowing how things are working matters.

Things that could always come in mind and handy are what you will need to start your troubleshooting:

– Bandwidth Limitations or Performance:

https://technet.microsoft.com/en-us/library/dn592150(v=exchg.150).aspx

https://technet.microsoft.com/en-us/library/jj200581(v=exchg.150).aspx

– Exchange Configuration (MRS)

To troubleshoot the MRs, You need to know what kind of error you’re getting and to see this you can use the following powershell after you connect to Office 365 powershell.

Get-MoveRequest {email} | Get-MoveRequestStatistics -Diagnostic -IncludeReport | Export-Clixml c:\logfile.xml

The resultant report will reveal the error and shows you where is the exact culprit.

– Disk Latency
– Firewall Configuration (IPS/IDS)

From Exchange 2016 to 2019 or 2013 to 2016 The transient error might be related to MRSProxy or at least this is the case with me 90% of the time. To resolve this issue you will need to change the MRSProxy values on the target server and depending on the error might also be the Source server too.

SOLUTION:

===========

1. Some instability was detected in communications as well as saturation by the size of the link.
2. The procedure to increase the timeout for the service through the file MRSProxy

File: MsExchangeMailboxReplication.exe.config

Object / line: DataImportTimeout.

New Value: 00:10:00

clip_image001[4]

New Configuration

clip_image001[6]