In office 365 when you’re working on Exchange 2010,2013, 2016 or 2019 in a hybrid environment things might look easy but in a big enterprises where Internet security is something being taken into account very seriously. It might cause many issues that you don’t expect at all.
One of my clients whom I was doing Exchange Migration for had an issue with the Migration. The error was as follows:
Error occurs after Office 365 Exchange online connects to Exchange on-premises 2010 mailbox server
RPR05DG049-db131′, Mailbox server ‘DB3PR05MB0778.eurprd05.prod.outlook.com’
Version 15.1 (Build 466.0).RequestExpiryTimestamp : 03.04.2116 07:42:38
ObjectState : New
Troubleshooting:
To troubleshoot issues, You need to put so many things into account! The architecture of the infrastructure of where you are doing the project is very important and the need of knowing how things are working matters.
Things that could always come in mind and handy are what you will need to start your troubleshooting:
To troubleshoot the MRs, You need to know what kind of error you’re getting and to see this you can use the following powershell after you connect to Office 365 powershell.
The resultant report will reveal the error and shows you where is the exact culprit.
– Disk Latency
– Firewall Configuration (IPS/IDS)
From Exchange 2016 to 2019 or 2013 to 2016 The transient error might be related to MRSProxy or at least this is the case with me 90% of the time. To resolve this issue you will need to change the MRSProxy values on the target server and depending on the error might also be the Source server too.
SOLUTION:
===========
1. Some instability was detected in communications as well as saturation by the size of the link.
2. The procedure to increase the timeout for the service through the file MRSProxy
OABGen encountered an error while generating the changes.oab file for version 2 and 3 differential downloads of address list
RESOLUTION 1
Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
322756 How to back up and restore the registry in Windows
To resolve this issue, follow these steps on the server that is running Exchange Server 2003:
Click Start, click Run, type regedit in the Open box, and then click OK.
Locate and then right-click the following registry subkey:
Type OAL post full if diff fails to name the new value.
Right-click OAL post full if diff fails, and then click Modify.
In the Value data box, type 1, and then click OK.
Exit Registry Editor.
Dismount and then mount the Public Folder Store again. To dismount and then mount the public folder store, follow these steps:
Start Exchange System Manager.
Expand Servers, expand the server that you want, expand Storage_Group_Name, and then right-click Public Folder Store.
Note If administrative groups are defined, follow these steps:
Expand Administrative Groups.
Expand Administrative_Group_Name.
Expand Servers.
Expand the server that you want.
Expand Storage_Group_Name.
Right-click Public Folder Store.
Click Dismount Store, and then click Yes to continue.
Right-click Public Folder Store, click Mount Store, and then click OK.
A new parent Legacy Exchange DN container value ‘/o=HEMA/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients’ was found during generation of the differential update file for offline address list ‘\Global Address List’. This will force clients using this offline address list to do a full download of the offline address list.
– \Default Offline Address List
Resolution 2:
If the first resolution didn’t work, try to disable version 2 and 3 OAB, then update again and see if you get any errors.
I have previously done a Hybrid integration with Office 365 with my Exchange 2010 server and enabled Archiving online when I migrated my user to Exchange online but then I finished my demo and decided to bring the user back on-premises. Now I have deployed Exchange 2013 and wanted to migrate the same user to Exchange 2013 from 2010 but the migration request fails with the following message.
6/7/2015 1:23:24 PM [EXCH2K13] ” created move request.
6/7/2015 1:23:57 PM [EXCH2K13] The Microsoft Exchange Mailbox Replication service ‘EXCH2K13.demotesas.local’ (15.0.1076.6 caps:1FFF) is examining the request.
6/7/2015 1:23:59 PM [EXCH2K13] Connected to target mailbox ‘b6ee5dd7-beab-45a0-9933-8e926a694de3 (Primary)’, database ‘Mailbox Database 0439787427’, Mailbox server ‘EXCH2K13.demotesas.local’ Version 15.0 (Build 1076.0).
6/7/2015 1:23:59 PM [EXCH2K13] Connected to source mailbox ‘b6ee5dd7-beab-45a0-9933-8e926a694de3 (Primary)’, database ‘Database1’, Mailbox server ‘EXCH01.demotesas.local’ Version 14.3 (Build 174.0).
6/7/2015 1:23:59 PM [EXCH2K13] Cleared sync state for request b6ee5dd7-beab-45a0-9933-8e926a694de3 due to ‘CleanupOrphanedMailbox’.
6/7/2015 1:23:59 PM [EXCH2K13] Mailbox signature will not be preserved for mailbox ‘b6ee5dd7-beab-45a0-9933-8e926a694de3 (Primary)’. Outlook clients will need to restart to access the moved mailbox.
6/7/2015 1:25:10 PM [EXCH2K13] Mailbox ‘Mohammed JA. Hamada’ was loaded from domain controller ‘ad.demotesas.local’.
6/7/2015 1:25:18 PM [EXCH2K13] Fatal error UpdateMovedMailboxPermanentException has occurred.
On Exchange 2010, I launched Exchange Management shell and ran the following cmdlet which will show any attribute that has arch in it for the user Mohammed
Get-mailbox User | fl arch*
Since there’s no archive mailbox then the archive domain is invalid and I don’t even own it anymore as it has expired a while ago.
Resolution:
I will try to remove the archive domain object from the user’s properties using the following cmdlet
Set-mailbox mailboxname -ArchiveDomain $null
Using the above cmdlet seems to fail due to this property being administered by Exchange server so it’ll have to be removed manually.
I will open the user’s attribute and delete the value and try to continue the migration again.
Performance counters for the Client access role is not installed
To solve the problem
Open the Exchange Management Shell
Run the following cmd: add-pssnapin Microsoft.Exchange.Management.PowerShell.Setup
Run the following cmd: new-perfcounters –definitionfilename “C:\Program Files\Microsoft\Exchange Server\V14\Setup\Perf\RpcClientAccessPerformanceCounters.xml”
By running these cmds we will install the Performance Counters needed for the RPC Client Access Service. Once installed the error won’t be displayed anymore.
In a very interesting situation that I came through I had an environment with two DCs and Exchange 2010 that I had previously setup for Hybrid integration with Office 365 for demonstration with a trial subscription but I haven’t removed the integration after I finished my test and the trial expired and the tenant was deleted.
Next I intended to upgrade my existing Exchange 2010 to Exchange 2013 and setup coexistence between them however, I have stumbled in the step of preparation of AD schema for Exchange 2013. While trying to prepare the schema I got the following error
Welcome to Microsoft Exchange Server 2013 Cumulative Update 8 Unattended Setup
Copying Files…
File copy complete. Setup will now collect additional information needed for
installation.
Performing Microsoft Exchange Server Prerequisite Check
Prerequisite Analysis FAILED
A hybrid deployment with Office 365 has been detected. Please ensure that you are running setup with the /TenantOrganizationConfig switch. To use the TenantOrganizationConfig switch you must first connect to your Exchange Online tenant via PowerShell and execute the following command: “Get-OrganizationConfig | Export-Clixml -Path MyTenantOrganizationConfig.XML”. Once the XML file has been generated, run setup with the TenantOrganizationConfig switch as follows “/TenantOrganizationConfig MyTenantOrganizationConfig.XML”.
If you continue to see this this message then it indicates that either the XML file specified is corrupt, or you are attempting to upgrade your on-premises Exchange installation to a build that isn’t compatible with the Exchange version of your Office 365 tenant. Your Office 365 tenant must be upgraded to a compatible version of Exchange before upgrading your on-premises Exchange installation. For
)/ms.exch.setupreadiness.DidTenantSettingCreatedAnException.aspx The Exchange Server setup operation didn’t complete. More details can be found in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
The Office 365 Hybrid setup was still there in my Exchange Console and since I couldn’t follow MS’s recommended steps to connect to O365 tenant and get the XML file then I had to do things manually.
First I connected to the EMC and removed all the instances that were created during the Exchange Hybrid Wizard Configuration
1- Removing Organization Relationships
2- Removing Federation Trust
3- Removing Remote Domains
4- Removing Accepted Domains
5- Removing Send and Receive Connectors
6- Lastly the Hybrid Configuration object…
Since remove-hybridconfiguration cmdlet is not supported to remove the hybrid configuration object from AD then we have no choice but to use ADSIEDIT tool to do so.
I will navigate to Configuration > Services > Microsoft Exchange > First Organization > Delete “CN=Hybrid Configuration”
Restart MSExchangeServicehost
Now I will try again to prepare AD schema for Exchange 2013 but I got a different error
Extending Active Directory schema FAILED
The following error was generated when “$error.Clear);
RoleSchemaPrefix + “schema0.ldf”)” was run: “Microsoft.Exchange.Configuration.Tasks.TaskException: There was an error while running ‘ldifde.exe’ to import the schema file ‘C:\Windows\Temp\ExchangeSetup\Setup\Data\PostExchange2003_schema0.ldf’. The error code is: 8224. More details can be found in the error file: ‘C:\Users\Administrator.DEMOTESAS\AppData\Local\Temp\2\ldif.err’at Microsoft.Exchange.Configuration.Tasks.Task.ThrowError(Exception exception, ErrorCategory errorCategory, Object target, String helpUrl)at Microsoft.Exchange.Management.Deployment.InstallExchangeSchema.ImportSchem
cessRecord()at Microsoft.Exchange.Configuration.Tasks.Task.<ProcessRecord>b__b()at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)”.The Exchange Server setup operation didn’t complete. More details can be found
in ExchangeSetup.log located in the <SystemDrive>:\ExchangeSetupLogs folder.
Checking the ldif.err file mentioned in the error above it seems that Exchange is complaining because the changes of the AD schema is not being replicated to the other AD partners which is true since I have another additional DC that’s turned off.
After turning on the other DC we’ll see what happens
The other DC had another issue as I have turned it off for long time and it was not syncing due to expired Tomb stone life so I had to fix this issue as well and I have published it in a different article.
Please click here to see how the replication issue was fixed.
After DC migration and changing in the PDC and Schema master role server to the new DC and shut down the old DC for test. On Exchange 2010 server you might get the following error
An error caused a change in the current set of Active Directory settings. Restart The Exchange Management Console.
Exchange Console
Current deployment
Exchange 2010
New DC 2012 R2 with another Additional DC installed newly.
Two DC 2008R2 but have been shut down for testing.
Problem:
After you shutdown or demote the old PDC or Schema master Demote Domain Controller role, Microsoft Exchange Management Console fails to retrieve any Exchange information with error message “An error caused a change in the current set of Active Directory Server settings. Restart Exchange Management console.”
Cause
Microsoft Exchange management console caches the data in the user’s profile for quick access, So whenever you try to open EMC from an existing Exchange admin profile you will get the same error.
Resolution:
Navigate to the following folder and delete the Exchange Management Console file.
In an environment where one DC exist after adding Windows 2012 R2 Servers as additional servers, Exchange 2007 doesn’t show the new servers although they also hold GC.
Research:
To locate the problem you should search the event ID (2080) which shows the populated DCs and the permissions allowed on Exchange servers
In the below screenshot, the SACL right was not provided to the new DCs due to GPO problem.
After checking sites, Replication, all is healthy and no issue with it.
3 servers (Two 2012 servers) and one DC 2003 Server
Exchange 2010 SP3 servers.
Reason:
The Default Domain Controllers Policy was not linked to the Domain Controllers OU.
Resolution:
After Linking the Domain Controllers OU to the Default Controllers policy, the SACL permission was provided without any issue.
Now Exchange is reporting healthy and can read the new DCs which allow us to demote the old DCs