Exchange 2013 ECP fails with 500 Unexpected error after running Hybrid Configuration Wizard with Office 365

Symptoms

After you attempt to try Hybrid Configuration Wizard between Exchange 2013 SP1 and Exchange online (Office 365), You are unable to login to your OWA/ECP Page and instead you get an 500 unexpected error:

If you go to event viewer You might find Event ID 4 which shows the error:

Cause:

The HCW or “Hybrid Configuration Wizard” In Exchange 2013 (CU6 or 8) might cause some changes to your CAS folder in the following path and file.

“c:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\ecp\DDI”` the file name is RemoteDomains.xaml and multiply some variables which causes ECP to fail and report that error.

Error:


Current user: 'Domain.local/User'

Request for URL 'https://ex2k1301.Domain.local:444/ecp/default.aspx(https://mail.Domain.com/ecp/)' failed with the following error:

System.Web.HttpUnhandledException (0x80004005): Exception of type 'System.Web.HttpUnhandledException' was thrown. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Data.DuplicateNameException: A column named 'TargetDeliveryDomain' already belongs to this DataTable.

at System.Data.DataColumnCollection.RegisterColumnName(String name, DataColumn column)

at System.Data.DataColumnCollection.BaseAdd(DataColumn column)

at System.Data.DataColumnCollection.AddAt(Int32 index, DataColumn column)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase.CreateColumn(DataTable table, Dictionary`2 rbacMetaData)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase..ctor(Service profileBuilder)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase..ctor(String schemaFilesInstallPath, String schema)

at Microsoft.Exchange.Management.DDIService.WSListDataHandler..ctor(String schemaFilesInstallPath, String resourceName, String workflowName, DDIParameters parameters, SortOptions sortOptions)

at Microsoft.Exchange.Management.DDIService.DDIServiceHelper.GetListCommon(DDIParameters filter, SortOptions sort, Boolean forGetProgress)

--- End of inner exception stack trace ---

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)

at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

at Microsoft.Exchange.Management.ControlPanel.WebServiceReference.GetList(DDIParameters filter, SortOptions sort)

at Microsoft.Exchange.Management.ControlPanel.OrganizationCache.LoadTargetDeliveryDomain(AddValueHandler addValue, LogErrorHandler logError)

at Microsoft.Exchange.Management.ControlPanel.OrganizationCache.TryGetValue[T](String key, T& value)

at Microsoft.Exchange.Management.ControlPanel._Default.RenderMetroTopNav()

at ASP.default_aspx.__RendermainForm(HtmlTextWriter __w, Control parameterContainer)

at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)

at System.Web.UI.HtmlControls.HtmlForm.RenderChildren(HtmlTextWriter writer)

at System.Web.UI.HtmlControls.HtmlContainerControl.Render(HtmlTextWriter writer)

at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)

at ASP.default_aspx.__Render__control1(HtmlTextWriter __w, Control parameterContainer)

at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)

at System.Web.UI.Page.Render(HtmlTextWriter writer)

at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)

at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

at System.Web.UI.Page.HandleError(Exception e)

at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

at System.Web.UI.Page.ProcessRequest()

at System.Web.UI.Page.ProcessRequest(HttpContext context)

at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

at System.Web.UI.Page.HandleError(Exception e)

at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

at System.Web.UI.Page.ProcessRequest()

at System.Web.UI.Page.ProcessRequest(HttpContext context)

at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Data.DuplicateNameException: A column named 'TargetDeliveryDomain' already belongs to this DataTable.

at System.Data.DataColumnCollection.RegisterColumnName(String name, DataColumn column)

at System.Data.DataColumnCollection.BaseAdd(DataColumn column)

at System.Data.DataColumnCollection.AddAt(Int32 index, DataColumn column)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase.CreateColumn(DataTable table, Dictionary`2 rbacMetaData)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase..ctor(Service profileBuilder)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase..ctor(String schemaFilesInstallPath, String schema)

at Microsoft.Exchange.Management.DDIService.WSListDataHandler..ctor(String schemaFilesInstallPath, String resourceName, String workflowName, DDIParameters parameters, SortOptions sortOptions)

at Microsoft.Exchange.Management.DDIService.DDIServiceHelper.GetListCommon(DDIParameters filter, SortOptions sort, Boolean forGetProgress)

--- End of inner exception stack trace ---

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)

at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

at Microsoft.Exchange.Management.ControlPanel.WebServiceReference.GetList(DDIParameters filter, SortOptions sort)

at Microsoft.Exchange.Management.ControlPanel.OrganizationCache.LoadTargetDeliveryDomain(AddValueHandler addValue, LogErrorHandler logError)

at Microsoft.Exchange.Management.ControlPanel.OrganizationCache.TryGetValue[T](String key, T& value)

at Microsoft.Exchange.Management.ControlPanel._Default.RenderMetroTopNav()

at ASP.default_aspx.__RendermainForm(HtmlTextWriter __w, Control parameterContainer)

at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)

at System.Web.UI.HtmlControls.HtmlForm.RenderChildren(HtmlTextWriter writer)

at System.Web.UI.HtmlControls.HtmlContainerControl.Render(HtmlTextWriter writer)

at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)

at ASP.default_aspx.__Render__control1(HtmlTextWriter __w, Control parameterContainer)

at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)

at System.Web.UI.Page.Render(HtmlTextWriter writer)

at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)

at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)

at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)

at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

at Microsoft.Exchange.Management.ControlPanel.WebServiceReference.GetList(DDIParameters filter, SortOptions sort)

at Microsoft.Exchange.Management.ControlPanel.OrganizationCache.LoadTargetDeliveryDomain(AddValueHandler addValue, LogErrorHandler logError)

at Microsoft.Exchange.Management.ControlPanel.OrganizationCache.TryGetValue[T](String key, T& value)

at Microsoft.Exchange.Management.ControlPanel._Default.RenderMetroTopNav()

at ASP.default_aspx.__RendermainForm(HtmlTextWriter __w, Control parameterContainer)

at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)

at System.Web.UI.HtmlControls.HtmlForm.RenderChildren(HtmlTextWriter writer)

at System.Web.UI.HtmlControls.HtmlContainerControl.Render(HtmlTextWriter writer)

at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)

at ASP.default_aspx.__Render__control1(HtmlTextWriter __w, Control parameterContainer)

at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)

at System.Web.UI.Page.Render(HtmlTextWriter writer)

at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)

at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

System.Data.DuplicateNameException: A column named 'TargetDeliveryDomain' already belongs to this DataTable.

at System.Data.DataColumnCollection.RegisterColumnName(String name, DataColumn column)

at System.Data.DataColumnCollection.BaseAdd(DataColumn column)

at System.Data.DataColumnCollection.AddAt(Int32 index, DataColumn column)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase.CreateColumn(DataTable table, Dictionary`2 rbacMetaData)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase..ctor(Service profileBuilder)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase..ctor(String schemaFilesInstallPath, String schema)

at Microsoft.Exchange.Management.DDIService.WSListDataHandler..ctor(String schemaFilesInstallPath, String resourceName, String workflowName, DDIParameters parameters, SortOptions sortOptions)

at Microsoft.Exchange.Management.DDIService.DDIServiceHelper.GetListCommon(DDIParameters filter, SortOptions sort, Boolean forGetProgress)

at System.Data.DataColumnCollection.RegisterColumnName(String name, DataColumn column)

at System.Data.DataColumnCollection.BaseAdd(DataColumn column)

at System.Data.DataColumnCollection.AddAt(Int32 index, DataColumn column)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase.CreateColumn(DataTable table, Dictionary`2 rbacMetaData)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase..ctor(Service profileBuilder)

at Microsoft.Exchange.Management.DDIService.AutomatedDataHandlerBase..ctor(String schemaFilesInstallPath, String schema)

at Microsoft.Exchange.Management.DDIService.WSListDataHandler..ctor(String schemaFilesInstallPath, String resourceName, String workflowName, DDIParameters parameters, SortOptions sortOptions)

at Microsoft.Exchange.Management.DDIService.DDIServiceHelper.GetListCommon(DDIParameters filter, SortOptions sort, Boolean forGetProgress)

Flight info: Features:[[Global.DistributedKeyManagement, False],[Global.GlobalCriminalCompliance, False],[Global.MultiTenancy, False],[Global.WindowsLiveID, False],[Eac.AllowMailboxArchiveOnlyMigration, True],[Eac.AllowRemoteOnboardingMovesOnly, False],[Eac.BulkPermissionAddRemove, True],[Eac.CmdletLogging, True],[Eac.CrossPremiseMigration, False],[Eac.DevicePolicyMgmtUI, False],[Eac.DiscoveryDocIdHint, False],[Eac.DiscoveryPFSearch, False],[Eac.DiscoverySearchStats, False],[Eac.DlpFingerprint, False],[Eac.EACClientAccessRulesEnabled, False],[Eac.GeminiShell, False],[Eac.ManageMailboxAuditing, False],[Eac.ModernGroups, False],[Eac.Office365DIcon, False],[Eac.OrgIdADSeverSettings, False],[Eac.RemoteDomain, False],[Eac.UCCAuditReports, False],[Eac.UCCPermissions, False],[Eac.UnifiedAuditPolicy, False],[Eac.UnifiedComplianceCenter, False],[Eac.UnifiedPolicy, False],[Eac.UnlistedServices, False],], Flights:[], Constraints:[[LOC, EN-US],[MACHINE, EX2K1301],[MODE, ENTERPRISE],[PROCESS, W3WP],[USER, Username(EditeD)@],[USERTYPE, BUSINESS],], IsGlobalSnapshot: False

Solution:

Looking in the path I have wrote above “\Microsoft\Exchange Server\V15\ClientAccess\ecp\DDI” and opening the file that I have mentioned “RemoteDomains.xaml” you can clearly see there are incorrect format that have been duplicated 3 times.

To make sure that this was the cause, I have another server with CU8 on it so I went and checked the same file which was in the same location to find the result different.

This is the server that works in another environment and doesn’t have any issue.

So, the solution was to remove the two duplicates and correct the format of the variable line… I corrected the first line that includes

<Variable DataObjectName=”RemoteDomain” Name=”TargetDeliveryDomain” Type=”{x:Type s:Boolean}” />

And deleted the other two identical lines.. then I saved the file and closed notepad.

Next: Open IIS on the same server and go to “Application Pools” right click on the affected pools and Recycle them.. You don’t need to reset IIS as the fix should work right away.

Recycle ECP Pool

After recycling checking if the pool is reporting started or not…

Next I open ECP and it works without an issue

moh10ly

Recent Posts

Reset passwords for Active Directory Users

Reset and manage your Active Directory users' Passwords Active Directory is one of the most…

1 year ago

Finding Exchange Database hidden mailboxes. ​

Finding Exchange Database hidden mailboxes. Story:Maybe you have been in this situation before, trying to…

1 year ago

Setting up ADConnect and PTA (Password auth through) servers agents behind proxy

If you're using a Proxy server in your firewall or in your network and have…

1 year ago

Get Report of Active Directory Locked Accounts and Machine they logged in from

Story:I got some clients  that have reported some of their users being locked out and…

1 year ago

Checking and Providing Full and SendAs delegate access on O365 Exchange Online

Delegate Permissions This is a code that I have wrote recently to check if an…

1 year ago

Retrieving attachments from Exchange mailbox using python

Story: I got a request from a client who constantly gets CVs and have to…

2 years ago