You might have got a request to upgrade from ADFS 2012 R2 to Windows ADFS 2016.
This process can be complicated especially if you’ll have to migrate the Database as well and it would be more of an issue when the Database is WID (Windows Internal Database) since there’s no much documentation about troubleshooting issues involving WID on ADFS.
I have got a request from a client whom have done a migration with another consultant and obviously it was not done right.
On Windows 2016 ADFS when trying to update the ADFS SSL certificate I get the following error:
Set-AdfsSslCertificate -ThumbPrint A7etc : PS0159 : The Operation is not supported at the current Farm Behavior Level ‘1’. Raise the farm to at least version ‘2’ before retrying.
At line:1 char:1
Trying to update the database from 1 to 2,3 will also fail with the following error:
Database upgrade cannot be performed on AdfsServer.domain.com. Error: A database for the target behavior level already exists.
If you’re installing ADFS on WID (Windows Internal Database) you should run the following to get the database name/Connect String
As for Authentication, Use the Windows Authentication with the user you’re logged into if you know that’s a privileged user and can authenticate, If not try with a user which you’ve done the upgrade of ADFS with.
After authenticating, You will be able to see AdfsConfiguration , AdfsConfigurationV3 and AdfsArtifactStore. What we need to see is that AdfsConfigurationV3 has data in it and is not totally empty.
After checking and comparing the size between V1 and V3, It appeared that V3 database is empty. So what next?
Deleting the AdfsConfigurationV3 was the first thought that hit my mind however, before deleting anything I always take a snapshot of the VM since backing up the WID is more painful and takes more time than simply backing up the VM (Checkpoint, Snapshot).
I got a request to place users into Security Groups for management purposes, The client have already users active but many of those users have left the work place and still have E3 or E1 Licenses which they should not have since this is pricey licenses and backing up users details is the easiest and most cost effective way of handling this.
So, To start (Prerequisites):
The Group based licensing management is a new feature, Was introduced in 2019 and not many people know that it is there however, This feature doesn’t come for free as you know (Since it’s Microsoft) and you must have a license for it or at least have users with E3 licensing model. So the requirements are:
– Azure AD Premium P1 or Higher
– Office 365 E3 or Higher.
– EMS or Higher.
How does it work?
In order for you to get this to work you need to make sure you have planned from where you want to manage those groups and their licenses, Online? Or On-Premises?
If you’re going to do this online, then you need to create a group for each Licensing Model which represents the intended License and its users e.g. Office365-E1 is going to be created as a security group and dedicated to E1 License users.
Office365-E3 will also be created the same way and users of License type E3 will be added to it.
If you’re going to manage those groups on-premises, Then you must have ADConnect (Azure AD Sync) tool to sync those groups after creating them.
In my case I have created those groups in the following manner:
After creating those groups, You will need to sync them to Office 365 using ADConnect. To force this to sync immediately fire up Powershell on Azure Connect Server and type
Start-ADSyncSyncCycle -PolicyType delta
What If I have users already assigned with License?
If you have users already assigned licenses and want to manage them using Group Based licensing then you’re going to have to get a list of all your users with their Licenses information into a CSV file and Import those users to the groups you created base on the license they have.
I created a PowerShell that would match user’s names and based on the license mentioned in the CSV file would add them to the relevant group but first you need to export Users from Office 365.
Export Users and their license from Office 365
First of all we’ll connect to Office 365 MSOL Service using Online Powershell
So this is how my CSV look right after I exported the users, We need to do some tuning on this CSV file to clean it and get it ready for our PowerShell.
There are total of 6 columns in this folder, If for whatever reason you wanted to use the ProxyAddress to distinguish users feel free to keep them in the script but in my case I didn’t need them so I deleted the entire column.
So I will keep the following (Remove Spacing between License Type)
The Value of the License Type is usually formatted like this “TenantName: License” and in order to make this column useful I am going to remove the Tenant name from all the cells.
Find and Replace can easily remove and clean these values for you.
After cleaning the column, this is how it looks
This should be useful for us now along with the PowerShell to add the users to their relevant groups.
On Active Directory from an elevated PowerShell
Run PowerShell ISE from a privileged account and copy + paste this script in ISE,
if($user.UserPrincipalName -eq $ImportedUPN -and $License -match “EMS”)
Add-ADGroupMember -Identity $EMS -Members $Sam
Write-Host $($UPN) “User has EMS License and has been added to the Group EMS” -ForegroundColor DarkGreen -BackgroundColor White
ElseIf ($user.UserPrincipalName -eq $ImportedUPN -and $License -Contains “STANDARDPACK”)
Add-ADGroupMember -Identity $E1 -Members $Sam
Write-Host $($UPN) “User has E1 License and has been added to the Group E1” -ForegroundColor black -BackgroundColor green
ElseIf ($user.UserPrincipalName -eq $ImportedUPN -and $License -Contains “ENTERPRISEPACK”)
Add-ADGroupMember -Identity $E3 -Members $Sam
Write-Host $($UPN) “User has E3 License and has been added to the Group E3” -ForegroundColor Blue -BackgroundColor White
Enabling Group Based License from Azure Portal
After this script finishes, I can open Azure Portal
From Azure Active Directory > Licenses > All Products
I will choose the license which I want to assign to a group of which I have created on my on-premises AD
Click on the License (Office 365 E1) and choose Assign from top menu
Make sure you select assignment options and customize the license according to the products you want your group members to use then click on Users and Groups and select the relevant Group which you’ve created (In my case it’s E1-Office365)
Here, The group has been assigned
Click assign and you should be done
We will do the same for E3 Users
From now on, Removing any user from this group will revoke their license and any service connected to it, You must be very careful when removing users from this group.
Microsoft has done great job covering this thoroughly and in a great detail including Scripts to be able to do many things like grabbing users who have an inherited license from a group or manually assigned. I am writing down the references if you’re more curious into these.
I have got this client who constantly keeps on making the mistake of create user from Cloud and provision them with a license in an Exchange Hybrid environment.
Although this is not difficult to fix but it’s not the recommended approach when creating a new user especially in a Hybrid environment since Exchange on-premises won’t recognize this user and most likely will consider any incoming emails from it as spoof or spam.
How to Create a Cloud user from Exchange On-premises?
From Exchange on-premises ECP Admin panel you have the option to directly create user on-cloud which will also create a user object on on-premises AD.
Second option – Using Powershell
It’s not that much different than the Web UI option but it’s just for people who prefer using PowerShell than GUI
The reason to follow those two methods is due to the need of Exchange on-premises being aware of each of those users so mail flow between Exchange on-premises and Online would not get affected and route this users mail to the wrong place or flag it as spammed or spoof …etc.
The Real Question now is: How to Sync Cloud User to On-premises AD ?
If by mistake we created a user on Cloud (Office 365) and we forgot to create an AD User for this account, that user might already have started using his account on Office 365 (Sharepoint, Exchange, Teams) etc.
There also might be the intention of moving users from Cloud to On-premises Exchange in case the company wanted to decrease their spending on cloud users and in this case when Migrating a cloud user to on-premises you will get the following errors:
Error: MigrationPermanentException: Cannot find a recipient that has mailbox GUID ’03c9764e-8b8e-4f33-94d1-ef098c4de656’. –> Cannot find a recipient that has mailbox GUID ’03c9764e-8b8e-4f33-94d1-ef098c4de656’.
So how do we overcome this situation since syncing a user might require you to delete the cloud user and recreate it on AD?
To sync the user from the Cloud to on-premises you will need to follow these steps :
1- Create an on-premises Mailbox where the following attributes would be matching the cloud user
2- The Location of the OU where the On-premises user is going to be created must be provisioned by ADConnect (Azure AD Connect)
You can look which of these OU are provisioned by Starting AD Connect Sync Manager
By verifying the user you created in the AD is in the right OU, You can now start AD Sync from PowerShell to speed up the process.
Below, You can see the user has been successfully synchronized to the cloud without any issue.
Now we’ll see it from the portal to confirm the user is synced with AD
Depending on the Source anchor being used in ADConnect there might be a GUID conflict or not, You will get an error similar to when trying to migrate the user in the beginning however you can solve this by replacing the cloud user’s GUID (ImmutableID) with the on-premises user which will force the user to merge with the On-prem user.
Let’s confirm in our case if the user on-cloud has a matching GUID with the one on-premises.
From CMD or Powershell you can use the following command to get the user’s ImmutableID (ObjectGUID) .
During a project of Hybrid migration from Exchange on-premises to Exchange online, I was almost about to finalize the project by moving the last remaining users mailboxes however had an interesting issue to deal with where a user was failing with the following error:
I had a project few weeks ago where my client wanted to install Skype for Business 2019 but had installed Lync before and removed the server without doing proper decommissioning which kept dirty records in AD database and had to be removed manually in order to make a new clean installation of Skype for Business 2019
To do so:
There are two days of doing so, One is using ADSIEdit and ADUC to remove Computer Objects and Users related attributes and Security Groups.
I normally would prefer PowerShell but since we can demonstrate both ways for people who like to work with GUI
Starting with GUI
Removing Legacy Lync server from the AD Schema
Using a domain or enterprise admin
Access to the ADSIEdit.
Goal of removing Legacy Lync server from your AD environment.
Preparing AD schema and domain for a new deployment after you improperly deleted Lync Servers without uninstalling them.
Cleaning Users’ Lync related attributes for the new deployment.
Step#1: Remove permissions
This step removes the original Lync permissions from the active director.
Open Active Directory Users and Computers
Right click on your top level domain being cleaned and select Properties
From the Properties windows, select the Security tab.
Remove all security users titled RTC*
These are usually
In Exchange MRSPROXY.SVC FAILED BECAUSE NO SERVICE WAS LISTENING ON THE SPECIFIED ENDPOINT. THE REMOTE SERVER RETURNED AN ERROR: (404) NOT FOUND
Exchange 2010 / 2013
You get an error when you’re trying to setup Hybrid configuration between your Exchange On-premises or Online.
After I had one issue like this I did some research and used Fiddler / Wireshark to check for traffic I noticed that the traffic on the server is not encrypted and testing the Migration Server Availability was reporting that the MRS service was not listening on the supposed port which is 443.
This problem may occur if the ExchangeGUID property of the Exchange Online MailUser object does not match the ExchangeGUID property of the on-premises mailbox. To successfully move a mailbox, the value of the ExchangeGUID property in the Exchange Online mailbox and in the associated on-premises remote mailbox must match.
In this case the solution was pretty easy, but still you’ll have to make a hard choice of choosing to place Exchange behind a load balancing with SSL Offloading on or not.
In my case I had to turn off the SSL Offloading on the Load balancer and that alone was enough to get this working.
Make sure that SSL Offloading is disabled on OWA/OA and Load balancer if there’s one.
In office 365 when you’re working on Exchange 2010,2013, 2016 or 2019 in a hybrid environment things might look easy but in a big enterprises where Internet security is something being taken into account very seriously. It might cause many issues that you don’t expect at all.
One of my clients whom I was doing Exchange Migration for had an issue with the Migration. The error was as follows:
Error occurs after Office 365 Exchange online connects to Exchange on-premises 2010 mailbox server
RPR05DG049-db131′, Mailbox server ‘DB3PR05MB0778.eurprd05.prod.outlook.com’
Version 15.1 (Build 466.0).RequestExpiryTimestamp : 03.04.2116 07:42:38
ObjectState : New
To troubleshoot issues, You need to put so many things into account! The architecture of the infrastructure of where you are doing the project is very important and the need of knowing how things are working matters.
Things that could always come in mind and handy are what you will need to start your troubleshooting:
The resultant report will reveal the error and shows you where is the exact culprit.
– Disk Latency
– Firewall Configuration (IPS/IDS)
From Exchange 2016 to 2019 or 2013 to 2016 The transient error might be related to MRSProxy or at least this is the case with me 90% of the time. To resolve this issue you will need to change the MRSProxy values on the target server and depending on the error might also be the Source server too.
1. Some instability was detected in communications as well as saturation by the size of the link.
2. The procedure to increase the timeout for the service through the file MRSProxy
Skype for Business Edge server deployment and Hybrid integration with Skype for Business Online
In the last Skype for Business post I have upgraded my Lync 2013 to Skype for Business (Click here to go to that post). in this article I am going to install Edge server for Skype for Business to the same Lync Environment where I have done the Upgrade to Skype for Business.
Configuring Edge Server
In order to configure Skype 4 Business Edge, we’ll have to change the Netbios to give it the name of our Domain but we won’t join it to the domain.
Edge Server must have 2 NICs, one Local NIC will point out to the Front end server but must not have Default gateway so traffic can only flow through the DMZ out to the internet and back in. but still it must be able to ping to the FE from Edge and vice versa.
The DMZ network could have a single DMZ address (Public Address to be pointing to) or three DMZ addresses for public IP addresses with standard https ports.
Edit the Edge server’s host file to include Lync FE and DC’s IP addresses and Hostname
Microsoft .Net Framework 3.5
Now I will go back to Skype for Business FE server, I’ll launch the topology builder and add new Edge server
I will add the first Edge pool which contains of a single Edge server
Next, you will have to choose if you want to enable federation with partners or other service providers …e.g. (Google)
I am intending to use a single Public IP address with a different ports (nonstandard) since this is a lab. For production use it’s recommended to have 3 public IP addresses, One is for Access Edge, AV and WebConf services.
Next I will choose the last option which says that the Edge pool is translated by NAT. I will configure my firewall to NAT ports to the Edge’s DMZ IP addresses from the Public so I am choosing this option.
This is the FQDN’s the default configuration .. It’ll only use a single FQDN for all services if you’re going to use a single public IP address with a different ports.
When you use a single IP address with a different ports, the Access Edge port will normally change to 5061 (Not 443 like in the _sip._tls.domain.com) SRV record which will cause failure if you forgot to change this port to match the one in your Topology’s Access Edge settings.
Next I’ll have to enter my Edge server’s Local IP address.
Next I will be asked to enter the DMZ’s IP address which the wizard calls (Private External IP address)
Here I am going to place the NAT IP address which is my Public IP address.
Next I’ll have to choose which Lync FE pool will be used as the next hop to the Edge pool. In this case I’ll be choosing my main pool since the second is only for resilience purpose.
Then I’ll associate the mediation pool for Edge server for external media traffic. I can assign both in this case.
Now I’ll click on Finish and right click on the Site name’s properties to enable the SIP federation and XMPP federation then Publish the topology.
Now I will setup Azure Active Directory Sync on my DC server in order to sync the required users for the test purpose.
My domain is adeo.local so I want to change the UPN for users to match the synced domain. (Adeo-office365.ga) and moh10ly.com
Installing Azure Active Directory Sync
Now I will install the prerequisites which consist of the following
Net framework 4.5.2 is required for AADS but it’s already installed on my server
Next I will install Microsoft Online Service Sign in assistant
Next I will install Azure AD Module
Finally Azure AD Sync
Before moving forward, I’ll have to go to the Office 365 portal and activate DirSync
Then use a global admin credentials from O365.
Adding the forest using an enterprise admin user account
Due to the fact that my domain adeo-office365.ga’s public dns host doesn’t have SRV configuration because it’s hosted by the famous free domain service (Freenom) so I’ll have to add my original domain moh10ly.com as Lync (S4B) requires SRV records to point to the on-premises lync.
I will only sync one OU, so I will untick the Sync now box and click on Finish
I will go to the following path
“C:\Program Files\Microsoft Azure AD Sync\UIShell” and create a shortcut for the GUI application of AADS on the desktop
“C:\Program Files\Microsoft Azure AD Sync\UIShell\miisclient.exe”
To get this GUI app to work, you will have to sign out of your account and sign back in as your username will be added to the local administrators and have the authority to open it
Log off, log back in
Next I will go to the connectors tab and double click on the ADDS connector (Adeo.local)
I will go to the Configure Directory Partitions and under Credentials I’ll choose “Alternate credentials for this directory partition” then enter my on-premises AD Enterprise admin credentials
I’ll click on Containers
I’ll untick the DC=Adeo,Dc=Local box and only choose Dirsync OU then click OK and apply
Before I start syncing my AD , I will go to Skype for Business Server and add my domain moh10ly.com as a SIP domain
Next I am going to change the FQDN of the SIP access edge for public domain to moh10ly.com and the default port for the Access Edge to 443 and publish the topology
I needed to finally check if all my FE servers are replicating. So then I can move to Edge server to install Lync components
On the Edge server, I’ll use ISO for Skype 4 business to install the setup
First thing I’ll install the local Configuration Store
I’ll click on Run and then I’ll be asked to import the configuration file which I’ll must export from Lync FE (Skype 4 b FE) server
In this case, I’ll go to Lync FE and open Lync Management shell and enter the following Cmdlet
Export-CsConfiguration -FileName c:\top.zip
This cmdlet will export a file to the root C drive . I’ll copy this file to the edge server.
I’ll click next to continue, this should start installing the local store
Next I’ll request a certificate for Internal NIC For edge server
I’ll take the CSR (Certificate sign request) code and get a certificate from my local CA
I’ll open MMC and add Certificates console and import the PKCS certificate
After importing the certificate I’ll assign it to the internal NIC by clicking on Assign to the Edge Internal
Once we assign the certfiicate to the internal edge. The replication service for Edge and FE will start working
Now I’ll import my Public Certificate to Edge Server’s DMZ NIC
I already imported my public certificate, now I’ll go to the S4B wizard and assign it there
Unlike IN lync 2013 when you Click on Start service in the Wizard all services start on their own but on Skype for business you ‘ll have to start the services manually by yourself.
So Instead I used the service console to start the services.
Now I’ll go back to the FE And enable remote connectivity to Skype for Business from outside and make sure that replication works fine by checking the Topology or from cmdlet
After the replication is finished, I was able to login remotely with my Skype for Business on-premises accounts.
Setting up Hybrid integration with Skype online for Business (O365)
In order to allow Hybrid environment to function properly, we’ll have to federate our Skype for Business on-premises’s Edge server as Microsoft says below
Federation allows users in your on-premises deployment to communicate with Office 365 users in your organization. To configure federation, run the following cmdlets in the Skype for Business Server Management Shell:
Next cmdlet will create a new public federated provider for skype for business online.. However it already exists by default as in the below snapshot but just to avoid any issues I will delete the default provider from control panel and recreate it again.
I’ll delete the hosted provider “Skype for Business Online”
I’ll try the cmdlet again after deleting the provider ..
According to Microsoft’s configuration of the Public DNS, you will have to configure only the SRV records to point to your edge server however, running a simple wireshark on your Skype for business client machine you can notice the following:
Microsoft Lync / Skype client first requires the Lyncdiscover / Lyncdiscoverinternal record in order to see where the user is located… then gets redirected to webdir.online.lync.com which is the Cname value to the Lyncdiscover Cname in the public DNS and tries to login the user through Login.microsoftonline.com then finds no user there and logs in using the SRV eventually in the end as in the below snapshot which I’ve used Wireshark for to monitor the DNS traffic that the Lync Client requests upon login request.
What have me confused here is that Microsoft says only SRV records must be pointing to your On-premises Lync/Skype for Business Edge server.. So you must enter something else other than SIP.domain.com (Which in normal cases might be the common name of your Edge certificate) for the value of the SRV Record since the SIP.domain.com and Lyncdiscover.domain.com must be pointing to Office 365.
I tried using the Public IP address of my Edge server just to check if my on-premises user will connect without any issue however I did have an issue with the Certificate saying “There was a problem verifying the certificate from the server”.
Luckily the Public certificate that I had on my edge server had multiple SANs (Subject Alternative Names) and one of them was WAC.moh10ly.com which I was intending to use for the WAC Server (Office Web Apps Server) and then I created an A record on my public DNS WAC.moh10ly.com that points to my Edge server’s Public IP address…. although the Wac.moh10ly.com is not a common name but it worked and I was able to federate with Office 365 users and was able to move users from on-premises to office 365 and back to on-premises as demonstrated later in the article.
“When creating DNS SRV records for hybrid deployments, the records, _sipfederationtls._tcp.<domain> and _sip._tls.<domain>, should point to the on-premises Access Proxy.”
Now I have changed all the SRV records to direct to the new A record
And finally deleted the A sip record and created a new CNAME record that points to sipdir.online.lync.com
I have already a user synced from my local AD to the cloud (office 365) that’s not enabled for Skype for business on-premises .. Once this user is synced and have been assigned a license it should be directly enabled for Skype for Business Online and I should be able to sign in to it without any issue.
In order for both users (homed online and On-premises) to see eachother’s presence the synced user must be enabled on the On-premises Server before moved to the cloud or else the presence and M will fail.
Time to test, I was able to sign in to the Online homed user (admin) and now I’ll be adding the on-premises homed user to the list to check the presence, IM ..etc
Here I added the user admin to my other account Mohammed.hamada and vice versa.
The Presence appears to be working fine for user homed on-premises as it shows when I changed it to “busy, be right back..etc” on the cloud user’s Client however the Office 365 homed user’s presence takes time to change on the on-premises user’s list and the IM doesn’t seem to work properly as messages sometimes doesn’t go through and fail.
Sending a message from the on-premises User (Mohammed Hamada) to (ADMIN)
Now sending an IM from Admin to Mohammed Hamada
To make sure that the issue is not within my on-premises server, I will use a different Skype for Business online account and see if IM work both ways.
This is my other user.. The presence information seems to work properly and now I’ll test the IM
IM between my On-premises and another user on another Office 365 tenant seems to be working fine back and forth as in the below snapshots so the issue might be related to Office 365 tenant which I am using for this test (could be related to trial version)
I am going to open a case with MS and see why this issue happens since my on-premises work fine with other tenants.
Now It’s time to move users from and to cloud and on-premises to check how easy, flexible or hard this process is.
I currently have 2 users, one on cloud and one synced and homed online (Office 365)
In order to move users, you can go to Users tab after the hybrid config is finished and find the user you want to move then click on Actions and chose to move the users to the Skype for Business Online as in the below snapshot
Before you move the user to Office 365, you must assign license to the user or else the move will fail.
You can move the user back from Office 365 to your on-premises Skype for Business server with the same process exactly except that you’ll have to choose which pool you need to move the user to.
Checking where the user is hosted from Skype for business Management shell
The Hosting Provider will show you where the user is working from now.