Notes from the field: August 2017

Hi All,

First things first, a Big Red WARNING: The following content applies to the specific context encountered in one of my client's O365 tenant. If you think you should follow it, please be extra careful that you understand the whats and whys of this post. It worked in my environment for my situation. It may not work in yours, or you may have to take additional steps to avoid data loss. If not careful, you MAY LOOSE DATA.

I do NOT accept responsibility for any data loss. Proceed at your own risk.

I recently worked on an issue where the O365 admin portal reported DirSync errors...,

... but when I looked at the detailed list of objects in error, the list was blank:

While I don't know what exactly happened, here is what I think the customer has done, as it makes the most sense:

Some users left the company. Their external contacts kept sending e-mail, and these e-mails had to be delivered somewhere so that business wasn't affected.
The client's admin decided to delete their mailbox and AD object. Created a distribution list, assigned the departed user's e-mail address, and populated the distribution list with recipients assigned to handle incoming e-mails for the departed users. No idea what happened to the departed users' mailbox content, and it is totally irrelevant for the purpose of this blog.
AADConnect deleted the O365 objects by means of AD sync. Consequently O365 moved objects into the Dumpster.
AADConnect tried to synch the new distribution groups, however an email address conflict was detected between the objects in the dumpster. The sync succeeded but with errors. For more details see the Duplicate Attribute Resiliency feature described in the first bullet point in the References section at the end of this post.

From this moment on, no matter what was done, the DirSync error persisted.

To fix it, I had to:

Purge the old accounts from the O365 dumpster.
Delete the distribution groups that were in error from O365 by moving them out of the AADConnect sync scope and having it do an AD sync.
Recreate the distribution groups by moving them back into the AADConnect sync scope and having it do an AD sync.

Here comes the long story. You'll need the Windows Azure Active Directory Module for Windows PowerShell.

Step 1: Connect to the O365 tenant with a tenant admin account:

Connect-MsolService

Step 2: List the objects in error:

Get-MsolDirSyncProvisioningError

The command gives us a couple of useful tips:

- a DisplayName

- the conflicting properties in the ProvisioningErrors field

I also searched recipients for any conflicting addresses - no joy.

I checked the O365 dumpster for any deleted accounts. Bingo! Some entries had property values also found in the objects in error.

Step 3: List the objects in the dumpster, along with their proxy addresses. Note that I used the -Wrap parameter in the FT (Format-Table) command in case the list of proxy addresses is too long, to avoid truncation (type it all on one line).

Get-MsolUser -ReturnDeletedUsers | ft DisplayName,ProxyAddresses -Wrap

In my case the tenant deleted some user accounts for staff who moved on, and replaced them with distribution groups populated with other users who then got the deleted user's e-mails, assigning the departed user's address to the distribution group. Therefore the same e-mail address was on the new distribution group, as well as on the deleted object in the dumpster which wasn't yet purged.

Hint: Purging the dumpster and re-synching AD doesn't solve the problem. It looks like it's a bug in the O365 identity management system. We still need to purge the dumpster but some additional steps are needed, so read on.

Step 4: To remove objects from the dumpster, use the command (type it all on one line):

Get-MsolUser -ReturnDeletedUsers | Remove-MsolUser -RemoveFromRecycleBin -Force

The command has no output:

Once the dumpster is purged and a new AD sync is initiated (or wait for the next cycle), you would think it's all sorted. Unfortunately it's not so. Once you do a re-sync and issue the Get-MsolDirSyncProvisioningError command, you'll still see that the conflicting objects are still in error.

No, the sync process isn't (yet) that evolved. A few more steps to go to get it sorted.

Step 5: Delete the object that is reported by Get-MsolDirSyncProvisioningError as being in conflict:

In the on-premises AD, move the objects in conflict outside the scope of the AADConnect sync scope (you didn't set the scope to the entire domain, right?)
Initiate an AD sync or wait for it to happen on schedule.
Confirm that the objects have been deleted in the O365 tenant.
Check the dumpster again. If the objects were moved there, then purge them as detailed at point 4 above.

Step 6: Run the Get-MsolDirSyncProvisioningError command again. It should return nothing:

Step 7: Move the object back into the AADConnect sync scope in the local AD and initiate a sync.

Step 8: Confirm that the objects have been re-created in O365.

Step 9: Confirm that the Get-MsolDirSyncProvisioningError command returns no more entries, and that's also reflected in the portal:

That's it.

Please note that in my case I only had to deal with Distribution Groups. Deleting and recreating them in O365 did not involve any user data, contrary to mailbox objects. That would have complicated things. Please see my warning at the top of this post.

References:

https://docs.microsoft.com/en-us/azure/active-directory/connect/active-directory-aadconnectsyncservice-duplicate-attribute-resiliency
https://docs.microsoft.com/en-us/powershell/module/msonline/get-msoluser?view=azureadps-1.0 (focus on the ReturnDeletedUsers property)
https://docs.microsoft.com/en-us/powershell/module/msonline/remove-msoluser?view=azureadps-1.0 (focus on the RemoveFromRecycleBin property)

Happy error hunting!

Hi There,

This is another one of many posts about AAD Connect failing to synchronise passwords, this time with some additional clarifications.

The error:

The context:

The admin configured his own account in the AD-DS connector in the management agent.
The admin changed his password over time. AD sync broke.
A new service account has been created, dedicated for AD access, and configured the connector to use it to correct the above problem. AD sync started working again.

What didn't happen is permission wasn't granted for the new account to synchronise passwords. User properties were synchronised, but not password hashes.

There were informational 611 events in the Application event log by Directory Synchronization:

The relevant bit: RPC Error 8453 : Replication access was denied. There was an error calling _IDL_DRSGetNCChanges

This is due to the fact that the connector account did not have the following permissions - see https://msdn.microsoft.com/en-us/library/azure/dn757602.aspx:

Replicating Directory Changes
Replicating Directory Changes All

These permissions are granted on the domain root.

Open Active Directory Users and Computers and in the View menu enable Advanced Features.

Right-click on the domain name, Properties, Security. Add the account and grant the permissions:

Wait for the next synchronization cycle or kick one off manually. Passwords should now sync successfully.

One last thing: the account you have to give permissions to is NOT what's configured in the Microsoft Azure AD Sync service:

Instead, the permissions have to be granted to the account configured on the AD connector:

References:

Happy syncing!

Notes from the field

Wednesday, 23 August 2017

NUMA Optimization on HP Proliant Gen9 Servers

O365 IDM Bug: DirSync Errors but No Objects in Error

Friday, 11 August 2017

I Moved to a New ISP and Mail Stopped Flowing - What Now?

Password Sync - No Recent Synchronization