Error Sending Email to Subdomains from O365

Suppose you have an on-premise Exchange and SharePoint server. You configure SharePoint to allow adding content by e-mail (see here). Your main domain is yourdomain.com and the SharePoint domain is sp.yourdomain.com. A sample SharePoint address may look like documents@sp.yourdomain.com.

You set up a dedicated Send connector and scope it for the sp.yourdomain.com domain, using the SharePoint server as smart host.

It works just fine.

Then you enable Hybrid Exchange, move a couple of mailboxes into the cloud, and you switch your MX record(s) to Exchange Online Protection.

In this scenario users with mailboxes in the cloud may fail to submit documents to SharePoint via e-mail and they'll get the following error:

LED=550 5.4.101 Proxy session setup failed on Frontend with '554 5.4.4 SMTPSEND.DNS.Non-ExistentDomain; nonexistent domain'

Here is what fixed it for me, and hopefully it will fix it for you too:

1. In the O365 Exchange Admin Center, under Mail flow / Accepted domains:

• Change your parent domain (yourdomain.com) from Authoritative to Internal Relay and enable Accept mail for all subdomains (see here - a tad old, due for an update).

2. Under Mail Flow / Connectors:

• Edit the connector which routes mail from O365 to your organization and click Next until the Edit Connector page is displayed.

• Add the *.yourdomain.com domain to the list.

• Click next and work your way through the wizard. Don't change anything else. Save the settings. In the process you'll need to provide a valid e-mail address to validate the connection. Provide one in the yourdomain.com domain.

3. On the on-premise Exchange server configure the following:

• Create a new accepted domain of type "External Relay" for sp.yourdomain.com:

• There should already be a dedicated Send connector. If not, create a new dedicated Send connector for the sp.yourdomain.com address space:

• On the Send connector, ensure that the SharePoint server is listed as smart host. Add it if it isn't.

E-mail will now start flowing as expected.

Your environment  might be slightly different, but the principle still applies.

SIDHistory and Exchange Resource Forests

The Environment

I have been recently involved in a business consolidation project where two businesses joined forces and formed a new entity. Inevitably, it had a strong migration component.

My client stipulated that assuming the new identity in the early stages is critical and it has to be reflected in communications. Therefore e-mail was the first component to be migrated. The plan was to let users log on and access resources such as file, print, application etc. as normal, with their LEGACY\oldlogon account, while sending/receiving e-mail should happen under their NEW\newlogon identity.

To achieve this first milestone, the new AD forest has been created and Exchange 2013 CU8 has been installed and configured.

The timeline was unclear as to when will users start to log on with their NEW\newlogon accounts, and in relation to that, whether users will start logging on to the new forest *before* or *after* other resources will have been migrated.

For this reason user accounts have been migrated along with the SIDHistory attribute in a “disabled” state, and their mailboxes moved cross-forest as Linked Mailboxes. Effectively an Account/Resource forest topology has been built. For more on Account/Resource forests see https://technet.microsoft.com/en-us/library/aa998031(v=exchg.150).aspx.

The Issue

All worked well, except delegation. Delegation was flaky at best, users regularly losing access, and admins unsuccessfully trying to re-establish permissions. While initially Get-MailboxPermission displayed the correct LEGACY\oldlogon accounts, they’ve been replaced, automagically, with NEW\newlogon as time passed. Inconsistency was the only constant.

I searched the Internet far and wide, asked on professional forums, however my efforts yielded no useful results.

I ended up opening a case with Microsoft.

The Workaround

For each SID in the shared mailbox’s ACL Exchange tries to find its corresponding AD account, and it searches its own forest first. It finds the SID of the LEGACY\oldlogon account in the SIDHistory attribute of the migrated NEW\newaccount, and it says “BINGO! This is the account!” Therefore it stops searching and does NOT follow the chain of trusted forests to identify the real owner of the SID. Therefore it incorrectly assigns permissions to NEW\newlogon in the resource forest instead of LEGACY\oldlogon in the account forest.

If the SID on the shared mailbox’s ACL isn’t found in the Exchange server’s home forest, then, and only then, Exchange will move on to find the account in one of the trusted forests.

To date the only way to get delegated access to work again is to clear the sIDHistory attribute of the NEW\newlogon account. There are a number of ways to clear the SIDHistory. My preferred tool is Ashley McGlone’s PowerShell module.

WARNING: Do NOT clear the SIDHistory unless the effects are fully understood AND if the effects will not affect the business in a negative way! Otherwise consider a career change.

While clearing the SIDHistory solves the immediate issue, it introduces new challenges later on when the merger/migration is refocused onto accounts and resources. Further steps must be carefully planned and coordinated, user accounts potentially re-migrated to re-populate the sIDHistory attribute, otherwise users risk to lose access (and the admin his/her job).

In my opinion the logic by which Exchange finds delegated accounts is flawed, and therefore I see the removal of SIDHistory not as a “fix” but rather as a quick and dirty “workaround”, lacking strategic view of the bigger picture. Hope Microsoft will fix it one day.

Happy migration!

Confused QMM Exchange Native Move Jobs

Setting the Scene

In a recent migration engagement I've come across an error, the cause of which, to date, is unknown even to Dell, yet it seems to be associated with target Exchange server availability.

The target Exchange server crashed due to some underlying storage issues while a couple of Native Move mailbox migration jobs were in progress.

I confirmed that from an Exchange server perspective the mailbox moves survived the crash (Get-MoveRequest | Get-MoveRequestStatistics), because the databases failed over to the surviving DAG member. Once the server was recovered and the databases moved back to the original server, I clicked Retry Move in the QMM console. I was expecting that QMM will update itself by querying Exchange and getting fresh information. Instead, QMM stuck its tongue out at me with the following error:

"Parameter set cannot be resolved using the specified named parameters."

The Issue

When in Native Move migration mode, QMM is nothing more than an easy to drive GUI which behind the scenes builds New-MoveRequest PowerShell commands and submits them to the target server.

QMM keeps track of the migration progress in its own SQL database. In a way this is good because it provides access to quick status and configuration information. However, in the case of Native Move jobs, where QMM is no longer in control of data movement between servers, QMM still appears to look at its SQL database as THE source of truth, as opposed to what the Exchange server is reporting. As a result, things get out of sync quite easily with nasty consequences in terms of time and cost of the migration project.

In case of Exchange server failure QMM appears to pollute its SQL database with wrong data. Common sense dictates that if a move request has successfully been submitted and in progress for some time, one would query the Exchange server to obtain fresh progress information and update its records accordingly. QMM, however, appears to looks exclusively at its own database with no regard to Exchange status reports. When I click "Retry Move", QMM fails to detect that a move request is already in progress despite the fact that its very own database contains a record of a successful command submission, which in itself is a hint that QMM should go and ask Exchange for an update. Instead it attempts to build a new move request command using polluted data from its SQL database.

Specifically, this is what happened in my case:

• Fact: QMM submits New-MoveRequest commands on the target server. Data is "pulled" by the target server from the source server. Therefore parameters are interpreted from the target server's perspective.
• Considering the point above, the correct parameter to use would be -TargetDatabase. Instead QMM uses -RemoteTargetDatabase, as if data would be "pushed".
• When the target Exchange server tries to execute the command, it encounters the -RemoteTargetDatabase parameter pointing to its very own, local database.
• The target Exchange server scratches its head: "I own this database, why on earth is the command telling me that it's remote???"
• Quite rightly, when QMM attempts to submit the wrongly built command, Exchange gets confused and throws the error.

The Fix

Dell was unable to put the finger on the cause. However they did come up with the following workaround:

1. Open SQL Server Management Studio. In the QMM Exchange database, locate and open the MAILBOX table.

 





2. Find the failing mailbox in the MAILBOX table and take note of the value in the ID field.

3. From the MAILBOX_PROCESSING_PROPERTY table, delete the entry matching the ID identified above:

4. If there is a mailbox move in progress (check in the Exchange Management Shell), then cancel it. Then, in the QMM console, retry the mailbox move. This time the correct command is assembled and successfully submitted.

Other Errors

Some other errors I've come across with Native Move jobs:

Error: Mailbox 'Migrated_User' is already being moved to 'Target_Database'.

Reason: QMM spawns multiple New-MoveRequest commands in rapid succession, a couple of seconds apart, before Exchange gets a chance to process the first instance. Effectively QMM appears to be a tad too impatient, not allowing sufficient time for Exchange to process the first command, thus causing QMM to believe that its request went no-where. Thus it will issue the second instance of the same command. However, by then, Exchange would have already processed the first command, and the second attempt results in the above error.
Workaround: Highlight the mailbox with this error and click Retry Move. This forces QMM to query Exchange and update the status of the job.
Note: Dell did neither confirm nor deny this to be a bug. The case was closed without a resolution.

Error: Failed to retrieve RootDSE at 'Source.Domain' under 'DOMAIN\account'
Reason: The domain controller configured in QMM was down for maintenance or otherwise unavailable (e.g. network down).
Workaround: Highlight the mailbox with this error and click Retry Move. This forces QMM to retry the connection to the domain controller.
Note: Dell did neither confirm nor deny this to be a bug. The case was closed without a resolution.

These errors occurred for about 25-30% of the entire migrated user base. Cutover schedules (or mailbox switches) were dead in the water, given that a human had to push a button to retry the jobs.

Since one cannot rely on the schedule to switch over mailboxes, you'll have to pay someone to watch it overnight and push the button as needed, or inconvenience users during the day.

Final Thoughts

QMM for Exchange is a great tool and it can streamline much of the work. Its Native Move engine however is riddled with bugs and it virtually lacks any smarts to recover from Exchange and AD server or network failures. It also seems to have a flawed logic in keeping up with where the Exchange servers are up to. Its scheduling options are great, provided that during the migration the domain controllers and Exchange servers stay available.

A word of caution: QMM doesn't work well with DAGs. Failing over databases to other DAG members doesn't go down well with QMM. Regardless what others are saying, this was my experience. It may change in the future as Dell improves the product.

At the time of this writing, however, if you will only use it in Native Move mode, then carefully consider the cost of the software versus that of a consultant who would build the scripts which could be used in conjunction with QMM-AD.

You would have to use Native Move if:

• You have lots of remote users with poor links and large mailboxes, and thus you want to preserve the OST file.
• The target server is Exchange 2013.
• You migrate to a resource forest (users will keep logging in to their legacy domain with their legacy credentials).

If all three conditions are true then the only option that preserves the OST file is Native Move. In this instance you might find that using QMM-AD with custom Exchange PowerShell scripts will be more cost effective.

Hope this will save some headache for someone out there.

Tame Your Exchange 2013 Logzilla

First Things First

The term Logzilla, as used in this post, has nothing to do with the syslog analysis solution named LogZilla.

Big things are sometimes called Whateverzilla. Oversized Exchange 2013 Managed Availability logs beg for the name. Let me know if it is upsetting to anyone.

Now that IP has been taken care of, skip to The Solution section below if you know all about Logzilla and just want to tame it, or read on for some background information.

Short Primer

Managed Availability is a great feature of Exchange 2013 in large deployments where automation is important. However it can be a hindrance for the small business with limited resources and little experience.

In a nutshell, Managed Availability constantly monitors every aspect of an Exchange 2013 environment, and where appropriate, it automatically takes proactive action to ensure that service levels aren’t affected. Fix first, analyse later. When we consider that Microsoft has hundreds of thousands of servers in Office 365 servicing millions of users, you can immediately see the benefits.

Also, Microsoft’s stance is that it shortens service call resolution time if the events that caused an outage are caught as soon as they happen. We no longer have to turn on logging, reproduce the issue, then turn logging off, because Managed Availability logs will have captured the original event already. It makes perfect sense in the case of the odd issue that’s difficult to reproduce.

To meet these goals, Managed Availability continuously gathers system performance data. As we can imagine, detailed logs need quite some space to be stored. This is why Microsoft recommends “at least” (a.k.a. minimum) 30Gb free space for the disk on which Exchange is installed. As we know from past experience, when Microsoft says “minimum”, practically it means that the software will install, however it will be unable to do anything useful.

The Issue

Exchange can be installed in a variety of storage configurations. Traditionally, it is not uncommon to see a 50-60Gb C: drive, where along the OS, there is Exchange also. If logging needs to be turned on, then traditionally the log location could be easily redirected to another drive with sufficient free space so that C: doesn’t fill up.

With Exchange 2013 you will quickly find that C: will fill up steadily and you can do nothing about it. At least not easily. And you haven't even turned on any logging!

The cause is Managed Availability: logs just keep (re)growing like cancer. By default. Unstoppable (well, MA can be disabled but we don't want to).

At this point you have probably crawled the Internet for solutions. You’ve probably come across articles which tell you to reconfigure the log location in Perfmon or in the Registry. You even tried it, only to find that the logs are back on C: in a few days. In your desperation you want to disable Managed Availability altogether, but that's not a good thing either.

The Future

I have recently attended the O365 Summit in Sydney where I had the chance to ask, straight from the source, how to relocate Logzilla.

The bad:

• There is no supported way to move Managed Availability logs.
• There are no plans to enable administrators to configure log paths in any future Exchange 2013 patch.

The good:

• The Exchange team has heard the community and is considering granting administrators the option to move the logs. However it will only be available in Exchange v.Next.

The Solution

If you are installing a new Exchange 2013 server then install it on a drive other than C:. A 100Gb drive should be sufficient, then you shouldn’t have to worry about moving logs, queues etc – just leave them in their default location. Optionally you may still want to move or logrotate IIS logs as by default they are stored on C:, regardless where Exchange is installed.

However if you already have an existing deployment which struggles with free space, and you cannot move your e-mail to a new Exchange server, then here is what you can do to address the immediate issue:

1. Download Sysinternals’ Junction tool from here.
2. Identify the folders which hold the largest logs, and those which grow the fastest on your C: drive. You can use a tool like WinDirStat.
3. Add a new volume with sufficient space to the Exchange server (100Gb will do)
4. Create a folder structure where you will store the logs. I prefer something that resembles the original structure.
5. Restart the server in Safe Mode. This is needed because some services run in the System context (PID 4) and will not let you implement the changes for the files will be locked.
6. Delete the original log folder, for instance “ETLTraces“ in “C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Diagnostics\”. If you want to preserve the log files then move them first to the new location.
7. Create a junction point in the original location, pointing it to the new target. For example:

JUNCTION.EXE “C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Diagnostics\ETLTraces” “L:\Logs\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Diagnostics\ETLTraces”

NOTE: Just to make it very clear, to successfully replace the folder with a junction, the original folder must be deleted first, and the new target folder must already exist.

8. Restart the server.
9. Optional but recommended: migrate to a server with a properly designed storage as soon as practical and decommission the trouble server.

WARNING:

1. While I found this solution to work and I could identify no ill effects, it is NOT supported by Microsoft. Implement it at your own risk and test it in a lab first. I accept no responsibility for any damage, downtime or loss of any kind it may cause.
2. Read point 1 again.

There you have it. Hope it saves you some headache.

Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 7

Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 1 - Introduction and lab description
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 2 - Deploy and configure the PKI infrastructure
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 3 - Configure and test the Exchange 2013 Client Access role
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 4 - Install CentOS 7
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 5 - Install and configure HAProxy
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 6 - Make HAProxy highly availabile
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 7 - Demo (this page)

---

We reached the final stage of our journey. In this last part we'll explore some of the features that the solution can offer. It is structured as follows:

• Initial tests and basic stats troubleshooting
• Service level load balancing
• Service level failure handling
• HAProxy failure handling
• Stuff to be aware of
• Final thoughts

Initial Tests and Basic Stats Troubleshooting

We start the demo by creating an Outlook profile. Our only test account in this lab is Administrator, and as I mentioned earlier, it doesn't concern me for the purpose of this lab.

The expectation is that the settings are retrieved by means of the Autodiscover service and the profile is created with minimal effort. Let's rock and roll.

In Control Panel / Mail, we click Add to create a new profile:

Type a name for the profile:



Type a user name, e-mail address and password:

When prompted, type the credentials for LAB\Administrator. It is a workgroup computer, so this is the expected behaviour. On a domain joined workstation we wouldn't have to authenticate. Remember to tick the box to remember the password to minimise inconvenience.

We get the green tick for all stages. Looking good so far. Click Finish.

And we have our profile created.

We fire up Outlook. It opens nicely and logs us in to the mailbox. I already have some content there from previous tests - remember, in Part 6 we configured e-mail notifications for HAProxy HA events in keepalived.

Now we do a "Test E-mail Autoconfiguration" (Ctrl + RighClick on the Outlook icon in the notification bar), to confirm that we've got all the configuration in good order.

Finally we confirm that Outlook is connected  as expected, using the right protocol. We have Outlook 2013 SP1 connecting to an Exchange 2013 SP1 server that's enabled for MAPI-HTTP, therefore Outlook will attempt to connect first, by default, via MAPI-HTTP. The following screenshot confirms that we are indeed connected via MAPI-HTTP. For more on MAPI-HTTP and on how to check whether it is indeed used, see this Microsoft Exchange Team blog.

By now we should see some stats in HAProxy. We now access the HAProxy stats page, which in this lab is accessible via https://mail.digitalbrain.com.au/stats. Please remember:

• There are two HAProxy servers in a HA configuration (provided you followed everything in this blog to the letter).
• mail.digitalbrain.com.au points to the VIP associated with the HA pair of our HAProxy servers.
• We will be connected to the HAProxy server holding the VIP, therefore the stats will represent traffic seen by that HAProxy server only. It is NOT a combined view of both servers.

With all the work so far, including the profile creation, launching Outlook, and performing all the connectivity tests, we should see some load-balanced traffic for Autodiscover and MAPI-HTTP. Here is what we get:

Huston, we have a problem! My Outlook profile was clearly configured by means of Autodiscover, yet there is no Autodiscover traffic:

On the bright side, we have MAPI-HTTP traffic, and stats show that it is load-balanced between the two CAS servers:

So what is going on?

Well, if look at the very bottom section of the stats, be_ex2013, we see something going on there. If we go back to Part 5 and analyse the haproxy.conf file, we'll understand that this section acts like a "catch-all" for all traffic that doesn't match the ACLs defined in the "frontend" section, and it is defined there as "default_backend be_ex2013":

Looking at the IIS logs on the CAS servers we do see autodiscover traffic, and here is the catch: we defined our HAProxy ACL as Autodiscover...,

...yet in the IIS log we see autodiscover:

Outlook is posting requests in lower case! And yes, HAProxy ACLs are CaSE sENsiTIve. The case sensitivity of ACLs is documented in the official HAProxy documentation, but it is easy miss it, so I thought I'll bring it out with an example. Mystery solved.

All we have to do to fix it is to add another ACL with the correct capitalization. We don't want to delete the original line because there might be some clients/applications which send requests with a capital "A".

We need to update the haproxy.conf file on both HAProxy servers and restart the haproxy service:

systemctl restart haproxy.service

Following that I removed and recreated the Outlook profile, opened Outlook and did the tests again, for the sake of generating some traffic that I can see in the stats and to make sure that I can clearly differentiate it from previous tests.

IMPORTANT: Please be aware that restarting the HAProxy service will also reset the stats. They are not preserved between restarts.

So, after another round of Outlook profile creation and testing, we can happily confirm that our Autodiscover service is load balanced and it is correctly recorded in the right section of the stats:

Here we go, we now know where to look and how to fix reporting if we don't see what we expect. There are other ways to work out the correct capitalization - we can look at HAProxy logs, as well as at the URLs configured in Exchange - whatever works for you.

Service Level Load Balancing

It is time now to check how our other services are load balanced.

Testing MAPI-over-HTTP

We've already checked MAPI-HTTP and confirmed it is working, so no point doing it again.

Testing RPC-over-HTTP (Outlook Anywhere)

Our test computer has Office 2013 SP1, therefore by default it will connect via MAPI-HTTP. We can force it to use RPC-over-HTTP:

• On the workstation, open Regedit.exe.
• Navigate to HKCU\Software\Microsoft\Exchange
• Create the MapiHttpDisabled entry with a DWORD value of 1.

It is documented at http://blogs.technet.com/b/exchange/archive/2014/05/09/outlook-connectivity-with-mapi-over-http.aspx.

Recreate the Outlook profile, or create a new one. Open Outlook and check the connection status: it uses RPC-over-HTTP indeed.

In order to generate some traffic we click on a couple of e-mails, open them, and send some to ourselves. Then we check the stats: our RPC-over-HTTP traffic is correctly identified and load balanced.

Testing Outlook Web App

We open Internet Explorer on LAB-WS01 and browse to https://mail.digitalbrain.com.au/owa. We can see that there are a couple of e-mail there.

Again, to create some traffic, we poke around the mailbox, then we check the stats. OWA traffic is correctly identified and load balanced:

Testing Exchange Admin Center

The process is similar to that of testing OWA: we browse to https://mail.digitalbrain.com.au/ecp, and click a few controls around the interface.

Then we confirm that our EAC (a.k.a. ECP) traffic is load-balanced:

Testing Exchange ActiveSync

This bit is a tad tricky due to the fact that I cannot access my lab from a proper smart phone, and therefore I had to use an emulator. Ideally you'd use a real smart phone or tablet, but if you are in the same boat as I, then get the Windows Phone SDK. Please bear in mind that you should use a physical machine with virtualization features enabled in the BIOS. Setting it up is beyond the scope of this article, I'll let you work it out.

An EAS client is no different from a computer: in order to communicate with a secure web site, it must trust it. In my lab I use in-house CAs, therefore the smart phone wouldn't trust my Exchange servers. We have to load our root and intermediate CAs' certificates into the phone's certificate store. There are a number of ways to do it and it is documented in this MSDN article. I chose the first option, installing the certificates via Internet Explorer. I used LAB-EX01 to make the certificates available, and here is how:

• On LAB-EX01 I created a new web site and called it "Certificates" - you can call it anything you like:

 

• We specify the path to the site's files as C:\Temp. This is where my root and intermediate certificates were stored when we built the environment.

 

• In Site Bindings we configure the host name as cert.digitalbrain.com.au (or anything you like that is different from URLs used by Exchange). We will access it via HTTP (unencrypted), on port 80.

 

 

• We add a new MIME type, otherwise access to the certificate files will be blocked. To start with, we open the MIME Types configuration:

 

• Type .cer as the file name extension and set the MIME type to text/plain:

• Then we create a pin-point DNS zone called cert.digitalbrain.com.au, pointing it to the LAB-EX01 server's IP address, 10.30.1.11. For a refresher on my DNS setup see Part 3.

• On the phone, open Internet Explorer and browse to http://cert.digitalbrain.com.au. The emulator is a bit small...

 

• ...but we can zoom in:

• Click "RootCA.cer". When prompted, tap "Open":

•  Then tap "Install":

• When done, tap "OK":

• Tap "Back" to go back to the certificates page:

Repeat the process for the intermediate CA's certificate, LAB-CA02.lab.local_lab-LAB-CA02-CA.cer.

Before configuring Exchange ActiveSync, we can test whether the certificates work by browsing to the OWA page. There are two things that indicate that we are on the right track:

1. We are taken into the mailbox with no certificate warnings.
2. The secure OWA website has a padlock, showing that the site is secure.





We are now ready to set up EAS and test. Back on the home screen, tap on the envelope icon:

Tap Exchange:

Type the administrator's e-mail address and password, then tap "Sign In":

The account is added with no further interaction. Tap "Done".

Opening the EAS client, we are taken to our Inbox:

At this point we have an EAS client configured, some e-mails in the Inbox, and we should see them in the EAS client. However Administrator is a member of a number of protected groups and as such default AdminSDHolder security prevents it from using Exchange ActiveSync. As a result our emulated Windows Phone will not show us anything. We can see this in the IIS logs on the Exchange servers:

Since I can't use EAS with Administrator, I created a regular, test user just to test EAS load balancing. Removed the connection to the Administrator mailbox, created one for the test user, and sent a couple of test e-mails. Lo and behold: my emulated EAS client has downloaded all my test emails!

Checking the stats, we bump into the same issue as with MAPI-HTTP: no traffic is shown for EAS. The reason is the same: wrong capitalization in the ACL of the URL in the haproxy.conf file. Here is the original (and incorrect) content:

Checking the IIS logs (see above), we can work out the correct capitalization and add an additional ACL, or edit the existing one. I chose to add a second line:

Remember:

1. Make the change on both HAProxy servers.
2. Restart the HAProxy service on both servers.

Once we updated haproxy.conf and restarted HAProxy, we send some e-mails to the test user and sync it with the EAS client. Checking the stats again confirms that this time it works like a charm:

Testing EWS and OAB

EWS provides key functionality for Outlook. Outlook also downloads the Offline Address Book via HTTPS. Simply launching and using Outlook will inevitably generate traffic to these services. Our MAPI-over-HTTP and RPC-over-HTTP tests so far would have inevitably generated some traffic. Checking the stats confirms that they are correctly identified and are load balanced:

Other Traffic

Anything that doesn't match the ACLs defined in the haproxy.conf file will be handled by the default, catch-all backend definition. Traffic will be load balanced based on host availability, but not bound to any specific service. If our ACLs are correctly configured, then we shouldn't see anything here (or negligible traffic for irrelevant content). From previous tests and failures however, we do see some residual traffic:

If we still see stuff being handled by this section and we are concerned about it, we can check the HAProxy logs to identify what they are, with the following command:

grep 'fe_ex2013~ be_ex2013/lab-ex*' /var/log/haproxy.log

The output will reveal the requests which were handled by the default, catch-all definition. Update the ACLs as or if required. In our case we've got it right and there is nothing to worry about:

Service Level Failure Handling

Now we want to test the core of this series of articles: what happens if a single service becomes unavailable, while everything else keeps working?

We'll set two components to an inactive state:

• RpcProxy on LAB-EX01 - remember, we forced Outlook to use RPC-over-HTTP, which is still in place
• EwsProxy on LAB-EX02 - Outlook uses EWS for a number of functions

We'll then reset the stats by restarting the HAProxy service on both servers, and we'll launch Outlook.

First, we set the state of RpcProxy on LAB-EX01, and EwsProxy on LAB-EX02, to Inactive:

Set-ServerComponentState LAB-EX02 -Component RpcProxy  -Requester Maintenance -State Inactive
Set-ServerComponentState LAB-EX01 -Component EwsProxy  -Requester Maintenance -State Inactive

Next we restart HAProxy on both servers:

systemctl restart haproxy.service

Then we launch Outlook and generate some activity:

• Open a couple of emails.
• Open the calendar.
• Download the offline address book.
• Anything else you might want to do to get the bytes flowing.

Finally we check the stats:

 

The stats have a number of noteworthy features:

• There is traffic on the be_ex2013_rpc service. Outlook indeed uses RPC-over-HTTP - our registry hack to force it to use RPC-over-HTTP is still in place.
• The RpcProxy service isn't responding on LAB-EX01 - note the red stripe across it and no traffic at all in the be_ex2013_rpc section.
• All RPC-over-HTTPS traffic has been directed to LAB-EX02.
• The EwsProxy service isn't responding on LAB-EX02 - note the red stripe across it and no traffic at all in the be_ex2013_ews section.
• All EWS traffic has been directed to LAB-EX01.
• The OabProxy service is active and working well on both Exchange servers and OAB traffic is load-balanced across them - note the stats in the be-ex2013_oab section.

Now we re-activate the disabled services, "fixing" the fault or taking servers out of maintenance and putting them back into production, say, after patching. Then we poke around in Outlook to generate some action, and will check the stats again.

Re-activating disabled services:

Set-ServerComponentState LAB-EX02 -Component RpcProxy  -Requester Maintenance -State Active
Set-ServerComponentState LAB-EX01 -Component EwsProxy  -Requester Maintenance -State Active

This time we don't reset the HAProxy stats because we want to confirm that after service recovery traffic is re-distributed again on the fly. Here are the new stats:

Please note the following features:

• No more red stripes, all services are back online.
• Traffic is now being load-balanced.
• A disproportionate amount of traffic can be observed between servers within the RPC and EWS services due to the earlier service "fault" while only one server out of two serviced the affected services.
• We didn't have to do anything in HAProxy: it detected the resurrected services and started using them on the fly.

All along this test the Outlook user experience was flawless, continuous, responsive and reliable.

We've successfully tested the essence of this lab: we load-balanced client traffic at Layer 7 service level, making smart load balancing decisions based on individual service (un)availability within an Exchange CAS server, and provided uninterrupted and seamless user experience. We can report no service disruption and ask for a bonus.

HAProxy Failure Handling

So far we tested how HAProxy handles failures of Exchange services. It assumed a working HAProxy server. But what happens if HAProxy itself fails?

This section shows how keepalived, our high availability solution, keeps the load balancing service itself working.

The test is simple: faults don't warn us kindly, "Dear admin, be advised that your server will fail at 10:23PM this Saturday". So I simulate a real fault by switching off power to LAB-HAP01 and we'll watch the following:

• The VIP is transferred to the secondary node, LAB-HAP02.
• Roughly how long it takes for the failure to be detected and the VIP to be moved and become active.
• Administrative email notification is sent to LAB\Administrator. Remember, we configured keepalived email notifications to go to administrator@digitalbrain.com.au, see Part 6.

Confirming that LAB-HAP01 is holding the VIP and both load balancers are online:

I set up two PING sessions: one for LAB-HAP01 (10.30.1.13), and the other for the VIP (10.30.1.15). Whilst the PINGs are in progress, I switched off power to LAB-HAP01. Pings showed that LAB-HAP01 went offline, and the VIP flicked over to LAB-HAP02. Only one ping was lost, so that's pretty good.

Looking at the logs on LAB-HAP02, we confirm that it is now the MASTER node in the HAProxy cluster:

Remember, we configured email keepalived notifications, so we should see some coming in. Here it is, confirming the same:

And in case someone needs more facts, we can always check the IP addresses bound to LAB-HAP02's NIC:

During all this time load balancing and Outlook connectivity was maintained - that's how I could see the incoming e-mail notification from LAB-HAP02. Mind you, this might be only by accident as I have not implemented any mechanisms in my lab to share and maintain client stickiness information across the two load balancers. Nevertheless, it shows that if after a failure the new master connects me to the same CAS server (remember, I still use RPC-over-HTTP), my connection will keep working. Your experience might be different unless client stickiness data sharing is enabled, or if you switched back to using MAPI-over-HTTP. For more details see the "Stuff to Be Aware Of" section below.

Now we "fix" the failed master node and watch the service fail back. This is what happens:

1. LAB-HAP01 comes back online and it detects that it isn't the MASTER.
2. LAB-HAP01 determines that it has a higher priority than LAB-HAP01 - see the "priority" values in the keepalived.conf files of the two hosts. LAB-HAP01 therefore assumes the MASTER role.
3. LAB-HAP02 acknowledges the takeover and it enters the BACKUP state.

Stuff To Be Aware Of

Technically we completed our lab build and testing and we could potentially leave it here. However a couple of closing thoughts are in order.

Is the solution perfect? No, it isn't.
Can it be optimised? Yes, it can.

Here are some quick points.

ACL Case Sensitivity

From the tests above we've learned that ACLs are case sensitive. In our lab we used the url_beg directive to match the beginning of the various Exchange 2013 URLs with specific patterns. We saw that /Autodiscover and /autodiscover are treated as distinctive strings.

We created entries for various capitalization options and that fixed our issue.

Another, better way to achieve the same is to use case insensitive matches. We could replace the url_beg directive with url_reg to do a case insensitive regexp string match. Our ACLs would change as follows:

acl autodiscover url_reg (?i)\/autodiscover
acl mapi url_reg (?i)\/mapi
acl rpc url_reg (?i)\/rpc
acl owa url_reg (?i)\/owa
acl eas url_reg (?i)\/microsoft-server-activesync
acl ecp url_reg (?i)\/ecp
acl ews url_reg (?i)\/ews
acl oab url_reg (?i)\/oab

However we must be mindful of the performance impact that regexp matching might have in a large environment or on undersized load balancers.

HAProxy Service Failures and Client Stickiness

The purpose of keepalived in our lab is to activate the backup load balancer should the master fail. However the slave will be unaware of which client was connected to which Exchange server. Therefore if the master fails, there will likely be lots of angry users, re-connected to the wrong CAS server, and the helpdesk phones will ring like crazy. Please bear in mind though that this is only an issue if using RPC-over-HTTP. It is unlikely to happen if using MAPI-over-HTTP, because the CAS role becomes a light-weight, stateless HTTP proxy which no longer requires client stickiness.

Ideally stickiness information should be maintained across load balancers, so that clients will be seamlessly reconnected to the correct server, regardless of which load balancer is servicing their sessions.

HAProxy provides the mechanism to alleviate this situation, however we have not implemented it in our lab. See this blog for more details, as well as the "3.5. Peers" section in the HAProxy documentation.

In real life you should synchronise server stick tables or use source IP hashes in order to reconnect clients to the correct CAS servers.

Stats Don't Survive Reboots and Aren't Cumulative, Nor Aggregated Across Hosts

HAProxy stats are reset every time the service is (re)started.

Each HAProxy host maintains its own set of stats. When we view the stats page, what we see are the values of the load balancer that holds the MASTER role at the time, since its last service (re)start. Also there is no event correlation as stats are point-in-time values which are meaningless without context.

During my research I've come across Datadog who claims to overcome this issue by archiving HAProxy logs and correlating/aggregating them. I haven't tested it, have no affiliation, nor any interest in Datadog, so don't hold me responsible if it doesn't work out for you.

Final Thoughts

In this series of articles we built a solution using completely free and open source tools. The only cost would be the time of the individual building it. It is a potential low-cost alternative to more expensive commercial solutions, provided that the skills to build, support and maintain it are available, and its limitations are accepted.

For the sake of completeness I need to warn everyone who contemplates implementing it into production  that you'll need to consider company policy and strategy. Some of the concerns I can quickly think of:

Is open source allowed in the organization? - I've seen places where open source was banned outright for various reasons.

Is support continuity a concern? - Some organizations are very good at making Windows and appliance-type gear work, but they have a massive lack of in-house Linux skills. You'd agree by now that some degree of Linux knowledge is required to build and maintain such a solution. If something breaks, will you have the resources to fix it? Can you survive on community support only?

Other concerns? - There may be others that escape me right now.

Well, this is it. I hope you enjoyed it and found it useful as well as informative, and thank you for bearing with me.

---

Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 1 - Introduction and lab description
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 2 - Deploy and configure the PKI infrastructure
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 3 - Configure and test the Exchange 2013 Client Access role
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 4 - Install CentOS 7
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 5 - Install and configure HAProxy
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 6 - Make HAProxy highly availabile
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 7 - Demo (this page)