Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 3

Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 1 - Introduction and lab description
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 2 - Deploy and configure the PKI infrastructure
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 3 - Configure and test the Exchange 2013 Client Access role (this page)
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 4 - Install CentOS 7
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 5 - Install and configure HAProxy
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 6 - Make HAProxy highly available
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 7 - Demo

---

In Part 2 we installed and configured our lab PKI, consisting of a root and an intermediate CA. Not that we needed two, we technically can get away with only one, but it is a closer simulation of what we find in the real world.

In Part 3 we configure the Exchange environment. Here are the main steps:

• Add an accepted domain and configure the default email address policy.
• Configure internal and external URLs.
• Enable and configure MAPI-HTTP.
• Install certificates.
• Preliminary testing.

Add an accepted domain and configure the default email address policy

We want to receive e-mail on our public SMTP domain, digitalbrain.com.au, but by default all our lab Exchange servers know of is lab.local. We need to add the domain as an authoritative accepted domain and make it the default. Then we update the “Default Policy” email address policy (our one and only) to add a new default template of “%m@digitalbrain.com.au” (a.k.a. alias@digitalbrain.com.au) as the primary template – see this Microsoft article for more details. Finally we update our recipients with the new template. We use the following commands:

New-AcceptedDomain -DomainName digitalbrain.com.au -DomainType Authoritative -Name digitalbrain.com.au
Set-AcceptedDomain digitalbrain.com.au -MakeDefault $true
Set-EmailAddressPolicy "Default Policy" -EnabledPrimarySMTPAddressTemplate %m@digitalbrain.com.au
Update-EmailAddressPolicy “Default Policy”

We then confirm that users (Administrator in our case) have been indeed updated and are using the new address:

Get-Mailbox Administrator | fl name,PrimarySMTPAddress,EmailAddresses

A word of caution: PowerShell commands that manipulate email address templates and policies expect a list of objects when multiple values have to be specified. If unsure, on production servers use the GUI to avoid overwriting existing values, or make a note of the values of properties before altering them, just in case. For instance, our actions have removed the %m@lab.local template from the EnabledEmailAddressTemplates property and replaced it with %m@digitalbrain.com.au. If our intention was to “append” our new template without overwriting the existing ones, then it would be trouble. While I don’t mind it in this lab, it would be a career limiting move in production.

Configure internal and external URLs

At this point we have LAB-EX01 and LAB-EX02 with vanilla, unconfigured Exchange 2013 servers installed. The servers are domain members and have their TCP/IP settings configured as detailed in the diagram in Part 1.

We start by logging on to LAB-EX01 with an account that is a member of the Organization Management group (LAB\Administrator in this demo) and open the Exchange Management Shell. The following commands have to be run on only one server (LAB-EX01 in this case, but you could also run them on LAB-EX02 if you like). They will configure the CAS roles on both servers. In my lab I am using the digitalbrain.com.au domain for both internal and external access – I’ll explain later the purpose behind it.

Get-ClientAccessServer | Set-ClientAccessServer -AutoDiscoverServiceInternalUri https://mail.digitalbrain.com.au/Autodiscover/Autodiscover.xml
Get-OutlookAnywhere | Set-OutlookAnywhere -InternalHostname mail.digitalbrain.com.au -InternalClientAuthenticationMethod Ntlm -InternalClientsRequireSsl $true -ExternalHostname mail.digitalbrain.com.au -ExternalClientAuthenticationMethod Ntlm -ExternalClientsRequireSsl $true
Get-OwaVirtualDirectory | Set-OwaVirtualDirectory -InternalUrl https://mail.digitalbrain.com.au/owa -ExternalUrl https://mail.digitalbrain.com.au/owa
Get-OwaVirtualDirectory | Set-OwaVirtualDirectory -DefaultDomain lab.local -LogonFormat UserName
Get-ActiveSyncVirtualDirectory | Set-ActiveSyncVirtualDirectory -InternalUrl https://mail.digitalbrain.com.au/Microsoft-Server-ActiveSync -ExternalUrl https://mail.digitalbrain.com.au/Microsoft-Server-ActiveSync
Get-EcpVirtualDirectory | Set-EcpVirtualDirectory -InternalUrl https://mail.digitalbrain.com.au/ecp -ExternalUrl https://mail.digitalbrain.com.au/ecp
Get-WebServicesVirtualDirectory | Set-WebServicesVirtualDirectory -InternalUrl https://mail.digitalbrain.com.au/EWS/Exchange.asmx -ExternalUrl https://mail.digitalbrain.com.au/EWS/Exchange.asmx
Get-OabVirtualDirectory | Set-OabVirtualDirectory -InternalUrl https://mail.digitalbrain.com.au/OAB -ExternalUrl https://mail.digitalbrain.com.au/OAB
Get-PowerShellVirtualDirectory | Set-PowerShellVirtualDirectory -InternalUrl http://mail.digitalbrain.com.au/powershell -ExternalUrl http://mail.digitalbrain.com.au/powershell

Note that if we would be using a wildcard certificate then we would also need to run the following commands (not in this lab though):

Set-OutlookProvider EXPR -CertPrincipalName msstd:*.digitalbrain.com.au
Set-OutlookProvider EXCH -CertPrincipalName msstd:*.digitalbrain.com.au

The ECP virtual directory requires additional explanation. It provides access to the Exchange Admin Center. It is the new management GUI and it is web based, therefore easy to publish and access. As a security best practice, EAC shouldn’t be exposed to the public Internet. I am not concerned about it in this lab, but for the sake of completeness, you can (and should) turn off access to the Exchange Admin Center.

At this point we have configured internal and external URLs of virtual directories, as well as the Service Connection Point.

Enable and Configure MAPI-HTTP

MAPI-HTTP is a new client protocol introduced with Service Pack 1 for Exchange 2013, and it only works with Outlook 2013 SP1 or later clients. It eliminates the RPC protocol layer and it encapsulates MAPI requests straight into the HTTP protocol. Check out this blog for more details.

Enabling MAPI-HTTP is easy and it is described here.

First, according to the prerequisites, we “should” install the hotfix described in KB 2908387 (my lab is using Windows 2012 R2 servers). However I had difficulty obtaining this hotfix at the time of this writing. It is a .NET hotfix which addresses performance issues and it should be installed in a production environment. In this lab however I did not bother. In a production environment install this hotfix on all CAS servers which will be enabled for MAPI-HTTP access.

Next we create the “DisableRetStructPinning” registry key as a DWORD value of 1 under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework. We do this on both LAB-EX01 and LAB-EX02.

Next we create the COMPLUS_DisableRetStructPinning system environment variable with a value of 1. We do this on both LAB-EX01 and LAB-EX02.

Next, on LAB-EX01 (or LAB-EX02 if you like – it needs to be run on one server only) we run the following commands in Exchange Management Shell:

Get-MapiVirtualDirectory | Set-MapiVirtualDirectory -InternalUrl https://mail.digitalbrain.com.au/mapi -ExternalUrl https://mail.digitalbrain.com.au/mapi -IISAuthenticationMethods Ntlm,OAuth,Negotiate
Set-OrganizationConfig -MapiHttpEnabled $true
iisreset /noforce

Done. That was easy.

Install Certificates

Now we need to deploy some certificates to make it all work.

To start with, we need to install the root CA certificate from LAB-CA01 into the “Trusted Root Certification Authorities” store, and the intermediate CA certificate from LAB-CA02 into the “Intermediate Certification Authorities” store of the local computer account. We must do it on both LAB-EX01 and LAB-EX02. If we miss this step then our Exchange SAN certificate will not be trusted and it will be unusable. We’ve been through the procedure in Part 2 when we imported the root CA certificate from LAB-CA01 to LAB-CA02, so I’m not going to go into any more detail.

After importing these certificates, our certificate stores on LAB-EX01 and LAB-EX02 look like this:

Imported Root CA Certificate

Imported Intermediate CA Certificate

Next, we need to generate the certificate signing request (CSR) file. On LAB-EX01, we type the following Exchange Management Shell command:

Set-Content -path “C:\Temp\mail_digitalbrain_com_au.req” -Value (New-ExchangeCertificate -GenerateRequest -KeySize 2048 -SubjectName “c=AU, s=NSW, o=Digital Brain, ou=HAProxy Lab, cn=mail.digitalbrain.com.au” -DomainName mail.digitalbrain.com.au,autodiscover.digitalbrain.com.au -PrivateKeyExportable $True -FriendlyName "Exchange 2013 for HAProxy Lab")

The output of this command is the C:\Temp\mail_digitalbrain_com_au.req file. We now transfer this file to LAB-CA02, then we log on to LAB-CA02 and open the “Certification Authority” MMC console. Right-click Lab-Intermediate-CA, select “All Tasks”, then click “Submit new request”.

In the Open Request File dialog select our CSR file that’s been transferred from LAB-EX01 and click Open. In my lab it is C:\Temp\mail_digitalbrain_com_au.req.

Next, click to highlight the Pending Requests node. There we’ll see our pending request. Right-click it, then under “All Tasks” select Issue.

Next click to highlight the “Issued Certificates” node. There is our new certificate. Right-click it, then under “All Tasks” select “Export Binary Data”.

In the Export Binary Data dialog select “Binary Certificate” and “Save binary data to a file” then click OK.

In the Save Binary Data dialog select a folder, give the exported certificate a name, and click Save. In my lab the file is C:\Temp\mail_digitalbrain_com_au.cer.

We now copy the issued certificate to LAB-EX01. Then we use the following Exchange Management Shell commands to import the certificate into the store, assign it to the SMTP and IIS services, and finally to restart IIS for the changes to take effect:

Import-ExchangeCertificate -FileData ([Byte[]]$(Get-Content -Path C:\Temp\mail_digitalbrain_com_au.cer -Encoding byte -ReadCount 0))
Enable-ExchangeCertificate -Thumbprint DACD9A0ACD2A9DFE2E4C565BFDD14FB9DE288F0A -Services SMTP,IIS
iisreset /noforce

Note that the certificate file that we just issued and transferred to LAB-EX01 is C:\Temp\mail_digitalbrain_com_au.cer. Also, the iisreset command usually takes a couple of attempts to complete, so keep trying until successful – this is normal and expected.

How do I know what thumbprint to use? Well, it is listed on the issued certificate, but I can also use the Get-ExchangeCertificate cmdlet to view certificate properties, including their thumbprint, on the Exchange server.

Documentation says that we can import and assign a certificate in one go on multiple servers if using the GUI (a.k.a. EAC). My personal experience is that the private key didn’t make its way to the additional servers. I must have been doing something wrong, probably because I tried to import the CER file (which obviously doesn’t contain the private key). Should I have imported a PFX file that’s been exported from another Exchange server along with the private key, it probably would have worked. This detail is missing from the official documentation, so don’t get caught. For this reason I prefer the traditional way: export the certificate from an Exchange server to PFX, then import it on each additional CAS server. Paul Cunningham has a nice article here whichclears it up nicely.

We therefore export the certificate along with the private key from LAB-EX01. While still in Exchange Management Shell on LAB-EX01, we run the following command:

Set-Content -path “C:\Temp\mail_digitalbrain_com_au.pfx” -Value (Export-ExchangeCertificate -Thumbprint DACD9A0ACD2A9DFE2E4C565BFDD14FB9DE288F0A -BinaryEncoded:$true -Password (Get-Credential).password).FileData -Encoding Byte

We will be prompted for credentials to protect the exported private key. The output of the command is the password-protected C:\Temp\mail_digitalbrain_com_au.pfx file which contains the certificate and the private key.

Now we transfer the file to LAB-EX02, then log on to LAB-EX02, and import the certificate:

Import-ExchangeCertificate -FileData ([Byte[]]$(Get-Content -Path C:\Temp\mail_digitalbrain_com_au.pfx -Encoding byte -ReadCount 0)) -Password:(Get-Credential).password

Type the credentials when prompted:

Next we use the Enable-ExchangeCertificate cmdlet to assign the new certificate to services. In your lab use the thumbprint of your certificate.

Enable-ExchangeCertificate -Thumbprint DACD9A0ACD2A9DFE2E4C565BFDD14FB9DE288F0A -Services SMTP,IIS

When complete, we restart IIS:

iisreset /noforce

We now have two fully configured Exchange 2013 servers, ready to take user load. We have one mailbox, that of the account we used to install Exchange, Administrator’s. It is sufficient for the purpose of this lab.

Preliminary testing

Now that we have a functional messaging system, we want to make sure that everything is in good order before we move on.

To start with, we need to configure DNS. In particular, we need records for mail.digitalbrain.com.au and autodiscover.digitalbrain.com.au. There are many ways to achieve the same and DNS is probably one of the least understood components when it comes to correctly configuring it for client access.

In this lab I set out to demonstrate what I believe is best practice and what I always try to implement for my clients. To illustrate, here is a hypothetical but realistic scenario:

1. Users must have seamless and consistent experience regardless whether they are connected to the trusted corporate network or coming in via the public Internet. Users may use any supported connectivity method. Users must not have to make changes to their mail profile when they roam.
2. DNS and Exchange management overhead must be minimised.
3. Costs must be minimised.

We achieve these goals by using the same publicly registered SMTP domain for both the internal and external URLs on virtual directories. If you cast your eyes back to the beginning of this article where we configured the CAS virtual directories, you will notice that all internal and external URLs are configured with the digitalbrain.com.au domain, irrespective of the fact that in this lab the local AD domain is lab.local. I configured the lab this way on purpose to demonstrate that we don’t have to use the local AD domain as internal URL on virtual directories. We should keep things simple: same domain, internally and externally.

This raises an issue: we have to maintain a zone for the publicly registered domain on the internal DNS servers, called split-brain DNS. Or maybe not… In many cases that might well be the case. However if you don’t want to, you don’t have to. We can use pinpoint DNS zones instead. Please note that the supporting article I’m referring to is written for Lync but it is equally applicable to Exchange servers also. Additionally, I want to mention that according to the article, “dnscmd.exe is required because the DNS user interface does not support creation of pin-point zones.” Wrong. You can use the GUI too. Try it.

Let’s see how this helps us to meet our hypothetical goal.

1. Roaming users may not necessarily be domain-joined. They move freely between the corporate and external networks. Having the internal and external URLs configured with the same domain, it removes the need to reconfigure mail profiles every time they connect to different networks. This becomes a massive win when we consider what’s involved in changing profiles on hundreds or even thousands of smart phones as users roam between corporate and carrier networks.
2. If using a publicly registered domain as internal URL then we need to maintain public DNS records pointing to private IP addresses on our internal DNS servers. Split DNS is bad. It doubles the administration, and errors *will* happen. We don’t want split DNS. The alternative is pinpoint DNS. Administration overhead and human error is virtually eliminated.
3. Since internal and external URLs use the same domain, we can get away with a cheaper SAN certificate, with only two entries: mail.digitalbrain.com.au and autodiscover.digitalbrain.com.au. It is cheap, it does not expose our internal AD domain and server names to external eyes, and it is fully compliant with the new rules regarding what we can and cannot have on a commercial certificate – I highly recommend you to get acquainted with these new rules as they’ll kick in in 2015.

With these design points in mind, my lab DNS server has two pinpoint DNS zones: one for mail.digitalbrain.com.au and one for autodiscover.digitalbrain.com.au. For the purpose of this immediate test, the zones point to LAB-EX01 (10.30.100.11). They will be re-pointed later to the
HAProxy virtual IP. This is how one of my pinpoint DNS zones looks like:

We will use LAB-WS01 for testing, and it will need to trust our Exchange certificate. Therefore we must install the root and intermediate CA certificates, the same way as we did on LAB-EX01 and LAB-EX02. If you haven’t done it yet, do it now. After installing the root and intermediate CA certificates on LAB-WS01, browse to https://mail.digitalbrain.com.au/owa. We are presented with the OWA logon page:

There are a couple of things to notice:

1. It might sound trivial, but the fact that we typed in the OWA URL and we have actually been presented with the OWA logon page, proves that our pinpoint zone in the internal lab DNS server has actually worked: we landed on LAB-EX01’s OWA page. Remember, we configured the zone with LAB-EX01’s IP address (check the DNS server screenshot above).
2. We used our publicly registered domain URL while connected to the local lab.local network – and it worked! We CAN use the publicly registered DNS domain for Exchange access internally too.
3. The padlock indicates that the site is trusted. We didn’t receive a certificate warning. This means that our lab PKI is working (provided that the root and intermediate CA certificates have been imported on all servers and the workstation). I tried to draw attention to it on the screenshot with the red arrow.

If we click on the padlock then we can see that the certificate was issued to mail.digitalbrain.com.au by our Lab-Intermediate-CA, and it is valid:

It also has the proper Subject and SAN entries. It is recommended to include the subject name as a SAN entry also, and some commercial CAs include it by default (Symantec and others), hence my choice to mimic a commercial CA this way too and have the subject name as a SAN entry also:

We can see the thumbprint too – it is the same that we used to enable the certificate:

And finally, we can see the full certificate chain, fully trusted, just as in a commercial PKI system:

We are in business. If you feel like it, then re-point the pinpoint zones in DNS to the other Exchange server and repeat the test to confirm that both Exchange servers are working.

In this part we’ve done quite a bit: we configured Exchange, imported CA certificates, requested and installed the Exchange SAN certificate, configured DNS, and we’ve done a basic test.

In the next part we will start getting into the real deal: we’ll install the Linux servers which ultimately will host our HAProxy load balancers. Stay tuned.

---

Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 1 - Introduction and lab description
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 2 - Deploy and configure the PKI infrastructure
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 3 - Configure and test the Exchange 2013 Client Access role (this page)
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 4 - Install CentOS 7
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 5 - Install and configure HAProxy
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 6 - Make HAProxy highly available
Highly Available L7 Load Balancing for Exchange 2013 with HAProxy – Part 7 - Demo