Tuesday, 17 August 2021

PrintNightmare Nightmare

Hello folks, Zoltan again. This time with a printer story.

The short version: Have you installed the August 2021 update, KB5005030n update and started seeing the infamous "Do you trust this printer?" warning? Well, learn to live with it or disable the security measures implemented by the update.

The meat around the bone: as a consultant and system admin, I've deployed numerous systems where printers are deployed and administered centrally using Microsoft's Point and Print technology. Moved all local and shared printers from individual PCs to a central print server. Give a user a new laptop, join the domain, and (s)he'll get all the printers. Life's good. Or so it was until PrintNightmare has been identified and Microsoft released its August 2021 security update, KB5005030.

The update changes the the security model of the Point and Print driver installation, requiring administrative approval to install or update signed or unsigned printer drivers. Yes, signed drivers too. From the article:

Since then, August the 10th 2021 that is, users who installed this update and their printers are installed via Point and Print, have been plagued with this warning, unable to print their documents:

The gist of it: all, (and by that Microsoft means ALL) computers running the Print Spooler service, workstations included, are vulnerable to the PrintNightmare vulnerability and therefore Microsoft's approach is to block all non-administrative users from installing printer drivers. Or stop printing altogether (no joke, see more about it further down).

As at now there are two realistic options:

1 - Whenever the warning appears, call the admin to input administrative credentials to install the driver. Depending on your environment you may have hundreds or thousands of workstations, each with multiple printers, each requiring administrative attention.

2 - Roll out the following Registry value to practically undo Microsoft's security measures and revert back to the original behavior. Beware, this way you'll expose your network to the PrintNightmare threat:

Registry location: HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Windows NT\Printers\PointAndPrint
Name: RestrictDriverInstallationToAdministrators
Value: 0

More details in Microsoft's MSRC team's blog and in the KB5005652 (CVE-2021-34481) article.

Microsoft touts a 3rd option, disabling the Print Spooler service, practically going back to pre-printer era of making hard copies (a.k.a. pen and paper with your own two little hands). From the article:

Now that you know your risks and options, you can start working out your long-term corporate printing strategy.

Enjoy :-)

Friday, 5 March 2021

Stuck in Pending Reboot Cycle

Hello again,

Microsoft released a bunch of security updates on March 2, 2021 (KB5000871) addressing a number of Exchange server vulnerabilities. Details here. But that's not the point of this post.

Exchange admins, myself included, started rolling out the update. However in one instance a server was stuck in a "pending reboot" state:

The information for pendingreboot is stored in the Registry. Not wanting to rediscover the wheel, searched for an easy way out: a script which looks for entries that control pending reboots.

Came across Adam Bertram's article. He not only has a good list of Registry entries to check, but provided a PowerShell script also to make it easy.

Using his script I quickly found that the RebootPendinig Registry key was present. The script in its original form only shows whether it is pending reboot or not, but it doesn't show which key or value it is. I altered the script to show this detail also:

Checking in Registry Editor confirmed it:

Please note that it is an empty key. Its sheer presence will trigger the pending reboot condition.

For more on this key see this Microsoft Scripting Guy article.

Having rebooted the computer, it was safe to delete the key. But...:

The fix:

  1. Took ownership of the key. The original owner is System.
  2. Added myself with Full Access permissions.
  3. Delete the key.

Once the key was deleted, the patch installed just fine.

Until next time.

Wednesday, 21 August 2019

"Important" Update Bound to Break Your Exchange Server. Again.

Hi There,

I've done a routine manual patching of one of the Windows 2012 R2 servers, and, instead of hitting the Install button, as many of us do (they are all "Important" updates after all so why not just install them all, right?), I looked at what is going to be installed.

To my surprise I saw .Net Framework 4.8 on the list:

WHAAAAT??? .Net version upgrade as "Important"?

Yes, Microsoft is doing it again: they are shoving it down the throat without thinking of the consequences. They ruined a couple of Exchange servers about a year ago - more details here.

A word of advice:

  • Don't just blindly accept all updates just because they are listed as "Important".
  • Have your Exchange patched to the latest CU.
  • Consult the Exchange .Net supportability matrix.

The Exchange team has put up this warning on its .Net supportability matrix:

Specific for hybrid deployments, if you want to stay supported, you must run the latest, or the immediately previous release of Cumulative Updates / Update Rollups - see here. If you have, for instance, a Hybrid Exchange 2016 on CU12, hence supported (at the time of this writing), and don't want to update to CU13 just yet, then think again: a routine Windows Update exercise may bring your messaging system to its knees:

Thank you .Net team, yet another not-so-well-done upgrade.

Thursday, 18 July 2019

550 5.4.316 Message expired, connection refused (Socket error code 10061)

In a recent engagement of Exchange migration to O365 the client started experiencing random inbound delivery failures. The error in the NDR was that in the title:

Research pointed to a couple of articles that all suggest checking the firewall:

  • O365 sources blacklisted/quarantined by an over-zealous Fortigate IPS rule - https://pariswells.com/blog/research/office-365-failed-550-5-4-316-message-expired-connection-refusedsocket-error-code-10061
  • Microsoft's own experience and recommendation - https://docs.microsoft.com/en-us/office365/securitycompliance/mail-flow-intelligence-in-office-365
It turned out to be not a firewall, but a case of asymmetric routing. Close enough. A new device has been introduced into the customer's environment to set up a VPN with a sister company at around the same time when the first delivery error reports started to came in. Setting up the VPN affected routing, resulting in egress/ingress SMTP traffic to/from the same source took very different paths.

Once routing has been corrected, email started to flow normally again.

Till next time.

Saturday, 3 November 2018

New Office Build Causes Credentials Prompt Loop

Another day, another challenge.

We've just provisioned a brand new laptop with a brand new Office (latest build) for a user. Yet we could not configure a mail profile: it kept prompting for credentials, no matter what we did. We tried DOMAIN\samAccountName as well as UPN to no avail.

The user is in a hosted Exchange environment with a UPN of domain.local on Exchange 2013.

Interestingly other users worked well.

To cut the long story short, it looks like the newest Office build requires that the domain component of the user's UPN matches the domain component of the primary SMTP address. The user components need not match.

Here is what failed and what worked in my test:
  • DOMAIN\samAccountName FAILED
  • user@domain.local FAILED
  • UPN fully matches email address WORKED
  • domain components of UPN and email address match, user components are different WORKED
Office builds that displayed this behavior are 1809 and 1810. Older builds worked with any of the above combinations. In my troubleshooting I used the latest Office 365 suite, 16.0.11001.20064 at the time of this writing, which failed/worked as described above.

Is it a bug? Is it Microsoft silently pushing users to adhere to standards imposed by Office 365 EXO? Can't tell. The fact is that it looks like we are being forced to adopt consistency. Not necessarily a bad thing, although without adequate documentation on Microsoft's part it is going to cause some major headaches in a number of organizations.

Lessons learned: start sticking to de facto standards and best practice: match users' UPN to their primary SMTP address.

Have a nice weekend.

Friday, 27 July 2018

Microsoft Builds Licensing Engine into the Exchange HCW

Hi There,

I am critical when Microsoft bricks my servers with dodgy updates, but I also give them kudos when due. Happy to say that it is kudos time.

In its July 20, 2018 update of the HCW, Microsoft added a welcome feature. As you may already be aware, in a hybrid environment where there are no local mailboxes anymore and the on-prem Exchange server is there purely for administrative purposes, you're eligible for a free Hybrid license. If you didn't know then click here to find out more. It used to be a separate process, a separate tool, thus extra time and administration.

Not anymore. If you run the HCW on a new, unlicensesd server, the first thing you'll notice is a big red message telling you that you are running an Unlicensed Product, and the Next button is grayed out. Also, the Version information states that the server is a "Standard Evaluation Edition".

Don't freak out. If you look closer, you now get a link to license this server now:

Click the link and you'll be prompted to log on to your O365 tenant admin account:

The wizard then goes on to validate the environment, then obtain and install your free, brand spanking new Hybrid edition license:

Once the license is in, you are given access to the Next button, and the rest of the process is pretty much the same as before. As an added feature, you are also given a copy product key link which reveals not only the product key in clear text, but the entire PowerShell command that was used to install it also - it's there just as an FYI, the wizard did it all for you.

If you restart the HCW, you'll notice that Standard Evaluation version has been updated to Coexistence Edition:

Pretty cool, huh?

Till later,

Monday, 16 July 2018

.NET Framework 4.7.2 Breaks AAD Connect and Exchange

Hi There,

Time for a new post.

Microsoft made .Net Framework 4.7.2 available on Windows Update on 10 July 2018, just about a week ago. As an "Important / Recommended" update, it gets under the radar at many organizations where all "Important" updates are installed as default practice. .NET updates used to come as "Optional". This time, however, Microsoft deemed this update "Important" for whatever odd reason that escapes me.

Although Microsoft "strongly recommends" the installation of this update, reports have emerged that it doesn't play nicely with AAD Connect. and Exchange. Specifically, CPU utilization of the Microsoft.Identity.Health.AadSync.MonitoringAgent.Startup.exe process goes through the roof, grinding the server to a halt:

Secondly, Microsoft has not (yet) updated the Exchange server prerequisites to reflect support for .NET Framework 4.7.2 - see https://docs.microsoft.com/en-us/exchange/plan-and-deploy/system-requirements:

Sure enough, the update bricks the Exchange OWA and ECP portals too. After you log on, you get a pristine, white browser window, devoid from anything:

I thought OK, let's rebuild some virtual directories. Well, for that I need EMS - as long as it works. It fell flat too:

In fact, looking at the IIS logs, it becomes clear that pretty much everything has gone south.

As recovery steps, first I removed .Net 4.7.2 as some sources indicate on the Internet. Unfortunately that didn't fix the AAD Connect high CPU problem - it returned after an hour or so. And it certainly didn't fix the Exchange problem.

As far as Exchange is concerned, I tried the following:

  • Removed .Net 4.7.2
  • Removed and reinstalled .Net 4.7.1
  • Installed Exchange 2013 CU21 - the server was a tad outdated, on CU13

No joy. The screenshots above were taken after the recovery attempt.

My recommendation to you, dear reader, is to block the installation of .Net 4.7.2 for the time being. It is NOT an "important" update, no matter how much Microsoft would like you to believe.

The update can be blocked with a Registry setting, as documented at KB4342394.

I am in for rebuilding the Exchange server bricked by Microsoft's (not so) "important" .Net update. Thank you Mr. Microsoft, yet another .Net blunder to add to the list.

Happy patching!


Microsoft has come to its senses and re-published .NET Framework 4.7.2 where it belongs, under "Optional" updates.