Note 10/2022: This article also applies to Exchange Server 2019.
Introduction
Exchange Server 2013 introduced a new feature called Managed Availability, which is a built-in monitoring system with self-recovery capabilities. Managed Availability performs continuous tests (probes) that simulate end-user actions, to detect possible problems with Exchange components or their dependencies. If probes are failing, it performs gradual simple recovery actions to bring the affected component in healthy state. It uses special type of mailboxes, called Monitoring Mailboxes or health mailboxes, to simulate end-user kinds of tests. The life cycle of monitoring mailboxes is taken care entirely by Managed Availability components. In this post, we’ll see how Managed Availability takes care of monitoring mailboxes, what best practices to keep monitoring mailboxes happy are and some related troubleshooting.
Monitoring Mailboxes
Functioning of Managed Availability is implemented by Microsoft Exchange Health Manager Service running on each Exchange Server 2013 role. The Microsoft Exchange Health Manager Service is responsible for creating and maintaining monitoring mailboxes Let’s take a look at how the Health Manager creates them!
How do we create monitoring mailboxes?
The MS Exchange Health Manager Service runs in two processes, MSExchangeHMHost.exe and MSExchangeHMWorker.exe (let’s call it the HM worker). HM worker process, at the time of startup, checks for availability of monitoring mailboxes and creates monitoring mailboxes as needed. Starting with Exchange Server 2013 Cumulative Update 1, accounts for monitoring mailboxes are created under following container in the domain Exchange server resides:
<ADdomain>\Microsoft Exchange System Objects\Monitoring Mailboxes
Example: The logic HM worker process uses to detect and create monitoring mailboxes depends on Exchange Cumulative Update (CU) installed, Exchange role installed on the box and mailbox databases present. The Following logic was used to create monitoring mailboxes for Exchange Server 2013 servers between RTM to Cumulative Update 5:
- One monitoring mailbox per mailbox database copy, plus one for all of CAS servers.
Here’s an example of monitoring mailboxes created on Exchange Server 2013 SP1 server that hosts both CAS and mailbox role:
[PS] C:\>Get-Mailbox -Monitoring | ft displayname,whencreated
DisplayName WhenCreated
----------- -----------
HealthMailboxb285a119be6649b3a89574f078e947f5 11/10/2014 9:07:29 AM
HealthMailbox60d8a8d1285e41bfa5ce1ef1fb93d14e 11/10/2014 9:07:36 AM
The display name of the monitoring mailbox created for database copy contained the GUID of the mailbox database for which it was created. Example: The display name of the monitoring mailbox created for CAS server contained the GUID of the Exchange server for which it was created. Example: The following logic is used to create monitoring mailbox for Exchange Server 2013 Cumulative Update 6 and onwards:
- One monitoring mailbox for each mailbox database copy hosted on mailbox role, plus ten monitoring mailboxes for each CAS server.
The following naming convention is used for the display name of the monitoring mailbox created for a database:
“HealthMailbox-“ + host name of server + “-“ + database name.
At the startup, the HM worker checks for name of databases present on the server and then checks for presence of monitoring mailboxes with display name as explained above. If it doesn’t find a monitoring mailbox for a specific database, it creates a new monitoring mailbox. That means, if you rename DB1 to DB2, HM worker will create new monitoring mailbox for DB2 on next startup. And the following naming convention is used for display name of the monitoring mailbox created for a CAS server:
“HealthMailbox-“ + host name of the CAS server + “-001” through “010.”
We attempt to distribute the monitoring mailboxes created for CAS servers across the mailbox databases available. Let’s make this real: Imagine that you have only one Exchange server running Exchange 2013 CU6 or newer in the org (server is named EXCH1) that hosts both CAS and Mailbox role and has only one mailbox database, 11 monitoring mailboxes will be created. Example: The Health Manager Service will create more mailboxes according to the logic explained above as you go on adding more databases or server roles.
Password resets
HM Worker is responsible for maintaining the password for Monitoring mailboxes. HM worker uses a complex algorithm to generate password to be used for monitoring mailbox. The password for monitoring mailbox is reset under the following conditions:
- A new health mailbox is being created
- Each time HM Worker process starts and is not able to retrieve the existing password for the monitoring mailbox
- Any other scenario where HM Worker is not able to get hold of existing password for the monitoring mailbox
Best practices
Here are some best practices regarding management of user accounts associated with monitoring mailboxes as well as mailboxes themselves:
- Do not apply third party customized password policies to user accounts of monitoring mailboxes
- Exclude monitoring mailboxes from user account lockout policies
- Do not move the user accounts out from the Monitoring Mailboxes container
- Do not change the user account properties, like restricting password change etc.
- Do not change AD permission inheritance
- Since HM worker handles password resets for monitoring mailboxes, in a large environment, it is normal to see increased password reset traffic for monitoring mailbox accounts; note that doing one of the things above might increase the frequency of those resets
- Do not move the monitoring mailboxes between mailbox databases
- Do not apply mailbox quotas to monitoring mailboxes
- If applying a retention policy, ensure the data within the monitoring mailbox is retained for a minimum of 30 days before being deleted
If you see a mailbox size increasing significantly for specific monitoring mailbox; you can simply disable the specific mailbox. The HM worker will create a new mailbox at next startup.
Common tasks with monitoring mailboxes
How to list monitoring mailboxes
The Get-Mailbox command is provided with special parameter “-Monitoring” to list only the monitoring mailboxes. Here are some examples:
- To list all monitoring mailboxes present in the organization: Get-Mailbox –Monitoring
- To list monitoring mailboxes present on a database: Get-Mailbox –Monitoring -Database <database name>
However, the mailboxes listed above may not be associated with the server on which database is hosted. As explained in creation logic, the name of Client Access Server or the mailbox database is important in searching for monitoring mailbox associated and in creation of monitoring mailbox.
- Use the following command to list the monitoring mailboxes associated with a specific server:
Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-<servername>-*"}
Example:
Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-exch2-*"} | ft name,displayname,database
- Use the following command to list the monitoring mailbox associated with a specific database:
Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-<name of database>"}
Example:
Get-Mailbox -Monitoring | ?{$_.DisplayName -like "*-db2exch2"} | ft name,displayname,database,servername
Troubleshooting tips
Here are some troubleshooting methods for monitoring mailboxes.
How to re-create monitoring mailboxes (NOT considered regular maintenance!)
The mailbox database removal doesn’t cleanup the AD user account associated with monitoring mailboxes. This can then result in orphaned AD user accounts. This happens because of deny permission inherited on Monitoring Mailboxes container. KB article 3046530has details on this, as well as the workaround to resolve it. If there are too many orphaned monitoring mailbox accounts, you may want to re-create them. Steps: 1) Make sure “Monitoring Mailboxes” container is present
- Open Active Directory Users & Computers
- Click on View and select “Advanced Features”
- The Browse to Microsoft Exchange System Objects
- Verify the presence of the “Monitoring Mailboxes” container.
Example: If the Monitoring Mailboxes container is missing:
- Make sure you have Exchange Server 2013 CU1 or above installed.
- Perform PrepareAD with the Exchange Server 2013 version installed.
2) Stop the “Microsoft Exchange Server Health Manager” service on all Exchange Server 2013 servers. 3) Open Exchange Management Shell and use following command to disable existing health mailboxes:
Get-Mailbox -Monitoring | Disable-Mailbox
4) Go back to Active Directory users & computers, right click on domain and search for “HealthMailbox” 5) Delete the health mailbox user accounts. 6) Wait for AD replication or force AD replication. 7) Start the “Microsoft Exchange Server Health Manager” on all Exchange Server 2013 servers.
Bhalchandra Atre
Updates
- 6/2/15: Updated best practices to include information on aging out data via retention policies.