Exchange 2013: High Availability – When Maintenance Might Be Necessary

NOTE: This post – drafted, composed, written, and published by me – originally appeared on and is potentially (c) Microsoft.

During the course of your on-premises environment, it may become necessary to take a production Exchange server out of rotation and perform maintenance on it (i.e.: replace memory, patching, reconfiguration, etc.). During this time, it will become necessary to prevent RemoteMonitoring from testing against the server you need to repair.

To understand this, we need to cover High Availability (HA) differences in Exchange 2013. Any Health Set that falls under ‘Global Monitoring’ has probes, monitors, and responders that are initialized at run-time on all Exchange 2013 servers in the organization. Any Health Set that falls under ‘Server Monitoring’ has static probes, monitors, and responders that are created locally on the server, only. Remote Monitoring falls under ‘Server Monitoring’ and has probes that test against other Exchange servers within the organization:

Get-ServerHealth -Server E15MBX -HealthSet RemoteMonitoring | FT -AutoSize

Server  State         Name                         TargetResource         HealthSetName    AlertValue ServerComponent
——  —–         —-                         ————–         ————-    ———- —————
E15MBX  NotApplicable HealthManagerObserverMonitor RemoteMonitoring Healthy    None

If you need to disable this probe while you’re performing maintenance on a server, so an escalation isn’t created, you can create a local server monitoring override for this purpose.

To do so, you’re going to need the Identity of the probe. You can perform the following command to find information regarding the probe:

Get-MonitoringItemIdentity -Server E15MBX -Identity RemoteMonitoring | Where{$_.ItemType -contains ‘Probe’}

RunspaceId     : b475c539-d2ba-4a28-9ad2-5cf0a18024ce
Server         : E15MBX
HealthSetName  : RemoteMonitoring
Name           : HealthManagerObserverProbe
TargetResource :
ItemType       : Probe
Identity       : RemoteMonitoring\HealthManagerObserverProbe\
IsValid        : True
ObjectState    : New

Once we’ve found the information regarding the probe, we can commence to setting the necessary override. To do so, we’ll use the following command:

Add-ServerMonitoringOverride -Server E15MBX -Identity ‘RemoteMonitoring\HealthManagerObserverProbe\‘ -ItemType Probe -PropertyName Enabled -PropertyValue 0 -Duration 60.00:00:00 -Confirm:$FALSE

The ‘PropertyValue’ is, typically, a bitwise flag; meaning that ‘0’ is off and ‘1’ is on. The ‘ItemType’ can be either a Probe, a Monitor, or a Responder. The ‘Duration’ must be 60 days or less.


To confirm the override has been set, we can check:

Get-ServerMonitoringOverride -Server E15MBX | FT -AutoSize

Identity                                                                    ItemType PropertyName PropertyValue
——–                                                                    ——– ———— ————-
RemoteMonitoring\HealthManagerObserverProbe\          Probe    Enabled      0

We can also verify when the expiration time is:

(Get-ServerMonitoringOverride -Server E15MBX | Where{$_.Identity -ilike ‘RemoteMonitoring\HealthManagerObserverProbe\’}).ExpirationTime
9/15/2013 1:29:59 PM

We can remove the override just as easily as we have set it:

Remove-ServerMonitoringOverride -Server E15MBX -Identity ‘RemoteMonitoring\HealthManagerObserverProbe\’ -ItemType Probe -PropertyName Enabled -Confirm:$FALSE

And just like that, Exchange 2013 is regularly testing against the remote server, again.

More information about Adding a Server Monitoring Override can be found here.

Exchange 2013: High Availability, FAST Search, and the Windows Registry

NOTE: This post – drafted, composed, written, and published by me – originally appeared on and is potentially (c) Microsoft.

One of the things not mentioned about High Availability (HA) and Database Availability Groups (DAGs) is that the automatic reseed feature is also used to maintain Exchange FAST Search indexes, independently, by HA.

HA utilizes a registry key to keep the Content Index (CI) states in a central locale for referencing. It queries this key because determining if the service is running on each node is approximately 30% of the cost of the ‘Get-MailboxDatabaseCopyStatus’ command. The registry key, itself, is located at: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus

You can query this registry key using the following syntax in PowerShell or Exchange Management Shell:

Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus

Assuming your mailbox server is not a member of a DAG, when you run the command you’ll get outputs that look something like the following:

{79853a57-6c9e-46a2-a5e3-70db036754f1} : 1,1,4294967297,2013-07-16 00:33:17Z,0,
{115d21d9-8b88-4574-b71e-5a7142daaf78} : 1,1,4294967297,2013-07-16 00:33:16Z,0,
{aab61984-7df3-412f-9c0b-f44bf0eaadbc} : 1,1,4294967297,2013-07-16 00:33:16Z,0,

Assuming your mailbox server is a member of a DAG, when you run the command you’ll get outputs that look something like the following:

{b03445e5-58c1-4f46-9d3b-3db2cb3ee3e3} : 1,1,2013-07-16 14:27:56Z,0,
{94689a79-5207-42a0-8ebc-50afe9fbfb52} : 4,7,2013-07-16 14:39:05Z,0,SPECIAL:E15-DAGMEMBER-1/3875/94689A79-5207-42A0-8EBC-50AFE9FBFB5212/Single
{1bfb37b0-9a9c-4821-9d03-cf0c7f8f2bba} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-5/3875/1BFB37B0-9A9C-4821-9D03-CF0C7F8F2BBA12/Single
{c77b6a95-719b-477c-be95-504f81b5376d} : 1,1,2013-07-16 14:14:02Z,0,
{1194860a-35c9-452e-8e87-b040eea1909c} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-2/3875/1194860A-35C9-452E-8E87-B040EEA1909C12/Single

There’s five points of information that make this relevant to issues with FAST Seach:

1.)    The initial {GUID} is the GUID of the database.

  1. There will be 1 entry per database on the DAG node.

2.)    ‘SPECIAL’ means that the CI is reseeding

  1. The source server, port, source database, and cell instance trail this value

3.)    The double numeric values (1,1; 4,7; 8,8; etc.) signify the state[s] of the CI

4.)    ‘Single’ references which type of cell we are dealing with.

5.)    The two digit number, after the second GUID, signifies the schema version of the index (in this case, the schema version is 12). You will not be able to find the database in question, if you use the second GUID and leave the schema version appended to it.


The first number in the series is the Index Status. The second number in the series is the error code. The codes break-down as follows:

Index Status:

Unknown = 0
Healthy = 1
Crawling =2
Failed = 3
Seeding = 4
Failed And Suspended = 5
Suspended = 6
Disabled = 7
Auto-Suspended = 8
Healthy And Upgrading = 9

Error Code:

1 Success
2 Internal Error
3 Crawling Database
4 DatabaseOffline
5 MapiNetworkError
6 Catalog Corruption
7 Seeding Catalog
8 Catalog Suspended
9 Catalog Reseed
10 Index Not Enabled
11 Catalog Excluded
12 Activation Preference skipped
13  Lag Copy skipped
14 Recovery Database Skipped
15 Fast Error
16 Service Not Running
17 Index Timestamp too Old

When a CI is reseeding, the status will – obviously – be ‘Seeding’ when you run ‘Get-MailboxDatabaseCopyStatus’. In order to see where the seeding source is, find the ‘ContentIndexSeedingSource’ value for that specific copy via ‘Get-MailboxDatabaseCopyStatus’ or you can query the registry key. Keep in mind, though, that the source port and cell are not exposed via ‘Get-MailboxDatabaseCopyStatus’ and are only found in the registry.



  • Scott Oseychik, Principal Escalation Engineer, Customer Service and Support
  • Microsoft Exchange FAST Team
  • Microsoft Exchange Sustained Engineering Team