Exchange2013: Testing MRS Across A DAG

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

In Exchange 2013, it may become necessary to test the Mailbox Replication Service (MRS) against all members of a Database Availability Group (DAG). To do this, we can perform the following script:

#We obtain the member servers of the DAG
$Servers = ((Get-DatabaseAvailabilityGroup LAB-NAEX15-01).Servers).Name
#We perform a ‘foreach’ against each member
foreach($server in $Servers)
{

Write-Host -ForegroundColor Green $server.Name
Test-MRSHealth -Identity $server -MonitoringContext:$TRUE

}

When we perform the above script, we should receive output like the following:

PERSEPHONE
RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : ServiceCheck
Passed      : True
Message     : The Mailbox Replication Service is running.
Identity    : PERSEPHONE
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : RPCPingCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is responding to a RPC ping. Server version: 15.0.712.11 caps:3F.
Identity    : PERSEPHONE
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : QueueScanCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is scanning mailbox database queues for jobs. Last scan age: 00:14:22.0340000.
Identity    : PERSEPHONE
IsValid     : True
ObjectState : New

RunspaceId          : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Events              : {Source: MSExchange Monitoring MRSHealth
                      Id: 1000
                      Type: Success
                      Message: MRSHealth check passed.}
PerformanceCounters : {Object: MSExchange Monitoring MRSHealth
                      Counter: Last Scan Age (secs)
                      Instance: PERSEPHONE
                      Value: 862}

MINERVA
RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : ServiceCheck
Passed      : True
Message     : The Mailbox Replication Service is running.
Identity    : MINERVA
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : RPCPingCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is responding to a RPC ping. Server version: 15.0.712.11 caps:3F.
Identity    : MINERVA
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : QueueScanCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is scanning mailbox database queues for jobs. Last scan age: 00:03:35.9990000.
Identity    : MINERVA
IsValid     : True
ObjectState : New

RunspaceId          : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Events              : {Source: MSExchange Monitoring MRSHealth
                      Id: 1000
                      Type: Success
                      Message: MRSHealth check passed.}
PerformanceCounters : {Object: MSExchange Monitoring MRSHealth
                      Counter: Last Scan Age (secs)
                      Instance: MINERVA
                      Value: 215}

CHARON
RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : ServiceCheck
Passed      : True
Message     : The Mailbox Replication Service is running.
Identity    : CHARON
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : RPCPingCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is responding to a RPC ping. Server version: 15.0.516.29 caps:3F.
Identity    : CHARON
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : QueueScanCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is scanning mailbox database queues for jobs. Last scan age: 00:07:23.1520000.
Identity    : CHARON
IsValid     : True
ObjectState : New

RunspaceId          : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Events              : {Source: MSExchange Monitoring MRSHealth
                      Id: 1000
                      Type: Success
                      Message: MRSHealth check passed.}
PerformanceCounters : {Object: MSExchange Monitoring MRSHealth
                      Counter: Last Scan Age (secs)
                      Instance: CHARON
                      Value: 443}

AGAMEMNON
RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : ServiceCheck
Passed      : True
Message     : The Mailbox Replication Service is running.
Identity    : AGAMEMNON
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : RPCPingCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is responding to a RPC ping. Server version: 15.0.712.11 caps:3F.
Identity    : AGAMEMNON
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : QueueScanCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is scanning mailbox database queues for jobs. Last scan age: 00:07:51.9760000.
Identity    : AGAMEMNON
IsValid     : True
ObjectState : New

RunspaceId          : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Events              : {Source: MSExchange Monitoring MRSHealth
                      Id: 1000
                      Type: Success
                      Message: MRSHealth check passed.}
PerformanceCounters : {Object: MSExchange Monitoring MRSHealth
                      Counter: Last Scan Age (secs)
                      Instance: AGAMEMNON
                      Value: 471}

 

Exchange 2013: ‘Get-AgentLog’ Throws Exception When Run From The Front End

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

During the investigation of a not-so-recent case, it was discovered that Get-AgentLog – when run from a Front-End server – will throw an exception. What occurs is that when you attempt to run the command from EMS, you receive a notice about an exception and that a Watson dump is being generated; however, no Watson dump can be found on the server and there are no events in the Application or Crimson Logs:

Get-AgentLog -TransportService FrontEnd -StartDate ([System.DateTime]::Now).AddDays(-4) -EndDate ([System.DateTime]::Now)
WARNING: An unexpected error has occurred and a Watson dump is being generated: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
Specified cast is not valid.
    + CategoryInfo          : NotSpecified: (:) [Get-AgentLog], InvalidCastException
    + FullyQualifiedErrorId : System.InvalidCastException,Microsoft.Exchange.Management.AgentLog.GetAgentLog
    + PSComputerName        : <Server>

If you attempt to specify the location, you receive an error that the location cannot be found (note the fully qualified error):

VERBOSE: Connecting to <Server>
VERBOSE: Connected to <Server>.
Get-AgentLog -TransportService FrontEnd -StartDate ([System.DateTime]::Now).AddDays(-4) -EndDate ([System.DateTime]::Now) -Location “C:\exchsrvr\TransportRoles\Logs\FrontEnd\AgentLog\”
The location “C:\exchsrvr\TransportRoles\Logs\FrontEnd\AgentLog” does not exist. Please specify a valid file or directory to look for agent logs using the -Location parameter.
Parameter name: Location
    + CategoryInfo          : InvalidArgument: (:) [Get-AgentLog], ArgumentException
    + FullyQualifiedErrorId : [Server=<Server2>,RequestId=5b8d0c5b-6b0b-4b7d-ad94-008aab18485c,TimeStamp=11/27/2013 11:15:48 PM] 97084C54,Microsoft.Exchange.Management.AgentLog.GetAgentLog
    + PSComputerName        : <Server>

In the stack, you would see that the exception is a CLR exception, due to a conversion:

KERNELBASE!RaiseException+0x39
clr!RaiseTheExceptionInternalOnly+0x28b
clr!IL_Throw+0xe3
System_Management_Automation_ni!System.Management.Automation.LanguagePrimitives.ConvertStringToType(System.Object, System.Type, Boolean, System.Management.Automation.PSObject, System.IFormatProvider, System.Management.Automation.Runspaces.TypeTable)+0x126e67e

_message:00000000054d6638 (System.String) Length=89, String=”Cannot convert the “switchAttribute” value of type “System.String” to type “System.Type”.”
_HResult:0x80004002 (System.Int32)
errorId:0000000002d09190 (System.String) Length=27, String=”InvalidCastFromStringToType”

The cause: In Exchange 2013, commands are proxied from the Frontend Server (CAS) to a Backend Server (Mailbox). This is the reason that Server2 shows up in the Fully Qualified Error, when the command is run from Server.

The work-around is to open Windows PowerShell and add the PSSnapin (this bypasses command proxying). Then, you will receive the necessary output, sans any exceptions occurring.

Get-AgentLog -TransportService FrontEnd -StartDate ([System.DateTime]::Now).AddDays(-30) -EndDate ([System.DateTime]::Now) | FT -AutoSize

Timestamp             SessionId        IPAddress   MessageId P1FromAddress     P2FromAddresses    Recipients    Agent                Event              Action

———             ———        ———   ——— ————-     —————    ———-    —–                —–              ——

11/21/2013 8:47:25 PM 08D0B544432ACF98 <Redacted>             <Redacted>        <Redacted>        <Redacted>     Inbound Trust Agent   OnEndOfHeaders    ModifyHeaders

11/21/2013 8:47:25 PM 08D0B544432ACF98 <Redacted>             <Redacted>         <Redacted>         <Redacted>     Inbound Trust Agent   OnEndOfHeaders    ModifyHeaders

11/21/2013 8:48:27 PM 08D0B544432ACF9A <Redacted>             <Redacted>         <Redacted>         <Redacted>     Inbound Trust Agent   OnEndOfHeaders    ModifyHeaders

11/21/2013 8:48:27 PM 08D0B544432ACF9A <Redacted>             <Redacted>         <Redacted>         <Redacted>     Inbound Trust Agent   OnEndOfHeaders    ModifyHeaders

You can read more about this at these sites:

http://blogs.msdn.com/b/dvespa/archive/2013/02/21/install-error-transport-agents-exchange-2013-powershell.aspx

http://technet.microsoft.com/en-us/library/bb125175.aspx

O365: Manual Outlook Profile Creation in Wave 15

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

After the migration to Wave 15, Tenant Admins may find it necessary to create manual Outlook profiles; primarily, for troubleshooting and reproducing issues. It should be important to note that this is not a fully supported scenario but there are some steps available, if manual profile creation is required.

Server: mailbox.outlook.com
User: <User for the Profile>
Logon Network Security: Anonymous Authentication
Proxy Connection URL: outlook.office365.com
Principal Name for Certificates: msstd:outlook.com

Again, this is only for manual profile creation for Tenant Admins, as your CNAME should point to Autodiscover and it should take over the Outlook Profile Creation via Autoconfigure from the User’s ActiveDirectory attributes.

O365: On-Premises Transport Queuing Lessons Learned from CBL

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

I recently had a case where on-premises Transport had Queues that were ‘stuck’ going to the cloud. Specifically, they were going to Exchange Online Protection for routing from on-premises to cloud mailboxes. The on-premises transport servers were generating errors, in a unique way. For example, this is what we saw from the Queues in PowerShell:

(Get-TransportServer | Get-Queue | Where{$_.NextHopDomain -ilike ‘<cloud domain>’}  | Select LastError -Unique).LastError

451 4.4.0 Primary target IP address responded with: “451 5.7.3 Must issue a STARTTLS command first.” Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts.

This error is highly deceptive because it could be inferred that the client is not issuing a STARTTLS to the remote endpoint in the cloud; however, when we looked at the SMTP Send protocol logs, we could clearly see the inverse was true:

<Server>.mail.protection.outlook.com Microsoft ESMTP MAIL Service ready at Fri, 6 Dec 2013 21:54:51 +0000″,
2013-12-06T21:54:51.891Z,Outbound to Office 365,08D0C0DE40C1BF74,3,x.x.x.x:64526,x.x.x.x:25,>,EHLO <on-premises domain>,
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,4,x.x.x.x:64526,x.x.x.x:25,<,250-<Server>.mail.protection.outlook.com Hello [External I.P. Address of Client],
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,5,x.x.x.x:64526,x.x.x.x:25,<,250-SIZE 157286400,
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,6,x.x.x.x:64526,x.x.x.x:25,<,250-PIPELINING,
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,7,x.x.x.x:64526,x.x.x.x:25,<,250-DSN,
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,8,x.x.x.x:64526,x.x.x.x:25,<,250-ENHANCEDSTATUSCODES,
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,9,x.x.x.x:64526,x.x.x.x:25,<,250-AUTH,
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,10,x.x.x.x:64526,x.x.x.x:25,<,250-8BITMIME,
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,11,x.x.x.x:64526,x.x.x.x:25,<,250-BINARYMIME,
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,12,x.x.x.x:64526,x.x.x.x:25,<,250 CHUNKING,
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,13,x.x.x.x:64526,x.x.x.x:25,*,,Connector is configured to send mail only over TLS connections and remote doesn’t support TLS
2013-12-06T21:54:52.001Z,Outbound to Office 365,08D0C0DE40C1BF74,14,x.x.x.x:64526,x.x.x.x:25,>,QUIT,

Ah, so the STARTTLS command is not available from the remote endpoint. Since TLS is required for this send connector, this is why we’re queuing.

Let’s double-check our math and our sanity, really quick. I use nslookup to query for the MX record of the cloud domain and telnet to port 25 on one of the I.P. addresses. So far, so good. I issue ‘EHLO <on-premises domain>’ and get this response:

<I.P. Address> – 220 <Server>.mail.protection.outlook.com Microsoft ESMTP MAIL Service ready at Fri, 6 Dec 2013 22:01:59 +0000
EHLO <on-premises domain> 250-<Server>.mail.protection.outlook.com Hello [External I.P. Address of Client]
250-SIZE 157286400
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-AUTH
250-8BITMIME
250-BINARYMIME
250 CHUNKING

Sure enough, ‘STARTTLS’ isn’t an available ESMTP verb. O.k., but why?

Well, as it turns out, we use public block lists and this company was cited by CBL for RWBL and XBL; specifically, Pushdo was being used to perform a DDoS against port 80 on webservers from an infected machine behind their public I.P. address.

As soon as they were listed, we removed STARTTLS (and other ESMTP verbs).

The customer discovered it was probably a visitor (CBL lists time of last activity) and as it had been over 8 hours, delisted themselves. Immediately, TLS was available for use, again.