Server2012: Adding DAG Member Results in ‘The fully qualified domain name for node ‘EMEA-SWEEX15-01′ could not be found.’

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

To get around this, add the permissions by following this article for pre-staging the Cluster Name Object for a Database Availability Group. If you still receive the error, modify the ‘dNSHostName’ attribute to reflect the FQDN of your DAG, as referenced below from my lab (suffix removed from screenshot):

C# + EWS: Autodiscover Test (Exchange and O365)

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

In times of troubleshooting client-side issues, it may become necessary to query for the autodiscover response the user is receiving from either Exchange on-premises or Exchange in O365 – or, in the case of a redirection, both on-premises and O365 Exchange. This is a sample C#.NET Console Application, which will query for the Autodiscover response and use the TraceListener class to write the response to files.

There are two things the code doesn’t take into account for:

1. The condition wherein the user’s SMTP address and UPN are different.
2. ALL of the possible returns from Autodiscover for the UserSettings.

 

Thus, I have included the the source code for two reasons:

1. Promotion of writing .NET programs for both on-premises Exchange and O365 Exchange.
2. Customization of both the UserSettings one is targeting and the target delivery folder for which the files should be saved.

 

If you have any problems, questions, or concerns, feel free to reach out to me and I’ll try to address them as soon as possible.

The source code can be found here: http://gallery.technet.microsoft.com/C-EWS-Autodiscover-Test-870b4a8e

Server 2012: Using PowerShell to Check for KB2878635

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

In the course of an investigation, it became necessary to write a PowerShell script to check the current file version of the kernelbase.dll. I needed to compare it against the file version of a KB that was pushed for better resiliency (http://support.microsoft.com/kb/2878635).

The script uses some basic logic to get the file version and compare it with the known published version of the file, at the time the KB was published, and determine if we were below, at, or above that file version. I’m including links to the MSDN articles that helped me to write the script.

$FileVer = ([System.Diagnostics.FileVersionInfo]::GetVersionInfo(“C:\Windows\System32\kernelbase.dll”)).FileVersion
$pattern = ” ”
$intVer = [Regex]::Split($FileVer,$pattern)
$intFileVer = $intVer[0]
[version]$DLLVersion = $intFileVer
[version]$Target = “6.2.9200.20712”
$DLLVersion.CompareTo($Target)

For the return values, here’s the table from MSDN:

Return value Meaning
Less than zero The current Version object is a version before value.
Zero The current Version object is the same version as value.
Greater than zero The current Version object is a version subsequent to value.

O365/Exchange 2013: Obtaining Move Report After The Move Request Has Been Removed

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

Often times, we will get an issue, where we need to check if any issues have occurred due to a move request but the move request (and seemingly any information related to it) has been removed.

Luckily, we also write/log this information for later retrieval. We can obtain the log and place it in a text file on the desktop (assuming the ‘~’ signifies the default home directory in PowerShell) via the following method:

((Get-MailboxStatistics <user alias OR smtp> -IncludeMoveHistory -IncludeMoveReport).MoveHistory).Report | Out-File C:\Users\johnbai\Desktop\report.tx

The report will need some clean-up, for readability-sake. I have attached a sample from my O365 tenant for reference.

report.txt

Fermion: A Side-Project for Administration

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

NOTICE: Sample programs in this blog are not supported under any Microsoft standard support program or service. The sample program is provided AS IS without warranty of any kind. Microsoft disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the program and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the programs be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the programs or documentation, even if Microsoft has been advised of the possibility of such damages.

NOTE:
============================================================================
To prevent errors on the local system, PMU (Performance Monitor Users group) must be enabled on the local machine
GOTO: http://blogs.msdn.com/b/bclteam/archive/2006/09/08/746900.aspx
============================================================================

I wanted to share a program that’s been in beta development, for some time now, and – possibly – get feedback about it. Essentially, the program was meant to be a purposeful tool for small to medium size companies that did not use SCOM to monitor the performance of their systems.

An example scenario: An Exchange administrator receives a call from the Helpdesk, saying that users are noticing performance problems on the system (mailflow, access, etc.). The Exchange Administrator can target a specific machine and check the targeted performance counters in a single dashboard layout (in this case, a WPF form).

I, currently, have ambitions to add targeted counters for Exchange 2013 (currently, the counters are incomplete), SQL (all versions), SharePoint (all versions), and Lync (all versions) servers.

Here is a list of the current known issues I’m working on (when I can):

1. Specific objects have borders in Windows 7, only.
2. The checkmark boxes do not stay selected in Windows Server 2008 R2 (without desktop experience enabled).
3. The data pumper does not cease, at first.
4. The DNS method throws an exception in Windows7, despite finding the object in DNS.

So, now that you’ve read all of this, here’s the installer. (Currently, version 1.0.10.21)

Contributing credits – David Herraiz Diaz

Fermion.zip

Exchange 2013: eDiscovery Changes

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

With the release of Exchange 2013, there are some changes that are relevant to eDiscovery; whether it be for In-Place Holds or Litigation Queries to export to the Discovery Mailbox. Most notably, eDiscovery/Exchange Search does not support AQS – it switched to KQL. KQL is supported in the SearchQuery parameter (Keywords box in the Exchange Admin Center). However, Outlook still uses AQS.

Using KQL, we can perform searches that are beneficial to the eDiscovery and will save time/money/resources, without the need to invoke a third-party to process the data for you.

For example, if I perform a query for any messages that only have a word document as an attachment, I get the two messages I expect to find.

If perform the same query but, this time, define a subject or keyword I’m after, the messages are excluded because the primary rule hasn’t been met.

If I perform a third query with words that exist in the document (but not in the document name), these documents will return in my query, as well.

There is a limitation to the number of mailboxes that can be searched and it is 5,000*. Any number beyond this and the specified query will return the following error: An unknown error occurred on the search server. Please contact your administrator for assistance. The message from the search server is ‘The search exceeded the maximum number of mailboxes that can be searched at a time. Please try searching less than 5000 mailboxes.’.

*The maximum number of mailboxes that you can search can be changed in on-premises Exchange 2013. You can use the Set-ThrottlingPolicy command with the DiscoveryMaxMailboxes parameter to do so but this may come at a negative impact to performance.

As Exchange now uses the FAST Search index, we can query for what documents haven’t been processed and why. For example, if I what to query for the error where the document parser encountered a processing error, I would use the following command in Exchange Management Console:

Get-FailedContentIndexDocuments Administrator -ErrorCode 7 | FT -AutoSize

DocID Database                Mailbox       Subject           Description
—– ——–                ——-       ——-           ———–
3462  LAB-NAEX15-01 Store 002 Administrator Binaries Test     The document parser encountered a processing error.
3464  LAB-NAEX15-01 Store 002 Administrator FW: Binaries Test The document parser encountered a processing error.

Using this I can see what, precisely, caused the document to not be indexed:

$errorSevens = Get-FailedContentIndexDocuments Administrator -ErrorCode 7
$errorSevens[0].AdditionalInfo
 301002 Error parsing document ‘exchange://localhost/Attachment/34eb02b4-3bc6-4163-a40d-2587faa9e0db/135d5536-d180-4198-9ba8-574b53df8206/e08d777e-e710-4407-a53d-1f57a4a58d79/a654efa1-bb87-426a-aaca-9866be73
3ccd/438086667654.0/System.Data.dll’. Document has an undetectable format and will not be parsed. 301002 Error parsing document ‘exchange://localhost/Attachment/34eb02b4-3bc6-4163-a40d-2587faa9e0db/135d5536-
d180-4198-9ba8-574b53df8206/e08d777e-e710-4407-a53d-1f57a4a58d79/a654efa1-bb87-426a-aaca-9866be733ccd/438086667654.1/mscorlib.dll’. Document has an undetectable format and will not be parsed.

In this case, the documents are binaries attached to the email for testing in regards to another issue. FAST Search cannot reverse-engineer binaries, so it is safe to assume that these files aren’t necessary for my eDiscovery purposes. See here for a list of formats that Exchange FAST Search can index.

The error code enumerations are as follows:

0 – No problems.
1 – An error has occurred.
2 – A timeout has occurred.
3 – The message was not processed in a timely manner.
4 – The mailbox was offline.
5 – The attachment limit was reached.
6 – The item is only partially indexed.
7 – The document parser encountered a processing error.
8 – The document annotations aren’t valid.
9 – The document is suspected of being unable to be processed.
10 – The document processing failed due to a Rights Management error.
11 – The Store Session is not available.
12 – The mailbox is quarantined.
13 – The mailbox is locked.
14 – The operation is not supported.
15 – Search can’t sign in to the mailbox.
16 – Body conversion failed.

O365: Testing Problems With RpcProxy for Outlook Anywhere Migrations

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

Often times, it may be necessary to test the RpcProxy URL to verify connectivity to your servers. There are various reasons for doing this but the primary is to verify if the connectivity problem isn’t limited to a problem with Exchange in O365.

RpcProxy is a compound URL, meaning it comprises different parts to make the whole. Let’s use this example URL and I’ll break down what I mean:

https://mail.contoso.com/rpc/rpcproxy.dll?MailboxServer.contoso.com:6001

So, the easiest one is the ‘mail.contoso.com’ as it’s the vanity name for external access to the Exchange environment. We follow that with the rpcproxy path in IIS, ‘/rpc/rpcproxy.dll’. We use a delimiter ‘?’ between the vanity name and the server and this is to pass the server we want to hit in the backend (in 2007, this will be your node name). Then, we have the server name ‘MailboxServer.contoso.com’, followed by one of the Outlook Anywhere ports: 6001, 6002, or 6004.

How do we test it? Easy! Use your favorite web browser and browse to the target destination. IIS should require you to authenticate (use your on-premises credentials you use for the migration) and you should get a blank page.

How do we know it fails? Well, the first sign would be a HTTP 400 error. If you have ISA or a TMG, you might get 503 or 500 errors; you may also get “The server denied the specified Uniform Resource Locator (URL). Contact the server administrator.” The key here is: is the error reproducible from other sources and what is the error returned, if any?

Using PowerShell and .NET to Construct a DirectorySearcher

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

PowerShell and .NET are very interoperable and can help to save time, when you’re performing generic, every day tasks. For example, let’s say that I want to create a Date-Time value that’s thirty minutes ago and one for right now, we can do this in one fell swoop in Exchange Management Shell (read: PowerShell):

v2: Get-MessageTrackingLog -Start [System.DateTime]::Now.AddMinutes(-30) -End [System.DateTime]::Now

v3: Get-MessageTrackingLog -Start ([System.DateTime]::Now).AddMinutes(-30) -End ([System.DateTime]::Now)

As you can see, the syntax changes slightly but the methods are the same because they derive from the same .NET class.

Below, I demonstrate constructing a DirectorySearcher object for a specific case. The final script is published on OneScript, here, but I wanted to demonstrate that we can use PowerShell + .NET to solve some complex problems in a rather easy way.

============================================================================================================================================

In rare cases, removal of an Exchange Server from the forest doesn’t go according to plan and, without Exchange Management Shell (EMS), finding servers via Active Directory might be a bit of a pain-point. Enter DirectorySearcher.

Here is an example, finding Exchange 2013 mailbox servers in the forest:

$colMBX = @()
$CurrentDomain = [System.DirectoryServices.ActiveDirectory.Domain]::GetComputerDomain()
$ForestName = $CurrentDomain.Forest.Name
$ForestDC = $ForestName.Replace(“.”,”,DC=”)
$ForestLDAP = [ADSI]”LDAP://CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=$ForestDC”
$orgName = $ForestLDAP.psbase.children | where {$_.objectClass -eq ‘msExchOrganizationContainer’}
$Path = “LDAP://CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=” + $orgName.Name + “,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=” + $ForestDC
$Searcher = New-Object System.DirectoryServices.DirectorySearcher
$Searcher.Filter = ‘(&(&(&(objectCategory=msExchExchangeServer)(msExchCurrentServerRoles:1.2.840.113556.1.4.803:=54)(serialNumber=*15*))))’
$Searcher.PageSize = 10000
$Searcher.SearchScope = “OneLevel”
$Searcher.PropertiesToLoad.Add(“Name”) | Out-Null
$Searcher.SearchRoot = $Path
$ServerResult = $Searcher.FindAll()
foreach ($result in $ServerResult)
{
       $colMBX += $result.Properties.name[0]
}
$colMBX

You’ll notice that in the filter we’re using the numeric value ‘1.2.840.113556.1.4.803’ between the attribute and the value we’re seeking. This OID is an Extensible Matching Rule for the bitwise operator AND, which may also be referred to as ‘LDAP_MATCHING_RULE_BIT_AND‘. It is not required for use in your filter but does follow RFC convention.

In Exchange 2013, for Mailbox servers we can use the value ’54’ to search and for CAS servers we can use the value ‘16385’.

To explain the values, we can demonstrate via table:

Server role Role value
Mailbox role 2
Client Access role (CAS) 4
Unified Messaging role 16
Hub Transport role 32

The Mailbox role now has the previous roles in one server, so 2 + 4 + 16 + 32 = 54.

You can read more on PowerShell + DirectorySearcher here: http://technet.microsoft.com/en-us/library/ff730967.aspx

Exchange2013: Testing MRS Across A DAG

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

In Exchange 2013, it may become necessary to test the Mailbox Replication Service (MRS) against all members of a Database Availability Group (DAG). To do this, we can perform the following script:

#We obtain the member servers of the DAG
$Servers = ((Get-DatabaseAvailabilityGroup LAB-NAEX15-01).Servers).Name
#We perform a ‘foreach’ against each member
foreach($server in $Servers)
{

Write-Host -ForegroundColor Green $server.Name
Test-MRSHealth -Identity $server -MonitoringContext:$TRUE

}

When we perform the above script, we should receive output like the following:

PERSEPHONE
RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : ServiceCheck
Passed      : True
Message     : The Mailbox Replication Service is running.
Identity    : PERSEPHONE
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : RPCPingCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is responding to a RPC ping. Server version: 15.0.712.11 caps:3F.
Identity    : PERSEPHONE
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : QueueScanCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is scanning mailbox database queues for jobs. Last scan age: 00:14:22.0340000.
Identity    : PERSEPHONE
IsValid     : True
ObjectState : New

RunspaceId          : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Events              : {Source: MSExchange Monitoring MRSHealth
                      Id: 1000
                      Type: Success
                      Message: MRSHealth check passed.}
PerformanceCounters : {Object: MSExchange Monitoring MRSHealth
                      Counter: Last Scan Age (secs)
                      Instance: PERSEPHONE
                      Value: 862}

MINERVA
RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : ServiceCheck
Passed      : True
Message     : The Mailbox Replication Service is running.
Identity    : MINERVA
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : RPCPingCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is responding to a RPC ping. Server version: 15.0.712.11 caps:3F.
Identity    : MINERVA
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : QueueScanCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is scanning mailbox database queues for jobs. Last scan age: 00:03:35.9990000.
Identity    : MINERVA
IsValid     : True
ObjectState : New

RunspaceId          : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Events              : {Source: MSExchange Monitoring MRSHealth
                      Id: 1000
                      Type: Success
                      Message: MRSHealth check passed.}
PerformanceCounters : {Object: MSExchange Monitoring MRSHealth
                      Counter: Last Scan Age (secs)
                      Instance: MINERVA
                      Value: 215}

CHARON
RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : ServiceCheck
Passed      : True
Message     : The Mailbox Replication Service is running.
Identity    : CHARON
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : RPCPingCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is responding to a RPC ping. Server version: 15.0.516.29 caps:3F.
Identity    : CHARON
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : QueueScanCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is scanning mailbox database queues for jobs. Last scan age: 00:07:23.1520000.
Identity    : CHARON
IsValid     : True
ObjectState : New

RunspaceId          : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Events              : {Source: MSExchange Monitoring MRSHealth
                      Id: 1000
                      Type: Success
                      Message: MRSHealth check passed.}
PerformanceCounters : {Object: MSExchange Monitoring MRSHealth
                      Counter: Last Scan Age (secs)
                      Instance: CHARON
                      Value: 443}

AGAMEMNON
RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : ServiceCheck
Passed      : True
Message     : The Mailbox Replication Service is running.
Identity    : AGAMEMNON
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : RPCPingCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is responding to a RPC ping. Server version: 15.0.712.11 caps:3F.
Identity    : AGAMEMNON
IsValid     : True
ObjectState : New

RunspaceId  : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Check       : QueueScanCheck
Passed      : True
Message     : The Microsoft Exchange Mailbox Replication service is scanning mailbox database queues for jobs. Last scan age: 00:07:51.9760000.
Identity    : AGAMEMNON
IsValid     : True
ObjectState : New

RunspaceId          : b14e1547-c215-4dbb-8ee1-a3c4978ca7c3
Events              : {Source: MSExchange Monitoring MRSHealth
                      Id: 1000
                      Type: Success
                      Message: MRSHealth check passed.}
PerformanceCounters : {Object: MSExchange Monitoring MRSHealth
                      Counter: Last Scan Age (secs)
                      Instance: AGAMEMNON
                      Value: 471}

 

Exchange 2013: ‘Get-AgentLog’ Throws Exception When Run From The Front End

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

During the investigation of a not-so-recent case, it was discovered that Get-AgentLog – when run from a Front-End server – will throw an exception. What occurs is that when you attempt to run the command from EMS, you receive a notice about an exception and that a Watson dump is being generated; however, no Watson dump can be found on the server and there are no events in the Application or Crimson Logs:

Get-AgentLog -TransportService FrontEnd -StartDate ([System.DateTime]::Now).AddDays(-4) -EndDate ([System.DateTime]::Now)
WARNING: An unexpected error has occurred and a Watson dump is being generated: Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
Specified cast is not valid.
    + CategoryInfo          : NotSpecified: (:) [Get-AgentLog], InvalidCastException
    + FullyQualifiedErrorId : System.InvalidCastException,Microsoft.Exchange.Management.AgentLog.GetAgentLog
    + PSComputerName        : <Server>

If you attempt to specify the location, you receive an error that the location cannot be found (note the fully qualified error):

VERBOSE: Connecting to <Server>
VERBOSE: Connected to <Server>.
Get-AgentLog -TransportService FrontEnd -StartDate ([System.DateTime]::Now).AddDays(-4) -EndDate ([System.DateTime]::Now) -Location “C:\exchsrvr\TransportRoles\Logs\FrontEnd\AgentLog\”
The location “C:\exchsrvr\TransportRoles\Logs\FrontEnd\AgentLog” does not exist. Please specify a valid file or directory to look for agent logs using the -Location parameter.
Parameter name: Location
    + CategoryInfo          : InvalidArgument: (:) [Get-AgentLog], ArgumentException
    + FullyQualifiedErrorId : [Server=<Server2>,RequestId=5b8d0c5b-6b0b-4b7d-ad94-008aab18485c,TimeStamp=11/27/2013 11:15:48 PM] 97084C54,Microsoft.Exchange.Management.AgentLog.GetAgentLog
    + PSComputerName        : <Server>

In the stack, you would see that the exception is a CLR exception, due to a conversion:

KERNELBASE!RaiseException+0x39
clr!RaiseTheExceptionInternalOnly+0x28b
clr!IL_Throw+0xe3
System_Management_Automation_ni!System.Management.Automation.LanguagePrimitives.ConvertStringToType(System.Object, System.Type, Boolean, System.Management.Automation.PSObject, System.IFormatProvider, System.Management.Automation.Runspaces.TypeTable)+0x126e67e

_message:00000000054d6638 (System.String) Length=89, String=”Cannot convert the “switchAttribute” value of type “System.String” to type “System.Type”.”
_HResult:0x80004002 (System.Int32)
errorId:0000000002d09190 (System.String) Length=27, String=”InvalidCastFromStringToType”

The cause: In Exchange 2013, commands are proxied from the Frontend Server (CAS) to a Backend Server (Mailbox). This is the reason that Server2 shows up in the Fully Qualified Error, when the command is run from Server.

The work-around is to open Windows PowerShell and add the PSSnapin (this bypasses command proxying). Then, you will receive the necessary output, sans any exceptions occurring.

Get-AgentLog -TransportService FrontEnd -StartDate ([System.DateTime]::Now).AddDays(-30) -EndDate ([System.DateTime]::Now) | FT -AutoSize

Timestamp             SessionId        IPAddress   MessageId P1FromAddress     P2FromAddresses    Recipients    Agent                Event              Action

———             ———        ———   ——— ————-     —————    ———-    —–                —–              ——

11/21/2013 8:47:25 PM 08D0B544432ACF98 <Redacted>             <Redacted>        <Redacted>        <Redacted>     Inbound Trust Agent   OnEndOfHeaders    ModifyHeaders

11/21/2013 8:47:25 PM 08D0B544432ACF98 <Redacted>             <Redacted>         <Redacted>         <Redacted>     Inbound Trust Agent   OnEndOfHeaders    ModifyHeaders

11/21/2013 8:48:27 PM 08D0B544432ACF9A <Redacted>             <Redacted>         <Redacted>         <Redacted>     Inbound Trust Agent   OnEndOfHeaders    ModifyHeaders

11/21/2013 8:48:27 PM 08D0B544432ACF9A <Redacted>             <Redacted>         <Redacted>         <Redacted>     Inbound Trust Agent   OnEndOfHeaders    ModifyHeaders

You can read more about this at these sites:

http://blogs.msdn.com/b/dvespa/archive/2013/02/21/install-error-transport-agents-exchange-2013-powershell.aspx

http://technet.microsoft.com/en-us/library/bb125175.aspx