Exchange: Recovering a Database from Dirty Shutdown (the Easy Way)

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

In testing (and production), we invariably run into problems where the database will be in a ‘bad state’. In my case, I pulled Store service out from under it (disabled and stopped the service) and this caused the database to go into dirty shutdown:

eseutil.exe /mh “C:\Databases\EMEA-SWEEX15-01 Store 001\EMEA-SWEEX15-01 Store 001.EDB”

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode…
         Database: C:\Databases\EMEA-SWEEX15-01 Store 001\EMEA-SWEEX15-01 Store 001.EDB

DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x65021d61
  Actual Checksum: 0x65021d61

Fields:
        File Type: Database
         Checksum: 0x65021d61
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,20
 Engine ulVersion: 0x620,20
Created ulVersion: 0x620,20
     DB Signature: Create time:05/05/2014 13:08:25.805 Rand:306694709 Computer:
         cbDbPage: 32768
           dbtime: 10263756 (0x9c9ccc)
            State: Dirty Shutdown
     Log Required: 12131-12194 (0x2f63-0x2fa2)
    Log Committed: 0-12195 (0x0-0x2fa3)
   Log Recovering: 12195 (0x2fa3)
  GenMax Creation: 10/14/2014 07:55:56.052
         Shadowed: Yes
       Last Objid: 56187
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00.000
 Old Repair Count: 0
  Last Consistent: (0x263B,8B,19C)  08/28/2014 08:55:57.153
      Last Attach: (0x263C,2,268)  08/28/2014 08:55:57.294
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00.000
    Last ReAttach: (0x2DEA,2,0)  10/11/2014 04:46:44.436
             Dbid: 1
    Log Signature: Create time:05/05/2014 13:08:25.649 Rand:2111075525 Computer:
       OS Version: (6.2.9200 SP 0 NLS ffffffff.ffffffff)

Previous Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Copy Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Differential Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: none
Old bad Checksum Error Count: none

  Last checksum finish Date: 10/14/2014 08:02:23.469
Current checksum start Date: 00/00/1900 00:00:00.000
      Current checksum page: 0

Operation completed successfully in 0.187 seconds.

Using eseutil (and assuming the log is still available) we can recover from such a state like the following:

eseutil /r “E01” /l “C:\Databases\EMEA-SWEEX15-01 Store 001\Logs” /d “C:\Databases\EMEA-SWEEX15-01 Store 001”

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode…
    Logfile base name: E01
            Log files: C:\Databases\EMEA-SWEEX15-01 Store 001\Logs
         System files: <current directory>
   Database Directory: C:\Databases\EMEA-SWEEX15-01 Store 001

Performing soft recovery…
                      Restore Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|
          ……………………………………………

Operation completed successfully in 13.140 seconds.

And, once we’ve completed said step, we can verify if the database is – indeed – in clean shutdown:

eseutil.exe /mh “C:\Databases\EMEA-SWEEX15-01 Store 001\EMEA-SWEEX15-01 Store 001.EDB”

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode…
         Database: C:\Databases\EMEA-SWEEX15-01 Store 001\EMEA-SWEEX15-01 Store 001.EDB

DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x758dd2a3
  Actual Checksum: 0x758dd2a3

Fields:
        File Type: Database
         Checksum: 0x758dd2a3
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,20
 Engine ulVersion: 0x620,20
Created ulVersion: 0x620,20
     DB Signature: Create time:05/05/2014 13:08:25.805 Rand:306694709 Computer:
         cbDbPage: 32768
           dbtime: 13412964 (0xccaa64)
            State: Clean Shutdown
     Log Required: 0-0 (0x0-0x0)
    Log Committed: 0-0 (0x0-0x0)
   Log Recovering: 0 (0x0)
  GenMax Creation: 00/00/1900 00:00:00.000
         Shadowed: Yes
       Last Objid: 66933
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00.000
 Old Repair Count: 0
  Last Consistent: (0x2FA4,1,31)  10/14/2014 08:10:54.954
      Last Attach: (0x263C,2,268)  08/28/2014 08:55:57.294
      Last Detach: (0x2FA4,1,31)  10/14/2014 08:10:54.954
    Last ReAttach: (0x2DEA,2,0)  10/11/2014 04:46:44.436
             Dbid: 1
    Log Signature: Create time:05/05/2014 13:08:25.649 Rand:2111075525 Computer:
       OS Version: (6.2.9200 SP 0 NLS ffffffff.ffffffff)

Previous Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Copy Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Differential Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: none
Old bad Checksum Error Count: none

  Last checksum finish Date: 10/14/2014 08:02:23.469
Current checksum start Date: 00/00/1900 00:00:00.000
      Current checksum page: 0

Operation completed successfully in 0.235 seconds.

And, now, the database is safe to mount, again.

C#: Returning an Object via System.Threading.Tasks.Task

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

In programming something (in EWS), discovered the joys of what I believe was ‘lock contention’. To explain, first we need to cover Tasks.

Async Tasks (which call ‘await’ against other tasks) will only perform the action requested, if you do not specify a managed object to handle. For example, if I call an async for a string and do not specify that I am expecting an object in return, this is what the code will look like (and will introduce our problem):

string noDataReturns = await System.Threading.Tasks.Task.Run(() => this.GetMeTheStringAsync().Result);

private async Task GetMeTheStringAsync()

{

return await System.Threading.Tasks.Task.Run(() => this.GetMeTheString());

}

private string GetMeTheString()

{

return “This is not the string you’re looking for.”;

}

If, instead, I tell the task what object I expect to get in return, when the task completes, the result will be object that is passed back – instead of the result of the task:

string dataReturns = await System.Threading.Tasks.Task<string>.Run(() => this.GetMeTheStringAsync().Result);

private async Task<string> GetMeTheStringAsync()

{

return await System.Threading.Tasks.Task<string>.Run(() => GetMeTheString());

}

private string GetMeTheString()

{

return “This is the string you’re looking for.”;

}

This is probably common-place parlance in the .NET developer’s world but I had discovered this when I was spawning a Task that would return a DataTable object but did not specify that I expecting an object in return. As such, the windows form was rendered useless/locked, as I placed the DataTable’s data source as the result of the Task but there was no data to put in the data table.

 

C#: Returning ADSI COM Properties of a User Object (Or Any Object, Really)

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

A few months ago, I was working on a project to query for specific AD properties of a user-object programmatically. I knew the properties I wanted to return and I had worked with returning properties in ExBPA/HRC. I hadn’t, however, found the joys of receiving a System.__COM object in any of my returns until now.

To explain this, you should familiarize yourself with ADSI (Active Directory Service Interfaces), if you’ve never been exposed to it, before. As the previous link points out, ADSI is a COM (Component Object Model), and – as such – we will get a return of ‘System.__COM’, if we fail to pull the data in the expected format.

So, how was I to return the data from the COM? Well, as it turns out, there’s ‘deprecated’ documentation which helped me farther than any blog, technical bulletin, or reference did. You can find it, here. (Apologies, apparently all of the English versions have been pulled because it was deprecated.)

Using some ingenuity and a little luck, I found the answer to my problem: return the value I’m querying for as ‘ActiveDs.IADsLargeInteger’.

So, armed with this knowledge, I could now pull the properties I wanted to return in usable context, like the following:

ActiveDs.IADsLargeInteger recipientTypeDetails = (ActiveDs.IADsLargeInteger)userResult.Properties[“msExchRecipientTypeDetails”].Value as ActiveDs.IADsLargeInteger;

Once I had the data ‘out’, as it were, I could try to manipulate it:

long recipTypeDetails = recipientTypeDetails.LowPart;

 

I realize this doesn’t mean much to some people (maybe a lot) but I figured if I could save one person the hours it took me to find the answer, then it was well worth the time in writing this.

Happy coding!

ExRCA: Analyzing Messages via Headers

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

ExRCA (https://testconnectivity.microsoft.com/ or http://www.exrca.com/ [Redirects to the former.]) has a feature that you can use to analyze where message delays may have occurred, aptly named ‘Message Analyzer’.

You’ll get a table with each hop that the message passed through and the time between each, like the following:

You can also get the Analyzer installed as an App for Office.

Exchange 2013: MailboxTransportMBTDeliveryPercentPermanentFailedMonitor Fires When User is Over Quota

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

In Exchange 2013, there may be an instance where a user is over-quota and causes the Mailbox Transport responder to escalate. This issue is specific only to Mailbox Delivery and there are some simple steps to prove that this is a ‘by-design’ scenario.

Check the current value of the counter, run command:

Get-Counter “\MSExchange Delivery Store Driver\Percent of Permanent Failed Deliveries within the last 5 minutes”

Perform Message Tracking to find the failure event(s), run command:

 

Get-MessageTrackingLog -Start ([System.DateTime]::Now.AddMinutes(-10)) -End ([System.DateTime]::Now) | Where{$_.Sender -notlike ‘*Health*’ -and $_.Recipients -notlike ‘*Health*’ -and $_.EventID -contains ‘FAIL’} | Sort TimeStamp | FT –AutoSize

If you find failure events in the message tracking log, obtain the MessageID of the failure events, run command:

Get-MessageTrackingLog -Start ([System.DateTime]::Now.AddMinutes(-10)) -End ([System.DateTime]::Now) | Where{$_.Sender -notlike ‘*Health*’ -and $_.Recipients -notlike ‘*Health*’ -and $_.EventID -contains ‘FAIL’} | Sort TimeStamp | FL MessageID


Use the MessageIDs to find the reason for the failure(s), run command:

 

(Get-MessageTrackingLog -MessageId “<MessageID>” | Where{$_.EventID -contains ‘FAIL’}).RecipientStatus

 

If you see the following failure context for the recipient status, this is the ‘by-design’ scenario where a user’s mailbox is over-quota and has been for quite some time.

[{LRT=};{LED=554 5.2.2 mailbox full; STOREDRV.Deliver.Exception:QuotaExceededException.MapiExceptionShutoffQuotaExceeded; Failed to process message due to a permanent exception with message Cannot open mailbox