O365 and Exchange 2016/Exchange 2013: Understanding the UserPhoto API

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

We recently had an issue for an Enterprise Cloud customer, in which the photo was not rendering for the user – which was uploaded to AD (and synced over via MMSSPP to the managed environment). It was sussed that the issue was customer-caused, as the customer was modifying the photo via the PowerShell commandlets and had deleted the photo.

Despite the fact that the customer had disabled the OWA functionality to change the user photo in OWA, the customer was using the PowerShell commandlets to modify this object, which is the same interface/commands that OWA utilises behind the scenes.

To explain this behaviour in further detail: When you delete a photo from a mailbox in Exchange (or Exchange Online), it changes the UserPhotoCacheId MAPI property value to ‘0’ and this signifies to the UserPhoto API that the photo has been deleted. When this occurs, we will not fall-back to the ADPhotoHandler to call the photo from AD because the user explicitly deleted the photo. This scenario is by-design of the UserPhoto API.

To rectify this behaviour, clear the MAPI property by running ‘Remove-UserPhoto -ClearMailboxPhotoRecord’.

Keep in mind, as well, that Exchange uses a CachingPhotoHandler and that the photos stored on disk have a TTL of 7 days.

I have written a script to help ascertain the MAPI properties on a given mailbox, obtain the location of the cached photos on disk, and obtain the IPM.UserPhoto item from the user’s mailbox (on-premises):


function Get-UserPhotoDataOnPremises
{
param([string]$User)
Write-Warning -Message "The source of our powers comes from a Cracker Jack™ box, so this will take a minute. Please be patient..."
if([System.String]::IsNullOrEmpty($User))
{
throw [System.NullReferenceException]::new("The user value cannot be null.")
}
else
{
$mbx = Get-Mailbox $User
}

[string]$scriptPath = $env:ExchangeInstallPath + “Scripts\ManagedStoreDiagnosticFunctions.ps1”
# Dot-load script into function
. $scriptPath

[GUID]$guid = $mbx.ExchangeGuid.Guid
[int]$mbxNumber = Get-StoreQuery -Database $mbx.Database -Query “SELECT MailboxNumber FROM Mailbox WHERE MailboxGuid=’$guid'” | Select -ExpandProperty MailboxNumber
[string]$folderId = Get-StoreQuery -Database $mbx.Database -Query “SELECT FolderId FROM Folder WHERE MailboxNumber=’$mbxNumber’ AND DisplayName=’$([System.String]::Empty)'” | Select -ExpandProperty FolderId
$global:item = Get-StoreQuery -Database $mbx.Database -Query “SELECT * FROM Message WHERE MailboxNumber=’$mbxNumber’ AND FolderId=’$folderId’ AND MessageClass=’IPM.UserPhoto'” -Unlimited
[string]$previewPhotoCachedId = (Get-StoreQuery -Database $mbx.Database -Query “SELECT UserPhotoPreviewCacheId FROM Mailbox WHERE MailboxNumber=’$mbxNumber'” -Unlimited).p7C1B0003
[string]$photoCacheId = (Get-StoreQuery -Database $mbx.Database -Query “SELECT UserPhotoCacheId FROM Mailbox WHERE MailboxNumber=’$mbxNumber'” -Unlimited).p7C1A0003

# Obtain files on disk (if any)
$smtp = $mbx.WindowsEmailAddress.Address
$smtpAtIndex = $smtp.IndexOf(“@”)
$smtpAtIndexPlusOne = $smtpAtIndex + 1
$smtpDotIndex = $smtp.LastIndexOf(“.”)
$smtpNewLength = $smtpDotIndex – $smtpAtIndexPlusOne
$subString = $smtp.Substring($smtpAtIndexPlusOne, $smtpNewLength)
$queryString = “_$subString”
$preSubString = $smtp.Substring(0, $smtpAtIndex)
$srvr = Get-MailboxDatabaseCopyStatus $mbx.Database.Name | Where{$_.Status -contains ‘Mounted’} | Select -ExpandProperty MailboxServer
$folderPathUnc = “\\$($srvr)\” + $env:ExchangeInstallPath.Replace(“:”, “$”) + “\ClientAccess\photos”
$obj = Get-ChildItem -Path $folderPathUnc -Filter “*$($queryString)*”
$folderPath = $obj.FullName
$obj2 = Get-ChildItem -Path $folderPath
$fullPaths = @()
foreach($o in $obj2)
{
$picObj = Get-ChildItem -Path $o.FullName -Filter “*$($preSubString)*”
$fullPaths += New-Object PSobject -Property @{
Name=$picObj.Name
Size=$picObj.Length
Location=$picObj.FullName
}
}

if([System.String]::IsNullOrEmpty($item.MessageId) -eq $FALSE)
{
$string = “IPM.UserPhoto item found in user’s mailbox. The object can be found in ” + “$” + “item”
Write-Host $string
}
if($previewPhotoCachedId)
{
Write-Host “UserPhotoPreviewCacheId found: $previewPhotoCachedId”
}
if($photoCacheId)
{
Write-Host “UserPhotoCacheId found: $photoCacheId”
}
Write-Host -ForegroundColor Green “Photo files found on disk on $($srvr) for $($smtp):”
$fullPaths | FL
}

Here’s an example as run from my lab:


[PS] E:\>Get-UserPhotoDataOnPremises -User Administrator
WARNING: The source of our powers comes from a Cracker Jack™ box, so this will take a minute. Please be patient...
IPM.UserPhoto item found in user's mailbox. The object can be found in $item
UserPhotoPreviewCacheId found: -88512737
UserPhotoCacheId found: -88512737
Photo files found on disk on [REDACTED] for [email protected]:

Name : _Administrator-8EEDD78A2D804372C17E9FABD151BCD5.jpg
Location : \\[REDACTED]\E$\exchsrvr\ClientAccess\photos\_contoso.se-BBED6250E228D4F38F5F19BF4F1A6823\HR648x648\_Administrator-8EEDD78A2D804372C17E9FABD151BCD5.jpg
Size : 79838

Name : _Administrator-8EEDD78A2D804372C17E9FABD151BCD5.jpg
Location : \\[REDACTED]\E$\exchsrvr\ClientAccess\photos\_contoso.se-BBED6250E228D4F38F5F19BF4F1A6823\HR96x96\_Administrator-8EEDD78A2D804372C17E9FABD151BCD5.jpg
Size : 3224

I’ve also written a method in C# to test obtain the photo via the EWS GetUserPhoto REST method:


///
/// Creates an EWS request for a user's photo at each standard size.
///

/// Vanity name of your endpoint.
/// Smtp Address of the user you’re targeting.
private static void GetPhotos(string vanityName, string username)
{
int[] sizeInts = new[] { 48, 64, 96, 120, 240, 360, 432, 504, 648 };
Parallel.ForEach(sizeInts, delegate (int i)
{
Uri targetUri = new Uri($”https://{vanityName}/EWS/Exchange.asmx/s/GetUserPhoto?email={username}&size=HR{i}x{i}”);
Console.WriteLine($”Targeting: {targetUri}”);
try
{
HttpWebRequest newWebRequest = (HttpWebRequest)WebRequest.Create(targetUri);
newWebRequest.UserAgent = “Enterprise Cloud UserPhoto EWS Client”;
newWebRequest.Credentials = new NetworkCredential(“[UserName]”, “[PassWord]”);
using (HttpWebResponse newWebResponse = (HttpWebResponse)newWebRequest.GetResponse())
{
if (newWebResponse.StatusCode == HttpStatusCode.OK)
{
Console.WriteLine($”Size {i} photo found.”);
}
else if(newWebResponse.StatusCode == HttpStatusCode.NotFound)
{
Console.WriteLine($”Photo API states that a photo cannot be found for the size {i}x{i}”);
}
else
{
Console.WriteLine($”Unexpected http response received: {newWebResponse.StatusCode}”);
}
}

}
catch (Exception e)
{
Console.WriteLine(e.Message);
}

});
}

If you run into any problems with the script or have any questions or concerns around the UserPhoto API, feel free to let know! 🙂

Exchange: Non-Discriminant Mailbox Moves

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

I had to rebuild my lab and in one of the invariable problems of doing this is that the default database is created and by the time you do any administration in Exchange, mailboxes exist on it.

I had to come up with a sure-fire way to balance moves between the two mount-points I had created, so as to not really favor one over the other (plus, manually alternating isn’t much fun).

Enter PowerShell. First, I had to make script to test if the iteration I was on was even or odd:

function Check-IfEvenOrOdd ($digit) {[bool]!($digit%2)}

Once I had the function, the next step was to iterate:

$MBXs = @()
$MBXs += Get-Mailbox -Database “Mailbox Database 1750300935”
$MBXs += Get-Mailbox -Database “Mailbox Database 1750300935” -Monitoring
$MBXs += Get-Mailbox -Database “Mailbox Database 1750300935” -Arbitration
$MBXs += Get-Mailbox -Database “Mailbox Database 1909919934” -Monitoring
$MBXs += Get-Mailbox -Database “Mailbox Database 1909919934” -Arbitration
($MBXs | Measure-Object).Count

for([int]$y = 0; $y -le 18; $y++)
{
 if(Check-IfEvenOrOdd $y = $true)
 {
  New-MoveRequest $MBXs[$y].Identity -TargetDatabase EURPROD01-db001
 }
 else
 {
  New-MoveRequest $MBXs[$y].Identity -TargetDatabase EURPROD01-db002
 }
}

And there you have it: Easy ‘balancing’ of mailbox moves to evacuate databases.

Exchange: Recovering a Database from Dirty Shutdown (the Easy Way)

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

In testing (and production), we invariably run into problems where the database will be in a ‘bad state’. In my case, I pulled Store service out from under it (disabled and stopped the service) and this caused the database to go into dirty shutdown:

eseutil.exe /mh “C:\Databases\EMEA-SWEEX15-01 Store 001\EMEA-SWEEX15-01 Store 001.EDB”

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode…
         Database: C:\Databases\EMEA-SWEEX15-01 Store 001\EMEA-SWEEX15-01 Store 001.EDB

DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x65021d61
  Actual Checksum: 0x65021d61

Fields:
        File Type: Database
         Checksum: 0x65021d61
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,20
 Engine ulVersion: 0x620,20
Created ulVersion: 0x620,20
     DB Signature: Create time:05/05/2014 13:08:25.805 Rand:306694709 Computer:
         cbDbPage: 32768
           dbtime: 10263756 (0x9c9ccc)
            State: Dirty Shutdown
     Log Required: 12131-12194 (0x2f63-0x2fa2)
    Log Committed: 0-12195 (0x0-0x2fa3)
   Log Recovering: 12195 (0x2fa3)
  GenMax Creation: 10/14/2014 07:55:56.052
         Shadowed: Yes
       Last Objid: 56187
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00.000
 Old Repair Count: 0
  Last Consistent: (0x263B,8B,19C)  08/28/2014 08:55:57.153
      Last Attach: (0x263C,2,268)  08/28/2014 08:55:57.294
      Last Detach: (0x0,0,0)  00/00/1900 00:00:00.000
    Last ReAttach: (0x2DEA,2,0)  10/11/2014 04:46:44.436
             Dbid: 1
    Log Signature: Create time:05/05/2014 13:08:25.649 Rand:2111075525 Computer:
       OS Version: (6.2.9200 SP 0 NLS ffffffff.ffffffff)

Previous Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Copy Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Differential Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: none
Old bad Checksum Error Count: none

  Last checksum finish Date: 10/14/2014 08:02:23.469
Current checksum start Date: 00/00/1900 00:00:00.000
      Current checksum page: 0

Operation completed successfully in 0.187 seconds.

Using eseutil (and assuming the log is still available) we can recover from such a state like the following:

eseutil /r “E01” /l “C:\Databases\EMEA-SWEEX15-01 Store 001\Logs” /d “C:\Databases\EMEA-SWEEX15-01 Store 001”

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode…
    Logfile base name: E01
            Log files: C:\Databases\EMEA-SWEEX15-01 Store 001\Logs
         System files: <current directory>
   Database Directory: C:\Databases\EMEA-SWEEX15-01 Store 001

Performing soft recovery…
                      Restore Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|
          ……………………………………………

Operation completed successfully in 13.140 seconds.

And, once we’ve completed said step, we can verify if the database is – indeed – in clean shutdown:

eseutil.exe /mh “C:\Databases\EMEA-SWEEX15-01 Store 001\EMEA-SWEEX15-01 Store 001.EDB”

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 15.00
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode…
         Database: C:\Databases\EMEA-SWEEX15-01 Store 001\EMEA-SWEEX15-01 Store 001.EDB

DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x758dd2a3
  Actual Checksum: 0x758dd2a3

Fields:
        File Type: Database
         Checksum: 0x758dd2a3
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,20
 Engine ulVersion: 0x620,20
Created ulVersion: 0x620,20
     DB Signature: Create time:05/05/2014 13:08:25.805 Rand:306694709 Computer:
         cbDbPage: 32768
           dbtime: 13412964 (0xccaa64)
            State: Clean Shutdown
     Log Required: 0-0 (0x0-0x0)
    Log Committed: 0-0 (0x0-0x0)
   Log Recovering: 0 (0x0)
  GenMax Creation: 00/00/1900 00:00:00.000
         Shadowed: Yes
       Last Objid: 66933
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00.000
 Old Repair Count: 0
  Last Consistent: (0x2FA4,1,31)  10/14/2014 08:10:54.954
      Last Attach: (0x263C,2,268)  08/28/2014 08:55:57.294
      Last Detach: (0x2FA4,1,31)  10/14/2014 08:10:54.954
    Last ReAttach: (0x2DEA,2,0)  10/11/2014 04:46:44.436
             Dbid: 1
    Log Signature: Create time:05/05/2014 13:08:25.649 Rand:2111075525 Computer:
       OS Version: (6.2.9200 SP 0 NLS ffffffff.ffffffff)

Previous Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Copy Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Previous Differential Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00.000

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: none
Old bad Checksum Error Count: none

  Last checksum finish Date: 10/14/2014 08:02:23.469
Current checksum start Date: 00/00/1900 00:00:00.000
      Current checksum page: 0

Operation completed successfully in 0.235 seconds.

And, now, the database is safe to mount, again.

C# + EWS: Autodiscover Test (Exchange and O365)

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

In times of troubleshooting client-side issues, it may become necessary to query for the autodiscover response the user is receiving from either Exchange on-premises or Exchange in O365 – or, in the case of a redirection, both on-premises and O365 Exchange. This is a sample C#.NET Console Application, which will query for the Autodiscover response and use the TraceListener class to write the response to files.

There are two things the code doesn’t take into account for:

1. The condition wherein the user’s SMTP address and UPN are different.
2. ALL of the possible returns from Autodiscover for the UserSettings.

 

Thus, I have included the the source code for two reasons:

1. Promotion of writing .NET programs for both on-premises Exchange and O365 Exchange.
2. Customization of both the UserSettings one is targeting and the target delivery folder for which the files should be saved.

 

If you have any problems, questions, or concerns, feel free to reach out to me and I’ll try to address them as soon as possible.

The source code can be found here: http://gallery.technet.microsoft.com/C-EWS-Autodiscover-Test-870b4a8e

Exchange 2013: High Availability, FAST Search, and the Windows Registry

NOTE: This post – drafted, composed, written, and published by me – originally appeared on https://blogs.technet.microsoft.com/johnbai and is potentially (c) Microsoft.

One of the things not mentioned about High Availability (HA) and Database Availability Groups (DAGs) is that the automatic reseed feature is also used to maintain Exchange FAST Search indexes, independently, by HA.

HA utilizes a registry key to keep the Content Index (CI) states in a central locale for referencing. It queries this key because determining if the service is running on each node is approximately 30% of the cost of the ‘Get-MailboxDatabaseCopyStatus’ command. The registry key, itself, is located at: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus

You can query this registry key using the following syntax in PowerShell or Exchange Management Shell:

Get-ItemProperty HKLM:\SOFTWARE\Microsoft\ExchangeServer\v15\Search\IndexStatus

Assuming your mailbox server is not a member of a DAG, when you run the command you’ll get outputs that look something like the following:

{79853a57-6c9e-46a2-a5e3-70db036754f1} : 1,1,4294967297,2013-07-16 00:33:17Z,0,
{115d21d9-8b88-4574-b71e-5a7142daaf78} : 1,1,4294967297,2013-07-16 00:33:16Z,0,
{aab61984-7df3-412f-9c0b-f44bf0eaadbc} : 1,1,4294967297,2013-07-16 00:33:16Z,0,

Assuming your mailbox server is a member of a DAG, when you run the command you’ll get outputs that look something like the following:

{b03445e5-58c1-4f46-9d3b-3db2cb3ee3e3} : 1,1,2013-07-16 14:27:56Z,0,
{94689a79-5207-42a0-8ebc-50afe9fbfb52} : 4,7,2013-07-16 14:39:05Z,0,SPECIAL:E15-DAGMEMBER-1/3875/94689A79-5207-42A0-8EBC-50AFE9FBFB5212/Single
{1bfb37b0-9a9c-4821-9d03-cf0c7f8f2bba} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-5/3875/1BFB37B0-9A9C-4821-9D03-CF0C7F8F2BBA12/Single
{c77b6a95-719b-477c-be95-504f81b5376d} : 1,1,2013-07-16 14:14:02Z,0,
{1194860a-35c9-452e-8e87-b040eea1909c} : 4,7,2013-07-16 14:39:04Z,0,SPECIAL:E15-DAGMEMBER-2/3875/1194860A-35C9-452E-8E87-B040EEA1909C12/Single

There’s five points of information that make this relevant to issues with FAST Seach:

1.)    The initial {GUID} is the GUID of the database.

  1. There will be 1 entry per database on the DAG node.

2.)    ‘SPECIAL’ means that the CI is reseeding

  1. The source server, port, source database, and cell instance trail this value

3.)    The double numeric values (1,1; 4,7; 8,8; etc.) signify the state[s] of the CI

4.)    ‘Single’ references which type of cell we are dealing with.

5.)    The two digit number, after the second GUID, signifies the schema version of the index (in this case, the schema version is 12). You will not be able to find the database in question, if you use the second GUID and leave the schema version appended to it.

 

The first number in the series is the Index Status. The second number in the series is the error code. The codes break-down as follows:

Index Status:

Unknown = 0
Healthy = 1
Crawling =2
Failed = 3
Seeding = 4
Failed And Suspended = 5
Suspended = 6
Disabled = 7
Auto-Suspended = 8
Healthy And Upgrading = 9

Error Code:

1 Success
2 Internal Error
3 Crawling Database
4 DatabaseOffline
5 MapiNetworkError
6 Catalog Corruption
7 Seeding Catalog
8 Catalog Suspended
9 Catalog Reseed
10 Index Not Enabled
11 Catalog Excluded
12 Activation Preference skipped
13  Lag Copy skipped
14 Recovery Database Skipped
15 Fast Error
16 Service Not Running
17 Index Timestamp too Old

When a CI is reseeding, the status will – obviously – be ‘Seeding’ when you run ‘Get-MailboxDatabaseCopyStatus’. In order to see where the seeding source is, find the ‘ContentIndexSeedingSource’ value for that specific copy via ‘Get-MailboxDatabaseCopyStatus’ or you can query the registry key. Keep in mind, though, that the source port and cell are not exposed via ‘Get-MailboxDatabaseCopyStatus’ and are only found in the registry.

 

Credits:

  • Scott Oseychik, Principal Escalation Engineer, Customer Service and Support
  • Microsoft Exchange FAST Team
  • Microsoft Exchange Sustained Engineering Team