Script to set retention tag on default folder items updated to v1.1.1

Articles in the "Retention tag on default folder items" series

  1. Use EWS to apply retention policy to items in a default folder
  2. Script to set retention tag on default folder items updated to v1.1.1 [This article]

When running v1.0 of the script in a folder with lots of items, it would keep stopping with no errors, but there were more items to process. I found that this was happening when the number of items to process changes because a new item was added to the folder. In other words, while processing deleted items and another item is added to the Deleted Items folder, the total number of items changes, resulting in Exchange not returning the next set of items correctly. v1.1 correctly accounts for this condition.

Additionally, I found that calendar items in the Deleted Items folder cannot be processed with the API. Trying to change any property returns an error that it can’t update calendar items that are already deleted. But since you can manually assign a tag to it in Outlook, I consider it a bug that you can’t update calendar items in the Deleted Items folder. So, I updated the search filter to exclude calendar items when not searching in the Calendar folder (processing meeting responses is okay; it is only appointment/meeting items that are affected).

You can now choose the default folder to process. If you don’t specify one, the Deleted Items folder is selected. I included all default folders that can have a retention policy tag assigned AND have a well-known folder ID. This means that you can’t use the script to process items in the Clutter or RSS Feeds folders. If you are interested in having the script work against those folders, let me know and I will add the code necessary to do so.

Download the updated version below. (The inline code of the first post has been updated, too.)

  Set-DefaultFolderItemsTag.zip (2.6 KiB)

Use EWS to apply retention policy to items in a default folder

Articles in the "Retention tag on default folder items" series

  1. Use EWS to apply retention policy to items in a default folder [This article]
  2. Script to set retention tag on default folder items updated to v1.1.1

When working with retention policies and the types of tags you can apply to folders and items, you can assign a personal tag to any item and to any custom folder. You cannot, however, assign a personal tag to a default folder (such as Deleted Items), even if a retention policy tag has not been assigned to the folder. This means that if no default policy tag has been assigned to the policy, the items in that folder will never expire. The only way for a user to expire items in that folder is to assign a personal tag to each and every item. For the deleted items folder, that can be a lot of items, and its contents are changing daily.

This script uses the EWS Managed API to get all items in the Deleted Items folder that do not have a tag assigned to them and then assign a specific tag to each. To start, you need to connect to EWS:

function Connect-WebServices ($smtpAddress)
	{
	$exchangeVersion = [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2013_SP1 
	$exchangeService = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService($exchangeVersion) 
 	$exchangeService.Credentials = New-Object Microsoft.Exchange.WebServices.Data.WebCredentials($EXOCreds)
    #Use hard-coded URL
    $exchangeService.Url = 'https://outlook.office365.com/EWS/Exchange.asmx'
    #Impersonate mailbox
	#$exchangeService.ImpersonatedUserId = New-Object Microsoft.Exchange.WebServices.Data.ImpersonatedUserId([Microsoft.Exchange.WebServices.Data.ConnectingIdType]::SmtpAddress, $smtpAddress)
    $folderID= New-Object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::$DefaultFolder,$smtpAddress)     
    [Microsoft.Exchange.WebServices.Data.Folder]::Bind($exchangeService,$folderID)
	}

I am using a function because I lifted this code from another one of my scripts, so calling this function returns an object for the Deleted Items folder. Based on the credentials format and EWS URL, you can see that I am connecting to Exchange Online. This can be easily changed to support on-premises. I am not using autodiscover because it is very slow when querying EXO, and since all EXO mailboxes can be accessed with the a single FQDN, it is simpler this way. I am not using impersonation because I am running this against my own mailbox, but you can uncomment the line if you choose to use it.

To search for items that do not have a tag assigned, this is used:

$policyTagProperty = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x3019,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Binary)
$itemView.PropertySet.Add($policyTagProperty)
$itemSearchFilter = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+Exists($policyTagProperty)
$itemSearchFilterNot = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+Not($itemSearchFilter)
$itemSearchResult = $folder.FindItems($itemSearchFilterNot,$itemView)

The MAPI property that indicates whether a tag has been assigned (whether implicitly or explicitly) is an extended property that you declare and add to a property set. The property is binary and contains the GUID of the tag, but since I am only looking for items without a tag, I only care if the property has a value. To do this, you first define a search filter object that says to include items where the property exists. Then to negate that, so I can find items without that property, you create another search filter object using the Not class that contains the other search filter.

Then you can apply the tag and its corresponding days until expiration value:

$policyTagGUID = New-Object Guid("{33CEDA03-0536-424C-8ECA-E839E0BC5945}")
$item.SetExtendedProperty($policyTagProperty, $policyTagGUID.ToByteArray())
$retentionPeriodProperty = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x301A,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Integer)
$item.SetExtendedProperty($retentionPeriodProperty, 30)
$item.Update([Microsoft.Exchange.WebServices.Data.ConflictResolutionMode]::AlwaysOverwrite)

To assign the tag, you need to know its GUID. You can get this from PowerShell, but if you don’t have access to Exchange to get this, you can manually assign the tag to an item, then use MFCMAPI and look at the item’s properties for the value in PR_POLICY_TAG (0x30190102). The RAW representation of the GUID will need to be converted to the proper byte order, which can be done in a variety of ways, but this site is an easy way. When assigning the tag you also have to set the property that contains the number of days after which the tag is configured to expire. In my case, it is a 30-day tag. (I tested not setting the property and the result in Outlook does show that the tag is assigned but it doesn’t show the expiration date. I don’t know if the property is only used to calculate the displayed date or if MRM actually uses it when expiring items.)

The complete script can be downloaded from the link or copied from the code below. It includes checking for the EWS Managed API and a progress bar (since this is not a fast operation).

  Set-DefaultFolderItemsTag.zip (2.6 KiB)

#Apply retention tag to items in deleted items folder that do not have one
#v1.1 4/2/15
#1.1 Exclude appointment items, detect changed search results, add help, choose folder to process

<#
	.Synopsis
		Assign personal tag to all items in a default folder.
	.Description
		Get all items in a folder that do not have a retention tag explicitly
        assigned and assign a personal tag them.
	.Parameter EmailAddress
		Email address of mailbox to process
	.Parameter DefaultFolder
		The name of the default folder to process.  Default value is DeletedItems.
	.Example
		Set-DefaultFolderItemsTag.ps1 user@domain
	.Example
		Set-DefaultFolderItemsTag.ps1 user@domain -DefaultFolder SentItems
	.Notes
		Version: 1.1
		Date: 4/2/15
	#>

Param (
	[Parameter(Position=0,Mandatory=$true,HelpMessage="Email address of mailbox")][string]$EmailAddress,
    [Parameter(Position=1,Mandatory=$false,HelpMessage="Name of default folder to process")]
    [ValidateSet('Calendar','ConversationHistory','DeletedItems','Inbox','Journal','JunkEmail','Notes','SentItems','SyncIssues','Tasks')]
    [string]$DefaultFolder = 'DeletedItems'
	)

#Region Variables

#Replace with GUID of tag to assign to items
$personalTagGUID = "{33CEDA03-0536-424C-8ECA-E839E0BC5945}"
#Replace with the number of days the tag is configured for retaining items
$personalTagRetentionDays = 30

#EndRegion

#Paths to EWS Managed API DLL
$ewsAPIVersions = '2.2','2.1','2.0','1.2','1.1','1.0'
$ewsAPIPath = 'C:\Program Files\Microsoft\Exchange\Web Services\_v_\Microsoft.Exchange.WebServices.dll'

#Test if any version of API is installed before continuing and load latest version
foreach ($version in $ewsAPIVersions)
	{
	$path = $ewsAPIPath.Replace('_v_',$version)
    if (Test-Path $path)
		{
		Add-Type -Path $path
		$apiFound = $true
		break
		}
	}

if (-not($apiFound))
	{
	Write-Error "The Exchange Web Services Managed API is required to use this script." -Category NotInstalled
	break
	}

function Connect-WebServices ($smtpAddress, $folder)
	{
	$exchangeVersion = [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2013_SP1 
	$exchangeService = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService($exchangeVersion) 
 	$exchangeService.Credentials = New-Object Microsoft.Exchange.WebServices.Data.WebCredentials($EXOCreds)
    #Use hard-coded URL
    $exchangeService.Url = 'https://outlook.office365.com/EWS/Exchange.asmx'
    #Enable tracing to debug
    #$exchangeService.TraceEnabled = $true
    #Impersonate mailbox
	#$exchangeService.ImpersonatedUserId = New-Object Microsoft.Exchange.WebServices.Data.ImpersonatedUserId([Microsoft.Exchange.WebServices.Data.ConnectingIdType]::SmtpAddress, $smtpAddress)
    $folderID= New-Object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::$folder,$smtpAddress)     
    [Microsoft.Exchange.WebServices.Data.Folder]::Bind($exchangeService,$folderID)
	}

function Apply-RetentionPolicy
    {
    #Bind to folder of mailbox
    $folder = Connect-WebServices $EmailAddress $DefaultFolder
    if (-not($folder))
        {
        Write-Error -Message "Error binding to folder in mailbox for $EmailAddress." -Category ConnectionError
        }
    else
        {
        #Set up paged search to stay below FindCountLimit
        $pageSize = 50
        $offset = 0
        $moreItems = $true
        $itemCount = 0
       
        $firstRun = $true

        while ($moreItems)
            {
            #Setup the view to do a paged search
            $itemView = New-Object Microsoft.Exchange.WebServices.Data.ItemView($pageSize,$offset,[Microsoft.Exchange.WebServices.Data.OffsetBasePoint]::Beginning)
            $itemView.Traversal = [Microsoft.Exchange.WebServices.Data.ItemTraversal]::Shallow
            $itemView.PropertySet = New-Object Microsoft.Exchange.WebServices.Data.PropertySet([Microsoft.Exchange.WebServices.Data.BasePropertySet]::IdOnly)
            #Define retention policy property to include in search
            $policyTagProperty = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x3019,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Binary)
            #Add property to property set
            $itemView.PropertySet.Add($policyTagProperty)
            #Create the search filter to find items with no tag set and are not appointments
            $itemSearchFilter1 = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+Exists($policyTagProperty)
            $itemSearchFilter2 = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo([Microsoft.Exchange.WebServices.Data.EmailMessageSchema]::ItemClass, 'IPM.Appointment')
            $itemSearchFilterCollection = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+SearchFilterCollection([Microsoft.Exchange.WebServices.Data.LogicalOperator]::Or)
            $itemSearchFilterCollection.Add($itemSearchFilter1)
            $itemSearchFilterCollection.Add($itemSearchFilter2)
            $itemSearchFilterNot = New-Object Microsoft.Exchange.WebServices.Data.SearchFilter+Not($itemSearchFilterCollection)
            #Search for items in folder matching search filter
            $itemSearchResult = $folder.FindItems($itemSearchFilterNot,$itemView)
            
            if ($firstRun)
                {
                if ($itemSearchResult.TotalCount -eq 0)
                    {
                    Write-Output "There are no items to process."
                    }
                else
                    {
                    $totalItems = $itemSearchResult.TotalCount
                    $firstRun = $false
                    }
                }
        
            #Detect changed result set
            #Indicator is when no results returned but the server says there is still a count
            #or when server's total count is greater than what is returned but it says there aren't more
            if (($itemSearchResult.Items.Count -eq 0 -and $itemSearchResult.TotalCount -gt 0) -or ($itemSearchResult.Items.Count -le $pageSize -and $itemSearchResult.TotalCount -gt $pageSize -and $itemSearchResult.MoreAvailable -eq $false))
                {
                $resultsChanged = $true
                $totalItems = $itemCount + $itemSearchResult.TotalCount
                }
            #Process each item in current search result collection
            foreach ($item in $itemSearchResult.Items)
                {
                $itemCount ++
                Write-Progress -Activity "Applying retention tag" -CurrentOperation "Updating item $itemCount of $totalItems" -PercentComplete ($itemCount/$totalItems*100)
                $policyTagGUID = New-Object Guid($personalTagGUID)
                $item.SetExtendedProperty($policyTagProperty, $policyTagGUID.ToByteArray())
                $retentionPeriodProperty = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x301A,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Integer)
                $item.SetExtendedProperty($retentionPeriodProperty, $personalTagRetentionDays)
                $item.Update([Microsoft.Exchange.WebServices.Data.ConflictResolutionMode]::AlwaysOverwrite)
                }
            
            #If the results have changed, set offset to 0
            if (-not($resultsChanged))
                {
                if ($itemSearchResult.MoreAvailable -eq $false)
                    {$moreItems = $false}
                if ($moreItems)
                    {$offset += $pageSize}
                }
            else
                {
                $resultsChanged = $false
                $offset = 0
                }
            }
        }
    }

if (-not($EXOCreds))
    {
    $EXOCreds = Get-Credential -Message 'Enter the credentials to use to access Exchange Online.'
    }
Apply-RetentionPolicy $EmailAddress

Delegate management module updated to v1.4.5

The module has been updated mostly for fixing issues when working with Exchange Online. The first version that supported it didn’t account account for object properties that are different compared to on-premises, as well as how to get user information. These are the changes in this version:

  • Fixed when using a default connection mode of EXO so that the rest of the module knows it.
  • Added option to not use autodiscover when using EXO since those lookups can sometimes add a lot of time to the cmdlets running.  If default mode is EXO and you don’t want to use autodiscover, uncomment that line below the default mode.  If using EXO on-demand, you can set the option with the DoNotUseAutodiscover switch parameter of Set-DelegateManagementMode. (The cmdlet’s help has been updated to reflect this.)
  • Added usage of the Azure Active Directory module when using EXO mode.  This means you need to have the WAAD module installed to work against Exchange Online.  Since that module is 64-bit only, you can only run the delegate management module in a 64-bit PowerShell session.
  • Fixed (hopefully and finally) the Write-Progress prompt that some people were getting that interrupted the cmdlets.  (Thanks, Jim.)
  • Fixed getting Send As permission in EXO due to it using a different cmdlet.
  • Fixed getting folder permissions in EXO due to the object properties being different.
  • Added removal of Deleted Items and Sent Items folder permissions when removing a delegate.

There are other things I have discovered need fixing: Exchange cmdlets loaded by the module are not accessible outside of the module; Exchange cmdlet errors are not caught in PowerShell 4 so the module cmdlets keep running after a terminating error would be detected if running in PowerShell 2; if multiple objects are pipelined to Get-MailboxDelegate and one of them does not have a mailbox, the cmdlet terminates without processing the remaining objects in the pipeline.

I also still intend to add support for hybrid mode.  It is more complicated, though, such as with adding delegates since I need to account for attempts at cross-premises delegation, which isn’t supported.

  DelegateManagement.zip (7.5 KiB)

Delegate management module updated to support Exchange Online

The module for managing Exchange mailbox delegates has been updated with support for Exchange Online. In its current version (v1.4) you can use one mode or the other. The default mode is on-premises, but you can change this on demand to use Exchange Online by using Set-DelegateMananagementMode, a new cmdlet added in this version. If you change it on demand, you will be prompted for your Office 365 credentials. If you will be exclusively working with Exchange Online, you can change the line near the top of the module to default to using that method. In that case, you will be prompted for credentials the first time you run a cmdlet.

It is my intention to update the module to support a hybrid environment, but I first need to set up one in my lab in order to test it.

  DelegateManagement.zip (7.5 KiB)

Database seeding completion estimate script updated

Articles in the "Database seeding estimation" series

  1. Estimate the time to complete seeding a mailbox database copy
  2. Database seeding completion estimate script updated [This article]

The script has been updated to include the average throughput from the sampling duration in the output. This will help gauge why the seeding completion time may vary between multiple executions of the script, such as during vs. outside of business hours. Additionally, some variable names have been updated (to better reflect their contents), and per scripting best practices, the help has an added example so that all parameters are used at least once among all the examples. The inline code has been updated, as has the download.

  Get-DatabaseSeedingCompletion.zip (1.5 KiB)

Estimate the time to complete seeding a mailbox database copy

Articles in the "Database seeding estimation" series

  1. Estimate the time to complete seeding a mailbox database copy [This article]
  2. Database seeding completion estimate script updated

Seeding a mailbox database copy, especially across a WAN, can take days to complete. The shell doesn’t give any indication of the time it will complete, though it does show a progress bar and the amount of data written. If you want to know when the seeding will actually finish, you can extrapolate this from the throughput of the seeding, dividing that into the amount of data remaining to be copied.

The throughput of a copy being seeded is available as a performance counter, as are the percentage complete and the amount written. Using this information I wrote a script to estimate the date and time the database seeding will complete. Because throughput can vary greatly from second to second, the default sample duration is 60 seconds. You can specify any number of seconds, choosing accuracy or impatience. Retrieving performance counters can take some time (not even counting any sampling you may do), so a progress indicator will indicate any time that is occurring. (Three separate queries for performance counters occur to provide the output.)

To run the script, provide a database copy and, optionally, a sample duration. The copy will be validated that it is being seeded, then the seeding throughput will be retrieved for 60 seconds or the duration you specified. Then the amount of data written so far is retrieved, comparing that to the total size of the database, so that the remaining amount can be determined. That amount is divided by the average throughput during the sample duration and calculating the time it will be done. Bear in mind that this calculation does not include the time to seed the content index or copy and replay transaction logs that have been generated since seeding began. The output will show the percentage of the database size that has been seeded thus far and the estimated date and time the seeding will complete.

You can copy the code below or download the script via the link.

  Get-DatabaseSeedingCompletion.zip (1.5 KiB)

<#
	.Synopsis
		Estimate time of completion for database seeding
	.Description
		Using a customizable sample duration of the seeding throughput for a database copy,
		the script estimates the date and time the seeding will complete.  This does
		not include the time to seed the content index or copy transaction logs that have
		been accruing since the seeding started.
	.Parameter Identity
		The identity of a database copy, using the format Database\Server.
	.Parameter SampleDuration
		The number of seconds (in one-second intervals) to monitor the seeding throughput
		in order to calculate the completion time.  The default is 60 seconds.
	.Example
		Get-DatabaseSeedingCompletion.ps1 -Identity database1\server2
	.Example
		Get-DatabaseSeedingCompletion.ps1 -Identity database2\serverA -SampleDuration 360
	.Notes
		Version: 1.1
		Date: 11/22/13
#>
param
	(
	[parameter(Mandatory=$true,Position=0)][string]$Identity,
	[parameter(Mandatory=$false,Position=1)][int]$SampleDuration = 60
	)
$database = $Identity.Substring(0,$Identity.IndexOf('\'))
$server = $Identity.Substring($Identity.IndexOf('\') + 1)
$showProgress = $true
while ($showProgress -eq $true)
	{
	Write-Progress -Activity 'Getting seeding progress percentage...' -Status 'Retrieving'
	$percentCompleteCounter = Get-Counter -ComputerName $server -Counter "\MSExchange Replica Seeder($database)\Database Seeding Progress %" -ErrorAction 'SilentlyContinue'
	$showProgress = $false
	Write-Progress -Activity 'Getting seeding progress percentage...' -Status 'Retrieving' -Completed
	}
if (-not($percentCompleteCounter))
	{
	Write-Warning -Message "$Identity is not currently being seeded."
	}
else
	{
	$percentComplete = $percentCompleteCounter.CounterSamples[0].CookedValue
	
	#Get seeding throughput
	$showProgress = $true
	while ($showProgress -eq $true)
		{
		Write-Progress -Activity 'Getting seeding throughput...' -Status "Sampling for $SampleDuration seconds"
		$kBytesWrittenPerSecondCounter = Get-Counter -ComputerName $server -Counter "MSExchange Replica Seeder($database)\Database Seeding Bytes Written (KB/sec)" -MaxSamples $SampleDuration
		$showProgress = $false
		Write-Progress -Activity 'Getting seeding throughput...' -Status 'Sampling' -Completed
		}
	$kBytesWrittenPerSecondAverage = $kBytesWrittenPerSecondCounter | ForEach-Object {$_.CounterSamples[0].CookedValue} | Measure-Object -Average | Select-Object -ExpandProperty Average
	if ($kBytesWrittenPerSecondAverage -eq 0)
		{
		Write-Warning -Message 'Cannot estimate completion time because throughput for the sampling period is 0 KB.'
		}
	else
		{
		#Get bytes written so far
		$showProgress = $true
		while ($showProgress -eq $true)
			{
			Write-Progress -Activity 'Getting seeding amount written so far...' -Status 'Retrieving'
			$kBytesWrittenCounter = Get-Counter -ComputerName $server -Counter "MSExchange Replica Seeder($database)\Database Seeding Bytes Written (KB)"
			$showProgress = $false
			Write-Progress -Activity 'Getting seeding amount written so far...' -Status 'Retrieving' -Completed
			}
		$kBytesWrittenValue = $kBytesWrittenCounter.CounterSamples[0].CookedValue
		
		#Get DB size to calculate remaining amount
		$dbSizeRaw = (Get-MailboxDatabase -Identity $database -Status).DatabaseSize
		if ($dbSizeRaw.GetType().Name -eq 'ByteQuantifiedSize') #Object type when using EMS
			{
			$dbSize = $dbSizeRaw.ToBytes()
			}
		else
			{ #Object type when using remoting is string
			$dbSize = [int64]($dbSizeRaw.Split("(")[1].Split()[0])
			}
		$KBRemaining = $dbSize/1KB - $kBytesWrittenValue
		$estimatedMinutesToCompletion = $KBRemaining/$kBytesWrittenPerSecondAverage/60
		$completionTime = (Get-Date).AddMinutes($estimatedMinutesToCompletion)
		$outputObject = "" | Select-Object -Property Identity,PercentComplete,'Throughput(MB/sec)',EstimatedCompletion
		$outputObject.Identity = $Identity
		$outputObject.PercentComplete = $percentComplete
		$outputObject.'Throughput(MB/sec)' = [math]::Round($kBytesWrittenPerSecondAverage/1024,1)
		$outputObject.EstimatedCompletion = "{0:g}" -f $completionTime
		$outputObject
		}
	}