About…

Search Service crashing on RDS / Server 2012R2

I’ve recently been experiencing many issues with the Windows Search service on Server 2012R2 – The search index service would crash and cause entire RDP sessions to hang whenever typing something in the start menu. Of course on RDS servers this was a major issue and my clients got very upset due to not being able to work for multiple hours without calling us for assistance.

This issue was actually rather difficult to troubleshoot as all of the servers experiencing this had all Windows Updates installed and no similar software. We even found the issue on a brand new fresh install without any external software once.

After several days of constant troubleshooting we’ve found the following symptoms to be true in all cases;

  • The Search Service itself does not crash – a subprocess of the search service(FilterHost) crashes.
  • Restarting the Search service resolves the issue temporarily.
  • Deleting and recreating the index does not resolve the issue.
  • The Search Index grows to extreme sizes(50GB+).
  • Offline Dedrag of the Search Databases often helps in shrinking the size, but does not resolve the issue.

Of course we’ve tried everything to resolve this – We’ve performed clean boots on servers, used procmon to find if another process causes the crash, enabled crash dumps and increased the time-out value on the search service to check if this was not a performance issue. Unfortunately none of these items helped in the slightest. The crashes kept on coming and our client actually considered removing the Windows Servers and moving to a different provider.

Afterwards I’ve been reading the details and I found we may have a classic scenario of the built-In search feature causing NTDLL thread pool exhaustion. Windows Search uses the NTDLL thread pool to achieve natural language process such as word breaking. It is possible that at the time of the crash there were many queued requests for Search. Maybe the server got too many requests and search couldn’t handle it. The Windows search has a limit of 512 threads to cater to the search operations. Seeing our clients have large operations with many Outlook and full-text-indexes for files this could be the case.

to resolve we restricted the search to use only a single thread per-query, which should resolve the problem if its indeed reaching the threads threshold due to many search queries.

This can be done by configuring this Registry value:

Name : CoreCount
Type : DWORD
Value : 1
Location : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Search

After configuring this value and rebooting we’ve not seen these issues at all – And deployed the solution to our entire RDS farm, Of course some credits should go to my co-worker Maarten van der Horst as he was relentless and did not let this issue go. 🙂

Stop AADSync logs from clogging up your servers disk space

I’ve been rolling out a lot of large AADSync deployments recently – I love how AADSync gives a SSO experience to the SMB markets without having to deploy ADFS. But as always these deployments in SMB markets have some downsides; The default configuration for AADSync/Dirsync is that it logs everything using tracing and ForeFront MSSQL Logs.

On smaller deployments or deployments where diskspace is expensive you might want to limit the usage of these logs. Of course my advice as always is to only make these changes when your DirSync/AADSync environment is running well and not experiencing any issues whatsoever.

To prevent the SQL database from growing to absurd sizes:

param([int]$DaysToKeep=2)
$DirSync = Get-WmiObject -Class “MIIS_SERVER” -Namespace “root\MicrosoftIdentityIntegrationServer”
$DirSync.ClearRuns([DateTime]::Today.AddDays(-$DaysToKeep)) | Format-Table ReturnValue

Save this script as .\ClearDirSyncDB.ps1 and run it. Of course you can set this as a schedulded task to automate it.

To prevent the TRACING folder from filling with logs:

  • Stop FIMSynchronizationService service via services.msc.
  • Run Notepad or your favorite text editor as administrator
  • Open the MIISERVER.Config file at X:\Program Files\Microsoft Online Directory Sync\SYNCBUS\Synchronization Service\Bin\miiserver.exe.config (Where X is the drive you installed AADSYNC)
  • Paste the following after
  • </appSettings>

<system.diagnostics>
  <trace autoflush=”true”>
  <listeners>
  <add name=”DirectorySynchronizationTraceFile” type=”System.Diagnostics.TextWriterTraceListener” initializeData=”C:\Temp\DirectorySynchronizationTrace.log” traceOutputOptions=”DateTime” />
  </listeners>
  </trace>
  <sources>
  <source name=”passwordSync” switchName=”sourceSwitch” switchType=”System.Diagnostics.SourceSwitch”>
  <listeners>
  <add name=”console” type=”System.Diagnostics.ConsoleTraceListener”>
  <filter type=”System.Diagnostics.EventTypeFilter” initializeData=”Information” />
  </add>
  <add name=”sharedTextLogger” type=”System.Diagnostics.TextWriterTraceListener” initializeData=”PasswordSync.log”>
  <filter type=”System.Diagnostics.EventTypeFilter” initializeData=”Verbose” />
  </add>
  <remove name=”Default” />
  </listeners>
  </source>
  </sources>
  <switches>
  <add name=”sourceSwitch” value=”Verbose” />
  </switches></system.diagnostics>

  • Re-start FIMSynchronizationService service via services.msc.
  • Reactivate your syncronisation by running the configuration wizard and entering your domain details

And then you’re all done! dirsync should not longer eat up your precious diskspace 🙂