Search Service crashing on RDS / Server 2012R2

I’ve recently been experiencing many issues with the Windows Search service on Server 2012R2 – The search index service would crash and cause entire RDP sessions to hang whenever typing something in the start menu. Of course on RDS servers this was a major issue and my clients got very upset due to not being able to work for multiple hours without calling us for assistance.

This issue was actually rather difficult to troubleshoot as all of the servers experiencing this had all Windows Updates installed and no similar software. We even found the issue on a brand new fresh install without any external software once.

After several days of constant troubleshooting we’ve found the following symptoms to be true in all cases;

  • The Search Service itself does not crash – a subprocess of the search service(FilterHost) crashes.
  • Restarting the Search service resolves the issue temporarily.
  • Deleting and recreating the index does not resolve the issue.
  • The Search Index grows to extreme sizes(50GB+).
  • Offline Dedrag of the Search Databases often helps in shrinking the size, but does not resolve the issue.

Of course we’ve tried everything to resolve this – We’ve performed clean boots on servers, used procmon to find if another process causes the crash, enabled crash dumps and increased the time-out value on the search service to check if this was not a performance issue. Unfortunately none of these items helped in the slightest. The crashes kept on coming and our client actually considered removing the Windows Servers and moving to a different provider.

Afterwards I’ve been reading the details and I found we may have a classic scenario of the built-In search feature causing NTDLL thread pool exhaustion. Windows Search uses the NTDLL thread pool to achieve natural language process such as word breaking. It is possible that at the time of the crash there were many queued requests for Search. Maybe the server got too many requests and search couldn’t handle it. The Windows search has a limit of 512 threads to cater to the search operations. Seeing our clients have large operations with many Outlook and full-text-indexes for files this could be the case.

to resolve we restricted the search to use only a single thread per-query, which should resolve the problem if its indeed reaching the threads threshold due to many search queries.

This can be done by configuring this Registry value:

Name : CoreCount
Type : DWORD
Value : 1
Location : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Search

After configuring this value and rebooting we’ve not seen these issues at all – And deployed the solution to our entire RDS farm, Of course some credits should go to my co-worker Maarten van der Horst as he was relentless and did not let this issue go. πŸ™‚

31 thoughts on “Search Service crashing on RDS / Server 2012R2

  1. Bart Verhelst

    We are experiencing the same issue. Can you please provide me info on
    bart at itprovider.be
    if the problem of huge EDB files (+20Gb) is solved by this registry tweak?

    We don’t have issue with the RDS server ‘hanging’ but for some RDS customers we have huge windows.edb file

    Reply
    1. TeGek Post author

      Hi Bart – I’ve sent you an email with some details about this but in short; Yes. For us it resolved edbs that used to be 40/50 Gb and are now around 5-10GB.

      Reply
  2. Alex

    Hello,

    have the same problem. we have TS farm and index crashes very often. EDB ib about 105 GB.
    also we use indexing for large PST colletion (about 3TB)

    do you think your solution can help us?

    Reply
    1. TeGek Post author

      3TB of PST’s is a problem on its own, Maybe you should look into an archiving solution? ;). I’m not sure about the size of a 3TB pst index but it did help us with .OST indexes of around 1TB. I woulnd’t expect miracles but it could help and does not harm performance by anything if you do not enable “DisableBackOff”.

      Reply
      1. Alex

        we have 3TB not in one file. but after add this registryu setting i have 2 500 000 items indexed and 25 Gb index size…

        Reply
  3. Wayne Small

    Great Article – I’ve been investigating a very similar issue to this. Would love to know if there is more to it and how you found the core count registry key. Any chance you could email me please? Many thanks

    Reply
    1. TeGek Post author

      Hi Andreas,

      All the information is in the blog post. Wayne simply needed some more information about the troubleshooting process and what our situation was compared to his.

      Good luck, and let me know if it worked!

      Reply
  4. Will Peters

    I owe you a beer. Thanks for posting your findings! This has been driving me crazy off and on for a couple months.

    Reply
  5. Dennis

    Hi TeGek
    I think we might be facing the same issues.
    We are using Office 365 with the Office 2013 suite. On our Windows 2012 R2 RDS server we experience a dramatic drop in performance.
    After a restart, The first day everything is running perfectly. After about a week we experience a lot of problems. Search in Outlook fail. Programs not responding (often Outlook). The icons on the desktop turns white.
    Then it’s time for a restart.
    I have restarted the server today, and the Windows.edb is already 1.8 GB on a RDS server for 7 users.
    I have tried to determine the subprocess of the search service (FilterHost) crashes. But I can’t see that.
    I would like to try to implement this fix. Could you please provide me more info? Have a nice day.
    Best regards.

    Reply
    1. TeGek Post author

      Hi Dennis,

      To process this fix all you’d need is the information in the blog post. But seeing you are also noticing a major drop in performance you should check the following:

      1.) Are you using cached mode? If no, enable it. enabling caching gives you a MAJOR increase in performance. If you are worried about disk space consider using Folder Redirection for the profile folders or UPD.

      2.) You’re also noticing a change in icons, which might point to a disk performance issue. If you are using Antivirus software. Please exclude .OST files from this. OST scanning can cause major performance issues.

      3.) If it is still not resolved, You should make performance logs using perfmon. Quite often these issues result from under-dimensioning servers.

      Reply
    2. Robert

      Hi Dennis

      Reading your comment was the hope we as a team needed to carry on fighting this issue, having someone else reporting the icons going white along with the degredation. Did you / Have you / Can you found / find a solution to this which you were able to box off the complaints for?

      Dennis. We need you. You’re our only hope.

      Reply
    1. TeGek Post author

      Hi Matt,

      I’ve actually never seen users disconnect from sessions due to the search host. I have seen complete session hangs though.

      Reply
  6. Dennis

    Hi TeGek
    It was a nice try, but unfortunately it didn’t solve my issue. It is very strange. Our Windows 2012 R2 RDS server is running on an IBM X3650 M5 – Dual Xeon – with 10K SAS disks. It has been running from November 2014. The symptoms we are facing are the same as you describe. The server works perfectly, but within a week it slows incredible down. We are only 7 users. We have a lot of Outlook activity and share our mailbox across all coworkers, so the OST files are big – 430 GB for 10 mailboxes.
    The Windows Search Index grows to 5 GB in 3-4 days, and that is slowing server down. If I rebuild the index everything works fine for the week or so.
    There is a lot of disk activity on the searchindexer.exe and, on the *.ost files and C:\ProgramData\Microsoft\Search\Data\Applications\Windows\windows.edb

    Reply
  7. Rudy Ooms

    I just applied the reg fix to multiple servers…. I really hope it will fix the:

    -The Search Index grows to extreme sizes(50GB+).

    How did you think of the CoreCount dword key? I could not find it anywhere except on this forum

    Reply
  8. Jarrod Cavner

    We are seeing this issue as well and are considering your fix. Did you see a performance degradation in searches after you put the registry key in place? Once we restart the search service to temporarily clear up the issue, we see multiple searches fire off each of them using between 10-15 threads.

    Reply
  9. Jason Johnson

    If the value in the registry does not exist do you need to create it. I am not see the entry in the registry?

    Reply
  10. Alessandro

    We have same problems described, file very large and often search crash.. We applied the fix adding the registry value and deleting the index files. After a server reboot the search started with indexing slowing than normal,. Today, after two days, i’ve checked the indexing process and it’s more fast.. Verifyng on registry, the CoreCount value it’s not more present.. Nobody have deleted the registry value!.. Maybe some windows application can control and remove the registry? I’ll try to insert the CoreCont value againg….

    Reply
  11. NZ sys admin

    Hi

    we are running outlook 2013 with gmail app synch on RDS (2012R2).

    every time the users log in outlook seems to be rebuilding the index. By the end of the day most people have some sort of search functionality but nowhere near as good as the stable index on a local install of outlook. but it’s a real groundhog day experience with it starting from scratch every day.

    We also have a large index file, although not sure we have the same performance issue. That said we have an entire G9 DL380 64GB just for the RDS server (30 users), but the users gmail accounts are typically 20-40GB each. so there may be a performance issue but it is not affecting us.

    we have researched this before and as far as we can see it appears to be a known problem with no solution(for the last 3-4 years). We are not interested in switching to exchange.

    any ideas to avoid the search file reindexing?

    Reply
  12. Lefteris Krashias

    Hi I’ve the below scenario:

    Windows Server 2012 R2 on Microsoft Azure.
    Server is setup with Terminal Services with 20 Users running Microsoft Office 2010 Standard.

    Problem: The Windows.edb file is increasing and after reaching a limit (don’t know) it rebuild by its own without reason. How to prevent this?

    Please help me.

    Thanks

    Reply
    1. TeGek Post author

      Hi Lefteris,

      When running in Azure I suggest disabling the search service on the server and disable cached Exchange mode.

      Azure has direct backend access to Office-365 and Online Mode capabilities are near perfect in this case. In online mode you won’t need the search service as you’ll be performing online searches.

      I haven’t heard about the search service rebuilding itself without reason, but you can always try to perform a defrag of the search database to see if it helps in reducing the size.

      Regards,

      Reply
  13. Edward Smith

    Can this fix be applied to workstations as well?

    I have several Windows 8.1 workstations where the Windows Search EDB file is growing to 100GB to 200GB in size, then it barfs, shrinks, and regrows. The OST files are between 25GB and 50GB with a *lot* of email attachments. Configuring the search to only index the OST does not change the behavior of the EDB file.

    This is eating up disk I/O and disk space and causing issues.

    On one machine we redirected a daily set of email reports with large attachments to a shared mailbox and that reduced the OST from 50+GB to 25GB, but the Windows Search EDB is stil 100GB in size.

    All machines are operating off Exchange 2013 in cached mode.

    Thanks

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *