Search Service crashing on RDS / Server 2012R2

I’ve recently been experiencing many issues with the Windows Search service on Server 2012R2 – The search index service would crash and cause entire RDP sessions to hang whenever typing something in the start menu. Of course on RDS servers this was a major issue and my clients got very upset due to not being able to work for multiple hours without calling us for assistance.

This issue was actually rather difficult to troubleshoot as all of the servers experiencing this had all Windows Updates installed and no similar software. We even found the issue on a brand new fresh install without any external software once.

After several days of constant troubleshooting we’ve found the following symptoms to be true in all cases;

  • The Search Service itself does not crash – a subprocess of the search service(FilterHost) crashes.
  • Restarting the Search service resolves the issue temporarily.
  • Deleting and recreating the index does not resolve the issue.
  • The Search Index grows to extreme sizes(50GB+).
  • Offline Dedrag of the Search Databases often helps in shrinking the size, but does not resolve the issue.

Of course we’ve tried everything to resolve this – We’ve performed clean boots on servers, used procmon to find if another process causes the crash, enabled crash dumps and increased the time-out value on the search service to check if this was not a performance issue. Unfortunately none of these items helped in the slightest. The crashes kept on coming and our client actually considered removing the Windows Servers and moving to a different provider.

Afterwards I’ve been reading the details and I found we may have a classic scenario of the built-In search feature causing NTDLL thread pool exhaustion. Windows Search uses the NTDLL thread pool to achieve natural language process such as word breaking. It is possible that at the time of the crash there were many queued requests for Search. Maybe the server got too many requests and search couldnโ€™t handle it. The Windows search has a limit of 512 threads to cater to the search operations. Seeing our clients have large operations with many Outlook and full-text-indexes for files this could be the case.

to resolve we restricted the search to use only a single thread per-query, which should resolve the problem if its indeed reaching the threads threshold due to many search queries.

This can be done by configuring this Registry value:

Name : CoreCount
Type : DWORD
Value : 1
Location : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Search

After configuring this value and rebooting we’ve not seen these issues at all – And deployed the solution to our entire RDS farm, Of course some credits should go to my co-worker Maarten van der Horst as he was relentless and did not let this issue go. ๐Ÿ™‚

37 Comments

  1. Bart Verhelst August 4, 2015 at 7:50 pm

    We are experiencing the same issue. Can you please provide me info on
    bart at itprovider.be
    if the problem of huge EDB files (+20Gb) is solved by this registry tweak?

    We don’t have issue with the RDS server ‘hanging’ but for some RDS customers we have huge windows.edb file

    1. TeGek August 4, 2015 at 8:16 pm

      Hi Bart – I’ve sent you an email with some details about this but in short; Yes. For us it resolved edbs that used to be 40/50 Gb and are now around 5-10GB.

  2. Alex September 4, 2015 at 10:53 pm

    Hello,

    have the same problem. we have TS farm and index crashes very often. EDB ib about 105 GB.
    also we use indexing for large PST colletion (about 3TB)

    do you think your solution can help us?

    1. TeGek September 9, 2015 at 9:32 am

      3TB of PST’s is a problem on its own, Maybe you should look into an archiving solution? ;). I’m not sure about the size of a 3TB pst index but it did help us with .OST indexes of around 1TB. I woulnd’t expect miracles but it could help and does not harm performance by anything if you do not enable “DisableBackOff”.

      1. Alex September 9, 2015 at 7:41 pm

        we have 3TB not in one file. but after add this registryu setting i have 2 500 000 items indexed and 25 Gb index size…

  3. Wayne Small November 27, 2015 at 4:56 am

    Great Article – I’ve been investigating a very similar issue to this. Would love to know if there is more to it and how you found the core count registry key. Any chance you could email me please? Many thanks

    1. TeGek November 27, 2015 at 7:36 am

      Hi Wayne,

      I’ve sent you an e-mail with some of my findings, Hope that helps!

      Regards,
      TeGek

  4. Andreas Furtenbacher December 15, 2015 at 12:16 pm

    Hi,

    we have +50GB index databases as well. Can you send me the details too ๐Ÿ™‚ Please!!

    Thank you,
    Andreas

    1. TeGek December 15, 2015 at 12:31 pm

      Hi Andreas,

      All the information is in the blog post. Wayne simply needed some more information about the troubleshooting process and what our situation was compared to his.

      Good luck, and let me know if it worked!

  5. Will Peters January 15, 2016 at 5:20 am

    I owe you a beer. Thanks for posting your findings! This has been driving me crazy off and on for a couple months.

    1. TeGek January 15, 2016 at 8:22 am

      No problem! Hope you enoy the solution and tell everyone you know about it ๐Ÿ˜‰

  6. Dennis February 10, 2016 at 2:52 pm

    Hi TeGek
    I think we might be facing the same issues.
    We are using Office 365 with the Office 2013 suite. On our Windows 2012 R2 RDS server we experience a dramatic drop in performance.
    After a restart, The first day everything is running perfectly. After about a week we experience a lot of problems. Search in Outlook fail. Programs not responding (often Outlook). The icons on the desktop turns white.
    Then itโ€™s time for a restart.
    I have restarted the server today, and the Windows.edb is already 1.8 GB on a RDS server for 7 users.
    I have tried to determine the subprocess of the search service (FilterHost) crashes. But I canโ€™t see that.
    I would like to try to implement this fix. Could you please provide me more info? Have a nice day.
    Best regards.

    1. TeGek February 18, 2016 at 8:33 pm

      Hi Dennis,

      To process this fix all you’d need is the information in the blog post. But seeing you are also noticing a major drop in performance you should check the following:

      1.) Are you using cached mode? If no, enable it. enabling caching gives you a MAJOR increase in performance. If you are worried about disk space consider using Folder Redirection for the profile folders or UPD.

      2.) You’re also noticing a change in icons, which might point to a disk performance issue. If you are using Antivirus software. Please exclude .OST files from this. OST scanning can cause major performance issues.

      3.) If it is still not resolved, You should make performance logs using perfmon. Quite often these issues result from under-dimensioning servers.

      1. Dennis February 22, 2016 at 8:37 pm

        Thank you very much for your help. I will post the result! :0)

    2. Robert September 12, 2017 at 3:08 pm

      Hi Dennis

      Reading your comment was the hope we as a team needed to carry on fighting this issue, having someone else reporting the icons going white along with the degredation. Did you / Have you / Can you found / find a solution to this which you were able to box off the complaints for?

      Dennis. We need you. You’re our only hope.

  7. roux February 12, 2016 at 3:04 pm

    Very Good Thank you for your help

    1. TeGek February 18, 2016 at 8:33 pm

      No problem! thanks for the reply ๐Ÿ™‚

  8. Matt February 16, 2016 at 9:37 pm

    does this fix the (4) disconnect problem?
    it seems that windows search service is the main cause of the (4)disconnects

    1. TeGek February 18, 2016 at 8:34 pm

      Hi Matt,

      I’ve actually never seen users disconnect from sessions due to the search host. I have seen complete session hangs though.

  9. Dennis March 3, 2016 at 11:30 am

    Hi TeGek
    It was a nice try, but unfortunately it didnโ€™t solve my issue. It is very strange. Our Windows 2012 R2 RDS server is running on an IBM X3650 M5 โ€“ Dual Xeon โ€“ with 10K SAS disks. It has been running from November 2014. The symptoms we are facing are the same as you describe. The server works perfectly, but within a week it slows incredible down. We are only 7 users. We have a lot of Outlook activity and share our mailbox across all coworkers, so the OST files are big โ€“ 430 GB for 10 mailboxes.
    The Windows Search Index grows to 5 GB in 3-4 days, and that is slowing server down. If I rebuild the index everything works fine for the week or so.
    There is a lot of disk activity on the searchindexer.exe and, on the *.ost files and C:\ProgramData\Microsoft\Search\Data\Applications\Windows\windows.edb

  10. Rudy Ooms March 21, 2016 at 2:48 pm

    I just applied the reg fix to multiple servers…. I really hope it will fix the:

    -The Search Index grows to extreme sizes(50GB+).

    How did you think of the CoreCount dword key? I could not find it anywhere except on this forum

  11. Jarrod Cavner March 22, 2016 at 2:13 pm

    We are seeing this issue as well and are considering your fix. Did you see a performance degradation in searches after you put the registry key in place? Once we restart the search service to temporarily clear up the issue, we see multiple searches fire off each of them using between 10-15 threads.

    1. TeGek August 11, 2016 at 8:22 pm

      We did not see a performance degradation after adding the key.

  12. Jason Johnson April 19, 2016 at 3:06 pm

    If the value in the registry does not exist do you need to create it. I am not see the entry in the registry?

    1. TeGek August 11, 2016 at 8:16 pm

      Yes ๐Ÿ™‚

  13. Alessandro December 29, 2016 at 11:47 am

    We have same problems described, file very large and often search crash.. We applied the fix adding the registry value and deleting the index files. After a server reboot the search started with indexing slowing than normal,. Today, after two days, i’ve checked the indexing process and it’s more fast.. Verifyng on registry, the CoreCount value it’s not more present.. Nobody have deleted the registry value!.. Maybe some windows application can control and remove the registry? I’ll try to insert the CoreCont value againg….

  14. NZ sys admin January 3, 2017 at 8:47 pm

    Hi

    we are running outlook 2013 with gmail app synch on RDS (2012R2).

    every time the users log in outlook seems to be rebuilding the index. By the end of the day most people have some sort of search functionality but nowhere near as good as the stable index on a local install of outlook. but it’s a real groundhog day experience with it starting from scratch every day.

    We also have a large index file, although not sure we have the same performance issue. That said we have an entire G9 DL380 64GB just for the RDS server (30 users), but the users gmail accounts are typically 20-40GB each. so there may be a performance issue but it is not affecting us.

    we have researched this before and as far as we can see it appears to be a known problem with no solution(for the last 3-4 years). We are not interested in switching to exchange.

    any ideas to avoid the search file reindexing?

  15. Lefteris Krashias March 21, 2017 at 9:46 am

    Hi I’ve the below scenario:

    Windows Server 2012 R2 on Microsoft Azure.
    Server is setup with Terminal Services with 20 Users running Microsoft Office 2010 Standard.

    Problem: The Windows.edb file is increasing and after reaching a limit (don’t know) it rebuild by its own without reason. How to prevent this?

    Please help me.

    Thanks

    1. TeGek March 22, 2017 at 9:32 am

      Hi Lefteris,

      When running in Azure I suggest disabling the search service on the server and disable cached Exchange mode.

      Azure has direct backend access to Office-365 and Online Mode capabilities are near perfect in this case. In online mode you won’t need the search service as you’ll be performing online searches.

      I haven’t heard about the search service rebuilding itself without reason, but you can always try to perform a defrag of the search database to see if it helps in reducing the size.

      Regards,

  16. Edward Smith May 23, 2017 at 5:54 pm

    Can this fix be applied to workstations as well?

    I have several Windows 8.1 workstations where the Windows Search EDB file is growing to 100GB to 200GB in size, then it barfs, shrinks, and regrows. The OST files are between 25GB and 50GB with a *lot* of email attachments. Configuring the search to only index the OST does not change the behavior of the EDB file.

    This is eating up disk I/O and disk space and causing issues.

    On one machine we redirected a daily set of email reports with large attachments to a shared mailbox and that reduced the OST from 50+GB to 25GB, but the Windows Search EDB is stil 100GB in size.

    All machines are operating off Exchange 2013 in cached mode.

    Thanks

    1. TeGek May 29, 2017 at 9:13 pm

      To be honest I have not tried it on workstations, so can’t comment on it. Sorry!

  17. TheAdmin February 16, 2018 at 6:11 pm

    We applied this fix in november 2017 because our RDS server was showing this exact same problem about 3 times a week.

    After applying the registry fix, we didn’t experience the issue anymore until now.
    Today, 16-2-2018, the issue is back while the registry key is still in place.

    Is anyone else experiencing this?

  18. Nathan Gracie-Raitt March 21, 2018 at 12:06 am

    Kelvin, thanks for the helpful blog and your various responses to other people that have reached out after experiencing similar issues. At least half the blogs I’ve read elsewhere about this issue seem to end up pointing here at some point either by the original author by comments submitted.

    My situation is that at the same time as onboarding my client to our MSP we fork-lifted their RDS VM (and other servers) out of a NZ local IaaS facility up to Azure, so the idea you had about turning off Outlook cached mode because the RDS host is now sitting so close the Exchange servers in O365 that just running in Online Mode will work well, is looking quite attractive.

    Before I suggest trying that approach I’d like to understand if you were seeing the massive EDB files that others have commented on, or if you were troubleshooting only the performance/crash issues described in your initial blog. I see someone has already tried implementing the ‘CoreCount’ registry key and because the VM is fairly well spec’d (16 vCPU, 128GB RAM, only ~40 concurrent users), I’m not seeing too much of a performance issue, but the drive where the index files have been moved to is shared with some applications that get upset when they can’t write to logfiles because the entire volume has been filled with an exploding Windows Search EDB file.

    Thanks,
    Nathan

    1. Kelvin Tegelaar March 26, 2018 at 10:27 am

      Hi Nathan,

      We did also see very large search index files. It was a happy coincidence that this seems to have been partily fixed by the corecount registry key, some of our RDS servers no longer experience it, others still grow to insane sizes.

      If you have an RMM system that can monitor using PowerShell files I suggest using the following blog to try and automatically resolve the issues: https://www.cyberdrain.com/370/ – It does give some slight user disruption but overal we’ve found that our clients don’t notice the rebuilds all that much.

      Regards,
      Kelvin

  19. LoneWolf August 20, 2018 at 2:33 pm

    In addition to the OP’s helpful suggestion, I have implemented one other change:

    https://support.microsoft.com/en-us/help/2838018/the-windows-edb-file-grows-very-large-in-windows-8-or-windows-server-2

    This shows how to do an offline defragmentation of the EDB: I have scripted the procedure using our remote management software; for others, I recommend calling Microsoft’s steps as a scripted batch file in Scheduled Tasks. Delete and rebuild your EDB once. Once you have done that, set this scheduled task to run once-per-week (or more often as you need) during after-hours. This should provide additional help. I hope it benefits you all.

  20. Andy September 26, 2019 at 10:09 am

    We implemented this registry modification but the service still errors and stops at least once a month.
    The event IDs are always in pairs: 7040 and 7042. It would be useful to know if it was having problems with a particular file or extension but I don’t see a way of finding that.

  21. Andy May 31, 2020 at 4:20 pm

    I fixed my issue using “Microsoft Support and Recovery Assistant” (former OffCAT).

    Started MS Office diagnostic and got:

    Possible Outlook search or indexing issue (event 10023 or 10024)
    The PDF iFilter is registered and Event 10023 or 10024 was found in the Application Event log. This has been known to cause indexing issues in Outlook. Click the ‘Click here…’ link if you are unable to search for Outlook items or your indexing of Outlook items does not finish.

    Removed Adobe Acrobat and finaly got working WSearch.

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.