Microsoft Exchange Search Host Controller Failing

Got a call from a client that reported user searches in OWA and Outlook were failing. This particular client had recently has some severe SAN issues. So, who knows were the problem could be. However, the place to start is with the indexes, and the index on this server showed as healthy.

Next, I looked at the services. Microsoft Exchange Search was running properly, but Microsoft Exchange Search Host Controller was stopped. When I started Microsoft Exchange Search Host Controller, it showed as running for a few seconds and then stopped.

In the event log, two related events were showing:

Event ID: 1026
Source: .NET Runtime
Application: hostcontrollerservice.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: Microsoft.Ceres.HostController.Controller.HostControllerException
Stack:
   at Microsoft.Ceres.HostController.WcfServer.WcfService.StartService()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

Event ID: 1000
Source: Application Error Faulting application name: hostcontrollerservice.exe, version: 15.0.4454.1006, time stamp: 0x50d08ef5 Faulting module name: KERNELBASE.dll, version: 6.2.9200.16857, time stamp: 0x530e784b
Exception code: 0xe0434352
Fault offset: 0x0000000000047b8c
Faulting process id: 0x413c
Faulting application start time: 0x01cf86480e45a787
Faulting application path: C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\hostcontrollerservice.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 4cc1a310-f23b-11e3-943e-00155d09980c
Faulting package full name:
Faulting package-relative application ID:

I found a couple of references to a corrupt and empty ini file that can cause this issue. The file in question is C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data\Nodes\Fsis\node.ini. However, mine appeared to be good. There was content in the file in any case.

My fix ended up being removing and reinstalling the search services. This was based on advice given to another person by Microsoft support. The steps were:

1. Ensure that both Microsoft Exchange Search and Microsoft Exchange Search Host Controller are stopped. I had to manually kill noderunner.exe processes that were started by Microsoft Exchange Search Host Controller but not stopped when the service failed.
2. Create a backup copy of InstallConfig.ps1 in C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Installer.
3. Edit InstallConfig.ps1 and modify the line for the baseport variable to be $script:baseport:3800. On this system the original value was 17000. I'm not sure if that's consistent or varies across systems.
4. Uninstall search by running: .\InstallConfig.ps1 -u -datafolder "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"
5. Reinstall search by running:   .\InstallConfig.ps1 -i -datafolder "C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\HostController\Data"

Other links for research into this issue:

• http://social.technet.microsoft.com/Forums/exchange/en-US/74aef23c-fe67-4f2f-a74e-4e895db28fa9/microsoft-exchange-search-host-controller-service-stopped-no-update-installed?forum=exchangesvrgeneral
• http://plasmasphere.co.uk/exchange-2013/exchange-2013-hostcontrollerservice-exe-crashes/