Sharepoint applied

While playing with backup/restore scenarios I needed to remove the content database for one of my web applications. On re-attaching the database and navigating to the sites hosted in the web application I was greeted by a "Service Unavailable" error. All other web applications and sites worked continued to work fine.

I always find it interesting to follow people’s troubleshooting steps so below I’m listing the ones I took when investigating the issue. You can scroll down to the end of the article if all you’re after is the solution to the issue:)

The following steps were taken to troubleshoot the issue:

1) Checked the status of the database used by the problematic web application (Sharepoint Central Administration > "Manage content Databases" section) – the status was reported as "ready"

2) Checked event logs – found only 1 error related to the problem:

Event ID: 2436
Source WSS search
Category: Gatherer
The start address <sts3s://[site collection address]/contentdbid={84d79b68-13eb-4cd5-b421-6c44c0c69bed}> cannot be crawled.
Context: Application ‘Search index file on the search server’, Catalog ‘Search’
Details:
An unrecognized HTTP status was received. Check that the address can be accessed using Internet Explorer.   (0×80041204)

I could see the gatherer cannot connect to the website to crawl site. This didn’t however, tell me why the connection is refused/non-operational

3) Restarted the web application’s IIS website in IIS manager
- IIS reset was not an option in the middle of the day.

4) Looked through SharePoint logs - no errors found

5) Launched Sharepoint Manager 2007, navigated to the web application – confirmed I can drill down to subsites, lists etc. The sites and content, therefore, were definitely there.

6) As I couldn’t find anything obvious in event logs or sharepoint logs navigated to C:\WINDOWS\system32\LogFiles\HTTPERR folder and looked through httperr.log files.

What is httperr file? I only learned about httperr logs recently when troubleshooting some pesky IIS issue.

Httperr.log is [pasted from MS site] "a new feature in IIS 6.0. If HTTP.sys is unable to write an error to a Web site log because the request did not reach user mode, the error will be entered in Httperr.log. Errors that are written to Httperr.log include the following:

• 503 errors
• idle connection time-outs
• URL parsing errors
• the last 10 requests to a failed application pool.

If you need some help in decoding the errors logged in the flie you can refer to the following MS article: Error logging in HTTP API which includes a nice description of the different fields and the order of the fields in httperr log files.

Looking through the files I the following two errors getting logged repeatedly

HTTP/1.1 POST /_vti_bin/sitedata.asmx 503 1790625552 Disabled SharePoint+-+90
HTTP/1.1 POST /_vti_bin/sitedata.asmx 503 1370408763 Disabled SharePoint+-+90

So basically attempts to connect to the sitedata.asmx web service resulted in 503 errors (Service Unavailable). I googled IIS 1790625552 and IIS 1370408763 hoping to get some more info but couldn’t find any articles matching the codes.

A bit of searching led me to the following MS article:
503-Service Unavailable (IIS 6.0)

The article lists the different error codes and causes of 503-Service Unavailable IIS errors.

Looking through the error table I found that the "Disabled" word maps to the following cause "The administrator has stopped the application pool.".

So the line in my httperr log:

HTTP/1.1 POST /_vti_bin/sitedata.asmx 503 1790625552 Disabled SharePoint+-+90

was basically telling me my application pool "Sharepoint – 90" was stopped.

I opened IIS Manager, expanded the Application Pools node and noticed the application pool used for the problematic web application was indeed stopped.

(notice the little red icon next to my Sharepoint – 90 application pool)

Right clicking on the application pool and choosing "Start" resolved the issue restarted it and suddenly my sites started working fine again.