HTTP Status Codes for SEO

HTTP status codes are something you really need to understand if you want to stand a hope of having your website ranking consistently well in search engines.

In this post I will explain all of the major codes, when to use them and as importantly when not to use certain codes.

HTTP Status Code terminology

• HTTP – Hypertext Transfer Protocol. The protocol a server uses to give data to the browser so that your visitor can view your website.
• Status Code – Each response a web server sends will come with a status code, this code can be interpreted by the web browser or other programs so it knows how to handle the rest of the data.
• Client – Normally this means a web browser, but it could be any program that is sending requests to a server
• Server – Or Web Server, this is a computer that serves up the content of your website to the browser.

The five major types of HTTP Status Code

HTTP Status Codes are 3 digit numbers, the first of those numbers correspond to a broad major type for the code.

Status Code 100

These are informational codes and they indicate a provisional response. These are not something most people will need to deal with most of the time, you should very rarely explicitly send a 1xx response.

Status Code 200

These are success messages. This means that the action requested by the client was received, understood, accepted and processed successfully.

The most common of these is 200, which means “OK” in HTTP request parlance, this is what will be returned by default when a web page is successfully served up.

Status Code 300

These are all redirection messages. This means that the client needs to perform an additional action in order to correctly complete the request.

These are something that in general you don’t have to deal with as a web developer, but getting a handle on redirects has some really lovely benefits for SEO, which I will cover later.

Status Code 400

These are all client error messages. The most well known of these is the 404 message, but there are others, which I will get into in more details later.

Status Code 500

These are all server error messages. This means that even though the request appeared to be valid something went wrong at the server level and it wasn’t able to return anything.

A fun way to remember these codes

• 1xx coming now
• 2xx here you go
• 3xx go away
• 4xx you screwed up
• 5xx we screwed up

Important Status Codes for SEO

Don’t worry, even though there is the potential for lots of codes there are really only a handful of them that are used regularly and that we will need to care about.

I am going to talk about seven of them today, if you think there are others I should be talking about please let me know in the comments.

200 OK

200 OK is the standard response for a successful request. When the request is genuinely OK then this is perfect.

If you are serving up a 404 page and you have caught the fact something was missing but don’t tell your server to send a 404, then your 404 pages will be coming up as 200, which means if Google follows a link which should be 404 it may try and index your 404 page.

Another time I have seen 200 being used incorrectly is when returning permission denied errors (like 403), if the developer is writing their own REST server they can sometimes return custom messages that will be 200 by default. Again this can send mixed messages to Google so should be avoided.

301 Moved Permanently

301 Moved Permanently means that a resource used to be available at a particular URL but now is available at another URL.

With 301 redirects not only will users be moved to the right place, any authority that particular page had with search engines will move too.

This is the preferred status code when redirecting content on your website

302 Found

302 Found is also known as a “temporary redirect”. From a user’s point of view this is exactly the same as a 301 Redirect because they will enter one page but be moved to another.

The big difference is that search engines will not bother passing any link value to the new location because they assume the redirect is only temporary.

304

304 Not Modified tells the client that the content has not been modified since a desired time. This is incredibly important when you want to consider crawl efficiency. When Google hits a page it will pass in the date that it last tried to access the page. If no content has changed since that time and a 304 comes back it will know it doesn’t need to download your content again.

403 Forbidden

403 Forbidden tells the client that the request was valid but the server refuses to respond to it. This is the standard response you should return to protected files and folders if you do not want them to be index.

There are of course other ways you can tell search engines not to index pages, but pages that return 403 will almost certainly not appear in search results.

404 Not Found

404 Not Found is the status code that almost everyone knows, even if they don’t know what a status code is.

404 Not Found means the resource could not be found at this moment in time. Sometimes this can be a temporary problem and because of this search engines will often try to grab the content from a particular URL again even if it responds with a 404.

You should take care to monitor what 404’s are happening on your website so that you can 301 redirect the user to appropriate content.

410 Gone

410 Gone tells the client that the resource was specifically removed.

This should be used when a resource has been intentionally removed and the resource should be purged from things like search engine results.

Naturally you should be careful using this and it should never be the default behaviour, but this can be a powerful status code to use when you definitely want particular URLs purged from search engine results.

500 Internal Server Error

500 Internal Server Error is something that both users and developers hate, this normally means something has went wrong at a code or database level.

This catch-all status code leaves search engines confused because they cannot infer what was meant to happen. If your website is returning 500 errors at all this needs to be addressed.

One common use-case I have seen is if you throw an error when a resource can’t be found in Ruby on Rails instead of catching the error and returning a 404 instead.

503 Service Unavailable

503 Service Unavailable tells the client that the server is currently unavailable.

This status code should be used during planned server maintenance to let search engines know that even though they may not be able to see something at the moment, there is a good chance if they try again later that it will work.

Returning the correct Status Code

Here is a non-exhaustive list of different ways you can set the status code within various frameworks, if you want me to add your framework of choice to this list please just leave a comment.

Ruby on Rails (Ruby)

	render :status => 503

Sinatra (Ruby)

	get ‘/foo’ do
	  status 418
	  headers \
	    “Allow”   => “BREW, POST, GET, PROPFIND, WHEN”,
	    “Refresh” => “Refresh: 20; http://www.ietf.org/rfc/rfc2324.txt”
	  body “I’m a tea pot!”
	end

CodeIgniter (PHP)

	$this->output->set_status_header(‘401’);

Laravel (PHP)

	return response($content, $status)
		->header(‘Content-Type’, $value);

Express (JavaScript)

	res.status(404).send(“Oh uh, something went wrong”);

Django (Python)

	from django.http import HttpResponse

	class HttpResponseNoContent(HttpResponse):
	    status_code = 204

	def my_view(request):
	    return HttpResponseNoContent()

Spark (Java)

	response.status(401);

Testing Status Code

If you want to test a status code for a specific request you can do this easily with the curl command.

Here is a valid page;

	curl -I tosbourn.com
	HTTP/1.1 200 OK
	...

Compared to a page that doesn’t exist;

	curl -I tosbourn.com/i-do-not-exist
	HTTP/1.1 404 Not Found
	...

I have written about a post and recorded a video about getting status codes from the command line.

418 – I am a tea pot

This isn’t related to SEO in the slightest, but I couldn’t write about status codes without talking about 418 – This code should be returned by tea pots requested to brew coffee. You can read more about it here.

More Status Code Resources

If you want to learn more about all the status codes you can chose to use on your web server then a great start is the Wikipedia page.

Some of the key points from this article have been summed up nicely in this 2011 infographic put out by Moz.

There is a really nice reference site for status codes over at httpstatus.es.

If you want to see some dogs acting out the various status codes, you can!

Thanks

Thank you to Gerry White for pointing out I had missed the 304 Not Modified code out of my list.