DougHughes.net

From time to time I’ve been able to debug difficult problems by manually making an HTTP request. For example, say you’re using cfcontent to serve binary documents and forget the reset=”yes” attribute. You might end up with some invalid data before your actual binary data which would corrupt the document.

Obviously, in this example you would be smart enough to add the reset=”yes” attribute and get on with your life. But, maybe you’ve forgotten about it or there’s some other problem dealing with the HTTP response which you want to debug. In this case you could make a manual HTTP request to the affected URL and take a look at what exactly is coming back. A browsers job is to hide this from you, so you’ve got to go a layer lower.

A 30 Second Explanation/Review of HTTP:

HTTP, or Hypertext Transport Protocol, is the protocol used to send data to and from web servers. Any time you request a webpage or image over the web you’re using HTTP. That’s where the little http:// you’re always typing comes from.

HTTP connections are really TCP/IP connections over which HTTP instructions are sent. Without getting into details (where I would probably trip up), TCP/IP is a means of making connections to other computers. With TCP/IP you connect to a specific “port” on which another computer listens for a connection. These ports are numbered. Certain port numbers are usualy used for specific purposes. In the case of the web the standard port is 80. This means your web server listens for requests on Port 80 and your web browser makes connections to port 80.

HTTP instructions are a set of commands which tell the web server what it is that you’re requesting so that it can return it. I’m not going to get into details here, but the standard which is used everywhere is HTTP 1.1. The full HTTP standard (known as an RFC) can be seen in a number of places. Here’s one: http://www.w3.org/Protocols/rfc2616/rfc2616.html

When you request a page from a web server your browser makes and TCP/IP connection to the web server. HTTP instructions are sent to the web server and the web server responds with the data requested.

The Anatomy of a Simple HTTP Request

Here’s a very simple HTTP request:

GET /index.cfm?test=1 HTTP/1.1
host: www.myserver.com

Note: The request is followed by a blank line.

This request is, for all intents and purposes, the same thing as typing the following url into your browser:

http://www.myserver.com/index.cfm

Here is the breakdown of the first line of the HTTP request:

GET

This is the method to use when making the HTTP request. When sending manually you will probably want to use GET if possible as POST is a bit more complicated. There are several other methods that are less well known such as HEAD, PUT, DELETE. Each has it’s own purpose, but it’s out side the scope of this article to get into them.

/index.cfm?test=1

This is the absolute path to the file you’re trying to access on the server. URL parameters can be attached to this the way you’re used to doing. In this example URL variable test is passed as “1”. If no file is to be named then just pass “/”.

HTTP/1.1

This tells the server that this is an HTTP 1.1 request, which is pretty much guaranteed to be what you want. Http 1.0 is not sufficient for servers that use virtual hosts. To the best of my knowledge, no other versions of HTTP are widely supported or in use.

Everything which follows the first line are request headers. Request headers add additional data which may modify the server response. These can be things like host headers, cookies, instructions on the type of data the browser will accept and more. Aside from the host header, I’m not going to get into these in this article. If you want more details (and you want to sleep well) then read the RFC.

Headers follow the pattern of:

Header-name: header-value

The example HTTP request above shows a header named “host” with a value of “www.myserver.com”. This tells your server which domain name the request is for and it uses that to look up which virtual server to use.

Each header is put on its own line. Following the headers there must be one blank line. This is always blank.

Everything following the blank line is the body of the HTTP request. This is where posted form data is sent if you’re using the POST method. This can be very complicated if you’re sending binary data and I’m not going to get into that. Let it suffice to say that if you need to send form data the form values are usually sent in the same format as URL variables.

Here’s a quick example of a POST request which also sends URL variables:

POST /index.cfm?test=1 HTTP/1.1
host: www.myserver.com

postedVar=postedValue&anotherVar=Another%20Value

Note: This request has an empty line after the headers and after the request body.

Sending the HTTP Request

Now that you know how to craft HTTP requests sending them to the server is the next logical step. To do this, simply telnet to your server on port 80 and send the request.

I suggest first typing your request out in a text editor before making the connection. This is because you won’t see what you type into the opened connection echoed back to you.

You would establish a connection by opening a command prompt and typing:

telnet www.myserver.com 80

This will open a TCP/IP connection to port 80 on your web server. When you see a message saying you’re connected to the server, simply paste the request you wrote in text editor. Hit the enter key if you forgot the last empty line in the request and you will see an HTTP response come back from the server.

The response will include the full HTTP headers and the raw data sent by the server. This data can be binary data and will probably look like garbled text. In some cases you may need to increase the buffer in your command window so you can see the entire response.

At this point you can see the full HTTP response and you might be able to use the information in it to debug your problem.

Good Luck!