From time to time I’ve been able to debug difficult problems by manually making an HTTP request. For example, say you’re using cfcontent to serve binary documents and forget the reset=”yes” attribute. You might end up with some invalid data before your actual binary data which would corrupt the document.
Obviously, in this example you would be smart enough to add the reset=”yes” attribute and get on with your life. But, maybe you’ve forgotten about it or there’s some other problem dealing with the HTTP response which you want to debug. In this case you could make a manual HTTP request to the affected URL and take a look at what exactly is coming back. A browsers job is to hide this from you, so you’ve got to go a layer lower.
A 30 Second Explanation/Review of HTTP:
HTTP, or Hypertext Transport Protocol, is the protocol used to send data to and from web servers. Any time you request a webpage or image over the web you’re using HTTP. That’s where the little http:// you’re always typing comes from.
HTTP connections are really TCP/IP connections over which HTTP instructions are sent. Without getting into details (where I would probably trip up), TCP/IP is a means of making connections to other computers. With TCP/IP you connect to a specific “port” on which another computer listens for a connection. These ports are numbered. Certain port numbers are usualy used for specific purposes. In the case of the web the standard port is 80. This means your web server listens for requests on Port 80 and your web browser makes connections to port 80.
HTTP instructions are a set of commands which tell the web server what it is that you’re requesting so that it can return it. I’m not going to get into details here, but the standard which is used everywhere is HTTP 1.1. The full HTTP standard (known as an RFC) can be seen in a number of places. Here’s one: http://www.w3.org/Protocols/rfc2616/rfc2616.html
When you request a page from a web server your browser makes and TCP/IP connection to the web server. HTTP instructions are sent to the web server and the web server responds with the data requested.
The Anatomy of a Simple HTTP Request
Here’s a very simple HTTP request:
GET /index.cfm?test=1 HTTP/1.1 host: www.myserver.com
Note: The request is followed by a blank line.
This request is, for all intents and purposes, the same thing as typing the following url into your browser:
Here is the breakdown of the first line of the HTTP request:
This is the method to use when making the HTTP request. When sending manually you will probably want to use GET if possible as POST is a bit more complicated. There are several other methods that are less well known such as HEAD, PUT, DELETE. Each has it’s own purpose, but it’s out side the scope of this article to get into them.
This is the absolute path to the file you’re trying to access on the server. URL parameters can be attached to this the way you’re used to doing. In this example URL variable test is passed as “1”. If no file is to be named then just pass “/”.
This tells the server that this is an HTTP 1.1 request, which is pretty much guaranteed to be what you want. Http 1.0 is not sufficient for servers that use virtual hosts. To the best of my knowledge, no other versions of HTTP are widely supported or in use.
Everything which follows the first line are request headers. Request headers add additional data which may modify the server response. These can be things like host headers, cookies, instructions on the type of data the browser will accept and more. Aside from the host header, I’m not going to get into these in this article. If you want more details (and you want to sleep well) then read the RFC.
Headers follow the pattern of:
The example HTTP request above shows a header named “host” with a value of “www.myserver.com”. This tells your server which domain name the request is for and it uses that to look up which virtual server to use.
Each header is put on its own line. Following the headers there must be one blank line. This is always blank.
Everything following the blank line is the body of the HTTP request. This is where posted form data is sent if you’re using the POST method. This can be very complicated if you’re sending binary data and I’m not going to get into that. Let it suffice to say that if you need to send form data the form values are usually sent in the same format as URL variables.
Here’s a quick example of a POST request which also sends URL variables:
POST /index.cfm?test=1 HTTP/1.1 host: www.myserver.com postedVar=postedValue&anotherVar=Another%20Value
Note: This request has an empty line after the headers and after the request body.
Sending the HTTP Request
Now that you know how to craft HTTP requests sending them to the server is the next logical step. To do this, simply telnet to your server on port 80 and send the request.
I suggest first typing your request out in a text editor before making the connection. This is because you won’t see what you type into the opened connection echoed back to you.
You would establish a connection by opening a command prompt and typing:
telnet www.myserver.com 80
This will open a TCP/IP connection to port 80 on your web server. When you see a message saying you’re connected to the server, simply paste the request you wrote in text editor. Hit the enter key if you forgot the last empty line in the request and you will see an HTTP response come back from the server.
The response will include the full HTTP headers and the raw data sent by the server. This data can be binary data and will probably look like garbled text. In some cases you may need to increase the buffer in your command window so you can see the entire response.
At this point you can see the full HTTP response and you might be able to use the information in it to debug your problem.
Comments on: "Tutorial on Sending Manual HTTP Requests" (10)
Other handy tools for debugging HTTP requests are LiveHTTPHeaders/Mozilla, wget and Ethereal.
LiveHTTPHeaders is a plugin for Mozilla/Firefox that allows you to see the headers exchanged in requesting/receiving a given URL
wget is available on Windows through Cygwin; on OS X through fink; it allows you to submit HTTP requests and to see a full log of all HTTP headers exchanged (using the -S option)
Ethereal is a packet-sniffer, you can use it to monitor all network traffic; it is particularly handy in that you can have it monitor all incoming/outgoing HTTP requests and see both the headers and data exchanged; this is a very powerful tool
I really want to say something to mock you for doing things the easy way. 😉
Actually, I really appreciate the information. I will check Ethereal and LiveHTTPHeaders out ASAP. Thanks!
I got the information on HTTP requests. I would like to know how to add custom into request headers?
Gopi – It’s quite simple. In the example above I show this http request:
POST /index.cfm?test=1 HTTP/1.1
If you wanted to add a custom header in there you’d simply add it after the host and before the empty line. IE:
POST /index.cfm?test=1 HTTP/1.1
That request has a custom header named “customHeader” with a value of “foobar”
Result #2 on Google search “crafting http requests”
I’ve got the w3c spec in another tab, but this just saved me an hour of reading =)
Good article. Thanks
A POST request shouldn’t work without the compulsory Content-Length header.
after entering >telnet http://www.google.com 80
i can’t see the typed body ..
Thanks for the post…
I run into nice free tool to send http request and get the response http headers, http cookie and more:
Try it you can learn alot about HTTP using this tool.