I've talked a little about HTTP and how it works here, but I don't think I ever really covered the basics of requests over HTTP.

IP/TCP & HTTP

Requests over the internet generally will use HTTP, it is the standard for web communication. Let's start with how the HTTP requests and responses are sent between a client and a server.

A connection between client and server is layered. IP finds the server, TCP opens a bridge between the client and server, then information is exchanged via HTTP.

IP is the first layer. IP (Internet Protocol) specifies how datagrams (packets of data) are sent between hosts (client and server in this case). Each packet contains the IP Address of the source host and destination host. IP is responsible for routing the packet from the source to the destination, as the packet travels through the network, any node it reaches checks the destination IP address and sends it on its way. IP only ensures that a single packet reaches it's destination.

TCP is the next layer on top of IP. IP only uses best effort to get a packet to its destination. That means that packets may reach the destination in a different order than the one in which they were sent, reach the destination multiple times, or never reach the destination at all. TCP (Transmission Control Protocol) sits on top of IP, and provides a reliable, ordered, error-checked delivery of the packets. TCP connections allow data to flow in both directions.

HTTP is the final layer. It uses TCP to connect the client and server, and then requests and responses can be sent back and forward.

HTTP Requests

To explain HTTP a little more in depth, lets start with a basic example, visiting a website in your web browser.

You open up your browser of choice, the client, and you put https://joshglasson.co.uk into the URL bar at the top. Once you press enter, the first thing that happens is your browser extracts the https to decide the protocol it should use to attempt to make a connection. My previous blog post mentioned in detail how encryption over HTTPS works, and I don't want to go into that level of detail at the moment, so please check out this blog if you want to read more on that.

Running curl -v 'https://joshglasson.co.uk will show me the below.

> GET / HTTP/2 
> Host: joshglasson.co.uk

As a side note to this, if I tried instead curl -v 'http://joshglasson.co.uk, using http instead of https, I would see the below:

> GET / HTTP/1.1
> Host: joshglasson.co.uk

< HTTP/1.1 301 Moved Permanently

Redirecting to https://joshglasson.co.uk/

Because my website doesn't support http (as it is old, outdated, and insecure), if you try to connect to it via http, you will receive an error and be redirected to the https version.

So now that the browser knows how it should be communicating, it can figure out where it needs to send request. In the url, the user has put joshglasson.co.uk, but computers don't work using the human readable url that we enter. Instead, they use IP addresses. The browser will ask a DNS server (there are a few options, such as Cloud Flare and Google) what the IP address is for the domain name joshglasson.co.uk, and if that domain is registered, it will respond with the correct IP address. Again this is covered in more detail here, including more info on IP addresses.

Now that the browser has the IP address, it is able to open the TCP connection to it. There is some handshaking that needs to be done, especially with https, more detail on that can be found here. Once the TCP connection is open, the browser can start sending requests. Lets have a look at the request:

The request line

> GET / HTTP/2 

This is showing that the request that is being sent is a GET request, and that we are using HTTP 2. I will cover the different request methods later on. This is the Request line, the main part of the request.

The request header

Along with the request you can send request headers. Most are optional, except Host (and Connection prior to HTTP 2):

> Host: joshglasson.co.uk

Headers allow for additional information to be sent, for example if you are accessing sensitive data you may need an authorisation token, which would be passed in the Authorization header.

The request body

We are looking at a GET request at the moment, which does not include a request body. I will cover the request body when I come to looking at the other HTTP methods.

The response

< HTTP/2 200

As I was able to successfully connect to the server and my request was valid, I received the above response. 200 represents OK, meaning that the server sent the response, and nothing else. Different response codes will show you other states, such as 201 which means that a resource has been created on the server based on the request sent.

This response should then contain all of the code for the website, which the browser will then parse into what you see.

HTTP Methods

GET

The GET method requests a representation of the specified resource. Requests using GET should only retrieve data.

HEAD

The HEAD method asks for a response identical to that of a GET request, but without the response body.

POST

The POST method is used to submit an entity to the specified resource, often causing a change in state or side effects on the server.

PUT

The PUT method replaces all current representations of the target resource with the request payload.

DELETE

The DELETE method deletes the specified resource.

CONNECT

The CONNECT method establishes a tunnel to the server identified by the target resource.

OPTIONS

The OPTIONS method is used to describe the communication options for the target resource.

TRACE

The TRACE method performs a message loop-back test along the path to the target resource.

PATCH

The PATCH method is used to apply partial modifications to a resource.

REST

I talked about REST in another post, but I wanted to add a little more around REST standards.

When a client request is made via a RESTful API, it transfers a representation of the state of the resource to the requester or endpoint. Headers are also very important in RESTful APIs as they contain information that is useful to the server.

For an API to be RESTful, it has to conform to these criteria:

1. Client–server – By separating the user interface concerns from the data storage concerns, we improve the portability of the user interface across multiple platforms and improve scalability by simplifying the server components.
2. Stateless – Each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client.
3. Cacheable – Cache constraints require that the data within a response to a request be implicitly or explicitly labeled as cacheable or non-cacheable. If a response is cacheable, then a client cache is given the right to reuse that response data for later, equivalent requests.
4. Uniform interface – By applying the software engineering principle of generality to the component interface, the overall system architecture is simplified and the visibility of interactions is improved. In order to obtain a uniform interface, multiple architectural constraints are needed to guide the behavior of components. REST is defined by four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state.
5. Layered system – The layered system style allows an architecture to be composed of hierarchical layers by constraining component behavior such that each component cannot “see” beyond the immediate layer with which they are interacting.
6. Code on demand (optional) – REST allows client functionality to be extended by downloading and executing code in the form of applets or scripts. This simplifies clients by reducing the number of features required to be pre-implemented.

Source: https://restfulapi.net/.

Naming Conventions

REST APIs use URIs to address resources. When naming the API, a RESTful URI should refer to a resource that is a thing (noun) instead of referring to an action (verb) because nouns have properties which verbs do not have – similar to resources have attributes.

Lets pretend we have a Spotify like music service, Notify, that we are trying to design an API for. The first part of the URI is the DNS name:

http://api.notify.com

The API needs to be able to manage users, and the playlists on their accounts. So we could have:

http://api.notify.com/user-management
http://api.notify.com/song-management

If we wanted to get a list users, or one specific user, the URI would look like the below:

GET http://api.notify.com/user-management/users
GET http://api.notify.com/user-management/users/{userId}

userId in the above example referring to a users id that would be passed in with the request to the API.

Similarly we could have endpoints for viewing all playlists, or a specific playlist for a user:

GET http://api.notify.com/song-management/users/{userId}/playlists
GET http://api.notify.com/song-management/users/{userId}/playlists/{playlistId}

We may also have an API for the user to control the player:

GET http://api.notify.com/song-management/users/{userId}/playlists/{playlistId}/play

Some more conventions for naming are:

• Use / to indicate a hierarchical relationship
• No trailing /'s
• Use hyphens between words to improve readability
• Use hyphens and not underscores
• Use lowercase letters
• Do not use file extensions

CRUD expressions should never be used in names either. The CRUD function should be defined by the HTTP method as in the below example:

// WRONG
GET http://api.notify.com/song-management/users/{userId}/playlists/{playlistId}/add?song=songId1,songId2

// RIGHT
PUT http://api.notify.com/song-management/users/{userId}/playlists/{playlistId}
Request Body:
{
    songs: [
        songId1,
        songId2
    ]
}

This means you can use the same endpoint to do multiple operations depending on the HTTP method:

// ADD
PUT http://api.notify.com/song-management/users/{userId}/playlists/{playlistId}/{songId}
// REMOVE
DELETE http://api.notify.com/song-management/users/{userId}/playlists/{playlistId}/{songId}

You can also use query parameters to filter:

http://api.notify.com/user-management/users?region=USA

API Call Flow

Here is an example of how an API call using HTTP may look:

Let's go through it step by step to see what is going on.

Firstly, we have our client, where the request is being made. This could be your web browser, or an app on your phone.

The client then makes a request to an api. The request in this case is a simple GET call to the api https://api.com/endpoint. The request is made using HTTP.

The client needs to turn the Domain Name (api.com) into an IP address, so it send the Domain Name to a DNS Resolver.

The DNS Resolver returns an IP Address to the client, so it knows where to send the request.

The request is sent using IP/TCP to make the network connection. The IP Address leads to an API Gateway, which is an API Management tool that sits between the client and the backend. The API Gateway allows you to have a single entry point for all incoming requests, and monitor and rate limit incoming requests. You can also do authentication and validation here if you want.

After passing through the API Gateway, the request then reaches a load balancer. The load balancer sits in front of the cloud application. Assuming your application has multiple servers/instances, the load balancer will decide which one to send the request to. This could be based on the current load of each server, a round robin approach, or a number of other algorithms.

The load balancer sends the request onto one of the servers. The server receives the message and it is sent on to the backend application that is running on the server.

The application may have some authentication it wants to do based on who can access the endpoint. If the authentication is successful, then the application can process the request.

The response is then sent back over IP/TCP, first going through the API Gateway. Again this allows you to do logging, and perhaps some validation if you want.

The response then reaches the client, who processes it and shows it to the user.

HTTP Status Codes

HTTP response status codes indicate if a request has been successful. There are 5 classes of response:

1. 1xx - Informational Responses (100 - 199).
2. 2xx - Successful Responses (200 - 299).
3. 3xx - Redirects (300 - 399).
4. 4xx - Client Errors (400 - 499).
5. 5xx - Server Errors (500 - 599).

See this website (or my preference) for a full list of HTTP Status Codes.

The main ones however are:

2xx - Success

200 - OK

The request has succeeded. The information returned with the response is dependent on the method used in the request.

201 - Created

The request has been fulfilled and resulted in a new resource being created. There may or may not be information returned.

204 - No Content

The server successfully processed the request, but is not returning any content.

4xx - Client Error

400 - Bad Request

The request cannot be fulfilled due to bad syntax. General error when fulfilling the request would cause an invalid state. Domain validation errors, missing data, etc. are some examples.

401 - Unauthorized

The request requires user authentication via a header. 401 is used when the request is missing or has an invalid authentication token.

403 - Forbidden

The server understood the request, but is refusing to fulfill it. 403 is used when the user has passed something valid n the header, but they are not authorised to perform the operation.

404 - Not Found

The server has not found anything matching the request.

409 - Conflict

The request could not be completed due to a conflict with the current state of the resource. The response should contain information on the details of the conflict and how they could resolve it.

418 - I'm a teapot

5xx - Server Error

500 - Internal Server Error

The server encountered an unexpected condition which prevented it from fulfilling the request. The general catch-all error when the server-side throws an exception.

503 - Service Unavailable

The server is currently unable to handle the request. May indicate a problem with a downstream system.