How the web works

Table of Contents

• Introduction
• When you start typing into the search bar
• You hit the enter key
• Opening of a socket + TLS handshake
• Making the actual HTTP GET request
• Getting response from the server
• Rendering the UI on the screen
• Conclusion

Introduction

This article gives a concise guide on what your browser does when you search for a web page. There are some things that are known to the general public while others are not so quite common. Lets discuss it here.

Most of the things discussed here are browser independent. But for simplicity, I assume it to be the Google Chrome browser, due to its popularity and widespread adoption.

When you start typing into the search bar

Let suppose you want to go to google.com and you start typing 'g...'. As you go on typing, the browser gives you autocomplete suggestions which may be based on a lot of factors like

• browser search history
• items that are most related to your search history, bookmarks etc.
• browser bookmarks, favorites, frequently visited sites

The DNS queries is not done from scratch every time. It uses all sorts of caching mechanisms available to it. The browser searches for a cache (in the order top to bottom in the list below)

• Browser cache (typically stored for 2-30 minutes)
• OS Cache
• Router Cache
• ISP DNS cache

Most of the browsers continuously make DNS queries for your search text in the background to give more accurate results. This step is very important from a usability standpoint because

• Browsers have a single search box which can be used for both "search by word/keywords/text" as well as "search by website URL"
• And as you type, there are no definite ways to know if the search query is a valid URL. So, if the DNS query for that search term resolves to an IP address, it is a website
• Even when the browser is confident that such and such website exists, it also gives an option to search the same website using a word search

You hit the enter key

The browser parses the search text. It already has some knowledge based on the DNS queries.

• If you directed it as a website URL, it makes a DNS query (if it hasn't already) to get the IP address.
• Else it assumes it to be a text search, and redirects to the browser's default search engine. Search engines have a URL to search by text. For example: In case of google it is https://www.google.com/search?q={{ search query }}

At this point, you have an IP address to make a network call.

Opening of a socket + TLS handshake

• The browser makes a requests to the system library for a TCP Socket stream
• The browser sends a ClientHello with the TLS version, list of cipher suites, compression methods available etc.
• The server replies with a ServerHello message with the TLS version, selected cipher, selected compression methods and server's public certificate (which contains the server's public key that will be used by the browser to encrypt further requests)
• The browser verifies the server certificate against its list of trusted CA so that chain of trust can be established. From this point, the generation of symmetric keys is initiated
• The browser generates a string of pseudo-random bytes, encrypts it with server's public key and then sends to the server.
• The server decrypts it with its own private key and uses these bytes to generate its own copy of the symmetric master key
• The browser sends a finished message to the server encrypting a hash of the transmission up to this point with the symmetric key. The server generates its own hash, decrypts the browser-sent hash to verify if it matches. If it matches, the server sends its own finished message to the browser encrypted with the symmetric key
• The TLS session is established and further communication can be done in a secure encrypted way.

Making the actual HTTP GET request

The browser identifies itself (attaches the User-Agent header), states the types of accepted responses (Accept and Accept-Encoding headers). The Connection header asks the server to keep the TCP connection open for further requests. The browser also attaches any cookies associated with the domain.

All of this data is crafted in the form of a HTTP request format according to the HTTP specifications

GET http://www.github.com/ HTTP/1.1
Accept: text/html
User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) [...]
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
Host: github.com

The server may either close the connection or if headers sent by the browser requested it, keep the connection open to be reused for further requests.

If the HTTP headers sent by the web browser included sufficient information for the web server to determine if the version of the file cached by the web browser has been unmodified since the last retrieval (ie. if the web browser included an ETag header), it may instead respond with a request of the form: 304 Not Modified and no payload, and the web browser instead retrieves the HTML from its cache.

Getting response from the server

• The server verifies that it can accept GET (or any other specified method) requests on the requested endpoint and that the client (browser) is allowed to use this method (by IP, auth etc.)
• The server pulls the content corresponding to the request and returns it as a response

Once the browser receives the HTML file back, it starts parsing it

• Goes from top to bottom
• Upon encountering any links (to other pages, CSS, JavaScript, Images or other assets, prefetch etc.) it repeats the above cycle again over and over until all assets are received.
• The requests for these static assets can be configured in the HTML code with attributes like async and defer

Rendering the UI on the screen

The browser starts to render the received HTML. The steps involves

• Constructing the DOM tree
• Render tree
• Layout of the render tree
• Painting the render tree

This is such a vast topic and has a lot of details involved. This definitely requires a complete article in itself. It would be discussed in a future article.

Conclusion

Although I might have missed a lot of details, but this should give the readers a fair amount of details as to how the web works. Starting from a search query to a final User-Interface, the browser has to do a lot of heavy-lifting, what most of the people take for granted.

I hope upon reading the article, you would appreciate the browser's work and all the engineering done by the very-smart people overtime.