Understanding Web Requests

isab3113

isab3113

— 5 min read
Understanding Web Requests
An astronaut drifts through space.

Web requests form the backbone of how web applications communicate with servers. Understanding how these requests work is crucial for attacking and defending web applications as a security professional. In this post, we'll dive deep into the mechanics of web requests, HTTP/HTTPS protocols, and tools like cURL and Browser DevTools. We’ll also explore how to use different HTTP request methods with real-world examples using cURL and cover various types of HTTP headers that impact web security.

HyperText Transfer Protocol (HTTP)

The HyperText Transfer Protocol (HTTP) is the application-level protocol that powers communication over the World Wide Web. Most web applications, whether simple websites or complex systems, use HTTP to exchange data between the client (your browser) and the server (where the application is hosted).

HTTP operates on a client-server model:

  • The client sends a request.
  • The server processes the request and returns a response.

The default port for HTTP communication is port 80, though this can be configured to other ports.

URLs

A URL (Uniform Resource Locator) is the address used to access resources on the Internet. It consists of several components that direct the client to the correct resource on the server.

Components of a URL:
  1. Scheme: Defines the protocol (http:// or https://).
  2. User Info: Optional credentials (username:password@).
  3. Host: The domain name or IP address (example.com).
  4. Port: Specifies the communication port (:80 or :443).
  5. Path: The resource accessed (/index.html).
  6. Query String: Contains parameters (?search=term).
  7. Fragment: References a specific section of the page (#section1).

HTTP Flow

The HTTP flow describes how a client and server communicate. Here’s a step-by-step outline of the basic HTTP flow:

  1. DNS Resolution: When you type a URL in the browser, the browser queries the DNS server to resolve the domain name into an IP address.
  2. HTTP Request: The browser (client) sends an HTTP request to the server, requesting a resource (e.g., a webpage or image).
  3. Server Processing: The server processes the request and returns a response, typically starting with an index file (like index.html).
  4. HTTP Response: The server sends the HTTP response back to the client with a status code (e.g., 200 OK), headers, and the requested resource.

cURL: Command-Line HTTP Requests

cURL is a command-line tool for sending HTTP requests. It allows developers and penetration testers to interact with servers, download files, and make API requests directly from the command line. This tool is invaluable for web penetration tests, simplifying sending and receiving raw HTTP requests.

Common cURL Commands:
  • Basic GET Request: curl http://example.com
  • Verbose Output: curl -v http://example.com (displays the complete request and response details).
  • Download a File: curl -O http://example.com/file.zip
  • Send Custom Headers: curl -H "User-Agent: CustomAgent" http://example.com

Using cURL, you can efficiently perform actions such as testing login forms, accessing APIs, and downloading files.

Hypertext Transfer Protocol Secure (HTTPS)

One of HTTP's major flaws is that it transmits data in clear text, which means that any attacker positioned between the client and the server can intercept sensitive data. To prevent this, HTTPS (Hypertext Transfer Protocol Secure) was introduced.

HTTPS Overview

HTTPS uses SSL/TLS (Secure Socket Layer / Transport Layer Security) to encrypt the communication between the client and server. This ensures that even if an attacker intercepts the traffic, they cannot read the contents without decrypting it.

Today, most browsers enforce HTTPS, and HTTP is being phased out due to its security risks. Websites that implement HTTPS can be identified by URLs starting with https://.

HTTPS Flow

When a user attempts to connect to a website using HTTPS, the following sequence takes place:

  1. Initial Request: The client sends a request to the server. If the client requests an http:// URL, the server typically redirects the client to the secure https:// version.
  2. Client Hello: The client initiates the SSL/TLS handshake by sending a "Client Hello" message containing information about supported encryption methods.
  3. Server Hello: The server responds with a "Server Hello," containing its SSL certificate.
  4. Key Exchange: The client verifies the server's certificate and exchanges keys to establish an encrypted connection.
  5. Encrypted Communication: All subsequent communication between the client and server is encrypted once the handshake is complete.

This secure flow protects data like login credentials and credit card information from being intercepted or tampered with by attackers.

cURL for HTTPS

When using cURL with HTTPS, the tool automatically handles SSL/TLS handshakes and encryption. However, there are some useful flags to control SSL behavior:

  • Skip SSL Certificate Check: Use the -k flag to bypass certificate validation (use with caution).
curl -k https://example.com
  • View HTTPS Request and Response: You can use -v to view verbose output, which includes SSL/TLS handshake details.
curl -v https://example.com

HTTP Requests and Responses

HTTP Requests

The client makes an HTTP request to request a resource from the server. Each request has a method (such as GET or POST) that indicates the action to be taken.

Here’s a breakdown of an HTTP request:

  1. Method: The type of action to perform (e.g., GET, POST).
  2. Path: The location of the resource (e.g., /home).
  3. Version: The HTTP version (HTTP/1.1).
  4. Headers: Key-value pairs that provide additional details (e.g., Host, User-Agent).
  5. Body: The data sent to the server (only with POST, PUT, etc.).

HTTP Responses

Once the server processes the request, it sends back a response. An HTTP response includes:

  1. Status Code: Indicates the result of the request (e.g., 200 OK, 404 Not Found).
  2. Headers: Additional information about the response (e.g., Content-Type, Server).
  3. Body: The response content (e.g., HTML, JSON, images).

HTTP Headers

HTTP headers are crucial for both requests and responses. They contain metadata that informs the server or client how to process the communication. Headers are categorized as General Headers, Entity Headers, Request Headers, Response Headers, and Security Headers.

General Headers

General headers are used in both requests and responses. They describe the communication itself rather than the content.

  • Date: Indicates when the message originated (e.g., Date: Wed, 16 Feb 2022 10:38:44 GMT).
  • Connection: Controls whether the network connection remains open (Connection: keep-alive or Connection: close).

Entity Headers

Entity headers describe the content being transferred between the client and the server. These are often used with POST/PUT requests and server responses.

  • Content-Type: Describes the data format sent (e.g., Content-Type: application/json).
  • Content-Length: Specifies the size of the content (e.g., Content-Length: 385).

Request Headers

Request headers provide information to the server about the request being made.

  • Host: Specifies the domain being queried (Host: example.com).
  • User-Agent: Identifies the client making the request (User-Agent: Mozilla/5.0).
  • Authorization: Used for passing authentication credentials (Authorization: BASIC YWRtaW46YWRtaW4=).

Response Headers

Response headers provide additional context about the server's response to the client.

  • Server: Information about the web server (e.g., Server: Apache/2.2.14).
  • Set-Cookie: Passes cookies to the client (Set-Cookie: sessionId=abcd1234).

Security Headers

Security headers are essential for enforcing security policies between the browser and server.

  • Content-Security-Policy (CSP): Prevents cross-site scripting (XSS) by limiting the sources of content (Content-Security-Policy: script-src 'self').
  • Strict-Transport-Security (HSTS): Forces browsers to use HTTPS (Strict-Transport-Security: max-age=31536000).

Examples of HTTP Methods Using cURL

Here are practical examples of how to use cURL for different HTTP request methods.

1. GET Request:

curl http://example.com

2. POST Request:

curl -X POST -d "username=admin&password=secret" http://example.com/login

3. PUT Request:

curl -X PUT -H "Content-Type: application/json" -

d '{"username":"newuser"}' http://example.com/api/users/1

4. DELETE Request:

curl -X DELETE http://example.com/api/users/1

5. HEAD Request:

curl -I http://example.com

6. OPTIONS Request:

curl -X OPTIONS http://example.com/api

7. PATCH Request:

curl -X PATCH -H "Content-Type: application/json" -d '{"username":"patcheduser"}' http://example.com/api/users/1

Conclusion

Understanding web requests, HTTP/HTTPS protocols, and the tools that allow us to manipulate and analyze these requests (like cURL and Browser DevTools) is essential for web security professionals. These skills form the foundation of web security, allowing you to assess vulnerabilities, identify security flaws, and protect against potential threats. Whether it's understanding HTTP methods, headers, or the details of SSL encryption in HTTPS, mastering these concepts will enhance your ability to secure web applications effectively.

Read more