API Request Cookbook

Categories: #web  #database 
Tags: #python  #requests  #sqlite 

Resources

• requests: The standard library for making HTTP requests
• pandas: For data manipulation and analysis
• sqlite3: For local database storage
• json: For parsing and handling JSON data
• aiohttp: For asynchronous HTTP requests
• httpx: Modern HTTP client with async support
• beautifulsoup4: For parsing HTML responses
• pydantic: For data validation and settings management
• xmltodict: For parsing XML responses

1. Making API requests with requests
2. Processing the response data with pandas
3. Storing results in SQLite for persistence
4. Using json for parsing API responses

This combination gives a robust foundation for handling API data workflows, from retrieval to storage and analysis.

1import requests
2import pandas as pd
3import json

Links:
https://requests.readthedocs.io/en/latest/api/

Interactive Notebooks:
../../notebooks/

API Calls

1#Simple API Call to an endpoint
2endpoint = "https://www.michaelmuratov.com"
3response = requests.get(endpoint)
4data = response.json()

The endpoint variable contains the target API URL, which in this example is https://www.michaelmuratov.com.

Using the requests library, send an HTTP GET request to this endpoint with requests.get(endpoint).

When the server responds, store the result in the response variable. Since modern APIs typically use JSON for data exchange, I convert the response text into a structured format using response.json().

Base64 Auth

Base64 encoding is commonly used in HTTP requests for authentication and data transfer. Here’s how it works:

1. Basic Authentication:

   • Username and password are combined with a colon: username:password
   • The string is encoded to Base64
   • Added to the Authorization header as: Basic <base64_string>

2. Common Use Cases:

   • HTTP Basic Authentication
   • API Key encoding
   • Binary data transfer in JSON
   • File uploads

1from requests.auth import HTTPBasicAuth
2import base64
3
4#with library
5response = requests.get(endpoint, auth=HTTPBasicAuth('user', 'pass'))
6
7#manually
8auth_string = "userid:password"
9b64Val = base64.b64encode(auth_string)

Over Proxy

When making requests through a proxy server, there are several important considerations:

1. Proxy Configuration:

   • HTTP and HTTPS proxies can be configured separately
   • Proxy authentication may be required
   • Some proxies may require specific headers

2. Common Issues:

   • Proxy connection timeouts
   • SSL/TLS verification with proxies
   • Proxy authentication failures
   • DNS resolution through proxy

3. Best Practices:

   • Always verify proxy connectivity before making requests
   • Handle proxy authentication securely
   • Consider proxy performance impact
   • Monitor proxy connection status

When working with proxies in Python requests, these are the important factors to consider:

• Basic Proxy Configuration
• Proxy Authentication
• SSL/TLS Verification
• Proxy Connection Timeout
• Proxy Error Handling

Basic Proxy Configuration

 1# Define where the proxy is located
 2http_proxy  = "http://10.10.1.10:3128"
 3https_proxy = "https://10.10.1.11:1080"
 4
 5# Define that the same proxy is handling both http and https requests
 6proxies = { 
 7  "http"  : http_proxy,
 8  "https" : https_proxy,
 9}
10
11# Proxy Authentication
12auth = HTTPProxyAuth("user", "pass")
13
14# SSL/TLS Verification
15verify_ssl = True
16
17# Proxy Connection Timeout
18timeout = 5.0
19
20try:
21    # Making a request through a proxy
22    response = requests.get(endpoint, proxies=proxies, auth=auth, verify=verify_ssl, timeout=timeout)
23    # Raise an exception if the request was unsuccessful
24    response.raise_for_status()
25except Exception as e:
26    print(e)

With SSL Verification

When working over long timeframes with SSL/TLS certificates, there are also some other important considerations:

1. SSL/TLS Certificate Validation:

   • Verify the server’s SSL certificate
   • Check the certificate chain

2. Common Issues:

   • Certificate validation errors
   • SSL/TLS handshake failures
   • Certificate expiration
   • Certificate revocation

3. Best Practices:

   • Always verify SSL/TLS certificates
   • Handle certificate validation errors gracefully
   • Monitor certificate status

Different SSL Approaches

1#no SSL verification
2requests.get(endpoint, verify=False)
3
4#Use default SSL verification
5requests.get(endpoint, verify=True)
6
7#Use custom SSL verification
8requests.get(endpoint, verify=True, cert="cert.pem")

How to obtain a website’s SSL certificate

1. Open the website in a browser
2. Click on the i or settings icon in the address bar
3. Click on the Connection is secure in the dropdown
4. Select Certificate is valid

5. Go to the Details tab
   
6. Select the top root certificate
7. Click on the Copy to File button
8. Select Base64-encoded ASCII, certificate chain file format and save the file
9. Move the file to the same directory as the script and rename it to cert.pem
10. Reference the file in the script with cert="cert.pem"

This will allow you to make requests over SSL/TLS without having to worry about the certificate. The certificate chain format allows you to submit the entire chain of trust from the root certificate to the server certificate.

API Caching

API caching is a technique used to store and retrieve API responses to avoid redundant requests. This can be particularly useful when dealing with frequently accessed data or when working with large datasets.

It is also useful during development to avoid hitting rate limits or quotas and to speed up the development process. It can be extremely annoying to wait for a request to complete, especially when working with large datasets.

Storing API Requests in a Standardized Format

To efficiently store API requests and their responses, you can use a SQLite database. This allows you to keep a structured record of each request, which can be useful for debugging, analytics, or auditing purposes. Below is an example of how you can set up a SQLite database to store API requests:

To set up a SQLite database for storing API requests, you can follow these steps:

Create a Database Connection:
Use the sqlite3 library in Python to connect to a SQLite database file. If the file does not exist, it will be created.

Establish the Header Columns:

• Epoch time
• Request Method (Get, Post, etc.)
• Request Headers JSON
• Endpoint
• Query
• Input Data
• Response Code
• Response JSON

Create and Commit

 1import sqlite3
 2
 3# Connect to the SQLite database (or create it if it doesn't exist)
 4connection = sqlite3.connect('api_requests.db')
 5
 6# Create a cursor object using the connection
 7cursor = connection.cursor()
 8
 9# Create a table to store API requests if it doesn't already exist
10cursor.execute('''
11CREATE TABLE IF NOT EXISTS api_requests (
12    id INTEGER PRIMARY KEY AUTOINCREMENT,
13    request_method TEXT NOT NULL,
14    request_headers TEXT,
15    endpoint TEXT NOT NULL,
16    query TEXT NOT NULL,
17    response_code INTEGER,
18    response TEXT,
19    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
20)
21''')
22
23# Commit the changes and close the connection
24connection.commit()
25connection.close()

Related content:

Basic Reverse Shell Guide

AI Web Security Lab

Launching my AI Web Security Lab — Here’s Why I’m Building It

Binary Search in Python

And you don’t need to memorize how binary search works