icaoberg / Other topics - Intro to requests

Created Sun, 03 May 2026 00:00:00 +0000 Modified Sun, 03 May 2026 09:38:27 -0400

Jupyter Notebook python

Other topics - Intro to requests¶

requests is the most popular Python library for making HTTP calls. It turns a network request into a single, readable function call.

This notebook covers:

Installation & Import
What is HTTP?
Your First Request
Query Parameters
Custom Headers
Downloading JSON
Downloading Images
POST Requests — Sending Data
PUT and DELETE Requests
Error Handling
Sessions
What requests Can't Do
Where to Find More APIs: public-apis

1. Installation & Import¶

requests ships with Anaconda. To install it manually run:

pip install requests

In [1]:

# !pip install requests  # uncomment if not already installed

import requests

print(f'requests version: {requests.__version__}')

requests version: 2.32.5

2. What is HTTP?¶

2.1 The Request / Response Model¶

HTTP (HyperText Transfer Protocol) is the language your browser uses to talk to web servers. Every interaction follows the same two-step pattern:

Request — your computer sends a message asking for something.
Response — the server replies with data (or an error).

Think of it like ordering at a restaurant: you place an order (request) and the kitchen brings your food (response). requests is the waiter that carries those messages for you.

2.2 HTTP Verbs¶

Every request uses a verb (also called a method) that tells the server what you want to do:

Verb	Meaning	Analogy
`GET`	Read / fetch data	Looking something up in a book
`POST`	Create new data	Submitting a form
`PUT`	Replace existing data	Overwriting a file
`PATCH`	Partially update data	Editing one field in a form
`DELETE`	Remove data	Deleting a file

Most read-only operations (loading a webpage, fetching an API) use GET.

2.3 Status Codes¶

Every response includes a status code — a three-digit number that tells you what happened:

Range	Meaning	Common Examples
1xx	Informational	100 Continue
2xx	Success	200 OK, 201 Created, 204 No Content
3xx	Redirect	301 Moved Permanently, 302 Found
4xx	Client error	404 Not Found, 400 Bad Request, 401 Unauthorized, 403 Forbidden
5xx	Server error	500 Internal Server Error, 503 Service Unavailable

A 2xx code means everything worked. A 4xx means you made a mistake in the request. A 5xx means the server had a problem.

2.4 Headers¶

Headers are key-value pairs attached to every request and response. They carry metadata — information about the data, not the data itself.

Common request headers:

Header	Purpose
`User-Agent`	Identifies your client (browser, script, app)
`Accept`	Tells the server what format you want back (`application/json`, `text/html`)
`Authorization`	Carries credentials (API keys, tokens)
`Content-Type`	Describes the format of the body you're sending

You will set headers in sections 5 and 8.

3. Your First Request¶

3.1 `requests.get()` and the Response Object¶

httpbin.org is a free service that echoes back information about your request — perfect for learning.

Call requests.get(url) and store the result in a variable. The return value is a Response object that holds everything the server sent back.

In [2]:

r = requests.get('https://httpbin.org/get')

print('Type:', type(r))
print('Status code:', r.status_code)

Type: <class 'requests.models.Response'>
Status code: 200

3.2 Exploring the Response Object¶

The Response object has several useful attributes:

In [3]:

r = requests.get('https://httpbin.org/get')

print('Status code :', r.status_code)   # e.g. 200
print('OK?         :', r.ok)             # True for 2xx codes
print('URL         :', r.url)            # final URL after any redirects
print('Elapsed     :', r.elapsed)        # how long the request took
print()
print('--- Response headers ---')
for key, value in r.headers.items():
    print(f'  {key}: {value}')

Status code : 200
OK?         : True
URL         : https://httpbin.org/get
Elapsed     : 0:00:00.074348

--- Response headers ---
  Date: Sun, 03 May 2026 13:31:06 GMT
  Content-Type: application/json
  Content-Length: 317
  Connection: keep-alive
  Server: gunicorn/19.9.0
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Credentials: true

In [4]:

import json

r = requests.get('https://httpbin.org/get')

# r.text is the raw response body as a string
print('r.text (first 200 chars):')
print(r.text[:200])
print()

# r.json() parses the JSON body into a Python dict
data = r.json()
print('r.json() type  :', type(data))
print('r.json() keys  :', list(data.keys()))

r.text (first 200 chars):
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate, br, zstd", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.32.5", 
    "X-Amzn-Trace-Id":

r.json() type  : <class 'dict'>
r.json() keys  : ['args', 'headers', 'origin', 'url']

4. Query Parameters¶

Query parameters are key-value pairs appended to a URL after a ?, separated by &:

https://example.com/search?q=python&page=2

You could build this string manually, but requests does it safely for you with the params keyword.

In [5]:

params = {
    'name': 'Alice',
    'lang': 'Python',
    'year': 2026,
}

r = requests.get('https://httpbin.org/get', params=params)

print('Actual URL:', r.url)  # requests built the query string for you
print()
print('Args echoed back by httpbin:')
print(json.dumps(r.json()['args'], indent=2))

Actual URL: https://httpbin.org/get?name=Alice&lang=Python&year=2026

Args echoed back by httpbin:
{
  "lang": "Python",
  "name": "Alice",
  "year": "2026"
}

5. Custom Headers¶

Pass a dictionary to the headers keyword argument to add or override request headers. A common reason is to identify your script with a User-Agent, or to tell the API you want JSON back.

In [6]:

custom_headers = {
    'User-Agent': 'intro-requests-notebook/1.0',
    'Accept': 'application/json',
    'X-Custom-Header': 'hello',
}

r = requests.get('https://httpbin.org/headers', headers=custom_headers)

print('Headers the server saw:')
print(json.dumps(r.json()['headers'], indent=2))

Headers the server saw:
{
  "Accept": "application/json",
  "Accept-Encoding": "gzip, deflate, br, zstd",
  "Host": "httpbin.org",
  "User-Agent": "intro-requests-notebook/1.0",
  "X-Amzn-Trace-Id": "Root=1-69f74e1a-5e5d346702c79b5d541eac28",
  "X-Custom-Header": "hello"
}

6. Downloading JSON¶

JSONPlaceholder is a free fake REST API. It returns realistic JSON data and is great for practising.

JSON (JavaScript Object Notation) is the most common data format for web APIs. Python's r.json() method converts the JSON text directly into a Python dict or list.

In [7]:

# Fetch a single post
r = requests.get('https://jsonplaceholder.typicode.com/posts/1')

post = r.json()
print('Type:', type(post))
print(json.dumps(post, indent=2))

Type: <class 'dict'>
{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

In [8]:

# Fetch all posts — the response is a list
r = requests.get('https://jsonplaceholder.typicode.com/posts')

posts = r.json()
print(f'Number of posts: {len(posts)}')
print('First post:')
print(json.dumps(posts[0], indent=2))

Number of posts: 100
First post:
{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}

In [9]:

# Access nested data — fetch a user
r = requests.get('https://jsonplaceholder.typicode.com/users/1')

user = r.json()
print('Name    :', user['name'])
print('Email   :', user['email'])
print('City    :', user['address']['city'])
print('Company :', user['company']['name'])

Name    : Leanne Graham
Email   : Sincere@april.biz
City    : Gwenborough
Company : Romaguera-Crona

In [10]:

# Use query params to filter — get posts by a specific user
r = requests.get(
    'https://jsonplaceholder.typicode.com/posts',
    params={'userId': 1}
)

user_posts = r.json()
print(f'Posts by user 1: {len(user_posts)}')
for p in user_posts[:3]:
    print(f'  [{p["id"]}] {p["title"]}')

Posts by user 1: 10
  [1] sunt aut facere repellat provident occaecati excepturi optio reprehenderit
  [2] qui est esse
  [3] ea molestias quasi exercitationem repellat qui ipsa sit aut

6.5 Anime Quotes — Animechan¶

Animechan is a free API that returns quotes from anime series. No authentication is required.

Rate limit: 5 requests/hour on the free tier.

In [11]:

# Fetch a single random quote
r = requests.get('https://api.animechan.io/v1/quotes/random')
r.raise_for_status()

quote = r.json()['data']
print(f'Anime    : {quote["anime"]["name"]}')
print(f'Character: {quote["character"]["name"]}')
print(f'Quote    : {quote["content"]}')

Anime    : Gintama
Character: Hijikata Toushirou
Quote    : *while eating Super Spicy Senbei* It's spicy... It's spicy. Damn it, why am I... It's so spicy I have tears in my eyes.

In [12]:

# Fetch 3 random quotes
for _ in range(3):
    r = requests.get('https://api.animechan.io/v1/quotes/random')
    r.raise_for_status()
    d = r.json()['data']
    print(f'[{d["anime"]["name"]}] {d["character"]["name"]}: "{d["content"][:80]}"')
    print()

[My Teen Romantic Comedy SNAFU] Hachiman Hikigaya: "Do you know the phrase “enough specks of dust creates a mountain”? Or maybe “thr"

[A Lull in the Sea] Chisaki Hiradaira: "Does becoming adult mean you lose a lot of things?"

[Fruits Basket] Saki Hanajima: "...If Tohru-chan were to die, I...I wonder if I'd be able to smile again one yea"

7. Downloading Images¶

picsum.photos returns random placeholder images. A URL like https://picsum.photos/400/300 returns a 400×300 JPEG.

Images are binary data, not text. Use r.content (bytes) instead of r.text (string). Trying to decode binary data as UTF-8 text will give you garbled output or an error.

In [13]:

r = requests.get('https://picsum.photos/400/300')

print('Status code :', r.status_code)
print('Content-Type:', r.headers.get('Content-Type'))
print('Size (bytes):', len(r.content))
print('Type of r.content:', type(r.content))

Status code : 200
Content-Type: image/jpeg
Size (bytes): 22150
Type of r.content: <class 'bytes'>

In [14]:

# Save the image to disk
with open('/tmp/sample_image.jpg', 'wb') as f:
    f.write(r.content)

print('Image saved to /tmp/sample_image.jpg')

Image saved to /tmp/sample_image.jpg

In [15]:

# Display the image inline in the notebook
from IPython.display import Image, display

display(Image(r.content))

No description has been provided for this image

Why r.content and not r.text?

Attribute	Type	Use for
`r.text`	`str`	Text responses: HTML, JSON, XML, CSV
`r.content`	`bytes`	Binary responses: images, PDFs, ZIP files
`r.json()`	`dict` / `list`	JSON responses (parses `r.text` automatically)

7.5 Cat Pictures — Cataas¶

Cataas (Cat as a Service) serves random cat photos. Append ?json=true to get a JSON object with the image URL instead of the raw bytes directly.

In [16]:

# Get metadata for a random cat image
r = requests.get('https://cataas.com/cat?json=true')
r.raise_for_status()

meta = r.json()
print('Cat ID  :', meta['id'])
print('Tags    :', meta.get('tags', []))
print('URL     :', meta['url'])

Cat ID  : RaAraGK7NRcy70HS
Tags    : []
URL     : https://cataas.com/cat/RaAraGK7NRcy70HS?position=center

In [17]:

# Download and display the cat image
img_url = meta['url']
r_img = requests.get(img_url)
r_img.raise_for_status()

from IPython.display import Image, display
display(Image(r_img.content))

In [18]:

# Cataas also lets you request cats with a text overlay
r = requests.get('https://cataas.com/cat/says/Hello%20World?json=true')
r.raise_for_status()
meta = r.json()

r_img = requests.get(meta['url'])
display(Image(r_img.content))

8. POST Requests — Sending Data¶

Use requests.post() to create a new resource. Pass your data as a Python dict to the json= keyword — requests will serialise it to JSON and set the Content-Type: application/json header automatically.

JSONPlaceholder accepts POST requests and returns what the new resource would look like (with a fake id).

In [19]:

new_post = {
    'title': 'My first post',
    'body': 'This is the content of the post.',
    'userId': 1,
}

r = requests.post(
    'https://jsonplaceholder.typicode.com/posts',
    json=new_post
)

print('Status code:', r.status_code)  # 201 Created
print('Response:')
print(json.dumps(r.json(), indent=2))

Status code: 201
Response:
{
  "title": "My first post",
  "body": "This is the content of the post.",
  "userId": 1,
  "id": 101
}

json= vs data=

Keyword	Sends	Content-Type
`json={'key': 'val'}`	JSON string	`application/json`
`data={'key': 'val'}`	Form-encoded string	`application/x-www-form-urlencoded`

Use json= for REST APIs. Use data= when submitting HTML forms.

9. PUT and DELETE Requests¶

PUT replaces an existing resource entirely. DELETE removes a resource. Both follow the same pattern as GET and POST.

Note: JSONPlaceholder simulates these operations — no real data is changed on the server.

In [20]:

# PUT — replace post 1 with new content
updated_post = {
    'id': 1,
    'title': 'Updated title',
    'body': 'Updated body.',
    'userId': 1,
}

r = requests.put(
    'https://jsonplaceholder.typicode.com/posts/1',
    json=updated_post
)

print('Status code:', r.status_code)  # 200 OK
print('Response:')
print(json.dumps(r.json(), indent=2))

Status code: 200
Response:
{
  "id": 1,
  "title": "Updated title",
  "body": "Updated body.",
  "userId": 1
}

In [21]:

# DELETE — remove post 1
r = requests.delete('https://jsonplaceholder.typicode.com/posts/1')

print('Status code:', r.status_code)  # 200 OK
print('Response body:', r.text)       # usually empty or {}

Status code: 200
Response body: {}

10. Error Handling¶

10.1 `raise_for_status()`¶

By default, requests does not raise an exception for error status codes like 404 or 500. Call r.raise_for_status() right after every request to automatically raise an HTTPError if the status code indicates failure.

In [22]:

# Request a URL that returns 404
r = requests.get('https://httpbin.org/status/404')

print('Status code:', r.status_code)  # 404 — no exception yet

try:
    r.raise_for_status()              # raises here
except requests.exceptions.HTTPError as e:
    print('HTTPError caught:', e)

Status code: 404
HTTPError caught: 404 Client Error: NOT FOUND for url: https://httpbin.org/status/404

10.2 Timeouts¶

By default requests waits forever for a response. Always pass a timeout (seconds) so your script doesn't hang if the server is slow or unreachable.

httpbin.org/delay/N artificially delays its response by N seconds — useful for testing timeouts.

In [23]:

try:
    # Server waits 10 s; our timeout is 3 s → Timeout exception
    r = requests.get('https://httpbin.org/delay/10', timeout=3)
except requests.exceptions.Timeout:
    print('Request timed out after 3 seconds.')

Request timed out after 3 seconds.

10.3 Full try / except Pattern¶

Combine raise_for_status() and exception handling into a reusable pattern:

In [24]:

url = 'https://jsonplaceholder.typicode.com/posts/1'

try:
    r = requests.get(url, timeout=10)
    r.raise_for_status()           # raise on 4xx / 5xx
    data = r.json()
    print('Success! Title:', data['title'])

except requests.exceptions.Timeout:
    print('The request timed out.')
except requests.exceptions.ConnectionError:
    print('Could not connect to the server.')
except requests.exceptions.HTTPError as e:
    print(f'HTTP error {r.status_code}: {e}')
except requests.exceptions.RequestException as e:
    # Catch-all for any other requests error
    print(f'Something went wrong: {e}')

Success! Title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit

11. Sessions¶

Every requests.get() call opens a new TCP connection to the server. A Session reuses the same connection for multiple requests to the same host — faster and more efficient.

Sessions also let you set headers, cookies, and authentication once and have them applied to every request automatically.

In [25]:

# Without a session — each call opens a new connection
r1 = requests.get('https://jsonplaceholder.typicode.com/posts/1')
r2 = requests.get('https://jsonplaceholder.typicode.com/posts/2')
print('Two separate connections made.')

Two separate connections made.

In [26]:

# With a session — one connection, shared headers
with requests.Session() as session:
    session.headers.update({
        'User-Agent': 'intro-requests-notebook/1.0',
        'Accept': 'application/json',
    })

    r1 = session.get('https://jsonplaceholder.typicode.com/posts/1')
    r2 = session.get('https://jsonplaceholder.typicode.com/posts/2')

print('Post 1 title:', r1.json()['title'])
print('Post 2 title:', r2.json()['title'])
print('Both requests used the same session headers.')

Post 1 title: sunt aut facere repellat provident occaecati excepturi optio reprehenderit
Post 2 title: qui est esse
Both requests used the same session headers.

When to use a Session:

You make more than one request to the same host.
You need to share headers, cookies, or authentication across requests.
You want the performance benefit of connection pooling.

The with statement ensures the session is closed cleanly when you're done.

12. What `requests` Can't Do¶

requests is excellent for straightforward HTTP calls, but it has limits:

Async / concurrent requests — requests is synchronous: it blocks your program until each response arrives. For async I/O or sending many requests in parallel, use httpx or aiohttp.
JavaScript-rendered pages — requests fetches the raw HTML the server sends. It cannot run JavaScript, so pages that load data dynamically (single-page apps) will appear empty. Use Selenium or Playwright to control a real browser.
WebSockets — requests only speaks HTTP request/response. It does not support persistent bidirectional WebSocket connections. Use the websockets library instead.

13. Where to Find More APIs: public-apis¶

public-apis is a community-maintained GitHub repository with over 430,000 stars — one of the most-starred repos on all of GitHub. It lists hundreds of free public APIs organised by category: Animals, Anime, Books, Finance, Games, Geocoding, Music, Science, Weather, and many more.

Each entry in the list tells you:

Column	Meaning
Auth	`No` = no key needed, `apiKey` = free key required, `OAuth` = OAuth flow
HTTPS	Whether the API supports HTTPS
CORS	Whether the API can be called from a browser

There is also a JSON API for the list itself at https://api.publicapis.org/entries.

In [27]:

# Query the public-apis directory — filter by category
# Note: api.publicapis.org is no longer active; fallback mock data illustrates the response shape
try:
    r = requests.get(
        'https://api.publicapis.org/entries',
        params={'category': 'Animals', 'https': 'true'},
        timeout=10
    )
    r.raise_for_status()
    data = r.json()
except (requests.exceptions.ConnectionError, requests.exceptions.HTTPError):
    data = {
        'count': 3,
        'entries': [
            {'API': 'AdoptAPet',  'Auth': 'apiKey', 'Description': 'Resource to help get pets adopted'},
            {'API': 'Axolotl',    'Auth': '',       'Description': 'Collection of axolotl pictures and facts'},
            {'API': 'Cat Facts',  'Auth': '',       'Description': 'Daily cat facts'},
        ]
    }

print(f"Total entries in 'Animals' category: {data['count']}")
print()
print('First 5 entries:')
for entry in data['entries'][:5]:
    auth = entry['Auth'] or 'None'
    print(f"  {entry['API']:<30} Auth={auth:<8} {entry['Description'][:50]}")

Total entries in 'Animals' category: 3

First 5 entries:
  AdoptAPet                      Auth=apiKey   Resource to help get pets adopted
  Axolotl                        Auth=None     Collection of axolotl pictures and facts
  Cat Facts                      Auth=None     Daily cat facts

In [28]:

# List all available categories
# Note: api.publicapis.org is no longer active; fallback mock data illustrates the response shape
try:
    r = requests.get('https://api.publicapis.org/categories', timeout=10)
    r.raise_for_status()
    categories = r.json()
except (requests.exceptions.ConnectionError, requests.exceptions.HTTPError):
    categories = [
        'Animals', 'Anime', 'Anti-Malware', 'Art & Design', 'Authentication & Authorization',
        'Blockchain', 'Books', 'Business', 'Calendar', 'Cloud Storage & File Sharing',
        'Continuous Integration', 'Cryptocurrency', 'Currency Exchange', 'Data Validation',
        'Development', 'Dictionaries', 'Documents & Productivity', 'Email', 'Entertainment',
        'Environment', 'Events', 'Finance', 'Food & Drink', 'Games & Comics',
        'Geocoding', 'Government', 'Health', 'Jobs', 'Machine Learning', 'Music',
        'News', 'Open Data', 'Open Source Projects', 'Patent', 'Personality',
        'Phone', 'Photography', 'Science & Math', 'Security', 'Shopping',
        'Social', 'Sports & Fitness', 'Test Data', 'Text Analysis', 'Tracking',
        'Transportation', 'URL Shorteners', 'Vehicle', 'Video', 'Weather'
    ]

print(f'Total categories: {len(categories)}')
for cat in sorted(categories):
    print(' ', cat)

Total categories: 50
  Animals
  Anime
  Anti-Malware
  Art & Design
  Authentication & Authorization
  Blockchain
  Books
  Business
  Calendar
  Cloud Storage & File Sharing
  Continuous Integration
  Cryptocurrency
  Currency Exchange
  Data Validation
  Development
  Dictionaries
  Documents & Productivity
  Email
  Entertainment
  Environment
  Events
  Finance
  Food & Drink
  Games & Comics
  Geocoding
  Government
  Health
  Jobs
  Machine Learning
  Music
  News
  Open Data
  Open Source Projects
  Patent
  Personality
  Phone
  Photography
  Science & Math
  Security
  Shopping
  Social
  Sports & Fitness
  Test Data
  Text Analysis
  Tracking
  Transportation
  URL Shorteners
  Vehicle
  Video
  Weather

Licensed under the GNU General Public License v2.0 (GPL-2).
You may redistribute and/or modify this work under the terms of GPL-2.
This work is distributed WITHOUT ANY WARRANTY.
Happy computing.