Pittsburgh Weather

One of my favorite things to do with GitHub Actions is let the workflow write back to its own repository. Not deploy, not publish, not send a notification — actually commit files and push them to main. The pittsburgh-weather repository is a clean example of that pattern applied to something practical: an hourly Pittsburgh weather feed that archives its own data.

What the repo does

The core idea is simple. A Python script calls the Open-Meteo API — no API key required — and retrieves hourly forecasts for Pittsburgh: temperature, precipitation, and wind speed over the next several days. It then:

Appends the new records to a monthly Parquet file (data/YYYYMM.parquet), deduplicating any rows already fetched today
Generates a dual-axis plot — temperature as a red line, precipitation as blue bars — and writes it as data/YYYYMMDD.png
Copies the latest plot to data/today.png so there’s always a stable link to the current visualization

None of that is remarkable on its own. What makes it interesting is what happens next.

The GitHub Actions loop

The workflow runs on a cron schedule — every hour. When the script finishes, the workflow stages the updated Parquet file and the new plot, commits them, and pushes back to main:

- name: Commit and push changes
  run: |
    git config user.name "github-actions[bot]"
    git config user.email "github-actions[bot]@users.noreply.github.com"
    git add data/
    git diff --cached --quiet || git commit -m "chore: update weather data [skip ci]"
    git push

The [skip ci] tag in the commit message is important: it tells GitHub Actions not to re-trigger the workflow on its own commits, which would create an infinite loop.

The result is a repository whose data/ directory grows automatically. Every hour, without any manual intervention, the archive gets one row richer and the plot gets refreshed. The repo itself becomes a living dataset — version-controlled, diffable, and auditable.

This pattern generalizes. Anywhere you would otherwise reach for a cron job on a server, a self-updating GitHub Actions repo can be a lighter-weight alternative — especially for small data feeds that don’t need a database.

Why Parquet and not JSON, TSV, or pkl

The script accumulates data over time, one fetch per hour, organized into monthly files. The choice of Parquet for that accumulation is deliberate.

JSON

JSON is the native format of the Open-Meteo API response, and it’s tempting to just write each response to a dated file and be done with it. The problem is that JSON is a poor format for tabular time-series data:

No schema enforcement. Every record is self-describing, which means field names and types are repeated in every row. A month of hourly data becomes verbose fast.
No efficient append. To add a new row to a JSON file you have to read it, deserialize it, modify it, and rewrite the whole thing.
Slow to query. Reading a JSON file into pandas means parsing every byte. There’s no column pruning, no predicate pushdown.

Storing raw API responses as JSON makes sense for an audit log or a cache. It’s not a good accumulation format.

TSV / CSV

TSV solves the verbosity problem — field names appear once in the header, rows are compact. But it introduces its own issues:

No type information. Every value is a string until your parser decides otherwise. Datetimes, floats, and integers all look the same on disk.
Fragile with special characters. Values that contain tabs, newlines, or quotes require careful escaping. Weather data is clean enough that this rarely bites, but it’s still a latent hazard.
Append is almost as awkward. You can echo a new row to the end of a CSV file, but deduplication and schema evolution require reading and rewriting the file.

TSV is fine for small, static exports meant for humans to open in a spreadsheet. For a growing archive queried by code, it’s the wrong tool.

Pickle (pkl)

Pickle serializes Python objects directly. That sounds convenient — round-trip a DataFrame with zero configuration — but the convenience is a trap:

Python-version and library-version dependent. A pickle written by pandas 1.x may not load correctly under pandas 2.x. A pickle written by Python 3.10 may fail under 3.12.
Not interoperable. Pickle is Python-only. You cannot open a pkl file in R, DuckDB, Spark, or any other tool without going through Python.
A security risk at boundaries. Pickle deserialization executes arbitrary code. That’s fine when you fully control the file, but it’s a habit worth avoiding.

Pickle is reasonable for short-lived caching within a single environment. It’s not appropriate for data you intend to keep, share, or query from multiple tools.

Parquet

Parquet was designed for exactly this use case: columnar storage of typed, structured, append-friendly tabular data.

Typed schema on disk. Timestamps are timestamps, floats are floats, integers are integers. No parsing ambiguity.
Columnar layout. Reading only the temperature column from a year of data reads only the temperature bytes from disk. Everything else is skipped at the storage layer.
Efficient compression. Columnar data compresses extremely well because repeated values in a column (like a fetch_date that’s the same for every row in a batch) can be represented with run-length encoding.
Interoperable. Parquet is a first-class format in pandas, Spark, DuckDB, Polars, Arrow, and most analytics tooling. The monthly files from this repo can be queried directly with DuckDB without loading them into Python first:

SELECT AVG(temperature_2m), date_trunc('day', timestamp)
FROM read_parquet('data/202504.parquet')
GROUP BY 2
ORDER BY 2;

Ecosystem support. GitHub doesn’t render Parquet files, but every data tool does. The format is stable, widely adopted, and not tied to any single language or library version.

The tradeoff is that Parquet is not human-readable — you can’t cat a Parquet file and make sense of it. For a dataset like this, that’s an acceptable cost. The data is meant to be queried, not inspected raw.

The bigger pattern

The pittsburgh-weather repo is small by design, but it demonstrates something worth internalizing: GitHub Actions is not just a CI/CD tool. It’s a scheduler with access to a git repository. Any periodic task that produces structured output — scraping a page, pulling an API, running a simulation — can be wired up this way.

The self-updating repo pattern is particularly useful when:

The dataset is small enough to fit in a git repository (rule of thumb: under a few hundred MB)
You want version history on the data itself, not just the code
You don’t want to operate a database or object storage bucket

For Pittsburgh weather at hourly resolution, a monthly Parquet file grows at roughly 50–100 KB per month. A year of data fits comfortably in a repo. Ten years of data probably still fits. At that point you’d want to reconsider — but that’s a good problem to have.

icaoberg / Pittsburgh Weather