Chapter 1 - Introduction

What is this book about?

This series of posts is about using Rust for data engineering tasks for people who are already familiar with Python and are curious about Rust. It will not cover every aspect of Rust, or of Python. Instead, it aims to give practical examples of how common engineering tasks done in Python might be done in Rust, along with representative benchmarks.

The current chapters cover:

• Getting data from an API
• Parsing data and using structs
• Transforming data with Polars

Concurrency is a work in progress. Writing data and web scraping are planned topics.

This book is not an introduction to either Rust or Python. There are many great resources to both out there. If you are not familiar with Python, the official Python Tutorial is a great starting point.

As for Rust, the Rust Book is a great introduction to the language and a must read. I can also recommend Rust in Action as well.

In particular, I think it’s important to understand some of the core principles behind static and dynamic typing, as well as memory safety and ownership. The borrow-checker in Rust is well-known as a steep hurdle to climb, but once you manage to understand it, you start writing better code. Don’t be discouraged, it takes time and I am still on the learning journey with you.

Should I use Rust for Data Engineering?

Probably not. Rust is a great language, it is fun, it is pleasant to use, and it is fast. But choosing a language for a project is more than choosing a language that is fun. There are cautionary tales about using Rust at a startup, and I think they are worth reading.

There are many reasons why you might not want to use Rust for data engineering. Start with the libraries and integrations your project needs: check whether they support the features you use, how they are maintained, and what support is available. An earlier version of this chapter said there were no Rust libraries for querying Snowflake; that is no longer true, as projects such as snowflake-connector-rs demonstrate. If your team already knows Python, learning Rust also adds time to development and onboarding.

There may be good reasons to use Rust for data engineering, however. Rust can reduce runtime and memory use for some workloads, particularly when replacing work done in Python loops. But a Python program may already do most of its work inside a native library such as Polars, or spend most of its time waiting on a network. Changing the language alone does not guarantee an improvement.

The examples in this book compare particular implementations on particular workloads. We will look at what each benchmark includes, whether the programs do equivalent work, and where the time goes. Treat the results as measurements to investigate and reproduce, rather than promises about your own pipeline.

I can’t tell you when to use Rust and when to use Python, but I do believe that by understanding both languages, their merits and pitfalls, you will be better positioned to make that decision for yourself.

Why Should I Learn Rust?

Because it is fun to learn new things. I can’t promise you that anything you learn here will ever have a material impact on your life or career. But if you enjoy learning and tinkering, then you might want to tinker with this. If you are like me, and you like learning for learning’s sake, then you will enjoy this experience too. I learned vim and lua not because it was useful, but because I was curious about it. I did end up benefiting from it, but I never approached it from a purely utilitarian perspective. There are better ways to spend your time if your goal is purely career advancement.

But, if you are curious about Rust, and if you like to have fun, then I think you will be pleasantly surprised by what Rust has to offer.

Prerequisites

Installing Rust and Python

You will need Rust and Python installed to follow along with the examples here.

For Rust, go to rustup.rs. This gives you rustc and cargo. Follow the platform-specific instructions there for a linker and other build prerequisites, too.

For Python, I recommend uv. It replaces the pile of tools I used to reach for — pyenv for interpreters, virtualenv for environments, pip for packages — with a single one, and it will install the right Python for you if you don’t have it.

# macOS / Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

Open a new terminal after installation if the commands are not found. Check that the tools are available:

rustc --version
cargo --version
uv --version

For Windows installation instructions, see the uv installation guide.

A small practice project looks like this. Create it outside the book’s repository; the book’s sample projects are already set up.

# Create a project
uv init --no-package --python 3.13 somepyproj
cd somepyproj

# Pin the Python version for this project
uv python pin 3.13

# Add a dependency
uv add polars

# Run the generated script inside the project environment
uv run python main.py

You should see Hello from somepyproj!. The --no-package flag explicitly selects a simple script layout with main.py; it avoids relying on uv’s default project layout. Choosing Python 3.13 at creation also keeps the project’s Python requirement compatible with the version we pin afterward.

uv creates the virtual environment when a command such as uv add or uv run needs it, so there is no separate “activate the venv” step. uv add writes the dependency into pyproject.toml and records the exact resolved version in uv.lock, which is the Python equivalent of Rust’s Cargo.lock.

This practice project resolves the currently available Polars release. To use the book’s dependency versions, use the checked-in projects below instead.

A note on versions

Both sample projects pin their direct dependencies exactly and commit their lockfiles to record resolved dependency versions. The versions listed for the examples are:

Both of these ecosystems move quickly, and polars in particular has changed its API substantially over the years. Use the committed lockfiles rather than recreating the dependencies with uv add or cargo add.

The Python project selects the 3.13 series in .python-version, rather than an exact patch release. The repository does not pin the Rust compiler with a rust-toolchain.toml file. Lockfiles do not pin your compiler, operating system, hardware, or build flags, and do not guarantee identical benchmark times. Record those details when comparing results.

Installing the Code

The code for this book is on GitHub. You will need Git installed. From the directory where you keep your projects:

git clone https://github.com/PedramNavid/rust-for-data.git
cd rust-for-data

# Install the Python dependencies from the committed lockfile
uv sync --locked --project wxpy

# Check that the Rust examples compile, using the committed lockfile
cargo check --locked --manifest-path wxrs/Cargo.toml --bins

The Rust examples live in wxrs, and the Python examples live in wxpy. The first Rust check can take a while because it compiles dependencies. cargo check checks the code without producing runnable binaries; later chapters cover running examples and building release binaries for benchmarks.

--locked makes these commands fail if the lockfile needs updating, rather than silently changing the dependency resolution. Keep the lockfile; any dependency upgrade needs the examples and benchmarks to be checked again.

The repository’s convenience commands also require make. From the repo root, make setup performs the same locked Python installation. You do not need an API key or the extracted bird dataset for these setup checks; those requirements are introduced with the examples that use them.

Fetching from an API

One of the simplest examples to start with is fetching data from an API endpoint. This is often the beginning of many data pipeline journeys.

We will use the OpenWeatherMap Air Pollution API to fetch the current air pollution for a configurable location by providing a latitude and longitude on the command line. Given that I’m in California, air pollution felt like the natural place to start.

You will need to sign up for a free account to get an API key. Once you’ve signed up, create an API key. The free tier allows a fixed number of requests per day; check the current pricing page for the limits before you start running benchmarks in a loop.

The chapter walks through the same small program in both languages: read the arguments, make the request, print the response, fail when something goes wrong, and finally measure both.

Where the Code Lives

If you followed the Prerequisites chapter, you already have the code checked out and both projects set up. The Rust project is wxrs, and the Python project is wxpy. The code for this chapter lives at:

wxrs/src/bin/ch3.rs
wxpy/wxpy/ch3/fetch_api.py

In Rust, a project usually has a src/main.rs file that runs the program, with additional code imported as modules from other files. There is a good convention for package layouts in Rust. Since we want one runnable program per chapter, this project has no main.rs at all. Every file in src/bin is compiled into its own binary, so ch3.rs becomes a binary named ch3.

In Python, wxpy/wxpy is a package, and each chapter is a sub-package inside it. That lets us run each chapter’s file as a module with python -m.

Starting a Project from Scratch

You do not need to run anything in this section, but it is worth seeing what creating these projects looks like, because it is one of the first differences between Rust and Python you will experience.

In Rust, this is as simple as running

# Create the project
cargo init wxrs

# Add a dependency
cd wxrs
cargo add reqwest --features blocking

This creates a new directory called wxrs with a Hello World example.

It also adds the reqwest crate to our dependencies, similar to pip install. Unlike a bare pip install though, this will also update Cargo.toml with our dependency, and create a Cargo.lock file that pins the reqwest crate to a specific version.

The --features flag is used to express optional compilation features. Reqwest has several options, described in the crate’s documentation.

We will use the blocking feature, which gives us a simpler interface to reqwest instead of futures that require an async runtime. We will eventually use async to show the power of Rust’s fearless concurrency.

Python too allows optional features, for example pip install snowflake-connector-python[pandas].

This used to be the part where I explained that Python makes you do all of this by hand: create the directory, create a virtual environment, hand-write a pyproject.toml, name your dependencies, then install the package locally. It was a genuinely unflattering comparison.

uv has since closed most of that gap, and the Python side now looks a lot like the Rust side:

# Create the project as an importable package
uv init --lib wxpy
cd wxpy

# Add a dependency
uv add requests

Like cargo add, uv add writes the dependency into pyproject.toml and records the exact resolved version in a lockfile: uv.lock here, Cargo.lock there. The --lib flag asks for a package layout rather than a single script, which is what lets us organise chapters as sub-packages. The exact layout uv generates has changed between versions, which is one more reason this book ships the projects ready to go rather than asking you to recreate them.

Now, admittedly we can skip all of the above steps, create a random file anywhere we want and run it with python myfile.py, but the goal here is to build a more stable distribution that can be packaged, shared, and tested.

It is worth being honest about what this does to the comparison. For years the Rust story here was simply better, and that was a real argument in Rust’s favour. It isn’t much of one anymore.

The two manifests in the repository have grown a few dependencies for later chapters, but the shape is the same:

# Cargo.toml
[package]
name = "wxrs"
version = "0.1.0"
edition = "2021"

# Direct dependencies are pinned exactly; Cargo.lock records the resolved graph.
# Benchmark results also depend on the toolchain, build flags, and machine.
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
polars = { version = "=0.55.2", features = ["lazy", "csv", "strings"] }
reqwest = { version = "=0.13.4", features = ["blocking", "json"] }
serde = { version = "=1.0.229", features = ["derive"] }
serde_json = "=1.0.151"

# pyproject.toml
[project]
name = "wxpy"
version = "0.0.1"
requires-python = ">=3.11"

# Direct dependencies are pinned exactly; uv.lock records the resolved graph.
# Benchmark results also depend on the interpreter, native builds, and machine.
dependencies = [
    "requests==2.34.2",
    "polars==1.44.1",
    "pandas==3.0.5",
]

[dependency-groups]
data = ["py7zr==1.1.3"]

Fetching Air Pollution Data

To fetch from an API, we will use the requests package in Python and the reqwest crate in Rust.

Both programs read the API key from the OWM_APPID environment variable, and take the latitude and longitude from the command line arguments.

Python

# wxpy/wxpy/ch3/fetch_api.py
import os
import sys

import requests

API_KEY = os.getenv("OWM_APPID")
URL = "https://api.openweathermap.org/data/2.5/air_pollution"


def get_air_pollution(lat, lon):
    params = {"lat": lat, "lon": lon, "appid": API_KEY}
    response = requests.get(URL, params=params, timeout=10)
    response.raise_for_status()
    return response.text


if __name__ == "__main__":
    usage = f"Usage: python {__file__} <lat> <lon>"

    if not API_KEY:
        print("Please set OWM_APPID environment variable")
        sys.exit(1)

    if len(sys.argv) != 3:
        print(usage)
        sys.exit(1)

    try:
        lat = float(sys.argv[1])
        lon = float(sys.argv[2])
    except ValueError:
        print(usage)
        sys.exit(1)

    try:
        body = get_air_pollution(lat, lon)
    except requests.RequestException as err:
        print(f"Request failed: {err}")
        sys.exit(1)

    print(body)

Rust

// wxrs/src/bin/ch3.rs
use std::time::Duration;

pub fn get_air_pollution(lat: f32, lon: f32, api_key: &str) -> Result<String, reqwest::Error> {
    let client = reqwest::blocking::Client::builder()
        .timeout(Duration::from_secs(10))
        .build()?;

    let url = format!(
        "https://api.openweathermap.org/data/2.5/air_pollution?lat={}&lon={}&appid={}",
        lat, lon, api_key
    );

    let body = client.get(url).send()?.error_for_status()?.text()?;

    Ok(body)
}

pub fn main() {
    let usage = format!("Usage: {} [lat] [lon]", std::env::args().next().unwrap());

    let api_key = std::env::var("OWM_APPID").expect(
        "Environment Variable OWM_APPID not set. Please set it to your
    OpenWeatherMap API key. https://home.openweathermap.org/api_keys",
    );

    let lat = std::env::args()
        .nth(1)
        .expect(&usage)
        .parse::<f32>()
        .expect(&usage);

    let lon = std::env::args()
        .nth(2)
        .expect(&usage)
        .parse::<f32>()
        .expect(&usage);

    match get_air_pollution(lat, lon, &api_key) {
        Ok(body) => println!("{}", body),
        Err(err) => {
            eprintln!("Request failed: {}", err);
            std::process::exit(1);
        }
    }
}

A few choices are shared by both programs, and they matter more than they look:

• The URL uses https. The API key is part of the query string, and over plain http it would travel across the network in clear text.
• Both requests have an explicit ten second timeout. Requests has no timeout by default and will wait forever on a silent server. The blocking reqwest client defaults to thirty seconds. Setting it explicitly makes the two programs behave the same way.
• Both programs check the HTTP status before trusting the body. OpenWeatherMap returns a JSON error document with a 401 or 400 status when something is wrong, and without the check that error document would be printed as if it were a result.

Setting the API Key

Both programs read the key from the environment. For a single session you can export it directly:

export OWM_APPID=your-api-key

The repository also ignores a .env file at its root, so you can keep the key there instead of in your shell history:

# .env, in the repository root
OWM_APPID=your-api-key

A .env file is just a list of assignments. Neither program reads it directly; you load it into your shell before running anything. set -a marks every variable defined afterwards for export, and set +a turns that back off:

set -a
. ./.env
set +a

The benchmark instructions later in the chapter assume you have done one of these two things.

Running the Program

Running the program is simple in both languages. We’ll provide the latitude and longitude of beautiful Fairfax, CA, birthplace of mountain biking, and nestled in the foothills of Mount Tamalpais.

Google gives the coordinates as 37.9871 and -122.5889

Python

In Python, we can use -m to run the module directly. uv run takes care of creating and using the project environment.

# in wxpy/
uv run python -m wxpy.ch3.fetch_api 37.9871 -122.5889

> {"coord":{"lon":-122.5889,"lat":37.9871},"list":[{"main":{"aqi":2},"components":{"co":181.29,"no":0.03,"no2":0.48,"o3":87.65,"so2":0.61,"pm2_5":2.63,"pm10":8.53,"nh3":0},"dt":1788747595}]}

Rust

In Rust, we must first compile the program before running it. cargo build builds a debug version of every binary in src/bin and places it under ./target/debug, so our ch3.rs becomes ./target/debug/ch3.

We can also compile and run in one step with cargo run, naming the binary we want.

# in wxrs/
cargo build
./target/debug/ch3 37.9871 -122.5889
> {"coord":{"lon":-122.5889,"lat":37.9871},"list":[{"main":{"aqi":2},"components":{"co":181.29,"no":0.03,"no2":0.48,"o3":87.65,"so2":0.61,"pm2_5":2.63,"pm10":8.53,"nh3":0},"dt":1788747595}]}

# or
cargo run --bin ch3 37.9871 -122.5889
> {"coord":{"lon":-122.5889,"lat":37.9871},"list":[{"main":{"aqi":2},"components":{"co":181.29,"no":0.03,"no2":0.48,"o3":87.65,"so2":0.61,"pm2_5":2.63,"pm10":8.53,"nh3":0},"dt":1788747595}]}

Discussion

Looking at both programs, we can see a fairly similar approach to solving this problem.

Both programs use an external library or crate (not-so-coincidentally named requests/reqwest).

In both programs, we’ve created a function that takes a latitude and longitude, fetches the results from an API and returns the results as text. We’ll cover handling structured data from JSON soon.

Types

One obvious difference is that in Rust, we declare the types of the lat and lon arguments, and in Python we do not.

#![allow(unused)]
fn main() {
pub fn get_air_pollution(lat: f32, lon: f32, api_key: &str) -> Result<String, reqwest::Error> {
    let client = reqwest::blocking::Client::builder()
        .timeout(Duration::from_secs(10))
        .build()?;

    let url = format!(
        "https://api.openweathermap.org/data/2.5/air_pollution?lat={}&lon={}&appid={}",
        lat, lon, api_key
    );

    let body = client.get(url).send()?.error_for_status()?.text()?;

    Ok(body)
}
}

Both lat and lon are f32, 32-bit floating-point numbers. That is plenty of precision for a coordinate. The Python function has no such declaration; it will accept whatever it is given, and the url line will happily format a string, a number, or None into the query string.

It is easy to read too much into this. Three separate things are going on when a program accepts a coordinate from the command line, and the languages only differ on one of them.

Parsing turns the text "37.9871" into a number. Command line arguments are always strings, in both languages, so this is a step somebody has to take. Rust’s parse::<f32>() and Python’s float() do the same job and both reject "nice". An earlier version of the Python program skipped this step and sent the strings straight to the API, which then rejected them with an HTTP 400. That was a choice made in the program, not a limitation of Python.

Type checking asks whether the program is consistent about what it passes around. In Rust, once lat is an f32 it stays one, and calling get_air_pollution with a string is a compile error. Python checks types at runtime and only on the operations that care, so a wrong type surfaces when something finally chokes on it, or never, if nothing does. mypy and similar tools bring static checking to Python as an optional layer on top; they change what gets caught before you run the program, not how the program runs.

Domain validation asks whether the value makes sense. A latitude of 91 parses perfectly well as an f32 and as a float, and neither of our programs rejects it. Both send it to the API, which answers with a 400. Rust’s type system does not do this for you either. You could define a Latitude type whose constructor refuses values outside -90 to 90, and later chapters will lean on that pattern, but it is work you do on top of the language, in either language.

So the honest summary is: Rust makes you parse, because there is no other way to get an f32, and then holds you to that type for the rest of the program. Python lets you defer parsing indefinitely, and it is on you to remember to do it.

Memory

The type declaration has one more consequence worth a paragraph. An f32 is four bytes, and the compiler knows that when it builds the program, so it can lay out lat and lon without any bookkeeping at runtime. A Python float is an object: the value plus a header holding a reference count and a type pointer, around 24 bytes in total, allocated on the heap and freed when nothing refers to it anymore. For two coordinates the difference is not worth noticing. It becomes interesting once you have millions of them, and we will return to memory properly when we look at ownership in the next chapter.

Handling Errors

Another subtle but important difference is the handling of errors.

In Python, errors are exceptions. A function that can fail raises, and the caller catches. Requests documents its exceptions well: everything it raises inherits from requests.RequestException, and raise_for_status raises an HTTPError for a 4xx or 5xx response. Our program catches that one base class and exits. What the language does not give you is any hint, at the call site, that requests.get can raise at all. You know because you read the documentation, or because it raised on you once.

In Rust, errors are values. get_air_pollution returns Result<String, reqwest::Error>, which says in the signature that the function can fail and exactly how. Inside the function, each ? returns the error to the caller as soon as one appears, and error_for_status turns a bad HTTP status into that same error type. The caller has to decide what to do with the Result; Result is marked #[must_use], so quietly ignoring one is a compiler warning rather than something you silently overlook.

#![allow(unused)]
fn main() {
    match get_air_pollution(lat, lon, &api_key) {
        Ok(body) => println!("{}", body),
        Err(err) => {
            eprintln!("Request failed: {}", err);
            std::process::exit(1);
        }
    }
}

The two programs behave the same way with a bad key:

OWM_APPID=bad uv run python -m wxpy.ch3.fetch_api 37.9871 -122.5889
> Request failed: 401 Client Error: Unauthorized for url: https://api.openweathermap.org/data/2.5/air_pollution?lat=37.9871&lon=-122.5889&appid=bad

OWM_APPID=bad ./target/debug/ch3 37.9871 -122.5889
> Request failed: HTTP status client error (401 Unauthorized) for url (https://api.openweathermap.org/data/2.5/air_pollution?lat=37.9871&lon=-122.5889&appid=bad)

Both exit with a non-zero status, which is what lets the benchmark harness later in the chapter trust that a timed run actually succeeded.

Not every failure in the Rust program is a Result, though. The argument parsing in main uses a different tool:

#![allow(unused)]
fn main() {
    let lat = std::env::args()
        .nth(1)
        .expect(&usage)
        .parse::<f32>()
        .expect(&usage);
}

expect says that if the value is missing or parse fails, the program should panic with the usage message. That is what happens with bad input:

./target/debug/ch3 nice birds

> thread 'main' panicked at src/bin/ch3.rs:37:10:
Usage: ./target/debug/ch3 [lat] [lon]: ParseFloatError { kind: Invalid }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The Python program does the equivalent with a try/except ValueError around float() and prints the usage line without the backtrace. You will see expect and its cousin unwrap used frequently in Rust. They are fine at the edge of a program where the only sensible response is to quit, and useful while debugging, but they are not error handling. Compare the two halves of main: the fetch returns a Result and lets the caller choose, while the parsing decides for the caller. We’ll cover error handling in more detail soon.

Benchmarks

Let me preface this by saying speed isn’t everything. No doubt someone familiar in Python will spend far more time learning Rust than they might ever save by running a slightly more optimized program. But it is nice to get a sense of the difference, and to watch how it changes as the programs get less trivial.

Let’s use hyperfine to benchmark the two programs. We’ll run each program 10 times after five warmups and take the average. Before we benchmark the Rust application, we’ll compile it using --release which builds a release rather than a debug version and it should provide us with a faster application.

cargo build --release

Every benchmark in this book is generated by the Makefile in benchmarks/, so you can reproduce them yourself:

# from the repo root, with OWM_APPID exported
make build-release
make -C benchmarks online 'BENCHMARK_CMD=hyperfine --warmup 5 --runs 10'

Under the hood that is just hyperfine comparing the two binaries:

hyperfine --warmup 5 --runs 10 \
    '../wxrs/target/release/ch3 30 -140' \
    '../wxpy/.venv/bin/python ../wxpy/wxpy/ch3/fetch_api.py 30 -140' \
    --export-markdown ch3_fetch_api.md

These results were generated against the live API after switching both programs to HTTPS and adding the status check, using five warmups and ten measured runs per implementation. Because both programs now exit non-zero on an HTTP error, hyperfine aborts the benchmark if any run fails, so every timed run below was a successful request.

On this run, Python averaged 171.0ms and Rust 152.5ms, making Rust about 1.12x faster for this complete command. Run-to-run standard deviations were 10.7ms and 27.7ms respectively, and the Rust run was the noisier of the two. These timings include startup, the TLS handshake, the HTTP request, and printing the response.

That is a much smaller gap than the earlier plain http version of this benchmark showed, and the reason is instructive. User CPU time was about 56.8ms for Python and 31.7ms for Rust. Over plain http the Rust program used about 6ms of CPU; the rest is the cost of setting up a TLS connection, which reqwest does with the pure-Rust rustls library in this build, while Python uses the system OpenSSL through its ssl module. Python’s interpreter startup and imports are still a plausible contributor to the remaining difference, but this benchmark does not isolate them from the other work each program does.

Both programs also wait on the network, and requests happen sequentially against a live service. Latency variation affects the result, and with a gap this small a different network day could plausibly reorder the two. A long-lived process that reuses a client would be a different workload, and one where the handshake cost is paid once rather than every run. No peak-memory measurements were taken in this run.

Again, this is a trivial application with trivial requirements and performance is not a key factor in deciding what language to build. But as we build more intensive applications we’ll keep an eye on memory and performance to see how the gap changes.

Summary

In this chapter we’ve built a simple application that fetches data from an API and returns the results. We’ve seen how Rust and Python differ in their approach to types and to handling errors, made both programs fail loudly and safely on bad input or a bad response, and measured the runtime of both complete programs against the live API.

Serializing Data

In the last chapter we fetched data from the OpenWeather API in order to get Air Pollution data. The astute observer will have noticed that we parsed the response as pure text, although the response was in JSON format.

The goal of this chapter is to walk through how we would take raw data and serialize it into a structured data format, such as JSON.

We’ll dive into theory in a little but let’s start with practice.

Serialization

Serialization is the process of taking data and encoding it into a known format that can later be retrieved. There are many ways to encode data, but largely these are broken into human-readable and binary formats.

CSVs, JSON, XML, and YAML are all human-readable serialization formats. Conversely, many binary formats exist, such as Parquet, Avro, and Protocol Buffers. Binary formats trade reduced readability for improved performance and size.

In the end, any data that needs to be persisted outside of a computer’s memory requires some type of serialization.

Let’s look at how serialization varies across both Rust and Python.

Python

In Python, we can serialize nearly any arbitrary data structure to JSON using the json module.

In [1]: import json

In [2]: my_obj = [{'a': 1, 'b': None}, "foo", "bar", ("baz", "baz")]

In [3]: json.dumps(my_obj)
Out[3]: '[{"a": 1, "b": null}, "foo", "bar", ["baz", "baz"]]'

Here’s the updated project code that serializes the response from the OpenWeather API.

import os
import sys

import requests

API_KEY = os.getenv("OWM_APPID")


def get_air_pollution(lat, lon):
    url = f"https://api.openweathermap.org/data/2.5/air_pollution?lat={lat}&lon={lon}&appid={API_KEY}"
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    return response.json()


def parse_air_pollution(body):
    aqi = body["list"][0]["main"]["aqi"]
    components = body["list"][0]["components"]
    return (aqi, components)



if __name__ == "__main__":
    usage = f"Usage: python {__file__} <lat> <lon>"

    if not API_KEY:
        print("Please set OWM_APPID environment variable")
        sys.exit(1)

    if len(sys.argv) != 3:
        print(usage)
        sys.exit(1)

    lat = sys.argv[1]
    lon = sys.argv[2]
    body = get_air_pollution(lat, lon)
    aqi, components = parse_air_pollution(body)

    print(f"Air Quality Index: {aqi}")
    print("Components:")
    for k, v in components.items():
        print(f"  {k}: {v}")

There are a few key things to note here.

First, we’re assuming the request was successful, that there is a JSON response body, and that it can parse correctly. If any of these assumptions are incorrect an exception will be raised, and we have no obvious way of knowing what these exceptions are or which method might raise one.

def parse_air_pollution(body):
    aqi = body["list"][0]["main"]["aqi"]
    components = body["list"][0]["components"]
    return (aqi, components)

When parsing the response, we slice into the response body to get various components. We’re explicitly fetching keys from a dictionary under the assumption that the payload is properly formed. There are safer dictionary methods to use, such as .get() which will return None if the key is missing rather than an exception, but in our case an Exception is warranted since we can’t do anything with the data if it’s missing.

We also haven’t explicitly typed the response from the API. This is something we can do with mypy or other tools like pydantic, but the Python interpreter itself has no type-guarantees.

Let’s look at how we might do this in Rust.

Rust

In Rust, we’ll need to install the serde crate as well as the json feature for reqwest.

cargo add serde --features derive
cargo add serde_json
cargo add reqwest --features json

Because Rust is a typed language, we will define the struct that represents the data we expect. The API response looks like the following:

{
    "coord": {
        "lon": -122.5889,
        "lat": 37.9871
    },
    "list": [
        {
            "main": {
                "aqi": 2
            },
            "components": {
                "co": 168.56,
                "no": 0.14,
                "no2": 0.75,
                "o3": 80.11,
                "so2": 0.7,
                "pm2_5": 3.48,
                "pm10": 5.58,
                "nh3": 0
            },
            "dt": 1687308878
        }
    ]
}

We can define a struct that represents this data as follows:

#![allow(unused)]
fn main() {
use serde::{Deserialize, Serialize};
#[derive(Debug, Serialize, Deserialize)]
pub struct AirPollution {
    pub coord: Coord,
    pub list: Vec<List>,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Coord {
    pub lon: f32,
    pub lat: f32,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct List {
    pub main: Main,
    pub components: Components,
    pub dt: usize,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Main {
    pub aqi: u8,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Components {
    pub co: f32,
    pub no: f32,
    pub no2: f32,
    pub o3: f32,
    pub so2: f32,
    pub pm2_5: f32,
    pub pm10: f32,
    pub nh3: f32,
}
}

As you can see, the struct mirrors the underlying JSON structure. The serde crate gives us a lot of flexibility here, in particular the section on Attributes and the Examples are worth spending some time on.

The reqwest crate also provides a json method that will automatically deserialize the response body into a struct.

#![allow(unused)]
fn main() {
pub fn get_air_pollution(lat: f32, lon: f32) -> AirPollution {
    let api_key = std::env::var("OWM_APPID").expect(
        "Environment Variable OWM_APPID not set. Please set it to your
    OpenWeatherMap API key. https://home.openweathermap.org/api_keys",
    );

    let url = format!(
        "https://api.openweathermap.org/data/2.5/air_pollution?lat={}&lon={}&appid={}",
        lat, lon, api_key
    );

    let client = reqwest::blocking::Client::builder()
        .timeout(std::time::Duration::from_secs(10))
        .build()
        .expect("client failed");

    client
        .get(url)
        .send()
        .expect("request failed")
        .error_for_status()
        .expect("server returned an error")
        .json()
        .expect("json failed")
}

Our function now returns an AirPollution struct, instead of a String, and reqwest’s json method will automatically deserialize the response body to the correct type.

Rust uses type inference to reduce the amount of syntax required. While function parameters and signatures always require types, local variables can usually be inferred by the compiler.

Let’s look at how returning a typed Struct changes how we interact with the data

#![allow(unused)]
fn main() {
pub fn parse_air_pollution(body: &AirPollution) -> (&Main, &Components) {
    let main = &body.list[0].main;
    let components = &body.list[0].components;
    (main, components)
}
}

We can access the underlying fields in the struct directly. Unlike a Python dictionary, the compiler will ensure that the fields we’re accessing exist.

If we add a missing field, for example:

#![allow(unused)]
fn main() {
let foo = &body.list[0].foo;
}

And run cargo check we’ll get the following error:

error[E0609]: no field `foo` on type `List`
  --> src/bin/ch4.rs:65:29
   |
65 |     let foo = &body.list[0].foo;
   |                             ^^^ unknown field
   |
   = note: available fields are: `main`, `components`, `dt`

For more information about this error, try `rustc --explain E0609`.
error: could not compile `wxrs` (bin "ch4") due to previous error

Compare to Python where we’d only get a run-time error if we tried to access a missing field, unless we opt-in to type hints using mypy.

Here’s the full Rust code for reference

use serde::{Deserialize, Serialize};
#[derive(Debug, Serialize, Deserialize)]
pub struct AirPollution {
    pub coord: Coord,
    pub list: Vec<List>,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Coord {
    pub lon: f32,
    pub lat: f32,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct List {
    pub main: Main,
    pub components: Components,
    pub dt: usize,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Main {
    pub aqi: u8,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Components {
    pub co: f32,
    pub no: f32,
    pub no2: f32,
    pub o3: f32,
    pub so2: f32,
    pub pm2_5: f32,
    pub pm10: f32,
    pub nh3: f32,
}

pub fn get_air_pollution(lat: f32, lon: f32) -> AirPollution {
    let api_key = std::env::var("OWM_APPID").expect(
        "Environment Variable OWM_APPID not set. Please set it to your
    OpenWeatherMap API key. https://home.openweathermap.org/api_keys",
    );

    let url = format!(
        "https://api.openweathermap.org/data/2.5/air_pollution?lat={}&lon={}&appid={}",
        lat, lon, api_key
    );

    let client = reqwest::blocking::Client::builder()
        .timeout(std::time::Duration::from_secs(10))
        .build()
        .expect("client failed");

    client
        .get(url)
        .send()
        .expect("request failed")
        .error_for_status()
        .expect("server returned an error")
        .json()
        .expect("json failed")
}

pub fn parse_air_pollution(body: &AirPollution) -> (&Main, &Components) {
    let main = &body.list[0].main;
    let components = &body.list[0].components;
    (main, components)
}

pub fn main() {
    let usage = format!("Usage: {} [lat] [lon]", std::env::args().next().unwrap());

    let lat = std::env::args()
        .nth(1)
        .expect(&usage)
        .parse::<f32>()
        .expect(&usage);

    let lon = std::env::args()
        .nth(2)
        .expect(&usage)
        .parse::<f32>()
        .expect(&usage);

    let body = get_air_pollution(lat, lon);
    let (main, components) = parse_air_pollution(&body);

    println!("Air Quality Index: {}", main.aqi);

    println!("Carbon Monoxide: {} μg/m³", components.co);
    println!("Nitrogen Monoxide: {} μg/m³", components.no);
    println!("Nitrogen Dioxide: {} μg/m³", components.no2);
    println!("Ozone: {} μg/m³", components.o3);
    println!("Sulfur Dioxide: {} μg/m³", components.so2);
    println!("Particulate Matter < 2.5 μm: {} μg/m³", components.pm2_5);
    println!("Particulate Matter < 10 μm: {} μg/m³", components.pm10);
    println!("Ammonia: {} μg/m³", components.nh3);
}

Serialization Formats

Something worth mentioning about the Rust serde crate is that it does not come with any built-in serialization formats. Instead, it provides a framework for serialization. We installed serde_json but there are many other formats available, such as serde_yaml and serde_avro.

Why Bother?

You might be wondering why we’d go through the trouble of defining a struct and serializing the response body into a struct. In Python, we avoid the boilerplate, we access fields directly, we can throw a little type-hinting at our code, we get to use # type: ignore freely, and if our application crashes, well, we’ll just fix it and run it again.

You are absolutely right! This is all true. However, any seasoned Python programmer is also aware of all the ways that poorly typed code can go wrong.

If you’ve ever created a compute-intensive application that operates on many gigabytes of data, you’ve probably run into a situation where you’ve had to re-run the application because it crashed. Type-safety helps prevent these types of issues, but types also provide another nice benefit: improved performance.

The compiler can optimize code based on the types it knows about. In Python, we can use type-hints to help the compiler, but ultimately the Python interpreter is still dynamically resolving types at runtime. In Rust, the compiler knows the types at compile-time and can optimize prior to running.

What’s that little & doing?

Ah, yes, the &. Now we are getting into the heart of Rust. Let’s look at the code for parsing air pollution again:

#![allow(unused)]
fn main() {
pub fn parse_air_pollution(body: &AirPollution) -> (&Main, &Components) {
    let main = &body.list[0].main;
    let components = &body.list[0].components;
    (main, components)
}
}

parse_air_pollution is a function that takes a reference to an AirPollution struct. The & is the syntax for creating a reference. In Rust, references are a way of passing a value to a function without transferring ownership of the value. This is a key concept in Rust, and it’s what allows Rust to guarantee memory safety.

In Python, values are passed around as references and tracked with counters. Every object carries a reference count, which is incremented each time a new name points at it and decremented whenever one of those names goes out of scope. When the count reaches zero, the object is freed immediately. Python also ships a separate cycle-detecting garbage collector, which runs occasionally to clean up groups of objects that reference each other and so never reach a count of zero on their own.

In Rust, there is no garbage collector. Instead, the compiler keeps track of the lifetime of every variable. When a variable goes out of scope, the compiler will automatically free the memory associated with the variable.

This means that you cannot use a variable after transferring ownership. For a deeper dive into the concept of ownership, read the Rust Book.

For example, if we tried print the value of body after assigning it, the compiler would give us an error:

#![allow(unused)]
fn main() {
fn parse_air(body: AirPollution) {
    let foo = body;
    println!("{:?}", body);
}
}

error[E0382]: borrow of moved value: `body`
  --> src/bin/ch4.rs:71:22
   |
69 | fn parse_air(body: AirPollution) {
   |              ---- move occurs because `body` has type `AirPollution`, which does not implement the `Copy` trait
70 |     let foo = body;
   |               ---- value moved here
71 |     println!("{:?}", body);
   |                      ^^^^ value borrowed here after move

It’s beyond the scope of this post to explain all the details of ownership and references, but it’s important to understand that Rust’s compiler is keeping track of the lifetime of every variable, and will not allow you to use a variable after it’s been moved.

Instead, we can use a reference to a variable. This keeps the underlying data in the same place in memory, but allows us to pass it to a function as a reference to the original value.

#![allow(unused)]
fn main() {
fn parse_air(body: &AirPollution) {
    let foo = body;
    println!("{:?}", body);
}
}

This has some really nice benefits when it comes to processing large amounts of data, as data engineers tend to do.

In Python, it’s not always clear when data is being copied, moved, or referenced. In Rust, copying code is very explicit. If we didn’t want to borrow a reference in the code above, we could also copy.

#![allow(unused)]
fn main() {
fn parse_air(body: AirPollution) {
    let foo = body.clone();
    println!("{:?}", body);
}
}

For the above code to work, we would also need to implement the Clone trait for the AirPollution struct and all of its fields:

#![allow(unused)]
fn main() {
#[derive(Debug, Clone, Deserialize)]
pub struct AirPollution {
...

}

Understanding ownership, references, and borrowing can be an uphill battle for new Rust programmers who are used to dynamically-typed languages, but with time and patience, it will come to you too.

Performance

To benchmark our code, we’re going to change our code to fetch an entire forecast rather than a single day, increasing the payload from 0.5kb to about 13kb.

In Python, we change the url and then iterate over every element in the list provided.

def get_air_pollution(lat, lon):
    url = f"https://api.openweathermap.org/data/2.5/air_pollution/forecast?lat={lat}&lon={lon}&appid={API_KEY}"
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    return response.json()


def parse_air_pollution(body):
    res = []
    print(body)
    for row in body["list"]:
        res.append((row["main"]["aqi"], row["components"], row["dt"]))
    return res


def print_air_pollution(main, components, dt):
    print("---")
    print(f"Air pollution forecast for {dt}")
    print(f"Air quality index: {main}")
    print("Components:")
    for k, v in components.items():
        print(f"  {k}: {v}")

In Rust, we also change the url and use the common iter().map().collect() pattern.

#![allow(unused)]
fn main() {
pub fn get_air_pollution(lat: f32, lon: f32) -> AirPollution {
    let api_key = std::env::var("OWM_APPID").expect(
        "Environment Variable OWM_APPID not set. Please set it to your
    OpenWeatherMap API key. https://home.openweathermap.org/api_keys",
    );

    let url = format!(
        "https://api.openweathermap.org/data/2.5/air_pollution/forecast?lat={}&lon={}&appid={}",
        lat, lon, api_key
    );

    let client = reqwest::blocking::Client::builder()
        .timeout(std::time::Duration::from_secs(10))
        .build()
        .expect("client failed");

    client
        .get(url)
        .send()
        .expect("request failed")
        .error_for_status()
        .expect("server returned an error")
        .json()
        .expect("json failed")
}

pub fn parse_air_pollution(body: AirPollution) -> Vec<(Main, Components, usize)> {
    body.list
        .iter()
        .map(|x| (x.main, x.components, x.dt))
        .collect()
}
}

These results were regenerated against the live forecast API after both programs were switched to HTTPS with an explicit timeout and status check, using five warmups and ten measured runs per program. Both programs exit non-zero on an HTTP error, so every timed run was a successful request. The response contains 96 records in about 13KB.

Rust averaged 183.4ms and Python 213.5ms, a ratio of about 1.16x. The run-to-run standard deviations were 27.6ms and 32.1ms respectively. User CPU time was 34.3ms for Rust and 61.6ms for Python, while system CPU time was 6.4ms and 12.8ms.

As in the previous chapter, these are whole-program measurements that include startup, a TLS handshake, and a live HTTP request. Most of the Rust CPU time is the handshake, which is why the gap is far narrower than the offline results below. They also include output formatting: the Python example prints the decoded response before printing individual records, while the Rust example prints individual records. This is not an isolated, equal-output comparison of deserialization speed.

To actually measure the parsing we need to get the network out of the way.

Offline Benchmarks

Benchmarking against a network connection can be a bit iffy. It also makes it hard to test larger and larger payloads, so we’ll create a large payload file and use that for an offline benchmark.

I’ve created a 9mb JSON file that mirrors the payload from the OpenWeather API, and created offline versions of the Rust and Python code to read from a local file. The code for both can be found in the sample repository under wxpy/wxpy/ch4/serialized_offline_benchmark.py and wxrs/src/bin/ch4_offline_benchmark.rs.

Here are the results of the offline benchmarks:

With the network out of the picture and a much larger payload, the refreshed run takes about 16.7ms in Rust and 102.4ms in Python, a ratio of about 6.1x. This measures the whole program: startup, file reading, JSON decoding, formatting, and writing output, rather than decoding alone. Both programs process the same records, but their output labels and formatting differ.

An aside: Rust is not automatically faster

The first time I ran this benchmark, Rust lost, and it is worth explaining why, because it is a mistake that is very easy to make.

Both programs print a line per record, and there are a lot of records. Rust’s standard output is line buffered, which means every println! costs a write syscall. Python’s standard output, when it is not attached to a terminal, is block buffered, so it batches those same lines into far fewer, larger writes. The result was a Rust program that spent most of its life in the kernel:

The following measurements are from that earlier buffering experiment; they were not rerun during the dependency refresh.

Look at the system time. That is the whole story: the parsing was never the problem. Wrapping stdout in a BufWriter is the idiomatic fix, and it is what the code in the repository now does.

#![allow(unused)]
fn main() {
let stdout = std::io::stdout();
let mut out = BufWriter::new(stdout.lock());
}

The general lesson is one worth internalising before you rewrite anything in Rust for performance: a language that is capable of being faster will still happily let you write something slower, and the bottleneck is very often I/O rather than the computation you were focused on. Measure, and look at where the time actually goes.

Transforming Data using Polars

In this chapter, we’ll look at how to transform data using Polars in both Python and Rust.

Polars is a “blazing fast DataFrame library” available in both Python and Rust. When I first wrote this chapter it was reasonable to describe it as a faster pandas with fewer features; that framing has not aged well. Polars has since reached 1.0 on the Python side and covers most of what you would reach for pandas to do.

The Polars documentation is a great resource for getting started, and the API docs have even more detail on syntax.

One thing worth knowing up front: the two languages are on different version numbers for the same project. The Python package is at 1.44.1 and the Rust crate is at 0.55.2. They are not as far apart as that makes them look.

Getting the data

This chapter uses the Project FeederWatch dataset, which is checked into the repository as a 7z archive because the extracted CSV is about 1.4GB. Unpack it first:

# from the repo root
make data

A note on lazy vs eager

Both languages give you two ways to work: eager, where each operation runs immediately, and lazy, where you describe the whole query and let Polars optimise it before running anything. Lazy is where the interesting work happens — it can push our column selection and our valid == 1 filter down into the CSV reader, so it never materialises the columns and rows we are going to throw away.

Both versions below use the lazy API, which keeps the comparison honest. It is also how you would write this in practice.

Let’s look at some key differences between the syntax in Python and Rust.

Python

import os

import polars as pl

script_path = os.path.dirname(os.path.realpath(__file__))
bird_path = os.path.join(script_path, "../../../lib/PFW_2016_2020_public.csv")
codes_path = os.path.join(script_path, "../../../lib/species_code.csv")

# The columns we care about, in the casing the CSV actually uses.
COLS = [
    "LATITUDE",
    "LONGITUDE",
    "SUBNATIONAL1_CODE",
    "Month",
    "Day",
    "Year",
    "SPECIES_CODE",
    "HOW_MANY",
    "VALID",
]

birds = pl.scan_csv(bird_path).select([pl.col(c).alias(c.lower()) for c in COLS])

codes = pl.scan_csv(codes_path, infer_schema_length=None).select(
    [
        pl.col("SPECIES_CODE").alias("species_code"),
        pl.col("PRIMARY_COM_NAME").alias("species_name"),
    ]
)

birds_df = (
    birds.filter(pl.col("valid") == 1)
    .group_by(["subnational1_code", "species_code"])
    .agg(
        [
            pl.col("how_many").sum().alias("total_species"),
            pl.col("how_many").count().alias("total_sightings"),
        ]
    )
    .join(codes, on="species_code", how="inner")
    .sort("total_species", descending=True)
    .collect()
)

print(birds_df)

The Python code is very concise. pl.scan_csv gives us a lazy frame, columns can be selected as a list of expressions, sort takes a simple descending argument, and nothing actually runs until the final collect().

I’ve also included an attempt at the same logic in pandas. While largely similar, there are a few differences, for example, in how we filter for valid results. Pandas has no lazy mode, so it does all of the work eagerly.

import os

import pandas as pd

script_path = os.path.dirname(os.path.realpath(__file__))
bird_path = os.path.join(script_path, "../../../lib/PFW_2016_2020_public.csv")
codes_path = os.path.join(script_path, "../../../lib/species_code.csv")

# adding usecols reducing memory usage and runtime from 13s to 7s
birds = pd.read_csv(
    bird_path,
    usecols=[
        "LATITUDE",
        "LONGITUDE",
        "SUBNATIONAL1_CODE",
        "Month",
        "Day",
        "Year",
        "SPECIES_CODE",
        "HOW_MANY",
        "VALID",
    ],
).rename(columns=lambda x: x.lower())

codes = pd.read_csv(codes_path)[["SPECIES_CODE", "PRIMARY_COM_NAME"]].rename(
    columns={"SPECIES_CODE": "species_code", "PRIMARY_COM_NAME": "species_name"}
)

birds = birds[
    [
        "latitude",
        "longitude",
        "subnational1_code",
        "month",
        "day",
        "year",
        "species_code",
        "how_many",
        "valid",
    ]
]

birds = birds[birds["valid"] == 1]
birds = (
    birds.groupby(["subnational1_code", "species_code"])
    .agg(total_species=("how_many", "sum"), total_sightings=("how_many", "count"))
    .reset_index()
)

birds = pd.merge(birds, codes, on="species_code", how="inner").sort_values(
    "total_species", ascending=False
)


print(birds)

Now let’s compare the above to Rust code.

Rust

use polars::prelude::*;

// Resolved at compile time relative to this crate, so the program works no
// matter which directory you run it from.
const BIRD_PATH: &str = concat!(
    env!("CARGO_MANIFEST_DIR"),
    "/../lib/PFW_2016_2020_public.csv"
);
const CODES_PATH: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/../lib/species_code.csv");

// The columns we care about, in the casing the CSV actually uses.
const COLS: [&str; 9] = [
    "LATITUDE",
    "LONGITUDE",
    "SUBNATIONAL1_CODE",
    "Month",
    "Day",
    "Year",
    "SPECIES_CODE",
    "HOW_MANY",
    "VALID",
];

fn main() -> PolarsResult<()> {
    let birds = LazyCsvReader::new(BIRD_PATH.into())
        .with_has_header(true)
        .finish()?
        .select(
            COLS.iter()
                .map(|name| col(*name).alias(name.to_lowercase()))
                .collect::<Vec<_>>(),
        );

    let codes = LazyCsvReader::new(CODES_PATH.into())
        .with_has_header(true)
        .with_infer_schema_length(None)
        .finish()?
        .select([
            col("SPECIES_CODE").alias("species_code"),
            col("PRIMARY_COM_NAME").alias("species_name"),
        ]);

    let joined = birds
        .filter(col("valid").eq(lit(1)))
        .group_by([col("subnational1_code"), col("species_code")])
        .agg([
            col("how_many").sum().alias("total_species"),
            col("how_many").count().alias("total_sightings"),
        ])
        .join(
            codes,
            [col("species_code")],
            [col("species_code")],
            JoinArgs::new(JoinType::Inner),
        )
        .sort(
            ["total_species"],
            SortMultipleOptions::default().with_order_descending(true),
        )
        .collect_with_engine(Engine::Streaming)?
        .unwrap_single();

    println!("{}", joined);
    Ok(())
}

The shape of the query is identical — scan, select, filter, group, aggregate, join, sort — but the Rust version is roughly 60% longer.

Almost all of that extra length is types and error handling rather than logic. A few things worth pointing out:

• main returns PolarsResult<()>, which lets us use ? after every fallible call. An earlier version of this chapter was littered with unwrap; this reads better and behaves better.
• sort takes a SortMultipleOptions builder rather than a bare keyword argument, because Rust has no keyword arguments.
• LazyCsvReader::new wants a PlRefPath, not a PathBuf, so the paths go through .into().
• The paths themselves are built with concat!(env!("CARGO_MANIFEST_DIR"), ..), which resolves them at compile time relative to the crate. Python gets the same effect at runtime from __file__.

Overall the APIs are close enough that translating between them is mostly mechanical.

Benchmarks

Let’s look at some benchmarks for polars in both Python and Rust, as well as similar code in Pandas.

These results were regenerated after updating to Rust Polars 0.55.2 and Python Polars 1.44.1. The three implementations were checked against each other: all 7,724 aggregate rows agree across all five columns after sorting independently of display order.

Both Polars versions beat pandas on this query. Rust Polars takes about 1.390s, Python Polars 0.436s, and pandas 4.237s. That makes Rust Polars about 3.0x faster than pandas and Python Polars about 9.7x faster. This is evidence for this workload, rather than a guarantee for every pandas program.

Python Polars is still the fastest of the three, about 3.2x faster than our Rust build. It would be easy to quietly drop that finding. It is more interesting to sit with it.

Both Polars implementations execute their data operations in Rust. The Python package provides bindings to Polars, so this is not a comparison between a Python loop and a Rust loop. It compares the Python distribution of Polars with our local Rust build, including their execution settings. The user CPU time exceeds wall time for both Polars programs, consistent with work running across multiple threads inside the engine.

An earlier investigation tried a different Rust allocator and switched the Rust query to the streaming engine. Neither change closed the gap in that run. Those experiments were not repeated during this dependency refresh; the checked-in Rust example continues to request the streaming engine, while Python calls collect() with its default settings.

Compiler tuning, enabled features, and execution settings are possible contributors to the remaining difference. We have not isolated their effects, so the timings do not establish that build tuning is the cause. Pinning the package versions alone does not make these two implementations identical.

The lesson is similar to the BufWriter example in the last chapter: reaching for Rust does not hand you performance. When Python already calls a native library, changing your application language may buy you little. Measure the complete workload and verify that the results agree before interpreting the timing differences.

Concurrent Programming

One of Rust’s major goals as a language is to enable fearless concurrency. So much so that an entire chapter of the Rust Book is devoted to it.

In Python, concurrency is possible however we are impacted by the GIL.

What’s really fascinating (to me, anyways) is how decisions about how memory is managed in both languages has a direct impact on how concurrency is handled.

Before we dig into concurrency, let’s take a step back and talk about memory.

Memory

Every programming language stores objects in memory. Whether it’s variables, functions, or other data, we store these in memory to allow fast access to them when we need them.

How languages manage memory defines the flavor and performance characteristics of the language.

The GIL and Python’s Memory Management

In Python, the infamous Global Interpreter Lock (or GIL) exists because objects in Python are reference counted. This means that every object has a counter associated with it that is incremented as it is referenced and decremented as it is removed from scope. When an object has 0 references, it is cleared from memory, freeing up space.

Those counters are not themselves thread-safe: if two threads incremented or decremented the same count at once, the object could be freed while still in use, or leak forever. Rather than lock every object individually, CPython takes a single global lock, the GIL, which guarantees that only one thread executes Python bytecode at a time. This has the effect of serializing execution and effectively making CPU-bound Python code single-threaded, no matter how many threads you spawn.

To work around these limitations, CPU-bound Python code has to reach for separate processes, typically via the multiprocessing module, which sidesteps the GIL by giving each process its own interpreter. That comes with its own set of limitations and overhead costs, since data has to be pickled and copied between processes rather than simply shared.

About the Author

This Rust for Data book was created by me, Pedram Navid.

You can find me on Twitter @pdrmnvd

and on LinkedIn @pedramnavid

and on GitHub @pedramnavid

and on Substack @databased.

and on my website pedramnavid.com.