The happy path is a lie

Every pipeline I wrote in 2025 had the same missing branch. except Exception is why I couldn’t see it.

For about seven months a small nightly job did exactly what I asked of it. It read the day’s threads from a public discussion forum, kept the six fields I cared about, ranked them by likes, and wrote the top five into a table that a dashboard picked up the next morning. It never paged me, so I stopped reading its logs.

Then one Tuesday the dashboard showed a quiet week. Activity down, three days running. I am a sociologist by training, so I did what I am trained to do: I opened a notebook and started drafting a paragraph about seasonal decline in forum engagement.

There was no decline. The forum had reorganised its API. The endpoint still answered, still returned valid JSON, and no longer carried the key my parser wanted. My code caught the KeyError, wrote one line into a log nobody reads, returned [], and exited zero. Three nights in a row it had overwritten real data with nothing — on schedule, successfully.

I spent that morning treating a measurement artifact as a social fact. The pipeline had given me no way to tell the two apart, because an empty result and a quiet week are the same object. That was true of every pipeline I wrote in 2025, and I wrote a lot of them.

Four failures wearing one coat

The bug was not the KeyError. The bug was the shape of the function around it:

def scrape(source: str) -> list[dict]:
    try:
        ...
    except Exception as error:
        log.warning("scrape failed: %s", error)
        return []

That signature promises list[dict] and the body underneath it can fail four unrelated ways. The network can be down, so nothing arrives. The host can be up and answer 500. Bytes can arrive that are not parseable JSON. The JSON can parse cleanly and not have the shape I assumed.

Those are four different facts about the world, and they want four different responses. Wait and try again. Back off and try again much later. Try again right now, because a body that stops mid-object is almost always transport. Stop entirely, go read the upstream changelog, and change the code — because the contract moved and no amount of retrying will put the key back.

except Exception erases all four distinctions at the one moment they still exist. After that line the caller holds an empty list, and the list does not know why it is empty. Nothing downstream can recover what was thrown away upstream. That is the missing branch: not a line I forgot to write, but a difference I declined to carry.

Giving the failures names

The fix is boring, which is the best thing about it. Four frozen dataclasses and one match:

@dataclass(frozen=True)
class NetworkError:
    cause: str

@dataclass(frozen=True)
class ApiError:
    status: int

@dataclass(frozen=True)
class ParseError:
    detail: str

@dataclass(frozen=True)
class ShapeError:
    detail: str

ScrapeError = NetworkError | ApiError | ParseError | ShapeError

def classify(error: Exception) -> ScrapeError:
    match error:
        case httpx.HTTPStatusError():
            return ApiError(status=error.response.status_code)
        case httpx.HTTPError() | OSError():
            return NetworkError(cause=str(error))
        case json.JSONDecodeError():
            return ParseError(detail=str(error))
        case KeyError() | TypeError() | ValueError():
            return ShapeError(detail=repr(error))
        case _:
            return NetworkError(cause=repr(error))

Each name carries a payload, because a label without evidence is not much better than a stack trace. ApiError(status=503) and ApiError(status=404) deserve different treatment, and the caller is the one who should decide which.

classify runs once, at the edge, as the last step of the pipeline:

def scrape(source: str, top_n: int = 5) -> Result[list[dict], ScrapeError]:
    return (
        read_source(source)
        .bind(parse_json)
        .bind(extract_topics)
        .map(lambda ts: [clean_topic(t) for t in ts])
        .map(lambda ts: rank_topics(ts, n=top_n))
        .alt(classify)
    )

The library here is returns, which gives me Result, Success, Failure, and .bind. That part is replaceable; a hand-rolled tagged union would do the same work. What is not replaceable is the signature. It now says out loud that this function has two outcomes, and it names the second one.

Both versions, run on the same three files

import sys
sys.path.insert(0, ".")

from railway.pipeline import scrape

for fixture in ("topics.json", "broken.json", "wrong-shape.json"):
    print(f"{fixture:20} {scrape(f'fixtures/{fixture}', top_n=2)}")

topics.json          <Success: [{'posts_count': 8, 'views': 5100, 'like_count': 88, 'id': 2, 'title': 'Why our nightly ETL lies about success', 'created_at': '2026-06-11T17:02:00Z'}, {'posts_count': 12, 'views': 3400, 'like_count': 41, 'id': 1, 'title': 'Retry storms in a queue worker', 'created_at': '2026-06-02T09:14:00Z'}]>
broken.json          <Failure: ParseError(detail='Expecting value: line 2 column 1 (char 28)')>
wrong-shape.json     <Failure: ShapeError(detail="KeyError('topic_list')")>

Figure 1 — frozen at render time, 8 Aug 2026

One call site, three inputs, three labelled outcomes. topics.json is a normal response and comes back as Success with the two highest-liked threads. broken.json is a body that stopped mid-write; it comes back as ParseError carrying the character offset where the parser gave up, and retrying it is reasonable, because that is usually a connection that died. wrong-shape.json is valid JSON without a topic_list; it comes back as ShapeError, and that is my Tuesday morning. Retrying it produces the same KeyError forever. Something upstream changed and a human has to look.

try/except can catch both of those. What it cannot do is hand the difference to the caller as a value.

The two names the figure does not show, NetworkError and ApiError, work the same way; they are only harder to stage from a file on disk.

And none of this means try/except is bad. The fetch layer of this same package is a bare try: ... except Exception as error: return Failure(error). Catching broadly at the boundary and converting the throw into a value is exactly what the construct is for. The failure mode is try/except as the caller’s entire vocabulary for what went wrong.

What it costs

Every signature gets longer and stays longer. list[dict] becomes Result[list[dict], ScrapeError], and every caller has to open the box before it can do anything, including the callers that only ever wanted the happy path. Tests get an extra layer of unwrapping. Notebooks get uglier.

.bind chains read strangely for a good while. I still reach for .map where .bind belongs, and the type checker is usually the only reason I notice before the tests do.

The taxonomy has to be agreed before it pays anything, and four categories is a design decision, not a discovery. If two services in the same system disagree about what ShapeError means, you are worse off than with except Exception, because the labels look authoritative and are not.

Mine is not finished either. Look at the last case in classify: an exception it does not recognise is labelled NetworkError. That tells the caller “transient, retry” about a failure nobody has understood yet. It is wrong, it is mine, and the honest fix is a fifth case for the unknown, which I have not written. A taxonomy with a permissive default branch quietly lies at its edges.

And the job still fails exactly as often as it did before. It fails on the same nights, for the same reasons, at the same rate. What changed is that the failure now arrives with a name attached, and the three days I lost to a decline that never happened would have been three lines in a log saying ShapeError.

The full version of this argument — the box, the two tracks, and how you test code shaped like this — is a short book I wrote around this same scraper, The Happy Path Is a Lie.

The code above is vendored verbatim from the railway package, from the book The Happy Path Is a Lie.