Memory Safety’s Hardest Problem

Uplifting a lobsters comment for easier reference.

The central memory safety counter example, the hardest case to solve, doesn’t have anything to do with destructors or heap:

const std = @import("std");

const E = union(enum) {
    a: u128,
    b: []const u8,
};

pub fn main() void {
    const bad_addr: u128 = @intFromPtr(&main);

    var e: E = .{ .b = "hello" };
    const oh_no_pointer: *const []const u8 = switch (e) {
        .a => unreachable,
        .b => |*p| p,
    };
    e = .{ .a = (16 << 64) + bad_addr };
    const oh_no: []const u8 = oh_no_pointer.*;
    std.debug.print("{s}\n", .{oh_no});
}

$ zig run main.zig
��C�� �

This sort of example also breaks Ada:

https://www.enyo.de/fw/notes/ada-type-safety.html

We have a tagged union, which can hold either A or B. We initialize the union as A, take a pointer to its internals, overwrite the original with B, and then use the pointer. The pointer is still typed as A, but the bytes it points to now belong to B: a type confusion.

This being said, we care about memory unsafety primarily because it leads to exploitable software, and it’s unclear just how impactful the example above is in practice. It is a happy coincidence that by far the most exploitable memory error in practice, the infamous buffer overflow, is also trivial to fix with compiler-inserted bounds checks. The biggest miss of the industry when it comes to memory safety is not listening to Walter Bright:

https://digitalmars.com/articles/C-biggest-mistake.html

I bet that, had we got char a[..] syntax around C11, quite a few issues wouldn’t have happened!

See also What is Memory Safety?

CSS: Unavoidable Bad Parts

An ersatz CSS tutorial for people who need to style a web page, but aren’t web developers. I am a wrong person to write this kind of thing, as I have neither the time, nor experience. I’d much rather read a book about this. Alas, I had to learn all this stuff from trawling MDN, so perhaps it is valuable to document what I have so far.

CSS, HTML and Web APIs are truly vast, and it takes a career to become a professional. The good news is that modern web has a reasonably-sized, learnable subset which is enough for simple tasks like a programming blog or a simple GUI. I haven’t seen a resource that teaches just this subset, but it’s not too hard to figure this out. The bad news is that there’s also a nasty set of gotchas, which will mess up your page, which you won’t suspect to exist, and which will need days of debugging to figure out. Still, it’s not that bad. I am quite happy with the styling on this site, and it’s only about 200 of readable CSS.

Good: HTML5 semantic tag names
It’s worth looking through MDN Elements Reference. There aren’t that many elements, and things like main, article, nav, kbd make it much easier to structure your page. Less obvious:

• ul for any kind of list, like site’s sections in header > nav.
• details for table-of-contents (check the source of MDN).
• dl/dt for list of pairs.

Bad: Wrappers
If you “View Source” on any “real” website, you’ll notice that everything has layers and layers of wrapper elements, so you might be tricked into thinking that wrappers are how you solve layout problems. I can’t really agree or disagree here, as I never wrote “production” CSS, but, in my experience, it’s much easier to understand if you do the opposite — restrict yourself to using only markup-meaningful semantic tags, and then figure out CSS which works with the markup you have.

Bad: Layout
This one is not an exclusively Web problem, layout is a struggle in every GUI framework I know. Imagine a fixed sized raster image, and a paragraph of text describing it. There are many ways to arrange these two elements on the screen’s rectangle. Generally, for every given width and height, you can do a decent job, as long as the total area is enough. A typical GUI is a hierarchy of such boxes, with a lot of “layout freedom”. The problem though is that layout of each box affects the layouts of all other boxes, as you generally want all boxes to meet exactly, without gaps and overlaps. An important negative realization is that the layout algorithm doesn’t exist. There isn’t a fully general solution to positioning and sizing GUI boxes. Rather, different systems use different sets of heuristics to do the job, from simple RectCut, to fully general constraint solvers, with everything in between. It is hard to get the mental model of how layout works, in general. So, don’t think “how can I do my layout in a given system”, think instead “what possible layouts are allowed by the system”.

Bad: Browser defaults
Let’s start with a bare (but still semantic) HTML markup of a blog article, without any CSS. If you open it in a browser, it will show something. The content isn’t unstyled — the text is of a certain color, font and size. Headers are bigger than the main text, links are underlined, etc. These are the default styles of your browser. They are helpful! The problem is that these styles differ between the browsers. So, even when you add your own CSS, and the end result looks fine in your browser, I might see something different, because you might rely on a browser default, without knowing it. The last bit is the killer here — the problem is in something you didn’t write.

The general solution here is a CSS reset, or normalization — starting your CSS with an explicit set of rules, overriding defaults. Not because defaults are inherently bad, because they are inconsistent. I don’t know which set of rules you need to override in practice, it’s a good idea to compare several existing CSS resets.

This touches on the big question: should you style your web page? There are two competing views of the Web platform — some people treat it as a flexible, adaptive, primarily visual medium for expressing design, others would prefer if the Web focused on delivering the content, allowing each user to customize the presentation. My personal answer here is pragmatic — by default, an unstyled page is poorly usable and looks bad. I would have preferred the world where CSS-less pages were readable as is, but, in this world, I think it is helpful to style the content. At the same time, it’s a good idea to allow advanced users to bring their own CSS. Make sure that your HTML markup is reasonable, that you don’t overfit your HTML to CSS (vice-versa is fine), and that your page functions in reader mode.

Good: Classless CSS
You can’t reset styles to true neutral nothing: if you make the text invisible (white or transparent), it is still a style. So you might as well embrace it: after reset, style common HTML elements directly. For example, to set your favorite font for all code snippets:

code { font-family: "JetBrains Mono", monospace; }

If you use main, header, footer, nav tags you can set the overall page layout without writing any CSS selectors. This of course requires making assumptions, in CSS, about the structure of your HTML, but, like, this is your HTML and your CSS, you can do whatever, and, if you don’t like the result, you can always change it!

Bad: CSS selectors
In programming, we collectively came around to distrust inheritance and prefer composition. Default CSS is like supercharged inheritance, each design element on your web page is affected by multiple rules, and you can always “monkey patch” existing elements by appending to your CSS. There’s an unfortunate gap between CSS affordances, and what you actually want to do. The two reasonable approaches are:

1. Conclude that CSS selectors add abstraction capability along the wrong axis, and stick to classless CSS and inline styles, using something like Tailwind to make writing inlines prettier, and something like JSX (or any other templating engine supporting composition) to avoid repetition in HTML.

2. Use CSS nesting to avoid writing “far reaching” selectors and style component-per-component:

header { /* Site Header */
    margin-bottom: 2rem;
    & nav {
        /* Styles, specific to nav in the Header. */
    }
}

Bad: box-sizing
UIs are recursive rectangles, layout is the process of figuring out where each rectangles goes, and it is determined by the sizes of rectangles themselves. So, understanding what is the size is quite fundamental. Sadly, by default the definition of size in HTML is very unintuitive: element’s width and height do not include element’s border and padding, which leads to surprising results: everything looks perfect at first, but increasing padding somewhere shifts the entire layout unexpectedly. For this reason, * { box-sizing: border-box; } deserves to be the first line in your CSS reset. It makes elements encapsulated, such that adding borders is a local-only change.

Chaotic Good: margin collapsing
Suppose you want to have a 8px gap around an element. You would think that you need to set the padding property. But that would be wrong — if you have two such elements next to each other, the gap between them would be 16px. The paddings would add, creating a visual gap larger than intended. You want something more akin to social distancing, where if one person is more introverted, this person’s bigger radius of exclusion is what defines the distance. And that’s how the margin property works. Two neighboring margins are combined using max rather than sum. Margin collapsing is very useful, but it can surprise you. E.g. I think child margin can stick beyond parent’s? To be honest, I don’t have a good intuitive understanding of margins, but I know enough to at least identify when it is the problem.

Margins are also one of the indirect inspirations for this post. In

Moving away from Tailwind, and learning to structure my CSS

Julia Evans writes that you generally don’t want to set margin on an element, and should rather let the parent control the inter-element margin of the children, using the so-called owl selector:

section > *+* {
  margin-top: 1rem;
}

That is, add margin to all section’s children exempting the first one. I didn’t know that! And, given all the pain that margin gave me so far, I actually get why you want to do this, and why this is a good idea. But it bugs me that you can’t learn that without becoming “professional” web developer, or reverse-engineering someone else’s CSS framework.

Bad: Default (flow) layout
Layout in general is tricky, because there’s no universal “layout algorithm”, just a bunch of special cases. But what does HTML actually do? The default layout algorithm I think goes back to the origin of HTML as a language for documents, and overfits a use-case of producing papers — mostly text content with some illustrations, where the text can flow around the pictures. That’s actually what you want for the main body of text of your blog, but, as soon as you want to actually control the spatial arrangement of the elements on your page, you want something different, for example…

Good: flexbox
This is really what separates modern web-development from the olden days, where you’d need a CSS PhD or a full-blown opaque CSS framework to be able to say “this goes to the left, and this goes to the right”. This layout allows you to arrange a series of elements either vertically or horizontally, adapting to the available space. It is rather complex and I can’t use flexbox without referencing MDN all the time, but usually I am able to get things done in the end.

Bad: responsive design
Modern CSS allows querying screen size, and implementing conditional logic based on that — a design that “responds” to user-agent constraints. This probably what you should use for “real” CSS, but note that HTML is inherently responsive. Unlike PostScript (PDF), it will automatically reflow the paragraphs when you change window size. So, it’s a good idea to avoid writing explicit responsive rules, and just rely on layout to do the reasonable thing. For example, this blog looks OK on mobile, tablet and desktop without any explicit @media queries. Unconditionally setting max-width on the main column of text is all that it takes.

Lawful Evil: pixels
1px does what you want, but not what it says. It’s not a size of one physical pixel on your screen. Rather, it’s a measure of visual angle. That is, 1px should look perceptually the same on any screen, and it is converted to different number of physical pixels, depending on the screen size, its pixel density, and the typical viewing distance. So you can just size everything in pixels, without thinking about different displays’ pixel densities. It gets weirder. CSS allows “real” units like centimeters or inches, but they are also angles, because everything is defined in terms of pixels.

Doubleplusungood: font-size
Flexbox is a good way to layout UI-elements. Flow layout works ok for laying out paragraphs of text. But what happens on the level of individual lines and glyphs is, in my opinion, a train wreck and a noob trap. Let’s start with the basics: if you write font-size: 16px then 16px is the size of what? Sadly, the answer is “nothing in particular” — this is a size of a virtual box around the glyph, but the box isn’t tight, and the size of the glyph varies, depending on the font. Luckily, font-size-adjust property can fix it, and make font-size consistent across fonts. See these two posts for details:

• font-size-adjust Is Useful
• Font size is useless; let’s fix it

Though, at the moment font-size-adjust seems to be very niche, so, while personally I’d put font-size-adjust: ex-height 0.53; right next to box-sizing, few pages do that.

The next issue with font-size is a thorny question of defaults. The good news is that it’s one of the properties that is fairly consistent across browsers, with 16px being the overwhelming default. The bad news is that, depending on the font, 16px can be on the smaller size. Not completely illegible, but very close to the lower bound. What’s worse, some default fonts are particularly small. For example, on Apple, font-family: serif looks much smaller than sans-serif, and is almost uncomfortable to read at 16px.

Can you just set font-size: 18px or whatever works best for your chosen font? I think the answer is yes, but there are some caveats to keep in mind. Refer to Accessibility: px or rem? for details. The issue is that modern browsers support two ways of making text on a page bigger:

• Zoom, which has a dedicated UI element, shortcuts/gestures, per-page persistence/overrides and a global default.
• Changing default font-size, a global setting buried deeply in the configuration page.

Setting font-size in your CSS disables that second approach.

Taking everything together: don’t assume that text on your page will be readable by default, check different configurations. Set font-size-adjust to reduce the number of degrees of freedom and to pin down the meaning of font-size. If the result looks fine with your chosen (or your user’s default) font and default font-size of 16px, then you are done. Otherwise, set font-size to a bigger number. Afterwards, check that the page is readable in reader mode as well.

Bad: line-height
Despite the name, line-height doesn’t set the height of a line. It is a height of a run of glyphs, set in the same font. The two coincide when all the text is in the same font. But if you have, e.g., some words set in monospace font, you are in for a surprise. While font-size-adjust fixes the size of a glyph inside the box, it still leaves its relative position unspecified. So, when two runs of text in different fonts are aligned vertically to share the baseline, their line-height line-boxes get shifted relative to each other: one sticks below, one sticks above. The line height overall becomes larger that what you’d expect, as it is configured as a union. See

Deep dive CSS: font metrics, line-height and vertical-align

for a thorough explanation of this effect.

Bad: vertical rhythm
If you google long enough this cluster of problems, sooner or later you’ll come across the idea of vertical rhythm, that you should make sure that lines are in the same relative position across different paragraphs, even if you have headings, images, and what not. As if there’s invisible lined paper behind your web-page. As far as I can tell, this is pure voodoo and is not useful. If you do two-column layout, then you want lines on opposite sides to align, but it makes no sense to jump through hoops for a single-column layout (hat tip to @chrismorgan).

Bad: word-break
The genius of the flow layout is its dynamism. It takes a moment of reflection to appreciate the technical marvel of text breaking itself neatly into lines as the window is resized to be narrower. Getting that to work for the first time ever in the world of durably printed text must have felt incredible. But the magic has its limits — you can only break the line at the whitespace, or at the hyphenation points. And some long spans, like inline code or URLs, might be unbreakable. This leads to overflow annoyance on mobile devices, something you notice only after you publish your work. There’s no one trick to fix it, but some tips are available here: Against Horizontal Scroll for details.

And … that’s all I remember so far? I reiterate my request for someone to write a short 100-page book explaining just enough of HTML&CSS to make a simple blog without getting collapsed by the margins!

TIL: Symlinking NixOS Dotfiles

The standard answer to managing dotfiles on NixOS is home-manager. I’ve never used it, due to two aesthetic and one practical objection:

• I avoid dependencies, especially in nix, which rivals Python in the number of approaches to dependency management.
• home-manager installs packages for the current user only, which makes sense on non-NixOS systems. But on a single-user desktop system, I prefer having just one set of packages.
• Having a source of truth for dotfiles be in nix store requires rebuilding your system to change config, which gets in the way of Emacs-style direct tinkering.

The approach I like is storing dotfiles in the same repository as flake.nix / configuration.nix and symlinking them in place.

The problem here is that NixOS seemingly doesn’t have a “native” way to say that /a/b/c should be a symlink to /c/d/e. Or has it?

If you search options for symlink, you’ll learn about environment.etc which allows you to configure symlinks, but only for things in /etc, not your ~/.config.

For the latter, you can use gnu stow or some other dotfile link manager, but the complexity of the problem of just managing symlinks doesn’t warrant yet another dependency. It’s fine to do this manually.

But wouldn’t it be nice if this framework for declarative configuration of your system allowed you to declaratively configure symlinks? Turns out this is possible, in roundabout way. Inaptly-named systemd-tmpfiles allows creating symlinks from a declarative config, and you can use NixOS to configure systemd-tmpfiles itself (thanks, NobbZ!).

For example, if I want to symlink ~/dotfiles/git/config to .config/git/config:

{
  systemd.tmpfiles.rules = [
    "L+ /home/matklad/.config/git/config - - - - /home/matklad/dotfiles/git/config"
  ];
}

No opinion at this point how this compares to a bespoke script or something more purpose-built.

Always Be Blaming

A few tips on 4D-ing your code comprehension skills.

I wrote on the importance of reading code before: Look Out For Bugs My default approach to reading is “predictive”: I don’t actually read the code line by line. Rather, I try to understand the problem that it wants to solve, then imagine my own solution, and read the “diff” between what I have in my mind and what I see in the editor. Non-empty “diff” signifies either a bug in my understanding, or an opportunity to improve the code.

This is 2D reading, understanding a snapshot of code, frozen in time. This is usually enough to spot “this feels odd” anomalies, worthy of further investigation.

Ideal code is memoryless — it precisely solves the problem at hand. Most real code is Markov — the shape of the code at time T depends not only on the problem statement, but also on the shape of the code at time T - 1. The 3D step is to trace the evolution of code over time, Where Do We Come From? What Are We? Where Are We Going?.

The step after that is to understand the why. What were we thinking back then, when we wrote this code? It’s useful to have the “theory of mind” concept ready here. I personally learned the term way too late in my life, so let me give a short intro for today’s lucky 10 000. Theory of mind is the ability to imagine yourself in someone else’s skin. Not just in their shoes (“I certainly would have acted differently in that situation”), but with their mind (“I wouldn’t have acted that way, but I get why they did”). This is something people learn. The experimental setup here is to have a child in a room with toys, with a doll sitting near the opposite end of the room, and asking the child “what does the doll see?”. Younger children describe the room from their perspective, older begin to intuit that doll’s perspective is different.

So this is the goal of reading code — understanding what the original author was thinking, and why.

End of the mumbo-jumbo, some practical advice. First, read Every line of code is always documented, it is very good.

Second, make sure it is effortless for you to find out how a given snippet of code evolved. This is harder than it seems! Just git blame isn’t an answer — mind the gap between the problem that’s easy to solve, and the problem in need of solving.

git blame answers spatial question of “how each line appeared in this file”, because there’s a relatively straightforward UI for this — annotate each line with a commit hash. But this is not the question you are asking most of the time! You don’t care about the file! There’s a small snippet of code in the middle, and you want a temporal history of that.

As much as I don’t like working in the browser GitHub’s web interface for blaming is probably better than what you get locally by default. It starts with the y shortcut, which resolves a symbolic reference like

https://github.com/tigerbeetle/tigerbeetle/blob/main/src/vsr/replica.zig

into the one which has a commit hash in the URL:

https://github.com/tigerbeetle/tigerbeetle/blob/c54f613a2eb2a127a0ba212704e3fa988c42e5cb/src/vsr/replica.zig

This commit hash is critical, because it anchors the entire repository — if you open a different file from the web UI, it will be shown as of that commit. This enables you to not myopically focus on just the diff in question, but to absorb the entire context at that point in time.

So my usual web workflow is:

• ctrl+f to find the line I am interested in
• b to toggle blame
• Click “blame prior to change” a couple of times, repeating ctrl+f to go back to the snippet I am curious about.
• cmd-click on the commits that are potentially relevant, pinning their commit hashes in the URL in new tabs.
• Then, from the commit page, “Browse files” button to then go and t to other files. Or, cmd+l to focus browser’s address bar, and s/commit/tree/ (or back!) as needed, to switch between diff and snapshot views.

Again, my goal here is not to annotate a diff on a file but rather to get a “virtual checkout” as of the interesting commit.

This web approach is what I was using throughout most of my career, but I’ve finally found a way to replicate it locally. The idea is to make blaming “in-place”. Instead of git blame annotating lines of code, I directly switch to a historical commit. I have the following devil hydra of shortcuts:

, b l blames line. It notes the $line the cursor is at, runs git blame -L $line,$line to find $commit that introduced the line, and then runs git switch --detach $commit to check it out. I have a dedicated worktree for code archeology, so I don’t worry about trashing my work. There’s also a half-hearted attempt to maintain “logical” cursor position, but it doesn’t work very well. Is there some git command that tells me directly “what’s the equivalent of $file:$line:column in $sha-A for $sha-B?”

, b p blames parent. Which is just switching to the parent commit of the current HEAD, what “blame before this change” does on GitHub (it works slightly differently because it assumes that , b l was the previous command)

, b u undoes the last blaming operation, switching to the previous point. I really love that, on the web, I can cmd-click to create an alternative branch of exploration. In theory, this is replicatable locally, but I prefer to destructively mutate a single working tree on disk. A big reason for preferring in-place blame is that LSP, ./zig/zig build test, rg and the like just work. That’s more important for me than the garden of forking paths, and undo is an acceptable work-around.

Finally, , b w copies GitHub link to the current commit and line, which I can paste into the browser. An enormous problem with modern version control landscape is that absolutely critical information in the form of code review comments is not a part of the git repository, and is locked in someone else’s proprietary database. I failed to solve this problem in one weekend, and had to begrudgingly adapt. Opening the commit in a browser links you to the PR and its discussion as well.

Implementing this blame workflow required a bit of custom code. Feel free to use it, but beware that it’s somewhat crufty, especially around maintaining current cursor position. Making a production-ready version of this sounds like a fun project ;-)

Catch Flakes On Main

A small Mechanical Habit today:

For an example, see the “Flakes” link on https://devhub.tigerbeetle.com

Flaky tests are tests that fail intermittently, once in a thousand runs. This might be due to a genuine bug (assumptions about scheduling that mostly hold) or due to instability of underlying infrastructure (e.g., inability to download a release from GitHub, or to delete a folder on Windows). In either case, flaky tests are a huge productivity drain — as the size and complexity of test suite grows, more and more CI runs fail spuriously, even as each individual test almost always passes.

Flaky tests are challenging to deal with — if you are working on landing a PR and your CI fails due to an obvious flake, the temptation to just re-run the test suite is enormous, especially if there’s a certain background dissatisfaction with infrastructure stability.

If you are of a mind to do some flake squashing, then your PRs will be green just to spite you! And working off of others’ PRs would require first to separate flakes from genuine failures.

This is why the merge queue is powerful: if there’s a guarantee that every commit on the main branch passes the tests, then every failure on main is a flake, by definition. Collecting all such failures into a single list compresses time, allows to prioritize the most impactful sources of instability, and reveals correlations between failures.

Learning Software Architecture

In reply to an email asking about learning software design skills as a researcher physicist:

I was attached to a bioinformatics lab early in my career, so I think I understand what you are talking about, the phenomenon of “scientific code”! My thoughts:

First meta observation is that “software design” is something best learned by doing. While I had some formal “design” courses at the University, and I was even “an architect” for our course project, that stuff was mostly make-believe, kindergarteners playing fire-fighters. What really taught me how to do stuff was an accident of my career, where my second real project (IntelliJ Rust) propelled me to a position of software leadership, and made design my problem. I did make a few mistakes in IJ Rust, but nothing too horrible, and I learned a lot. So that’s good news — software engineering is simple enough that an inquisitive mind can figure it out from first principles (and reading random blog posts).

Second meta observation, the bad news: Conway’s law is important. Softwaregenesis repeats the social architecture of the organization producing software. Or, as put eloquently by neugierig,

I suspect that the difference you perceive between industrial and scientific software is not so much about software-building knowledge, but rather about the field of incentives that compels people to produce the software. Something like “my PhD needs to publish a paper in three months” is perhaps a significant explainer?

Two things you can do here. One, at times you get a chance to design or nudge an incentive structure for a project. This happens once in a blue moon, but is very impactful. This is the secret sauce behind TIGER_STYLE, not the set of rules per se, but the social context that makes this set of rules a good idea.

Two, you can speedrun the four stages of grief to acceptance. Incentive structure is almost never what you want it to be, but, if you can’t change it, you can adapt to it. This is also true about most industrial software projects — there’s never a time to do a thing properly, you must do the best you can, given constraints.

Let me use rust-analyzer as an example. The physical reality of the project is that it’s simultaneously very deep (it’s a compiler! Yay!) and very wide (opposite to an LLM, a classical IDE is a lot of purpose-built special features). The social reality is that “deep compiler” can attract a few brilliant dedicated contributors, and that the “breadth features” can be a good fit for an army of weekend warriors, people who learn Rust, who don’t have sustained capacity to participate in the project, but who can sink an hour or two to scratch their own itch.

My insistence that rust-analyzer doesn’t require building rustc, that it builds on stable, that it doesn’t have any C dependencies, and that the entire test suite takes seconds, was in the service of the goal of attracting high-impact contributors. I was wrangling the build system to make sure people can work on the borrow checker without thinking about anything else.

To attract weekend warriors, the internals of rust-analyzer are split into multiple independent features, where each feature is guarded by catch_unwind at runtime. The thinking was that I explicitly don’t want to care too much about quality there, that the bar for getting a feature PR in is “happy path works & tested”. It’s fine if the code crashes, it will only attract further contributors, provided that:

• the quality is isolated to a feature, and doesn’t spill over,
• at runtime, the crash is invisible to the user (it’s crucial that rust-analyzer features work with an immutable snapshot, and can’t poison the data).

In contrast, when working on the core spine which provided support for features, I was very relatively more pedantic about quality.

A word of caution about adapting to, rather than fixing incentive structure — the future is uncertain, and tends to happen in the least convenient manner. The original motivation behind rust-analyzer experiment was to avoid the need to write a parallel compiler (the one in IntelliJ Rust), and to prototype a better architecture for LSP, so that the learnings could be backported to rustc. So, even in core (especially in core), the code was very experimental. Oh well. Stuck with one more compiler now, I guess?

I might hazard a guess that something similar happened to uutils project, which started as the primary destination for people learning Rust, and ended up as Ubuntu coreutils implementation.

Third, now to some concrete recommendations. Sadly, I don’t know of a single book I can recommend which contains the truths. I suspect one can only find such a book in an apocryphal short story by Borges: practice seems to be an essential element here. But here are some things worth paying attention to:

Boundaries talk by Gary Bernhardt is all-time favorite. It contains solid object-level advice, and, for me, it triggered the meta inquiry.

How to Test is something I wish I had. I immediately understood the importance of testing, but it took me a long time to grow arrogant enough to admit that most widely-cited testing advice is shamanistic snake-oil, and to conceptualize what actually works.

∅MQ guide and, more generally, writings by Pieter Hintjens introduced me to Conway’s Law thinking. That “feature development” architecture of rust-analyzer? – optimistic merging, applied.

Reflections on a decade of coding by Jamii is excellent, goes very meta. It is intentionally the first of my links.

Ted Kaminski blog is the closest there is to a coherent theory of software development, appropriately framed as a set of notes to a non-existing book!

As for the actual books, Software Engineering at Google and Ousterhout’s The Philosophy of Software Design are often recommended. They are good. SWE, in particular, helped me with a couple of important names. But they weren’t ground breaking for me.

Steering Zig Fmt

Two tips on using zig fmt effectively. Read this if you are writing Zig, or if you are implementing a code formatter.

For me, zig fmt is better than any other formatter I used: rustfmt, the one in IntelliJ, deno fmt. zig fmt is steerable. For every syntactic construct, it has several variations for how it might be laid out. The variation used is selected by looking at what’s currently in a file.

Easier to show a pair of examples:

    f(1, 2,
      3);

// -> zig fmt ->

    f(1, 2, 3);

    f(1, 2,
      3,);

// -> zig fmt ->

    f(
        1,
        2,
        3,
    );

Depending on the trailing comma, function call is formatted on a single line, or with one argument per line.

The way this plays out in practice is that you decide how you want to lay out the code, add a couple of ,, hit the reformat shortcut (, p is mine), and zig fmt does the rest. For me, this works better than the alternative of the formatter guessing. 90% of great formatting are blank lines between logical blocks and tasteful choice of intermediate variables, so you might as well lean into key choices, rather than eliminate them.

I know of one non-trivial formatting customization point: columnar layout for arrays:

    .{ 1, 2, 3,
       4, 5, 6, 7, 8, 9, 10, 11,  };

One would think that trailing comma would lead to a number-per-line layout, but, for arrays, zig fmt also takes note of the first line break. In this case, the line break comes after the first three items, so we get three numbers per line, aligned:

    .{
        1,  2,  3,
        4,  5,  6,
        7,  8,  9,
        10, 11,
    };

How cool is that!

Furthermore, with judicious use of ++ (array concatenation), you can vary the number of items per line. When I need to pass --key value pairs to subprocess, I often go for formatting like this:

try run(&(.{ "aws", "s3", "sync", path, url } ++ .{
    "--include",            "*.html",
    "--include",            "*.xml",
    "--metadata-directive", "REPLACE",
    "--cache-control",      "max-age=0",
}));

Minimal Viable Zig Error Contexts

fn process_file(io: Io, path: []const u8) !void {
    errdefer log.err("path={s}", .{path});

    const fd = try Io.Dir.cwd().openFile(io, path, .{});
    defer fd.close(io);

    // ...
}

Out of the box, Zig provides minimal and sufficient facilities for error handling — strongly-typed error codes. Error reporting is left to the user. Idiomatic solution is to pass a Diagnostics out parameter (“sink”) to materialize human-readable strings as needed.

Diagnostics pattern works well for “production” code, but for more script-y code it adds too much friction relative to the default option of a plain try fallible(), which of course gives a less than ideal message on failure:

λ zig build
error: FileNotFound
~/.cache/zig/p/../lib/std/Io/Threaded.zig:4866:35: 0x1044126c7 in dirOpenFilePosix (fail)
                        .NOENT => return error.FileNotFound,
                                  ^
~/.cache/zig/p/../lib/std/Io/Dir.zig:578:5: 0x104347d8b in openFile (fail)
    return io.vtable.dirOpenFile(io.userdata, dir, sub_path, options);
    ^
~/fail/main.zig:10:16: 0x10443da5f in f (fail)
    const fd = try Io.Dir.cwd().openFile(io, path, .{});
               ^
~/fail/main.zig:6:5: 0x10443db47 in main (fail)
    try process_file(io, "data.txt");
    ^

Error trace is helpful, but knowing which file is the problem is even more so.

The first attempt at finding a middle ground between fully-fledged diagnostics sink pattern and a plain try is something like this:

const fd = dir.openFile(io, path, .{}) catch |err| {
    log.err("failed to open file '{s}': {t}", .{path, err});
    return err;
}

Unsatisfactory. The friction is high, you need to come up with a reasonably-sounding error message, the “happy path” of the code is obscured, and you need to repeat this for every fallible operation.

A worse-is-better version of the above code is

errdefer log.err("path={s}", .{path});
const fd = try dir.openFile(io, path, .{});

That is, just log error context as key=value pairs, guarded by errdefer. The result is not pretty, but passable:

λ zig build
error: path=./data.txt
error: FileNotFound
~/.cache/zig/p/../lib/std/Io/Threaded.zig:4866:35: 0x1044126c7 in dirOpenFilePosix (fail)
                        .NOENT => return error.FileNotFound,
                                  ^
~/.cache/zig/p/../lib/std/Io/Dir.zig:578:5: 0x104347d8b in openFile (fail)
    return io.vtable.dirOpenFile(io.userdata, dir, sub_path, options);
    ^
~/fail/main.zig:10:16: 0x10443da5f in f (fail)
    const fd = try Io.Dir.cwd().openFile(io, path, .{});
               ^
~/fail/main.zig:6:5: 0x10443db47 in main (fail)
    try process_file(io, "data.txt");
    ^

The friction is reduced a lot:

• No need to come up with any error messages beyond existing variable names.
• No need to change any of the trys.
• The context is set per-block. If a function does several fallible operations on a file, the path needs to be specified only once.
• The context is “telescopic” every function in the call-stack can add its own context.

There’s one huge drawback though — the error message is logged, even if the error is subsequently handled. This is especially important in Zig 0.16, where cancelation (serendipitous-success) is a possible error for any IO-ing operation, and which is intended to be handled, rather than reported.

Generalizing:

• Happy path adds context to all operations in-progress.
• Errors materialize current context.

This does feel like a better error management strategy than decorating errors individually, when they happen. I wonder which language features facilitate this style?

This article https://goldstein.lol/posts/error-progress/ rather convincingly argues that the answer might be “none”?

256 Lines or Less: Test Case Minimization

Property Based Testing and fuzzing are a deep and science-intensive topic. There are enough advanced techniques there for a couple of PhDs, a PBT daemon, and a client-server architecture. But I have this weird parlor-trick PBT library, implementable in a couple of hundred lines of code in one sitting.

This week I’ve been thinking about a cool variation of a consensus algorithm. I implemented it on the weekend. And it took just a couple of hours to write a PBT library itself first, and then a test, that showed a deep algorithmic flaw in my thinking (after a dozen trivial flaws in my coding). So, I don’t get to write more about consensus yet, but I at least can write about the library. It is very simple, simplistic even. To use an old Soviet joke about Babel and Bebel, it’s Gogol rather than Hegel. But for just 256 lines, it’s one of the highest power-to-weight ratio tools in my toolbox.

Read this post if:

• You want to stretch your generative testing muscles.
• You are a do-it-yourself type, and wouldn’t want to pull a ginormous PBT library off the shelf.
• You would pull a library, but want to have a more informed opinion about available options, about essential and accidental complexity.
• You want some self-contained real-world Zig examples :P

Zig works well here because it, too, is exceptional in its power-to-weight.

const std = @import("std");
const assert = std.debug.assert;

entropy: []const u8,

pub const Error = error{OutOfEntropy};

const FRNG = @This();

pub fn init(entropy: []const u8) FRNG {
    return .{ .entropy = entropy };
}

pub fn fill(prng: *PRNG, bytes: []u8) void { ... }

pub fn bytes(frng: *FRNG, size: usize) Error![]const u8 {
    if (frng.entropy.len < size) return error.OutOfEntropy;
    const result = frng.entropy[0..size];
    frng.entropy = frng.entropy[size..];
    return result;
}

pub fn array(frng: *FRNG, comptime size: usize) Error![size]u8 {
    return (try frng.bytes(size))[0..size].*;
}

const builtin = @import("builtin");

pub fn int(frng: *FRNG, Int: type) Error!Int {
    comptime {
        assert(@typeInfo(Int).int.signedness == .unsigned);
        assert(builtin.cpu.arch.endian() == .little);
    }
    return @bitCast(try frng.array(@sizeOf(Int)));
}

pub const cpu: std.Target.Cpu = .{
    .arch = .aarch64,
    .model = &std.Target.aarch64.cpu.apple_m3,
    // ...
}

pub fn boolean(frng: *FRNG) Error!bool {
    return (try frng.int(u8)) & 1 == 1;
}

pub fn int_inclusive(frng: *FRNG, Int: type, max: Int) Error!Int

pub fn int_inclusive(frng: *FRNG, Int: type, max: Int) Error!Int {
    comptime assert(@typeInfo(Int).int.signedness == .unsigned);
    if (max == std.math.maxInt(Int)) return try frng.int(Int);

    const bits = @typeInfo(Int).int.bits;
    const less_than = max + 1;

    var x = try frng.int(Int);
    var m = std.math.mulWide(Int, x, less_than);
    var l: Int = @truncate(m);
    if (l < less_than) {
        var t = -%less_than;

        if (t >= less_than) {
            t -= less_than;
            if (t >= less_than) t %= less_than;
        }
        while (l < t) {
            x = try frng.int(Int);
            m = std.math.mulWide(Int, x, less_than);
            l = @truncate(m);
        }
    }
    return @intCast(m >> bits);
}

pub fn range_inclusive(
    frng: *FRNG, Int: type,
    min: Int, max: Int,
) Error!Int {
    comptime assert(@typeInfo(Int).int.signedness == .unsigned);
    assert(min <= max);
    return min + try frng.int_inclusive(Int, max - min);
}

pub fn index(frng: *FRNG, slice: anytype) Error!usize {
    assert(slice.len > 0);
    return try frng.range_inclusive(usize, 0, slice.len - 1);
}

const World = struct {
    frng: *FRNG,
    replicas: []Replica,
    // ...
};

fn simulate_request(world: *World) !void {
    const replica = try world.frng.index(world.replicas);
    const payload = try world.frng.int(u64);

    world.send_payload(replica, payload);
}

fn step(world: *World) !void {
    const action = try world.frng.weighted(.{
        .request = 10,
        .message = 20,
        .crash = 1,
    });
    switch (action) {
        .request => try world.simulate_request(),
        .message => { ... },
        .crash => { ... },
    }
}

pub fn weighted(
    frng: *FRNG,
    weights: anytype,
) Error!std.meta.FieldEnum(@TypeOf(weights)) {
    const fields =
        comptime std.meta.fieldNames(@TypeOf(weights));

    var total: u32 = 0;
    inline for (fields) |field|
        total += @field(weights, field);
    assert(total > 0);

    var pick = try frng.int_inclusive(u64, total - 1);
    inline for (fields) |field| {
        const weight = @field(weights, field);
        if (pick < weight) {
            return @field(
                std.meta.FieldEnum(@TypeOf(weights)),
                field,
            );
        }
        pick -= weight;
    }
    unreachable;
}

const S = struct {
    foo: bool,
    bar: u32,
    baz: []const u8
};

const E = enum { foo, bar, baz };

pub fn swarm_weights(frng: *FRNG, Weights: type) Error!Weights {
    var result: Weights = undefined;
    inline for (comptime std.meta.fieldNames(Weights)) |field| {
        @field(result, field) = try frng.range_inclusive(u32, 1, 100);
    }
    return result;
}

fn run_test(gpa: Allocator, frng: *FRNG) !void {
    var world = World.init(gpa, &frng) catch |err|
        switch (err) {
            error.OutOfEntropy => return,
            else => return err,
        };
    defer world.deinit(gpa);

    while (true) {
        world.step() catch |err| switch (err) {
            error.OutOfEntropy => break,
        };
    }
}

const World = struct {
    frng: *FRNG,
    weights: ActionWeights,

    // ...

    const ActionWeights = struct {
        request: u32,
        message: u32,
        crash: u32,
        // ...
    };

    pub fn init(gpa: Allocator, frng: *FRNG) !void {
        const weights = try frng.swarm_weights(ActionWeights);
        // ...
    }

    fn step(world: *World) error{OutOfEntropy}!void {
        const action = try world.frng.weighted(world.weights);
        switch (action) {
            .request => { ... },
            // ...
        }
    }
};

// NOT ACTUALY ZIG :(

var world = try World.init(gpa, &frng)
    catch (error.OutOfEntropy) return;

pub fn main(init: std.process.Init) !void {
    const gpa = init.gpa;
    const io = init.io;

    var stdin_reader = std.Io.File.stdin().reader(io, &.{});
    const entropy = try stdin_reader.interface
        .allocRemaining(gpa, .unlimited);
    defer gpa.free(entropy);

    var frng = FRNG.init(entropy);

    var world = World.init(gpa, &frng, .{}) catch |err|
        switch (err) {
            error.OutOfEntropy => return,
            else => return err,
        };
    defer world.deinit(gpa);

    world.run();
}

pub const Driver = struct {
    io: std.Io,
    sut: []const u8,
    buffer: []u8,

    const log = std.log;
};

fn run_once(driver: Driver, options: struct {
    size: u32,
    seed: u64,
    quiet: bool,
}) !enum { pass, fail } {
    assert(options.size <= driver.buffer.len);
    const entropy = driver.buffer[0..options.size];

    var rng = std.Random.DefaultPrng.init(options.seed);
    rng.random().bytes(entropy);

    var child = try std.process.spawn(driver.io, .{
        .argv = &.{driver.sut},
        .stdin = .pipe,
        .stderr = if (options.quiet) .ignore else .inherit,
    });

    try child.stdin.?.writeStreamingAll(driver.io, entropy);
    child.stdin.?.close(driver.io);
    child.stdin = null;

    const term = try child.wait(driver.io);
    return if (success(term)) .pass else .fail;
}

fn success(term: std.process.Child.Term) bool {
    return term == .exited and term.exited == 0;
}

fn run_multiple(driver: Driver, options: struct {
    size: u32,
    attempts: u32,
}) !union(enum) { pass, fail: u64 } {
    // ...
}

assert(options.size <= driver.buffer.len);

for (0..options.attempts) |_| {
    var seed: u64 = undefined;
    driver.io.random(@ptrCast(&seed));

    const outcome = try driver.run_once(.{
        .seed = seed,
        .size = options.size,
        .quiet = true,
    });
    switch (outcome) {
        .fail => return .{ .fail = seed },
        .pass => {},
    }
}
return .pass;

fn search(driver: Driver, options: struct {
    attempts: u32 = 100,
}) !union(enum) {
    pass,
    fail: struct { size: u32, seed: u64 },
} {
    // ...
}

var found_size: ?u32 = null;
var found_seed: ?u64 = null;

var pass: bool = true;
var size: u32 = 16;
var step: u32 = 16;
for (0..1024) |_| {
    if (step == 0) break;
    const size_next = if (pass) size + step else size -| step;
    if (size > driver.buffer.len) break;

    const outcome = try driver.run_multiple(.{
        .size = size_next,
        .attempts = options.attempts,
    });
    switch (outcome) {
        .pass => log.info("pass: size={}", .{size_next}),
        .fail => |seed| {
            found_size = size_next;
            found_seed = seed;
            log.err("fail: size={} seed={}", .{ size_next, seed });
        },
    }
    const pass_next = (outcome == .pass);

    if (pass and pass_next) {
        step *= 2;
    } else if (!pass and !pass_next) {
        // Keep the step.
    } else {
        step /= 2;
    }

    if (pass or !pass_next) {
        size = size_next;
        pass = pass_next;
    }
} else @panic("safety counter");

if (found_size == null) return .pass;
return .{ .fail = .{
    .size = found_size.?,
    .seed = found_seed.?,
} };

pub fn main(
    gpa: std.mem.Allocator,
    io: std.Io,
    sut: []const u8,
    operation: union(enum) {
        replay: struct { size: u32, seed: u64 },
        search: struct {
            attempts: u32 = 100,
            size_max: u32 = 4 * 1024 * 1024,
        },
    },
) !void {
    const size_max = switch (operation) {
        .replay => |options| options.size,
        .search => |options| options.size_max,
    };

    const buffer = try gpa.alloc(u8, size_max);
    defer gpa.free(buffer);

    var driver: Driver = .{
        .io = io,
        .buffer = buffer,
        .sut = sut,
    };

    switch (operation) {
        .replay => |options| {
            const outcome = try driver.run_once(.{
                .size = options.size,
                .seed = options.seed,
                .quiet = false,
            });
            log.info("{t}", .{outcome});
        },
        .search => |options| {
            const outcome = try driver.search(.{
                .attempts = options.attempts,
             });
            switch (outcome) {
                .pass => log.info("ok", .{}),
                .fail => |fail| {
                    log.err("minimized size={} seed={}", .{
                        fail.size, fail.seed,
                     });
                },
            }
        },
    }
}

Consensus Board Game

I have an early adulthood trauma from struggling to understand consensus amidst a myriad of poor explanations. I am overcompensating for that by adding my own attempts to the fray. Today, I want to draw a series of pictures which could be helpful. You can see this post as a set of missing illustrations for Notes on Paxos, or, alternatively, you can view that post as a more formal narrative counter-part for the present one.

The idea comes from my mathematics of consensus lecture, with versions in English and Russian.

The Vote

The fundamental idea underpinning consensus is simple majority vote. If R0, … R4 are the five committee members, we can use the following board to record the votes:

A successful vote looks like this:

Here, red collected 3 out of 5 votes and wins. Note that R4 hasn’t voted yet. It might, or might not do so eventually, but that won’t affect the outcome.

The problem with voting is that it can get stuck like this:

Here, we have two votes for red, two votes for blue, but the potential tie-breaker, R4, voted for green, the rascal!

To solve split vote, we are going to designate R0 as the committee’s leader, make it choose the color, and allow others only to approve. Note that meaningful voting still takes place, as someone might abstain from voting — you need at least 50% turnout for the vote to be complete:

Here, R0, the leader (marked with yellow napoleonic bicorne), choose red, R2 and R3 acquiesced, so the red “won”, even as R1 and R4 abstained (x signifies absence of a vote).

The problem with this is that our designated leader might be unavailable itself:

The Board

Which brings us to the central illustration that I wanted to share. What are we going to do now is to multiply our voting. Instead of conducting just one vote with a designated leader, the committee will conduct a series of concurrent votes, where the leaders rotate in round-robin pattern. This gives rise to the following half-infinite 2D board on which the game of consensus is played:

Each column plays independently. If you are a leader in a column, and your cell is blank, you can choose whatever color. If you are a follower, you need to wait until column’s leader decision, and then you can either fill the same color, or you can abstain. After several rounds the board might end up looking like this:

The benefit of our 2D setup is that, if any committee member is unavailable, their columns might get stuck, but, as long as the majority is available, some column somewhere might still complete. The drawback is that, while individual column’s decision is clear and unambiguous, the outcome of the board as whole is undefined. In the above example, there’s a column where red wins, and a column where blue wins.

So what we are going to do is to scrap the above board as invalid, and instead require that any two columns that achieved majorities must agree on the color. In other words, the outcome of the entire board is the outcome of any of its columns, whichever finishes first, and the safety condition is that no two colors can reach majorities in different columns.

Let’s take a few steps back when the board wasn’t yet hosed, and try to think about the choice of the next move from the perspective of R3:

As R3 and the leader for your column, you need to pick a color which won’t conflict with any past or future decisions in other columns. Given that there are some greens and blues already, it feels like maybe you shouldn’t pick red… But it could be the case that the three partially filled columns won’t move anywhere in the future, and the first column gets a solid red line! Tough choices! You need to worry about the future and the infinite number of columns to your right!

Luckily, the problem can be made much easier if we assume that everyone plays by the same rules, in which case it’s enough to only worry about the columns to your left. Suppose that you, and everyone else is carefully choosing their moves to not conflict with the columns to the left. Then, if you chose red, your column wins, and subsequently some buffoon on the right chooses green, it’s their problem, because you are to their left.

So let’s just focus on the left part of the board. Again, it seems like blue or green might be good bets, as they are already present on the board, but there’s a chance that the first column will eventually vote for red. To prevent that, what we are going to do is to collect a majority of participants (R0, R2, R3) and require them to commit to not voting in the first columns. Actually, for that matter, let’s prevent them from voting in any column to the left:

Here, you asked R0, R2 and R3 to abstain from casting further votes in the first three columns, signified by black x. With this picture, we can now be sure that red can not win in the first column — no color can win there, because only two out of the five votes are available there!

Still, we have the choice between green and blue, which one should we pick? The answer is the rightmost. R2, the participant that picked blue in the column to our immediate left, was executing the same algorithm. If they picked blue, they did it because they knew for certain that the second column can’t eventually vote for green. R2 got a different majority of participants to abstain from voting in the second column, and, while we, as R3, don’t know which majority that was, we know that it exists because we know that R2 did pick blue, and we assume fair play.

---

That’s all for today, that’s the trick that makes consensus click, in the abstract. In a full distributed system the situation is more complicated. Each participant only sees its own row, the board as a whole remains concealed. Participants can learn something about others’ state by communicating, but the knowledge isn’t strongly anchored at time. By the time a response is received the answer could as well be obsolete. And yet, the above birds-eye view can be implemented in a few exchanges of messages.

Please see the Notes for further details.