On Git Refs

I have recently improved my mental model of Git. Consider these two git commands:

$ git fetch origin master
$ git switch -c my-feature origin/master

Do you understand why is it origin master in one command, and origin/master in the other? I didn’t, until a few weeks ago!

My understanding was that git is a content-addressable database. Git stores commits, a commit is identified by the hash of its content, and the content of a commit is, primarily:

• a memory-less snapshot of a state of the codebase at a given point in time,
• a list of (hashes of) parent commits.

That was enough git for me to understand git log output and get me out of any botched rebase without having to re-clone the repo (For roughly half of my career, I was re-cloning the repo. No shame in that! Learning git is useful, but it’s not the highest priority thing to learn when you start).

I now understand that git not only comes with an append-only (“immutable”) content-addressable database, but is also a boring mutable key-value store.

Git has a mutable map whose keys are strings, and whose values are content-addressed objects. The keys are conventionally formatted as file system paths, and you can usually inspect the state of the mapping by listing .git/refs directory:

$ eza -T .git/refs
.git/refs
├── heads
│   ├── make
│   ├── master
│   ├── my-feature
│   └── pbd-adt
├── origin
├── remotes
│   └── origin
│       ├── context-switches
│       ├── gh-pages
│       ├── HEAD
│       ├── make
│       └── master
└── tags

$ cat .git/refs/heads/master
b59148228e52f7c615ead7fdd4e91001994ad50f

$ git show-ref refs/heads/master
b59148228e52f7c615ead7fdd4e91001994ad50f refs/heads/master

What makes this refs KV infrastructure confusing is that:

• It powers many distinct user-visible git features, but refs themselves are an implementation detail.
• It is a bit of a leaky abstraction, refs are almost invisible in the day-to-day usage.
• Git CLI uses shorthand notation for refs and many default arguments, which makes it not obvious that a particular CLI argument is a ref.
• And, as usual, git likes to give several names to one thing, and re-uses the same name for distinct things.

Branches, tags, and git notes are all just refs!

The structure becomes much more obvious once you elaborate all CLI shortcuts. The original command

$ git fetch origin master

then becomes

$ git fetch \
    https://github.com/matklad/matklad.github.io \
    refs/heads/master:refs/remotes/origin/master

The first argument of fetch (https://...) is a location of a remote repository. Git will “dial” that address, and will transfer some data from that computer locally over the network.

The second argument is a source:target pair of string keys (refs). The source is a key on the remote repo, the target is the name of a local key, and fetch as a whole asks git to read a value from a remote repository and save it locally under a different name.

To avoid typing repository URLs all the time, git assigns them symbolic names, with origin being the conventional name for the primary remote repository:

$ git fetch origin \
    refs/heads/master:refs/remotes/origin/master

refs/heads/master is a fully elaborated name of a branch on the remote repo. That is, branch my-feature is just a refs/heads/my-feature ref. It could have been refs/branch/my-feature, but it isn’t :)

I don’t know the specific shorthand rules, but, generally, git allows you to spell only the suffix of a ref:

$ git fetch origin \
    master:refs/remotes/origin/master

refs/remotes/origin/master is the name of the local ref we’ll use to store the result. It would seem natural to just use the same name locally as the one on the remote, but this only works if there’s a single remote. If there are two upstream repositories (for example, your fork, and the original repo you forked from), their ref names will collide. That’s why we want to namespace the refs for remote called foo under refs/remotes/foo. And origin is just a conventional name for the remote in simple setups.

Again, it would be more natural to directly mirror remote ref structure locally:

refs/ heads/my-branch -> refs/ remotes/origin/ heads/my-branch

but git strips the redundant heads component. And this -heads, +remotes/$remote mapping is built in, which compresses the command to

$ git fetch origin master

It’s worth reflecting why it works this way. Git model is offline first. What’s more, it assumes explicit synchronization points. Rather than synchronizing with the remote repository in background when there’s connectivity, git requires explicit fetch and push operations to transfer bytes over the wire. In this paradigm, it is useful to model the state of the remote party at the moment when we talked to them the last time. Theory of mind!

This hopefully deconfuses git’s concept of local and remote branches. Consider the main branch. It exists on the remote named origin as refs/heads/main. When you synchronize your local repository with origin, you get refs/remotes/origin/main — you current best knowledge about the the state of main on the origin.

And then there’s your local refs/heads/main. It typically starts pointing at the same commit as refs/remotes/origin/main. But, when you make a commit, refs/heads/main advances, but refs/remotes/origin/main stays the same.

When you try to push your local commit to origin, you will get a conflict, if the main branch on the origin advanced in the meanwhile. In that case, git automatically updates refs/remotes/origin/main (as that’s just a local mirror of the remote state), but then it’s on you to update refs/heads/main and push it again.

Revisiting the full example:

$ git fetch origin master
$ git switch -c my-feature origin/master

The first command looks up the URL for the origin remote in .git/config and makes a network request to that machine. As a result, the local refs/remotes/origin/master gets updated to the same commit as refs/heads/master remotely (the commit and its ancestors are transferred locally as a result).

The second command creates a refs/heads/my-feature ref (a branch), whose starting point is refs/remotes/origin/master. It is an example of a leaky abstraction.

The second argument there is a (shorthand of a) ref, so you can do

$ git switch -c my-feature \
    refs/remotes/origin/master

But, although the first argument creates a ref, it isn’t a ref itself. In other words, if you try to elaborate it as well

$ git switch -c refs/heads/my-feature \
    refs/remotes/origin/master

you’ll get

refs/heads/refs/heads/my-feature

That’s all! I am pretty sure this isn’t particularly useful, but maybe it is interesting!

Benchmark In Milliseconds

How long should a micro benchmark run? My rule of thumb is to tweak the input size until the benchmark takes about 300ms, for the following reasons:

• Milliseconds are integers ranging from 1 to 999. Enough precision to notice even a small improvement, and easy to scan visually. No need for different units or floating points (compare 1.31s with 239ms).
• Anything faster than, say, 10ms risks being skewed by fixed costs (e.g, interpreter startup). Hundreds of milliseconds is an eternity for a computer, usually enough to make one-off overheads irrelevant without using fancier (= less robust) techniques to explicitly account for them.
• For a human, hundreds of milliseconds is fast, but noticeable. Pushing numbers into human-perceptible range allows me to use my intuitive sense of time and speed, it doesn’t rely exclusively on numeracy. It’s plain fun to see, as a result of optimization work, how a previously lagging CLI command becomes “instant”.
• But anything longer than a second makes iterating on the benchmark slower than it needs to be. Running a benchmark 10 times in a row to eyeball variance should be fast!

The imminently-to-be-stated assumption here is that the purpose of benchmarking isn’t so much a precise measurement of performance, but rather providing the author with enough intuition to make a correct decision.

Finding Bugs

Are generative (randomized) tests significantly more effective than example-based unit-tests at discovering bugs? There’s an interesting discussion about this on lobste.rs. One argument in favor of unit tests is, paraphrasing

To me, it seems that generative testing should shake out that particular creature, so I wrote a lil fuzzer of my own, and it indeed discovered another bug in that version of regex, and then the one I was after. I didn’t find anything in the latest version. I like to do a write up about the process, as it is a good case study for how one approaches a problem like this.

I want to be extra clear that my argument is very weak here, as I know exactly the bug I am after, and I even know that fuzzers can find it. My primary goal is to teach you the techniques, leaving it to your judgment just how effective they are. That being said, I think finding a second bug validates the approach somewhat.

I also want to emphasize that writing fuzzers to find known bugs is far from an idle amusement. While I believe that generative testing is very powerful, relative to its cost, it’s always a question whether a particular test is thorough enough. And it never is, you will find more bugs elsewhere (that’s why defense in depth and runtime mitigations are critical). And, whenever you have a pest that dodged your fuzzers, your first order of business is to treat this event as a bug in the fuzzer, and change it so that it can find this and related bugs. Only then you are allowed to add a fix and a unit test!

use regex;

fn main() {
    let r = regex::Regex::new(".abb|b").unwrap();

    let m = r.find("zabb").unwrap();

    // Fails with regex-automata=0.4.15:
    assert_eq!(m.as_str(), "zabb")
}

use fastrand::Rng;

use fastrand::Rng;

fn main() {
    let mut rng = Rng::new();

    // Re-use the same memory for all tests.
    let mut text_alphabet: Vec<u8> = vec![];
    let mut text: Vec<u8> = vec![];


    for _ in 0..1_000_000 {
        // It's unlikely that a counter example with
        // 7 different letters exists, while there
        // isn't one with just 6.
        alphabet_swarm(&mut rng, b"abcdef", &mut text_alphabet);
        let text =
            gen_string(&mut rng, &text_alphabet, &mut text);
    }
}

fn alphabet_swarm<'a>(
    rng: &mut Rng,
    all: &[u8],
    pick: &'a mut Vec<u8>,
) {
    pick.clear();
    pick.extend(all);
    rng.shuffle(pick);
    let count = rng.usize(1..=pick.len());
    pick.truncate(count);
}

fn gen_string<'a>(
    rng: &mut Rng,
    alphabet: &[u8],
    result: &'a mut Vec<u8>,
) -> &'a str {
    result.clear();
    // Again, this is a short string.
    // Longer failures are not likely.
    let count = rng.usize(0..8);
    for _ in 0..count {
        result.push(alphabet[rng.usize(0..alphabet.len())]);
    }
    str::from_utf8(result).unwrap()
}

#[derive(Default, Debug)]
struct ReOptions {
    alt: u16, // |
    rep: u16, // *
    any: u16, // .
    lit: u16, // 'a'
    sum: u16,
    alphabet: Vec<u8>,
}

impl ReOptions {
    fn swarm(&mut self, rng: &mut Rng, alphabet_full: &[u8]) {
        // We _still_ want to enable a few features at a time.
        self.alt = if rng.bool() { 0 } else { rng.u16(0..100) };
        self.rep = if rng.bool() { 0 } else { rng.u16(0..100) };
        self.any = if rng.bool() { 0 } else { rng.u16(0..100) };
        self.lit = rng.u16(1..100);
        self.sum = self.alt + self.rep + self.any + self.lit;
        assert!(self.sum > 0);
        alphabet_swarm(rng, alphabet_full, &mut self.alphabet);
    }
}

fn gen_re(
    rng: &mut Rng,
    options: &ReOptions,
    result: &mut Vec<u8>,
) {
    result.clear();
    let size = rng.u8(0..8);
    gen_re_rec(rng, options, result, size);

}

fn gen_re_rec(
    rng: &mut Rng,
    options: &ReOptions,
    result: &mut Vec<u8>,
    size: u8,
) {
    if size == 0 {
        return; // Base case, empty regex.
    }

    // Pick one of the features, according to weights.
    let mut p = rng.u16(0..options.sum);
    if p < options.alt {
        // Alternation distributes the size
        // among the two children.
        let size_left = rng.u8(0..=size - 1);
        let size_right = size - size_left - 1;
        assert!(size == size_left + 1 + size_right);

        result.push(b'(');
        gen_re_rec(rng, options, result, size_left);
        result.extend(b")|(");
        gen_re_rec(rng, options, result, size_right);
        result.push(b')');
        return;
    }
    p -= options.alt;

    if p < options.rep {
        result.push(b'(');
        gen_re_rec(rng, options, result, size - 1);
        result.extend(b")*");
        return;
    }
    p -= options.rep;

    if p < options.any {
        gen_re_rec(rng, options, result, size - 1);
        result.push(b'.');
        return;
    }
    p -= options.any;

    if p < options.lit {
        gen_re_rec(rng, options, result, size - 1);
        let index = rng.usize(0..options.alphabet.len());
        let lit = options.alphabet[index];
        result.push(lit);
        return;
    }
    unreachable!();
}

fn main() {
    let mut rng = Rng::new();

    let mut options = ReOptions::default();
    let mut text_alphabet: Vec<u8> = vec![];
    let mut text: Vec<u8> = vec![];
    let mut re: Vec<u8> = vec![];

    let mut test_count: u32 = 0;
    for _ in 0..1_000_000 {
        options.swarm(&mut rng, b"abcdef");
        alphabet_swarm(&mut rng, b"abcdefx", &mut text_alphabet);

        gen_re(&mut rng, &options, &mut re);

        let re = str::from_utf8(&re).unwrap();
        let r1 = regex::Regex::new(re).unwrap();
        let r2 = regex_lite::Regex::new(re).unwrap();

        for _ in 0..1000 {
            test_count += 1;
            let text =
                gen_string(&mut rng, &text_alphabet, &mut text);

            let m1 = r1.find(text)
                .map_or("not found", |it| it.as_str());
            let m2 = r2.find(text)
                .map_or("not found", |it| it.as_str());

            if m1 != m2 {
                eprintln!("err re={re} text={text} m1={m1} m2={m2}");
                return;
            }

            if test_count % 500_000 == 0 {
                eprintln!("ok  re={re} text={text}");
            }
        }
    }
}

err re=(e)|(fee) text=xxfee

err re=(f..)*.d text=xfcbdd

use fastrand::Rng;

fn main() {
    let mut rng = Rng::new();

    let mut options = ReOptions::default();
    let mut text_alphabet: Vec<u8> = vec![];
    let mut text: Vec<u8> = vec![];
    let mut re: Vec<u8> = vec![];

    let mut test_count: u32 = 0;
    for _ in 0..1_000_000 {
        options.swarm(&mut rng, b"abcdef");
        alphabet_swarm(&mut rng, b"abcdefx", &mut text_alphabet);

        gen_re(&mut rng, &options, &mut re);

        let re = str::from_utf8(&re).unwrap();
        let r1 = regex::Regex::new(re).unwrap();
        let r2 = regex_lite::Regex::new(re).unwrap();

        for _ in 0..1000 {
            test_count += 1;
            let text =
                gen_string(&mut rng, &text_alphabet, &mut text);

            let m1 = r1.find(text)
                .map_or("not found", |it| it.as_str());
            let m2 = r2.find(text)
                .map_or("not found", |it| it.as_str());

            if m1 != m2 {
                eprintln!("err re={re} text={text} m1={m1} m2={m2}");
                return;
            }

            if test_count % 500_000 == 0 {
                eprintln!("ok  re={re} text={text}");
            }
        }
    }
}

fn alphabet_swarm<'a>(
    rng: &mut Rng,
    all: &[u8],
    pick: &'a mut Vec<u8>,
) {
    pick.clear();
    pick.extend(all);
    rng.shuffle(pick);
    let count = rng.usize(1..=pick.len());
    pick.truncate(count);
}

fn gen_string<'a>(
    rng: &mut Rng,
    alphabet: &[u8],
    result: &'a mut Vec<u8>,
) -> &'a str {
    result.clear();
    let count = rng.usize(0..8);
    for _ in 0..count {
        result.push(alphabet[rng.usize(0..alphabet.len())]);
    }
    str::from_utf8(result).unwrap()
}

#[derive(Default, Debug)]
struct ReOptions {
    alt: u16, // |
    rep: u16, // *
    any: u16, // .
    lit: u16, // 'a'
    sum: u16,
    alphabet: Vec<u8>,
}

impl ReOptions {
    fn swarm(&mut self, rng: &mut Rng, alphabet_full: &[u8]) {
        self.alt = if rng.bool() { 0 } else { rng.u16(0..100) };
        self.rep = if rng.bool() { 0 } else { rng.u16(0..100) };
        self.any = if rng.bool() { 0 } else { rng.u16(0..100) };
        self.lit = rng.u16(1..100);
        self.sum = self.alt + self.rep + self.any + self.lit;
        assert!(self.sum > 0);
        alphabet_swarm(rng, alphabet_full, &mut self.alphabet);

    }
}

fn gen_re(
    rng: &mut Rng,
    options: &ReOptions,
    result: &mut Vec<u8>,
) {
    result.clear();
    let size = rng.u8(0..8);
    gen_re_rec(rng, options, result, size);

}

fn gen_re_rec(
    rng: &mut Rng,
    options: &ReOptions,
    result: &mut Vec<u8>,
    size: u8,
) {
    if size == 0 {
        return; // Base case, empty regex.
    }

    // Pick one of the features, according to weights.
    let mut p = rng.u16(0..options.sum);
    if p < options.alt {
        // Alternation distributes the size
        // among the two children.
        let size_left = rng.u8(0..=size - 1);
        let size_right = size - size_left - 1;
        assert!(size == size_left + 1 + size_right);

        result.push(b'(');
        gen_re_rec(rng, options, result, size_left);
        result.extend(b")|(");
        gen_re_rec(rng, options, result, size_right);
        result.push(b')');
        return;
    }
    p -= options.alt;

    if p < options.rep {
        result.push(b'(');
        gen_re_rec(rng, options, result, size - 1);
        result.extend(b")*");
        return;
    }
    p -= options.rep;

    if p < options.any {
        gen_re_rec(rng, options, result, size - 1);
        result.push(b'.');
        return;
    }
    p -= options.any;

    if p < options.lit {
        gen_re_rec(rng, options, result, size - 1);
        let index = rng.usize(0..options.alphabet.len());
        let lit = options.alphabet[index];
        result.push(lit);
        return;
    }
    unreachable!();
}

Static Allocation, Constant Work

In reply to this email:

Yes, object pools are an interesting case to think about, as they clarify the relation between memory safety and more general correctness.

First, consider the case where no object pool is used, and we malloc and free order objects. In this case, the logical error of use-after-free turns into physical type confusion, and can easily lead to arbitrary code execution and the like. If you have two objects of different types sharing the same memory location, a user-controlled integer in one object might be a function pointer in the other: an exploitable goto primitive

Now, what happens if we introduce an object pool which stores a list of “dead” objects of type T? Logical use-after-free is still possible, but its physical effect is now different — we still get aliasing of memory, but there’s no type confusion. You can’t necessarily fiddle with an integer and change a function pointer, unless you additionally hit the hard case, where the object in question stores an inline enum. Assuming that doesn’t hapen, you get a perfectly defined, deterministic behavior, even if you are not happy about the result.

This suggests an interesting solution for hardening code, which I’ve learned from Fil. If your allocation function is typed (it takes a T comptime parameter or runtime type witness, rather than a runtime type-erased size and alignment), you can write an allocator that uses type-segregated pools internally. This will be somewhat less memory efficient, as the allocator won’t be able to re-use freed memory of objects of type U for objects of type T, but the memory overhead will probably be small (rare object types do not matter, popular object types will have a lot of intra-type re-use), you might actually gain in memory locality, and solve most of type confusions. Again, inline enums break this, but, curiously, if you always heap allocate enum variants, then this works again. Fil-C can’t use this, because C allocator’s interface is untyped, but someone else could :P

But this is academic. How do we avoid the bugs? Generational indexes are a popular remedy, but I have never used them, so I don’t have any non-common knowledge insights about this pattern. Instead, I will share another pair of tricks from TigerStyle. I have only a vague understanding of what an order matching engine is, but I suspect these tricks might help there

$ order-engine --orders-max=1_000_000

const orders: []Order = try gpa.alloc(Order, cli_args.orders_max);

const OrderPool = struct {
    orders: []Order,
    free: DynamicBitSet,

    fn acquire(pool: *OrderPool) ?*Order { ... }
    fn release(pool: *OrderPool, order: *Order) { ... }
};

const OrderPool = struct {
    orders: []union {
        order: Order,
        next_free: ?u32,
    },
    first_free: ?u32,
};

const Order = {
    id: u128,
    price: u32,
    count: u32,

    tag: enum { bid, ask, reserved },

    pub const reserved: Order = .{
        .id = 0,
        .price = 0,
        .count = 0,
        .tag = .reserved,
    };
};

for (orders) |order| {
    process(order)
}

for (orders_active) |order_index| {
    const order = orders[order_index];
    process(order);
}

const item = for (items) |item| {
    if (predicate(item)) break item;
} else null;

Cancelation Terminology

A short note explaining the difference between synchronous cancelation, asynchronous cancelation, and graceful shutdown. I am not too attached to these specific three terms, but I want to call your attention to the three things behind them, which are important not to confuse with each other.

synchronous cancelation is an (often implicit) control flow structure. It unwinds the stack and looks like this:

task.cancel();
// The task will have finished by this point.

Synchronous cancelation is a bit like Molière’s prose — we do it all the time, but not necessarily in full consciousness. The primary source of synchronous cancelation is error handling — every time an Exception is thrown or an error returned, the code promptly breaks out of all the loops, ifs, and blocks, invoking the necessary cleanup actions via RAII, finally, with/try with resources or defer.

asynchronous cancelation is a communication protocol between two parties. One party requests cancelation (synchronously), but then it has to wait until the other party acknowledges it and winds down. It looks like this:

task.request_cancelation();
// The task could still be running here.
task.join().await;
// After the requisite wait, the task is finished.

Like synchronous cancelation, this is a relatively low-level concern when implementing a concurrent program in a way that doesn’t crash or hang. I know two central example where an asynchronous cancelation is required.

First is the CPU thread pool. Imagine you have offloaded encrypting a buffer to a separate thread as a part of handling user’s request. Some time later, you learn that the request must be canceled (perhaps the user had left). You can’t just abandon the encrypting thread. First, it would be smart not to waste CPU cycles for useless work, but, more importantly, the underlying buffer must remain tied up. If it were to be freed as a result of request cancelation, something else might re-use that memory, leading to data races.

But you also can’t just cancel that thread synchronously! It’s in the middle of a hyper-optimized SIMD loop, and you really don’t want it to check the cancelation flag before reading every byte. What you’d want is to split the buffer into reasonably-sized chunks, and check the cancelation status after every chunk. But that means that the party that requested the cancelation must wait for at least one chunk’s worth of work!

Another example here is io_uring. It has exactly the same shape: if you submit a write with a buffer to the kernel, that buffer must remain tied up until the write finishes (and you can cancel the write to make it finish faster). While io_uring is still at least a somewhat exotic technology (though, arguably, it’s the interfaces we have had before which are byzantine), the thread pool example demonstrates that the phenomenon of asynchronous cancelation itself is rather mundane.

Asynchronous cancelation comes up all the time when writing concurrent software. Because it affects the overall shape of the code, it’s useful to identify it early. Conversely, it is useful to ask yourself whether you need asynchronous cancelation at all, and whether synchronous one can be made to work. This is especially important in Rust, which makes synchronous cancelation too easy, and doesn’t provide great mechanisms for asynchronous one.

Finally, graceful shutdown is an application programming pattern for handling connections. It lives on a higher level of abstraction than the two cancelations. If you are implementing a web service, you can implement shutdown by stopping your accept loop (rejecting new connections), but continuing to serve all existing connections until their respective clients disconnect. If the load balancer is configured to route new connection requests to different instances of the service, this pattern allows you to do rolling upgrades without service disruptions.

As a bonus point, a related idea is that of crash-only software. Cancelation is all good, but your entire program can get SIGKILLed arbitrarily by an OOM killer, and the entire computer might get rebooted on powerloss. Reliable software has to handle ungraceful shutdown without losing data. But, if you can survive powerloss, you might as well implement the Quit button by SIGKILLing yourself, simultaneously simplifying the implementation and increasing testing coverage for powerloss scenarios.

To give some examples from TigerBeetle, Grid.cancel is an asynchronous cancelation. It takes a callback to notify the caller when the cancelation is done. This API is used during state sync. When a replica determines that that cluster is so far ahead that event based transfer doesn’t work, and that a state transfer is required to catch up, it must cancel all outstanding grid read operations. A read can be backed either by replica’s local disk, or by transparent fetch of the data from a neighboring replica. In the first case, we have to wait until the read is done. In the second case, we need to abandon the read — remote read getting stuck is probably the reason for us to state sync in the first place.

StateMachine.reset is an example of a synchronous cancelation. This is the part of the same flow as Grid.cancel, and is an example of how you can simplify the code if you think clearly about asynchronous vs synchronous cancelation. Ultimately, StateMachine sits on top of the Grid, but there’s a bunch of intermediate layers (Forest, Tree, Compaction, Scan, etc). A naive approach would be to notice that Grid requires asynchronous cancelation and propagate asynchrony throughout the stack. What we do instead is asynchronously canceling just the Grid directly, and then synchronously reseting everything else.

Another example of asynchronous cancelation is Client.shutdown. When an application using TigerBeetle “drops” the Client object, we need to free all OS resources. Our client also uses io_uring, so we must first wait for all outstanding syscalls to complete. In the comment, we call it “graceful shutdown”, but I think this is wrong, and this is the motivation for writing down this article. We don’t do graceful shutdown at TigerBeetle — it’s crash only all the way. Tail latency tolerance (asking several nodes for an answer and picking the fastest one) is a more general solution, as it handles not only crash faults, but also gray failures. In a distributed system, a very slow node looks exactly the same as a crashed one. A crash is just a degree of slowness.

Take aways:

• Synchronous cancelation is control flow operator
• Asynchronous cancelation is a communication protocol
• Graceful shutdown is an application-level design pattern

Rust Glancer

Rust Glancer, a functional LSP server for Rust which uses two orders of magnitude less RAM, is incredibly cool. Go check it out! This post started as a comment on lobste.rs, but I figured it out that it’s better to publish it somewhat more prominently. Don’t expect polished writing though!

Some thoughts:

Yeah, rowan is garbage :P I was really thinking about

• incremental parsing,
• incremental, DOM-mutation style refactorings,

And Rowan is pretty good for that. But that’s 1% use case. The 99% use case is all the code in your 6666 dependencies which you won’t ever look at, but which needs to be at least shallowly analyzed. Even for incremental tool whose main goal is refactoring, the primary AST structure should be just a list of arrays. There might be a real post about that at some point, see https://youtu.be/G93oYL1ry70 as a teaser.

If I understand correctly, Rust Glancer wants to process each function body. I think that part can perhaps be made lazy (but not incremental!) with little overhead? Index all items, but, for functions, do only the currently opened file? This might combine some of the better parts of both worlds.

Would be interesting to compare memory usage with Rust Rover. Net of the IDE GUI itself, I would expect RR to be more compact.

I might be rationalizing/misremembering things, but IIRC it’s exactly around adding proc macros that the thing began to feel unreasonably bulky. Expanding proc macros is slow as we are running real code, we can’t really do normal IDE cheats. And proc macros generate a lot of code. At one point I measured, it was like 30% of rust-analyzer binary size was attributed to JSON parsing code. If no one sees the code, it can’t harm anybody, right?

One potential approach here is to pull the Sorbet trick, where you don’t run meta programming at all, and instead have a plugin interface to “explain” the effects of what that would have done. Instead of running serde, we just add a shim that injects imp Serialize for T {} with an empty body.

Rust analyzer’s core data model is very pedantic about always observing consistent snapshots of the code, and does its best to ensure that the language client and server have a shared, strictly serializable view of the world. It’s a shame that LSP doesn’t allow that to be correct, only heuristically right, unlike the older Dart Analyzer protocol, which has sound data synchronization.

However our implementation of file watching is sketchy! First, there are two backends: we can ask the editor to do watching for us, or we can use server side watching. Try changing this option and see if it helps? But then, yeah, my recollection is that our native watcher’s API was fundamentally racy, and I didn’t do the messy platform-specific work of making it correct.

But the main thing I want to write, and why I moved from the cozy lobste.rs text area to the luxurious comforts of an Emacs buffer, is that right now rust-analyzer is a bit like that half-drawn horse meme, except that it’s only the head half of the horse.

One Big Idea of IntelliJ is that its PSI API (essentially AST with resolved types) is really an interface, and there are multiple providers. And in a typical usage, there’re at least three backends in play:

• For the files opened in the editor, actively modified by the user, the PSI is backed by the concrete syntax trees.
• For the rest of the project files, the PSI is backed by the so called Stub Tree, a compact on disk representation storing only the “externally visible” parts of the file (so, without function bodies). If the user navigates to a new file, its PSI transparently switches from stubs to syntax tree.
• For dependencies, the PSI is often backed by the compiled .class files, produced by javac. If you navigate there, the IDE just decompiles stuff for you! Super cool!

This is how I think such things should work. rust analyzer shouldn’t use salsa for all those 6666 dependencies you still haven’t looked at. It should just use rustc’s .rmeta files, switching to salsa, transparently, only when the user starts messing around their ~/.cargo/registry/src folder.

The prerequisite for that is defining the abstract API for accessing Rust code. That was always the plan, and we did start on that at some point:

https://hackmd.io/ytd82QNiT_Ku2XFr1EAtiQ

But I don’t think that work was ever completed.

This still seems to me to be the lowest-hanging watermelon here — split the world into arcy-pointy incremental tip of the iceberg, and mostly read-only, on disk, compact, dark, moist breeding ground for supply chain attacks.

Such glance analyzer architecture would be great, imo!

Better Batteries

One of the eternal schisms in programming is over the question of whether the standard library should be minimal or encompassing. This is the wrong question to ask. The right one is:

Python is always brought up as example of leaky batteries exploding in slow motion, but this has nothing to do with size. The problem with Python’s stdlib is its, ahem, uneven quality. Some standard library modules don’t follow language naming conventions! You know which unittest module I am talking about :-)

But even that is not a mistake. It’s actually Python core’s advantage — that it makes functionality available early, not thinking about the future too much. That’s how we ended up with ossified cAPI which makes CPython the language, but that is also how we ended up with Python powering data scientific revolution.

The Go standard library is similarly encompassing, but it is held in a high regard. Go team has institutional capacity to deliver well-designed API for the standard library, and then some: https://pkg.go.dev/golang.org/x

Rust is an interesting case. The 1.0 standard library APIs are brilliant. Collections and iterators are a work of art. But it also feels that, while the current team has the capacity to preserve existing APIs and fill in some gaps, the capacity to execute design decisions is limited. While golang.org/x captures excess capacity, rust-lang-nursery is a graveyard. Maybe I am over-indexing on my favorite hobby-horse, but it seems that the reason for Rust not having an API to get a stream of random bytes from the OS in 2026 is that, while it is an easy technical problem, it requires tricky organization architecture (including getting money in peoples’ pockets, of course) to actually get solved in the high-stakes environment of a world-wide coordination problem called a programming language.

Printing Lists

To print a comma-separated list, a concise idiom is to optionally print the comma first, before the element:

for (items, 0..) |item, item_index| {
    if (item_index > 0) std.debug.print(", ");
    std.debug.print("{}", .{item});
}

Zig’s Io.Threaded is Neat

std.Io.Threaded is one of the implementations of Zig’s new Io interface that enables concurrency. This is a boring “just use threads” impl. I personally find it neat though — it does this weird thing that I wanted to do for ages, that to my knowledge no one else is doing properly, and implements it better than I thought to be possible.

Io.Threaded uses blocking syscalls and fully supports cancelation.

use rayon::prelude::*;
fn sum_of_squares(input: &[i32]) -> i32 {
    input.par_iter()
         .map(|i| i * i)
         .sum()
}

while (true) {
    if (is_canceled()) return error.Canceled; /// Easy!
    ...
}

const read_size = try read(fd, buffer); // ???

Memory Safety’s Hardest Problem

Uplifting a lobsters comment for easier reference.

The central memory safety counter example, the hardest case to solve, doesn’t have anything to do with destructors or heap:

const std = @import("std");

const E = union(enum) {
    a: u128,
    b: []const u8,
};

pub fn main() void {
    const bad_addr: u128 = @intFromPtr(&main);

    var e: E = .{ .b = "hello" };
    const oh_no_pointer: *const []const u8 = switch (e) {
        .a => unreachable,
        .b => |*p| p,
    };
    e = .{ .a = (16 << 64) + bad_addr };
    const oh_no: []const u8 = oh_no_pointer.*;
    std.debug.print("{s}\n", .{oh_no});
}

$ zig run main.zig
��C�� �

This sort of example also breaks Ada:

https://www.enyo.de/fw/notes/ada-type-safety.html

We have a tagged union, which can hold either A or B. We initialize the union as A, take a pointer to its internals, overwrite the original with B, and then use the pointer. The pointer is still typed as A, but the bytes it points to now belong to B: a type confusion.

This being said, we care about memory unsafety primarily because it leads to exploitable software, and it’s unclear just how impactful the example above is in practice. It is a happy coincidence that by far the most exploitable memory error in practice, the infamous buffer overflow, is also trivial to fix with compiler-inserted bounds checks. The biggest miss of the industry when it comes to memory safety is not listening to Walter Bright:

https://digitalmars.com/articles/C-biggest-mistake.html

I bet that, had we got char a[..] syntax around C11, quite a few issues wouldn’t have happened!

See also What is Memory Safety?