Rust Glancer

Rust Glancer, a functional LSP server for Rust which uses two orders of magnitude less RAM, is incredibly cool. Go check it out! This post started as a comment on lobste.rs, but I figured it out that it’s better to publish it somewhat more prominently. Don’t expect polished writing though!

Some thoughts:

Yeah, rowan is garbage :P I was really thinking about

• incremental parsing,
• incremental, DOM-mutation style refactorings,

And Rowan is pretty good for that. But that’s 1% use case. The 99% use case is all the code in your 6666 dependencies which you won’t ever look at, but which needs to be at least shallowly analyzed. Even for incremental tool whose main goal is refactoring, the primary AST structure should be just a list of arrays. There might be a real post about that at some point, see https://youtu.be/G93oYL1ry70 as a teaser.

If I understand correctly, Rust Glancer wants to process each function body. I think that part can perhaps be made lazy (but not incremental!) with little overhead? Index all items, but, for functions, do only the currently opened file? This might combine some of the better parts of both worlds.

Would be interesting to compare memory usage with Rust Rover. Net of the IDE GUI itself, I would expect RR to be more compact.

I might be rationalizing/misremembering things, but IIRC it’s exactly around adding proc macros that the thing began to feel unreasonably bulky. Expanding proc macros is slow as we are running real code, we can’t really do normal IDE cheats. And proc macros generate a lot of code. At one point I measured, it was like 30% of rust-analyzer binary size was attributed to JSON parsing code. If no one sees the code, it can’t harm anybody, right?

One potential approach here is to pull the Sorbet trick, where you don’t run meta programming at all, and instead have a plugin interface to “explain” the effects of what that would have done. Instead of running serde, we just add a shim that injects imp Serialize for T {} with an empty body.

Rust analyzer’s core data model is very pedantic about always observing consistent snapshots of the code, and does its best to ensure that the language client and server have a shared, strictly serializable view of the world. It’s a shame that LSP doesn’t allow that to be correct, only heuristically right, unlike the older Dart Analyzer protocol, which has sound data synchronization.

However our implementation of file watching is sketchy! First, there are two backends: we can ask the editor to do watching for us, or we can use server side watching. Try changing this option and see if it helps? But then, yeah, my recollection is that our native watcher’s API was fundamentally racy, and I didn’t do the messy platform-specific work of making it correct.

But the main thing I want to write, and why I moved from the cozy lobste.rs text area to the luxurious comforts of an Emacs buffer, is that right now rust-analyzer is a bit like that half-drawn horse meme, except that it’s only the head half of the horse.

One Big Idea of IntelliJ is that it’s PSI API (essentially AST with resolved types) is really an interface, and there are multiple provides. And in a typical usage, there’s at least three backends in play:

• For the files opened in the editor, actively modified by the user, the PSI is backed by the concrete syntax trees.
• For the rest of the project files, the PSI is backed by the so called Stub Tree, a compact on disk representation storing only the “externally visible” parts of the file (so, without function bodies). If the user navigates to a new file, its PSI transparently switches from stubs to syntax tree.
• For dependencies, the PSI is often backed by the compiled .class files, produced by javac. If you navigate there, the IDE just decompiles stuff four you! Super cool!

This is how I think such things should work. rust analyzer shouldn’t use salsa for all those 6666 dependencies you still haven’t looked at. It should just use rustc’s .rmeta files, switching to salsa, transparently, only when the user starts messing around their ~/.cargo/registry/src folder.

The prerequisite for that is defining the abstract API for accessing Rust code. That was always the plan, and we did start on that at some point:

https://hackmd.io/ytd82QNiT_Ku2XFr1EAtiQ

But I don’t think that work was ever completed.

This still seems to me to be the lowest-hanging watermelon here — split the world into arcy-pointy incremental tip of the iceberg, and mostly read-only, on disk, compact, dark, moist breeding ground for supply chain attacks.

Such glance analyzer architecture would be great, imo!