Tinkering Down Under

apticrate: search for crates in your apt repositories

Sun, 24 Dec 2023 22:15:00 +1100

About six weeks ago I published a post called Rust without crates.io about how you could skip the central crates.io repository and develop directly against Debian packages. This annoyed quite a lot of people. By my rough reckoning, about one third of people liked it, one third of people hated it, and the rest only read the comments and upvoted whatever sounded good without checking what I actually wrote. Not too shabby for a day on the internet. It’s fine, the beauty of the suggested approach is that I don’t need anyone’s permission.

Anyway it turns out that if you’re used to searching on crates.io, figuring out what packages are available to you on Debian (and at what versions) is not trivial. The package names generally take the form of librust-crate-name-dev and the package version matches the crate version, so it is possible to search for them but it quickly devolves into a bunch of faffing about with apt commands. There’s a nice Rust package overview on the Debian site but this isn’t particularly easy to use either.

You’re left with ambiguity over whether multi-word crate names are separated by a hyphen or an underscore and it takes a second level of lookup to figure out what actual version librust-tokio-dev represents. Then there’s a series of metapackages that represent the dependencies required for different features. For example if you want to use the color feature of structopt then this requires more dependencies—you can install librust-structopt+color-dev, which will pull in the required librust-clap-2+color-dev (provided by librust-clap-2-dev).

To make this all a bit easier I hacked together a tool called… apticrate. If you want to use it, here is the repository. I’m just going to paste a couple of invocations here and you’ll get the idea.

$ apticrate struct
struct-patch 0.4.1              --         librust-struct-patch-dev
struct-patch-derive 0.4.1       --         librust-struct-patch-derive-dev
structmeta 0.2.0                --         librust-structmeta-dev
structmeta-derive 0.2.0         --         librust-structmeta-derive-dev
structopt 0.3.26                installed  librust-structopt-dev
  deps for feat "color"         --         librust-structopt+color-dev
  deps for feat "debug"         --         librust-structopt+debug-dev
  deps for feat "default"       installed  librust-structopt+default-dev
  deps for feat "doc"           --         librust-structopt+doc-dev
  deps for feat "no_cargo"      --         librust-structopt+no-cargo-dev
  deps for feat "suggestions"   --         librust-structopt+suggestions-dev
  deps for feat "wrap_help"     --         librust-structopt+wrap-help-dev
  deps for feat "yaml"          --         librust-structopt+yaml-dev
structopt-derive 0.4.18         installed  librust-structopt-derive-dev
synstructure 0.12.3             installed  librust-synstructure-dev
  deps for feat "proc-macro"    installed  librust-synstructure+proc-macro-dev
synstructure_test_traits 0.1.0  --         librust-synstructure-test-traits-dev

$ apticrate clap
carapace_spec_clap 0.1.12  --         librust-carapace-spec-clap-dev
clap 2.34.0                installed  librust-clap-2-dev
clap 3.2.25                --         librust-clap-3-dev
clap 4.4.6                 --         librust-clap-dev
clap-num 1.0.2             --         librust-clap-num-dev
clap-verbosity-flag 2.0.1  --         librust-clap-verbosity-flag-dev
clap_builder 4.4.6         --         librust-clap-builder-dev
clap_complete 3.1.1        --         librust-clap-complete-3-dev
clap_complete 4.4.3        --         librust-clap-complete-dev
clap_complete_fig 4.3.1    --         librust-clap-complete-fig-dev
clap_derive 3.2.25         --         librust-clap-derive-3-dev
clap_derive 4.4.2          --         librust-clap-derive-dev
clap_lex 0.5.0             --         librust-clap-lex-dev
clap_mangen 0.2.12         --         librust-clap-mangen-dev

So you can either run it directly to get a list of all Rust crate packages on your system or you can pass a filter to search for specific crates and find out what versions are available. In either case it will tell you what the name of the package is that you have to install, and neatly group up the feature metapackages in case you want to install those ones too.

As of today, I finally got around to fixing some bugs and it is now correctly picking up all the Rust crates available in testing (trixie) and stable (bookworm), relying entirely on offline package data that you can query through apt-cache.

$ cargo run | wc -l
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
     Running `target/debug/apticrate`
2423
$ apt-cache search librust-* | grep -vE "^parsec-service" | wc -l
2423

$ cargo run | wc -l
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
     Running `target/debug/apticrate`
1949
$ apt-cache search librust-* | wc -l
1949

Fair warning: the code does not look very good. It is an unfortunate reality that the Debian packages indicate which crates they provide in inconsistent ways, and in some cases not at all so I had to hard-code a short list of them. Probably I should extract some of this apt data and write test cases, but for today it’s working.

On the bright side, however, I have ensured that this apticrate code will build against the rust-all compiler provided by both Debian stable and testing, and I have so far resisted the urge to add any external crates at all (even regex) which means that you don’t need any other dependencies to use this tool.

I am actively using apticrate to work on some things but I have limited hobby time at the moment so it may be some time before I can make a full report on the efficacy of developing against the Debian subset of crates. It will be influenced heavily by what you’re working on. Early indications are that much is available and also much is not. If there was an easy way to “pass through” a specific crates.io crate in addition to the Debian packages then I would add it but supporting this looks messy right now. I think the spirit of the approach is to stick with what I’m given and if I can’t make that work, best to give up and go grovelling back to crates.io. In theory the situation should only get better as more Rust software gets added to Debian and the dependency base grows.

Rust without crates.io

Tue, 14 Nov 2023 21:00:00 +1100

Rust is a lovely programming language but I’ve never quite come to terms with crates.io, or any other of these language-specific repositories where everyone uploads and downloads code willy-nilly. I have several objections:

If crates.io goes down or access is otherwise disrupted then the Rust community will stop work. It is profoundly unresilient to have a single point of failure like this. Certainly some people will have vendored their deps and others will have a panamax mirror handy, but for most, Rust as we know it stops if this one particular web service goes down.
There is no mediation of any kind between when a new library/version is published and when it is consumed. You need only one author in your maybe-hundreds-of-dependencies tree to be hacked, coerced or in a malicious mood for you to have a really bad day.
Any tampering with crates.io itself (espionage, disgruntlement, national security) could have an incredibly wide blast radius, or a incredibly wide set of targets from which to choose.
Since crates.io is the source for crates, it is normal for both developers and CI machines to be hitting this web service all the time. Opportunities for mischief are exacerbated when clients are phoning home so frequently.

So what’s the alternative? I think we all need to take a step back from the altar of developer velocity and take a deep breath. I don’t want dependencies hot off the press. Ideally I want someone independent of the authors playing a curatorial role.

Now, actually getting some human review of dependency updates is quite a hard thing to do. cargo-crev has been trying for years to make this happen. I would love if it was the solution but it isn’t yet, and I think it’s a little ambitious. Yes we would like to have super-experienced software developers reviewing all our libraries with cryptographic stamps of approval, but if they’re not available we could be the target of remote shell in a build.rs. Surely there’s a middle ground here?

What’s interesting is that this problem is largely solved for C and C++: Linux distributions like Debian package such a wide range of libraries that for many things that you want to develop or install, you don’t need any third-party libraries at all. It’s just a matter of finding the right apt-get incantations and off you go. Even if you can get 95% of your libraries from a common trusted source then your risk is decreased considerably.

Rust libraries don’t work quite the same as C/C++ ones. Normal Rust code can’t be dynamically linked—a binary will have all of its dependencies statically linked at build time, so you won’t typically see .so files for Rust libraries that are going to be consumed by other Rust code. Since there is no .so file, Debian has no package that installs the library. However if they want to ship a binary that was written in Rust, their builders can’t just be downloading stuff from crates.io. They need a way to package all of the software that represents that Debian release. To solve that problem they’ve taken all these little dependencies and put their full Rust source code in packages with names like librust-cratename-dev.

Hmm, how many such packages? Running on trixie (testing)…

$ aptitude search librust- | grep -vE "^v " | wc -l
2336

This is starting to look like a serious curation of the most important Rust crates, available from any Debian mirror. There are some double-ups to be sure, since in some cases multiple incompatible versions of the same crate had to be packaged. Still. Maybe there is enough Rust in Debian now that it’s viable to write interesting Rust software independently of crates.io? That would solve basically all my concerns and the situation is only going to improve as more Rust software gets packaged.

To be clear, I don’t expect that Debian Developers are auditing these packages in the manner of cargo-crev. The good thing is that they don’t actually need to for it to be a major improvement.

A DD isn’t going to upload a new patch release just ‘cause. It’s going to be because it has an important fix or because some other program has depended on it. On crates.io a maintainer is free to create new releases for any reason and cargo update is not going to evaluate how good that reason is.
A simple time delay will allow egregious malware like malicious build.rs scripts to be caught, whether that’s the super-long Debian stable cycle or even the several days required to migrate from unstable to testing. I assume that an urgent security issue would be distributed the same as any other Debian update.
They might decide to give the diff at least a cursory look, which is better than nothing.

How do we do this? It’s actually quite easy because the big-brained Debian developers have arranged all the Rust dependencies to follow the format of a cargo Directory Source. That is, all of the packages are installed in their own directories under /usr/share/cargo/registry, including implementing a cheeky workaround for the required .cargo-checksum.json files.

You can then add some brief incantations to your .cargo/config.toml, whether on a project- or user-wide basis:

[net]
offline = true

[source]

[source.apt]
directory = "/usr/share/cargo/registry"

[source.crates-io]
replace-with = "apt"

This overrides the default crates.io source and ensures dependencies can only be fulfilled locally by installing the relevant packages. This happily doesn’t require any changes to your projects themselves—you just have to be careful to use versions in your Cargo.toml (and Cargo.lock) that are resolvable on Debian, since it is a subset of those available on the wider crates.io.

I am quite certain that Debian wouldn’t have enough coverage yet for the monorepo at work, but I gave this a go on my one of my little CLI projects that has half a dozen dependencies. Apart from having to downgrade copypasta from 0.8.2 to 0.8.1 in the Cargo.lock, this builds and runs just fine. What a treat.

This little investigation has given me much more confidence in using Rust generally into the future. I feared that the “grab any dependency version you like” approach facilitated by crates.io would render Rust impervious to any sort of curation effort, such that anyone who was serious about my earlier concerns would have to stick to a language used to the old ways like C++. Fortunately, Debian is here to prove me wrong. A+ work by their Rust packaging team.

All power to those who like to live on the edge; I’ll be over here trying to minimise different types of dependencies.

Autonomy #1 - Intro and Backups

Sat, 11 Nov 2023 20:00:00 +1100

Douglas Adams once made this amusing observation about how we view technology:

Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.

Anything that’s invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.

Anything invented after you’re thirty-five is against the natural order of things.

I happen to be thirty-five now so if I’ve developed a nagging feeling that computing took a wrong turn maybe fifteen years ago then it shouldn’t come as a huge surprise. So yes I fit the stereotype but I’m gonna go for it anyway, over a series of blog posts.

What is it, then, that changed since the noughties? Is it all nostalgia or is there something interesting there?

Modern computing lets many of us down in various ways. There’s the developer who can’t work when GitHub is down, the customer whose data is leaked in an open S3 bucket, the libertarian who worries about mass surveillance of their cloud data, the user who is baffled whenever Slack revamps its UI, the business owner who can’t accept payments when the internet goes down, the supply chain attack delivered via crates.io, and others like these.

In a nutshell this is what I think: the problem is that computer users are losing their autonomy, and the root cause is the increasing use of internet servers. Most of what I consider to be the problems with modern computing stem from the use of servers where they’re not absolutely necessary, and we can improve autonomy by reducing the number of servers or our reliance on them where possible.

In debates about internet servers today, a lot of emphasis is placed on who is running the servers, how they handle data, and what the business model is that enables those servers to be operating. I would claim that for computing autonomy more generally, these factors make only a limited difference. Replacing proprietary servers with open source ones or locked-in servers with interoperable ones is laudable but doesn’t get to the heart of the problem and maybe even perpetuates it. Why? As soon as you bring an internet server into the mix, a computer-literate layperson is unable to responsibly maintain it; not without granting complete trust to somebody else.

“Autonomy” means different things in different situations. I use this flexible word intentionally because it is the common link between a lot of unpleasant scenarios. Rather than try to explain everything all at once, what I intend to do is write about some modest changes I’ve been making as an individual to improve my computing autonomy and use those changes to explain what I mean. By framing the problem as internet servers I expect to find a common thread through many modern ills like those I mentioned above.

More about that later. Suffice to say, we can all do our bit as individuals and organisations to reduce our dependence on servers. Where do we start?

We must start by acknowledging one benefit of the cloud, as a constellation of internet servers—it serves as a backup of our data, and often the tech companies do a better job of not losing our data than we do as individuals. Phones get smashed, laptops get lost, but datacentres with RAID storage and 24/7 technicians go on. All computers are fallible so we need backups. If you use the cloud for backups, you need the cloud. Let’s do something about that.

What I have done, and what I recommend, is to choose one computer that is responsible for my data. I use my desktop PC where I invested in 4 TB of storage. My goal is to store all my data on this computer, then back up that computer thoroughly. Make no mistake—performing a thorough independent backup requires a little bit of ongoing work—but it is a very useful baseline. Since I have a backup regime that I understand and trust, any other server-bound data that I “liberate” onto my own hard drive will be automatically captured and protected.

It’s easy to over-think this kind of thing. I don’t strictly follow the 3-2-1 of backups. I have two backup USB hard drives, one on-site and one off-site, and I rotate them occasionally in case the building burns down. Because I’m a computer nerd I choose to create incremental snapshots on LUKS-encrypted volumes using rdiff-backup. Did your eyes glaze over reading the last sentence? Don’t stress. If you have a Mac, you’ll do just as well alternating between two Time Machine drives. When I was using Windows a while back I did something much simpler. I had around 200 GB of data and a 3 TB external. That external drive looks like this:

Each time I did a backup I created a new folder named after the current date and did a copy-paste of the relevant folders from my computer. For bonus points the drive was Bitlocker-encrypted, though even that was probably overkill. Each backup would take a couple of hours to complete but it didn’t matter—it was quick to kick off, and importantly it was extremely simple. Virtually everybody understands copying files on and off drives. You simply don’t need fancy backup software to do backups if you’re prepared to take a couple of minutes every week to make it happen.

Having two backup drives is good. If your computer fails you have two opportunities to get your data back. You can have three or more if you really want to. At no point have you granted a cloud provider the opportunity to show your files to intelligence agencies, scan it for copyrighted content, train AI models on it, or mess up their own backups. You have direct access to your own backup no matter what. This backup forms the bedrock of your autonomous computing experience. If your photos are in the cloud and you wish they weren’t—well, download them into a folder on your hard drive. Now they will be picked up in your backup strategy and you can be confident about it. If you want to sync your notes with your laptop—fine, just make sure they sync back to your computer where the backups happen and they will be safe.

Putting in place a good backup system is the most onerous and most important part of autonomous computing. Think of it as another small physical chore that we have to do regularly, like taking out the rubbish or watering the plants. Get it right, and we will not hesitate to bring other data under our own control.

You might wonder how this works if the data you’re trying to back up doesn’t take the form of neat files on disk, because it’s wrapped up in the internal format of some application. I work around this by backing up my entire home directory but it’s also helpful to change the way we operate so that data is stored in easily-accessible self-contained directories where possible. We’ll explore this further later.

Memories from old LAN parties

Sat, 09 Sep 2023 21:30:00 +1000

I participated in a lot of LANs in the 2000s, sometimes at private homes and sometimes at dedicated LAN venues. Many strange and wonderful things happened at these events that wouldn’t make a lick of sense today. Here is a collection of unrelated memories which may be altered to protect the guilty.

A man who runs a LAN facility will generally not give you a Windows 98 SE CD key, even if you need to reinstall Windows on your PC during an event.

A LAN isn’t legitimate unless at least one person has to reinstall Windows along the way.

Multi-round tournaments seem like a good idea but you will never get anybody into their allocated matches at the times they’re supposed to be there.

If you mod your ATX case so that the power button is gigantic and illuminated with a red LED, someone will come along while you’re gaming and wonder out loud “what does this button do?” while pressing it.

Having a PC with a window in the side of the case does in fact give you credibility.

If you take a 500 GB drive to a public LAN where the average drive size is 80 GB then it is possible that it will be stolen from your box while you’re taking a break go-karting.

If you use a crack that generates malicious traffic from your IP then you may be forced to reinstall Windows before you can reconnect to the network.

If you don’t have enough warez to reach the minimum share limit for the DC++ server you can always add the directories for your games installed under Program Files.

The desk you’re seated at may have a badge at the back telling you which static IP to use, but you can ignore that if the venue has upgraded to DHCP.

If you’re using a 10 Mbit hub and copy the same directory to two Windows SMB hosts simultaneously it is somehow smart enough to make the transfers coincide so it can transmit the same data to both at once. To this day I have no idea what heuristic it used but honest to god, it slowed down the earlier transfer and then went in lockstep file by file for the remainder.

If you throw an old motherboard onto the 11 kV power lines on a nearby pole it will toast some components but probably not cause a power outage.

Windows 98 SE requires that you reboot after changing your IP address settings.

Windows 2000 is quite good for gaming and has the added benefit that you will be prompted to press Ctrl-Alt-Del to logon, which feels corporate (or school-like) and cool.

If a pirated game comes with a crack.exe then you run the crack.exe. What could go wrong?

There were websites which aggregated downloads of keygens for various games. These worked more often than you would expect.

In an attempt to reduce piracy for multiplayer, game producers such as Blizzard let you install “spawn” copies of the software which could only participate in a multiplayer game and didn’t require the CD to be in the drive.

Alcohol 120% was the free tool of choice to emulate a CD drive for ISOs of games which required the CD to be inserted for copy protection, or rip a CD for that purpose. This only became popular once hard drives became big enough for people to spend many gigabytes imaging their CDs.

The cool kids had a Barton Athlon XP 2500+ and a Radeon 9600. The rich kids had a 9800.

Internet access wasn’t a thing until the later years. LANs which tried to provide internet had strict quotas or bandwidth limits and generally found it difficult to manage.

If you had the right ICQ number you could repeat it multiple times to generate a valid Starcraft CD key.

Sometimes LANs would get terrible sponsorship deals and nerds would fight over bounty like a CD of Visual C++ 6 Enterprise Edition.

A man who runs a LAN facility might sell you an RTL8139 PCI card for $15 if you want to upgrade to 100 Mbps. This card might work great for a decade+ after.

If you’re a poor high school student who wants to upgrade your home network from a hub to a switch, you could pay the lion’s share of the $55 for the right to keep it at your house between LANs.

WINE can basically only run Starcraft.

If someone has shared their installation of Warcraft III over SMB it will run much faster if you copy it to your local machine first rather than execute it directly from the network folder.

Motherboards had headers to connect USB ports built into the case but they were completely unstandardised so you had to slot them in pin-by-pin in the correct locations following your motherboard manual. If you got this wrong you could short out your PSU through thin wires by plugging in a USB mouse, which would create an awful whirring sound.

A commercial LAN venue would advertise itself on a local pop music radio station by having a dialogue of people arguing about which IP addresses to use.

If you had the oldest PC in your group, you might be connecting to the hub via coax and a BNC connector while everyone else is using CAT5.

What’s WiFi?

A few notes on Veilid

Tue, 15 Aug 2023 21:35:00 +1000

A few days ago cDc launched the Veilid framework at DEF CON. The basic idea is that it’s a distributed network of nodes working peer-to-peer to provide a storage and communication layer capable of supporting modern social applications, without central servers or blockchains. Normally an app built on Veilid will embed a fully-functioning node, such that even iOS and Android users become part of the Veilid network, helping with the routing and data management while the app is running. The embedded node is written in cross-platform Rust (with Flutter bindings) and there’s also a headless server binary that you can control with a JSON API. Probably the best overview right now is the slide deck.

I’ve been looking at the docs and code and there’s a lot of clever stuff. Safety Routes and Private Routes provide Tor-like onion routing which allow a request’s sender and receiver to individually opt in or out of obscuring their location in the network. Support is built in to help nodes who are stuck behind NAT, or communicate with other peers on LAN.

The most basic form of communication is the App Message and the App Call. These are a “statement” and a “request/response” respectively. You can target them at a specific peer, or if someone gave you a Private Route then you can funnel your message into that, so that they receive it without you ever knowing their real peer identity. This creates the fascinating possibility of sending/receiving messages and responding to requests without each side knowing who the other is.

Mostly though you aren’t supposed to think in terms of physical peers. One of Veilid’s main features is its DHT, into which you can publish records corresponding to one of two schema types. One schema is owned by a single key; whoever created the record can update its value again later. The other consists of multiple “members”, representing other users (public keys) who are allowed to set the value of their corresponding subkeys. This way you have a single DHT record that supports concurrent writing by different people.

When you set a DHT record in the network, a node will take responsibility for storing and serving it if their key is sufficiently “close” to that of the record. Least recently used data will be evicted, and of course nodes could go offline at any time, so Veilid creates some redundancy. Ultimately it’s up to the owner of the DHT record to reinsert it periodically if they want it to stick around.

While flexible, compared with your average embedded database these tools are somewhat primitive/unusual and the trick is to figure out how to write a full modern app on top of them. The as-yet-unreleased VeilidChat should offer a comprehensive example of how it’s done.

At this point, sadly, I’m losing enthusiasm. From this brief description you can see that every Veilid app is participating in the network and your device might become temporarily responsible for hosting a DHT record for something completely unrelated. These records can consist of subkey values up to 32 kB each with a total size limit of 1 MB, which is potentially a substantial document. In most sensible applications the record values will be opaque ciphertext so this would be pretty uncontroversial. However, you can of course insert a DHT record containing anything you like. I confirmed this suspicion on the Discord earlier.

<thombles> Am I following the code correctly when it seems there is no layer of encryption/hashing between the DHT key and the peer asked to store it? Or to put it more directly, if I choose to put a 32kB JPEG of Richard Stallman in a subkey and set it, at least one node in the network now has a picture of Richard Stallman in an sqlite database on their hard drive if they go to look for it?

<DilDog> Yes. Whether or not to encrypt the data and how is up to the key writers

<DilDog> Hmm

<DilDog> That said there is a device encryption key on the tabledb where storage keeps records.

<DilDog> And the key is In the protected store

<DilDog> So its not just hanging out there. But there’s not much we can do about this i dont think.

He’s quite right of course. And there’s no “encryption/hashing” trick available: if you’re hosting someone else’s ciphertext then it’s arbitrary binary so they can have you store whatever bytes they want.

It hard to see how Veilid could get around this. The designers needed some sort of persistence beyond when your device is online without relying on centralisation. You can’t just magic storage into existence. You have to borrow it from somebody. Thus you can send peers arbitrary data for (temporary) safekeeping/serving and the problem arises.

I’m being coy of course; I’m much less concerned about Richard Stallman than I am about JPEGs that would be illegal for me to possess. Personally I have a pretty low risk tolerance for those randomly existing on my PC or phone, even temporarily and in an encrypted form (since my device has the key). To some degree you accept this risk whenever you host an online service that accepts arbitrary data, but the risk feels elevated in the context of software that is explicitly designed to provide anonymous data sharing with onion routing.

I can’t square this circle. Communal storage networks are not going to be a solution for me. For all its flaws I’m happier with the federated approach: you know where your content is going to be stored and it will probably stay online for longer than on a DHT.

rdiff-backup's new interface

Sun, 09 Jul 2023 10:00:00 +1000

My favourite incremental backup tool is rdiff-backup, mostly because it’s so conceptually simple. When you back up a source directory to a destination, the destination directory becomes an exact replica. The only difference is it contains an additional subdirectory rdiff-backup-data, inside which it stashes all the metadata plus data pertaining to earlier increments in case you wanted to restore something earlier than the latest snapshot.

I find it profoundly reassuring to be able to see and interact with the backup copy of the files. If I need to restore from the latest backup I don’t need any special software—it’s just copying files. It’s old, stable, and packaged everywhere. A+.

I feed it my entire homedir, which means there are often 10s to 100s of thousands of files changed in each increment, mostly inconsequential. rdiff-backup churns through them all in short order so I don’t waste any time worrying about it.

Perhaps its weakest point has been the convoluted CLI syntax with lots of long-format options that weren’t so easy to work out. Happily it has recently had a revamp and the CLI started warning me that I should switch to using the new syntax.

My backup script is quite simple and I only needed to make a handful of changes. Much tidier!

# Back up directory - old vs new
$ rdiff-backup --print-statistics $srcdir $destdir
$ rdiff-backup backup --print-statistics $srcdir $destdir

# Remove old increments, allowing deleting more than one at once - old vs new
$ rdiff-backup --remove-older-than 52W --force $destdir
$ rdiff-backup --force remove increments --older-than 52W $destdir

# List all increments including cumulative size to keep them - old vs new
$ rdiff-backup --list-increment-sizes $destdir
$ rdiff-backup list increments --size $destdir

If I was Meta and wanted to make fedi implode

Fri, 07 Jul 2023 12:30:00 +1000

Step 1

Implement ActivityPub on threads.net. Despite carrying Facebook’s reputation, get federated by relying on a couple of groups: fediverse enthusiasts greedy for growth, and those who are indifferent because they think it will be possible to back out later if it goes wrong.

Step 2

Get people who are popular on Threads well established with followers on the outside network. This one-way action will flood the fediverse with lots of people and content that were previously unavailable. Anyone who was opposed will look kind of silly in the face of this bounty. Any future attempts by outsiders to defederate will be met with internal pushback “but I’ll lose access to X, Y and Z!” The infighting would sap volunteer energy. Inconsistent federation rules would lead to server-hopping and a drain of users to pro-Threads servers, or Threads itself.

Step 3

Begin complaining of major problems with spam, trolls, sockpuppets and sybils. The fact that ActivityPub can originate from any domain with any number of users means that you have to rely on reputation and servers’ own moderation policies, which are highly variable. This is entirely incompatible with Meta’s commercial mode of operation where it has to deliver a certain kind of service so as not to spook the customers (that is, the advertisers). Even though these problems are probably occurring, it is easy to fabricate traffic if Threads isn’t suffering the levels of spam required to garner sympathy.

Step 4

Require that each individual user who wants their content to federate with Threads accepts a terms of service by sending a message to a specific account. The code will be obtained by visiting a Meta website where you enter your email and agree to the acceptable use policy, understanding that you may be blocked later if you fail to comply. For bonus points, make this as low-friction as possible, not even requiring a Facebook account.

Step 5

Block federation with all users who have not accepted the terms of service. The majority, who want their fediverse experience to continue as it did before, will take the individually-convenient path of signing up with the terms. Users who are blocked will have the frustration of being shadowbanned in conversations when viewed within Threads, adding to the pressure to agree. This will lead to a widespread perception that the non-Threads experience is “broken”. Some servers will defederate despite the warnings of step 2. The fediverse will be irrevocably split between a vassal of Meta, who can claim to regulators that they are interoperable and therefore not anticompetitive, and a handful of nerds who will slouch off to build anew without the others.

Querying UniFi channel utilisation

Mon, 03 Jul 2023 06:30:00 +1000

I recently needed to do some optimisation work on a Ubiquiti UniFi setup. Some workloads were causing the capacity of a single AP to be exceeded, or more precisely, the channel was getting so fully occupied that connections were beginning to drop. I was trying to evaluate whether a particular configuration would fix the problem and it was slightly annoying. The UniFi web console does report channel utilisation but it makes it hard to see the actual numbers (mouse hover) and it updates extremely lackadaisically. I wanted hard data, dangit.

Happily for me, this information can be had with a little effort. First we must enable SSH access to the AP under site options. This lets me shell in and run commands directly on the device.

The AP has a number of custom utilities starting with the prefix mca-. The one that’s relevant to us today is called mca-dump. This outputs an enormous blob of JSON containing all the metrics for the device including all the radios and connected stations, in the order of 600 kB in my case.

After combing through this data it was clear that each WiFi radio had its own block of statistics, identifiable by the channel information. I was interested in the 5 GHz channel which turned out to be keyed under the interface name wifi1. Harder to find were the utilisation statistics; they use the acronym cu which makes sense in hindsight but didn’t stick out while reading the JSON.

Putting this together, we can process it with a tool like jq to get the information we want.

$ ssh $AP mca-dump | jq '.radio_table[] | select(.name == "wifi1") | .athstats | {cu_self_rx,cu_self_tx,cu_total}'
{
  "cu_self_rx": 0,
  "cu_self_tx": 1,
  "cu_total": 1
}

Although this data is not documented anywhere that I could find, my understanding is that these values represent the percentage of time the AP spends receiving and transmitting respectively, plus the total utilisation of the channel including interference.

Book review: Stolen Focus by Johann Hari

Thu, 08 Jun 2023 19:55:00 +1000

I recently read Stolen Focus after it was recommended via the local LUG’s chatroom. I’ve been exploring this theme recently with Cal Newport’s Deep Work and Digital Minimalism, along with Jon Haidt’s research into the alarming (alleged) impacts of smartphones and social media on youth mental health. It’s fair to say I came into Hari’s more recent book with the attitude “it’s gotta be the phones, man”.

It really is a fantastically researched piece. Stolen Focus is chockers with choice quotes from attention-adjacent researchers that Hari clearly went to enormous effort to locate and interview. By comparison, Newport writes well but his anecdotal style feels like an extended blog post. The book has a very wide remit; it tries to distance itself from self-help titles by looking at broader societal problems that affect our ability to concentrate, many of which you certainly can’t self-help your way out of.

He hooks you from the start with a story about his godson and another about his own three-month digital detox. Throughout the first half his reflections on the detox are cleverly woven into the insights from his research. I nodded along at his familiar observations, raised eyebrows at his notes about sleep, and grimaced in sympathy with his candid recollections of addictive behaviour. It’s simply good writing.

However there is an exception to my enthusiasm. Chapters 7 “The Rise of Cruel Optimism” and 8 “The First Glimpses of the Deeper Solution” address the question of what to do about the phones, and I’m sorry to report that it goes completely off the rails. This interlude is full of non sequiturs and strange analogies that probably scan okay if you’re disposed to the same conclusions as Hari, but if you needed convincing then it falls well short. Just as abruptly as these chapters arrive, Hari switches back to sharing his research and everything is pretty reasonable for the second half. It’s a wild ride.

To explain what I mean, this is the basic overall narrative:

Why is everyone looking at their dang phones all the time?
Oh I guess that’s me too. Let’s try a long detox and see what it’s like.
That was amazing but it didn’t stick. Am I a bad person?
We can relax everyone! Our phone addiction was Big Tech’s fault all along!
We need to regulate or nationalise Big Tech urgently.
Also we need a UBI, 4-day work week, reduced pollution, improved psychiatric care, more sleep, free range kids, and if we stop pursuing infinite economic growth maybe we’ll be able to think clearly enough to fix global warming.

Yeah so it gets a bit grandiose towards the end but it’s pleasantly written so I’ll allow it.

The first real sin of Stolen Focus is that it discusses lots of interesting phenomena without making any effort to rank these by their level of impact or by how much agency we have to do something about them. He attempts to rouse us with inspiring stories of feminism and the labour movement but frankly it doesn’t work. It is completely disspiriting and disempowering to blame so many different causes that are intractable in their own ways, then cheerfully encourage us to take on big problems in the manner of Greenpeace. He ends with an example of people getting arrested on purpose. I suppose it’s inspirational but it feels like at some point we’ve become unmoored from what started as a normal conversation about people spending too much time on Twitter.

For my money: it’s gotta be the phones, man. There are a lot of problems in the world but when I look at my own behaviour and that of people around me, it’s the tech and social media that’s shattering most of the attention while also being one of the things most under our control. Hari does give individualistic measures occasional credit but it’s clearly not his preferred stance. He revealed that after his three months offline, some four months later he was back on his phone four hours a day. If that’s your outcome then I think you’ve got to accept a little bit of personal responsibility. I’m quite sure Hari read Digital Minimalism—if he’d followed the detox instructions there then maybe he would have had the same success as many of Newport’s followers.

The book’s second sin is its analysis of what to do about phones and social media is completely underwhelming. I disagree with what the problem is, disagree with the rhetoric employed, and disagree that the solutions proposed will actually help. My impression is that Hari spent a lot of time with his revered ex-Big Tech whistleblowers, who in their rush to mea culpa have somewhat overestimated their own depravity. The two chapters feel like they were back-calculated from the overall vibe of the book: that the general solution to improving focus is lobbying the government.

If you follow along in this section it would seem that the world consists of two kinds of people:

Hapless well-meaning citizens who have been entranced by evil Big Tech and need the government to step in to help;
A handful of privileged libertarians who uniquely have the time and energy to switch off evil Big Tech’s intrusions.

This quote captures the sentiment well:

One day Aza [Raskin] said to me: “The fundamental thing is that no one likes the way they are spending time or making decisions with the way technology currently is. …”

This is plainly bollocks. Many people enjoy using technology the way it is. And who are you or I to say that they’re wrong? If you start with this sort of assumption then you end up saying one wrong thing after another and I think that’s what’s happened here.

I have a particular circle of friends—happy and well-adjusted, who catch up face-to-face all the time—who are also consummate lovers of social media. When the new app BeReal appeared most of them leapt at the opportunity to have additional notifications and surveillance capitalism on their phones, simply because it was a fun thing to do with their friends¹. They don’t need help or fixing, least of all from me or the Australian government.

If an adult spends four hours a night watching YouTube and following its recommendations, who am I to say that this is bad²? It would be elitist on two fronts—firstly by implying that there’s something objectively “better” this person should be doing with their time, secondly by restricting what may be one the few forms of entertainment available for a poor person who hasn’t much else except a smartphone and some data.

As a second example I present Mastodon³ which is a social network technology inspired directly by Twitter. Yes, Twitter, one of the worst attention sinks so far invented: a 24/7 party of dopamine, outrage and virtue signalling, with a demonstrated ability to turn its users’ brains into glue⁴. Open source hackers looked at Twitter and decided that its biggest problems were the central servers run by capitalists and its susceptibility to overbearing regulation, while accepting the overall idea as basically okay.

To be fair to Mastodon it has some tweaks that turn down the heat: content warnings, no quote tweets, and longer posts by default. However if you spend much time on the network you quickly find the same posturing, subtooting and dunking, plus the uniquely Mastodonian drama of which instances are blocking or muting which. If Twitter is the “hellsite”, Mastodon is summer in Vegas⁵.

So suppose the US government compulsarily acquires Twitter, bans profile-based advertising so you have to pay to use these services, and dictates the way they can show content. What do you suppose happens next? Will the Twitterati be relieved that finally they don’t get notifications about inane things? Of course not. Either they get angry or they jump ship to Mastodon, probably to a free-of-charge community server where they can burn their attention without limits.

What if we tried to regulate open source software too? I’m not going to explore that here but I don’t think this would go very well. Suffice to say it’s not just Big Tech and not just surveillance capitalism that is grabbing our attention. Even if you target the big companies with red tape there are plenty of of other options to get your dopamine hits, and there are plenty of people who like it this way.

Hari muddies the waters by talking about how YouTube’s recommendation engine promotes radicalising content, and how Facebook allows neo-Nazi groups to associate openly, or even how they would like to track your eyeballs using cameras. This stuff is important but in a discussion about focus it’s only tangentially relevant. Sure, controversial content may be a prime choice to help grow engagement but it’s not like they can’t shatter your attention with more wholesome alternatives. Just look at TikTok. In the context of this chapter, Hari’s simply throwing dirt to make the punitive measure of heavy regulation sound good.

He spends a great deal of time discussing “cruel optimism”, a real concept, but using a completely inappropriate analogy. The idea is that if you’re suffering from stress due to work expectations, or lack of health insurance, or all-consuming family duties, someone might suggest you do some mindfulness and meditation without helping to address any of the actual stressors. If it doesn’t work, maybe it’s your own fault because you didn’t mindful hard enough. A pretty crappy concept, right?

These situations are completely incomparable to what we’re actually talking about. In chapter 7 Hari is interviewing Nir Eyal, a technologist who advocates for self control.

“Two-thirds of people with a smartphone never change their notification settings. What? Right? This is not hard stuff. We just need to do this kind of stuff.”

“For God’s sake, push the fucking button that says ‘do not disturb’ for an hour if you’re going to have a meeting with your colleagues. Is that so difficult?”

Hari introduces the argument via Eyal while also trying to disprove it. The main argument he puts forward is that Eyal’s solution is “cruel optimism”—but I see very little resemblance between apps on your phone that fight for your attention and serious personal situations like inability to pay for medical costs or being expected to do unpaid overtime responding to emails. These are so far apart it borders on offensive to people who are actually in dire straits.

Like Eyal says, we can push the fucking button. Sure, there’s an addictive aspect, but we must keep in mind that if we let tech have access to our minds then we did so through our own actions and we have agency to limit or remove that access, every single day. This is very different from any kind of stress pushed upon you from outside, or Jaron Lanier comparing technology with lead paint which is a universal health risk you’re subjected to without choice.

The rhetorical device that Hari has used is that Eyal isn’t a trustworthy source to solve our focus woes—he was part of a class at Stanford that studied persuasive technologies and he previously wrote a book about how to write software to hook users. Therefore anything he says is clearly hypocritical. Why not interview any of the other millions of people who feel quite in control of their smartphone? Again, that would distract from the book’s overall goal of lobbying the government to regulate big tech.

I want to wheel out a third anecdotal example. Often tech is not even meant to be addictive but it turns out to be anyway. A few years ago I had an Apple Watch. When I first set it up it displayed all the notifications from my phone. It was ridiculous to be looking at my wrist all the time so I quickly turned that off. What remained on the watch was my fitness tracking: how many kilojoules I’d burned during the day with my physical activity.

I looked at this damn watch compulsively. It was fun to press the button and make the screen light up and see what the fitness numbers were. Well, not really fun. But it was a shiny distraction that kept drawing my attention away from whatever actually mattered and it wasn’t getting better after 4 or 5 months. Does Apple engineer it to be addictive to play with? Probably—but I really can’t fault them for whatever they did. It did exactly what I asked it to do. And I was addicted to tapping on it to look at a number all the same, so I got rid of it.

Stolen Focus doesn’t spend any time engaging with anti-tech sentiment. The logic is that we could have our smartphones and everything would be good if everyone would just design their software to be better. As my Apple Watch demonstrates that’s just not true. If you’re getting any sort of message over a wireless network, either you have notifications or you have intermittent rewards. It’s an intrinsic property of how we have to interact with communications technology. This creates addictive or distracting elements, no matter how boring the software actually is.

Someone who decides (on a permanent basis) not to have a smartphone has avoided the problem quite comprehensively. Yes they might lose some convenience, but rearchitecting the global economy is also going to entail some loss of convenience so maybe it deserves some thought? I wonder if these kinds of people didn’t show up in the book just because they’re so frustratingly smug. They’re a walking demonstration that individual control is actually possible, and it wouldn’t help the argument that the government needs to own Facebook.

The idea that smartphones were basically a mistake isn’t a mainstream one, and not one that I actually expected Hari to reach for. However at the end of the book he starts talking about stopping infinite growth of the economy, which should raise serious questions about how much smartphone technology we humans should be making anyway. To be fair there isn’t really space for this discussion in Stolen Focus. Perhaps we can look forward to a sequel called Focus During Degrowth or something like that?

So if I’m so smart, what would I do about the damn phones? In short, I think the problem will eventually take care of itself. If you’re worried, this is what I reckon we should do:

Research and document the hell out of technological harm, specifically what it does to our attention and mental health.
Take all available individual measures to engage with technology with intent and to support the things in our lives that matter.
Change the culture.

I don’t think we’re far away from a substantial hipster culture that eschews all digital technology. Of course, hipsters aren’t the mainstream by definition but there’s enough cultural exchange that within a few years I expect it to become uncool in many quarters to take out your phone in a social situation, just like it was when mobile phones were new.

The evidence is also accumulating that smartphones and social media are harming our kids. If there’s anything in this world that will provoke a reaction, it’s a health risk to kids. Increasing numbers of schools and towns are agreeing not to let their kids have smartphones until a particular age and I think this trend will accelerate. This process is nothing special—it’s society catching up with scientific research.

As adults we should be very aware of our relationship with our hardware, software and services and actually think about how much time we spend, just like any other potentially addictive activity that we have to navigate when living in a liberal democracy.

It’s a good book. It’s a big problem. But let’s not spend taxpayer money on Facebook, for goodness’ sake.

To their credit they made a half-hearted attempt to convince me to join, knowing that I wouldn’t. I looked up how it worked on Wikipedia and politely declined. ↩
I would readily apply different rules to children. ↩
I’m aware this is properly called the fediverse, along with the identica, GNU social, pleroma, etc. ↩
I offer no sources for this observation but it has been sad to watch the Twitter Mindset take hold in people I care about. The bile and tribal thinking seeps in deep, then begins to show up in different social contexts. Yes, on both the political left and right. ↩
If it sounds like I’m projecting, yes indeedy. I may have little love for social media at the moment but I’ve written plenty of favourable things about fedi in the past. ↩

To block_on or await?

Thu, 13 Apr 2023 22:15:00 +1000

A problem

I recently faced a conundrum in some Rust at the boundary between sync and async code. The library in question uses a persistent tokio executor internally and you interact with it via a C-style API. Since there is no such thing as an async C function these entry points to the library are all implemented as sync Rust. If they need to do something async on the internal runtime before they return, they execute a future using the runtime’s block_on method then wait for the result by blocking the current thread.

So far there is no issue in what I’ve described. Provided the caller expects the functions to block for as long as they do, everything is working as intended.

This library also does something else. You can register callbacks such that it will spontaneously call some function pointer that you provided when certain events occur. These callbacks have be delivered on somebody’s thread and in this case the library will use one of its own. Since the triggers for these callbacks are often async I/O or timers it’s often the case that it’s a tokio worker thread that ends up invoking the callback. This is allowed because the C callback is a sync function, and async functions can call sync functions.

In isolation these two aspects are valid but when you combine them a problem arises.

A callback is triggered, which the library delivers on a tokio worker thread by calling the user’s function.
Inside the callback, the user code calls back into the library in direct response to the event.
This “tromboned” library function tries to use block_on.
The app panics because you can’t use block_on in the context of a tokio runtime.

This is a tricky situation. The function could be called from either a sync or async context. Worse, you might not even notice these combinations of callback-and-call-in unless you tried to do something specific. It’s just a crash lurking in the dark for someone to bump into it.

A wrong turn

Like others before me, my first thought was this: what if these functions could detect whether they’re in the tokio runtime and convert the block_on call to something that “awaits”? I put “awaits” in quotes because the basic idea is clearly impossible—a sync function like this cannot .await on anything unless it was compiled as a future in the first place. However if we take a step back from reality you could imagine some sort of mechanism where the thread talks to the runtime and between them they agree to park this particular future for a while until some other future completes; a dynamically-added await point if you will.

It was at this point I found an old discussion on the Rust users forum, “Accessing the current tokio runtime to call block_on”. In this exciting thread, Alice Rhyl of tokio states unequivocally that you shouldn’t try to do this, and 2e71828 provides a brilliant/slightly cursed workaround involving crossbeam. I’m going to reproduce a slightly-fixed version here for the sake of discussion:

let future_result = match tokio::runtime::Handle::try_current() {
    Err(_) => tokio::runtime::new().block_on( the_future ),
    Ok(handle) => {
        crossbeam::scope(move |scope| {
            scope.spawn(move || {
                handle.block_on( the_future )
            })
        })
    }
};

Alice said:

In principle, the crossbeam::scope thing will “work” […]

How would it work then? Well first we try to get the Handle for the current async context. If it fails (because we’re on a regular sync thread) then we spin up a whole new tokio Runtime to execute our future, which is overkill but allowed. If getting the handle succeeds, we know we can use Handle::block_on to run a future to completion, but only if we’re starting from a sync thread. So we use crossbeam to spawn a separate thread from which we can safely make that call. (I guess using a scoped thread means the future doesn’t have to be 'static?) Kind of genius but not very efficient in either situation.

As far as I know there’s no “good” way to do this because Rust’s async model doesn’t allow you to yield from a future inbetween await points. It just doesn’t make sense. Even so, suppose I was especially bored and I wanted to hack support for this into tokio in some way. What could I feasibly do?

2e71828’s solution points to one approach: even if we block the tokio worker (which is usually verboten) you can kind of get away with it if you spin up an additional thread to compensate. Creating a new thread for every function is pretty inefficient though. If you had some sort of pool of them… wait, this sounds familiar—ah yes, tokio’s blocking threads. I propose that you could have a separate threadpool that expands up to the same size as the number of regular workers. Then you could have handle.block_on_or_await(future) which acquires a thread from that pool and uses it to poll the future, while allowing the original thread, which has a sync function on top of its stack, to sleep until woken with the result. By reusing these threads you can save the cost of creating them repeatedly.

What if we didn’t want to create extra threads though? We don’t want to starve the executor by taking the current worker thread out of service. The solution: spawn the inner future, then while waiting for its result, take some other future that’s ready to poll from the executor and do that on top of the stack of the currently-polled future. Then we can check if the result came through inbetween polls of other futures, allowing us to reduce the stack by a layer. Normally spawned futures have to be 'static… I feel like you could unsafely assert that references are stable in this case since it’s buried in the stack of another thread but I would leave that to decision to somebody who knows things about unsafe.

With either approach, I don’t like my chances of getting that PR merged.

A little common sense

In reality the problem with the design is at the opposite end—invoking the callbacks. If there’s any possibility that a sync function will block for a significant period of time then it should not be called directly from a worker thread since it could starve the executor. The user callback is a perfect example of an unpredictable sync function. They might not allow it to return for ten seconds and that’s my problem to deal with.

Therefore the solution is to ensure that all callbacks are delivered on non-worker threads, either via tokio’s spawn_blocking API or some other std::threads created for the purpose. This solves the original problem neatly: if the user never has the opportunity to run their own code on threads belonging to the internal tokio runtime then they can always safely call a function that performs block_on. Unfortunately this isn’t the sort of problem that the compiler can catch but it’s something you can look out for in review.

If you want an additional incentive not to expose your workers consider this hair-raising scenario: if the user code operating the C API is also Rust, it’s possible for them to call tokio::spawn inside the sync function (using their version of the tokio library) to inject a future into your runtime (using your version of the tokio library). This often works because thread local storage is great like that, and again it’s not something where the compiler will help. A surprisingly easy mistake to make if both sides are very async code.