A few notes on Veilid

A few days ago cDc launched the Veilid framework at DEF CON. The basic idea is that it’s a distributed network of nodes working peer-to-peer to provide a storage and communication layer capable of supporting modern social applications, without central servers or blockchains. Normally an app built on Veilid will embed a fully-functioning node, such that even iOS and Android users become part of the Veilid network, helping with the routing and data management while the app is running. The embedded node is written in cross-platform Rust (with Flutter bindings) and there’s also a headless server binary that you can control with a JSON API. Probably the best overview right now is the slide deck.

I’ve been looking at the docs and code and there’s a lot of clever stuff. Safety Routes and Private Routes provide Tor-like onion routing which allow a request’s sender and receiver to individually opt in or out of obscuring their location in the network. Support is built in to help nodes who are stuck behind NAT, or communicate with other peers on LAN.

The most basic form of communication is the App Message and the App Call. These are a “statement” and a “request/response” respectively. You can target them at a specific peer, or if someone gave you a Private Route then you can funnel your message into that, so that they receive it without you ever knowing their real peer identity. This creates the fascinating possibility of sending/receiving messages and responding to requests without each side knowing who the other is.

Mostly though you aren’t supposed to think in terms of physical peers. One of Veilid’s main features is its DHT, into which you can publish records corresponding to one of two schema types. One schema is owned by a single key; whoever created the record can update its value again later. The other consists of multiple “members”, representing other users (public keys) who are allowed to set the value of their corresponding subkeys. This way you have a single DHT record that supports concurrent writing by different people.

When you set a DHT record in the network, a node will take responsibility for storing and serving it if their key is sufficiently “close” to that of the record. Least recently used data will be evicted, and of course nodes could go offline at any time, so Veilid creates some redundancy. Ultimately it’s up to the owner of the DHT record to reinsert it periodically if they want it to stick around.

While flexible, compared with your average embedded database these tools are somewhat primitive/unusual and the trick is to figure out how to write a full modern app on top of them. The as-yet-unreleased VeilidChat should offer a comprehensive example of how it’s done.

At this point, sadly, I’m losing enthusiasm. From this brief description you can see that every Veilid app is participating in the network and your device might become temporarily responsible for hosting a DHT record for something completely unrelated. These records can consist of subkey values up to 32 kB each with a total size limit of 1 MB, which is potentially a substantial document. In most sensible applications the record values will be opaque ciphertext so this would be pretty uncontroversial. However, you can of course insert a DHT record containing anything you like. I confirmed this suspicion on the Discord earlier.

<thombles> Am I following the code correctly when it seems there is no layer of encryption/hashing between the DHT key and the peer asked to store it? Or to put it more directly, if I choose to put a 32kB JPEG of Richard Stallman in a subkey and set it, at least one node in the network now has a picture of Richard Stallman in an sqlite database on their hard drive if they go to look for it?

<DilDog> Yes. Whether or not to encrypt the data and how is up to the key writers

<DilDog> Hmm

<DilDog> That said there is a device encryption key on the tabledb where storage keeps records.

<DilDog> And the key is In the protected store

<DilDog> So its not just hanging out there. But there’s not much we can do about this i dont think.

He’s quite right of course. And there’s no “encryption/hashing” trick available: if you’re hosting someone else’s ciphertext then it’s arbitrary binary so they can have you store whatever bytes they want.

It hard to see how Veilid could get around this. The designers needed some sort of persistence beyond when your device is online without relying on centralisation. You can’t just magic storage into existence. You have to borrow it from somebody. Thus you can send peers arbitrary data for (temporary) safekeeping/serving and the problem arises.

I’m being coy of course; I’m much less concerned about Richard Stallman than I am about JPEGs that would be illegal for me to possess. Personally I have a pretty low risk tolerance for those randomly existing on my PC or phone, even temporarily and in an encrypted form (since my device has the key). To some degree you accept this risk whenever you host an online service that accepts arbitrary data, but the risk feels elevated in the context of software that is explicitly designed to provide anonymous data sharing with onion routing.

I can’t square this circle. Communal storage networks are not going to be a solution for me. For all its flaws I’m happier with the federated approach: you know where your content is going to be stored and it will probably stay online for longer than on a DHT.