From GNU social to Mastodon
There’s been another surge of Twitter users moving to Mastodon in the last few days and for the first time it feels like it’s getting some mainstream traction. It’s an interesting time—the fediverse has never operated at this scale before. Many are hoping it could fill the role of Twitter 2.0 but that’s still very uncertain. As a community we’re going to suck it and see.
If you’ll indulge me in some navel-gazing (this is my blog after all) I want to jot down some notes about what I’ve seen in the fediverse in the last few years. The purpose isn’t to be some hipster who liked federation before it was cool. In fact, there were significant problems that were fixed recently and now we have more clients than we know what to do with. (Push notifications, woohoo!) This also isn’t intended as a complete or balanced history. If you’re interested in the lineage of the software there are better articles.
I simply want to write down some anecdotes from my perspective before I forget them. It’s been a hectic little chapter of computing history.
Note: Although I use the past tense to describe software I used before switching to Mastodon, most of it hasn’t gone anywhere. It’s still actively used today.
gnusocial.no running the default GNU social theme
I jumped into the story in August 2015. At this point GNU social and its predecessors had already been around for years. I found out about it because a friend was setting up a server for the local linux users group. By my estimate there were thousands of active users and the community was diverse enough that they discussed all kinds of topics. The first test of any new communication software is whether it’s used to talk about anything other than itself. They’d long since achieved that.
Mastodon wasn’t going to exist for another year but we already had the federated universe, or fediverse. This was the word used to describe the different types of software on different servers that all spoke the same protocol. You could make an account on one server then follow and communicate with people on any other server.
GNU social was the most popular, maintained by Mikael Nordfeldth (mmn). It was designed to be accessed through your browser and also had a basic HTTP API. The interface was spartan but functional. There was no AJAX so you needed to refresh the page or click a link to see any new notices. It did support posts longer than 140 characters, image attachments, hashtags and groups.
This was not the state of the art, however. Hannes Mannerheim (hannes2peer) had created a plugin for GNU social called Qvitter. This let users switch on an AJAX-enabled web interface that looked very similar to how Twitter did at the time. You could keep the tab open and it would let you know when you had new notifications or quips to read1. It was even responsive—I used it on mobile with varying degrees of success. Some GNU social instances had the plugin; some didn’t. He and mmn worked together to fix issues spanning their respective codebases.
quitter.is homepage running Qvitter
By far the biggest server was quitter.se with over ten thousand registered users. This one was run by Hannes himself and it always had the latest version of Qvitter. This set the stage for a very left-wing community: Qvitter servers proudly described themselves as “a federation of microbloggers who care about ethics and solidarity and want to quit the centralised capitalist services.” Any right-wing agitators were promptly ejected.
There was a pattern in the naming of the popular servers of the time—the likes of quitter.se and quitter.is ran Qvitter. gnusocial.de, gnusocial.no and so on tended not to. loadaverage.org was another popular server that didn’t run Qvitter but had its own distinctive theme. As with Mastodon there were many small servers run by individuals and organisations. GNU social is distributed under the GNU AGPL licence, so anyone who sets up a server needs to share their own modifications.
The multitude of servers was often poorly understood by newcomers—they thought the web service was called “Quitter” and that they needed to make an account on each “Quitter” to follow their friends.
If you were the sort of person who liked the federation concept and enjoyed talking to strangers on the Internet, especially about free software, this is was an interesting place to hang out.
To be clear, GNU social, Qvitter and some of these servers still exist, maintained to varying degrees. quitter.se shut down recently after being overloaded by the influx of federated Mastodon users.
New users have been coming to the fediverse in waves. There’s always been a background level of organic growth but most of the action happens when Twitter does something stupid. Twitter does stupid things on a fairly regular schedule so this constitutes the bulk of the growth. People search for Twitter alternatives and sign up. Many of them get bored and leave again. Some stay, and the process continues.
I saw my first big wave in February 2016, about six months after I joined up. Buzzfeed published an article suggesting that Twitter was going to display tweets out of chronological order for the first time. As we know, that definitely happened. There was a trending hashtag #RIPTwitter and a number of potential refugees showed up on quitter.se and even started creating their own instances.
I described that event in a blog post at the time. Looking back on it I clearly thought that “this could be it for the fediverse!”. Obviously, in hindsight, it was not “it” for the fediverse. We still didn’t have Mastodon and it just wasn’t impressive enough to capture significant numbers of users.
It was then that shitposter.club came into existence, who wound up being somewhat notorious. They picked a lot of arguments with the early Mastodon crowd and Eugen in particular, causing some early adopters to wonder why they bothered interoperating with GNU social at all. Thankfully a complete schism was averted and as far as I know they’re still going strong. They’re the only well-established counter-culture GNU social instance that I knew about. At the same time they were fierce proponents of federation. With some exceptions they weren’t actively malicious, which forced admins to start thinking about how they wanted to handle content that wasn’t obviously spam or abuse but they may not want to have on their own federated timelines.
After watching all this play out I formed a theory about how the fediverse was going to expand. What I thought would happen was that groups of people with various strongly held views (politics, etc.) would band together in their own instances. There would probably be some sniping between them but on the whole we would be happier than on Twitter, since we would self-select into appropriate servers and mostly hang out with people who share our views.
It hasn’t really happened that way. I think there’s a fairly uniform lefty vibe across most of Mastodon—certainly no opposing servers at war with each other like I suspected would happen. I’m still trying to understand why many types of people I’ve seen on Twitter are staying away from the technology. Maybe some of them are the same people but acting differently?
!grow group hosted on social.umeahackerspace.se
There was a cool feature in GNU social called groups that has never existed on Twitter or on Mastodon. A user can register a group on their server—for example there was one about gardening called “!grow”. If you subscribe to the group and then put “!grow” in one of your messages, all the other subscribers see it in their home timeline. It’s similar to subscribing to a hashtag except the group’s creator can moderate members.
You could create groups for any topic. It was a great way to meet new people from across the fediverse who shared the same interests as you.
Groups addressed a problem fundamental to hashtags in the federation—a problem that still exists in Mastodon. When you browse a hashtag you only see posts that your server knows about. By comparison, groups are owned by a particular server that is responsible for sending notifications to all the subscribers. Everybody gets to see all messages.
There are some practical problems. If a server goes offline or is shut down, any groups it was hosting go silent and subscribers are left hanging. There is no mechanism in GNU social for moving or sharing groups between servers. Different groups on different servers can use the same tag names and this gets confusing.
It’s kind of fascinating to me that this feature wound up in GNU social even though they didn’t have solutions for long term management. It enabled some cool interactions. It caused a bunch of consternation when things went wrong. Users bumbled their way through.
Is it a good design? Eh, probably not. I imagine that’s why they’re not supported in Mastodon.
LÆMEUR’s fediverse conversation assembler tool (link below)
One of the recurring difficulties in GNU social was that the server did not retrieve the full context of conversations from other servers. Suppose users A and B are on another server and they’re talking to each other. You only follow A so you see half of the conversation—but if there is someone else on your server who follows B you can click on it and see the whole thing. It’s complicated—what you see depends on what other people on your server do!
This led to interesting workarounds. One was to click through to the server of one of the participants and view the conversation there. This is kind of tedious, and you’re still unable to reply to a message that only exists on a remote server.
Adam Moore (LÆMEUR) wrote a webpage that would attempt to assemble an entire conversation given the URL of one of the messages. Others built “follow bots” that would try to follow everyone so that as much of the fediverse as possible was on their server.
It was an annoying problem but happily this is no longer an issue in Mastodon. I haven’t reviewed how it works but I’ve never had any trouble viewing entire conversations directly from my own server.
I worked on a couple of my own projects to improve the GNU social ecosystem. One of these was successful. The other was not.
The successful one was some HTML documentation: The Unofficial GNU Social Manuals. At the time there were a few webpages here and there but no complete introduction for new users. I set about writing one. It’s not super detailed but it attempts to cover the basics of federation. I had some assistance from LUG friends and then a user named drymer translated the whole thing into Spanish (!). I still see a steady trickle of activity in my web server logs.
Working logo for Flyweight
The failed project was an open source native client for iOS, which I named Flyweight. I wanted to help improve adoption of GNU social. My theory was firstly that we needed higher quality user interfaces, and secondly that we needed to reach out to people who weren’t free software nerds, including iPhone users. When you’re surrounded by a lot of free software nerds both of these ideas are controversial. “Why should the appearance of the software matter? It’s about the freedom from corporate Twitter.” “Why should we encourage people to keep using proprietary Apple platforms?”
Regardless, I was convinced that I was onto something. As one of the few professional iOS developers in the community I figured I was well placed to have a stab at it.
It turned out that I was developing Flyweight in parallel with Eugen Rochko developing Mastodon. I think I already knew about Mastodon at the time but it was still weird and new, and had only a few users compared with the rest of the fediverse.
In the end though it was a lack of data in GNU social’s APIs that stopped my project. You see, my app could download data from the server in either JSON or XML. The JSON was a Twitter-compatible API that was easy to consume in web browsers but it wasn’t very clean. Things like group references were “baked in” to the HTML. Extracting these reliably in a native app would be annoying so I opted for the much tidier XML.
I knew that if I was building a good quality client I needed some sort of “Notifications” stream where you can see your mentions, repeats, likes and follows. We already had this in the Qvitter web interface. I assumed that all I had to do was tap into the XML equivalent.
I then discovered that Qvitter adds an entirely new JSON API to the server! This meant the notifications data was only in the JSON format, not the cleaner XML, and it was simply unavailable on the many servers that didn’t have Qvitter installed. I only had access to “Mentions”, which wasn’t good enough.
To solve this problem I needed to add new XML streams to GNU social, align those with Qvitter, get it merged, convince all the existing GNU social servers to upgrade to this new version, and then finally use it in my client with some sort of fallback behaviour if it was missing. Along with some other tweaks I wanted, suddenly this was a lot more work than I bargained for. I was still mulling it over when Mastodon suddenly got bigger than GNU social.
I had a look at the Mastodon API—it was clean, complete and much simpler than the GNU social one. There were already Mastodon iOS clients underway by others. I decided to abandon mine.
It was a little sad but ultimately I felt vindicated in my theory—a much more polished UI was what we needed to get more users in the fediverse. What I hadn’t expected was that it would come in the form of a brand new server implementation.
I mentioned that even in 2015 there were multiple compatible servers. GNU social was the most popular but it certainly wasn’t the only one. Others supporting the OStatus protocol were Hubzilla and Friendica. Neither of those projects were really optimised for microblogging—they had other goals in mind—but they could follow GNU social users and vice versa.
A later experiment was quit.im—this was a GNU social site modified to only allow posting images, not text. As far as I know it was first fediverse site that attempted to replicate Instagram.
Maiyannah Bishop, admin of community.highlandarrow.com, began making major improvements to the GNU social codebase. She created a fork called postActiv, whose development continues today.
Nowadays we have a W3C standard called ActivityPub that different kinds of servers can use to interoperate with social feeds. It’s supported by Mastodon, PixelFed and PeerTube. There appears to be work underway to add it to GNU social. Pleroma is another ActivityPub microblogging server compatible with Mastodon. To be honest I haven’t been keeping up with ActivityPub developments so I won’t say any more about it.
I’ve spent the last year or so primarily on Mastodon so I can’t vouch for the latest developments in GNU social land. I have noticed several long-running instances shutting down so Mastodon is taking its toll one way or another.
Nonetheless there continues to be a well-established and proud GNU social community. Many old timers view Mastodon as nothing but a particularly well-populated offshoot of what they already built. The desire to “beat Twitter” has never been universal. Some people found everything they wanted in their smaller GNU social servers and no doubt they’ll remain there happily for years to come.
There were also many evangelists, people who truly believed in the superiority of the community-driven fediverse over Twitter, who had limited success pre-Mastodon. No doubt the lack of volunteer developers and designers was a problem. I think they were also hamstrung by a faith in free software ideology to draw people in. The truth is that software needs to be pretty polished to get a look in either way.
Right now I’m waiting to see what happens when Mastodon comes under attack from motivated and intelligent troublemakers. Offhand I can think of all kinds of ways the network could be disrupted. The more it grows the more interesting a target it becomes. No doubt developers and admins will push back but it’s going to be a big challenge in such a distributed environment. Time will tell I suppose.
2022-11 Update: Some four years after posting this I stumbled on some old screenshots of Flyweight, the prototype iOS client mentioned above. I thought it would be nice to put them up and show it wasn’t completely vapourware. :)
Flyweight login screen
Flyweight showing a scrollable feed
-
Posts have had lots of names. Identica called them “dents”, GNU social “notices”, Qvitter “quips” (internally, “queets”) and now Mastodon has “toots”. ↩