Rudy: as a content moderator, you see the worst of humanity. Black Sky moderators are black, moderating anti black content, so it's pretty intense.
Day one it was pretty intense, but it was just ten reports. At a peak last week, it was a thousand reports. that takes a lot of time
[00:00:19] Intro
Andrew: Hello, welcome to the DevToolsFM podcast. This is a podcast about developer tools and the people who make them. I'm Andrew. And this is my cohost, Justin.
Justin: Hey everyone, uh, we're really excited to have Rudy Fraser join us today. Rudy, uh, you're the creator of Black Sky, uh, and I'm incredibly,~~ incredibly~~ excited to talk about that. But before we dive in, would you mind telling our audience a little bit more about yourself?
Rudy: Yeah. Um, first off, thanks for having me. Really appreciate it. Um, yeah, I'm Rudy. I'm a native New Yorker, born and raised. ~~Uh, ~~Black Sky's kind of been a side project that has really taken off. But, uh, my day job at the moment has been like, uh, enterprise Salesforce, uh, Uh, so like I, as a technical architect, um, and yeah, I've been working on black sky for about a year and a half now, and it has been super fulfilling.
Um, I view it as an extension of the community building kind of work that I do. So,~~ um,~~ I also am a mutual aid organizer in, uh, in New York. So I also want to give a little nod to we, the people at NYC, they inspire me constantly, um, and in the work I do.
Andrew: Sweet.
[00:01:26] What is Black Sky?
Andrew: So let's, let's take it from a really high level. Cause a lot of people probably don't know what black sky is and don't really know that like blue sky can enable this sort of like, Community within a community on its platform. So like from a really high level, what is black sky and why did you create it?
Rudy: Yeah, so I would say Black Sky is two things.~~ So Uh, but the ~~it's firstly like a community. Um, so it's a community of the black users on the blue sky app. ~~Um, but ~~and kind of separately, I refer to as like black sky algorithms, which is kind of the team behind it and creating the tools that enable black sky to exist.
And those tools are an app protocol implementation. Um, um, specifically called risky. Which enables us to build community, uh, kind of the, the TLDR is that, um, trying to enable community building on decentralized apps, um, particularly centering like marginalized communities. And that usually ends up meaning building in safety tools for them.
Uh, because you're building, you know, app protocol. For folks who don't know is a, um, uh, social media protocol, a decentralized social media protocol that at least thus far has prioritized public conversations. Um, there's plans in the future to have private accounts, private posts, um, You know, some parts of it are like DMS are private, but they're still like a centralized piece.
So they're not really a part of the protocol per se yet, but so everything's really public. And so when you're doing something like public, like a black sky, that just comes with harassment. And so, um, a key thing that, uh, I've had to build in from the very beginning, almost, um, has been safety tools.
Andrew: So other social networks really don't give us like the levers to pull to even like build in safety tools. So like there was black Twitter, but black Twitter is really just a bunch of people on Twitter that happened to be in the same area, talking to each other. Like what, what tools does blue sky give us to like moderate that community?
Rudy: Uh, and so I would even say that like blue sky has like a couple, there's a couple of things that it allows for. Uh, but a lot of stuff that has kept black sky safe has kind of just been like hacking on top of that. But the core, the core pieces that, um, black sky has been leveraging, uh, so far has been a custom feeds.
It was one of the first custom feeds launched. Before even custom feeds were available in the blue sky app. Um, and that's what was, so with custom feeds, the way I would think of it for folks who are not familiar with blue sky is blue sky is like if multiple companies were building the same social media.
Uh, app. And so each different part, when you go into, um, blue sky and you load up the custom feeds, those feeds could all be built by, they're all built by different people. Some are built by blue sky themselves. Some are built using feed builder tools, uh, that developers enable to allow other people to create, um, feeds without needing technical, the technical knowledge to do that.
Um, and some feeds are like black sky, which, um, I built from scratch. And so that's, that's like one component. And I, there's like kind of like a philosophy that went into why I thought custom feeds would be good for building a community like black sky. ~~Um, ~~cause there's, there's like a bit of, uh, You know, some people, the first thing people often ask after they maybe understand about the blue sky is like, why not activity pub or like mastodon for building that similar community.
And, you know, barrier to entry aside, like, I think one of the things is like discoverability. What black Twitter had was it was a, it was a black community in a very, that was very visible. Um, but still had built up its own defense mechanisms, um, over time that were very like the individual basis. And so people actually brought a lot of that, those like practices over, but with black sky, it allowed me to having like observed how black Twitter operated and some of the issues that faced just proactively build.
Um, some of those things into how, how the feeds work.
[00:05:35] Challenges and Innovations in Moderation
Rudy: Uh, the other component that Blue Sky has is composable moderation. So you don't, you're not stuck with just blue skies out of the box mod. Like you don't have to rely on just the Blue Sky core moderators. You know, you could, you could, you, you're able to choose who moderates content for you.
Um. And that could mean labeling content, hiding it, um, there's different protocol nuances of how much a moderator can can do and take action on, ~~um, ~~but that's been a big sell for for Black Sky as well, that we're able to offer context aware content moderation for, um, particularly protecting against anti Black content.
Harassment and misogynoir, which is the intersection between misogyny and anti black harassment. ~~Uh, ~~and, and we're able to do that faster than blue sky is able to, you know, they, they recently made a post that they've gotten there. They got like 40, 000 moderation reports, like in a day, um, the other day, I think.
And for the context, they had like, maybe I think, what was it like 300 K and all of 2023. Um, and so, I mean, we're all seeing that, like, there's just been a Lots of different factors that have led to it, but, uh, we've all had to kind of scale up our defense mechanisms a bit. But yeah, so for, for black users, they're finding that like Black Sky, where maybe another moderator may not take an action because they're unsure, they're unclear if this would be harassment.
Um, we're able to be pretty decisive on almost everything that comes across our desk. And we are, we have so far been able to do it faster than like blue sky would be able to again because they're dealing with a 000. We're dealing with a smaller subset of that. And yeah, there's this stackable approach to moderation too.
So we're able to like interweave things that I think that's a piece that like is less understood by a lot of folks. But to give an example, um, people are able to report content. To black sky, and it's not just stuff where like it would be labeled as anti black harassment. It could be that someone is spamming the black sky feed.
And so I've kind of directed people to use subscribe to the label or service report this content of something that shouldn't be in the feed, or it could be someone's being a troll. It could be that like this is a toxic conversation that's happening. Um, and they're able to also use, um, like use the in app reporting to do that really conveniently.
And we're able to handle it in a scalable way. Um, using the same, like moderation, like create different moderation cues. So like, this is a cue of stuff that needs to be removed from the feed. This is a cue of stuff that, um, is maybe needs to be labeled.~~ Um,~~ Maybe this is like so, so vile content that it needs to be.
This whole account needs to be put on a block list so that people can automatically block it. Um, there's some stuff that we've built that prevents certain people from even viewing the feed. And so that's like, that, that one is, that one is a, that was more of a hack. That's not like a thing that people really, no one else has really done that.
Um, but that goes back to the philosophy of like, okay, we're going to build this very publicly because there's this kind of cultural expectation or question that people ask of like, uh, who all going to be there, which is just like wanting to know who you're going to be in community with. Before you join a community again, that's goes back to the tension with discover ability and like a activity pub.
Um, so you're able to see the community before you start engaging with it. ~~Um, ~~but because of that, um, harassment, so harass that the stuff is still a vector of harassment because people who don't belong or who would be a threat to that community can also see that content. But as soon as they show their hand basically, and, and are discovered as being abusive or whatever the case may be, there's the in app reporting that can get triggered.
Um, there's proactive, there's some pro, there's Proactive, uh, automation that we could trigger for identifying certain things as well. And then, um, the This was actually kind of a hack that people made to, ~~uh, ~~if your feet is down, you usually just see a generic message of pipe through network error. Um, cause the upstream services broken.
and so some people kind of made a clever way to just show like a post. So it's a human readable error message. I took that. And said like, okay, what if I could put people on like a ban list and then show that error message instead of showing the content that the feed would normally show? Um, so that's like another one.
We call that ban from TV. So some so if you know, there's all these different things that can be triggered. You can get labeled as anti black harassment. You can get put on the green list. So people who subscribe can auto block you. You can get banned from viewing the feed so that you're not able to. Keep using it to find people, um, and it can really like isolate and, and stop harm, especially since we can't take down accounts ourself.
So yeah, none of that is possible with like any, any other platform.
Justin: Yeah, that's a lot of clever tooling usage. I especially like the matching user accounts on the feed and just giving them like a custom post that is like an error message. I imagine that wasn't necessarily intended, but it's like a very clever usage of the tools that you have available to you.
~~Um, ~~
Rudy: And really quick, that's because, cause you can build personalized feeds, right? Like if you wanted to make a for you algorithm, you'd have to know who the user was, so you know what likes they have. So when the. Every request to your feed is like proxy through blue sky, but it comes along with your, uh, which called your did.
Um, so it's like your unique identifier on the network. And so then we could say like this, this did is not allowed to view this content. Sorry to cut you off.
[00:11:09] Ad
Andrew: We'd like to stop and think our sponsor for the week. Mux. Mux is an awesome platform that makes adding video to your product as seamless as can be. They handle everything from uploads, hosting streaming. And even playback. They've got everything you need to build a comprehensive video product. And where that gets really interesting is with analytics. Analytics or something you add at the end, and then you have to constantly futz with and fiddle with, to get that important information to your stakeholders.
But with MCSE They have an amazing treasure trove of data That lets you gain insights on how your viewers are viewing your videos. The recently announced that they're opening up the flood gates.
And instead of just retaining 35 days of data, you get a full hundred, you can see how your users have been using your videos for three full months. That's pretty cool. And it didn't stop there. They added a whole bunch of new metrics. Also, ~~you can view the average. ~~
You can view the average playtime and the average completed time. You can even drill down into the details and see where people dropped off your videos to gauge whether your content is engaging or not. Pretty cool. Pretty powerful. And super useful if you're running a business. So if you want to get started with mux, head over to Dot com right now. And with that, let's get back to the episode.
[00:12:22] Technical Infrastructure of Black Sky
Justin: ~~awesome. ~~So to support all this, you have a few different like software components. So you mentioned the labeler earlier. You have the feed. Um, what, what are you actually having to run to sort of like maintain this? And like, you know, what does that require?
Rudy: Yeah, there's so there's the experimental stuff. And then there is, uh, the kind of core bread and butter. So the custom feed, um, you need two things. Uh, you need a firehose consumer. So Blue Sky has this public publicly accessible, locked open, uh, firehose that is sending, you know, at this point, gigabytes of data, uh, over on a daily basis through the network that hundreds of folks are subscribed to and listened for every like.
Post follow profile, update image, upload on the network. And then you, you, what you want to do is you want to filter that down to the content that you want to store in your custom feed. Otherwise it would be an enormous amount of data. So I have a firehose consumer that listens for everything on the network, um, has been really.
Scalable so far. Uh, and then it sends over to, uh, I, people do this a little differently. It's all up to you and how you want to design it. Some people use the firehose consumer safe straight down to the database. I send over to the, uh, uh, basically have a feed gen server that does two things. It both like receives message messages from the firehose consumer and then saves that down to the database.
And it also serves up the posts from the database, um, in response to like feed generator requests. So when you view black sky, when you view black sky trending, um, that's all a server. Um, and I keep them separate because especially like a, you know, another example of that was like, I'm able to like with this big spike in traffic, like The Firehose consumer has done pretty fine on its own, but I actually had to scale the feed gen horizontally, um, in response to all the traffic that came in, um, I had mentioned off recording, but like the Tyson fight was a big night for Black Sky.
It was coming right on the heels of a big wave of black Twitter users migrating over. And then it was just a big cultural event. And so it was the first time the feed ever had a million views in a single day. Um, and that broke a lot of things, uh, but that's when I was able to, okay, like the, the firehose consumer is fine.
Like, you know, that can, that's on some good, uh, hardware, but the feed gen, like I need to separate this out because I was finding that. Which is again, dealing with scale has been a learning curve for me, uh, of like this, the server is responding slowly. All of a sudden, like, what do I do about that? Um, is it that I need to, like, scale it vertically?
Like, do I pay more for this individual server? Do I scale it horizontally? Um, and figuring all that out on the fly while it's breaking and people are talking about me because it's, and so that's the, the other tricky thing with the network is like, you get. No, I think like the startup advice is like, talk to your users.
My users are always talking to me. And, uh, and it's funny cause Netflix was glitching out during the Tyson fight. And so people were posting like, uh, it actually made me feel better. Cause I said, Netflix was struggling more than the black sky feed. Um, but yeah, so that's, so the feed gen, the firehose consumer, uh, and, uh, so it's a couple of things.
So I've built a custom app protocol implementation in rust. Um, so that's risky. So the firehose consumer is risky firehose. The feed gen is risky feed gen. ~~Um, ~~there are other, there's another component that's more experimental that I'm working with that we can get into that's like the, been the. Most difficult to build and most complex and all the things, but I felt was very important enough for the long term vision.
I have with black sky, which is a P. D. S. which you can basically think of the back end half of a of a social media. Um, but I, what I run from, like, just out of the box blue sky is their, um, their ozone instance. So that's the labeler service. ~~Um, ~~That's where you can log in and have like a moderation queue and you can have, you can add moderators to, to help you out with stuff.
And then you could like label the content. Um, so that I don't have a custom version of that. That's just out of the box. Um, And then I have a bunch of scripts that do like a bunch of Python scripts that do different like jobs of like removing like a, uh, auto mod script that removes content from the feed based on certain rules.
I've just basically just like a hundred SQL queries that like remove wordle posts, uh, you know, remove sensitive, uh, topics, um, remove posts that have like 20 hashtags in them. Um, there's stuff that like, there's, you know, obviously again, since messages are coming through the fire hose, it's doing some decision making there of like this person included this hashtag.
So include this post in the feed, this person added themselves as a member and it tracks membership in the backend on the database. Also Postgres, that's the other thing, obviously. ~~Um, and, uh, da, da, da, da, da, um, It's ~~so it's that's tracking some content and then there's like, uh, there's some protections that are built in there because we found like, you know, once threats evolve over time, we didn't really have to worry at a certain point, you know, when black sky was like 200 users posting in the feed 10, 000 people viewing it.
It wasn't people trying to like, like use the like n word and like post using the black sky hashtag but like that happened. around the time of the million views and it was very public and someone was like, this shouldn't, this shouldn't have happened. And I know how quickly something like, first of all, I can empathize with like, feeling maybe like a bit of betrayal of like, Y'all said this was a safe space.
Why am I seeing this? Um, and then, uh, I can understand the, like, this shouldn't even come through the feed. So responded to that and basically put in some slur filters to, to not even, no matter what you, what your content matches, it won't even make it into the database at that point, you know, stuff like that.
Um, but then there's like retroactive stuff as well, um, that are deleting stuff and then like ozone. I won't be too hard on ozone. It is very bare bones. ~~Um, ~~and I have, I've submitted a lot of issues to the GitHub. Uh, and I should work on some myself. Um, but it's just been really hectic lately. ~~Um, ~~but like ozone doesn't email you when you get, uh, an email.
Like a new moderation, like report. And we, uh, mentioned, I mentioned off camera, like we went from getting, when I launched the ozone instance, we were getting, I knew, I knew if I launched an ozone instance, it would blow up. I knew people would use it. Um, I know that like there would be content to report.
But, and it was crazy content right off the bat. It's, it's really, moderation, people underappreciate content moderation so much. You, as a content moderator, you see the worst of humanity. Um, and, uh, Black Sky moderators are black, so, and dealing with like, like, moderating anti black content, so it's pretty intense.
Um, day one it was pretty intense, but it was just ten reports. At a peak last week, it was a thousand reports. and it takes, that takes a lot of time and so you, you don't get notifications, um, by default with, with ozone. Um, I think the, the blue sky team, they have like a S they use, I guess they use Slack for their moderators.
And so they have like a Slack bot built in go. But so I just made like a quick Python script that like sends, uh, email notifications, gives context to the team of like this post is showing up in the feed or it's not, here's the comment, here's the person's username, et cetera. So different little tools like that are, are all there.
Um, and like I mentioned, like the backend, so using. Using Postgres. A read on a couple read only replicas now, and a primary node all
host. I was gonna say, all hosts on Digital Ocean. 'cause I got into the Digital Ocean startup, uh, hatch program. So wherever I can get free credits.
[00:20:16] Scaling and Community Engagement
Andrew: Yeah, that's, that's kind of what I want to talk about next is that like when from the outside looking in, not having created a feed at first, I kind of assumed the shape of it would be a lot different. Like in general, when I use an API for something, it's like an API. And I like say, Hey, blue sky, give me some posts.
The setup of a custom feed is like, as you said, kind of completely different. It's just a server just listens to the fire hose and like chooses to store things or not. How do you think that's going to scale? Cause like, even right now we're not like the biggest network and it still costs money. Like, do you foresee it being a problem as the network grows, having to kind of like host all of your own infrastructure for it?
Rudy: yeah. I would say so. I agree. It's not obvious that you're not relying on Blue Sky to like serve up the posts, which is why that like people underestimate that like custom feeds. One, and I've had, what's been really cool with the migration of users. I've, I've like, uh, I'm not used to being in the public eye very much.
Uh, and, um, I went from like having the biggest following on for any network was like a thousand followers to now like 30, 000 and, uh, I still have yet to like introduce myself to people or like explain like the project or anything. But the users, like they. I think they've, they've done a good job of teaching each other about this.
And what's been really cool is like the mainstream adoption of decentralized. technology is like allowing people are learning about it and then explaining to each other. So someone made this really good thread from black sky was just like, Hey, custom feeds are basically their own app. Like you view it inside the blue sky app, but they're their own app.
And I was like, I didn't think people would get that if I explained it that way. Um, they did. And they're like, um, yeah, so you can really, you can do whatever you want with it. You know, um, that's, and that leaves a lot of. Creativity and freedom to design what you want to do. Um, you know, I think an interesting thing was like, you know, with custom feeds, it's like, how do you make a, how do you, you, you, then you get agency.
So how do you make an engaging custom feed? Right. Like that's one thing. ~~Um, ~~and like, there's no like out of the box, like here, I think the Twitter algorithm is like insanely complicated. Um, but just like, yeah. How do you, how do you choose? Do you go off likes? Um, do you go off like. other interactions? Do you add waiting to it?
Do you add like age of the post? Um, do you want to surface posts that are not getting as many likes, but they could be potentially interesting to people? Do you add personalization that this person doesn't, you know, this is maybe a really pop popular political post, but the person may not what like politics or want to see that right now.
Um, none of that exists today. So people create feeds and they're like, Like, I think the technical infrastructure is a question, but the reality is, I guess I go back to where I was talking about, where I was like, no one, like, no one tells you how to deal with scale. They just kind of tell you, you probably won't make, get to the scale where it would be a problem.
A majority of the 50, 000 custom feeds on blue sky are not being used. So as long as they kind of work and work for the, like couple people who are using them, it's probably fine. But if you want to make a really popular one that gets a lot of usage. And like, that's where you start to like have to. Figure things out as it goes.
And, and there's not really, and there's not really the bootstrapping of how do you get it to popularity, right? You don't really even, let's say you've wanted, let's say you want to make a custom feed and you're like, I don't want to learn how to be a dev per se, but I want to make a custom feed and then I want to promote that feed and I want people to get really engaged with it.
And I want to build a community around it, whatever idea or topic, because people did that very early on. Um, The game dev feeds or, uh, 3d printing feeds or anything like that. Um, and then it's like, okay, maybe you figured out how to include your topic. Or your community, but have you figured out how to, do you have, do you even have the tools to make the algorithm interesting to people, ~~um, ~~and like choose what that means for you particularly for me, it was like, how do we keep it interesting, but also keep it safe because it could easily be corrupted, um, but not everyone's going to have that consideration, but, uh, yeah, I think more about that.
And so blue sky, I've seen them talk about this, where it is like, they think that one of the devs mentioned that he doesn't think. Custom feeds are optimal right now. I think threads have recently launched custom feeds to kind of copy. I mean, literally copy what blue sky has done. And, uh, I think there's some people who like respect what they built and how the ease of use of it.
For me, I'm like, ease of use, but not really, is it actually going to, it could never produce a black sky. So I'm just kind of like, I don't know. Um, it's not really for me, but, I do think that there is room for tools that allow people to create engaging feeds and build communities around, uh, and then. I ultimately think it'll be like a service that does that for people.
Um, and that we'll need to figure out it's funding and monetization or otherwise, whether it's, and I think, so that's something that's been interesting.
[00:25:37] Sustainability and Future of Decentralized Social Media
Rudy: Uh, I got, I went to earlier this year, I went to this really interesting. Uh, that's, it's really stuck with me. This decentralized social media workshop at Princeton.
shout out to Nathan Schneider. He, and he was who got me in there at the last minute, but it was like, no stir people, not activity pub folks. Me and Brian Newbold, who's like a core BlueSky person, we're the only BlueSky folks holding it down for App Protocol. Uh, and like one of the, for like, it was an unconference and like one of the things that we, my table talked about at one point was like, how do you actually fund all this?
You know, some folks in Noster have gotten big money, um, to run their stuff. Uh, you know, Farcaster. Which wasn't at the known from forecast. It was there, but they got like, what? 180 million or something crazy like that. Um, you know, blue sky themselves, people always come for them, but they're like VC funded, you got the social web foundation, they're like nonprofit.
But, you know, for with Black Sky, for me, I was just like, I, I want to build something engaging and interesting, but I'm like, I'm not gonna go broke doing it, you know, and it's like, I've, I go back to again, like, mutual aid organizing. It's like, it's supposed to be, if it's for the community, like, I don't want to, here's the thing, I'm going to give it away for free and like, give it back to the community, but I don't also want, you know, Unlike a charity, like I'm not trying to just, you know, have the burden be on me.
Like we all, like, we all got to take a piece of this and make this work. Um, so from the early days, like Black Sky was the first project to be, Black Sky was fiscally hosted by a nonprofit as of right now, ~~um, ~~uh, open source collective. And so all of our funding and stuff is transparent over there. So it's been donation focused, but honestly, like the donations only recently started spiking.
~~Um, and I, I think it's, I think the, ~~What I think people are donating for is like they appreciate the work and the stuff that I built and there's been a bunch of more usage and stuff, but at least for me personally, like, I want that money to go to moderators because I believe in paying for content moderation because it's terrible work, like I mentioned, and most like Reddit mods do that for free.
you know, I think we should be able to have community moderators to get paid. Kind of point blank period, the technical stuff and all that, that's something I'm trying to figure out separately, but there are opportunities. Um, so I think I'm also probably one of the first feeds to like, do something like a sponsored post.
There was all these conversations about advertising. Um, and the consensus has always been like, advertising isn't even possible on the network. And I disagree with that because, uh, Anything is possible. If you just like where there's a will, there's a way you can hack something together. Um, and I don't think that, like, I think it kind of depends on what you advertise, but long story short, I was just like, I want to get us past the conversation of, can we do this to, should we do this?
Um, so the way that I did that was like, I have like, when you view the black sky feed, any of the feeds now, and even I've started like hosting feeds for other people, um, randomly about like 0. 1 percent of the time. Sponsored posts will pop up in the feed and it will be for a donation to black sky. Um, and I think that's like a people may not want ads, but I'm like, what if you, what if you just did it like that?
Like what if the advertising businesses impressions that lead to like people paying for something to another company? Why can't it just be like, Hey, you're using this social media product that I built. Chip in every once in a while. Um, and like, it's, it's in your face, like as you're using it. So that was kind of a thought process.
I'm sure people will come up with other creative things. There's someone who's built a co op, um, who like founded a co op for their app protocol projects. Uh, once upon a time there was a project who got into Y Combinator. Um, they ended up pivoting shortly after. Uh, but yeah, I think there's going to be interesting business models that come with that protocol.
Cause there's also like an openness to. experiment with that and an understanding that like, we have to find some way to sustain it all. Um, and what that will be, I think it'll, we'll see how it all shakes out.
Justin: This is such a reoccurring theme that we have on our podcast. We talk a lot about like open source sustainability. How do people who work on open source tools or trying to build a business around them make money on that? Uh, it's interesting. I was at a un conference myself in LA, uh, several weeks ago, uh, for an independent research conferences like Inc and Switch and basically people in the HCI space who are not, Not necessarily academic, but who are doing like computer science research.
And funding was another conversation that came up. It's like, how do we, how do we make this effort sustainable? Um, so a lot of parallels between what we're talking about here and what I'm hearing in other communities. And, um, I think your, your approach to this is really interesting is like thinking like creatively about how you can use the technology to sort of like self sustain it.
Um, what is your sort of long term plan here? I mean, do you want to turn this into a. Business or a nonprofit or like, how are you thinking about like the organization around the technology? Cause I mean, at some point, like scale does hit you and stuff does get expensive. And, you know,
Rudy: Yeah, yeah. I, um, for me, I, the way I think of it is like, I want to just work on Black Sky full time. Like I, I want to get to that point. Like it's, it's the thing that I, I spent so much time thinking about all the different problems and where it can go. And like, I also like, I feel like there's been much kind of hate and doubt that has Happened.
I mean, it happens with like any publicly visible project, both for like black sky and like app protocol itself. I'm like, I kind of just want to like prove some people wrong. Um, and, and like build something that people really fall in love with. And that's already kind of happening. ~~Um, ~~and so it's like, that's working.
So, and that's with me doing it part time late nights. I'm I have two hours, I'm on two hours of sleep right now. Um, this, uh, my second world's strongest cup of coffee. Um, And yeah, I would like to be able to get some rest and be able to work on things. Um, it's been interesting, like, you know, for the longest I got, like, uh, I've dealt with lots of like rejection for like larger funding.
And so that's where I like leaned more into the grassroots. I'm like, you know, if, if no one understands the thing that I'm building, the users understand. So, you know, let's see if they can support, and that's been cool, like that's gotten up to. I think right now that's around the estimate open collective shows like an estimated annual budget.
That's like 30, 000. 35, 000 right now. But I'm like, I feel like that. I want that to like cover the moderators, um, basically. And, and I, I can see the moderation queue as the network scales, that scaling. Um, so as of right now, I'm kind of thinking of it there, but the other thing with the growth and the, and the attention has been like, there's been angel investors who have reached out.
Um, so I'm starting to have those conversations and, uh, I, I think there's a world where I get to just like work on the tooling behind, like stay working in the tooling and then building for like other devs and try to find, um, try to find sustainability there and then keep the black sky community aspect, like that be its own thing, that's not corrupted by any, like any kind of funding that would come through, but, uh, there's devs in the ecosystem.
There is no tooling. We're all doing this from scratch, learning everything the hard way each time. And so that's something I'm kind of like looking forward to.
[00:32:52] Exploring Sponsored Posts and Ads
Andrew: Yeah, the, the sponsored post idea seems really interesting to me. I feel like blue sky as a service and a protocol will start need to like having things where you can like say, Oh, this is an ad. Cause like having a feed with a bunch of ads in it and you don't know their ads seems wrong. Um, so I, I, I do hope they add something there, but as you said, I could see, I could see big businesses happening around just like curating and putting feeds out there.
It's, it's a very new frontier.
Rudy: Yeah. Yeah. ~~And, um, ~~and then if people were like, really are like, the thing is like, it, it has the opt out, like lots of ways to opt out. Right. So if someone's making a feed that. Maybe it was popular and then there's ads in it. Like maybe if you don't even like the, maybe you're a dev and you don't like that, I built black sky and I started putting these like donation posts in it.
The code's open source, so like you can just fork it, remove the sponsored posts piece, or I mean, you can literally like run it your own yourself, like it's an environment variable to like turn off the sponsored posts. So like you could, um, I think as long as people continue building that practice as well as like giving people freedom of choice and like, uh, A way to opt out then and at a protocol level, I think we won't run into like the platform decay problem and like the race to the bottom.
I think that there are lots of mechanisms that prevent that kind of thing from happening. Um, but I do agree that like, probably like, like, I can see fandoms trying to like, create. feeds for their, for the fandoms, um, very easily. That's kind of already happening. Uh, I'm very, I'm, I don't know where I stand on brand accounts.
~~Um, ~~if someone like adds their like black owned business into the black sky feed, then I'm pretty chill about that. But I'm very wary of like anything else. There was actually like the, like Hawaiian punch has an account and is doing the, like, you know, quippy, you know, dunk, poke people that are funny, but I'm just like, I don't know that I want, like, I remember when threads launched and it was just like brand accounts talking to brand accounts.
I just think that's like, I don't know. It makes, it gives me the ick.~~ Uh, ~~but I do think, yeah, there's lots of new opportunities. Um, cause the other thing too, is like, because it's an open protocol, you could do. You don't even need to milk, build a whole separate app. You know, like something I, a hack, like a hacky thing, or I think, I think it's cool.
It's like, all right, so I write a blog about my work with black sky often. And something that someone built was like, you could take, you could make a blog post on your blog and then you could post the link to that as a blue sky post. And then you could put a component on your blog so that as people replied to the blue sky post, the replies show up on.
on the blog page. And so from there, it's like when you, you just think about the different ways the content is just easily available. The content is just an open API. There's a fire hose. You could, you could do all these crafty things. So, um, I'm curious to see how people start incorporating the content from the network in other places that are not just like within the app.
And obviously we haven't even talked about like other people, like people have built other apps. There's like front page. That's like a thing. That's like the Reddit. Kind of, uh, for app proto, um, you know, someone made a, like a Lincoln bio kind of, uh, app, uh, there's smoke signal that's like RSVPs. Um, and so, yeah, there's, there's all of those things that are being built all without people having to create new accounts and they can take their, their identity with them.
Um, something I think of with black skies, like you can take, you can take your community with you because we're all like you, you're not creating new accounts and stuff. Um, But then like, I don't know, maybe Netflix starts pulling in content, like on a live stream or something as like, or, or Spotify recently launched.
Like I got, got invited to like a little, little Uzi Vert listening party. That was like, I don't even know if he was in there. Like he spoke, but like, it could have easily been like a pre recorded thing. Cause he just said one or two things and then had like dropped a couple of emojis in the chat, but it was just the music playing, but they could also easily just pull that as like.
That could just have been a custom feed on the side, um, pulling from, from blue sky and it would be really easy to do. So, uh, yeah, I'm like, there's lots of possibilities. So
Justin: this is all like super exciting. I especially like the sort of like. Ingenuity of all the tooling that you've built for black sky, uh, Andrew, all the stuff that you've done for blue sky, like there's this really interesting tooling and the one question that it begs is how does blue sky, the company maintain itself?
And then, uh, is there a risk to these communities based on blue skies, actual financial risk? Um, so one of the questions that we sort of were thinking with you is like, what do you think? about the life of black sky or this community sort of like outside of blue sky, the company, uh, do you think like the app protocol lives beyond them or is it like, what does that look like?
Rudy: that's why I like, I, that's kind of why I wanted to build this company is why I wanted to build Risky, which is, um, that I want to build an alternative app protocol implementation, um, for various different reasons. Uh, again, there's, I think Risky is probably one of the. Most developed or like active projects that is trying to pursue that.
So that's like a cool opportunity, especially since a lot of the, um, pushback on app protocol is that it's just one company building the protocol. ~~Um, ~~and. You know, while that is true, like, I mean, they, they accept like PRs to their core code base, but they are still like selective with it. They have rules around that, you know, so I, I want to, I want to be able to build Risky and hold them accountable to building the protocol.
~~Um, ~~and because if, if you only ever run their code, I don't really feel like that's. That you don't you at least don't get the power without arguing semantics about things you're not getting the power of decentralization because they'll just you're just going to always push whatever update that they have.
Um, and so you're just, you're just hosting a version of it but you're not really, you know, they could just still change the protocol at a whim to whatever they want it to be so I want to be able to hold them accountable to a protocol. And be able to build that as an alternative that also helps ensure the longevity of a community like black sky.
I mean, today we prop like without me building all that, like we could probably just host all our own infrastructure and like have a separate running, could host the front end client, like have like a, probably don't even need the relay, like just have a PDS and the front end client and have that just be disconnected from everybody.
Uh, So like that's possible today. And then, well, the one that's like trick, there's a couple of sticking points because there's stuff that they, they have not yet made available to the protocol, like DMs, like the video transcriptions and a DID PLC, which is the core identity service, that's all hosted by them.
I think there's people building stuff to like export. And backup, um, stuff from the PLC directory and the PLC directory that has like your, your keys. So yeah, like your cryptographic keys to control your account. So there's some people like trying to build that resiliency there. I would say like if it happened today, there's risk.
That that's how I've seen other people talk about it too. Like if, if blue sky went bad today, that's risk. Uh, If they did it like a year from now, hopefully that's like lessened. And then I think a couple of years out, there should be, we should see a lot of stuff play out. Like, do people have the funding to keep working on these things and like build the alternatives and like have the support to make that happen?
Uh, and then it like, I think the protocol is kind of a dream right now. And then the closer we can get to making that dream reality, then I think, you know, then I think we're good. It just has to like last. last that long. And if they did, if they did run out of money kind of immediately, which I'd have a hard time imagining that happen, but if they did, like we could rebuild.
Um,
[00:40:59] Future of Black Sky and Blue Sky
Rudy: and I think the community around Black Sky has been so far that like there would be a bunch of people who would come along for that ride. Uh, it would obviously cause disruption. People would fall off, but, um, I, yeah, I don't think it would be the end of it and like pack everything up, move on at that point.
Andrew: I kind of find it funny that your plan to de risk Black Sky is called risky. Uh, that's just a, a fun point. Uh, but you mentioned that you're like rebuilding all of that proto or all of those parts that they put in their TypeScript library, that's a, a big ask. Uh, you mentioned that one part of that has been particularly hard.
Uh, can you talk more about that?
Rudy: Yeah. Um, I mean, I like a challenge and if, and I've done all this with like, as like while doing two consulting engagements on the side and all the other things that have happened and like got married, like got erected for protesting, like still made this all work. Um, but yeah, but PDS, uh, was like 100, 000 lines of code at this point.
Um, so again, once again, like a large amount. Contributions to making that work. Uh, and it's still like, there's still some kinks to work out. Um, but I think that also kind of like shows. The that it's a not impossible. And like, uh, you know, for me, it's like, it's, it's engaging. Uh, I wasn't like, uh, I've seen people throw a couple little shots at like how I've like written the code again, first time open source, I never really written, written, uh, wrote code in rust before, uh, but I just like the rust programming language.
So I used it as the opportunity of like, I'll just do both. I'll like learn app protocol, build app protocol, learn rust, build it. Um, and been having a good time doing that. But yeah, the PDS is complicated. So the PDS is the personal data server. Um, it is analogous to an instance in activity pub for folks that are familiar, but it is different.
Um, so once upon a time, the easiest way to explain is like once upon a time, like the PDS was kind of the core. Back end component like people talk about the relay now and how that's centralized and stuff The PDS was it was at one point the app was just the PDS and a front end client and everyone was on one PDS Then they added a bunch of different abstractions like like layers in between that and then they added the relay and stuff But at its core piece like the PDS is the back end service.
So it hosts your like your account your password Um, it hosts this like important piece called the, um, your, it's your repository. And it's like a GitHub repository. Um, but it has this like fairly complex data structure. I'm again, like I mentioned, like a more of a hacky, like, not like a, like, like a staff engineer, like senior engineer by any means.
And so I never really worked with tree like data structures before. Um, and so, and I hate recursion from the bottom of my heart.~~ Uh, ~~But yeah, so the Merkle search tree has like both of those, but it's this like computer science concept that they like took from a white paper and implemented, um, that it's supposed to be able to, uh, kind of maintain it's like speed and be able to be consistently updated, uh, with like, uh, for a pretty like large amount of data essentially.
And within that, like there's, so there's already those components and then they're, they're, it uses, ~~uh, ~~Seaborg, uh, to do like in, uh, it's encoding. So the pointer, the, there's pointers in the tree that are like Seaborg encodings of the data. So it's like, I think I'm explaining that right. Um, so, There's already taking this like theoretical concept, they've implemented it in TypeScript, translating that over to Rust.
There's all the different libraries that don't really come across. Well, someone explained to me, like, there's like a bunch of different CBOR libraries in Rust, but like, they're all kind of old. ~~Um, ~~and so that there's, there's just trickiness with that. ~~Um, ~~what else was challenging? Uh, the sequencer. So, you know, these are like your, oh, the sequencer plus the.
The WebSocket stream. So you're also doing like WebSockets, uh, which is like basically like the relay is, uh, the word is like slurping, but it's like, it's consuming from your, your PDS. Um, so when your PDS comes online, you, you turn on server, create an account, make a post you spoke internally, there's a sequencer that's listening for the events, um, that are added to the database.
And then, uh, if there are, and then it creates like a queue. Uh, of those events, uh, and then for anyone listening. It's going to like output those events based on the cursor that they provide. Um, and there's like a window of how many, like, if you allow like events from 72 hours ago, et cetera. Uh, but you as a consumer, if you, you could, you could do, you could run your firehose consumer straight from a PDS, right?
The, the relay is like a convenience. So the relay is listening to all the PDS is on the network essentially. But if you wanted, you could just consume from one PDS. And so, uh, the relay is just another consumer, um, essentially. But in mass. And so that was a tricky thing. I ran into all these deadlock problems and, um, it was just like, there's just like new programming issues that I hadn't encountered before, um, dealing with web sockets, never used those before.
~~Uh, yeah. So those, those are, ~~those are some, there's just like efficiency, like even though I felt like I translated the code fairly decently over to Rust. Um, It just didn't run the same, um, for whatever reason. And so then trying to like identify and troubleshoot those problems. Then there's the protocol piece.
So that's like something where the tool, the tooling that I'd like to build out, I think on activity pub and mastodon, if you like want to build a active, um, activity pub implementation, I think that's like a. Good way of describing that you could build, uh, there's like test suites that can say, like, this is compliant with activity pub.
There are no test suites for app protocol. So it was a lot of trial and error seeing, like, can I get the relay? Can I get the did PLC server to accept assigned did document? Did I sign the document correctly, uh, with the, with the keys? Um, then like getting the relay to accept my messages and like the relay is not like outputting like really great error messages or anything like that.
So then I had to run the relay locally on my machine and like. Add deep, like a bunch of print statements, you know, at least this is the way I did it. I had a bunch of print statements to see like what is coming through on its end and like, why is it what's its internal errors of why it's rejecting stuff and I spent the most time i've ever spent looking at raw bytes and being like, oh It didn't accept the did plc server did not accept My sign did document because the way surdi serialized The, the, was that it, it didn't handle the nesting of it correctly.
And I had to serialize it as a string and then serialize it to JSON and, or, and add a respect sort order, like to the like crate as a feature, uh, and that got the correct order of bytes. Cause I looked at it and I was like, What it's supposed to. And then I had like the core blue sky thing running, doing the same events.
And then I looked at the bytes from both of them and they're the same numbers. They're just slightly like five of them are the wrong order. And like, that was because of like the sort order. And so that was like one of those things where it's like, it took me really long to figure out. Um, and I don't know that there was an obvious tool to do that.
I've like started to find little tools that help with some things like that. We're like. Sibo. Um, it's, it's like you do this in coding, but it has like this, this, it's like this, this tagged bites. So it has like, it starts with like the number 42, and I just found that like sometimes I, when you view the SIBO or when you view like a, a pointer, basically there's different rep ways to represent it.
You could represent it as the bites. But like, it's usually represented as a string, uh, format of it. And so like a hex, uh, string. And so like just passing, choosing which of those you're actually passing and, um, how you're supposed to interpret those. That was like, again, another hard problem to like figure out.
Cause there's not something that's just telling you, Hey, this date, this specific field that came through, like this is wrong and that's why you're getting these errors. ~~Um, ~~yeah. And then there's some things that just aren't clear. Like. You know, with the Merkle search tree implementation, like does the pointer, there's like a field called outdated pointer.
So one would expect that if you change the underlying data, it's okay that the pointer is outdated at a certain point. But when I tried to do their test scripts and I did my test scripts, I was finding that like, it wasn't the expected result. And then I had to like force the pointer to update in certain pieces.
And I'm like, I don't know if I did something wrong elsewhere. And now I'm like compensating for the first error or if I'm, you know, Or there's something else going on. So a lot of like non obvious and like being probably one of the only, at least at the time, it was like the only person who was trying to build like a full PDS implementation and not just like pieces and components of it.
It was a rough and lonely road. I spent a lot of time, like I would hit up the blue sky devs themselves, which kind of plays back into like people saying like, Oh, it's centralized, but it's like, I don't know, they designed it. And. We're trying to figure out, make it decentralized, but that means like, I need to go back to them.
Like, Hey, like, why, why am I encountering, encountering this? Or like, yeah. And like, sometimes things, I thought things were a problem and they weren't a problem. Um, and it's like, then there was stuff that's specific to their implementation. So all of these things, that was what made it tricky, new technology, new libraries, me being new to certain concepts, like fundamental programming.
Problems. And then, like, you're trying to build in a, an ecosystem that's a protocol. So, like, you need things to be compliant with the protocol and there's no, like, compliance testing. So, you just have to, like, try, at least for me, it was, like, learning things the hard way. Yeah,
Justin: Yeah, that sounds like a lot. Uh, you've, you've bitten off a big chunk of complexity and props for using Rust to do it. Like Rust is not an easy language to just build anything in to begin with. But, uh, Doing this and figuring out the PDS and like, you know, working on the protocol and like trying to get all that to work.
That's fantastic. And the community will be better for it, which is also one of the things that I'm excited about by Blue Sky in general. It's like, uh, these efforts hopefully will like build, uh, so the work you're doing here is going to help someone else. And hopefully it like all comes back together.
[00:52:06] Final Thoughts and Future Prospects
Justin: Um, as we're wrapping up, we always like to ask a future facing question.
And for you, there's, uh, A few different ways that we could go with this. Uh, there's, there's obviously the work that you're doing around Black Sky, both the community and the technology that enables the community. Uh, there's sort of like the features that you still want to tackle, uh, coming up to sort of continue to improve the community and maybe like what the future of this space is in general.
Um, so I'm gonna just leave that with you and let you pick up on the thread that you think would be most interesting to talk about.
Rudy: I would say like something that I'll answer kind of all three, I guess. One thing that I'd like to tackle probably that feels more urgent than the PDS, but I just started the PDS. So I like want to finish it before I pick up anything else. There was tons of stuff in the news recently of how blue sky struggled overall with the, the network traffic.
Um, so like it was super frustrating these last weeks where it was like, people were like, Rudy, is it your feet that's down or is it like all of blue sky that's down? And like, sometimes it was me and it was like, okay, like now I got to figure out the scaling, like a Tyson fight. That was me other days. It wasn't, uh, other days it was like intermittent issues with just the, like the blue sky app view.
I think one of the times it was like a underwater, like, internet cable got cut, um, it was just like, there's just so many things going wrong and all different, literally every different part of the stack in the network. But then like one of the ones that happened that had to meet, like, we literally all had to shut down like our, uh, firehose consumers was the relay, um, jazz from the blue sky team.
Um, they have like done the, so I mentioned like the relay produces like gigabytes of, of bandwidth, um, daily and multiply that by the 250. People doing like consuming the fire hose, like blue skies relay is like responsible for like outputting terabytes of data daily. Uh, and it's also archiving a lot of the data.
And so there's like a bunch of like, there's like heavy, like disk operations. I think that they end up going through and like, Burning through their, their servers, uh, at a certain point. So jazz launched jet stream, which is, uh, with the fire hose again, this is like authenticated transfer protocol. Like there's, uh, so there, it comes with verifiable, all the posts that come, all the records that come through are verifiable because it brings through the whole Merkle search tree, all the blocks, um, from it.
Uh, and so if you wanted to. You could verify every single thing that comes through the network. Um, and that's the intention of it.~~ Uh,~~
Most people consuming the Firehose don't need that per se, uh, because they, maybe they, they philosophically trust Blue Sky, right? Uh, and they also, they're really just trying to hack on like a hobby, custom feed, and so they just want to like get the posts or whatever the case may be. And so, Jazz launched Jetstream, which allows people to, um, Like consume the fire hose with just Jason and it's much lighter.
And so it's like significant, it's like a hundred or some crazy number lower. ~~Uh, ~~blue sky is trying to get people to use jet stream, but in the, like, in, as in the process of trying to encourage folks to do that, like the relay just kind of like crapped out. Over these last couple of days. So they're trying to basically encourage people now, I think, to run their own relay.
And so for me, I'd like to run my own relay. Um, if it's, especially if it's not like the over, like at mines wouldn't need to be archival, like it doesn't need to store posts. It would just be like another kind of like pipe of the, the data from the network. Um, And it would allow Black Sky, like it used to be fun when like every other feed broke and like Black Sky was the only feed that worked.
That always gave me a little bit of pride. Um, and so I'd like to be able to like get that built in. So I'd probably work on some of those. I think they're, there's like a tool called Rainbow is what they're, um, like, like Splitter that's, that they're working on. So there's that. Uh, I think the other piece that you had mentioned was, for this space, I think just a lot more.
A lot more companies and people building stuff. Hopefully I'd like to see more front end apps. There's stuff I wanted to, before this big traffic wave, I was going to like work on like a little front end app that was going to be like a Pinterest for app protocol. Um, uh, but like, yeah, more of those like creative.
Creative things. I think we probably see more connection between activity. I mean, hopefully I feel like it's really, it's like a lot of just interpersonal conflict. I feel like between the communities, but I like to, I'm like a part of like, if that's the, uh, federated trust and safety, um, or I think like knowledge sharing amongst like moderate, like I've, I've met with moderators from like Reddit and discord, like communities.
And like, I think that knowledge sharing is really important to keep the whole internet safe. It's like, we should know. Like, what are the threats that are happening? I'm actually wearing like a shirt from my friends who built this, have this project called open measures, um, that like, they provide like analytics tools for researchers to track stuff on the open web, uh, to like, for trends for like, like white supremacist campaigns or like QAnon or stuff like that.
Um, they can like choose what, uh, keywords and stuff and like, be able to have visualizations to be able to understand those threats. I think all of those kinds of like that kind of intelligence. Is going to be super important so that none of these spaces become places where like extremism can start to thrive.
Um, I think that would be really cool for the internet. I feel like you had one other one, but maybe, maybe I missed it. That's stuff I would want to build. That is where I think the space ends up. That's one part of the space. I think that ends up growing more.
Justin: You made it Andrew.
Andrew: I'm bad about that. Sorry. Uh, well that wraps it up for our questions this week. Thanks for coming on Rudy. This was a very interesting dive into all the work that you're doing. It's super commendable how you've grown such a community while also rebuilding all of at Proto in a language that you didn't know before.
So thanks for coming on and talking about it.
Rudy: Oh, thanks for having me. I really appreciate y'all. This is dope.
Andrew: Justin, you're muted now.
Justin: ~~ ~~Just to repeat what Andrew said. Uh, yeah, you're the work you're doing is awesome. The, the, the amount of time that you're doing at it in is mind blowing. Uh, you mentioned a few times, you're like, I'm, you know, I'm not a staff engineer or somewhere. Well, you could be, cause this is impressive. So, uh, yeah, thanks for all the work that you're doing.
It makes the whole community better. Uh, and I'm excited to see what you do.
Rudy: Appreciate y'all. Thank you.