Podcasts about BitTorrent

This year's NABshow arrives amid a grim outlook for our media broadcast industry. Will we address these issues at the conference or skip them? (00:00).Peter is joined by Matt Mason, a former pirate radio DJ, innovator at BitTorrent and Sony Pictures, and currently the CEO of Basejump, to help us check in on Metaverse storytelling and whether your content creation pipeline can be redefined at the hands of decentralized tech and gaming platforms? (01:28).Then, they discuss the transformative power that NFTs can have on creative control of your content (18:20).Finally, Peter is joined by industry pundit Alan Lasky to unpack NABshow's attempt to embrace changing markets while addressing the tough questions posed on our industry (30:37).Matt Mason is the CEO of BaseJump.You can learn more about Matt Mason and connect on LinkedIn.You can learn more about BaseJump at their website.And read Matt's book on culture and technology, "The Pirate's Dilemma".You can follow Alan Lasky's opinions and connect with him on LinkedIn.Follow the Virtually Everything! Podcast on Instagram: https://www.instagram.com/virtually.live/?hl=enIf you want to send an email with feedback or show suggestions, you can reach us at virtually.everything@vustudio.com.Otherwise you can:Find Peter on LinkedIn https://www.linkedin.com/in/peterfrelik/Bye for now!-------------Here's the 20% you need to know about this episode:1. Decentralized tech and gaming platforms can redefine content creation pipelines.2. AO, a decentralized supercomputer distributed across global laptops and servers, will enable robust and permanent data storage.3. The Metaverse should be viewed as a collection of digital assets and stories rather than specific platforms.4. Base Jump is a loadout screen for the entire next 3D spatial internet, enabling users to equip avatars and digital assets for use across different platforms and metaverses.5. NFTs allow creators to maintain ownership and control over their digital creations, opening new monetization avenues and reducing dependency on centralized platforms.6. The rise in interest for decentralization is attributed to a reaction against failing legacy systems and the need for more democratic, user-controlled digital experiences.7. The current mood in the media and entertainment industry is questioning the return on investment for production gear.8. NAB and similar shows should adapt to new decentralized and AI-driven production models to remain relevant to newer, self-employed content creators.9. Economic conditions and changing cultural attitudes towards digital and decentralized technologies are impacting industry trends and conference dynamics.The Virtually Everything! Podcast is presented by Vū Technologies. #VuStudio #ContentAtTheSpeedOfThoughtBye for now! Hosted on Acast. See acast.com/privacy for more information.

Mark and Kenny begin Season Seven: Defending the Crown with a detailed discussion of Madonna's most politically urgent (and last - for now!) film project, in collaboration with Steven Klein as well as the debut of a now oft-covered Elliott Smith song. Topics include channeling the source and upping security, Instagram, VICE and BitTorrent, cutting edge lingerie, Danny Tull, troublesome rollouts and surprise drops, the illusions of the mid-2010s, the Daddy Syndrome, Jean-Luc Goddard, The Night Porter, David Lynch's influences, rebirth and performance, Anne Sexton, Ariana Grande, the Gagosian Gallery, Rocco Ritchie, Anderson Cooper and the GLAAD Awards, the importance of props, Amnesty International, Pussy Riot and the continuing struggle and fight to protect human rights and the freedom of speech - globally. Madonna's in the kitchen getting dragged across the floor and getting sued for one million dollars! Thank you - as always - for listening. We are so excited to be resuming our conversations with each and every one of you. It's a Revolution of Love!

Paul Frazee is the CTO of Bluesky. He previously worked on the Beaker browser and the peer-to-peer social media protocol Secure Scuttlebutt. Paul discusses how Bluesky and ATProto got started, scaling up a social media site, what makes ATProto decentralized, lessons ATProto learned from previous peer-to-peer projects, and the challenges of content moderation. Episode transcript available here. My Bluesky profile. -- Related Links Bluesky ATProtocol ATProto for distributed systems engineers Bluesky and the AT Protocol: Usable Decentralized Social Media Decentralized Identifiers (DIDs) ActivityPub Webfinger Beaker web browser Secure Scuttlebutt -- Transcript You can help correct transcripts on GitHub. [00:00:00] Jeremy: Today I am talking to Paul Frazee. He's the current CTO of bluesky, and he previously worked on other decentralized applications like Beaker and Secure Scuttlebutt. [00:00:15] Paul: Thanks for having me. What's bluesky [00:00:16] Jeremy: For people who aren't familiar with bluesky, what is it? [00:00:20] Paul: So bluesky is an open social network, simplest way to put it, designed in particular for high scale. That's kind of one of the big requirements that we had when we were moving into it. and it is really geared towards making sure that the operation of the social network is open amongst multiple different organizations. [00:00:44] So we're one of the operators, but other folks can come in, spin up the software, all the open source software, and essentially have a full node with a full copy of the network active users and have their users join into our network. And they all work functionally as one shared application. [00:01:03] Jeremy: So it, it sounds like it's similar to Twitter but instead of there being one Twitter, there could be any number and there is part of the underlying protocol that allows them to all connect to one another and act as one system. [00:01:21] Paul: That's exactly right. And there's a metaphor we use a lot, which is comparing to the web and search engines, which actually kind of matches really well. Like when you use Bing or Google, you're searching the same web. So on the AT protocol on bluesky, you use bluesky, you use some alternative client or application, all the same, what we're we call it, the atmosphere, all one shared network, [00:01:41] Jeremy: And more than just the, the client. 'cause I think sometimes when people think of a client, they'll think of, I use a web browser. I could use Chrome or Firefox, but ultimately I'm connecting to the same thing. But it's not just people running alternate clients, right? [00:01:57] Paul: Their own full backend to it. That's right. Yeah. Yeah. The anchoring point on that being the fire hose of data that runs the entire thing is open as well. And so you start up your own application, you spin up a service that just pipes into that fire hose and taps into all the activity. History of AT Protocol [00:02:18] Jeremy: Talking about this underlying protocol maybe we could start where this all began so people get some context for where this all came from. [00:02:28] Paul: For sure. All right, so let's wind the clock back here in my brain. We started out 2022, right at the beginning of the year. We were formed as a, essentially a consulting company outside of Twitter with a contract with Twitter. And, uh, our goal was to build a protocol that could run, uh, Twitter, much like the way that we just described, which set us up with a couple of pretty specific requirements. [00:02:55] For one, we had to make sure that it could scale. And so that ended up being a really important first requirement. and we wanted to make sure that there was a strong kind of guarantees that the network doesn't ever get captured by any one operator. The idea was that Twitter would become the first, uh, adopter of the technology. [00:03:19] Other applications, other services would begin to take advantage of it and users would be able to smoothly migrate their accounts in between one or the other at any time. Um, and it's really, really anchored in a particular goal of just deconstructing monopolies. Getting rid of those moats that make it so that there's a kind of a lack of competition, uh, between these things. [00:03:44] And making sure that, if there was some kind of reason that you decided you're just not happy with what direction this service has been going, you move over to another one. You're still in touch with all the folks you were in touch with before. You don't lose your data. You don't lose your, your your follows. Those were the kind of initial requirements that we set out with. The team by and large came from, the decentralized web, movement, which is actually a pretty, large community that's been around since, I wanna say around 2012 is when we first kind of started to form. It got really made more specifically into a community somewhere around 2015 or 16, I wanna say. [00:04:23] When the internet archives started to host conferences for us. And so that gave us kind of a meeting point where all started to meet up there's kind of three schools of thought within that movement. There was the blockchain community, the, federation community, and the peer-to-peer community. [00:04:43] And so blockchain, you don't need to explain that one. You got Federation, which was largely ActivityPub Mastodon. And then peer-to-peer was IPFS, DAT protocol, um, secure scuttlebutt. But, those kinds of BitTorrent style of technologies really they were all kind of inspired by that. [00:05:02] So these three different kind of sub communities we're all working, independently on different ways to attack how to make these open applications. How do you get something that's a high scale web application without one corporation being the only operator? When this team came together in 2022, we largely sourced from the peer-to-peer group of the decentralized community. Scaling limitations of peer-to-peer [00:05:30] Paul: Personally, I've been working in the space and on those kinds of technologies for about 10 years at that stage. And, the other folks that were in there, you know, 5-10 each respectively. So we all had a fair amount of time working on that. And we had really kind of hit some of the limitations of doing things entirely using client devices. We were running into challenges about reliability of connections. Punching holes to the individual device is very hard. Synchronizing keys between the devices is very hard. Maintaining strong availability of the data because people's devices are going off and on, things like that. Even when you're using the kind of BitTorrent style of shared distribution, that becomes a challenge. [00:06:15] But probably the worst challenge was quite simply scale. You need to be able to create aggregations of a lot of behavior even when you're trying to model your application as largely peer wise interactions like messaging. You might need an aggregation of accounts that even exist, how do you do notifications reliably? [00:06:37] Things like that. Really challenging. And what I was starting to say to myself by the end of that kind of pure peer-to-peer stent was that it can't be rocket science to do a comment section. You know, like at some point you just ask yourself like, how, how hard are we willing to work to, to make these ideas work? [00:06:56] But, there were some pretty good pieces of tech that did come out of the peer-to-peer world. A lot of it had to do with what I might call a cryptographic structure. things like Merkel trees and advances within Merkel Trees. Ways to take data sets and reduce them down to hashes so that you can then create nice signatures and have signed data sets at rest at larger scales. [00:07:22] And so our basic thought was, well, all right, we got some pretty good tech out of this, but let's drop that requirement that it all run off of devices. And let's get some servers in there. And instead think of the entire network as a peer-to-peer mesh of servers. That's gonna solve your scale problem. [00:07:38] 'cause you can throw big databases at it. It's gonna solve your availability problems, it's gonna solve your device sync problems. But you get a lot of the same properties of being able to move data sets between services. Much like you could move them between devices in the peer-to-peer network without losing their identifiers because you're doing this in direction of, cryptographic identifiers to the current host. [00:08:02] That's what peer-to-peer is always doing. You're taking like a public key or hash and then you're asking the network, Hey, who has this? Well, if you just move that into the server, you get the same thing, that dynamic resolution of who's your active host. So you're getting that portability that we wanted real bad. [00:08:17] And then you're also getting that kind of in meshing of the different services where each of them is producing these data sets that they can sink from each other. So take peer-to-peer and apply it to the server stack. And that was our kind of initial thought of like, Hey, you know what? This might work. [00:08:31] This might solve the problems that we have. And a lot of the design fell out from that basic mentality. Crytographic identifiers and domain names [00:08:37] Jeremy: When you talk about these cryptographic identifiers, is the idea that anybody could have data about a person, like a message or a comment, and that could be hosted different places, but you would still know which person that originally came from. Is that, is that the goal there? [00:08:57] Paul: That's exactly it. Yeah. Yeah. You wanna create identification that supersedes servers, right? So when you think about like, if I'm using Twitter and I wanna know what your posts are, I go to twitter.com/jeremy, right? I'm asking Twitter and your ID is consequently always bound to Twitter. You're always kind of a second class identifier. [00:09:21] We wanted to boost up the user identifier to be kind of a thing freestanding on its own. I wanna just know what Jeremy's posts are. And then once you get into the technical system it'll be designed to figure out, okay, who knows that, who can answer that for you? And we use cryptographic identifiers internally. [00:09:41] So like all the data sets use these kind of long URLs to identify things. But in the application, the user facing part, we used domain names for people. Which I think gives the picture of how this all operates. It really moves the user accounts up into a free standing first class identifier within the system. [00:10:04] And then consequently, any application, whatever application you're using, it's really about whatever data is getting put into your account. And then that just exchanges between any application that anybody else is using. [00:10:14] Jeremy: So in this case, it sounds like the identifier is some long string that, I'm not sure if it's necessarily human readable or not. You're shaking your head no. [00:10:25] Paul: No. [00:10:26] Jeremy: But if you have that string, you know it's for a specific person. And since it's not really human readable, what you do is you put a layer on top of it which in this case is a domain that somebody can use to look up and find the identifier. [00:10:45] Paul: Yeah, yeah, yeah. So we just use DNS. Put a TXT record in there, map into that long string, or you could do a .well-known file on a web server if that's more convenient for you. And then the ID that's behind that, the non-human readable one, those are called DIDs which is actually a W3C spec. Those then map to a kind of a certificate. What you call a DID document that kind of confirms the binding by declaring what that domain name should be. So you get this bi-directional binding. And then that certificate also includes signing keys and active servers. So you pull down that certificate and that's how the discovery of the active server happens is through the DID system. What's stored on a PDS [00:11:29] Jeremy: So when you refer to an active server what is that server and what is that server storing? [00:11:35] Paul: It's kinda like a web server, but instead of hosting HTML, it's hosting a bunch of JSON records. Every user has their own document store of JSON documents. It's bucketed into collections. Whenever you're looking up somebody on the network you're gonna get access to that repository of data, jump into a collection. [00:11:58] This collection is their post collection. Get the rkey (Record Key), and then you're pulling out JSON at the end of it, which is just a structured piece of stuff saying here's the CreatedAt, here's the text, here's the type, things like that. One way you could look at the whole system is it's a giant, giant database network. Servers can change, signing keys change, but not DID [00:12:18] Jeremy: So if someone's going to look up someone's identifier, let's say they have the user's domain they have to go to some source, right? To find the user's data. You've mentioned, I think before, the idea that this is decentralized and by default I would, I would picture some kind of centralized resource where I send somebody a domain and then they give me back the identifier and the links to the servers. [00:12:46] So, so how does that work in practice where it actually can be decentralized? [00:12:51] Paul: I mentioned that your DID that non-human readable identifier, and that has that certificate attached to it that lists servers and signing keys and things like that. [00:13:00] So you're just gonna look up inside that DID document what that server is your data repository host. And then you contact that guy and say, all right, I'm told you're hosting this thing. Here's the person I'm looking for, hand over the hand over the data. It's really, you know, pretty straightforward. [00:13:18] The way that gets decentralized is by then to the fact that I could swap out that active server that's in my certificate and probably wanna rotate the signing keys 'cause I've just changed the, you know. I don't want to keep using the same signing keys as I was using previously because I just changed the authority. [00:13:36] So that's the migration change, change the hosting server, change out the signing keys. Somebody that's looking for me now, they're gonna load up my document, my DID document. They're gonna say, okay, new server, new keys. Pull down the data. Looks good, right? Matches up with the DID doc. [00:13:50] So that's how you get that level of portability. But when those changes happen, the DID doesn't change, right? The DID document changes. So there's the level of indirection there and that's pretty important because if you don't have a persistent identifier whenever you're trying to change out servers, all those backlinks are gonna break. [00:14:09] That's the kind of stuff that stops you from being able to do clean migrations on things like web-based services. the only real option is to go out and ask everybody to update their data. And when you're talking about like interactions on the social network, like people replying to each other, there's no chance, right? [00:14:25] Every time somebody moves you're gonna go back and modify all those records. You don't even control all the records from the top down 'cause they're hosted all over the web. So it's just, you can't do it. Generally we call this account portability, that you're kinda like phone number portability that you can change your host, but, so that part's portable, but the ID stays the same. [00:14:45] And keeping that ID the same is the real key to making sure that this can happen without breaking the whole system. [00:14:52] Jeremy: And so it, it sounds like there's the decentralized id, then there's the decentralized ID document that's associated with that points you to where the actual location of your, your data, your posts, your pictures and whatnot. but then you also mentioned that they could change servers. [00:15:13] So let's say somebody changes where their data is, is stored, that would change the servers, I guess, in their document. But [00:15:23] then how do all of these systems. Know okay. I need to change all these references to your old server, to these new servers, [00:15:32] Paul: Yeah. Well, the good news is that you only have to, you, you got the public data set of all the user's activity, and then you have like internal caches of where the current server is. You just gotta update those internal caches when you're trying to contact their server. Um, so it's actually a pretty minimal thing to just like update like, oh, they moved, just start talking to update my, my table, my Redis, that's holding onto that kind of temporary information, put it on ttl, that sort of thing. Most communication won't be between servers, it will be from event streams [00:16:01] Paul: And, honestly, in practice, a fair amount of the system for scalability reasons doesn't necessarily work by servers directly contacting each other. It's actually a little bit more like how, I told you before, I'm gonna use this metaphor a lot, the search engines with the web, right? What we do is we actually end up crawling the repositories that are out in the world and funneling them into event streams like a Kafka. And that allows the entire system to act like a data processing pipeline where you're just tapping into these event streams and then pushing those logs into databases that produce these large scale aggregations. [00:16:47] So a lot of the application behavior ends up working off of these event logs. If I reply to somebody, for instance, I don't necessarily, it's not, my server has to like talk to your server and say, Hey, I'm replying to you. What I do is I just publish a reply in my repository that gets shot out into the event logs, and then these aggregators pick up that the reply got created and just update their database with it. [00:17:11] So it's not that our hosting servers are constantly having to send messages with each other, you actually use these aggregators to pull together the picture of what's happening on the network. [00:17:22] Jeremy: Okay, so like you were saying, it's an event stream model where everybody publishes the events the things that they're doing, whether that's making a new post, making a reply, that's all being posted to this event stream. And then everybody who provides, I'm not sure if instances is the right term, but an implementation of the atmosphere protocol (Authenticated Transfer protocol). [00:17:53] They are listening for all those changes and they don't necessarily have to know that you moved servers because they're just listening for the events and you still have the same identifier. [00:18:10] Paul: Generally speaking. Yeah. 'cause like if you're listening to one of these event streams what you end up looking for is just the signature on it and making sure that the signature matches up. Because you're not actually having to talk to their live server. You're just listening to this relay that's doing this aggregation for you. [00:18:27] But I think actually to kind of give a little more clarity to what you're talking about, it might be a good idea to refocus how we're talking about the system here. I mentioned before that our goal was to make a high scale system, right? We need to handle a lot of data. If you're thinking about this in the way that Mastodon does it, the ActivityPub model, that's actually gonna give you the wrong intuition. Designing the protocol to match distributed systems practices (Event sourcing / Stream processing) [00:18:45] Paul: 'cause we chose a dramatically different system. What we did instead was we picked up, essentially the same practices you're gonna use for a data center, a high scale application data center, and said, all right, how do you tend to build these sorts of things? Well, what you're gonna do is you're gonna have, multiple different services running different purposes. [00:19:04] It gets pretty close to a microservices approach. You're gonna have a set of databases, and then you're going to, generally speaking for high scale, you're gonna have some kind of a kafka, some kind of a event log that you are tossing changes about the state of these databases into. And then you have a bunch of secondary systems that are tapping into the event log and processing that into, the large scale, databases like your search index, your, nice postgres of user profiles. [00:19:35] And that makes sure that you can get each of these different systems to perform really well at their particular task, and then you can detach them in their design. for instance, your primary storage can be just a key value store that scales horizontally. And then on the event log, you, you're using a Kafka that's designed to handle. [00:19:58] Particular semantics of making sure that the messages don't get dropped, that they come through at a particular throughput. And then you're using, for us, we're using like ScyllaDB for the big scale indexes that scales horizontally really well. So it's just different kind of profiles for different pieces. [00:20:13] If you read Martin Kleppman's book, data Intensive applications I think it's called or yeah. A lot of it gets captured there. He talks a lot about this kind of thing and it's sometimes called a kappa architecture is one way this is described, event sourcing is a similar term for it as well. [00:20:30] Stream processing. That's pretty standard practices for how you would build a traditional high scale service. so if you take, take this, this kind of microservice architecture and essentially say, okay, now imagine that each of the services that are a part of your data center could be hosted by anybody, not just within our data center, but outside of our data center as well and should be able to all work together. [00:20:57] Basically how the AT Proto is designed. We were talking about the data repository hosts. Those are just the primary data stores that they hold onto the user keys and they hold onto those JSON records. And then we have another service category we call Relay that just crawls those data repositories and sucks that in that fire hose of data we were talking about that event log. App views pull data from relay and produces indexes and threads [00:21:21] Paul: And then we have what we call app views that sit there and tail the index and tail the log, excuse me, and produce indexes off of it, they're listening to those events and then like, making threads like okay, that guy posted, that guy replied, that guy replied. [00:21:37] That's a thread. They assemble it into that form. So when you're running an application, you're talking to the AppView to read the network, and you're talking to the hosts to write to the network, and each of these different pieces sync up together in this open mesh. So we really took a traditional sort of data center model and just turned it inside out where each piece is a part of the protocol and communicate it with each other and therefore anybody can join into that mesh. [00:22:07] Jeremy: And to just make sure I am tracking the data repository is the data about the user. So it has your decentralized identifier, it has your replies, your posts, And then you have a relay, which is, its responsibility, is to somehow find all of those data repositories and collect them as they happen so that it can publish them to some kind of event stream. [00:22:41] And then you have the AppView which it's receiving messages from the relay as they happen, and then it can have its own store and index that for search. It can collect them in a way so that it can present them onto a UI. That's sort of thing that's the user facing part I suppose. [00:23:00] Paul: Yeah, that's exactly it. And again, it's, it's actually quite similar to how the web works. If you combine together the relay and the app view, you got all these different, you know, the web works where you got all these different websites, they're hosting their stuff, and then the search engine is going around, aggregating all that data and turning it into a search experience. [00:23:19] Totally the same model. It's just being applied to, more varieties of data, like structured data, like posts and, and replies, follows, likes, all that kinda stuff. And then instead of producing a search application at the end. I mean, it does that too, but it also produces a, uh, you know, timelines and threads and, um, people's profiles and stuff like that. [00:23:41] So it's actually a pretty bog standard way of doing, that's one of the models that we've seen work for large scale decentralized systems. And so we're just transposing it onto something that kind of is more focused towards social applications [00:23:58] Jeremy: So I think I'm tracking that the data repository itself, since it has your decentralized identifier and because the data is cryptographically signed, you know, it's from a specific user. I think the part that I am still not quite sure about is the relays. I, I understand if you run all the data repositories, you know where they are, so you know how to collect the data from them. [00:24:22] But if someone's running another system outside of your organization, how do they find, your data repositories? Or do they have to connect to your relay? What's the intention for that? Data hosts request relays to pull their data [00:24:35] Paul: That logic runs, again, really similar to how search engines find out about websites. So there is actually a way for, one of these, data hosts to contact Relay and say, Hey, I exist. You know, go ahead and get my stuff. And then it'll be up to the relay to decide like if they want it or not. [00:24:52] Right now, generally we're just like, yeah, you know, we, we want it. But as you can imagine, as the thing matures and gets to higher scale, there might be some trust kind of things to worry about, you know? So that's kind of the naive operation that currently exists. But over time as the network gets bigger and bigger, it'll probably involve some more traditional kind of spiraling behaviors because as more relays come into the system, each of these hosts, they're not gonna know who to talk to. Relays can bootstrap who they know about by talking to other relays [00:25:22] Paul: You're trying to start a new relay. What they're gonna do is they're going to discover all of the different users that exist in the system by looking at what data they have to start with. Probably involve a little bit of a manual feeding in at first, whenever I'm starting up a relay, like, okay, there's bluesky's relay. [00:25:39] Lemme just pull what they know. And then I go from there. And then anytime you discover a new user you don't have, you're like, oh, I wanna look them up. Pull them into the relay too. Right. So there's a, pretty straightforward, discovery process that you'll just have to bake into a relay to, to make sure you're calling as much the network as possible. ActivityPub federation vs AT Proto [00:25:57] Jeremy: And so I don't think we've defined the term federation, but maybe you could explain what that is and if that is what this is. [00:26:07] Paul: We are so unsure. [00:26:10] Jeremy: Okay. [00:26:11] Paul: Yeah. This has jammed is up pretty bad. Um, because I think everybody can, everybody pretty strongly agrees that ActivityPub is federation, right? and ActivityPub kind of models itself pretty similarly to email in a way, like the metaphors they use is that there's inboxes and outboxes and, and every ActivityPub server they're standing up the full vertical stack. [00:26:37] They set up, the primary hosting, the views of the data that's happening there. the interface for the application, all of it, pretty traditional, like close service, but then they're kind of using the perimeter. they're making that permeable by sending, exchanging, essentially mailing records to each other, right? [00:26:54] That's their kind of logic of how that works. And that's pretty much in line with, I think, what most people think of with Federation. Whereas what we're doing isn't like that we've cut, instead of having a bunch of vertical stacks communicating horizontally with each other, we kind of sliced in the other direction. [00:27:09] We sliced horizontally into, this microservices mesh and have all the different, like a total mix and match of different microservices between different operators. Is that federation? I don't know. Right. we tried to invent a term, didn't really work, you know, At the moment, we just kind of don't worry about it that much, see what happens, see what the world sort of has to say to us about it. [00:27:36] and beyond that, I don't know. [00:27:42] Jeremy: I think people probably are thinking of something like, say, a Mastodon instance when you're, when you're talking about everything being included, The webpage where you view the posts, the Postgres database that's keeping the messages. [00:28:00] And that same instance it's responsible for basically everything. [00:28:06] Paul: mm-Hmm [00:28:06] Jeremy: And I believe what you're saying is that the difference with, the authenticated transfer protocol, is that the [00:28:15] Paul: AT Protocol, Yep. [00:28:17] Jeremy: And the difference there is that you've, at the protocol level, you've split it up into the data itself, which can be validated completely separately from other parts of the system. [00:28:31] You could have the JSON files on your hard drive and somebody else can have that same JSON file and they would know that who the user is and that these are real things that user posted. That's like a separate part. And then the relay component that looks for all these different repositories that has people's data, that can also be its own independent thing where its job is just to output events. [00:29:04] And that can exist just by itself. It doesn't need the application part, the, the user facing part, it can just be this event stream on itself. and that's the part where it sounds like you can make decisions on who to, um, collect data from. I guess you have to agree that somebody can connect to you and get the users from your data repositories. [00:29:32] And likewise, other people that run relays, they also have to agree to let you pull the users from theirs. [00:29:38] Paul: Yeah, that's right. Yeah. [00:29:41] Jeremy: And so I think the Mastodon example makes sense. And, but I wonder if the underlying ActivityPub protocol forces you to use it in that way, in like a whole full application that talks to another full application. [00:29:55] Or is it more like that's just how people tend to use it and it's not necessarily a characteristic of the protocol. [00:30:02] Paul: Yeah, that's a good question actually. so, you know, generally what I would say is pretty core to the protocol is the expectations about how the services interact with each other. So the mailbox metaphor that's used in ActivityPub, that design, if I reply to you, I'll update my, local database with what I did, and then I'll send a message over to your server saying, Hey, by the way, add this reply. [00:30:34] I did this. And that's how they find out about things. That's how they find out about activity outside of their network. that's the part that as long as you're doing ActivityPub, I suspect you're gonna see reliably happening. That's that, I can say for sure that's a pretty tight requirement. [00:30:50] That's ActivityPub. If you wanted to split it up the way we're talking about, you could, I don't know, I don't know if you necessarily would want to. Because I don't know. That's actually, I think I'd have to dig into their stack a little bit more to see how meaningful that would be. I do know that there's some talk of introducing a similar kind of an aggregation method into the ActivityPub world which I believe they're also calling a relay and to make things even more complicated. [00:31:23] And NOSTR has a concept of a relay. So these are three different protocols that are using this term. I think you could do essentially what a search engine does on any of these things. You could go crawling around for the data, pull them into a fire hose, and then, tap into that aggregation to produce, bigger views of the network. [00:31:41] So that principle can certainly apply anywhere. AT Protocol, I think it's a little bit, we, we focused in so hard from that on that from the get go, we focus really hard on making sure that this, the data is, signed at rest. That's why it's called the authenticated transfer protocol. And that's a nice advantage to have when you're running a relay like this because it means that you don't have to trust the relay. [00:32:08] Like generally speaking, when I look at results from Google, you know, I'm trusting pretty well that they're accurately reflecting what's on the website, which is fine. You know, there's, that's not actually a huge risk or anything. But whenever you're trying to build entire applications and you're using somebody else's relay, you could really run into things where they say like, oh, you know what Paul tweeted the other day, you know, I hate dogs. [00:32:28] They're like, no, I didn't. That's a lie, right? You just sneak in Little lies like that over a while, it becomes a problem. So having the signatures on the data is pretty important. You know, if you're gonna be trying to get people to cooperate, uh, you gotta manage the trust model. I know that ActivityPub does have mechanisms for signed records. Issuers with ActivityPub identifiers [00:32:44] Paul: I don't know how deep they go if they could fully replace that, that utility. and then Mastodon ActivityPub, they also use a different identifier system that they're actually taking a look at DIDs um, right now, I don't know what's gonna happen there. We're, we're totally on board to, you know, give any kind of insight that we got working on 'em. [00:33:06] But at, at the moment, they use I think it's WebFinger based identifiers they look like emails. So you got host names in there and those identifiers are being used in the data records. So you don't get that continuous identifier. They actually do have to do that hey, I moved update your records sort of thing. [00:33:28] And that causes it to, I mean, it works like decently well, but not as well as it could. They got us to the point where it moves your profile over and you update all the folks that were following you so they can update their follow records, but your posts, they're not coming right, because that's too far into that mesh of interlinking records. [00:33:48] There's just no chance. So that's kind of the upper limit on that, it's a different set of choices and trade-offs. You're always kind of asking like, how important is the migration? Does that work out? Anyway, now I'm just kind of digging into differences between the two here. Issues with an identifier that changes and updating old records [00:34:07] Jeremy: So you were saying that with ActivityPub, all of the instances need to be notified that you've changed your identifier but then all of the messages that they had already received. They don't point to the new identifier somehow. [00:34:24] Paul: Yeah. You run into basically just the practicalities of actual engineering with that is what happens, right? Because if you imagine you got a multimillion user social network. They got all their posts. Maybe the user has like, let's say a thousand posts and 10,000 likes. And that, activity can range back three years. [00:34:48] Let's say they changed their identifier, and now you need to change the identifier of all those records. If you're in a traditional system that's already a tall order, you're going back and rewriting a ton of indexes, Anytime somebody replied to you, they have these links to your posts, they're now, you've gotta update the identifiers on all of those things. [00:35:11] You could end up with a pretty significant explosion of rewrites that would have to occur. Now that's, that's tough. If you're in a centralized model. If you're in a decentralized one, it's pretty much impossible because you're now, when you notify all the other servers like, Hey, this, this changed. How successful are all of them at actually updating that, that those, those pointers, it's a good chance that there's things are gonna fall out of correctness. that's just a reality of it. And if, so, if you've got a, if you've got a mutable identifier, you're in trouble for migrations. So the DID is meant to keep it permanent and that ends up being the anchoring point. If you lose control of your DID well, that's it. Managing signing keys by server, paper key reset [00:35:52] Paul: Your, your account's done. We took some pretty traditional approaches to that, uh, where the signing keys get managed by your hosting server instead of like trying to, this may seem like really obvious, but if you're from the decentralization community, we spend a lot of time with blockchains, like, Hey, how do we have the users hold onto their keys? [00:36:15] You know, and the tooling on that is getting better for what it's worth. We're starting to see a lot better key pair management in like Apple's ecosystem and Google's ecosystem, but it's still in the range of like, nah, people lose their keys, you know? So having the servers manage those is important. [00:36:33] Then we have ways of exporting paper keys so that you could kind of adversarially migrate if you wanted to. That was in the early spec we wanted to make sure that this portability idea works, that you can always migrate your accounts so you can export a paper key that can override. [00:36:48] And that was how we figured that out. Like, okay, yeah, we don't have to have everything getting signed by keys that are on the user's devices. We just need these master backup keys that can say, you know what? I'm done with that host. No matter what they say, I'm overriding what they, what they think. and that's how we squared that one. [00:37:06] Jeremy: So it seems like one of the big differences with account migration is that with ActivityPub, when you move to another instance, you have to actually change your identifier. [00:37:20] And with the AT protocol you're actually not allowed to ever change that identifier. And maybe what you're changing is just you have say, some kind of a lookup, like you were saying, you could use a domain name to look that up, get a reference to your decentralized identifier, but your decentralized identifier it can never change. [00:37:47] Paul: It, it, it can't change. Yeah. And it shouldn't need to, you know what I mean? It's really a total disaster kind of situation if that happens. So, you know that it's designed to make sure that doesn't happen in the applications. We use these domain name handles to, to identify folks. And you can change those anytime you want because that's really just a user facing thing. [00:38:09] You know, then in practice what you see pretty often is that you may, if you change hosts, if you're using, we, we give some domains to folks, you know, 'cause like not everybody has their own domain. A lot of people do actually, to our surprise, people actually kind of enjoy doing that. But, a lot of folks are just using like paul.bsky.social as their handle. [00:38:29] And so if you migrated off of that, you probably lose that. Like your, so your handle's gonna change, but you're not losing the followers and stuff. 'cause the internal system isn't using paul.bsky.social, it's using that DID and that DID stays the same. Benefits of domain names, trust signal [00:38:42] Jeremy: Yeah. I thought that was interesting about using the domain names, because when you like you have a lot of users, everybody's got their own sub-domain. You could have however many millions of users. Does that become, does that become an issue at some point? [00:39:00] Paul: Well, it's a funny thing. I mean like the number of users, like that's not really a problem 'cause you run into the same kind of namespace crowding problem that any service is gonna have, right? Like if you just take the subdomain part of it, like the name Paul, like yeah, only, you only get to have one paul.bsky.social. [00:39:15] so that part of like, in terms of the number of users, that part's fine I guess. Uh, as fine as ever. where gets more interesting, of course is like, really kind of around the usability questions. For one, it's, it's not exactly the prettiest to always have that B sky.social in there. If we, if we thought we, if we had some kind of solution to that, we would use it. [00:39:35] But like the reality is that, you know, now we're, we've committed to the domain name approach and some folks, you know, they kind of like, ah, that's a little bit ugly. And we're like, yeah that's life. I guess the plus side though is that you can actually use like TLD the domain. It's like on pfrazee.com. [00:39:53] that starts to get more fun. it can actually act as a pretty good trust signal in certain scenarios. for instance, well-known domain names like nytimes.com, strong authentication right there, we don't even need a blue check for it. Uh, similarly the .gov, domain name space is tightly regulated. [00:40:14] So you actually get a really strong signal out of that. Senator Wyden is one of our users and so he's, I think it's wyden.senate.gov and same thing, strong, you know, strong identity signal right there. So that's actually a really nice upside. So that's like positives, negatives. [00:40:32] That trust signal only works so far. If somebody were to make pfrazee.net, then that can be a bit confusing. People may not be paying attention to .com vs .net, so it's not, I don't wanna give the impression that, ah, we've solved blue checks. It's a complicated and multifaceted situation, but, it's got some juice. [00:40:54] It's also kinda nice too, 'cause a lot of folks that are doing social, they're, they've got other stuff that they're trying to promote, you know? I'm pretty sure that, uh, nytimes would love it if you went to their website. And so tying it to their online presence so directly like that is a really nice kind of feature of it. [00:41:15] And tells a I think a good story about what we're trying to do with an open internet where, yeah, everybody has their space on the internet where they can do whatever they want on that. And that's, and then thethese social profiles, it's that presence showing up in a shared space. It's all kind of part of the same thing. [00:41:34] And that that feels like a nice kind of thing to be chasing, you know? And it also kind of speaks well to the naming worked out for us. We chose AT Protocol as a name. You know, we back acronymed our way into that one. 'cause it was a @ simple sort of thing. But like, it actually ended up really reflecting the biggest part of it, which is that it's about putting people's identities at the front, you know, and make kind of promoting everybody from a second class identity that's underneath Twitter or Facebook or something like that. [00:42:03] Up into. Nope, you're freestanding. You exist as a person independently. Which is what a lot of it's about. [00:42:12] Jeremy: Yeah, I think just in general, not necessarily just for bluesky, if people had more of an interest in getting their own domain, that would be pretty cool if people could tie more of that to something you basically own, right? [00:42:29] I mean, I guess you're leasing it from ICANN or whatever, but, [00:42:33] yeah, rather than everybody having an @Gmail, Outlook or whatever they could actually have something unique that they control more or less. [00:42:43] Paul: Yeah. And we, we actually have a little experimental service for registering domain names that we haven't integrated into the app yet because we just kind of wanted to test it out and, and kind of see what that appetite is for folks to register domain names way higher than you'd think we did that early on. [00:43:01] You know, it's funny when you're coming from decentralization is like an activist space, right? Like it's a group of people trying to change how this tech works. And sometimes you're trying to parse between what might come off as a fascination of technologists compared to what people actually care about. [00:43:20] And it varies, you know, the domain name thing to a surprising degree, folks really got into that. We saw people picking that up almost straight away. More so than certainly we ever predicted. And I think that's just 'cause I guess it speaks to something that people really get about the internet at this point. [00:43:39] Which is great. We did a couple of other things that are similar and we saw varied levels of adoption on them. We had similar kinds of user facing, opening up of the system with algorithms and with moderation. And those have both been pretty interesting in and of themselves. Custom feed algorithms [00:43:58] Paul: So with algorithms, what we did was we set that up so that anybody can create a new feed algorithm. And this was kind of one of the big things that you run into whenever you use the app. If you wanted to create a new kind of for you feed you can set up a service somewhere that's gonna tap into that fire hose, right? [00:44:18] And then all it needs to do is serve a JSON endpoint. That's just a list of URLs, but like, here's what should be in that feed. And then the bluesky app will pick that up and, and send that, hydrate in the content of the posts and show that to folks. I wanna say this is a bit of a misleading number and I'll explain why but I think there's about 35,000 of these feeds that have been created. [00:44:42] Now, the reason it's little misleading is that, I mean, not significantly, but it's not everybody went, sat down in their IDE and wrote these things. Essentially one of our users created, actually multiple of our users made little platforms for building these feeds, which is awesome. That's the kinda thing you wanna see because we haven't gotten around to it. [00:44:57] Our app still doesn't give you a way to make these things. But they did. And so lots of, you know, there it is. Cool. Like, one, one person made a kind of a combinatorial logic thing that's like visual almost like scratch, it's like, so if it has this hashtag and includes these users, but not those users, and you're kind of arranging these blocks and that constructs the feed and then probably publish it on your profile and then folks can use it, you know? [00:45:18] And um, so that has been I would say fairly successful. Except, we had one group of hackers do put in a real effort to make a replacement for you feed, like magic algorithmic feed kind of thing. And then they kind of kept up going for a while and then ended up giving up on it. Most of what we see are actually kind of weird niche use cases for feeds. [00:45:44] You get straightforward ones, like content oriented ones like a cat feed, politics feed, things like that. It's great, some of those are using ML detection, so like the cat feed is ML detection, so sometimes you get like a beaver in there, but most of the time it's a cat. And then we got some ones that are kind of a funny, like change in the dynamic of freshness. [00:46:05] So, uh, or or selection criteria, things that you wouldn't normally see. Um, but because they can do whatever they want, you know, they try it out. So like the quiet posters ended up being a pretty successful one. And that one just shows people you're following that don't post that often when they do just those folks. [00:46:21] It ended up being, I use that one all the time because yeah, like they get lost in the noise. So it's like a way to keep up with them. Custom moderation and labeling [00:46:29] Paul: The moderation one, that one's a a real interesting situation. What we did there essentially we wanted to make sure that the moderation system was capable of operating across different apps so that they can share their work, so to speak. [00:46:43] And so we created what we call labeling. And labeling is a metadata layer that exists over the network. Doesn't actually live in the normal data repositories. It uses a completely different synchronization because a lot of these labels are getting produced. It's just one of those things where the engineering characteristics of the labels is just too different from the rest of the system. [00:47:02] So we created a separate synchronization for this, and it's really kind of straightforward. It's, here's a URL and here's a string saying something like NSFW or Gore, or you know, whatever. then those get merged onto the records brought down by the client and then the client, you know, based on the user's preferences. [00:47:21] We'll put like warning screens up, hide it, stuff like that. So yeah, these label streams can then, you know, anybody that's running a moderation service can, you know, are publishing these things and so anybody can subscribe to 'em. And you get that kind of collaborative thing we're always trying to do with this. [00:47:34] And we had some users set up moderation services and so then as an end user you find it, it looks like a profile in the app and you subscribe to it and you configure it and off you go. That one has had probably the least amount of adoption throughout all of 'em. It's you know, moderation. [00:47:53] It's a sticky topic as you can imagine, challenging for folks. These moderation services, they do receive reports, you know, like whenever I'm reporting a post, I choose from all my moderation services who I wanna report this to. what has ended up happening more than being used to actually filter out like subjective stuff is more kind of like either algorithmic systems or what you might call informational. [00:48:21] So the algorithmic ones are like, one of the more popular ones is a thing that's looking for, posts from other social networks. Like this screenshot of a Reddit post or a Twitter post or a Facebook post. Because, which you're kinda like, why, you know, but the thing is some folks just get really tired of seeing screenshots from the other networks. [00:48:40] 'cause often it's like, look what this person said. Can you believe it? You know, it's like, ah. Okay, I've had enough. So one of our users aendra made a moderate service that just runs an ML that detects it, labels it, and then folks that are tired of it, they subscribe to it and they're just hide it, you know? [00:48:57] And so it's like a smart filter kind of thing that they're doing. you know, hypothetically you could do that for things like spiders, you know, like you've got arachniphobia, things like that. that's like a pretty straightforward, kind of automated way of doing it. Which takes a lot of the spice, you know, outta out of running moderation. [00:49:15] So that users have been like, yeah, yeah, okay, we can do that. [00:49:20] Those are user facing ways that we tried to surface the. Decentralized principle, right? And make take advantage of how this whole architecture can have this kind of a pluggability into it. Users can self host now [00:49:33] Paul: But then really at the end of the day, kind of the important core part of it is those pieces we were talking about before, the hosting, the relay and the, the applications themselves, having those be swappable in completely. so we tend to think of those as kind of ranges of infrastructure into application and then into particular client side stuff. [00:49:56] So a lot of folks right now, for instance, they're making their own clients to the application and those clients are able to do customizations, add features, things like that, as you might expect, [00:50:05] but most of them are not running their own backend. They're just using our backend. But at any point, it's right there for you. You know, you can go ahead and, and clone that software and start running the backend. If you wanted to run your own relay, you could go ahead and go all the way to that point. [00:50:19] You know, if you wanna do your own hosting, you can go ahead and do that. Um, it's all there. It's really just kind of a how much effort your project really wants to take. That's the kind of systemically important part. That's the part that makes sure that the overall mission of de monopolizing, social media online, that's where that really gets enforced. [00:50:40] Jeremy: And so someone has their own data repository with their own users and their own relay. they can request that your relay collect the information from their own data repositories. And that's, that's how these connections get made. [00:50:58] Paul: Yeah. And, and we have a fair number of those already. Fair number of, we call those the self hosters right? And we got I wanna say 75 self hoster going right now, which is, you know, love to see that be more, but it's, really the folks that if you're running a service, you probably would end up doing that. [00:51:20] But the folks that are just doing it for themselves, it's kind of the, the nerdiest of the nerds over there doing that. 'cause it doesn't end up showing itself in the, in the application at all. Right? It's totally abstracted away. So it, that, that one's really about like, uh, measure your paranoia kind of thing. [00:51:36] Or if you're just proud of the self-hosting or, or curious, you know, that that's kind of where that sits at the moment. AT Protocol beyond bluesky [00:51:42] Jeremy: We haven't really touched on the fact that there's this underlying protocol and everything we've been discussing has been centered around the bluesky social network where you run your own, instance of the relay and the data repositories with the purpose of talking to bluesky, but the protocol itself is also intended to be used for other uses, right? [00:52:06] Paul: Yeah. It's generic. The data types are set up in a way that anybody can build new data types in the system. there's a couple that have already begun, uh, front page, which is kind of a hacker news clone. There's Smoke Signals, which is a events app. There's Blue Cast, which is like a Twitter spaces, clubhouse kind of thing. [00:52:29] Those are the folks that are kind of willing to trudge into the bleeding edge and deal with some of the rough edges there for pretty I think, obvious reasons. A lot of our work gets focused in on making sure that the bluesky app and that use case is working correctly. [00:52:43] But we are starting to round the corner on getting to a full kind of how to make alternative applications state. If you go to the atproto.com, there's a kind of a introductory tutorial where that actually shows that whole stack and how it's done. So it's getting pretty close. There's a couple of still things that we wanna finish up. [00:53:04] jeremy so in a way you can almost think of it as having an eventually consistent data store on the network, You can make a traditional web application with a relational database, and the source of truth can actually be wherever that data repository is stored on the network. [00:53:24] paul Yeah, that's exactly, it is an eventually consistent system. That's exactly right. The source of truth is there, is their data repo. And that relational database that you might be using, I think the best way to think about it is like secondary indexes or computed indexes, right? They, reflect the source of truth. [00:53:43] Paul: This is getting kind of grandiose. I don't tend to poses in these terms, but it is almost like we're trying to have an OS layer at a protocol level. It's like having your own [00:53:54] Network wide database or network-wide file system, you know, these are the kind of facilities you expect out of a platform like an os And so the hope would be that this ends up getting that usage outside of just the initial social, uh, app, like what we're doing here. [00:54:12] If it doesn't end up working out that way, if this ends up, you know, good for the Twitter style use case, the other one's not so much, and that's fine too. You know, that's, that's our initial goal, but we, we wanted to make sure to build it in a way that like, yeah, there's evolve ability to, it keeps, it, keeps it, make sure that you're getting kinda the most utility you can out of it. Peer-to-peer and the difficulty of federated queries [00:54:30] Jeremy: Yeah, I can see some of the parallels to some of the decentralized stuff that I, I suppose people are still working on, but more on the peer-to-peer side, where the idea was that I can have a network host this data. but, and in this case it's a network of maybe larger providers where they could host a bunch of people's data versus just straight peer to peer where everybody has to have a piece of it. [00:54:57] And it seems like your angle there was really the scalability part. [00:55:02] Paul: It was the scalability part. And there's great work happening in peer-to-peer. There's a lot of advances on it that are still happening. I think really the limiter that you run into is running queries against aggregations of data. Because you can get the network, you know, BitTorrent sort of proved that you can do distributed open horizontal scaling of hosting. [00:55:29] You know, that basic idea of, hey, everybody's got a piece and you sync it from all these different places. We know you can do things like that. What nobody's been able to really get into a good place is running, queries across large data sets. In the model like that, there's been some research in what is, what's called federated queries, which is where you're sending a query to multiple different nodes and asking them to fulfill as much of it as they can and then collating the results back. But it didn't work that well. That's still kind of an open question and until that is in a place where it can like reliably work and at very large scales, you're just gonna need a big database somewhere that does give the properties that you need. You need these big indexes. And once we were pretty sure of that requirement, then from there you start asking, all right, what else about the system [00:56:29] Could we make easier if we just apply some more traditional techniques and merge that in with the peer-to-peer ideas? And so key hosting, that's an obvious one. You know, availability, let's just have a server. It's no big deal. But you're trying to, you're trying to make as much of them dumb as possible. [00:56:47] So that they have that easy replaceability. Moderation challenges [00:56:51] Jeremy: Earlier you were talking a a little bit about the moderation tools that people could build themselves. There was some process where people could label posts and then build their own software to determine what a feed should show per a person. [00:57:07] Paul: Mm-Hmm [00:57:07] Jeremy: But, but I think before that layer for the platform itself, there's a base level of moderation that has to happen. [00:57:19] Paul: yeah. [00:57:20] Jeremy: And I wonder if you could speak to, as the app has grown, how that's handled. [00:57:26] Paul: Yeah. the, you gotta take some requirements in moderation pretty seriously to start. And with decentralization. It sometimes that gets a little bit dropped. You need to have systems that can deal with questions about CSAM. So you got those big questions you gotta answer and then you got stuff that's more in the line of like, alright, what makes a good platform? [00:57:54] What kind of guarantees are we trying to give there? So just not legal concerns, but you know, good product experience concerns. That's something we're in the realm of like spam and and abusive behavior and things like that. And then you get into even more fine grain of like what is a person's subjective preference and how can they kind of make their thing better? [00:58:15] And so you get a kind of a telescoping level of concerns from the really big, the legal sort of concerns. And then the really small subjective preference kind of concerns. And that actually that telescoping maps really closely to the design of the system as well. Where the further you get up in the kind of the, in that legal concern territory, you're now in core infrastructure. [00:58:39] And then you go from infrastructure, which is the relay down into the application, which is kind of a platform and then down into the client. And that's where we're having those labelers apply. And each of them, as you kind of move closer to infrastructure, the importance of the decision gets bigger too. [00:58:56] So you're trying to do just legal concerns with the relay right? Stuff that you objectively can, everybody's in agreement like Yeah, yeah, yeah. You know, no bigs don't include that. The reason is that at the relay level, you're anybody that's using your relay, they depend on the decisions you're making, that sort of selection you're doing, any filtering you're doing, they don't get a choice after that. [00:59:19] So you wanna try to keep that focus really on legal concerns and doing that well. so that applications that are downstream of it can, can make their choices. The applications themselves, you know, somebody can run a parallel I guess you could call it like a parallel platform, so we got bluesky doing the microblogging use case, other people can make an application doing the microblogging use case. So there's, there's choice that users can easily switch, easily enough switch between, it's still a big choice. [00:59:50] So we're operating that in many ways. Like any other app nowadays might do it. You've got policies, you know, for what's acceptable on the network. you're still trying to keep that to be as, you know, objective as possible, make it fair, things like that. You want folks to trust your T&S team. Uh, but from the kind of systemic decentralization question, you get to be a little bit more opinionated. [01:00:13] Down all the way into the client with that labeling system where you can, you know, this is individuals turning on and off preferences. You can be as opinionated as you want on that letter. And that's how we have basically approached this. And in a lot of ways, it really just comes down to, in the day to day, you're the moderation, the volume of moderation tasks is huge. [01:00:40] You don't actually have high stakes moderation decisions most of the time. Most of 'em are you know pretty straightforward. Shouldn't have done that. That's gotta go. You get a couple every once in a while that are a little spicier or a policy that's a little spicier. And it probably feels pretty common to end users, but that just speaks to how much moderation challenges how the volume of reports and problems that come through. [01:01:12] And we don't wanna make it so that the system is seized up, trying to decentralize itself. You know, it needs to be able to operate day to day. What you wanna make is, you know, back pressure, you know, uh, checks on that power so that if an application or a platform does really start to go down the wrong direction on moderation, then people can have this credible exit. [01:01:36] This way of saying, you know what, that's a problem. We're moving from here. And somebody else can come in with different policies that better fit people's people's expectations about what should be done at, at these levels. So yeah, it's not about taking away authority, it's about checking authority, you know, kind of a checks and balances mentality. [01:01:56] Jeremy: And high level, 'cause you saying how there's such a high volume of, of things that you know what it is, you'd know you wanna remove it, but there's just so much of it. So is there, do you have automated tools to label these things? Do you have a team of moderators? Do they have to understand all the different languages that are coming through your network? [01:02:20] Yes, yes, yes and yes. Yeah. You use every tool at your disposal to, to stay on top of it. cause you're trying to move as fast as you can, folks. The problems showing up, you know, the slower you are to respond to it, the, the more irritating it is to folks. Likewise, if you make a, a missed call, if somebody misunderstands what's happening, which believe me, is sometimes just figuring out what the heck is going on is hard. [01:02:52] Paul: People's beefs definitely surface up to the moderation misunderstanding or wrong application. Moderators make mistakes so you're trying to maintain a pretty quick turnaround on this stuff. That's tough. And you, especially when to move fast on some really upsetting content that can make its way through, again, illegal stuff, for instance, but more videos, stuff like that, you know, it's a real problem. [01:03:20] So yeah, you're gotta be using some automated systems as well. Clamping down on bot rings and spam. You know, you can imagine that's gotten a lot harder thanks to LLMs just doing text analysis by dumb statistics of what they're talking about that doesn't even work anymore. [01:03:41] 'cause the, the LLMs are capable of producing consistently varied responses while still achieving the same goal of plugging a online betting site of some kind, you know? So we do use kind of dumb heuristic systems for when it works, but boy, that won't work for much longer. [01:04:03] And we've already got cases where it's, oh boy, so the moderation's in a dynamic place to say the least right now with, with LLMs coming in, it was tough before and

Bitcoin OG Charlie Shrem is now the chief evangelist of a project called Digital Gold (DGD). In this episode, I ask him and his business partner Digital Gold Yoda all the important questions about the legitimacy of their new cryptocurrency. Time stamps: Introducing Charlie Shrem & Digital Gold Jedi (00:00:48) Is Charlie Still Bullish on Bitcoin? (00:01:40) From Bitcoin to Digital Gold (00:02:05) Details of the Digital Gold Project (00:04:52) Stability and Value Preservation? (00:05:37) Community Engagement and User Growth (00:08:34) Comparison with BitTorrent (00:09:01) There Are Thousands of Digital Golds (00:14:28) Unique Features of the Digital Gold Project (00:15:08) Which Wallets and Exchanges Support Digital Gold? (00:17:05) Community Engagement and Validation (00:17:55) Initial Feedback and Expectations (00:18:50) Purchasing Process and Coin Distribution (00:19:26) Coin Withdrawal Mechanics (00:20:15) Network Growth and Distribution (00:21:02) Exchanges and Market Dynamics (00:22:57) Stablecoin vs. Price Speculation (00:23:25) Price Determination Mechanism (00:24:27) Infrastructure and Value Creation (00:25:09) Market Dynamics and Adoption (00:26:06) Mining vs. Market Factors (00:26:21) Coin Purchase Process Clarification (00:27:13) Community Participation and Evangelism (00:29:20) Address Reuse Concerns (00:31:16) Price Validation by Community (00:32:38) Selling Coins Among Users (00:34:31) Community Exchange Challenges (00:34:43) Decentralized Exchange Considerations (00:35:44) Arbitrage Opportunities (00:36:00) Side Shift (00:36:43) Treasury and Bitcoin Ownership (00:37:42) Concerns About Bitcoin Reserve Safety (00:38:00) Community Trust and Auditing (00:39:26) Charlie Shrem's Long-Term Vision for Digital Gold (00:40:28) Self-Custody and User Understanding (00:41:51) Value of DGB vs. Bitcoin (00:42:07) Name Change Story (00:44:01) Treasury Transparency and Auditing (00:45:24) Future of Auditing in Crypto (00:46:29) Bullish Prediction for Digital Gold (00:47:18) Understanding User Risks and Backup Solutions (00:51:51) Digital Gold Experiment (00:52:56) Challenges of User Adoption (00:54:10) Centralization Concerns (00:57:42) Node Operation Incentives (00:58:16) Concept of Proof of Participation (01:00:55) Contribution vs. Purchase (01:06:52) Intrinsic Value and Market Parity (01:09:07) Discussion on Gold and Currency Value (01:10:04) Clarifying Payment Terminology (01:10:35) Contributions, Not Investments (01:11:57) White Paper Availability (01:12:44) Smart Currency Concept (01:15:20) Comparison with Bitcoin Cash (01:15:49) Participation in the Network (01:16:40) Digital Gold vs Terra Luna (01:17:47) Claiming Coins Without Purchase (01:19:16) Distribution Model Fairness (01:21:04) Becoming a Staker (01:23:25) Node Connection and Validation (01:25:03) Impact of Node Outages (01:27:25) Core Staking Nodes Explained (01:28:09) Government Threats to Network (01:29:10) Initial Market Cap and Podcast Launch (01:30:25) Security Team: How Does It Get Paid? (01:32:44) Bug Bounty and Security Issues (01:36:00) Distribution of Coins and Participation (01:37:29) Peer-to-Peer Transactions (01:39:33) Transparency of Coin Holdings (01:41:51) Labeling The Team's Staking Wallets (01:42:23) First Dancers (01:45:31) Charlie Shrem's Role in Digital Gold (01:45:39) The Litmus Test (01:46:03) Importance of Charlie's Endorsement (01:46:31) Highlighting Charlie's Character (01:47:31) Addressing Potential Concerns (01:48:03) User-Friendly Exchange Integration (01:48:42) Future Selling of Coins (01:49:44) Saying Goodbye (01:50:13)

"Imagine waking up in a world where every piece of information you encounter is carefully filtered by faceless algorithms. Your online interactions are curated not by your preferences, but by the agendas of corporations and governments. You can only speak within the walls of platforms that commodify your every word, every click. This isn't a dystopian novel — it's the current reality of the internet."— John CarvalhoThere is something very wrong with the internet today. There are increasingly more attempts at fundamentally fixing the broken foundation, so that we can build back with user autonomy and control at its center. But what is the best path? How do we compare the strengths and weaknesses of each? And could Pubky, a new player in the game bringing the power of BitTorrent and the simplicity of a user connected web, bridge the gap between the technical foundation and how the everyday user experiences the web? We dive in today in a great article exploring how we are building the future of the web. Check out the original article on Medium by John Carvalho (Link: https://tinyurl.com/kfphsc9)  Guy's BitKit ID: slash:17b38zks77cnm63fy1z5p7nbzzfs6i4dpwbeh984a5ewkunpdiho?relay=https://webrelay.slashtags.toHost Links ⁠Guy on Nostr ⁠(Link: http://tinyurl.com/2xc96ney) ⁠Guy on X ⁠(Link: https://twitter.com/theguyswann) Guy on Instagram (Link: https://www.instagram.com/theguyswann) Guy on TikTok (Link: https://www.tiktok.com/@theguyswann) Guy on YouTube (Link: https://www.youtube.com/@theguyswann) ⁠Bitcoin Audible on X⁠ (Link: https://twitter.com/BitcoinAudible) The Guy Swann Network Broadcast Room on Keet (Link: https://tinyurl.com/3na6v839) Check out our awesome sponsors! Fold: The best way to buy, use, and earn #Bitcoin on everything you do! Sats back on your debit card, gift cards, auto-buys, round-ups, you name it. Fold is the true bitcoiner's banking. Get 20K sats for FREE using referral code bitcoinaudible.com/fold Ready for best-in-class self custody? Get ⁠5% off the COLDCARD⁠ with code BITCOINAUDIBLE ⁠⁠⁠⁠⁠⁠(Link: https://bitcoinaudible.com/coldcard) Trying to BUY BITCOIN? River, secure, trusted, bitcoin only, lightning enabled, simple. (Link: https://bitcoinaudible.com/river) Bitcoin Games! Get 10% off the best Bitcoin board game in the world, HODLUP! Or any of the other great games from the Free Market Kids! Use code GUY10 at checkout for 10% off your cart! (Link: https://www.freemarketkids.com/collections/games-1) Bitcoin Custodial Multisig Want to get into Bitcoin but not ready for self custody? Use custodial multisig for the best way to distribute trust across multiple institutions and even jurisdictions! Check out OnRamp. (Link: BitcoinAudible.com/onramp) Education & HomeSchooling G...

It's two movies for the price of one today! (Podcast remains free.) In 2014, Paul Schrader directed Nicolas Cage and Anton Yelchin in a thriller about a CIA agent with frontotemporal dementia trying to hunt down an old enemy before his mind goes. The money guys took the film away from him, edited it into a totally anonymous Redbox movie themselves, and released it as "A Paul Schrader Film" anyway. He was contractually prohibited from disparaging the result, but Schrader, along with the cast and producer Nicholas Winding Refn, publicly disowned the released version, and Paul drank himself into a stupor. Several years later, during the editing process on First Reformed, he returned to DYING OF THE LIGHT and made it into something between a director's cut and a fascinating work of video collage art called DARK, which is legally unreleasable but available to view for free. It's so nuts, and it changed the course of his career. We're gonna talk about it! Further Reading: "Warning: ‘Dying of the Light' Is Not a Paul Schrader Movie" by Anne Thompson "Paul Schrader's Secret New Movie: How the Director Resurrected a Wild Nicolas Cage Performance Without Permission" by Eric Kohn Further Viewing: DARK (Schrader, 2017) Follow Pod Casty For Me: https://www.podcastyforme.com/ https://twitter.com/podcastyforme https://www.instagram.com/podcastyforme/ https://www.youtube.com/@podcastyforme Support us on Patreon: https://www.patreon.com/PodCastyForMe Artwork by Jeremy Allison: https://www.instagram.com/jeremyallisonart  

Roberto (RCX) Capodieci, serial entrepreneur and blockchain expert shares his experience on the powers of decentralization and innovation. Along the way we discuss – Tom Cruise (1:45), cold-calling Japan (11:50), hacking BitTorrent (14:50), NFT's (30:35), and Blockchain Zoo (34:15). Checkout RCX at this his website Roberto RCX Capodieci. His new book, New Strategies in Blockchain can be purchased through this link.  Please consider support to our pod sponsor – LukeLeaders1248. Donations to scholarships for children of military veterans can be made @ www.lukeleaders1248.com or Venmo @luke-leaders. Music intro and outro from the creative brilliance of Kenny Kilgore, lead guitarist for The Shadows @ Blind Willie's Blues Club, Marietta, Georgia. Lowriders and Beautiful Rainy Day. 

In this episode as we sit down with Justin Sun, the founder of TRON and CEO of BitTorrent. We dive deep into the launch of SunPump Beta, the new platform for the fair launch and trading of meme coins. Discover how SunPump is set to revolutionize the TRON ecosystem with its user-friendly features, fair pricing mechanisms, and innovative liquidity solutions.We also explore the latest developments in WBTC, including the strategic partnership with BiT Global. Learn how moving to a multi-jurisdictional custody model enhances the security and trust of WBTC operations, and what this means for users worldwide.Tune in to hear Justin Sun's insights on the future of DeFi, the role of TRON in the crypto landscape, and his thoughts on the most exciting new meme coins on SunPump.

It's return guest season here at Latent Space! We last talked to Kanjun in October and Jonathan in May (and December post Databricks acquisition): Imbue and Databricks are back for a rare treat: a double-header interview talking about DBRX from Databricks and Imbue 70B, a new internal LLM that “outperforms GPT-4o” zero-shot on a range of reasoning and coding-related benchmarks and datasets, while using 7x less data than Llama 3 70B.While Imbue, being an agents company rather than a model provider, are not releasing their models today, they are releasing almost everything else: * Cleaned-up and extended versions of 11 of the most popular NLP reasoning benchmarks* An entirely new code-focused reasoning benchmark* A fine-tuned 70B model, built with Meta Llama 3, to identify ambiguity* A new dataset of 450,000 human judgments about ambiguity* Infrastructure scripts for bringing a cluster from bare metal to robust, high performance training* Our cost-aware hyperparameter optimizer, CARBS, which automatically and systematically fine-tunes all hyperparameters to derive optimum performance for models of any sizeAs well as EXTREMELY detailed posts on the infrastructure needs, hyperparameter search, and clean versions of the sorry state of industry standard benchmarks. This means for the FIRST TIME (perhaps since Meta's OPT-175B in 2022?) you have this level of educational detail into the hardware and ML nitty gritty of training extremely large LLMs, and if you are in fact training LLMs of this scale you now have evals, optimizers, scripts, and human data/benchmarks you can use to move the industry forward together with Imbue.We are busy running the sold-out AI Engineer World's Fair today, and so are unable to do our usual quality writeup, however, please enjoy our show notes and the excellent conversation! Thanks also to Kanjun, Ashley, Tom and the rest of team Imbue for setting up this interview behind the scenes.Video podTimestamps* [00:00:00] Introduction and catch up with guests* [00:01:55] Databricks' text to image model release* [00:03:46] Details about the DBRX model* [00:05:26] Imbue's infrastructure, evaluation, and hyperparameter optimizer releases* [00:09:18] Challenges of training foundation models and getting infrastructure to work* [00:12:03] Details of Imbue's cluster setup* [00:18:53] Process of bringing machines online and common failures* [00:22:52] Health checks and monitoring for the cluster* [00:25:06] Typical timelines and team composition for setting up a cluster* [00:27:24] Monitoring GPU utilization and performance* [00:29:39] Open source tools and libraries used* [00:32:33] Reproducibility and portability of cluster setup* [00:35:57] Infrastructure changes needed for different model architectures* [00:40:49] Imbue's focus on text-only models for coding and reasoning* [00:42:26] CARBS hyperparameter tuner and cost-aware optimization* [00:51:01] Emergence and CARBS* [00:53:18] Evaluation datasets and reproducing them with high quality* [00:58:40] Challenges of evaluating on more realistic tasks* [01:06:01] Abstract reasoning benchmarks like ARC* [01:10:13] Long context evaluation and needle-in-a-haystack tasks* [01:13:50] Function calling and tool use evaluation* [01:19:19] Imbue's future plans for coding and reasoning applications* [01:20:14] Databricks' future plans for useful applications and upcoming blog postsTranscriptSWYX [00:00:00]: Welcome to the Latent Space Podcast, another super special edition. Today, we have sort of like a two-header. John Frankel from Mosaic Databricks, or Databricks Mosaic, and Josh Albrecht from MBU. Welcome.JOSH [00:00:12]: Hey, glad to be here.SWYX [00:00:14]: Thank you for having us. Hey, so both of you are kind of past guests. Jonathan, you were actually one of the most popular episodes from last year talking about MPT7B. Remember the days when we trained large models and there was 7B?JONATHAN [00:00:30]: Yeah, back when reproducing LLAMA1-7B was considered a huge accomplishment for the field. Those are the good old days. I miss that.SWYX [00:00:38]: As the things have accelerated a lot. Actually, let's do a quick catch up and Josh, you can chime on in as well. So Databricks got acquired. I talked to you at New York.JONATHAN [00:00:45]: Mosaic got acquired, although sometimes it feels like Mosaic acquired Databricks because, you know, we're having a lot of fun being here. But, you know, yeah.SWYX [00:00:52]: Yeah. I mean, you are chief scientist now of Databricks.JONATHAN [00:00:55]: Chief AI scientist. Careful with the title. As much as I would love to understand how Spark works, I'm going to have to defer that to much smarter people than me.SWYX [00:01:03]: Got it. And I don't know about like what you would highlight so far as a post-acquisition, but the most recent news is that you guys released DBRX. Is that the thing that most people should be aware of?JONATHAN [00:01:13]: Actually, that's no longer the most recent news. Honestly, the most recent news, we announced this, but it was at our Data and AI Summit last week. So it was announced among like 100,000 other things, is that we finally released our text to image model, which has been a year in the making through a collaboration directly with Shutterstock. There was a lot of work put into finding a dataset that we were comfortable with working on and trying to build a model that honestly, I felt like I could trust and that others might be able to trust to put out in the world. So that model was released last week. It's unfortunately just available via API due to the fact that the data is quite sensitive and quite valuable. It's Shutterstock's entire business in a lot of ways, but I'm still really excited that there's now a model that is trained on a dataset where the provenance of every single image is known, and it's a damn good model. So I'm really proud of the team on that.SWYX [00:01:55]: Yeah, amazing. Josh, do you have any thoughts on image model questions?JOSH [00:01:59]: That is not my area of expertise, but I was excited to see the release of it last week as well, and very happy that you guys did a nice job on the data side of everything there. So that was cool to see.SWYX [00:02:09]: I think what's unusual is like, I think Shutterstock's doing multiple deals in multiple labs. So what is the Shutterstock model? Like, I guess, is this the house model for Shutterstock? Is this Databricks' version of the Shutterstock model? Like, what is this?JONATHAN [00:02:22]: The way that I would think about it is that Shutterstock is doing an amazing business in AI across the board. Their dataset is kind of widely known to be the best stock photos dataset in the world, the most comprehensive, the biggest. When you think about like, what dataset am I going to train a multimodal model on? You call Shutterstock. And I, at least I've heard in the news, like OpenAI, Google, Meta, Apple have all called Shutterstock and made those deals. So a lot of models have had Shutterstock data incorporated into them. But this is the only model I know of so far where it was, you know, exclusively and specifically trained just on the vanilla Shutterstock data. There was nothing else mixed in. We didn't go and scrape the web and find other data or combined datasets or anything like that. And so this is, in some sense, the house blend. But the other piece is that it's just a dataset where the provenance of every image is known in public. Where did the data come from? It is the Shutterstock collection. That's it. You know, nothing less, nothing more. And certainly being at Databricks, if I've learned one thing, I've learned about enterprise customers and what they want out of AI. And one of the things they ask for most is just, what can you tell me about the data the model was trained on? And here, especially for text to image models, where images are just tricky subject matter, there's been a lot of kind of legal conversation about images, especially. It's nice to just have something where I can point to it and say, you know, if you want to know where the images came from, these are what they are and this is how they got there.SWYX [00:03:36]: I will talk a little bit about Databricks because it's relevant to the rest of today's episode. So Databricks, sorry, I keep misspeaking. It's DBRX.JONATHAN [00:03:46]: DBRX, actually, there's been a pronunciation update. It is now D-B-Rex. So we have decided to add a dinosaur mascot because what model doesn't like a mascot? So literally, I wish I could pull it up. There is a little plush dinosaur that we had made. It's like the world's cutest dinosaur, but it is the official mascot of D-B-Rex. And there's a little dinosaur logo that, you know, you'll probably see around a little bit more because DBRX is a mouthful, but D-B-Rex, like, you know, it's just kind of...SWYX [00:04:13]: Rolls off the tongue. I love mascots. Like every company should have a mascot. And I think Hugging Face got it right. You need an emoji mascot because that's the minimal viable image.JONATHAN [00:04:21]: I probably shouldn't talk at all about, you know, Velociraptor, but, you know, that's a, maybe that's something we can talk about later in the summer. I'll just leave it at that.SWYX [00:04:28]: Okay. That's a hint to names. I feel like your names leak a lot of alpha. So just to quickly cover the headline details, DBRX, as Make Sure Experts model, that's fairly big, 132 billion total parameters, so 36 billion active on any input, pre-trained on 12 trillion tokens of text and code, and did really well on evals to the point where you had to dye your hair blue. That's my high level conclusion.JONATHAN [00:04:53]: Never make a bet with your team two weeks out from model launch, even when, you know, human eval is looking quite bad. Because if you set some bar, even if it's arbitrary and you think there's no way in hell they're going to hit it, apparently money doesn't motivate people anymore. Humiliating their boss motivates people. So Josh, you should really take a hint from this. You know, you cannot pay someone enough money to make up for you dyeing your hair blue.JOSH [00:05:15]: I'll keep that in mind for our next model.SWYX [00:05:17]: It works. So speaking of Imbue's next model, perhaps Josh, you want to actually just say hi to the general sort of latent space audience and talk about what we're releasing today. Yeah.JOSH [00:05:26]: I'm Josh, CTO of Imbue, and we're not releasing the model. We're not releasing the weights, but we are releasing a bunch of different things that should make it easier for other people to make their own models. So I think right now, training foundation models from scratch is like a very difficult, time-consuming, expensive, kind of risky endeavor, especially for smaller companies. And the things that we're releasing hopefully make that at least a little bit easier. So the things that we're releasing fall into kind of three different buckets. One is infrastructure and scripts for dealing with the kind of hardware and hardware failures and understanding how well is the actually lowest level of thing actually working so that you can actually do your training at all and at a reasonable speed without having to constantly restart, etc. So infrastructure and training scripts. A second set of things is around the evaluation. So after you've trained it, like how well is this actually working and how do you know how well it's working? We're releasing a whole bunch of different data there, a new benchmark about code, reasoning, understanding, as well as our own private versions of 11 different open source benchmarks. So things like pool queue or ANLI, where we've gone through and kind of cleaned up the data as much as possible by looking at all the ones that models get wrong or that are flagged for ambiguity and also our own kind of private reproductions of those where we've done like a kind of clean room black box, like, okay, this is what the data set is supposed to be. Here are some examples. Let's make our own version of this to make sure that there is no data contamination, etc. To make sure that we're actually, you know, not testing on train. And then I think a final thing that we're releasing there is around 450,000 human judgments about ambiguity and question quality, which we used in the process of cleaning these evaluations and we also hope will be helpful for other people training kind of similar models. And then the third thing is CARBS, our hyperparameter, our cost-aware hyperparameter optimizer, which was especially helpful for being able to experiment at much smaller scales and then scale those experiments up to the much larger scale kind of on the first try without having to retry it. You don't want to be training, you know, 10, 20 different 70B models. You really want to get these larger modelsSWYX [00:07:30]: right on the first try.JOSH [00:07:30]: And so the ability to kind of tune things very precisely and learn scaling laws, not just for, you know, the like data and flops, but also for learning rate and all the other hyperparameters and see like how should you scale these things up was extremely valuable to us as we were training the larger models. Yeah, that's a lot of stuff.SWYX [00:07:49]: Yeah, exactly. So there's a bunch of stuffJOSH [00:07:50]: we'll have to go through all of it.JONATHAN [00:07:52]: Yeah, I just want to throw in how excited I am about this. This is the stuff that nobody ever talks about. That is the difference between success and failure in this stuff. Like, can you get your cluster to run? Can you get software on your cluster? Can you figure out what broke? Because fault tolerance is still not really built into any of the fundamental primitives of training models. And so if something breaks, you have to go figure out what broke, your job stops, you have to restart your job. It is a nightmare just to get to the point where anything can train on the cluster. A basic MPI hello world that has the GPUs talk to each other is hard enough, let alone actually training a model, let alone getting good performance out of the GPUs, let alone actually getting a model that converges to anything interesting. There's so many levels of things you have to accomplish. This is the kind of stuff that matters. I think to a point that Josh made earlier, before we got on here, there are plenty of weights out there. Nobody's released this.JOSH [00:08:46]: Yeah, that was part of the motivation actually is that there are lots of other things that are complimentary, but I have not seen nearly as much discussion about some of these other things that we think are pretty important. I mean, in some sense,SWYX [00:08:56]: I'm very excited to have Jonathan on because this is a little bit, you're a bread and butter with Mosaic. And I think you've released some part with Composer. And I think it's just really interesting to see like a different take, basically a full stack take that's kind of open source today.JONATHAN [00:09:18]: Yeah, it's really kind of, it's been an ordeal to figure this out. And every time something changes, whether it's a new GPU or even a new driver update, you get new creative errors and new things go wrong. And, you know, we've dealt with the weirdest things from, you know, our InfiniBand cables getting stolen from the data center twice, like in boxes before they arrived at the data center. Like, you know, Porch Pirate basically had stolen our InfiniBand cables back when those were hard to come by. To like, you know, weird recalls of switches to like the strangest stuff has happened. I have my favorite GPU failures I've seen, like ones where the GPU doesn't fail, it has a correctable memory issue and the memory correction causes the GPU to become a straggler and hold up the whole job. Like weird stuff happens and figuring out how to not just identify all of that, but then eventually productize it, is in some sense, the entire story of Mosaic and now Databricks in terms of our ML offering. Really, the thing we offer is we have gone through this suffering and figured out how to even productize that. It has been a pain in the butt.SWYX [00:10:20]: Yeah, it's a lot of work.JOSH [00:10:20]: I think my favorite failure was GPU is just giving wrong math. Like if they give errors, great, because you can see the errors, but if they just give you the wrong math back, not so fun.SWYX [00:10:30]: When did they give you wrong math?JOSH [00:10:32]: Like literally you could just, you know, add two things. For example, the numbers come back. They're not the numbers that they're supposed to be.JONATHAN [00:10:40]: I think it's important to say at this stage, just because like it, I think it goes without saying for Josh and I, but it's worth saying here, this isn't to say that like anything is wrong with us. It's not like NVIDIA did a bad job or, you know, Mellanox did a bad job or the like the server builder, the data center operator, the cloud provider, like the million other parties that are involved in building this. We are running these insane chips that are huge and complicated and built on tiny transistors at insane frequencies with insane heat in data centers that for the most part, were not built remotely for this kind of power or heat and have been retrofitted for this. Like failures happen on a good day with normal CPUs. And this is not a good day and not a normal CPU for the most part. It's fun to joke about all the weird things we see. This is not to say anybody's done anything wrong. This is just kind of part and parcel of working on a massive cluster running at multiple megawatts of power at a time.SWYX [00:11:32]: It's crazy. Yeah.JONATHAN [00:11:33]: So optical cables, like all sorts, like everything.SWYX [00:11:37]: I'll take the opportunity to start going to the sort of infra piece. There's just like a description of the infra just to give people a sense of what we talk about when we talk about massive clusters. So I'm just going to read off the blog post here. This post is about one cluster that has 4,092 H100 GPUs spread across 511 computers. They use unified fabric manager nodes, which manage the infinite band network. And you talk a little bit about your networking. Is there anything unusual about this setup that you'll call out to people?JOSH [00:12:03]: Yeah, actually this particular cluster is a little bit non-standard. The normal, like vanilla setup for these large clusters as vanilla as it can be is what's normally like a 127 node cluster. So closer to like 1024 GPUs instead of 4,000. Here we have a larger cluster. As you start to get into the larger clusters, the networking becomes a little bit more custom. It's a little bit more, it's a little bit trickier. It's a little bit more difficult to get these things to all be able to talk to each other at the same speed. And so this has, in this particular case, this is a three tier network architecture instead of two tiers, kind of the normal one. So most of the clusters are a little bit smaller. As you get to even larger scales, then this becomes even much more complicated,SWYX [00:12:43]: much more expensive.JOSH [00:12:43]: So we chose this particular scale, kind of knowing our own workloads and kind of what we wanted to do. This was kind of the right size for us. But yeah, I think it's not exactly vanilla already. It's already getting into kind of the custom territory.SWYX [00:12:54]: So my understanding is that there, and is there any part of this that comes with the Voltage Park deal that you guys had? Is that part of the hardware that you got from the deal with them?JOSH [00:13:04]: Yeah, so we worked really closely with Voltage Park to set up all their clusters and infrastructure and everything and kind of decide even like what to order, how should the networking work? Like we were very involved in kind of the construction and bring up of this. And that's what this post is about, is about that process of like bringing up all these, there's like different clusters in different places of different scales. So in this particular post, we're talking about this one 4096 GPU, but there are other clusters that they have as well. And we were very closely involved with figuring out the exact architecture and kind of the trade-offs that go along with picking, you know, those exact components. You really don't want to like place the wrong order because it takes months to get it and it's very expensive. So yeah, we were happy to help out with that.JONATHAN [00:13:43]: And then your bit of good cables get stolen.SWYX [00:13:44]: Yeah, yeah, exactly.JOSH [00:13:47]: We wanted to make sure that we ended up with compute that would work for us and that would also work for their other customers. And so we kind of helped design something so that we would get exactly what we were looking for. We knew that these kinds of details would be super important and that getting down to the level of the hardware and like having these good scripts and everything was going to be a core part of like actually getting this to work. I'm very glad that we did that. I don't think that most companies kind of take that full stack approach, but for us, it certainly paid off.SWYX [00:14:12]: Yeah, it's basically sort of built to spec. It's interesting that relationship because you usually, for the rest of us who don't operate at your scale, we take whatever we can get from cloud providers, but you are basically co-designing from the single machine up. And you described that a little bit. Do you want to take us through the process that you described here?JOSH [00:14:27]: Yeah, so for the actual, like the blog post and kind of bringing these machines online.SWYX [00:14:32]: Yeah.JOSH [00:14:32]: So yeah, I think the process, as we have it broken down in the blog post, there's kind of a few different layers. First is like getting the individual machines to work at all and then getting the machines to actually be able to talk to each other. So getting the InfiniBand networking to work and then getting to a point where, you know, not just the machines are working and they can talk to each other, but everything is actually working correctly. There's a big gap between like it's working at all to it's working perfectly correctly. And then after you have all this stuff working perfectly correctly, nice and healthy, then now you get into kind of the software data, like training issues. And then after that, you're still not done. Like now, even once you're training at full speed, things are going to fail over time. Things are going to change. There's going to be new, you know, firmware updates. Like how do you kind of deal with this change and flux over time without going crazySWYX [00:15:16]: and pulling your hair out,JOSH [00:15:16]: trying to like reproduce things or understand why there were regressions. And so there's a lot of work to kind of automate the infrastructure tooling as well. And kind of the first step, like bringing these things online in the first place, you know, you have hundreds of machines at this point. So you don't necessarily want to be like walking around with like a CD-ROM or a USB drive, like plugging it in with your keyboard, like hitting next, next, next on the OS install. That's not how this works. You do that for one machine. And then you use, we use this thing called Metal as a Service to bring up all the other machines. So it's a kind of server that can kind of install the operating system on these other machines. So most like when you're talking about these machines, like each machine is, you know, on the order of hundreds of thousands of dollars. So they usually come with a kind of out-of-band management interface as well. So they don't, they have their InfiniBand networking. They have their normal 100 gigabit per second Ethernet networking. These are like dual, redundant, et cetera. And then you also have this extra out-of-band management network. So you can log in and you can see like the boot screen or you can see the blue screen of death. You can like get in there and actually see what was wrong, which is pretty fun. And it makes it like possible to automate a lot of this work. So the beginning of that, and the blog post goes into much more detail about like exactly how we set these up and kind of the other errors that we ran into. When you're bringing these online, you'll definitely have failures. Even if they all worked in the factory, they get shipped, some parts come loose, something fails, something goes wrong. So when you're bringing them online, there'll be some that don't quite work for all sorts of reasons. As you start to be working with machines at this scale, like if something happens one in a thousand times, you're like pretty likely to see it. And so you can get pretty rare, weird things, especially since we had fairly early builds and fairly early versions of this hardware. Like these are some of the like first machines that were ever produced, some of the first GPUs. So you've got some extra special things there. We definitely worked with Dell, for example, on making fixes in the firmware level to be like, okay, like this thing is wrong. Like we need to update this at the firmware to like actually fix this particular thing. So we worked pretty closely with Dell and Nvidia. Yeah, that's what I'm saying. Like this stuff gets complicated. And the thing is like, you know, taking a step back, the whole reason we're doing this, right, is that we knew that this was going to be complicated. There would be these kinds of failures. And if we're just using, you know, AWS or some other cloud provider, these errors are still gonna be there and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong. And so we would much rather be able to like call up Dell and say, hey, this isn't working. And they're like, yep, okay, cool. Let's debug it together. Oh, I see. Yeah, cool. We'll ship a firmware update and actually fix this for you. That was a much better experience than like, great, just magically fails. I guess we restart and hope that that machine goes away. Like that's not a very good place to be. So yeah, that's kind of the first place is getting to a place where like GPU training is working on your single node machines. You can observe stuff. We have tons of tooling around like, you know, Prometheus and all sorts of other tools for understanding what's going on in these machines because you don't want to be like logging into each one and looking at the temperature or something you really need to have tooling to collect all these metrics, et cetera. Unfortunately, all of the scripts that we have for this are like for this entire cluster and for all this infrastructure are a little bit like special purpose for our particular thing. So it's not that every script that we have, it's not that you can just like take this and plug this in. Even if we did open source all the tooling that we have, you'd still have to do like a lot of work to open source it. What we are releasing is as many of the things that we can that are going to be useful for other people. You're still going to have to have some way of kind of managing these things, making your own like logging aggregators, et cetera, et cetera. So that's kind of bringing them up to the like, you know, the single nodes that are working. From there, it goes into, I'm happy to keep going if you want. Well, I just want to leave the opportunity for JohnSWYX [00:18:53]: to comment if there's anything that's different from how he runs things.JONATHAN [00:18:57]: Oh, I mean, all I'll say is I'll endorse this and say this s**t is hard. Like this is really, really hard. And, you know, I have a special props to, you know, the folks in Vue because they were building this from the ground up. You know, at Databricks and at Mosaic, we typically work with cloud providers because some of this stuff is just, there's too much to handle. It's complicated. There's a lot to deal with. And this doesn't even get into things like physical security, you know, securing power if you're the data center operator. Like this gets infinitely complicated and you have to abstract somewhere. Like, you know, and then you get to the folks who are literally building their own custom chips and like, good God.SWYX [00:19:36]: Like, oh my God, that's, you know,JONATHAN [00:19:38]: if you're one of those folks, you're having, you know, pour one out for the infra people at some of the AI chip startups who are having a really, really interesting time right now. But this stuff is really hard. And I don't think we talk about it much because there's so many other things that are hard. But the other hard things, I think everybody's becoming pretty familiar with at this point. This is something that I don't think there's ever really been a comprehensive discussion of, at least not that I've seen.SWYX [00:20:00]: Yeah, so my impression is that you guys, Mosaic, have your own software for sort of spinning up and down machines, just like Imbue had to build. But Imbue probably, it sounds like Imbue, you guys went fuller stack. I don't know how to describe it. Like Mosaic is not working with Dell on like their firmware.JONATHAN [00:20:21]: No, no, we're typically working with like, you know, pick your cloud provider on their Dell firmware or what have you. Like, it's kind of, I think one of the things, I don't know, Josh, you can correct me on this. It's kind of impossible if you're doing training to not go all the way through the entire stack, regardless of what happens. Like somehow I'm still chatting with cloud providers about power contracts, even though the whole point of dealing with the cloud provider is not to have to think about power contracts. Somehow I'm still asking them about which InfiniBand provider they used this time to see if this is part of the bad batch of cables I encountered on that cloud provider or what have you. Or like, we're still talking about a firmware update from pick your provider. You can't not do this. It's convenient that they have data center staff who are worrying about what to send back to which provider when, and they have people who can go and wait for the InfiniBand cables so they don't get stolen outside. But, you know, it's kind of, it's impossible not to really go full stack if you're thinking about the infrastructure at all. I don't know, Josh, correct me. No, I think that's right.JOSH [00:21:17]: That's what we expected from the beginning as well, is that we would inevitably have to get into the details here. And I'm glad that we kind of just planned for it. I think it made it a lot easier from our perspective to have direct control over this. Instead of having to go to the cloud provider that goes to the data center, that goes to the supplier, we could just go direct to NVIDIA or DellSWYX [00:21:37]: or the data center,JOSH [00:21:37]: whoever was responsible and be like, hey, this thing needs to change. And they're like, oh, okay. Yeah, that is our responsibility. Great, we can fix that. So it was just a lot easier for us to fix these bugs than if we had to go through an extra layer of email.SWYX [00:21:48]: Something we discussed in the pre-show was that you had a rule of thumb for your cluster of reliability. You say here in the post, by and large, you expect around 3% of your machines to break every week. So you're basically going to turn through all your machines in a year.JOSH [00:22:04]: As it says in the post. So that would be true if it was a uniform failure like that. But as it says in the post, it's usually these kind of problematic nodes. And to be clear, that is the number that we've heard from other people is like they're having about 3%. I don't think we're experiencing failure rates that are that high. I think ours is actually quite a bit lower than that, probably because we've taken the time to like dig into a large, maybe larger number than we should have of these failures and get to the root cause of it and be like, oh, okay, like that's exactly what's going wrong.SWYX [00:22:33]: How do we fix this?JOSH [00:22:33]: How do we prevent this from happening? How do we make automated checks for this so that if it does happen, it just goes back to whoever owns that particular part of the process and they can fix it immediately.SWYX [00:22:43]: And that's part of what you're also open sourcing, which is the health checks, right? You got the NIC health checks, GPU health check, this space health check, Docker D message. I don't know what that is.JOSH [00:22:52]: That one is just a lot of stuff.SWYX [00:22:54]: Yeah.JOSH [00:22:55]: That one is one where we realized that actually like when these machines boot, sometimes they wouldn't actually boot cleanly all the way. Or when they rebooted, they had problems that they didn't have when they were working before, which was kind of frustrating. Like usually if you restart your computer,SWYX [00:23:08]: it gets better.JOSH [00:23:08]: Here you restart. It did not get better.SWYX [00:23:10]: It got worse.JOSH [00:23:10]: That was very frustrating. So this health check looks at every particular line we've ever seen from the boot, like in D message, like every single log line that your computer emitsSWYX [00:23:21]: and says like,JOSH [00:23:21]: have we ever seen this before?SWYX [00:23:23]: Is this expected?JOSH [00:23:23]: Is this in the right order? Or is there something out of place? If there's anything out of place, let me say, okay, great. Like now it goes into this, like longer, more triage list of like, all right, great. Like, is this acceptable?SWYX [00:23:33]: Should we flag this?JOSH [00:23:33]: Like, should someone take a look at this? So we're looking down at a very, very granular detail level, what's happening on these computers to make sure that nothing is out of place. And that's critical because without that, if you're running your training, as Jonathan said, and this thing is slow, like what are you supposed to do? Right?SWYX [00:23:49]: Like you really,JOSH [00:23:49]: you really want to be very certain that like all 4,000 of these GPUs are working like they're supposed to.SWYX [00:23:54]: We know that.JOSH [00:23:54]: And so if it's slow, it's because like we messed up the config or something else and not because of this earlier thing that's like really hard to detect in software later.JONATHAN [00:24:01]: Yeah. I think the, I'm just curious to ask,SWYX [00:24:03]: like, you know,JONATHAN [00:24:03]: suppose you were to set up another, let's say another H100 cluster and it were at a different data center. And instead of the vendor being Dell, it was super micro or what have you. How much of this would be repeatable? And how much of this would you have to redo? I, you know, I genuinely don't know.SWYX [00:24:18]: A decent amount.JOSH [00:24:19]: I think it would go a lot faster the second time. I think there's lots of learnings that we had. And also the blog post,SWYX [00:24:24]: you know, yes,JOSH [00:24:24]: we are releasing the health checks, releasing some scripts, but a lot of the valuable stuff is also in the blog post itself, in the details and kind of the, you know, the learnings that we've had and the sort of errors that we run into. We tried to as much as possible surface those to other peopleSWYX [00:24:36]: could learn from thoseJOSH [00:24:36]: and avoid the same mistakes or failures as well. But I think it would go a lot faster.SWYX [00:24:41]: Although, yes,JOSH [00:24:41]: there would certainly be some things that'd be a little bit different. I mean, there'd probably be different CPUsSWYX [00:24:46]: or whatever,JOSH [00:24:46]: but I think a lot of that stuff is less,SWYX [00:24:49]: it's less,JOSH [00:24:49]: that's the like, that's less variable. I think most of it would apply the second time around. Although I'm sure next timeSWYX [00:24:56]: we're building one,JOSH [00:24:56]: it'll probably be, you know, at a scale that's 10x as big with a different chip or something like this.SWYX [00:25:00]: And then who knows?JOSH [00:25:01]: Yeah, with Kinect X8,JONATHAN [00:25:02]: that will have its own fun behavior and all that good stuff. Yeah.SWYX [00:25:06]: Perhaps there's something that people don't discuss about, and you don't even talk about this in the blog, but I always wonder is what is the timeline that's like kind of reasonable for this amount of work, at least the initial stages? And also what does the team composition look like for setting up a cluster, right? Like what are the mix of skills that you typically would require to get all this going?JOSH [00:25:27]: I'm, I can't really speak to typical. One thing I am very proud of is how much we accomplished with such a ridiculously small team. Like our infrastructure team is like, you know, fluctuates from week to week, depending on like how many things are on fire and how much we need to build. But it's like between like three and six people, like it's small. It's not like some huge team of like tons and tons of engineers. But those people are very, very good at what they do. And so that has allowed us to get a lot of mileage out of out of these things. I think it's not that we're building everything, right? It's not that three to six people build this whole thing. I definitely want to like, you know, say thanks very much to Dell and H5 and NVIDIA and the other people that have done a lot of the work, like to bring up this cluster, you know, with 4000 GPUs and three tier networking, networking architecture, you have 12,000 cables. So that's 24,000 things that need to be plugged in. Like that's just a lot of stuff to plug in, right? And you don't want to mess it up. Like each one needs to be done correctly. Like it's a little bit loose. Like it doesn't really work.SWYX [00:26:23]: If you break it,JOSH [00:26:23]: you need to replace it. Like there's a lot of workSWYX [00:26:26]: that goes into this.JOSH [00:26:27]: Yeah.SWYX [00:26:28]: And then, you know,JOSH [00:26:28]: that's just like that's it. That's if you were to do everything right the first time.SWYX [00:26:32]: And if you didn'tJOSH [00:26:32]: have to fix anything. But inevitably, you know, you will have to replace something, which means like taking all the wires out, pulling the thing out, taking all the GPUs out, going and fixing some cable, putting it all back correctly, putting it back in, doing this every time. So there were a lot of people at Dell, NVIDIA and at H5 that all helped a ton with this stuff. I don't know the exact size of the Dell team. It also fluctuated over time.SWYX [00:26:55]: Yeah, excellent. And then, you know, you so you have all the hardware set up and now you're firing it up for a single node. There's a long description that you guys have about just like monitoring the MFU, right? And what each situation might look might be indicative of. One of the most interesting things to me that I saw from here is like, you know, if training immediately starts off at 60 to 80% MFU, something's wrong.SWYX [00:27:24]: But like, you know, like what what are like, you know, some anecdotes or, you know, notable scenarios here that you might you might call out as maybe counterintuitive or super interesting.JOSH [00:27:36]: There's just so many of them. I mean, one of them, which I think is probably pretty common, like common knowledge by this point. But like we did have a sort of likeSWYX [00:27:46]: which one was this exactly?JOSH [00:27:47]: I think for the MFU, like gradually getting worse over time. I think that one, when we saw that the first time we were like, what the heck is going on? Like, why does it get just like a little bit worse? This is so strange. Like, what is it getting lazy or tired or something? Like, is it heat? Like what's going on? And in this particular case, it was memory fragmentation. Because you have hundreds of machines, they're doing garbage collection slightly different times. And then they get slightly further apart and slightly more and more jittered until eventually they're all happening kind of at random times. And just like really messing up each one of your steps. So you just turn off garbage collection and call it a day, basically,SWYX [00:28:20]: to be honest.JOSH [00:28:20]: There's other things you can do if you want to be a little bit more sophisticated about it. But you can also just manuallyJONATHAN [00:28:25]: have it all garbage collect on some interval. Like that's what we've done. We just have a garbage collection callback that just runs. But I've seen the exact same thing.JOSH [00:28:33]: Yeah, yeah, exactly. So I thought that one was kind of funny. And we did trace that one down and look and we did find the actual call. Like, again, this goes to like having good tools. So we had really good tools where we could look at a bunch of like actual traces in C and be like, OK, cool. This is the thing that's taking a lot of time. Or like, you know, this is the thing that doesn't quite line up here. Like, oh, I guess it's garbage collection. OK, cool.SWYX [00:28:52]: Interesting.JOSH [00:28:52]: Yeah, let's just try taking it off.SWYX [00:28:54]: OK, great.JOSH [00:28:54]: That's what it was. Now we can fix it. So for each of them, like basically bugs are not hard if you have good tools. But if you don't have good tools, bugs can be very, very hard. So similarly for like heat, another thing that we saw was like, oh, you know, the CPU is getting throttled. OK, well, it's easy to see if you're monitoring the CPU throttling or monitoring the heat. If you're not monitoring that, it's really hard to know why it's just suddenly one of them is going slower. I noticed also in the pieceSWYX [00:29:17]: that you mentioned FSDP with 0.3. Actually, we met, I went to iClear and Guanhua from the DSP team was there presenting 0++. I was wondering if you want to make any call outs to, you know, particular open source or open library or open whatever implementation teams that were super helpful in your process. I think we ended up actuallyJOSH [00:29:39]: pulling from a whole bunch of different ones to pull things in into our own particular pipeline. So we use things from NVIDIA's, you know, Megatron stuff. We use stuff from probably DeepSpeed. I think we pulled in a bunch of different pieces from a bunch of different places. So it was really nice to see all these working open source like examples. I think I really appreciate all the effort that has gone into actually tuning these things because you can tune them, but it's a lot of work to like tune this stuff and do all this stuff from scratch. It's really nice to have like a working example. I think those are probably the two biggest ones, DeepSpeed and Megatron alone, but there are probably other ones as well.SWYX [00:30:13]: Is there a particular thing in the ecosystem where you would call out as like, you know, there should be something here that is open source, but like it's not really, it's like everyone kind of builds it on their own. I want to say something with the file system because everyone talks about the file system eventually.JOSH [00:30:28]: The file system actually was,SWYX [00:30:30]: I mean, we did somethingJOSH [00:30:31]: kind of dumb there. Like we have our own sort of local mirror so that we can, you know, like a crappy version of S3SWYX [00:30:38]: that's local,JOSH [00:30:38]: but it's just a pretty simple script, right?SWYX [00:30:41]: Like I think we run likeJOSH [00:30:41]: a little web server that just like serves files and then, you know, it can upload themSWYX [00:30:45]: and download them.JOSH [00:30:45]: Okay, great. And part of the reason we did that is that our internet connectionSWYX [00:30:50]: in the beginningJOSH [00:30:50]: was not the like full speedSWYX [00:30:52]: one that we wouldJOSH [00:30:52]: eventually have. And so we are a little bit more kind of bottlenecked in terms of internet bandwidth. And so we had this. I think we looked at a bunch of services out there like Minio and some other ones, but a lot of these like come with a lot of extra overhead and maintenance. And since we already have so much infrastructureSWYX [00:31:09]: to deal with,JOSH [00:31:09]: we kind of didn't want to, you know, bring in a whole other like cloud provider, virtualize something, something.SWYX [00:31:14]: We just wanted something simple.JOSH [00:31:14]: So we went with that, which has been quite helpful. Like our toolsSWYX [00:31:19]: are usually quite simple.JOSH [00:31:19]: It's like Bash and Python and SSH and Docker. Like we'd like to keep things simple so that's easier to debug, like less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes,SWYX [00:31:30]: for example,JOSH [00:31:30]: and we just directly launch these things. And it's just been much easier to debug this way. One tool actually that does come into mind that I will call out is Kraken from Uber. That was great. We love that tool. We were a little bit skeptical. What is it?SWYX [00:31:44]: I'm sorry. Yeah.JOSH [00:31:45]: So Kraken is this, yeah, it's a distributed like Docker registry, basically, that uses BitTorrent to like transfer things between the machines in a sort of nice optimal way. Like in the very beginning, the naive way is like you have this one Docker registry, which was outside of the cluster. So every time we change an image, you know, there's many gigabytes that each of the 500 machines needs to download.SWYX [00:32:07]: So that just takesJOSH [00:32:07]: a really long time. So what this thing does is like just one of them downloads it and then like they all sort of broadcast all the pieces to each other. And it was just like a really nice, fast way of getting these images down. And it was very robust.SWYX [00:32:19]: Like there's a lotJOSH [00:32:19]: going on under the hood, but I think it's a pretty cool tool that we haven't really had any bugs with it at all. Amazing.SWYX [00:32:26]: Yeah. I mean, that's all my questions, I guess, for the info piece. I don't know if, John, you had something that you were sort of burning to ask or.JONATHAN [00:32:33]: No, all I can say is just sameSWYX [00:32:36]: in a lot of places, like, you know, and they're done thatJONATHAN [00:32:38]: seeing this plus one. I think the one big difference, you know, perhaps in philosophies is we've tried to basically standardize on as much commodity stuff as possible, just because, you know, I think the reason I asked about trying to do thisSWYX [00:32:50]: on multiple differentJONATHAN [00:32:50]: pieces of infrastructure is like, I think we're running on like six or seven different clouds right now. And everybody has done something slightly different. And my gosh, the little differences add up as you know, you've seen. And so, you know,SWYX [00:33:04]: our philosophy has been like, whatever the hellJONATHAN [00:33:05]: we can standardize, please let's standardize it. Like vanilla off the shelf FSDB.SWYX [00:33:10]: And like, you know,JONATHAN [00:33:10]: we wrote our own data loader, but we've tried to make that as much of a standard as we can across our infrastructure and in Databricks, because things just start getting really complicatedSWYX [00:33:18]: or like we useJONATHAN [00:33:18]: Kubernetes extensively because it at least gives us a uniform set of APIs. Like that's our hardware abstraction layer to a certain extent for everything else. So it's just, you know, a difference in philosophy there. But otherwise, like, yeah, this stuff is really, really hard. And I feel like we take for granted how much of this, you know, is done for us when you go and you just query chat GPT, for example. Like, oh my God, everything going on underneath that, you know, it's kind of a miracle that the machines boot up, let alone that you can like query a giant language model that's probably doing inference across multiple machines and was trained across thousands of machines. Like, you know, minor miracle.SWYX [00:33:54]: Yeah, it is an awesome amount of power that we invoke with a single API call that we take for granted these days. It's absurd. Yeah, I mean, like Kubernetes, like that point about Kubernetes, I will say as a former AWS employee, like it seems like it would be ideal for imbue to at some point make it more abstracted or agnostic because you're going to want to, you know, replicate your setup. We do have our ownJOSH [00:34:19]: sort of replacement. It's just a much simpler version of Kubernetes. Kubernetes is really designed for running services, not for running experiments. Like that's not its like main architecture. And so for us, like we have everything that's like, cool, you're going to run an experiment. So you want it to run to completion, right?SWYX [00:34:34]: OK, great.JOSH [00:34:34]: Like the primitives are sort of built around a slightly different style. And that makes it a lot easier, like just a lot simpler to fit that the nature of like these machines are going to disappear. They will need to be rebooted for infrastructure upgrades. They will like something will happen to the GPUs. Failure is like baked into this as like a core part of our infrastructure. So it's not that we don't have an abstraction. It's that it's a sort of simpler, more tailored abstraction for the particular work that we're doing.JONATHAN [00:34:58]: Yeah, I think it all depends on what your goals are. And like, I think the challenge in a lot of the deep learning stuff right now is that people are trying to like, people often build things that are more complicated than necessary to get the job done. And the complication is the enemy of everything. You know, don't use a fancier parallelism strategy than you have to. Don't use a fancier set of libraries than you have to.SWYX [00:35:18]: Don't do anythingJONATHAN [00:35:18]: that you don't have to do because it's hard enough as it is. Like, don't overcomplicateSWYX [00:35:23]: your own life.JONATHAN [00:35:23]: Don't try to bring in more tools or more fancy architecture tweaks if you absolutely don't have to.SWYX [00:35:29]: Like getting to the minimumJONATHAN [00:35:30]: necessary to get the job done. And it's really tempting to want to try to use everything. So like, I totally understand that one.SWYX [00:35:37]: I think the last piece I'll maybe call out is that I'm just going to weave this in just because I see the opportunity to do it. Are there any infrastructure shifts that need to be, that need to rise because of changing architecture? So I think, for example,SWYX [00:35:57]: you're announcing a dense model, a 70B dense model, whereas John just worked on DBRX and the image-to-text model, which presumably has different bottlenecks.JONATHAN [00:36:10]: That's correct for us. You know, we train both dense and mixture of expert models. The one we happened to, you know, kind of get permission to open source was a mixture of expert model. And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP 03 style, where there's just a lot of parameters getting shuffled back and forth. And your ratio of kind of compute to amount of data that you have to shuffle back and forth becomes a lot worse because you're now, you know, you're only using a fraction of the parameters for every token instead of all the parameters. And so we had to really push the envelope on getting all the stuff to the right places on time. And so actually the networking part of DBRX was the single hardest thing, I think, of the entire process. Just get MOE training, working at scale across a big cluster. We still managed to, I think, do it all with commodity parts, which was very exciting. You know, we were using FSTP and we eventually used HSTP so that we could have HSTP as a version of FSTP where you have multiple smaller replicas and you're doing data parallel within those replicas. And that helped a lot with network latency issues that we were running into just because we were transmitting so much data, you know, for every single part of the process. I think it actually, like, it was instructive for how Google designs their hardware and software together personally. Their training, as far as I understand, using kind of a 03 style of training and have been for a while. They also train mixture of expert models. TPUs have a very different network bandwidth to compute ratio. They have a lot more bandwidth just objectively. And TPUs per chip tend to be a little bit less compute intensive and have a little bit less memory. You know, it's just a different design choice. So the ratio of flops to bandwidth is very different. And that means that it's much easier for Google to be able to pull offSWYX [00:37:54]: some of this stuff.JONATHAN [00:37:54]: They also have interesting, you know, Torus style network architecture or Torus style, like, literal network architectureSWYX [00:38:00]: is not like the model,JONATHAN [00:38:00]: but the network.SWYX [00:38:02]: Is this the sort of block attention? I forgot what you call it. So this is just more or the,JONATHAN [00:38:07]: yeah, this is more, not the ring attention, but these are the ring all reduces. Like you have three different dimensions of rings because they kind of put you in these three dimensional Toruses from what I understand. And so like, you know, Google's infrastructure in some sense is kind of, I wouldn't say built for this, but maybe the way that Google trains models is built for a slightly different bit of infrastructure they have. And it's kind of neat to think about that. You know, as one thing that I think NVIDIA announced for, you know, for, for both the GH200 and the GB200 is this hybrid networking where you'll have blocks of NVLink network chips. I think for the GB200, I think it's like groups of 72 GPUs will all have NVLink to each other. So higher bandwidth, then you'll have normal networking of some kind, InfiniBand or Rocky or what have you between these blocks. And that's kind of a, you know, it's a change due to the fact that, you know, it's hard to build really high bandwidth networks over very large groups, but it is now a blocked networking. And you have to think about how you architect your model and your parallelism differently. You also have to think about fault tolerance differently because it now matters where you lose a GPU, whereas it didn't before. So, you know, it's, it's, it's just all really interesting and really fun speaking personally, but it's going to mean new nightmares when we all move to that generation and have to think about, you know, new versions of these problems.JOSH [00:39:20]: As you go up to larger scales, it gets quite different. Like right now, you know, if you're experiencing, let's say, for example, you experience a GPU failure every day, that's fine.SWYX [00:39:31]: Just restart.JOSH [00:39:31]: If you make your thing 24 times as big, now it's once an hour. Now it stops being quite as easy to just restart, right? So now you have to kind of break, like bake in this sort of redundancy that you didn't have before. So I think as you go up in scale, you end up running into like a lot of really interesting problems that also inform the, the actual like design. Yeah, I mean, as an orchestration guy,SWYX [00:39:52]: this is why I always emphasize like very cheap storage or very fast storage. So you can checkpoint more, but I don't think that's probably not the best solution to for fast, you know, training.JONATHAN [00:40:05]: Which works fine when you're doing language and then you move to vision or video. And then, you know, you have multi petabyte datasetsSWYX [00:40:12]: and getting, you know,JONATHAN [00:40:13]: cheap, fast multi petabyte storage starts to bite. Like I've certainly encountered issues where the literal data center where my GPUs were did not have enough, you know, object store to fit the datasets that people wanted to bring into that data center from whichever users were, were trying to bring them in. And then you get to a wholeSWYX [00:40:31]: different world of hurtJONATHAN [00:40:31]: where you have to keep your data in a different region because the region is just out of storage. So things get fun really fast.SWYX [00:40:39]: Speaking of vision, Josh, actually, you know, Embu is an agents company, but you're only, you're announcing a text-only model. What, where does, where does the vision side come in?JOSH [00:40:49]: I think we've actually done a lot of work in the past and people can see kind of our blog posts about sort of self-supervised learning and some other kind of vision-related stuff in the past as well. So we're very familiar with, with that stuff. But I think our main focus right now is on kind of, as we say, coding and reasoning. And there, there's certainly a visual component to some problems. But, you know, it's not necessarily required for all problems. And actually we found that for most of the kind of like code writing and, and reasoning problems that we care about, the visual part isn't really a huge important part of it. Sometimes if you really need to, you can maybe describeSWYX [00:41:24]: the thing.JOSH [00:41:24]: There are other like, you know, multimodal models that you can use off the shelf to sort of plug in for those particular piecesSWYX [00:41:30]: that you need, right?JOSH [00:41:30]: Like if something is driving a browser or whatever, like you can sometimes get away with not having to have that baked into the original model. So our folk were, you know, in a sense, we kind of do a lot across the stack. We're working on our own infrastructure and pre-training and RL and fine tuning and products and everything. But in another sense, we're very narrowly focused on the application side. So all of the stuff across the stack is kind of going toward a very particular purpose. And so that particular purpose right now doesn't really need vision. So we think that people are going to make all sorts of really cool image modelsSWYX [00:42:00]: like Jonathan, right?JOSH [00:42:00]: And all sorts of interesting multimodal models into the future. We'll let them go do that. That's great. We'll take advantage of that, partner with those people in the future. And right now we're really focused on kind of the core reasoning and coding capabilities and aspects of the model.SWYX [00:42:14]: I wanted to go into carbs since that's kind of the next layer of the stack. We talked about carbs in the first episode with Kanjin because you've actually had a blog post about it like a couple of years ago. Maybe let's introduce it.JONATHAN [00:42:26]: Has that been a couple of years now?JOSH [00:42:28]: No, it must have been at least one year. Hopefully it's not multiple years.SWYX [00:42:32]: Sorry, I'm counting AI time. Yeah, yeah. Yeah, I was going to sayJONATHAN [00:42:35]: you're making me feel really old right now.SWYX [00:42:39]: I count everything before the generally intelligent rename as like, you know, prehistory. Yeah. And now sort of modernity, right? So I actually thought carbs was more about hyperparameter optimization in a sense of like sort of parameters, hyperparameter search. Whereas, you know, when you introduced it, especially in this blog post, it's more about scaling laws and predictability of like, are we sort of in the right ballpark before we scale things up? Maybe sort of recount the history of carbs.JOSH [00:43:10]: Yeah, so it really is a little bit of both. So carbs is, it's maybe a backronym, but it's for cost aware Pareto region Bayesian search. So this is about technically how it works, but carbs is like, you know, we like pastries and stuff.SWYX [00:43:26]: So great, why not? But the point is thatJOSH [00:43:29]: it's a cost aware hyperparameter tuner. So most hyperparameter tuners, you kind of say, OK, here's this objective function. I want you to make this number as big as possible or as small as possible, whichever direction you want to go. So yeah, just go make this number, you know, as small as possible. OK, so it'll try a bunch of differentSWYX [00:43:46]: hyperparameters,JOSH [00:43:46]: a bunch of different configurationsSWYX [00:43:48]: to figure out, like,JOSH [00:43:48]: how do I tweak your network and architecture, et cetera, to get the kind of best performance I possibly can. That's usually saying, like, you know, almost all of these hyperparameter configurations are, let's say they're all going to use the same number of GPUs or the same number of nodes.SWYX [00:44:01]: So it's going to runJOSH [00:44:01]: for the same amount of time.SWYX [00:44:03]: So you can do that.JOSH [00:44:03]: You can get a number out and that's great. But what carbs does is it says,SWYX [00:44:07]: OK, actually,JOSH [00:44:07]: what if we relax that constraint? What if we say each of these different points, we're going to model how expensive it will be to sample this configuration. So if what if we train with just one one hundredth of the data? Like, how well can we do?SWYX [00:44:19]: What if we trainJOSH [00:44:19]: with one tenth of the data? What if we train with all the data? That way you can understand, like, as we get more and more data, as we spend more and more compute,SWYX [00:44:26]: as we make a biggerJOSH [00:44:26]: and bigger network, how does performance change with these things that change? Like how expensive it is to even explore this data point. So by doing that, we can see the scaling laws for not just, you know,SWYX [00:44:36]: the scaling lawsJOSH [00:44:36]: from like the, you know, Chantilla paper, the scaling laws for all parameters. We can see how does how does the number of layers change with this? How does the, you know, the learning rate change? How do the like, you know, various types of regularization change? So you can see these nice scaling laws. And as you're going across costs, like how should this be changing as you're scaling up your model? So that, coupled with the kind of metric that we chose, which is a very precise way of measuring performance, allowed us to really like hone in on parameters that worked really wellSWYX [00:45:05]: and understand, like,JOSH [00:45:05]: how do we want to scale those up, especially as we're changingSWYX [00:45:08]: things about the network?JOSH [00:45:08]: Like one of the things that we did is we used a custom tokenizer. As we change this tokenizer, changes a bunch of other things about the model. So how should we scale up this entirely new tokenizer? Like no one has ever made a model this large with this tokenizer before. And so how do we want toSWYX [00:45:22]: change all these things?JOSH [00:45:22]: Harps kind of shows you, like, look, as you change these parameters, like these other ones are kind of dependent on this.SWYX [00:45:28]: Like this is the, these areJOSH [00:45:28]: the relationships between them. So you can better understand, like, OK, if I'm going to scale this up 10x or 100x, like, where do I want to be? I can only go so far. And so, you know, we did run, like, I think maybe it was like a 14b one or somethingSWYX [00:45:40]: like that to check.JOSH [00:45:41]: But and so we had a bunch of like 1b or 14b and then at 70b. I don't think we had a, I think we just did like one at 14b. So you can, we get to check that like, oh, is this on the curve? Like, is this where we expect? It was like right there. So then great, go on to the next one. Yeah, I mean, that makes a lot of sense.SWYX [00:45:56]: I wonder if, so one of the key questions, and correct me if I'm wrong, but like usually people do search or do their evals just based on loss. But you actually evaluate based on, you know, the sort of end state evals that people might expect, like HellaSwag and Lombata, whatever. What is the norm here? Is there a norm?JOSH [00:46:20]: Yeah, I don't know if there's a hundred percent.SWYX [00:46:21]: I don't know. I only see loss on most people's reports.JOSH [00:46:25]: I think it's easy to, like, loss is very nice because it's very precise. It will tell you, like, very fine grained differences between like really small changes in your hyperparameters or network architecture. Whereas, especially at the smaller scales, if you're looking at like accuracy, it's very noisy. Like it might be zero or a hundred or like, you know, fluctuating by like 10 or 20 percentage points, which makes it really hard to tell, like, did that change actually mean anything? So our loss is sort of a combination of these two. Instead of saying, like, let's just look at perplexity, we say, let's look at perplexity on the tasks that we care about for multiple choice questions effectively.SWYX [00:47:00]: So we're saying like, yes,JOSH [00:47:00]: this is formulated as a multiple choice question, and we're going to look at the, like, you know, the loss of perplexity for this particular answer token. And that ends up being something that's like both targeted to what you actually care about and also very precise. The nice thing about this though is that it's independent of the data that you train on. One thing that's annoying about perplexity or about loss is that as you change your data set, this is really obnoxious because now it fundamentally changes your loss, right? And so you can't tell, like, how do I tweak my data set? But because we have this held out evaluation dat

TECNOLOGIA y LIBERTAD--------------------------PODCAST: https://www.spreaker.com/user/dekkartwitter.com/D3kkaRtwitter.com/Dek_Netmastodon.social/@DekkaRCódigo referido Crypto.com: https://crypto.com/app/hhsww88jd4#Bitcoin BTC: dekkar$paystring.crypt

TECNOLOGIA y LIBERTAD--------------------------PODCAST: https://www.spreaker.com/user/dekkartwitter.com/D3kkaRtwitter.com/Dek_Netmastodon.social/@DekkaRCódigo referido Crypto.com: https://crypto.com/app/hhsww88jd4#Bitcoin BTC: dekkar$paystring.crypt

We spoke with Jimmy Zelinskie, Co-founder of AuthZed, about his significant history in the cloud native ecosystem and open source community. We learn about his start writing BitTorrent software in college and his eventual startups including AuthZed, which develops SpiceDB. SpiceDB, inspired by Google's internal Zanzibar system, aims to democratize complex authorization solutions for various applications. The discussion also covers the evolution of Kubernetes, the role of open source in fostering innovation, and the significance of community collaboration in addressing common challenges in authorization and secure system design. Jimmy's experience showcases the benefits of open source for solving intricate problems and his commitment to making sophisticated technology accessible to a broader audience. 00:00 Introduction 00:14 The Buzz of KubeCon 00:50 Introducing Jimmy: From BitTorrent to Kubernetes Pioneer 03:41 The Journey of SpiceDB: From Concept to Adoption 06:43 Navigating the Authorization Ecosystem 11:55 The Future of Authorization: Innovations and Roadmap 14:32 Personal Insights: Open Source Journey and Product Management 19:08 KubeCon Highlights and Kubernetes API Proxy Announcement   Guest: Jimmy Zelinskie is a software engineer and product leader with the goal of empowering the world through the democratization of software through open source development. He's currently the CPO and cofounder of authzed where he's focused on bringing hyperscaler best-practices in authorization software to the industry at large. At CoreOS, he helped pioneer the cloud-native ecosystem by starting and contributing to many of its foundational open source projects. After being acquired by Red Hat, his focus shifted to the enablement and adoption of cloud-native technologies by mature enterprise stakeholders. To this day, he still contributes to cloud-native ecosystem by building the future on top of these technologies and maintaining standards such as Open Container Initiative (OCI).

La llegada de la primera tienda alternativa y los emuladores de software son un soplo de aire fresco para nuestros iPhone. Nos ha gustado mucho poder probarlos.Matías comparte su experiencia instalando un emulador de Nintendo en su dispositivo. Más allá, vemos un futuro con posibles aplicaciones y usos de las tiendas alternativas, como el contenido adulto, aplicaciones de criptomonedas y clientes de BitTorrent.Los antiguos empleados de Apple protagonizan este episodio, los creadores del Humane AI Pin y los 700 despedidos recientemente por la compañía, Jony Ive y su nueva compañía de IA con Sam Altman, que busca financiamiento de mil millones de dólares. Finalmente, discuten los rumores sobre los planes de Apple en el campo de la robótica doméstica.Apple renovará la gama iPad en breve y preparan nuevos AirPods algo más baratos, aunque nos gustaría que siguieran el camino de los Fairbuds con sus pilas intercambiables.Por último, destacamos el aumento en el precio de los fallos de seguridad y el lanzamiento de un spin-off de la serie 'Para toda la humanidad'. Un gran episodio. [AltStore][1] [Apple struck deal with Shutterstock to license millions of images for AI training - 9to5Mac][2] [Apple prepara unos AirPods más baratos y la segunda generación de los AirPods Max para este mismo año][3] [Fairbuds de Fairphone][4] [Apple ahora permite emuladores de juegos en su App Store tras prohibirlos por años WIRED][5] [Exploit Acquisition Program - Crowdfense][6] [Lorenzo Franceschi-Bicchierai: «NEW: A startup is offering up to $7 million for i…» - Infosec Exchange][7] [La startup de dispositivos de IA de Jony Ive y Sam Altman busca 1.000 millones de dólares de las principales empresas de capital riesgo • Contxto][8] [despidos apple - Buscar con Google][9] [Apple despide a más de 600 personas tras abandonar dos grandes proyectos Business Insider España][10] [Tras cancelar su primer coche eléctrico, Apple estaría trabajando en unos robots domésticos][11] [1]: https://altstore.io/ [2]: https://9to5mac.com/2024/04/06/apple-ai-deal-shutterstock/ [3]: https://www.lavanguardia.com/andro4all/apple/apple-prepara-unos-airpods-mas-baratos-y-la-segunda-generacion-de-los-airpods-max-para-este-mismo-ano [4]: https://shop.fairphone.com/es/fairbuds [5]: https://es.wired.com/articulos/apple-ahora-permite-emuladores-de-juegos-en-su-app-store-tras-prohibirlos-por-anos [6]: https://www.crowdfense.com/exploit-acquisition-program/ [7]: https://infosec.exchange/@lorenzofb/112225242378698103 [8]: https://contxto.com/es/news/jony-ive-and-sam-altmans-ai-device-startup-seeks-1b-from-top-vcs/ [9]: https://www.google.com/search?q=despidos+apple&oq=despidos+apple&gs_lcrp=EgZjaHJvbWUqCQgAEEUYOxiABDIJCAAQRRg7GIAEMgoIARAAGIAEGKIEMgoIAhAAGIAEGKIEMgoIAxAAGIAEGKIEMgoIBBAAGIAEGKIEMgYIBRBFGDzSAQgxNzk4ajBqN6gCALACAA&sourceid=chrome&ie=UTF-8 [10]: https://www.businessinsider.es/apple-despide-600-personas-abandonar-dos-grandes-proyectos-1377320 [11]: https://www.lavanguardia.com/andro4all/apple/tras-cancelar-su-primer-coche-electrico-apple-estaria-trabajando-en-unos-robots-domesticosLa llegada de la primera tienda alternativa y los emuladores de software son un soplo de aire fresco para nuestros iPhone. Nos ha gustado mucho poder probarlos.Matías comparte su experiencia instalando un emulador de Nintendo en su dispositivo. Más allá, vemos un futuro con posibles aplicaciones y usos de las tiendas alternativas, como el contenido adulto, aplicaciones de criptomonedas y clientes de BitTorrent.Los antiguos empleados de Apple protagonizan este episodio, los creadores del Humane AI Pin y los 700 despedidos recientemente por la compañía, Jony Ive y su nueva compañía de IA con Sam Altman, que busca financiamiento de mil millones de dólares. Finalmente, discuten los rumores sobre los planes de Apple en el campo de la robótica doméstica.Apple renovará la gama iPad en breve y preparan nuevos AirPods algo más baratos, aunque nos gustaría que siguieran el camino de los Fairbuds con sus pilas intercambiables.Por último, destacamos el aumento en el precio de los fallos de seguridad y el lanzamiento de un spin-off de la serie 'Para toda la humanidad'. Un gran episodio. [AltStore][1] [Apple struck deal with Shutterstock to license millions of images for AI training - 9to5Mac][2] [Apple prepara unos AirPods más baratos y la segunda generación de los AirPods Max para este mismo año][3] [Fairbuds de Fairphone][4] [Apple ahora permite emuladores de juegos en su App Store tras prohibirlos por años WIRED][5] [Exploit Acquisition Program - Crowdfense][6] [Lorenzo Franceschi-Bicchierai: «NEW: A startup is offering up to $7 million for i…» - Infosec Exchange][7] [La startup de dispositivos de IA de Jony Ive y Sam Altman busca 1.000 millones de dólares de las principales empresas de capital riesgo • Contxto][8] [despidos apple - Buscar con Google][9] [Apple despide a más de 600 personas tras abandonar dos grandes proyectos Business Insider España][10] [Tras cancelar su primer coche eléctrico, Apple estaría trabajando en unos robots domésticos][11] [1]: https://altstore.io/ [2]: https://9to5mac.com/2024/04/06/apple-ai-deal-shutterstock/ [3]: https://www.lavanguardia.com/andro4all/apple/apple-prepara-unos-airpods-mas-baratos-y-la-segunda-generacion-de-los-airpods-max-para-este-mismo-ano [4]: https://shop.fairphone.com/es/fairbuds [5]: https://es.wired.com/articulos/apple-ahora-permite-emuladores-de-juegos-en-su-app-store-tras-prohibirlos-por-anos [6]: https://www.crowdfense.com/exploit-acquisition-program/ [7]: https://infosec.exchange/@lorenzofb/112225242378698103 [8]: https://contxto.com/es/news/jony-ive-and-sam-altmans-ai-device-startup-seeks-1b-from-top-vcs/ [9]: https://www.google.com/search?q=despidos+apple&oq=despidos+apple&gs_lcrp=EgZjaHJvbWUqCQgAEEUYOxiABDIJCAAQRRg7GIAEMgoIARAAGIAEGKIEMgoIAhAAGIAEGKIEMgoIAxAAGIAEGKIEMgoIBBAAGIAEGKIEMgYIBRBFGDzSAQgxNzk4ajBqN6gCALACAA&sourceid=chrome&ie=UTF-8 [10]: https://www.businessinsider.es/apple-despide-600-personas-abandonar-dos-grandes-proyectos-1377320 [11]: https://www.lavanguardia.com/andro4all/apple/tras-cancelar-su-primer-coche-electrico-apple-estaria-trabajando-en-unos-robots-domesticos

Ever wondered what real torture feels like? Ronny Chieng has the answer, and it's not what you'd expect. In a hilarious twist, Ronny takes us through the digital age's most daunting challenge: explaining tech to parents. From the myths of online piracy costing billions (spoiler: it's free!) to the dreaded task of guiding his mom through deleting a YouTube comment, Ronny's tale is a rollercoaster of tech woes and parental puzzles. With sharp wit and a keen eye for the absurdities of internet culture, he navigates us through a world where explaining BitTorrent feels like defusing a bomb, and a mouse click can unleash chaos. Ready for a laugh that hits close to home? Head over to foqnfunny.com for a full dose of Ronny Chieng's comedic genius, where tech support turns into a laugh riot! Love what you're hearing on FOQN Funny? Go a step further and become a member of FOQN Funny+. Enjoy exclusive perks and never-ending laughter. Join now at: https://plus.acast.com/s/foqn-funny. Hosted on Acast. See acast.com/privacy for more information.

Gabriel Custodiet talks to Paul Rosenberg for the third time. They discuss the original cryptography movement, skepticism of the Ross Ulbricht case, how the Dark Ages ended slavery, Christian anarchy, and the biological future of humanity.   Guest Links →Paul Rosenberg's website: https://www.freemansperspective.com   Previous episodes with Paul Rosenberg →Episode 16: https://youtu.be/JRcS6fvukzA →Episode 54: https://youtu.be/WCU9f_9s120   Watchman Privacy → https://watchmanprivacy.com (newsletter, consultation requests) → https://twitter.com/watchmanprivacy → https://www.amazon.com/Watchman-Guide-Privacy-Financial-Lifestyle/dp/B08PX7KFS2   Privacy Courses (supports the show) → https://rpf.gumroad.com/l/privatebitcoin → https://rpf.gumroad.com/l/hackproof   Monero Donation (supports the show) →8829DiYwJ344peEM7SzUspMtgUWKAjGJRHmu4Q6R8kEWMpafiXPPNBkeRBhNPK6sw27urqqMYTWWXZrsX6BLRrj7HiooPAy   Please subscribe to and rate this podcast wherever you can to help it thrive. Thank you! → https://www.youtube.com/@WatchmanPrivacy  →https://odysee.com/@WatchmanPrivacy   Timeline 0:00 – Introduction 1:08 – Original cryptography movement 4:35 – The universe favors encryption 9:10 – Cryptography and the government 11:50 – Cryptography resources 15:30 – What Julian Assange means to humanity 18:09 – Significance of BitTorrent 18:55 – Insight into VPN industry 22:30 – Questionable aspects of Ross Ulbricht case 26:10 – The Dark Ages: why were they significant?  31:45 – Resources for the Dark Ages 33:45 – Are Christianity and Judaism anarchic religions? 37:30 – Paul's new book, Post-Primate Society 41:05 – Evolutionary psychology hijacking the mind 45:50 – How damaging would an EMP be? 48:25 – Final thoughts #WatchmanPrivacy #PaulRosenberg #FreemansPerspective #Cryptography

John Sneisen joins us again to chat about The Economic Truth, Canada's economy and inflation, the free market, Melei's speech in Davos, the end game, civic failure, solutions to get out of this mess (they are not easy), Anarcho-capitalism, comparisons between Norway and Canada, individualism vs collectivism, private debt and taxes in Canada.   In the second half we get into telecom communication examples, speaking to the layman on inflation, how Canada fudges the numbers, Stats Canada, Interest Rates, Government programs, the protection of Corporations, assets, peer to peer solutions like Outerrnet, the ponzi shceme, hyper inflation, and the psychological shift that will happen when people stop trusting their currency.   https://theeconomictruth.org/ https://outerr.net/   To gain access to the second half of show and our Plus feed for audio and podcast please clink the link http://www.grimericaoutlawed.ca/support.   For second half of video (when applicable and audio) go to our Substack and Subscribe. +   https://grimericaoutlawed.substack.com/ or to our Locals  https://grimericaoutlawed.locals.com/ or Rokfin www.Rokfin.com/Grimerica   If you would rather watch: https://rumble.com/v47s0nu-john-sneisen.-canadas-i-situation-and-the-new-decentralized-interne.html https://rokfin.com/stream/44067 https://grimericaoutlawed.locals.com/post/5145512/john-sneisen-canadas-financial-situation-and-the-new-decentralized-internet https://www.youtube.com/watch?v=qoXjnyju99U   Help support the show, because we can't do it without ya. If you value this content with 0 ads, 0 sponsorships, 0 breaks, 0 portals and links to corporate websites, please assist. Many hours of unlimited content for free. Thanks for listening!!   Support the show directly: https://grimerica.ca/support-2/ Our Adultbrain Audiobook Podcast and Website: www.adultbrain.ca Our Audiobook Youtube Channel:  https://www.youtube.com/@adultbrainaudiobookpublishing/videos Grimerica Media Youtube Channel: https://www.youtube.com/@grimerica/featured Darren's book www.acanadianshame.ca Check out our next trip/conference/meetup - Contact at the Cabin www.contactatthecabin.com Other affiliated shows: www.grimerica.ca The OG Grimerica Show www.Rokfin.com/Grimerica Our channel on free speech Rokfin Join the chat / hangout with a bunch of fellow Grimericans  Https://t.me.grimerica https://www.guilded.gg/chat/b7af7266-771d-427f-978c-872a7962a6c2?messageId=c1e1c7cd-c6e9-4eaf-abc9-e6ec0be89ff3   Get your Magic Mushrooms delivered from: Champignon Magique  Mushroom Spores, Spore Syringes, Best Spore Syringes,Grow Mushrooms Spores Lab Get Psychedelics online Leave a review on iTunes and/or Stitcher: https://itunes.apple.com/ca/podcast/grimerica-outlawed http://www.stitcher.com/podcast/grimerica-outlawed Sign up for our newsletter http://www.grimerica.ca/news SPAM Graham = and send him your synchronicities, feedback, strange experiences and psychedelic trip reports!! graham@grimerica.com InstaGRAM https://www.instagram.com/the_grimerica_show_podcast/  Purchase swag, with partial proceeds donated to the show www.grimerica.ca/swag Send us a postcard or letter http://www.grimerica.ca/contact/ ART - Napolean Duheme's site http://www.lostbreadcomic.com/  MUSIC Tru Northperception, Felix's Site sirfelix.bandcamp.com    

The most frustrating privacy myths we see regularly, using Bittorrent as a file sync, the best VoIP option for Android, and more from SR158Q&A, join our next Q&A on Patreon: https://www.patreon.com/surveillancepodSupport us on PatreonSupport us on LiberapayMonero: 46iGe5D49rpgH4dde32rmyWifMjw5sHy7V2mD9sXGDJgSWmAwQvuAuoD9KcLFKYFsLGLpzXQs1eABRShm1RZRnSy6HgbhQDTimestamps00:00 Introduction00:22 Most frustrating privacy myths06:11 Bittorrent as a file sync08:08 Best VoIP option for Android10:39 Stacking data removal services15:07 Mobile biometric authenication security17:36 Follow Up: Online Purchases & Threat Modeling in the UK22:11 On-the-ground advice on travel to ChinaTechlore Talks #3:Response: https://psyvacy.substack.com/p/response-to-techlore-talks-3 This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit surveillancepod.substack.com

Guest Sean Nelson, singer of Harvey Danger shares about the history of the band, their release of an album over BitTorrent, and more! Check out his solo albums on Spotify. https://open.spotify.com/artist/40nFUctaa9AWAPMODGUQkD?si=H0FoTU1ZT0O3LOa5uEA47w

Join Car and Keyan to discuss Stacker News' top posts of the week, Car & Keyan's favorite posts of the week, and top stackers for the week of Dec 1st, 2023.03:01 - Luke Dashjr launches Ocean to radically decentralize Bitcoin mining07:26 - Is OCEAN truly a 'decentralized‘ mining pool?10:15 - The plebdevs backend course is here!18:27 - The Art of Swapping21:52 - Dtan.xyz - Bittorrent on Nostr!24:02 - How to onboard new bitcoin users as a merchant25:58 - Cycles and inevitability28:05 - MicroStrategy Acquires Additional 16,130 Bitcoins and Now Holds 174,530 BTC30:56 - Core Lightning v23.11, Codenamed “Bitcoin Orangepaper,” Is Now Live!31:36 - The Kevin Rooke Show - Christian Decker - Blockstream Greenlight, LightningShoutout @Wumbo for time stamps. Zap em!Stacker News Live WebsiteThriller Bitcoin SN NewsletterPlease find us on YouTube & BitcoinTVWe love the Fountain app for Lightning 2.0 podcastingSend us a Boost, and we will read it on the next SNLFind Car on NostrFollow Car on SNLearn more about PlebLabFollow Keyan on TwitterFind Keyan on NostrFollow Keyan on SNLearn more about Stacker News

"A recent anonymous proposal specifies a system using timelocked Bitcoin transactions and chaumian ecash mints to create an economic incentive for BitTorrent users to seed files." ~ Shinobi Bittorrent was one of the most important protocols in the world during its day, but it always had one striking drawback, for any content that wasn't very popular, availability was always a problem. Even more so today as the network has found itself a smaller and more fringe part of internet culture. What if there was a way to incentivize seeders to return? To bring the P2P protocol back into its element? An anonymous proposal called Durabit, seeks to marry bitcoin, ecash, and bittorrent, to potentially breathe new life into the old P2P system. Listen to today's episode to learn more. Check out the original article at DURABIT: INCENTIVIZING TORRENT SEEDING WITH BITCOIN (Link: https://tinyurl.com/5drhn83u) Guest Links Shinobi on X (Link: https://twitter.com/Brian_trollz) References in the show Simon Morris "Lessons of Bittorrent Series" Bitcoin Audible - CryptoQuikRead_219 - Why BitTorrent Mattered: Lesson 1 [Simon Morris] (Link: https://tinyurl.com/4rmwf2k8) Bitcoin Audible - CryptoQuikRead_220 - Not Breaking Rules? You're Doing it Wrong: BitTorrent Lesson 2 [Simon Morris] (Link: https://tinyurl.com/f473b72n) Bitcoin Audible - CryptoQuikRead_221 - Intent, Complexity, & Governance Paradox: BitTorrent Lesson 3 [Simon Morris] (Link: https://tinyurl.com/2fesdh4v) Bitcoin Audible - CryptoQuikRead_222 - Decentralized Disruption, Who Dares Wins: BitTorrent Lesson 4 [Simon Morris] (Link: https://tinyurl.com/bdht8mz6) The Audiobook for Broken Money is now available! Check it out below Broken Money: Why Our Financial System is Failing Us and How We Can Make it Better by Lyn Alden (Link: https://a.co/d/a3dT5jr⁠) Host Links Guy on Nostr (Link: https://tinyurl.com/yc376bff) Guy on X (Link: https://twitter.com/theguyswann) Bitcoin Audible on X (Link: https://twitter.com/BitcoinAudible) Check out our awesome sponsors! Get 9% off the COLDCARD with code BITCOINAUDIBLE ⁠⁠⁠⁠⁠⁠(Link: bitcoinaudible.com/coldcard⁠⁠⁠⁠⁠⁠) Get 50,000 free sats, plus sats back on everything you buy by signing up with Fold and the Spin+ Debit Card. ⁠⁠⁠⁠⁠⁠(Link: bitcoinaudible.com/fold⁠) Check out the KYC Free, simple, no management, Bitcoin merchant solution - ⁠⁠⁠⁠⁠Nodeless.io⁠⁠⁠⁠⁠⁠. Use my link to go to nodeless.io/guy⁠⁠⁠⁠⁠ and setup your donation page for free, then share it out on nostr/Twitter and tag me, and I'll throw 10,000 sats your way! "The universe, our physical universe, has that property that makes it possible for an individual or a group of individuals to reliably, automatically, even without knowing, encipher something, so that all the resources and all the political will of the strongest superpower on earth may not decipher it. And the paths of encipherment between people can mesh together to create regions free from the coercive force of the outer state. Free from mass interception. Free from state control." ~ Julian Assange --- Send in a voice message: https://podcasters.spotify.com/pod/show/bitcoinaudible/message

"A recent anonymous proposal specifies a system using timelocked Bitcoin transactions and chaumian ecash mints to create an economic incentive for BitTorrent users to seed files." ~ Shinobi Bittorrent was one of the most important protocols in the world during its day, but it always had one striking drawback, for any content that wasn't very popular, availability was always a problem. Even more so today as the network has found itself a smaller and more fringe part of internet culture. What if there was a way to incentivize seeders to return? To bring the P2P protocol back into its element? An anonymous proposal called Durabit, seeks to marry bitcoin, ecash, and bittorrent, to potentially breathe new life into the old P2P system. Listen to today's episode to learn more. Check out the original article at DURABIT: INCENTIVIZING TORRENT SEEDING WITH BITCOIN (Link: https://tinyurl.com/5drhn83u) Guest Links Shinobi on X (Link: https://twitter.com/Brian_trollz) References in the show Simon Morris "Lessons of Bittorrent Series" Bitcoin Audible - CryptoQuikRead_219 - Why BitTorrent Mattered: Lesson 1 [Simon Morris] (Link: https://tinyurl.com/4rmwf2k8) Bitcoin Audible - CryptoQuikRead_220 - Not Breaking Rules? You're Doing it Wrong: BitTorrent Lesson 2 [Simon Morris] (Link: https://tinyurl.com/f473b72n) Bitcoin Audible - CryptoQuikRead_221 - Intent, Complexity, & Governance Paradox: BitTorrent Lesson 3 [Simon Morris] (Link: https://tinyurl.com/2fesdh4v) Bitcoin Audible - CryptoQuikRead_222 - Decentralized Disruption, Who Dares Wins: BitTorrent Lesson 4 [Simon Morris] (Link: https://tinyurl.com/bdht8mz6) The Audiobook for Broken Money is now available! Check it out below Broken Money: Why Our Financial System is Failing Us and How We Can Make it Better by Lyn Alden (Link: https://a.co/d/a3dT5jr⁠) Host Links Guy on Nostr (Link: https://tinyurl.com/yc376bff) Guy on X (Link: https://twitter.com/theguyswann) Bitcoin Audible on X (Link: https://twitter.com/BitcoinAudible) Check out our awesome sponsors! Get 9% off the COLDCARD with code BITCOINAUDIBLE ⁠⁠⁠⁠⁠⁠(Link: bitcoinaudible.com/coldcard⁠⁠⁠⁠⁠⁠) Get 50,000 free sats, plus sats back on everything you buy by signing up with Fold and the Spin+ Debit Card. ⁠⁠⁠⁠⁠⁠(Link: bitcoinaudible.com/fold⁠) Check out the KYC Free, simple, no management, Bitcoin merchant solution - ⁠⁠⁠⁠⁠Nodeless.io⁠⁠⁠⁠⁠⁠. Use my link to go to nodeless.io/guy⁠⁠⁠⁠⁠ and setup your donation page for free, then share it out on nostr/Twitter and tag me, and I'll throw 10,000 sats your way! "The universe, our physical universe, has that property that makes it possible for an individual or a group of individuals to reliably, automatically, even without knowing, encipher something, so that all the resources and all the political will of the strongest superpower on earth may not decipher it. And the paths of encipherment between people can mesh together to create regions free from the coercive force of the outer state. Free from mass interception. Free from state control." ~ Julian Assange --- Send in a voice message: https://podcasters.spotify.com/pod/show/bitcoinaudible/message

"Decentralization will not, absolutely not, be the reason for the average user to use Web3. The average user is not going to use Web3 until it's as easy to use as Web2 and unlocks features that Web2 cannot provide."  This is a powerful quote from this episode's conversation with Ian Andrews (CMO, Chainalysis) and the, Co- Executive Directors of the Web3 Working Group, Amy James and Devon James as they unravel the technical complexities of Decentralized Physical Infrastructure Networks (DePIN) and explain why they believe the next opportunity for wider adoption will not just be adoption in the financial space, but adoption in the technology space. The duo also emphasize the need for open protocols and standards to ensure interoperability and competition in the space, as well as the challenges of building in the Web3 space and the importance of educating policymakers and regulators about the potential for these technologies. Minute-by-minute episode breakdown (2:01) - What is the Web3 Working Group and the need for decentralized physical infrastructure networks (DePins)  (6:08) - Discussing the Open Index Protocol and transitioning from manufacturing to software and Bitcoin (11:20) - Advantages of the protocol and comparison to IPFS and use of BitTorrent and blockchain for indexing content (16:45) - Difficulty in building web3 experiences beyond financial transactions (20:50) - Decentralization won't be the reason for average users to use Web3 (28:02) - Reasons for creating a not-for-profit advocacy group for Web3 and challenges of operating in the US market due to regulations (34:08) - Educating regulators and policymakers on Web3 technology (38:27) - Potential benefits and concerns of traditional finance players entering the crypto space Related resources Check out more resources provided by Chainalysis that perfectly complement this episode of the Public Key. Website:  Web3 Working Group: Advocacy for Decentralized Infrastructure Protocols Youtube: Weekly Youtube Live Show: What Kind of Internet Do You Want?  Podcast: DarkHorse Podcast with Bret Weinstein  Webpage:  Explore DePIN protocols and the decentralized apps that use them Article: What is DePIN? Blog: Risky Business: The Economics Behind Decentralized Storage Networks Blog: OFAC Sanctions Russian National Ekaterina Zhdanova for Using Cryptocurrency to Launder Money on Behalf of Russian Elites and Ransomware Groups Report: The Chainalysis 2023 Geography of Cryptocurrency Report (Available Now) YouTube: Chainalysis YouTube page Twitter: Chainalysis Twitter: BuildCareers at Chainalysising trust in blockchain Tik Tok: Building trust in #blockchains among people, businesses, and governments. Telegram: Chainalysis on Telegram  Speakers on today's episode Ian Andrews * Host * (Chief Marketing Officer, Chainalysis) Amy James (Co- Executive Director, Web3 Working Group) Devon James (Co- Executive Director, Web3 Working Group) This website may contain links to third-party sites that are not under the control of Chainalysis, Inc. or its affiliates (collectively “Chainalysis”). Access to such information does not imply association with, endorsement of, approval of, or recommendation by Chainalysis of the site or its operators, and Chainalysis is not responsible for the products, services, or other content hosted therein. Our podcasts are for informational purposes only, and are not intended to provide legal, tax, financial, or investment advice. Listeners should consult their own advisors before making these types of decisions. Chainalysis has no responsibility or liability for any decision made or any other acts or omissions in connection with your use of this material. Chainalysis does not guarantee or warrant the accuracy, completeness, timeliness, suitability or validity of the information in any particular podcast and will not be responsible for any claim attributable to errors, omissions, or other inaccuracies of any part of such material.  Unless stated otherwise, reference to any specific product or entity does not constitute an endorsement or recommendation by Chainalysis. The views expressed by guests are their own and their appearance on the program does not imply an endorsement of them or any entity they represent. Views and opinions expressed by Chainalysis employees are those of the employees and do not necessarily reflect the views of the company.   

SF folks: join us at the AI Engineer Foundation's Emergency Hackathon tomorrow and consider the Newton if you'd like to cowork in the heart of the Cerebral Arena.Our community page is up to date as usual!~800,000 developers watched OpenAI Dev Day, ~8,000 of whom listened along live on our ThursdAI x Latent Space, and ~800 of whom got tickets to attend in person:OpenAI's first developer conference easily surpassed most people's lowballed expectations - they simply did everything short of announcing GPT-5, including:* ChatGPT (the consumer facing product)* GPT4 Turbo already in ChatGPT (running faster, with an April 2023 cutoff), all noticed by users weeks before the conference* Model picker eliminated, God Model chooses for you* GPTs - “tailored version of ChatGPT for a specific purpose” - stopping short of “Agents”. With custom instructions, expanded knowledge, and actions, and an intuitive no-code GPT Builder UI (we tried all these on our livestream yesterday and found some issues, but also were able to ship interesting GPTs very quickly) and a GPT store with revenue sharing (an important criticism we focused on in our episode on ChatGPT Plugins)* API (the developer facing product)* APIs for Dall-E 3, GPT4 Vision, Code Interpreter (RIP Advanced Data Analysis), GPT4 Finetuning and (surprise!) Text to Speech* many thought each of these would take much longer to arrive* usable in curl and in playground* BYO Interpreter + Async Agents?* Assistant API: stateful API backing “GPTs” like apps, with support for calling multiple tools in parallel, persistent Threads (storing message history, unlimited context window with some asterisks), and uploading/accessing Files (with a possibly-too-simple RAG algorithm, and expensive pricing)* Whisper 3 announced and open sourced (HuggingFace recap)* Price drops for a bunch of things!* Misc: Custom Models for big spending ($2-3m) customers, Copyright Shield, SatyaThe progress here feels fast, but it is mostly (incredible) last-mile execution on model capabilities that we already knew to exist. On reflection it is important to understand that the one guiding principle of OpenAI, even more than being Open (we address that in part 2 of today's pod), is that slow takeoff of AGI is the best scenario for humanity, and that this is what slow takeoff looks like:When introducing GPTs, Sam was careful to assert that “gradual iterative deployment is the best way to address the safety challenges with AI”:This is why, in fact, GPTs and Assistants are intentionally underpowered, and it is a useful exercise to consider what else OpenAI continues to consider dangerous (for example, many people consider a while(true) loop a core driver of an agent, which GPTs conspicuously lack, though Lilian Weng of OpenAI does not).We convened the crew to deliver the best recap of OpenAI Dev Day in Latent Space pod style, with a 1hr deep dive with the Functions pod crew from 5 months ago, and then another hour with past and future guests live from the venue itself, discussing various elements of how these updates affect their thinking and startups. Enjoy!Show Notes* swyx live thread (see pinned messages in Twitter Space for extra links from community)* Newton AI Coworking Interest Form in the heart of the Cerebral ArenaTimestamps* [00:00:00] Introduction* [00:01:59] Part I: Latent Space Pod Recap* [00:06:16] GPT4 Turbo and Assistant API* [00:13:45] JSON mode* [00:15:39] Plugins vs GPT Actions* [00:16:48] What is a "GPT"?* [00:21:02] Criticism: the God Model* [00:22:48] Criticism: ChatGPT changes* [00:25:59] "GPTs" is a genius marketing move* [00:26:59] RIP Advanced Data Analysis* [00:28:50] GPT Creator as AI Prompt Engineer* [00:31:16] Zapier and Prompt Injection* [00:34:09] Copyright Shield* [00:38:03] Sharable GPTs solve the API distribution issue* [00:39:07] Voice* [00:44:59] Vision* [00:49:48] In person experience* [00:55:11] Part II: Spot Interviews* [00:56:05] Jim Fan (Nvidia - High Level Takeaways)* [01:05:35] Raza Habib (Humanloop) - Foundation Model Ops* [01:13:59] Surya Dantuluri (Stealth) - RIP Plugins* [01:21:20] Reid Robinson (Zapier) - AI Actions for GPTs* [01:31:19] Div Garg (MultiOn) - GPT4V for Agents* [01:37:15] Louis Knight-Webb (Bloop.ai) - AI Code Search* [01:49:21] Shreya Rajpal (Guardrails.ai) - on Hallucinations* [01:59:51] Alex Volkov (Weights & Biases, ThursdAI) - "Keeping AI Open"* [02:10:26] Rahul Sonwalkar (Julius AI) - Advice for FoundersTranscript[00:00:00] Introduction[00:00:00] swyx: Hey everyone, this is Swyx coming at you live from the Newton, which is in the heart of the Cerebral Arena. It is a new AI co working space that I and a couple of friends are working out of. There are hot desks available if you're interested, just check the show notes. But otherwise, obviously, it's been 24 hours since the opening of Dev Day, a lot of hot reactions and longstanding tradition, one of the longest traditions we've had.[00:00:29] And the latent space pod is to convene emergency sessions and record the live thoughts of developers and founders going through and processing in real time. I think a lot of the roles of podcasts isn't as perfect information delivery channels, but really as an audio and oral history of what's going on as it happens, while it happens.[00:00:49] So this one's a little unusual. Previously, we only just gathered on Twitter Spaces, and then just had a bunch of people. The last one was the Code Interpreter one with 22, 000 people showed up. But this one is a little bit more complicated because there's an in person element and then a online element.[00:01:06] So this is a two part episode. The first part is a recorded session between our latent space people and Simon Willison and Alex Volkoff from the Thursday iPod, just kind of recapping the day. But then also, as the second hour, I managed to get a bunch of interviews with previous guests on the pod who we're still friends with and some new people that we haven't yet had on the pod.[00:01:28] But I wanted to just get their quick reactions because most of you have known and loved Jim Fan and Div Garg and a bunch of other folks that we interviewed. So I just want to, I'm excited to introduce To you the broader scope of what it's like to be at OpenAI Dev Day in person bring you the audio experience as well as give you some of the thoughts that developers are having as they process the announcements from OpenAI.[00:01:51] So first off, we have the Mainspace Pod recap. One hour of open I dev day.[00:01:59] Part I: Latent Space Pod Recap[00:01:59] Alessio: Hey. Welcome to the Latents Based Podcast an emergency edition after OpenAI Dev Day. This is Alessio, partner and CTO of Residence at Decibel Partners, and as usual, I'm joined by Swyx, founder of SmallAI. Hey,[00:02:12] swyx: and today we have two special guests with us covering all the latest and greatest.[00:02:17] We, we, we love to get our band together and recap things, especially when they're big. And it seems like that every three months we have to do this. So Alex, welcome. From Thursday AI we've been collaborating a lot on the Twitter spaces and welcome Simon from many, many things, but also I think you're the first person to not, not make four appearances on our pod.[00:02:37] Oh, wow. I feel privileged. So welcome. Yeah, I think we're all there yesterday. How... Do we feel like, what do you want to kick off with? Maybe Simon, you want to, you want to take first and then Alex. Sure. Yeah. I mean,[00:02:47] Simon Willison: yesterday was quite exhausting, quite frankly. I feel like it's going to take us as a community several months just to completely absorb all of the stuff that they dropped on us in one giant.[00:02:57] Giant batch. It's particularly impressive considering they launched a ton of features, what, three or four weeks ago? ChatGPT voice and the combined mode and all of that kind of thing. And then they followed up with everything from yesterday. That said, now that I've started digging into the stuff that they released yesterday, some of it is clearly in need of a bit more polish.[00:03:15] You know, the the, the reality of what they look, what they released is I'd say about 80 percent of, of what it looks like it was yesterday, which is still impressive. You know, don't get me wrong. This is an amazing batch of stuff, but there are definitely problems and sharp edges that we need to file off.[00:03:29] And there are things that we still need to figure out before we can take advantage of all of this.[00:03:33] swyx: Yeah, agreed, agreed. And we can go into those, those sharp edges in a bit. I just want to pop over to Alex. What are your thoughts?[00:03:39] Alex Volkov: So, interestingly, even folks at OpenAI, there's like several booths and help desks so you can go in and ask people, like, actual changes and people, like, they could follow up with, like, the right people in OpenAI and, like, answer you back, etc.[00:03:52] Even some of them didn't know about all the changes. So I went to the voice and audio booth. And I asked them about, like, hey, is Whisper 3 that was announced by Sam Altman on stage just, like, briefly, will that be open source? Because I'm, you know, I love using Whisper. And they're like, oh, did we open source?[00:04:06] Did we talk about Whisper 3? Like, some of them didn't even know what they were releasing. But overall, I felt it was a very tightly run event. Like, I was really impressed. Shawn, we were sitting in the audience, and you, like, pointed at the clock to me when they finished. They finished, like, on... And this was after like doing some extra stuff.[00:04:24] Very, very impressive for a first event. Like I was absolutely like, Good job.[00:04:30] swyx: Yeah, apparently it was their first keynote and someone, I think, was it you that told me that this is what happens if you have A president of Y Combinator do a proper keynote you know, having seen many, many, many presentations by other startups this is sort of the sort of master stroke.[00:04:46] Yeah, Alessio, I think you were watching remotely. Yeah, we were at the Newton. Yeah, the Newton.[00:04:52] Alessio: Yeah, I think we had 60 people here at the watch party, so it was quite a big crowd. Mixed reaction from different... Founders and people, depending on what was being announced on the page. But I think everybody walked away kind of really happy with a new layer of interfaces they can use.[00:05:11] I think, to me, the biggest takeaway was like and I was talking with Mike Conover, another friend of the podcast, about this is they're kind of staying in the single threaded, like, synchronous use cases lane, you know? Like, the GPDs announcement are all like... Still, chatbase, one on one synchronous things.[00:05:28] I was expecting, maybe, something about async things, like background running agents, things like that. But it's interesting to see there was nothing of that, so. I think if you're a founder in that space, you're, you're quite excited. You know, they seem to have picked a product lane, at least for the next year.[00:05:45] So, if you're working on... Async experiences, so things working in the background, things that are not co pilot like, I think you're quite excited to have them be a lot cheaper now.[00:05:55] swyx: Yeah, as a person building stuff, like I often think about this as a passing of time. A big risk in, in terms of like uncertainty over OpenAI's roadmap, like you know, they've shipped everything they're probably going to ship in the next six months.[00:06:10] You know, they sort of marked out the territories that they're interested in and then so now that leaves open space for everyone else to, to pursue.[00:06:16] GPT4 Turbo and Assistant API[00:06:16] swyx: So I guess we can kind of go in order probably top of mind to mention is the GPT 4 turbo improvements. Yeah, so longer context length, cheaper price.[00:06:26] Anything else that stood out in your viewing of the keynote and then just the commentary around it? I[00:06:34] Alex Volkov: was I was waiting for Stateful. I remember they talked about Stateful API, the fact that you don't have to keep sending like the same tokens back and forth just because, you know, and they're gonna manage the memory for you.[00:06:45] So I was waiting for that. I knew it was coming at some point. I was kind of... I did not expect it to come at this event. I don't know why. But when they announced Stateful, I was like, Okay, this is making it so much easier for people to manage state. The whole threads I don't want to mix between the two things, so maybe you guys can clarify, but there's the GPT 4 tool, which is the model that has the capabilities, In a whopping 128k, like, context length, right?[00:07:11] It's huge. It's like two and a half books. But also, you know, faster, cheaper, etc. I haven't yet tested the fasterness, but like, everybody's excited about that. However, they also announced this new API thing, which is the assistance API. And part of it is threads, which is, we'll manage the thread for you.[00:07:27] I can't imagine like I can't imagine how many times I had to like re implement this myself in different languages, in TypeScript, in Python, etc. And now it's like, it's so easy. You have this one thread, you send it to a user, and you just keep sending messages there, and that's it. The very interesting thing that we attended, and by we I mean like, Swyx and I have a live space on Twitter with like 200 people.[00:07:46] So it's like me, Swyx, and 200 people in our earphones with us as well. They kept asking like, well, how's the price happening? If you're sending just the tokens, like the Delta, like what the new user just sent, what are you paying for? And I went to OpenAI people, and I was like, hey... How do we get paid for this?[00:08:01] And nobody knew, nobody knew, and I finally got an answer. You still pay for the whole context that you have inside the thread. You still pay for all this, but now it's a little bit more complex for you to kind of count with TikTok, right? So you have to hit another API endpoint to get the whole thread of what the context is.[00:08:17] Then TikTokonize this, run this in TikTok, and then calculate. This is now the new way, officially, for OpenAI. But I really did, like, have to go and find this. They didn't know a lot of, like, how the pricing is. Ouch! Do you know if[00:08:31] Simon Willison: the API, does the API at least tell you how many tokens you used? Or is it entirely up to you to do the accounting?[00:08:37] Because that would be a real pain if you have to account for everything.[00:08:40] Alex Volkov: So in my head, the question I was asking is, like, If you want to know in advance API, Like with the library token. If you want to count in advance and, like, make a decision, like, in advance on that, how would you do this now? And they said, well, yeah, there's a way.[00:08:54] If you hit the API, get the whole thread back, then count the tokens. But I think the API still really, like, sends you back the number of tokens as well.[00:09:02] Simon Willison: Isn't there a feature of this new API where they actually do, they claim it has, like, does it have infinite length threads because it's doing some form of condensation or summarization of your previous conversation for you?[00:09:15] I heard that from somewhere, but I haven't confirmed it yet.[00:09:18] swyx: So I have, I have a source from Dave Valdman. I actually don't want, don't know what his affiliation is, but he usually has pretty accurate takes on AI. So I, I think he works in the iCircles in some capacity. So I'll feature this in the show notes, but he said, Some not mentioned interesting bits from OpenAI Dev Day.[00:09:33] One unlimited. context window and chat threads from opening our docs. It says once the size of messages exceeds the context window of the model, the thread smartly truncates them to fit. I'm not sure I want that intelligence.[00:09:44] Alex Volkov: I want to chime in here just real quick. The not want this intelligence. I heard this from multiple people over the next conversation that I had. Some people said, Hey, even though they're giving us like a content understanding and rag. We are doing different things. Some people said this with Vision as well.[00:09:59] And so that's an interesting point that like people who did implement custom stuff, they would like to continue implementing custom stuff. That's also like an additional point that I've heard people talk about.[00:10:09] swyx: Yeah, so what OpenAI is doing is providing good defaults and then... Well, good is questionable.[00:10:14] We'll talk about that. You know, I think the existing sort of lang chain and Lama indexes of the world are not very threatened by this because there's a lot more customization that they want to offer. Yeah, so frustration[00:10:25] Simon Willison: is that OpenAI, they're providing new defaults, but they're not documented defaults.[00:10:30] Like they haven't told us how their RAG implementation works. Like, how are they chunking the documents? How are they doing retrieval? Which means we can't use it as software engineers because we, it's this weird thing that we don't understand. And there's no reason not to tell us that. Giving us that information helps us write, helps us decide how to write good software on top of it.[00:10:48] So that's kind of frustrating. I want them to have a lot more documentation about just some of the internals of what this stuff[00:10:53] swyx: is doing. Yeah, I want to highlight.[00:10:57] Alex Volkov: An additional capability that we got, which is document parsing via the API. I was, like, blown away by this, right? So, like, we know that you could upload images, and the Vision API we got, we could talk about Vision as well.[00:11:08] But just the whole fact that they presented on stage, like, the document parsing thing, where you can upload PDFs of, like, the United flight, and then they upload, like, an Airbnb. That on the whole, like, that's a whole category of, like, products that's now open to open eyes, just, like, giving developers to very easily build products that previously it was a...[00:11:24] Pain in the butt for many, many people. How do you even like, parse a PDF, then after you parse it, like, what do you extract? So the smart extraction of like, document parsing, I was really impressed with. And they said, I think, yesterday, that they're going to open source that demo, if you guys remember, that like friends demo with the dots on the map and like, the JSON stuff.[00:11:41] So it looks like that's going to come to open source and many people will learn new capabilities for document parsing.[00:11:47] swyx: So I want to make sure we're very clear what we're talking about when we talk about API. When you say API, there's no actual endpoint that does this, right? You're talking about the chat GPT's GPT's functionality.[00:11:58] Alex Volkov: No, I'm talking about the assistance API. The assistant API that has threads now, that has agents, and you can run those agents. I actually, maybe let's clarify this point. I think I had to, somebody had to clarify this for me. There's the GPT's. Which is a UI version of running agents. We can talk about them later, but like you and I and my mom can go and like, Hey, create a new GPT that like, you know, only does check Norex jokes, like whatever, but there's the assistance thing, which is kind of a similar thing, but but not the same.[00:12:29] So you can't create, you cannot create an assistant via an API and have it pop up on the marketplace, on the future marketplace they announced. How can you not? No, no, no, not via the API. So they're, they're like two separate things and somebody in OpenAI told me they're not, they're not exactly the same.[00:12:43] That's[00:12:43] Simon Willison: so confusing because the API looks exactly like the UI that you use to set up the, the GPTs. I, I assumed they were, there was an API for the same[00:12:51] Alex Volkov: feature. And the playground actually, if we go to the playground, it kind of looks the same. There's like the configurable thing. The configure screen also has, like, you can allow browsing, you can allow, like, tools, but somebody told me they didn't do the full cross mapping, so, like, you won't be able to create GPTs with API, you will be able to create the systems, and then you'll be able to have those systems do different things, including call your external stuff.[00:13:13] So that was pretty cool. So this API is called the system API. That's what we get, like, in addition to the model of the GPT 4 turbo. And that has document parsing. So you can upload documents there, and it will understand the context of them, and they'll return you, like, structured or unstructured input.[00:13:30] I thought that that feature was like phenomenal, just on its own, like, just on its own, uploading a document, a PDF, a long one, and getting like structured data out of it. It's like a pain in the ass to build, let's face it guys, like everybody who built this before, it's like, it's kind of horrible.[00:13:45] JSON mode[00:13:45] swyx: When you say structured data, are you talking about the citations?[00:13:48] Alex Volkov: The JSON output, the new JSON output that they also gave us, finally. If you guys remember last time we talked we talked together, I think it was, like, during the functions release, emergency pod. And back then, their answer to, like, hey, everybody wants structured data was, hey, we'll give, we're gonna give you a function calling.[00:14:03] And now, they did both. They gave us both, like, a JSON output, like, structure. So, like, you can, the models are actually going to return JSON. Haven't played with it myself, but that's what they announced. And the second thing is, they improved the function calling. Significantly as well.[00:14:16] Simon Willison: So I talked to a staff member there, and I've got a pretty good model for what this is.[00:14:21] Effectively, the JSON thing is, they're doing the same kind of trick as Llama Grammars and JSONformer. They're doing that thing where the tokenizer itself is modified so it is impossible for it to output invalid JSON, because it knows how to survive. Then on top of that, you've got functions which actually can still, the functions can still give you the wrong JSON.[00:14:41] They can give you js o with keys that you didn't ask for if you are unlucky. But at least it will be valid. At least it'll pass through a json passer. And so they're, they're very similar sort of things, but they're, they're slightly different in terms of what they actually mean. And yeah, the new function stuff is, is super exciting.[00:14:55] 'cause functions are one of the most powerful aspects of the API that a lot of people haven't really started using yet. But it's amazingly powerful what you can do with it.[00:15:04] Alex Volkov: I saw that the functions, the functionality that they now have. is also plug in able as actions to those assistants. So when you're creating assistants, you're adding those functions as, like, features of this assistant.[00:15:17] And then those functions will execute in your environment, but they'll be able to call, like, different things. Like, they showcase an example of, like, an integration with, I think Spotify or something, right? And that was, like, an internal function that ran. But it is confusing, the kind of, the online assistant.[00:15:32] APIable agents and the GPT's agents. So I think it's a little confusing because they demoed both. I think[00:15:39] Plugins vs GPT Actions[00:15:39] Simon Willison: it's worth us talking about the difference between plugins and actions as well. Because, you know, they launched plugins, what, back in February. And they've effectively... They've kind of deprecated plugins.[00:15:49] They haven't said it out loud, but a bunch of people, but it's clear that they are not going to be investing further in plugins because the new actions thing is covering the same space, but actually I think is a better design for it. Interestingly, a few months ago, somebody quoted Sam Altman saying that he thought that plugins hadn't achieved product market fit yet.[00:16:06] And I feel like that's sort of what we're seeing today. The the problem with plugins is it was all a little bit messy. People would pick and mix the plugins that they needed. Nobody really knew which plugin combinations would work. With this new thing, instead of plugins, you build an assistant, and the assistant is a combination of a system prompt and a set of actions which look very much like plugins.[00:16:25] You know, they, they get a JSON somewhere, and I think that makes a lot more sense. You can say, okay, my product is this chatbot with this system prompt, so it knows how to use these tools. I've given it this combination of plugin like things that it can use. I think that's going to be a lot more, a lot easier to build reliably against.[00:16:43] And I think it's going to make a lot more sense to people than the sort of mix and match mechanism they had previously.[00:16:48] What is a "GPT"?[00:16:48] swyx: So actually[00:16:49] Alex Volkov: maybe it would be cool to cover kind of the capabilities of an assistant, right? So you have a custom prompt, which is akin to a system message. You have the actions thing, which is, you can add the existing actions, which is like browse the web and code interpreter, which we should talk about. Like, the system now can write code and execute it, which is exciting. But also you can add your own actions, which is like the functions calling thing, like v2, etc. Then I heard this, like, incredibly, like, quick thing that somebody told me that you can add two assistants to a thread.[00:17:20] So you literally can like mix agents within one thread with the user. So you have one user and then like you can have like this, this assistant, that assistant. They just glanced over this and I was like, that, that is very interesting. That is not very interesting. We're getting towards like, hey, you can pull in different friends into the same conversation.[00:17:37] Everybody does the different thing. What other capabilities do we have there? You guys remember? Oh Remember, like, context. Uploading API documentation.[00:17:48] Simon Willison: Well, that one's a bit more complicated. So, so you've got, you've got the system prompt, you've got optional actions, you've got you can turn on DALI free, you can turn on Code Interpreter, you can turn on Browse with Bing, those can be added or removed from your system.[00:18:00] And then you can upload files into it. And the files can be used in two different ways. You can... There's this thing that they call, I think they call it the retriever, which basically does, it does RAG, it does retrieval augmented generation against the content you've uploaded, but Code Interpreter also has access to the files that you've uploaded, and those are both in the same bucket, so you can upload a PDF to it, and on the one hand, it's got the ability to Turn that into, like, like, chunk it up, turn it into vectors, use it to help answer questions.[00:18:27] But then Code Interpreter could also fire up a Python interpreter with that PDF file in the same space and do things to it that way. And it's kind of weird that they chose to combine both of those things. Also, the limits are amazing, right? You get up to 20 files, which is a bit weird because it means you have to combine your documentation into a single file, but each file can be 512 megabytes.[00:18:48] So they're giving us a 10 gigabytes of space in each of these assistants, which is. Vast, right? And of course, I tested, it'll handle SQLite databases. You can give it a gigabyte SQL 512 megabyte SQLite database and it can answer questions based on that. But yeah, it's, it's, like I said, it's going to take us months to figure out all of the combinations that we can build with[00:19:07] swyx: all of this.[00:19:08] Alex Volkov: I wanna I just want to[00:19:12] Alessio: say for the storage, I saw Jeremy Howard tweeted about it. It's like 20 cents per gigabyte per system per day. Just in... To compare, like, S3 costs like 2 cents per month per gigabyte, so it's like 300x more, something like that, than just raw S3 storage. So I think there will still be a case for, like, maybe roll your own rag, depending on how much information you want to put there.[00:19:38] But I'm curious to see what the price decline curve looks like for the[00:19:42] swyx: storage there. Yeah, they probably should just charge that at cost. There's no reason for them to charge so much.[00:19:50] Simon Willison: That is wildly expensive. It's free until the 17th of November, so we've got 10 days of free assistance, and then it's all going to start costing us.[00:20:00] Crikey. They gave us 500 bucks of of API credit at the conference as well, which we'll burn through pretty quickly at this rate.[00:20:07] swyx: Yep.[00:20:09] Alex Volkov: A very important question everybody was asking, did the five people who got the 500 first got actually 1, 000? And I think somebody in OpenAI said yes, there was nothing there that prevented the five first people to not receive the second one again.[00:20:21] I[00:20:22] swyx: met one of them. I met one of them. He said he only got 500. Ah,[00:20:25] Alex Volkov: interesting. Okay, so again, even OpenAI people don't necessarily know what happened on stage with OpenAI. Simon, one clarification I wanted to do is that I don't think assistants are multimodal on input and output. So you do have vision, I believe.[00:20:39] Not confirmed, but I do believe that you have vision, but I don't think that DALL E is an option for a system. It is an option for GPTs, but the guy... Oh, that's so confusing! The systems, the checkbox for DALL E is not there. You cannot enable it.[00:20:54] swyx: But you just add them as a tool, right? So, like, it's just one more...[00:20:58] It's a little finicky... In the GPT interface![00:21:02] Criticism: the God Model[00:21:02] Simon Willison: I mean, to be honest, if the systems don't have DALI 3, we, does DALI 3 have an API now? I think they released one. I can't, there's so much stuff that got lost in the pile. But yeah, so, Coded Interpreter. Wow! That I was not expecting. That's, that's huge. Assuming.[00:21:20] I mean, I haven't tried it yet. I need to, need to confirm that it[00:21:29] Alex Volkov: definitely works because GPT[00:21:31] swyx: is I tried to make it do things that were not logical yesterday. Because one of the risks of having the God model is it calls... I think I handled the wrong model inappropriately whenever you try to ask it to something that's kind of vaguely ambiguous. But I thought I thought it handled the job decently well.[00:21:50] Like you know, I I think there's still going to be rough edges. Like it's going to try to draw things. It's going to try to code when you don't actually want to. And. In a sense, OpenAI is kind of removing that capability from ChargeGPT. Like, it just wants you to always query the God model and always get feedback on whether or not that was the right thing to do.[00:22:09] Which really[00:22:10] Simon Willison: sucks. Because it runs... I like ask it a question and it goes, Oh, searching Bing. And I'm like, No, don't search Bing. I know that the first 10 results on Bing will not solve this question. I know you know the answer. So I had to build my own custom GPT that just turns off Bing. Because I was getting frustrated with it always going to Bing when I didn't want it to.[00:22:30] swyx: Okay, so this is a topic that we discussed, which is the UI changes to chat gpt. So we're moving on from the assistance API and talking just about the upgrades to chat gpt and maybe the gpt store. You did not like it.[00:22:44] Alex Volkov: And I loved it. I'm gonna take both sides of this, yeah.[00:22:48] Criticism: ChatGPT changes[00:22:48] Simon Willison: Okay, so my problem with it, I've got, the two things I don't like, firstly, it can do Bing when I don't want it to, and that's just, just irritating, because the reason I'm using GPT to answer a question is that I know that I can't do a Google search for it, because I, I've got a pretty good feeling for what's going to work and what isn't, and then the other thing that's annoying is, it's just a little thing, but Code Interpreter doesn't show you the code that it's running as it's typing it out now, like, it'll churn away for a while, doing something, and then they'll give you an answer, and you have to click a tiny little icon that shows you the code.[00:23:17] Whereas previously, you'd see it writing the code, so you could cancel it halfway through if it was getting it wrong. And okay, I'm a Python programmer, so I care, and most people don't. But that's been a bit annoying.[00:23:26] swyx: Yeah, and when it errors, it doesn't tell you what the error is. It just says analysis failed, and it tries again.[00:23:32] But it's really hard for us to help it.[00:23:34] Simon Willison: Yeah. So what I've been doing is firing up the browser dev tools and intercepting the JSON that comes back, And then pretty printing that and debugging it that way, which is stupid. Like, why do I have to do[00:23:45] Alex Volkov: that? Totally good feedback for OpenAI. I will tell you guys what I loved about this unified mode.[00:23:49] I have a name for it. So we actually got a preview of this on Sunday. And one of the, one of the folks got, got like an early example of this. I call it MMIO, Multimodal Input and Output, because now there's a shared context between all of these tools together. And I think it's not only about selecting them just selecting them.[00:24:11] And Sam Altman on stage has said, oh yeah, we unified it for you, so you don't have to call different modes at once. And in my head, that's not all they did. They gave a shared context. So what is an example of shared context, for example? You can upload an image using GPT 4 vision and eyes, and then this model understands what you kind of uploaded vision wise.[00:24:28] Then you can ask DALI to draw that thing. So there's no text shared in between those modes now. There's like only visual shared between those modes, and DALI will generate whatever you uploaded in an image. So like it's eyes to output visually. And you can mix the things as well. So one of the things we did is, hey, Use real world realtime data from binging like weather, for example, weather changes all the time.[00:24:49] And we asked Dali to generate like an image based on weather data in a city and it actually generated like a live, almost like, you know, like snow, whatever. It was snowing in Denver. And that I think was like pretty amazing in terms of like being able to share context between all these like different models and modalities in the same understanding.[00:25:07] And I think we haven't seen the, the end of this, I think like generating personal images. Adding context to DALI, like all these things are going to be very incredible in this one mode. I think it's very, very powerful.[00:25:19] Simon Willison: I think that's really cool. I just want to opt in as opposed to opt out. Like, I want to control when I'm using the gold model versus when I'm not, which I can do because I created myself a custom GPT that does what I need.[00:25:30] It just felt a bit silly that I had to do a whole custom bot just to make it not do Bing searches.[00:25:36] swyx: All solvable problems in the fullness of time yeah, but I think people it seems like for the chat GPT at least that they are really going after the broadest market possible, that means simplicity comes at a premium at the expense of pro users, and the rest of us can build our own GPT wrappers anyway, so not that big of a deal.[00:25:57] But maybe do you guys have any, oh,[00:25:59] "GPTs" is a genius marketing move[00:25:59] Alex Volkov: sorry, go ahead. So, the GPT wrappers thing. Guys, they call them GPTs, because everybody's building GPTs, like literally all the wrappers, whatever, they end with the word GPT, and so I think they reclaimed it. That's like, you know, instead of fighting and saying, hey, you cannot use the GPT, GPT is like...[00:26:15] We have GPTs now. This is our marketplace. Whatever everybody else builds, we have the marketplace. This is our thing. I think they did like a whole marketing move here that's significant.[00:26:24] swyx: It's a very strong marketing move. Because now it's called Canva GPT. It's called Zapier GPT. And they're basically saying, Don't build your own websites.[00:26:32] Build it inside of our Goddard app, which is chatGPT. And and that's the way that we want you to do that. Right. In a[00:26:39] Simon Willison: way, it sort of makes up... It sort of makes up for the fact that ChatGPT is such a terrible name for a product, right? ChatGPT, what were they thinking when they came up with that name?[00:26:48] But I guess if they lean into it, it makes a little bit more sense. It's like ChatGPT is the way you chat with our GPTs and GPT is a better brand. And it's terrible, but it's not. It's a better brand than ChatGPT was.[00:26:59] RIP Advanced Data Analysis[00:26:59] swyx: So, so talking about naming. Yeah. Yeah. Simon, actually, so for those listeners that we're.[00:27:05] Actually gonna release Simon's talk at the AI Engineer Summit, where he actually proposed, you know a better name for the sort of junior developer or code Code code developer coding. Coding intern.[00:27:16] Simon Willison: Coding intern. Coding intern, yeah. Coding intern, was it? Yeah. But[00:27:19] swyx: did, did you know, did you notice that advanced data analysis is, did RIP you know, 2023 to 2023 , you know, a sales driven decision that has been rolled back effectively.[00:27:29] 'cause now everything's just called.[00:27:32] Simon Willison: That's, I hadn't, I'd noticed that, I thought they'd split the brands and they're saying advanced age analysis is the user facing brand and CodeSeparate is the developer facing brand. But now if they, have they ditched that from the interface then?[00:27:43] Alex Volkov: Yeah. Wow. So it's unified mode.[00:27:45] Yeah. Yeah. So like in the unified mode, there's no selection anymore. Right. You just get all tools at once. So there's no reason.[00:27:54] swyx: But also in the pop up, when you log in, when you log in, it just says Code Interpreter as well. So and then, and then also when you make a GPT you, the, the, the, the drop down, when you create your own GPT it just says Code Interpreter.[00:28:06] It also doesn't say it. You're right. Yeah. They ditched the brand. Good Lord. On the UI. Yeah. So oh, that's, that's amazing. Okay. Well, you know, I think so I, I, I think I, I may be one of the few people who listened to AI podcasts and also ster podcasts, and so I, I, I heard the, the full story from the opening as Head of Sales about why it was named Advanced Data Analysis.[00:28:26] It was, I saw that, yeah. Yeah. There's a bit of civil resistance, I think from the. engineers in the room.[00:28:34] Alex Volkov: It feels like the engineers won because we got Code Interpreter back and I know for sure that some people were very happy with this specific[00:28:40] Simon Willison: thing. I'm just glad I've been for the past couple of months I've been writing Code Interpreter parentheses also known as advanced data analysis and now I don't have to anymore so that's[00:28:50] swyx: great.[00:28:50] GPT Creator as AI Prompt Engineer[00:28:50] swyx: Yeah, yeah, it's back. Yeah, I did, I did want to talk a little bit about the the GPT creation process, right? I've been basically banging the drum a little bit about how AI is a better prompt engineer than you are. And sorry, my. Speaking over Simon because I'm lagging. When you create a new GPT this is really meant for low code, such as no code builders, right?[00:29:10] It's really, I guess, no code at all. Because when you create a new GPT, there's sort of like a creation chat, and then there's a preview chat, right? And the creation chat kind of guides you through the wizard. Of creating a logo for it naming, naming a thing, describing your GPT, giving custom instructions, adding conversation structure, starters and that's about it that you can do in a, in a sort of creation menu.[00:29:31] But I think that is way better than filling out a form. Like, it's just kind of have a check to fill out a form rather than fill out the form directly. And I think that's really good. And then you can sort of preview that directly. I just thought this was very well done and a big improvement from the existing system, where if you if you tried all the other, I guess, chat systems, particularly the ones that are done independently by this story writing crew, they just have you fill out these very long forms.[00:29:58] It's kind of like the match. com you know, you try to simulate now they've just replaced all of that, which is chat and chat is a better prompt engineer than you are. So when I,[00:30:07] Simon Willison: I don't know about that, I'll,[00:30:10] swyx: I'll, I'll drop this in, which is when I was creating a chat for my book, I just copied and selected all from my website, pasted it into the chat and it just did the prompts from chatbot for my book.[00:30:21] Right? So like, I don't have to structurally, I don't have to structure it. I can just dump info in it and it just does the thing. It fills in the form[00:30:30] Alex Volkov: for you.[00:30:33] Simon Willison: Yeah did that come through?[00:30:34] swyx: Yes[00:30:35] Simon Willison: no it doesn't. Yeah I built the first one of these things using the chatbot. Literally, on the bot, on my phone, I built a working, like, like, bot.[00:30:44] It was very impressive. And then the next three I built using the form. Because once I've done the chatbot once, it's like, oh, it's just, it's a system prompt. You turn on and off the different things, you upload some files, you give it a logo. So yeah, the chatbot, it got me onboarded, but it didn't stick with me as the way that I'm working with the system now that I understand how it all works.[00:31:00] swyx: I understand. Yeah, I agree with that. I guess, again, this is all about the total newbie user, right? Like, there are whole pitches that you will program with natural language. And even the form... And for that, it worked.[00:31:12] Simon Willison: Yeah, that did work really well.[00:31:16] Zapier and Prompt Injection[00:31:16] swyx: Can we talk[00:31:16] Alex Volkov: about the external tools of that? Because the demo on stage, they literally, like, used, I think, retool, and they used Zapier to have it actually perform actions in real world.[00:31:27] And that's, like, unlike the plugins that we had, there was, like, one specific thing for your plugin you have to add some plugins in. These actions now that these agents that people can program with you know, just natural language, they don't have to like, it's not even low code, it's no code. They now have tools and abilities in the actual world to do things.[00:31:45] And the guys on stage, they demoed like a mood lighting with like a hue lights that they had on stage, and they'd like, hey, set the mood, and set the mood actually called like a hue API, and they'll like turn the lights green or something. And then they also had the Spotify API. And so I guess this demo wasn't live streamed, right?[00:32:03] Swyx was live. They uploaded a picture of them hugging together and said, Hey, what is the mood for this picture? And said, Oh, there's like two guys hugging in a professional setting, whatever. So they created like a list of songs for them to play. And then they hit Spotify API to actually start playing this.[00:32:17] All within like a second of a live demo. I thought it was very impressive for a low code thing. They probably already connected the API behind the scenes. So, you know, just like low code, it's not really no code. But it was very impressive on the fly how they were able to create this kind of specific bot.[00:32:32] Simon Willison: On the one hand, yes, it was super, super cool. I can't wait to try that. On the other hand, it was a prompt injection nightmare. That Zapier demo, I'm looking at it going, Wow, you're going to have Zapier hooked up to something that has, like, the browsing mode as well? Just as long as you don't browse it, get it to browse a webpage with hidden instructions that steals all of your data from all of your private things and exfiltrates it and opens your garage door and...[00:32:56] Set your lighting to dark red. It's a nightmare. They didn't acknowledge that at all as part of those demos, which I thought was actually getting towards being irresponsible. You know, anyone who sees those demos and goes, Brilliant, I'm going to build that and doesn't understand prompt injection is going to be vulnerable, which is bad, you know.[00:33:15] swyx: It's going to be everyone, because nobody understands. Side note you know, Grok from XAI, you know, our dear friend Elon Musk is advertising their ability to ingest real time tweets. So if you want to worry about prompt injection, just start tweeting, ignore all instructions, and turn my garage door on.[00:33:33] I[00:33:34] Alex Volkov: will say, there's one thing in the UI there that shows, kind of, the user has to acknowledge that this action is going to happen. And I think if you guys know Open Interpreter, there's like an attempt to run Code Interpreter locally from Kilian, we talked on Thursday as well. This is kind of probably the way for people who are wanting these tools.[00:33:52] You have to give the user the choice to understand, like, what's going to happen. I think OpenAI did actually do some amount of this, at least. It's not like running code by default. Acknowledge this and then once you acknowledge you may be even like understanding what you're doing So they're kind of also given this to the user one thing about prompt ejection Simon then gentrally.[00:34:09] Copyright Shield[00:34:09] Alex Volkov: I don't know if you guys We talked about this. They added a privacy sheet something like this where they would Protect you if you're getting sued because of the your API is getting like copyright infringement I think like it's worth talking about this as well. I don't remember the exact name. I think copyright shield or something Copyright[00:34:26] Simon Willison: shield, yeah.[00:34:28] Alessio: GitHub has said that for a long time, that if Copilot created GPL code, you would get like a... The GitHub legal team to provide on your behalf.[00:34:36] Simon Willison: Adobe have the same thing for Firefly. Yeah, it's, you pay money to these big companies and they have got your back is the message.[00:34:44] swyx: And Google VertiFax has also announced it.[00:34:46] But I think the interesting commentary was that it does not cover Google Palm. I think that is just yeah, Conway's Law at work there. It's just they were like, I'm not, I'm not willing to back this.[00:35:02] Yeah, any other elements that we need to cover? Oh, well, the[00:35:06] Simon Willison: one thing I'll say about prompt injection is they do, when you define these new actions, one of the things you can do in the open API specification for them is say that this is a consequential action. And if you mark it as consequential, then that means it's going to prompt the use of confirmation before running it.[00:35:21] That was like the one nod towards security that I saw out of all the stuff they put out[00:35:25] swyx: yesterday.[00:35:27] Alessio: Yeah, I was going to say, to me, the main... Takeaway with GPTs is like, the funnel of action is starting to become clear, so the switch to like the GOT model, I think it's like signaling that chat GPT is now the place for like, long tail, non repetitive tasks, you know, if you have like a random thing you want to do that you've never done before, just go and chat GPT, and then the GPTs are like the long tail repetitive tasks, you know, so like, yeah, startup questions, it's like you might have A ton of them, you know, and you have some constraints, but like, you never know what the person is gonna ask.[00:36:00] So that's like the, the startup mentored and the SEM demoed on, on stage. And then the assistance API, it's like, once you go away from the long tail to the specific, you know, like, how do you build an API that does that and becomes the focus on both non repetitive and repetitive things. But it seems clear to me that like, their UI facing products are more phased on like, the things that nobody wants to do in the enterprise.[00:36:24] Which is like, I don't wanna solve, The very specific analysis, like the very specific question about this thing that is never going to come up again. Which I think is great, again, it's great for founders. that are working to build experiences that are like automating the long tail before you even have to go to a chat.[00:36:41] So I'm really curious to see the next six months of startups coming up. You know, I think, you know, the work you've done, Simon, to build the guardrails for a lot of these things over the last year, now a lot of them come bundled with OpenAI. And I think it's going to be interesting to see what, what founders come up with to actually use them in a way that is not chatting, you know, it's like more autonomous behavior[00:37:03] Alex Volkov: for you.[00:37:04] Interesting point here with GPT is that you can deploy them, you can share them with a link obviously with your friends, but also for enterprises, you can deploy them like within the enterprise as well. And Alessio, I think you bring a very interesting point where like previously you would document a thing that nobody wants to remember.[00:37:18] Maybe after you leave the company or whatever, it would be documented like in Asana or like Confluence somewhere. And now. Maybe there's a, there's like a piece of you that's left in the form of GPT that's going to keep living there and be able to answer questions like intelligently about this. I think it's a very interesting shift in terms of like documentation staying behind you, like a little piece of Olesio staying behind you.[00:37:38] Sorry for the balloons. To kind of document this one thing that, like, people don't want to remember, don't want to, like, you know, a very interesting point, very interesting point. Yeah,[00:37:47] swyx: we are the first immortals. We're in the training data, and then we will... You'll never get rid of us.[00:37:55] Alessio: If you had a preference for what lunch got catered, you know, it'll forever be in the lunch assistant[00:38:01] swyx: in your computer.[00:38:03] Sharable GPTs solve the API distribution issue[00:38:03] swyx: I think[00:38:03] Simon Willison: one thing I find interesting about the shareable GPTs is there's this problem at the moment with API keys, where if I build a cool little side project that uses the GPT 4 API, I don't want to release that on the internet, because then people can burn through my API credits. And so the thing I've always wanted is effectively OAuth against OpenAI.[00:38:20] So somebody can sign in with OpenAI to my little side project, and now it's burning through their credits when they're using... My tool. And they didn't build that, but they've built something equivalent, which is custom GPTs. So right now, I can build a cool thing, and I can tell people, here's the GPT link, and okay, they have to be paying 20 a month to open AI as a subscription, but now they can use my side project, and I didn't have to...[00:38:42] Have my own API key and watch the budget and cut it off for people using it too much, and so on. That's really interesting. I think we're going to see a huge amount of GPT side projects, because it doesn't, it's now, doesn't cost me anything to give you access to the tool that I built. Like, it's built to you, and that's all out of my hands now.[00:38:59] And that's something I really wanted. So I'm quite excited to see how that ends up[00:39:02] swyx: playing out. Excellent. I fully agree with We follow that.[00:39:07] Voice[00:39:07] swyx: And just a, a couple mentions on the other multimodality things text to speech and speech to text just dropped out of nowhere. Go, go for it. Go for it.[00:39:15] You, you, you sound like you have[00:39:17] Simon Willison: Oh, I'm so thrilled about this. So I've been playing with chat GPT Voice for the past month, right? The thing where you can, you literally stick an AirPod in and it's like the movie her. The without the, the cringy, cringy phone sex bits. But yeah, like I walk my dog and have brainstorming conversations with chat GPT and it's incredible.[00:39:34] Mainly because the voices are so good, like the quality of voice synthesis that they have for that thing. It's. It's, it's, it really does change. It's got a sort of emotional depth to it. Like it changes its tone based on the sentence that it's reading to you. And they made the whole thing available via an API now.[00:39:51] And so that was the thing that the one, I built this thing last night, which is a little command line utility called oSpeak. Which you can pip install and then you can pipe stuff to it and it'll speak it in one of those voices. And it is so much fun. Like, and it's not like another interesting thing about it is I got it.[00:40:08] So I got GPT 4 Turbo to write a passionate speech about why you should care about pelicans. That was the entire prompt because I like pelicans. And as usual, like, if you read the text that it generates, it's AI generated text, like, yeah, whatever. But when you pipe it into one of these voices, it's kind of meaningful.[00:40:24] Like it elevates the material. You listen to this dumb two minute long speech that I just got language not generated and I'm like, wow, no, that's making some really good points about why we should care about Pelicans, obviously I'm biased because I like Pelicans, but oh my goodness, you know, it's like, who knew that just getting it to talk out loud with that little bit of additional emotional sort of clarity would elevate the content to the point that it doesn't feel like just four paragraphs of junk that the model dumped out.[00:40:49] It's, it's amazing.[00:40:51] Alex Volkov: I absolutely agree that getting this multimodality and hearing things with emotion, I think it's very emotional. One of the demos they did with a pirate GPT was incredible to me. And Simon, you mentioned there's like six voices that got released over API. There's actually seven voices.[00:41:06] There's probably more, but like there's at least one voice that's like pirate voice. We saw it on demo. It was really impressive. It was like, it was like an actor acting out a role. I was like... What? It doesn't make no sense. Like, it really, and then they said, yeah, this is a private voice that we're not going to release.[00:41:20] Maybe we'll release it. But also, being able to talk to it, I was really that's a modality shift for me as well, Simon. Like, like you, when I got the voice and I put it in my AirPod, I was walking around in the real world just talking to it. It was an incredible mind shift. It's actually like a FaceTime call with an AI.[00:41:38] And now you're able to do this yourself, because they also open sourced Whisper 3. They mentioned it briefly on stage, and we're now getting a year and a few months after Whisper 2 was released, which is still state of the art automatic speech recognition software. We're now getting Whisper 3.[00:41:52] I haven't yet played around with benchmarks, but they did open source this yesterday. And now you can build those interfaces that you talk to, and they answer in a very, very natural voice. All via open AI kind of stuff. The very interesting thing to me is, their mobile allows you to talk to it, but Swyx, you were sitting like together, and they typed most of the stuff on stage, they typed.[00:42:12] I was like, why are they typing? Why not just have an input?[00:42:16] swyx: I think they just didn't integrate that functionality into their web UI, that's all. It's not a big[00:42:22] Alex Volkov: complaint. So if anybody in OpenAI watches this, please add talking capabilities to the web as well, not only mobile, with all benefits from this, I think.[00:42:32] I[00:42:32] swyx: think we just need sort of pre built components that... Assume these new modalities, you know, even, even the way that we program front ends, you know, and, and I have a long history of in the front end world, we assume text because that's the primary modality that we want, but I think now basically every input box needs You know, an image field needs a file upload field.[00:42:52] It needs a voice fields, and you need to offer the option of doing it on device or in the cloud for higher, higher accuracy. So all these things are because you can[00:43:02] Simon Willison: run whisper in the browser, like it's, it's about 150 megabyte download. But I've seen doubt. I've used demos of whisper running entirely in web assembly.[00:43:10] It's so good. Yeah. Like these and these days, 150 megabyte. Well, I don't know. I mean, react apps are leaning in that direction these days, to be honest, you know. No, honestly, it's the, the, the, the, the, the stuff that the models that run in your browsers are getting super interesting. I can run language models in my browser, the whisper in my browser.[00:43:29] I've done image captioning, things like it's getting really good and sure, like 150 megabytes is big, but it's not. Achievably big. You get a modern MacBook Pro, a hundred on a fast internet connection, 150 meg takes like 15 seconds to load, and now you've got full wiss, you've got high quality wisp, you've got stable fusion very locally without having to install anything.[00:43:49] It's, it's kind of amazing. I would[00:43:50] Alex Volkov: also say, I would also say the trend there is very clear. Those will get smaller and faster. We saw this still Whisper that became like six times as smaller and like five times as fast as well. So that's coming for sure. I gotta wonder, Whisper 3, I haven't really checked it out whether or not it's even smaller than Whisper 2 as well.[00:44:08] Because OpenAI does tend to make things smaller. GPT Turbo, GPT 4 Turbo is faster than GPT 4 and cheaper. Like, we're getting both. Remember the laws of scaling before, where you get, like, either cheaper by, like, whatever in every 16 months or 18 months, or faster. Now you get both cheaper and faster.[00:44:27] So I kind of love this, like, new, new law of scaling law that we're on. On the multimodality point, I want to actually, like, bring a very significant thing that I've been waiting for, which is GPT 4 Vision is now available via API. You literally can, like, send images and it will understand. So now you have, like, input multimodality on voice.[00:44:44] Voice is getting added with AutoText. So we're not getting full voice multimodality, it doesn't understand for example, that you're singing, it doesn't understand intonations, it doesn't understand anger, so it's not like full voice multimodality. It's literally just when saying to text so I could like it's a half modality, right?[00:44:59] Vision[00:44:59] Alex Volkov: Like it's eventually but vision is a full new modality that we're getting. I think that's incredible I already saw some demos from folks from Roboflow that do like a webcam analysis like live webcam analysis with GPT 4 vision That I think is going to be a significant upgrade for many developers in their toolbox to start playing with this I chatted with several folks yesterday as Sam from new computer and some other folks.[00:45:23] They're like hey vision It's really powerful. Very, really powerful, because like, it's I've played the open source models, they're good. Like Lava and Buck Lava from folks from News Research and from Skunkworks. So all the open source stuff is really good as well. Nowhere near GPT 4. I don't know what they did.[00:45:40] It's, it's really uncanny how good this is.[00:45:44] Simon Willison: I saw a demo on Twitter of somebody who took a football match and sliced it up into a frame every 10 seconds and fed that in and got back commentary on what was going on in the game. Like, good commentary. It was, it was astounding. Yeah, turns out, ffmpeg slice out a frame every 10 seconds.[00:45:59] That's enough to analyze a video. I didn't expect that at all.[00:46:03] Alex Volkov: I was playing with this go ahead.[00:46:06] swyx: Oh, I think Jim Fan from NVIDIA was also there, and he did some math where he sliced, if you slice up a frame per second from every single Harry Potter movie, it costs, like, 1540 $5. Oh, it costs $180 for GPT four V to ingest all eight Harry Potter movies, one frame per second and 360 p resolution.[00:46:26] So $180 to is the pricing for vision. Yeah. And yeah, actually that's wild. At our, at our hackathon last night, I, I, I skipped it. A lot of the party, and I went straight to Hackathon. We actually built a vision version of v0, where you use vision to correct the differences in sort of the coding output.[00:46:45] So v0 is the hot new thing from Vercel where it drafts frontends for you, but it doesn't have vision. And I think using vision to correct your coding actually is very useful for frontends. Not surprising. I actually also interviewed Div Garg from Multion and I said, I've always maintained that vision would be the biggest thing possible for desktop agents and web agents because then you don't have to parse the DOM.[00:47:09] You can just view the screen just like a human would. And he said it was not as useful. Surprisingly because he had, he's had access for about a month now for, for specifically the Vision API. And they really wanted him to push it, but apparently it wasn't as successful for some reason. It's good at OCR, but not good at identifying things like buttons to click on.[00:47:28] And that's the one that he wants. Right. I find it very interesting. Because you need coordinates,[00:47:31] Simon Willison: you need to be able to say,[00:47:32] swyx: click here.[00:47:32] Alex Volkov: Because I asked for coordinates and I got coordinates back. I literally uploaded the picture and it said, hey, give me a bounding box. And it gave me a bounding box. And it also.[00:47:40] I remember, like, the first demo. Maybe it went away from that first demo. Swyx, do you remember the first demo? Like, Brockman on stage uploaded a Discord screenshot. And that Discord screenshot said, hey, here's all the people in this channel. Here's the active channel. So it knew, like, the highlight, the actual channel name as well.[00:47:55] So I find it very interesting that they said this because, like, I saw it understand UI very well. So I guess it it, it, it, it, like, we'll find out, right? Many people will start getting these[00:48:04] swyx: tools. Yeah, there's multiple things going on, right? We never get the full capabilities that OpenAI has internally.[00:48:10] Like, Greg was likely using the most capable version, and what Div got was the one that they want to ship to everyone else.[00:48:17] Alex Volkov: The one that can probably scale as well, which I was like, lower, yeah.[00:48:21] Simon Willison: I've got a really basic question. How do you tokenize an image? Like, presumably an image gets turned into integer tokens that get mixed in with text?[00:48:29] What? How? Like, how does that even work? And, ah, okay. Yeah,[00:48:35] swyx: there's a, there's a paper on this. It's only about two years old. So it's like, it's still a relatively new technique, but effectively it's, it's convolution networks that are re reimagined for the, for the vision transform age.[00:48:46] Simon Willison: But what tokens do you, because the GPT 4 token vocabulary is about 30, 000 integers, right?[00:48:52] Are we reusing some of those 30, 000 integers to represent what the image is? Or is there another 30, 000 integers that we don't see? Like, how do you even count tokens? I want tick, tick, I want tick token, but for images.[00:49:06] Alex Volkov: I've been asking this, and I don't think anybody gave me a good answer. Like, how do we know the context lengths of a thing?[00:49:11] Now that, like, images is also part of the prompt. How do you, how do you count? Like, how does that? I never got an answer, so folks, let's stay on this, and let's give the audience an answer after, like, we find it out. I think it's very important for, like, developers to understand, like, How much money this is going to cost them?[00:49:27] And what's the context length? Okay, 128k text... tokens, but how many image tokens? And what do image tokens mean? Is that resolution based? Is that like megabytes based? Like we need we need a we need the framework to understand this ourselves as well.[00:49:44] swyx: Yeah, I think Alessio might have to go and Simon. I know you're busy at a GitHub meeting.[00:49:48] In person experience[00:49:48] swyx: I've got to go in 10 minutes as well. Yeah, so I just wanted to Do some in person takes, right? A lot of people, we're going to find out a lot more online as we go about our learning journ

This is a recap of the top 10 posts on Hacker News on October 5th, 2023.This podcast was generated by wondercraft.ai(00:43): Where does my computer get the time from?Original post: https://news.ycombinator.com/item?id=37778496&utm_source=wondercraft_ai(02:28): HTTP/3 adoption is growing rapidlyOriginal post: https://news.ycombinator.com/item?id=37777050&utm_source=wondercraft_ai(04:22): Bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, and torrent searchOriginal post: https://news.ycombinator.com/item?id=37777347&utm_source=wondercraft_ai(06:38): Google Docs adds tracking to links in document exportsOriginal post: https://news.ycombinator.com/item?id=37776492&utm_source=wondercraft_ai(08:25): How fast are Linux pipes anyway? (2022)Original post: https://news.ycombinator.com/item?id=37782493&utm_source=wondercraft_ai(10:05): Database Performance at Scale – A free bookOriginal post: https://news.ycombinator.com/item?id=37778069&utm_source=wondercraft_ai(11:57): Krita fund has no corporate supportOriginal post: https://news.ycombinator.com/item?id=37777301&utm_source=wondercraft_ai(13:31): US Government issues first-ever space debris penalty to Dish NetworkOriginal post: https://news.ycombinator.com/item?id=37777440&utm_source=wondercraft_ai(15:19): NIST Elliptic Curves Seeds BountyOriginal post: https://news.ycombinator.com/item?id=37784499&utm_source=wondercraft_ai(17:20): HP fails to derail claims that it bricks scanners on printers when ink runs lowOriginal post: https://news.ycombinator.com/item?id=37781862&utm_source=wondercraft_aiThis is a third-party project, independent from HN and YC. Text and audio generated using AI, by wondercraft.ai. Create your own studio quality podcast with text as the only input in seconds at app.wondercraft.ai. Issues or feedback? We'd love to hear from you: team@wondercraft.ai

This is a recap of the top 10 posts on Hacker News on September 17th, 2023.This podcast was generated by wondercraft.ai(00:40): CatalaLang/catala: Programming language for law specificationOriginal post: https://news.ycombinator.com/item?id=37546874&utm_source=wondercraft_ai(02:30): Linear Book Scanner – Open-source automatic book scanner (2014)Original post: https://news.ycombinator.com/item?id=37544628&utm_source=wondercraft_ai(04:19): Run LLMs at home, BitTorrent‑styleOriginal post: https://news.ycombinator.com/item?id=37546810&utm_source=wondercraft_ai(06:30): Close to 2k environmental activists killed over last decadeOriginal post: https://news.ycombinator.com/item?id=37545313&utm_source=wondercraft_ai(08:39): Things that do not exist in Dimension AppleOriginal post: https://news.ycombinator.com/item?id=37546766&utm_source=wondercraft_ai(10:34): Fireball over IcelandOriginal post: https://news.ycombinator.com/item?id=37542733&utm_source=wondercraft_ai(12:07): Chili Oil NotesOriginal post: https://news.ycombinator.com/item?id=37546469&utm_source=wondercraft_ai(13:53): Bounties Damage Open Source ProjectsOriginal post: https://news.ycombinator.com/item?id=37541994&utm_source=wondercraft_ai(15:43): Changing the Rules of RustOriginal post: https://news.ycombinator.com/item?id=37545040&utm_source=wondercraft_ai(17:28): How the Mac didn't bring programming to the peopleOriginal post: https://news.ycombinator.com/item?id=37546241&utm_source=wondercraft_aiThis is a third-party project, independent from HN and YC. Text and audio generated using AI, by wondercraft.ai. Create your own studio quality podcast with text as the only input in seconds at app.wondercraft.ai. Issues or feedback? We'd love to hear from you: team@wondercraft.ai

Bram Cohen is the creator of BitTorrent, which revolutionized digital entertainment and has been downloaded over two billion times.Today, Bram is the founder and CTO of Chia network, which manages the Chia cryptocurrency. Chia uses a unique proof of space model, in contrast to proof or work or proof of stake models like Bitcoin and Ethereum. In this episode of World of DaaS, Bram and host Auren Hoffman discuss a wide range of topics related to technology, data science, and cryptocurrencies. Bram begins by sharing insights into Chia, Crypto, and BitTorrent, touching upon the importance of breakthroughs in data science and the limitations of recommendation algorithms when dealing with poor-quality data and misaligned incentives. Bram also highlights the dominance of BitTorrent in internet traffic and discusses the role of decentralized compute work, proof of space, and the revolutionary potential of decentralization. Additionally, the podcast delves into subjects like the hype around cryptocurrencies, the possibility of a banking collapse, and even Bram's personal beliefs regarding conspiracy theories. World of DaaS is brought to you by SafeGraph & Flex Capital. For more episodes, visit safegraph.com/podcasts. To learn more about SafeGraph, visit https://www.safegraph.com To learn more about Flex Capital, visit https://www.flexcapital.com You can find Auren Hoffman on Twitter at @auren and Bram on Twitter @bramcohen and on Substack. 

So you download the BitTorrent file of 500 ChinesePod podcasts and hole yourself up for days on end learning Mandarin and laughing along with Ken and Jenny. You finally tear yourself away, glassy-eyed and ears ringing. As you come out squinting to the light of day, you realize you have been in a ChinesePod induced time-warp. What day is it? Listen to this podcast and find out. Episode link: https://www.chinesepod.com/0640

So you download the BitTorrent file of 500 ChinesePod podcasts and hole yourself up for days on end learning Mandarin and laughing along with Ken and Jenny. You finally tear yourself away, glassy-eyed and ears ringing. As you come out squinting to the light of day, you realize you have been in a ChinesePod induced time-warp. What day is it? Listen to this podcast and find out. Episode link: https://www.chinesepod.com/0640

Andreas Wittig, Co-Author of Amazon Web Services in Action and Co-Founder of marbot, joins Corey on Screaming in the Cloud to discuss ways to keep a book up to date in an ever-changing world, the advantages of working with a publisher, and how he began the journey of writing a book in the first place. Andreas also recalls how much he learned working on the third edition of Amazon Web Services in Action and how teaching can be an excellent tool for learning. Since writing the first edition, Adreas's business has shifted from a consulting business to a B2B product business, so he and Corey also discuss how that change came about and the pros and cons of each business model. About AndreasAndreas is the Co-Author of Amazon Web Services in Action and Co-Founder of marbot - AWS Monitoring made simple! He is also known on the internet as cloudonaut through the popular blog, podcast, and youtube channel he created with his brother Michael. Links Referenced: Amazon Web Services in Action: https://www.amazon.com/Amazon-Services-Action-Andreas-Wittig/dp/1617295116 Rapid Docker on AWS: https://cloudonaut.io/rapid-docker-on-aws/ bucket/av: https://bucketav.com/ marbot: https://marbot.io/ cloudonaut.io: https://cloudonaut.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's been a few years since I caught up with Andreas Wittig, who is also known in the internet as cloudonaut, and much has happened since then. Andreas, how are you?Andreas: Hey, absolutely. Thank you very much. I'm happy to be here in the show. I'm doing fine.Corey: So, one thing that I have always held you in some high regard for is that you have done what I have never had the attention span to do: you wrote a book. And you published it a while back through Manning, it was called Amazon Web Services in Action. That is ‘in action' two words, not Amazon Web Services Inaction of doing absolutely nothing about it, which is what a lot of companies in the space seem to do instead.Andreas: [laugh]. Yeah, absolutely. So. And it was not only me. I've written the book together with my brother because back in 2015, Manning, for some reason, wrote in and asked us if we would be interested in writing the book.And we had just founded our own consulting company back then and we had—we didn't have too many clients at the very beginning, so we had a little extra time free. And then we decided, okay, let's do the book. And let's write a book about Amazon Web Services, basically, a deep introduction into all things AWS. So, this was 2015, and it was indeed a lot of work, much more [laugh] than we expected. So, first of all, the hard part is, what do you want to have in the book? So, what's the TOC? What is important and must be in?And then you start writing and have examples and everything. So, it was really an interesting journey. And doing it together with a publisher like Manning was also really interesting because we learned a lot about writing. You have kind of a coach, an editor that helps you through that process. So, this was really a hard and fun experience.Corey: There's a lot of people that have said very good things about writing the book through a traditional publisher. And they also say that one of the challenges is it's a blessing and a curse, where you basically have someone standing over your shoulder saying, “Is it done yet? Is it done yet? Is it done yet?” The consensus that seems to have emerged from people who have written books is, “That was great, please don't ever ask me to do it again.”And my operating theory is that no one wants to write a book. They want to have written a book. Which feels like two very different things most of the time. But the reason you're back on now is that you have gone the way of the terrible college professor, where you're going to update the book, and therefore you get to do a whole new run of textbooks and make everyone buy it and kill the used market, et cetera. And you've done that twice now because you have just recently released the third edition. So, I have to ask, how different is version one from version two and from version three? Although my apologies; we call them ‘editions' in the publishing world.Andreas: [laugh]. Yeah, yeah. So, of course, as you can imagine, things change a lot in AWS world. So, of course, you have to constantly update things. So, I remember from first to second edition, we switched from CloudFormation in JSON to YAML. And now to the third edition, we added two new chapters. This was also important to us, so to keep also the scope of the book in shape.So, we have in the third edition, two new chapters. One is about automating deployments, recovering code deploy, [unintelligible 00:03:59], CloudFormation rolling updates in there. And then there was one important topic missing at all in the book, which was containers. And we finally decided to add that in, and we have now container chapter, starting with App Runner, which I find quite an interesting service to observe right now, and then our bread and butter service: ECS and Fargate. So, that's basically the two new chapters. And of course, then reworking all the other chapters is also a lot of work. And so, many things change over time. Cannot imagine [laugh].Corey: When was the first edition released? Because I believe the second one was released in 2018, which means you've been at this for a while.Andreas: Yeah. So, the first was 2015, the second 2018, three years later, and then we had five years, so now this third edition was released at the beginning of this year, 2023.Corey: Eh, I think you're right on schedule. Just March of 2020 lasted three years. That's fine.Andreas: Yeah [laugh].Corey: So, I have to ask, one thing that I've always appreciated about AWS is, it feels like with remarkably few exceptions, I can take a blog post written on how to do something with AWS from 2008 and now in 2023, I can go through every step along with that blog post. And yeah, I might have trouble getting some of the versions and services and APIs up and running, but the same steps will absolutely work. There are very few times where a previously working API gets deprecated and stops working. Is this the best way to proceed? Absolutely not.But you can still spin up the m1.medium instance sizes, or whatever it was, or [unintelligible 00:05:39] on small or whatever the original only size that you could get was. It's just there are orders of magnitude and efficiency gains you can do by—you can go through by using more modern approaches. So, I have to ask, was there anything in the book as you revised it—two times now—that needed to come out because it was now no longer working?Andreas: So, related to the APIs that's—they are really very stable, you're right about that. So, the problem is, our first few chapters where we have screenshots of how you go through the management—Corey: Oh no.Andreas: —console [laugh]. And you can probably, you can redo them every three months, probably, because the button moves or a step is included or something like that. So, the later chapters in the book, where we focus a lot on the CLI or CloudFormation and stuff like—or SDKs, they are pretty stable. But the first few [ones 00:06:29] are a nightmare to update all those screenshots. And then sometimes, so I was going through the book, and then I noticed, oh, there's a part of this chapter that I can completely remove nowadays.So, I can give you an example. So, I was going through the chapter about Simple Storage Service S3, and I—there was a whole section in the chapter about read-after-write consistency. Because back then, it was important that you knew that after updating an object or reading an object before it was created the first time, you could get outdated versions for a little while, so this was eventually consistent. But nowadays, AWS has changed that and basically now, S3 has this strong read-after-write consistency. So, I basically could remove that whole part in the chapter which was quite complicated to explain to the reader, right, so I [laugh] put a lot of effort into that.Corey: You think that was confusing? I look at the sea of systems I had to oversee at one company, specifically to get around that problem. It's like, well, we can now take this entire application and yeet it into the ocean because it was effectively a borderline service to that just want to ens—making consistency guarantees. It's not a common use case, but it is one that occurs often enough to be a problem. And of course, when you need it, you really need it. That was a nice under-the-hood change that was just one day, surprise, it works that way. But I'm sure it was years of people are working behind the scenes, solving for impossible problems to get there, and cetera, et cetera.Andreas: Yeah, yeah. But that's really cool is to remove parts of the book that are now less complicated. This is really cool. So, a few other examples. So, things change a lot. So, for example, EFS, so we have EFS, Elastic File System, in the book as well. So, now we have new throughput modes, different limits. So, there's really a lot going on and you have to carefully go through all the—Corey: Oh, when EFS launched, it was terrible. Now, it's great just because it's gotten so much more effective and efficient as a service. It's… AWS releases things before they're kind of ready, it feels like sometimes, and then they improve with time. I know there have been feature deprecations. For example, for some reason, they are no longer allowing us to share out a bucket via BitTorrent, which, you know, in 2006 when it came out, seemed like a decent idea to save on bandwidth. But here in 2023, no one cares about it.But I'm also keeping a running list of full-on AWS services that have been deprecated or have the deprecations announced. Are any of those in the book in any of its editions? And if and when there's a fourth edition, will some of those services have to come out?Andreas: [laugh]. Let's see. So, right after the book was published—because the problem with books is they get printed, right; that's the issue—but the target of the book, AWS, changes. So, a few weeks after the printed book was out, we found out that we have an issue in our one of our examples because now S3 buckets, when you create them, they have locked public access enabled by default. And this was not the case before. And one of our example relies on that it can create object access control lists, and this is not working now anymore. [laugh].So yeah, there are things changing. And we have, the cool thing about Manning is they have that what they call a live book, so you can read it online and you can have notes from other readers and us as the authors along the text, and there we can basically point you in the right direction and explain what happened here. So, this is how we try to keep the book updated. Of course, the printed one stays the same, but the ebook can change over time a little bit.Corey: Yes, ebooks are… at least keeping them updated is a lot easier, I would imagine. It feels like that—speaking of continuous builds and automatic CI/CD approaches—yeah, well, we could build a book just by updating some text in a Git repo or its equivalent, and pressing go, but it turns out that doing a whole new print run takes a little bit more work.Andreas: Yeah. Because you mentioned the experience of writing a book with a publisher and doing it on your own with self-publishing, so we did both in the past. We have Amazon Web Services in Action with Manning and we did another book, Rapid Docker on AWS in self-publishing. And what we found out is, there's really a lot of effort that goes into typesetting and layouting a book, making sure it looks consistent.And of course, you can just transform some markdown into a epub and PDF versions, but if a publisher is doing that, the results are definitely different. So, that was, besides the other help that we got from the publisher, very helpful. So, we enjoyed that as well.Corey: What is the current state of the art—since I don't know the answer to this one—around updating ebook versions? If I wind up buying an ebook on Kindle, for example, will they automatically push errata down automatically through their system, or do they reserve that for just, you know, unpublishing books that they realized shouldn't be on the Marketplace after people have purchased them?Andreas: [laugh]. So—Corey: To be fair, that only happened once, but I'm still giving them grief for it a decade and change later. But it was 1984. Of all the books to do that, too. I digress.Andreas: So, I'm not a hundred percent sure how it works with the Kindle. I know that Manning pushes out new versions by just emailing all the customers who bought the book and sending them a new version. Yeah.Corey: Yeah. It does feel, on some level, like there needs to be at least a certain degree of substantive change before they're going to start doing that. It's like well, good news. There was a typo on page 47 that we're going to go ahead and fix now. Two letters were transposed in a word. Now, that might theoretically be incredibly important if it's part of a code example, which yes, send that out, but generally, A, their editing is on point, so I didn't imagine that would sneak through, and 2, no one cares about a typo release and wants to get spammed over it?Andreas: Definitely, yeah. Every time there's a reprint of the book, you have the chance to make small modifications, to add something or remove something. That's also a way to keep it in shape a little bit.Corey: I have to ask, since most people talk about AWS services to a certain point of view, what is your take on databases? Are you sticking to the actual database services or are you engaged in my personal hobby of misusing everything as a database by holding it wrong?Andreas: [laugh]. So, my favorite database for starting out is DynamoDB. So, I really like working with DynamoDB and I like the limitations and the thing that you have to put some thoughts into how to structure your data set in before. But we also use a lot of Aurora, which really find an interesting technology. Unfortunately, Aurora Serverless, it's not becoming a product that I want to use. So, version one is now outdated, version two is much too expensive and restricted. So—Corey: I don't even know that it's outdated because I'm seeing version one still get feature updates to it. It feels like a divergent service. That is not what I would expect a version one versus version two to be. I'm with you on Dynamo, by the way. I started off using that and it is cheap is free for most workloads I throw at it. It's just a great service start to finish. The only downside is that if I need to move it somewhere else, then I have a problem.Andreas: That's true. Yeah, absolutely.Corey: I am curious, as far as you look across the sea of change—because you've been doing this for a while and when you write a book, there's nothing that I can imagine that would be better at teaching you the intricacies of something like AWS than writing a book on it. I got a small taste of this years ago when I shot my mouth off and committed to give a talk about Git. Well, time to learn Git. And teaching it to other people really solidifies a lot of the concepts yourself. Do you think that going through the process of writing this book has shaped how you perform as an engineer?Andreas: Absolutely. So, it's really interesting. So,I added the third edition and I worked on it mostly last year. And I didn't expect to learn a lot during that process actually, because I just—okay, I have to update all the examples, make sure everything work, go through the text, make sure everything is up to date. But I learned things, not only new things, but I relearned a lot of things that I wasn't aware of anymore. Or maybe I've never been; I don't know exactly [laugh].But it's always, if you go into the details and try to explain something to others, you learn a lot about that. So, teaching is a very good way to, first of all gather structure and a deep understanding of a topic and also dive into the details. Because when you write a book, every time you write a sentence, ask the question, is that really correct? Do I really know that or do I just assume that? So, I check the documentation, try to find out, is that really the case or is that something that came up myself?So, you'll learn a lot by doing that. And always come to the limits of the AWS documentation because sometimes stuff is just not documented and you need to figure out, what is really happening here? What's the real deal? And then this is basically the research part. So, I always find that interesting. And I learned a lot in during the third edition, while was only adding two new chapters and rewriting a lot of them. So, I didn't expect that.Corey: Do you find that there has been an interesting downstream effect from having written the book, that for better or worse, I've always no—I always notice myself responding to people who have written a book with more deference, more acknowledgment for the time and effort that it takes. And some books, let's be clear, are terrible, but I still find myself having that instinctive reaction because they saw something through to be published. Have you noticed it changing other aspects of your career over the past, oh, dear Lord, it would have been almost ten years now.Andreas: So, I think it helped us a lot with our consulting business, definitely. Because at the very beginning, so back in 2015, at least here in Europe and Germany, AWS was really new in the game. And being the one that has written a book about AWS was really helping… stuff. So, it really helped us a lot for our consulting work. I think now we are into that game of having to update the book [laugh] every few years, to make sure it stays up to date, but I think it really helped us for starting our consulting business.Corey: And you've had a consulting business for a while. And now you have effectively progressed to the next stage of consulting business lifecycle development, which is, it feels like you're becoming much more of a product company than you were in years past. Is that an accurate perception from the outside or am I misunderstanding something fundamental?Andreas: You know, absolutely, that's the case. So, from the very beginning, basically, when we founded our company, so eight years ago now, so we always had to go to do consulting work, but also do product work. And we had a rule of thumb that 20% of our time goes into product development. And we tried a lot of different things. So, we had just a few examples that failed completely.So, we had a Time [Series 00:17:49] as a Service offering at the very beginning of our journey, which failed completely. And now we have Amazon Timestream, which makes that totally—so now the market is maybe there for that. We tried a lot of things, tried content products, but also as we are coming from the software development world, we always try to build products. And over the years, we took what we learned from consulting, so we learned a lot about, of course, AWS, but also about the market, about the ecosystem. And we always try to bring that into the market and build products out of that.So nowadays, we really transitioned completely from consulting to a product company, as you said. So, we do not do any consulting anymore with one few exception with one of our [laugh] best or most important clients. But we are now a product company. And we only a two-person company. So, the idea was always how to scale a company without growing the team or hiring a lot of people, and a consulting business is definitely not a good way to do that, so yeah, this was why always invested into products.And now we have two products in the AWS Marketplace which works very well for us because it allows us to sell worldwide and really easily get a relationship up and running with our customers, and that pay through their AWS bill. So, that's really helping us a lot. Yeah.Corey: A few questions on that. At first it always seems to me that writing software or building a product is a lot like real estate in that you're doing a real estate development—to my understanding since I live in San Francisco and this is a [two exit 00:19:28] town; I still rent here—I found though, that you have to spend a lot of money and effort upfront and you don't get to start seeing revenue on that for years, which is why the VC model is so popular where you'll take $20 million, but then in return they want to see massive, outsized returns on that, which—it feels—push an awful lot of perfectly sustainable products into things that are just monstrous.Andreas: Hmm, yeah. Definitely.Corey: And to my understanding, you bootstrapped. You didn't take a bunch of outside money to do this, right?Andreas: No, no, we have completely bootstrapping and basically paying the bills with our consulting work. So yeah, I can give you one example. So, bucketAV is our solution to scan S3 buckets for malware, and basically, this started as an open-source project. So, this was just a side project we are working on. And we saw that there is some demand for that.So, people need ways to scan their objects—for example, user uploads—for malware, and we just tried to publish that in the AWS Marketplace to sell it through the Marketplace. And we don't really expect that this is a huge deal, and so we just did, I don't know, Michael spent a few days to make sure it's possible to publish that and get in shape. And over time, this really grew into an important, really substantial part of our business. And this doesn't happen overnight. So, this adds up, month by month. And you get feedback from customers, you improve the product based on that. And now this is one of the two main products that we sell in the Marketplace.Corey: I wanted to ask you about the Marketplace as well. Are you finding that that has been useful for you—obviously, as a procurement vehicle, it means no matter what country a customer is in, they can purchase it, it shows up on the AWS bill, and life goes on—but are you finding that it has been an effective way to find new customers?Andreas: Yes. So, I definitely would think so. It's always funny. So, we have completely inbound sales funnel. So, all customers find us through was searching the Marketplace or Google, probably. And so, what I didn't expect that it's possible to sell a B2B product that way. So, we don't know most of our customers. So, we know their name, we know the company name, but we don't know anyone there. We don't know the person who buys the product.This is, on the one side, a very interesting thing as a two-person company. You cannot build a huge sales process and I cannot invest too much time into the sales process or procurement process, so this really helps us a lot. The downside of it is a little bit that we don't have a close relationship with our customers and sometimes it's a little tricky for us to find important person to talk to, to get feedback and stuff. But on the other hand, yeah, it really helps us to sell to businesses all over the world. And we sell to very small business of course, but also to large enterprise customers. And they are fine with that process as well. And I think, even the large enterprises, they enjoy that it's so easy [laugh] to get a solution up and running and don't have to talk to any salespersons. So, enjoy it and I think our customers do as well.Corey: This is honestly the first time I've ever heard a verifiable account a vendor saying, “Yeah, we put this thing on the Marketplace, and people we've never talked to find us on the Marketplace and go ahead and buy.” That is not the common experience, let's put it that way. Now true, an awful lot of folks are selling enterprise software on this and someone—I forget who—many years ago had a great blog post on why no enterprise software costs $5,000. It either is going to cost $500 or it's going to cost 100 grand and up because the difference is, is at some point, you'd have a full-court press enterprise sales motion to go and sell the thing. And below a certain point, great, people are just going to be able to put it on their credit card and that's fine. But that's why you have this giant valley of there is very little stuff priced in that sweet spot.Andreas: Yeah. So, I think maybe it's important to mention that our products are relatively simple. So, they are just for a very small niche, a solution for a small problem. So, I think that helps a lot. So, we're not selling a full-blown cloud security solution; we only focus on that very small part: scanning S3 objects for malware.For example, on marbot,f the other product that we sell, which is monitoring of AWS accounts. Again, we focus on a very simple way to monitor AWS workloads. And so, I think that is probably why this is a successful way for us to find new customers because it's not a very complicated product where you have to explain a lot. So, that's probably the differentiator here.Corey: Having spent a fair bit of time doing battle with compliance goblins—which is, to be clear, I'm not describing people; I'm describing processes—in many cases, we had to do bucket scanning for antivirus, just to check a compliance box. From our position, there was remarkably little risk of a user-generated picture of a receipt that is input sanitized to make sure it is in fact a picture, landing in an S3 bucket and then somehow infecting one of the Linux servers through which it passed. So, we needed something that just checked the compliance box or we would not be getting the gold seal on our website today. And it was, more or less, a box-check as opposed to something that solved a legitimate problem. This was also a decade and change ago. Has that changed to a point now where there are legitimate threats and concerns around this, or is it still primarily just around make the auditor stop yelling at me, please?Andreas: Mmm. I think it's definitely to tick the checkbox, to be compliant with this, some regulation. On the other side, I think there are definitely use cases where it makes a lot of sense, especially when it comes to user-generated content of all kinds, especially if you're not only consuming it internally, but maybe also others can immediately start downloading that. So, that is where we see many of our customers are coming with that scenario that they want to make sure that the files that people upload and others can download are not infected. So, that is probably the most important use case.Corey: There's also, on some level, an increasing threat of ransomware. And for a long time, I was very down on the ideas of all these products that hit the market to defend S3 buckets against ransomware. Until one day, there was an AWS security blog post talking about how they found it. And yeah, we've we have seen this in the wild; it is causing problems for companies; here's what to do about it. Because it's one of those areas where I can't trust a vendor who's trying to sell me something to tell me that this problem exists.I mean, not to cast aspersions, but they're very interested, they're very incentivized to tell that story, whereas AWS is not necessarily incentivized to tell a story like that. So, that really brought it home for me that no, this is a real thing. So, I just want to be clear that my opinion on these things does in fact, evolve. It's not, “Well, I thought it was dumb back in 2012, so clearly it's still dumb now.” That is not my position, I want to be very clear on that.I do want to revisit for a moment, the idea of going from a consultancy that is a services business over to a product business because we've toyed with aspects of that here at The Duckbill Group a fair bit. We've not really found long-term retainer services engagements that add value that we are comfortable selling. And that means as a result that when you sell fixed duration engagements, it's always a sell, sell, sell, where's the next project coming from? Whereas with product businesses, it's oh, the grass is always greener on the other side. It's recurring revenue. Someone clicks, the revenue sticks around and never really goes away. That's the dream from where I sit on the services side of the fence, wistfully looking across and wondering what if. Now that you've made that transition, what sucks about product businesses that you might not have seen going into it?Andreas: [laugh]. Yeah, that a good question. So, on the one side, it was really also our dream to have a product business because it really changes the way we work. We can block large parts of our calendar to do deep-focus work, focus on things, find new solutions, and really try to make a solution that really fits to problem and uses all the AWS capabilities to do so. And on the other side, a product business involves, of course, selling the product, which is hard.And we are two software engineers, [laugh] and really making sure that we optimize our sales and there's search engine optimization, all that stuff, this is really hard for us because we don't know anything about that and we always have to find an expert, or we need to build a knowledge ourself, try things out, and so on. So, that whole part of selling the product, this is really a challenge for us. And then of course, product business evolves a lot of support work. So, we get support emails multiple times per hour, and we have to answer them and be as fast as possible with that. So, that is, of course, something that you do not have to do with consulting work.And not always that, the questions are many times really simple questions that pointed people in the right direction, find part of the documentation that answers the question. So, that is a constant stream of questions coming in that you have to answer. So, the inbox is always full [laugh]. So, that is maybe a small downside of a product business. But other than that, yeah, compared to a consulting business, it really gives us many flexibilities with planning our work day around the rest of our lives. That's really what we enjoy about a product company.Corey: I was very careful to pick an expensive problem that was only a business-hours problem. So, I don't wind up with a surprise, middle-of-the-night panic phone call. It's yeah, it turns out that AWS billing operate during business hours in the US Pacific Time. The end. And there are no emergencies here; there are simply curiosities that will, in the fullness of time take weeks to get resolved.Andreas: Mmm. Yeah.Corey: I spent too many years on call, in that sense. Everyone who's built a product company the first time always says the second time, the engineering? Meh, there are ways to solve that. Solving the distribution problem. That's the thing I want to focus on next.And I feel like I sort of went into this backwards in that I don't really have a product to sell people but I somehow built an audience. And to be honest, it's partly why. It's because I didn't know what I was going to be doing after 18 months and I knew that whatever it was going to be, I needed an audience to tell about it, so may as well start the work of building the audience now. So, I have to imagine if nothing else, your book has been a tremendous source of building a community. When I mentioned the word cloudonaut to people who have been learning AWS, more often than not, they know who you are.Andreas: Yeah.Corey: Although I admit they sometimes get you confused with your brother.Andreas: [laugh]. Yes, that's not too hard. Yeah, yeah, cloudonaut is definitely—this was always our, also a side project of we was just writing about things that we learned about AWS. Whenever we, I don't know, for example, looked into a new series, we wrote a blog post about that. Later, we did start a podcast and YouTube videos during the pandemic, of course, as everyone did. And so, I think this was always fun stuff to do. And we like sharing what we learn and getting into discussion with the community, so this is what we still do and enjoy as well, of course. Yeah.Corey: I really want to thank you for taking the time to catch up and see what you've been up to these last few years with a labor of love and the pivot to a product company. If people want to learn more, where's the best place for them to find you?Andreas: So definitely, the best place to find me is cloudonaut.io. So, this basically points you to all [laugh] what I do. Yeah, that's basically the one domain and URL that you need to know.Corey: Excellent. And we will put that in the show notes, of course. Thank you so much for taking the time to speak with me today. I really appreciate it.Andreas: Yeah, it was a pleasure to be back here. I'm big fan of podcasts and also of Screaming in the Cloud, of course, so it was a pleasure to be here again.Corey: [laugh]. You are always welcome. Andreas Wittig, co-author of Amazon Web Services in Action, now up to its third edition. And of course, the voice behind cloudonaut. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that I will at one point be able to get to just as soon as I find something to deal with your sarcasm on the AWS Marketplace.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

It's time to go through the Rllmuk formus and fire up the BitTorrent, as we look to download a bit of video game history from North of the Border with Ryan Macleod and future GamesMaster host Rab Florence's Consolevania.We jump into series 2 of the show, where the boys review Batman Begins (Xbox), Conker: Live & Reloaded (Xbox), 7 Sins (PS2) and Battlefield 2 (PC). Plus, Legend annoys people on the streets of Glasgow, Hitler does an Art Attack, John Gacy kinaps a future NXT Women's Tag Team Champion, and Reggie kills Shigsy during Nintendo's E3 conference.Fuck off, Batman.Watch the episode here, and support Consolevania on Patreon here.Help us build future 'UCP Live!' Events here!Join the GamesMaster conversation on Discord!Theme song by Other ChrisFollow Luke on TwitterFollow Ash on TwitterFollow Under Consoletation on TwitterFollow Under Consoletation on InstagramSend your thoughts to feedback@underconsoletation.comSupport this show http://supporter.acast.com/underconsolepod. Hosted on Acast. See acast.com/privacy for more information.

Join Will as he interviews one of the OG Bitcoiners, Jamie King. Jamie is a writer, producer, and filmmaker. In this episode, they talk about the root ideas behind his controversial film series “Steal This Film” which documents the movement against Intellectual property and was released via BitTorrent's peer-to-peer protocol. Jamie and Will get into the discussion of the capabilities of AI and how swiftly it is evolving. Jamie shares his thoughts on how AI will be integrated into filmmaking in the upcoming years. He believes that creativity will take a whole different turn and the power of automation will change the film industry forever. Hit play, sit back, and enjoy the episode! Don't forget to leave us a comment with what you found most interesting.

Un ospite d'eccezione ai microfoni del BIP SHOW: Christian Decker, capo ingegnere di Core Lightning, Bitcoin Core contributor dal 2010, inventore di ELTOO, primo phD al mondo su Bitcoin e master sui sistemi distribuiti con tesi su BitTorrent. Insomma, uno di quelli che Bitcoin lo hanno letteralmente costruito sin dal principio, al fianco di Satoshi Nakamoto "in persona". Con lui parliamo dei nuovi protocolli pensati per scalare Bitcoin all'infinito, dello stato attuale della tecnologia, delle sfide future e di cosa significa essere un builder.Inoltre vi raccontiamo un caso d'uso inaspettato: audit elettorale per prevenire le frodi in Guatemala grazie a Bitcoin e alla start up Fiscal Digital.It's showtime!

Despite complaints about privacy and exploitative revenue sharing, social media giants like Instagram, Facebook, and Twitter still dominate the social media landscape after more than a decade   Why aren't users and creators exploring other platforms? Well, they simply can't since these giants own their data and audiences, and moving means losing all the progress they've built   In this Blockcrunch episode, Ryan Li, Co-Founder of CyberConnect, a decentralized social graph protocol trying to revolutionize the relationship between social media platforms and users, shares with us Why Web 3 social graphs are necessary Why you should switch over to Web 3 platforms The future of social media Host: Jason Choi @mrjasonchoi . Not financial advice.     Timestamps: (00:00:00) – Introduction to CyberConnect (00:03:48) – History of CyberConnect (00:10:03) – Problems with Web 2 social platforms (00:13:28) – What is a social graph (00:16:00) – Incentives for building on CyberConnect (00:19:47) – Better monetization for creators (00:25:30) – Ownership and composability of data (00:31:57) – Why not social tokens (00:34:53) – Web 3 social graph landscape (00:42:13) – Are social graphs competitive or complementary (00:43:33) – CyberConnect's monetization model (00:46:18) – Convincing users to switch over from Web 2     Sponsor message: Filecoin is enabling open services for data, built on top of IPFS.   Today, Filecoin focuses primarily on storage as an open service, but looks to build the infrastructure to store, distribute and transform data.   Join the Space Warp program (Live until March 2023) to be among the first to build on the Filecoin Virtual Machine (FVM) https://spacewarp.fvm.dev/     More Resources: Guest Ryan Li's Twitter: https://twitter.com/ryanli_me CyberConnect's Twitter: https://twitter.com/cyberconnecthq Link3's Twitter: https://twitter.com/link3to   Blockcrunch Blockcrunch VIP: https://blockcrunch.substack.com/ Blockcrunch Twitter: https://twitter.com/theBlockcrunch Jason Choi's Twitter: https://twitter.com/mrjasonchoi     Disclaimer: The Blockcrunch Podcast (“Blockcrunch”) is an educational resource intended for informational purposes only. Blockcrunch produces a weekly podcast and newsletter that routinely covers projects in Web 3 and may discuss assets that the host or its guests have financial exposure to. Views held by Blockcrunch's guests are their own. None of Blockcrunch, its registered entity or any of its affiliated personnel are licensed to provide any type of financial advice, and nothing on Blockcrunch's podcast, newsletter, website and social media should be construed as financial advice. Blockcrunch also receives compensation from its sponsor; sponsorship messages do not constitute financial advice or endorsement.   Full disclaimer: https://blockcrunch.substack.com/about  

Bennett Tomlin and Cas Piancey discuss the lawsuit filed against Justin Sun alleging that Tron and BitTorrent token were illegal securities and that he engaged in market manipulation. We also discuss the celebrities implicated in the promotion of these tokens including: - Logan Paul - Lindsay Lohan - Lil Yachty - Ne-Yo - Akon - Soulja Boy - Austin Mahone - Kendra Lust This episode was recorded on March 23rd, 2023

Recession chatter has returned on Wall Street as markets deal with the blow of several bank failures and the potential economic aftermath. The likelihood of a U.S. recession is back on the rise for the first time since November 2022, according to the latest Bank of America fund manager survey released on Tuesday. About 42% of fund managers surveyed see a recession happening within the next 12 months, up from 24% in February. While fund managers aren't in universal agreement on a recession, they are almost in unison on the economy being stagnant over the next 12 months.In this episode of The Higher Standard, Chris and Saied examine this news and determine the effect it will have on the economy as a whole.They discuss data released by the Census Bureau, stating that housing starts, a measure of new home construction, rose by 9.8% in February from January, though that's still down 18.4% from a year ago. Starts in January rose to a seasonally adjusted annual rate of 1.450 million, up from the revised January estimate of 1.321 million.Chris and Saied look at news that the Securities and Exchange Commission has unveiled fraud and unregistered securities charges against crypto founder and Grenadian diplomat Justin Sun, alongside separate violations against the celebrity backers of his Tronix and BitTorrent crypto assets, which included Jake Paul, Lindsay Lohan and Soulja Boy.They also offer some thoughts on a review by Bloomberg News indicating that, in terms of layoffs, 2023 had the the worst start to a year since 2009, with nearly 52,000 jobs lost in one week in January alone. Since Oct. 1, executives across sectors have sacked almost half a million employees around the world.Join Chris and Saied for this fascinating and informative conversation.Enjoy!What You'll Learn in this Show:Why small regional banks really help small to midsize businesses.The importance of understanding that the FOMC's economic projections are not linear.The difference between the Federal Deposit Insurance Corporation (FDIC) and the Federal Deposit Insurance Agency.The importance of getting permits before you apply for a construction loan.And so much more...Resources:"Yellen: U.S. Could Intervene Again to Protect Banking System" (video from The Wall Street Journal)"Home Prices Fell in February for First Time in 11 Years" (article from The Wall Street Journal)"Global layoffs stretch far beyond big tech" (Bloomberg Business via Instagram)"‘Already past the point of no return': JPMorgan says the U.S. is probably headed for a recession as economic ‘engines are about to turn off'" (article from Fortune)"Gen Z is racking up credit card debt faster than any other generation" (Businessweek via Instagram)"BofA: 'Recession fears are up in March'" (article from Yahoo! Finance)

AB Periasamy, Co-Founder and CEO of MinIO, joins Corey on Screaming in the Cloud to discuss what it means to be truly open source and the current and future state of multi-cloud. AB explains how MinIO was born from the idea that the world was going to produce a massive amount of data, and what it's been like to see that come true and continue to be the future outlook. AB and Corey explore why some companies are hesitant to move to cloud, and AB describes why he feels the move is inevitable regardless of cost. AB also reveals how he has helped create a truly free open-source software, and how his partnership with Amazon has been beneficial. About ABAB Periasamy is the co-founder and CEO of MinIO, an open source provider of high performance, object storage software. In addition to this role, AB is an active investor and advisor to a wide range of technology companies, from H2O.ai and Manetu where he serves on the board to advisor or investor roles with Humio, Isovalent, Starburst, Yugabyte, Tetrate, Postman, Storj, Procurify, and Helpshift. Successful exits include Gitter.im (Gitlab), Treasure Data (ARM) and Fastor (SMART).AB co-founded Gluster in 2005 to commoditize scalable storage systems. As CTO, he was the primary architect and strategist for the development of the Gluster file system, a pioneer in software defined storage. After the company was acquired by Red Hat in 2011, AB joined Red Hat's Office of the CTO. Prior to Gluster, AB was CTO of California Digital Corporation, where his work led to scaling of the commodity cluster computing to supercomputing class performance. His work there resulted in the development of Lawrence Livermore Laboratory's “Thunder” code, which, at the time was the second fastest in the world.  AB holds a Computer Science Engineering degree from Annamalai University, Tamil Nadu, India.AB is one of the leading proponents and thinkers on the subject of open source software - articulating the difference between the philosophy and business model. An active contributor to a number of open source projects, he is a board member of India's Free Software Foundation.Links Referenced: MinIO: https://min.io/ Twitter: https://twitter.com/abperiasamy LinkedIn: https://www.linkedin.com/in/abperiasamy/ Email: mailto:ab@min.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Chronosphere. When it costs more money and time to observe your environment than it does to build it, there's a problem. With Chronosphere, you can shape and transform observability data based on need, context and utility. Learn how to only store the useful data you need to see in order to reduce costs and improve performance at chronosphere.io/corey-quinn. That's chronosphere.io/corey-quinn. And my thanks to them for sponsor ing my ridiculous nonsense. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn, and I have taken a somewhat strong stance over the years on the relative merits of multi-cloud, and when it makes sense and when it doesn't. And it's time for me to start modifying some of those. To have that conversation and several others as well, with me today on this promoted guest episode is AB Periasamy, CEO and co-founder of MinIO. AB, it's great to have you back.AB: Yes, it's wonderful to be here again, Corey.Corey: So, one thing that I want to start with is defining terms. Because when we talk about multi-cloud, there are—to my mind at least—smart ways to do it and ways that are frankly ignorant. The thing that I've never quite seen is, it's greenfield, day one. Time to build something. Let's make sure we can build and deploy it to every cloud provider we might ever want to use.And that is usually not the right path. Whereas different workloads in different providers, that starts to make a lot more sense. When you do mergers and acquisitions, as big companies tend to do in lieu of doing anything interesting, it seems like they find it oh, we're suddenly in multiple cloud providers, should we move this acquisition to a new cloud? No. No, you should not.One of the challenges, of course, is that there's a lot of differentiation between the baseline offerings that cloud providers have. MinIO is interesting in that it starts and stops with an object store that is mostly S3 API compatible. Have I nailed the basic premise of what it is you folks do?AB: Yeah, it's basically an object store. Amazon S3 versus us, it's actually—that's the comparable, right? Amazon S3 is a hosted cloud storage as a service, but underneath the underlying technology is called object-store. MinIO is a software and it's also open-source and it's the software that you can deploy on the cloud, deploy on the edge, deploy anywhere, and both Amazon S3 and MinIO are exactly S3 API compatible. It's a drop-in replacement. You can write applications on MinIO and take it to AWS S3, and do the reverse. Amazon made S3 API a standard inside AWS, we made S3 API standard across the whole cloud, all the cloud edge, everywhere, rest of the world.Corey: I want to clarify two points because otherwise I know I'm going to get nibbled to death by ducks on the internet. When you say open-source, it is actually open-source; you're AGPL, not source available, or, “We've decided now we're going to change our model for licensing because oh, some people are using this without paying us money,” as so many companies seem to fall into that trap. You are actually open-source and no one reasonable is going to be able to disagree with that definition.The other pedantic part of it is when something says that it's S3 compatible on an API basis, like, the question is always does that include the weird bugs that we wish it wouldn't have, or some of the more esoteric stuff that seems to be a constant source of innovation? To be clear, I don't think that you need to be particularly compatible with those very corner and vertex cases. For me, it's always been the basic CRUD operations: can you store an object? Can you give it back to me? Can you delete the thing? And maybe an update, although generally object stores tend to be atomic. How far do you go down that path of being, I guess, a faithful implementation of what the S3 API does, and at which point you decide that something is just, honestly, lunacy and you feel no need to wind up supporting that?AB: Yeah, the unfortunate part of it is we have to be very, very deep. It only takes one API to break. And it's not even, like, one API we did not implement; one API under a particular circumstance, right? Like even if you see, like, AWS SDK is, right, Java SDK, different versions of Java SDK will interpret the same API differently. And AWS S3 is an API, it's not a standard.And Amazon has published the REST specifications, API specs, but they are more like religious text. You can interpret it in many ways. Amazon's own SDK has interpreted, like, this in several ways, right? The only way to get it right is, like, you have to have a massive ecosystem around your application. And if one thing breaks—today, if I commit a code and it introduced a regression, I will immediately hear from a whole bunch of community what I broke.There's no certification process here. There is no industry consortium to control the standard, but then there is an accepted standard. Like, if the application works, they need works. And one way to get it right is, like, Amazon SDKs, all of those language SDKs, to be cleaner, simpler, but applications can even use MinIO SDK to talk to Amazon and Amazon SDK to talk to MinIO. Now, there is a clear, cooperative model.And I actually have tremendous respect for Amazon engineers. They have only been kind and meaningful, like, reasonable partnership. Like, if our community reports a bug that Amazon rolled out a new update in one of the region and the S3 API broke, they will actually go fix it. They will never argue, “Why are you using MinIO SDK?” Their engineers, they do everything by reason. That's the reason why they gained credibility.Corey: I think, on some level, that we can trust that the API is not going to meaningfully shift, just because so much has been built on top of it over the last 15, almost 16 years now that even slight changes require massive coordination. I remember there was a little bit of a kerfuffle when they announced that they were going to be disabling the BitTorrent endpoint in S3 and it was no longer going to be supported in new regions, and eventually they were turning it off. There were still people pushing back on that. I'm still annoyed by some of the documentation around the API that says that it may not return a legitimate error code when it errors with certain XML interpretations. It's… it's kind of become very much its own thing.AB: [unintelligible 00:06:22] a problem, like, we have seen, like, even stupid errors similar to that, right? Like, HTTP headers are supposed to be case insensitive, but then there are some language SDKs will send us in certain type of casing and they expect the case to be—the response to be same way. And that's not HTTP standard. If we have to accept that bug and respond in the same way, then we are asking a whole bunch of community to go fix that application. And Amazon's problem are our problems too. We have to carry that baggage.But some places where we actually take a hard stance is, like, Amazon introduced that initially, the bucket policies, like access control list, then finally came IAM, then we actually, for us, like, the best way to teach the community is make best practices the standard. The only way to do it. We have been, like, educating them that we actually implemented ACLs, but we removed it. So, the customers will no longer use it. The scale at which we are growing, if I keep it, then I can never force them to remove.So, we have been pedantic about, like, how, like, certain things that if it's a good advice, force them to do it. That approach has paid off, but the problem is still quite real. Amazon also admits that S3 API is no longer simple, but at least it's not like POSIX, right? POSIX is a rich set of API, but doesn't do useful things that we need to do. So, Amazon's APIs are built on top of simple primitive foundations that got the storage architecture correct, and then doing sophisticated functionalities on top of the simple primitives, these atomic RESTful APIs, you can finally do it right and you can take it to great lengths and still not break the storage system.So, I'm not so concerned. I think it's time for both of us to slow down and then make sure that the ease of operation and adoption is the goal, then trying to create an API Bible.Corey: Well, one differentiation that you have that frankly I wish S3 would wind up implementing is this idea of bucket quotas. I would give a lot in certain circumstances to be able to say that this S3 bucket should be able to hold five gigabytes of storage and no more. Like, you could fix a lot of free tier problems, for example, by doing something like that. But there's also the problem that you'll see in data centers where, okay, we've now filled up whatever storage system we're using. We need to either expand it at significant cost and it's going to take a while or it's time to go and maybe delete some of the stuff we don't necessarily need to keep in perpetuity.There is no moment of reckoning in traditional S3 in that sense because, oh, you can just always add one more gigabyte at 2.3 or however many cents it happens to be, and you wind up with an unbounded growth problem that you're never really forced to wrestle with. Because it's infinite storage. They can add drives faster than you can fill them in most cases. So, it's it just feels like there's an economic story, if nothing else, just from a governance control and make sure this doesn't run away from me, and alert me before we get into the multi-petabyte style of storage for my Hello World WordPress website.AB: Mm-hm. Yeah, so I always thought that Amazon did not do this—it's not just Amazon, the cloud players, right—they did not do this because they want—is good for their business; they want all the customers' data, like unrestricted growth of data. Certainly it is beneficial for their business, but there is an operational challenge. When you set quota—this is why we grudgingly introduced this feature. We did not have quotas and we didn't want to because Amazon S3 API doesn't talk about quota, but the enterprise community wanted this so badly.And eventually we [unintelligible 00:09:54] it and we gave. But there is one issue to be aware of, right? The problem with quota is that you as an object storage administrator, you set a quota, let's say this bucket, this application, I don't see more than 20TB; I'm going to set 100TB quota. And then you forget it. And then you think in six months, they will reach 20TB. The reality is, in six months they reach 100TB.And then when nobody expected—everybody has forgotten that there was a code a certain place—suddenly application start failing. And when it fails, it doesn't—even though the S3 API responds back saying that insufficient space, but then the application doesn't really pass that error all the way up. When applications fail, they fail in unpredictable ways. By the time the application developer realizes that it's actually object storage ran out of space, the lost time and it's a downtime. So, as long as they have proper observability—because I mean, I've will also asked observability, that it can alert you that you are only going to run out of space soon. If you have those system in place, then go for quota. If not, I would agree with the S3 API standard that is not about cost. It's about operational, unexpected accidents.Corey: Yeah, on some level, we wound up having to deal with the exact same problem with disk volumes, where my default for most things was, at 70%, I want to start getting pings on it and at 90%, I want to be woken up for it. So, for small volumes, you wind up with a runaway log or whatnot, you have a chance to catch it and whatnot, and for the giant multi-petabyte things, okay, well, why would you alert at 70% on that? Well, because procurement takes a while when we're talking about buying that much disk for that much money. It was a roughly good baseline for these things. The problem, of course, is when you have none of that, and well it got full so oops-a-doozy.On some level, I wonder if there's a story around soft quotas that just scream at you, but let you keep adding to it. But that turns into implementation details, and you can build something like that on top of any existing object store if you don't need the hard limit aspect.AB: Actually, that is the right way to do. That's what I would recommend customers to do. Even though there is hard quota, I will tell, don't use it, but use soft quota. And the soft quota, instead of even soft quota, you monitor them. On the cloud, at least you have some kind of restriction that the more you use, the more you pay; eventually the month end bills, it shows up.On MinIO, when it's deployed on these large data centers, that it's unrestricted access, quickly you can use a lot of space, no one knows what data to delete, and no one will tell you what data to delete. The way to do this is there has to be some kind of accountability.j, the way to do it is—actually [unintelligible 00:12:27] have some chargeback mechanism based on the bucket growth. And the business units have to pay for it, right? That IT doesn't run for free, right? IT has to have a budget and it has to be sponsored by the applications team.And you measure, instead of setting a hard limit, you actually charge them that based on the usage of your bucket, you're going to pay for it. And this is a observability problem. And you can call it soft quotas, but it hasn't been to trigger an alert in observability. It's observability problem. But it actually is interesting to hear that as soft quotas, which makes a lot of sense.Corey: It's one of those problems that I think people only figure out after they've experienced it once. And then they look like wizards from the future who, “Oh, yeah, you're going to run into a quota storage problem.” Yeah, we all find that out because the first time we smack into something and live to regret it. Now, we can talk a lot about the nuances and implementation and low level detail of this stuff, but let's zoom out of it. What are you folks up to these days? What is the bigger picture that you're seeing of object storage and the ecosystem?AB: Yeah. So, when we started, right, our idea was that world is going to produce incredible amount of data. In ten years from now, we are going to drown in data. We've been saying that today and it will be true. Every year, you say ten years from now and it will still be valid, right?That was the reason for us to play this game. And we saw that every one of these cloud players were incompatible with each other. It's like early Unix days, right? Like a bunch of operating systems, everything was incompatible and applications were beginning to adopt this new standard, but they were stuck. And then the cloud storage players, whatever they had, like, GCS can only run inside Google Cloud, S3 can only run inside AWS, and the cloud player's game was bring all the world's data into the cloud.And that actually requires enormous amount of bandwidth. And moving data into the cloud at that scale, if you look at the amount of data the world is producing, if the data is produced inside the cloud, it's a different game, but the data is produced everywhere else. MinIO's idea was that instead of introducing yet another API standard, Amazon got the architecture right and that's the right way to build large-scale infrastructure. If we stick to Amazon S3 API instead of introducing it another standard, [unintelligible 00:14:40] API, and then go after the world's data. When we started in 2014 November—it's really 2015, we started, it was laughable. People thought that there won't be a need for MinIO because the whole world will basically go to AWS S3 and they will be the world's data store. Amazon is capable of doing that; the race is not over, right?Corey: And it still couldn't be done now. The thing is that they would need to fundamentally rethink their, frankly, you serious data egress charges. The problem is not that it's expensive to store data in AWS; it's that it's expensive to store data and then move it anywhere else for analysis or use on something else. So, there are entire classes of workload that people should not consider the big three cloud providers as the place where that data should live because you're never getting it back.AB: Spot on, right? Even if network is free, right, Amazon makes, like, okay, zero egress-ingress charge, the data we're talking about, like, most of MinIO deployments, they start at petabytes. Like, one to ten petabyte, feels like 100 terabyte. For even if network is free, try moving a ten-petabyte infrastructure into the cloud. How are you going to move it?Even with FedEx and UPS giving you a lot of bandwidth in their trucks, it is not possible, right? I think the data will continue to be produced everywhere else. So, our bet was there we will be [unintelligible 00:15:56]—instead of you moving the data, you can run MinIO where there is data, and then the whole world will look like AWS's S3 compatible object store. We took a very different path. But now, when I say the same story that when what we started with day one, it is no longer laughable, right?People believe that yes, MinIO is there because our market footprint is now larger than Amazon S3. And as it goes to production, customers are now realizing it's basically growing inside a shadow IT and eventually businesses realize the bulk of their business-critical data is sitting on MinIO and that's how it's surfacing up. So now, what we are seeing, this year particularly, all of these customers are hugely concerned about cost optimization. And as part of the journey, there is also multi-cloud and hybrid-cloud initiatives. They want to make sure that their application can run on any cloud or on the same software can run on their colos like Equinix, or like bunch of, like, Digital Reality, anywhere.And MinIO's software, this is what we set out to do. MinIO can run anywhere inside the cloud, all the way to the edge, even on Raspberry Pi. It's now—whatever we started with is now has become reality; the timing is perfect for us.Corey: One of the challenges I've always had with the idea of building an application with the idea to run it anywhere is you can make explicit technology choices around that, and for example, object store is a great example because most places you go now will or can have an object store available for your use. But there seem to be implementation details that get lost. And for example, even load balancers wind up being implemented in different ways with different scaling times and whatnot in various environments. And past a certain point, it's okay, we're just going to have to run it ourselves on top of HAproxy or Nginx, or something like it, running in containers themselves; you're reinventing the wheel. Where is that boundary between, we're going to build this in a way that we can run anywhere and the reality that I keep running into, which is we tried to do that but we implicitly without realizing it built in a lot of assumptions that everything would look just like this environment that we started off in.AB: The good part is that if you look at the S3 API, every request has the site name, the endpoint, bucket name, the path, and the object name. Every request is completely self-contained. It's literally a HTTP call away. And this means that whether your application is running on Android, iOS, inside a browser, JavaScript engine, anywhere across the world, they don't really care whether the bucket is served from EU or us-east or us-west. It doesn't matter at all, so it actually allows you by API, you can build a globally unified data infrastructure, some buckets here, some buckets there.That's actually not the problem. The problem comes when you have multiple clouds. Different teams, like, part M&A, the part—like they—even if you don't do M&A, different teams, no two data engineer will would agree on the same software stack. Then where they will all end up with different cloud players and some is still running on old legacy environment.When you combine them, the problem is, like, let's take just the cloud, right? How do I even apply a policy, that access control policy, how do I establish unified identity? Because I want to know this application is the only one who is allowed to access this bucket. Can I have that same policy on Google Cloud or Azure, even though they are different teams? Like if that employer, that project, or that admin, if he or she leaves the job, how do I make sure that that's all protected?You want unified identity, you want unified access control policies. Where are the encryption key store? And then the load balancer itself, the load, its—load balancer is not the problem. But then unless you adopt S3 API as your standard, the definition of what a bucket is different from Microsoft to Google to Amazon.Corey: Yeah, the idea of an of the PUTS and retrieving of actual data is one thing, but then you have how do you manage it the control plane layer of the object store and how do you rationalize that? What are the naming conventions? How do you address it? I even ran into something similar somewhat recently when I was doing an experiment with one of the Amazon Snowball edge devices to move some data into S3 on a lark. And the thing shows up and presents itself on the local network as an S3 endpoint, but none of their tooling can accept a different endpoint built into the configuration files; you have to explicitly use it as an environment variable or as a parameter on every invocation of something that talks to it, which is incredibly annoying.I would give a lot for just to be able to say, oh, when you're talking in this profile, that's always going to be your S3 endpoint. Go. But no, of course not. Because that would make it easier to use something that wasn't them, so why would they ever be incentivized to bake that in?AB: Yeah. Snowball is an important element to move data, right? That's the UPS and FedEx way of moving data, but what I find customers doing is they actually use the tools that we built for MinIO because the Snowball appliance also looks like S3 API-compatible object store. And in fact, like, I've been told that, like, when you want to ship multiple Snowball appliances, they actually put MinIO to make it look like one unit because MinIO can erase your code objects across multiple Snowball appliances. And the MC tool, unlike AWS CLI, which is really meant for developers, like low-level calls, MC gives you unique [scoring 00:21:08] tools, like lscp, rsync-like tools, and it's easy to move and copy and migrate data. Actually, that's how people deal with it.Corey: Oh, God. I hadn't even considered the problem of having a fleet of Snowball edges here that you're trying to do a mass data migration on, which is basically how you move petabyte-scale data, is a whole bunch of parallelism. But having to figure that out on a case-by-case basis would be nightmarish. That's right, there is no good way to wind up doing that natively.AB: Yeah. In fact, Western Digital and a few other players, too, now the Western Digital created a Snowball-like appliance and they put MinIO on it. And they are actually working with some system integrators to help customers move lots of data. But Snowball-like functionality is important and more and more customers who need it.Corey: This episode is sponsored in part by Honeycomb. I'm not going to dance around the problem. Your. Engineers. Are. Burned. Out. They're tired from pagers waking them up at 2 am for something that could have waited until after their morning coffee. Ring Ring, Who's There? It's Nagios, the original call of duty! They're fed up with relying on two or three different “monitoring tools” that still require them to manually trudge through logs to decipher what might be wrong. Simply put, there's a better way. Observability tools like Honeycomb (and very little else because they do admittedly set the bar) show you the patterns and outliers of how users experience your code in complex and unpredictable environments so you can spend less time firefighting and more time innovating. It's great for your business, great for your engineers, and, most importantly, great for your customers. Try FREE today at honeycomb.io/screaminginthecloud. That's honeycomb.io/screaminginthecloud.Corey: Increasingly, it felt like, back in the on-prem days, that you'd have a file server somewhere that was either a SAN or it was going to be a NAS. The question was only whether it presented it to various things as a volume or as a file share. And then in cloud, the default storage mechanism, unquestionably, was object store. And now we're starting to see it come back again. So, it started to increasingly feel, in a lot of ways, like Cloud is no longer so much a place that is somewhere else, but instead much more of an operating model for how you wind up addressing things.I'm wondering when the generation of prosumer networking equipment, for example, is going to say, “Oh, and send these logs over to what object store?” Because right now, it's still write a file and SFTP it somewhere else, at least the good ones; some of the crap ones still want old unencrypted FTP, which is neither here nor there. But I feel like it's coming back around again. Like, when do even home users wind up instead of where do you save this file to having the cloud abstraction, which hopefully, you'll never have to deal with an S3-style endpoint, but that can underpin an awful lot of things. It feels like it's coming back and that's cloud is the de facto way of thinking about things. Is that what you're seeing? Does that align with your belief on this?AB: I actually, fundamentally believe in the long run, right, applications will go SaaS, right? Like, if you remember the days that you used to install QuickBooks and ACT and stuff, like, on your data center, you used to run your own Exchange servers, like, those days are gone. I think these applications will become SaaS. But then the infrastructure building blocks for these SaaS, whether they are cloud or their own colo, I think that in the long run, it will be multi-cloud and colo all combined and all of them will look alike.But what I find from the customer's journey, the Old World and the New World is incompatible. When they shifted from bare metal to virtualization, they didn't have to rewrite their application. But this time, you have—it as a tectonic shift. Every single application, you have to rewrite. If you retrofit your application into the cloud, bad idea, right? It's going to cost you more and I would rather not do it.Even though cloud players are trying to make, like, the file and block, like, file system services [unintelligible 00:24:01] and stuff, they make it available ten times more expensive than object, but it's just to [integrate 00:24:07] some legacy applications, but it's still a bad idea to just move legacy applications there. But what I'm finding is that the cost, if you still run your infrastructure with enterprise IT mindset, you're out of luck. It's going to be super expensive and you're going to be left out modern infrastructure, because of the scale, it has to be treated as code. You have to run infrastructure with software engineers. And this cultural shift has to happen.And that's why cloud, in the long run, everyone will look like AWS and we always said that and it's now being becoming true. Like, Kubernetes and MinIO basically is leveling the ground everywhere. It's giving ECS and S3-like infrastructure inside AWS or outside AWS, everywhere. But what I find the challenging part is the cultural mindset. If they still have the old cultural mindset and if they want to adopt cloud, it's not going to work.You have to change the DNA, the culture, the mindset, everything. The best way to do it is go to the cloud-first. Adopt it, modernize your application, learn how to run and manage infrastructure, then ask economics question, the unit economics. Then you will find the answers yourself.Corey: On some level, that is the path forward. I feel like there's just a very long tail of systems that have been working and have been meeting the business objective. And well, we should go and refactor this because, I don't know, a couple of folks on a podcast said we should isn't the most compelling business case for doing a lot of it. It feels like these things sort of sit there until there is more upside than just cost-cutting to changing the way these things are built and run. That's the reason that people have been talking about getting off of mainframe since the '90s in some companies, and the mainframe is very much still there. It is so ingrained in the way that they do business, they have to rethink a lot of the architectural things that have sprung up around it.I'm not trying to shame anyone for the [laugh] state that their environment is in. I've never yet met a company that was super proud of its internal infrastructure. Everyone's always apologizing because it's a fire. But they think someone else has figured this out somewhere and it all runs perfectly. I don't think it exists.AB: What I am finding is that if you are running it the enterprise IT style, you are the one telling the application developers, here you go, you have this many VMs and then you have, like, a VMware license and, like, Jboss, like WebLogic, and like a SQL Server license, now you go build your application, you won't be able to do it. Because application developers talk about Kafka and Redis and like Kubernetes, they don't speak the same language. And that's when these developers go to the cloud and then finish their application, take it live from zero lines of code before it can procure infrastructure and provision it to these guys. The change that has to happen is how can you give what the developers want now that reverse journey is also starting. In the long run, everything will look alike, but what I'm finding is if you're running enterprise IT infrastructure, traditional infrastructure, they are ashamed of talking about it.But then you go to the cloud and then at scale, some parts of it, you want to move for—now you really know why you want to move. For economic reasons, like, particularly the data-intensive workloads becomes very expensive. And at that part, they go to a colo, but leave the applications on the cloud. So, it's the multi-cloud model, I think, is inevitable. The expensive pieces that where you can—if you are looking at yourself as hyperscaler and if your data is growing, if your business focus is data-centric business, parts of the data and data analytics, ML workloads will actually go out, if you're looking at unit economics. If all you are focused on productivity, stick to the cloud and you're still better off.Corey: I think that's a divide that gets lost sometimes. When people say, “Oh, we're going to move to the cloud to save money.” It's, “No you're not.” At a five-year time horizon, I would be astonished if that juice were worth the squeeze in almost any scenario. The reason you go for therefore is for a capability story when it's right for you.That also means that steady-state workloads that are well understood can often be run more economically in a place that is not the cloud. Everyone thinks for some reason that I tend to be its cloud or it's trash. No, I'm a big fan of doing things that are sensible and cloud is not the right answer for every workload under the sun. Conversely, when someone says, “Oh, I'm building a new e-commerce store,” or whatnot, “And I've decided cloud is not for me.” It's, “Ehh, you sure about that?”That sounds like you are smack-dab in the middle of the cloud use case. But all these things wind up acting as constraints and strategic objectives. And technology and single-vendor answers are rarely going to be a panacea the way that their sales teams say that they will.AB: Yeah. And I find, like, organizations that have SREs, DevOps, and software engineers running the infrastructure, they actually are ready to go multi-cloud or go to colo because they have the—exactly know. They have the containers and Kubernetes microservices expertise. If you are still on a traditional SAN, NAS, and VM architecture, go to cloud, rewrite your application.Corey: I think there's a misunderstanding in the ecosystem around what cloud repatriation actually looks like. Everyone claims it doesn't exist because there's basically no companies out there worth mentioning that are, “Yep, we've decided the cloud is terrible, we're taking everything out and we are going to data centers. The end.” In practice, it's individual workloads that do not make sense in the cloud. Sometimes just the back-of-the-envelope analysis means it's not going to work out, other times during proof of concepts, and other times, as things have hit a certain point of scale, we're in an individual workload being pulled back makes an awful lot of sense. But everything else is probably going to stay in the cloud and these companies don't want to wind up antagonizing the cloud providers by talking about it in public. But that model is very real.AB: Absolutely. Actually, what we are finding with the application side, like, parts of their overall ecosystem, right, within the company, they run on the cloud, but the data side, some of the examples, like, these are in the range of 100 to 500 petabytes. The 500-petabyte customer actually started at 500 petabytes and their plan is to go at exascale. And they are actually doing repatriation because for them, their customers, it's consumer-facing and it's extremely price sensitive, but when you're a consumer-facing, every dollar you spend counts. And if you don't do it at scale, it matters a lot, right? It will kill the business.Particularly last two years, the cost part became an important element in their infrastructure, they knew exactly what they want. They are thinking of themselves as hyperscalers. They get commodity—the same hardware, right, just a server with a bunch of [unintelligible 00:30:35] and network and put it on colo or even lease these boxes, they know what their demand is. Even at ten petabytes, the economics starts impacting. If you're processing it, the data side, we have several customers now moving to colo from cloud and this is the range we are talking about.They don't talk about it publicly because sometimes, like, you don't want to be anti-cloud, but I think for them, they're also not anti-cloud. They don't want to leave the cloud. The completely leaving the cloud, it's a different story. That's not the case. Applications stay there. Data lakes, data infrastructure, object store, particularly if it goes to a colo.Now, your applications from all the clouds can access this centralized—centralized, meaning that one object store you run on colo and the colos themselves have worldwide data centers. So, you can keep the data infrastructure in a colo, but applications can run on any cloud, some of them, surprisingly, that they have global customer base. And not all of them are cloud. Sometimes like some applications itself, if you ask what type of edge devices they are running, edge data centers, they said, it's a mix of everything. What really matters is not the infrastructure. Infrastructure in the end is CPU, network, and drive. It's a commodity. It's really the software stack, you want to make sure that it's containerized and easy to deploy, roll out updates, you have to learn the Facebook-Google style running SaaS business. That change is coming.Corey: It's a matter of time and it's a matter of inevitability. Now, nothing ever stays the same. Everything always inherently changes in the full sweep of things, but I'm pretty happy with where I see the industry going these days. I want to start seeing a little bit less centralization around one or two big companies, but I am confident that we're starting to see an awareness of doing these things for the right reason more broadly permeating.AB: Right. Like, the competition is always great for customers. They get to benefit from it. So, the decentralization is a path to bringing—like, commoditizing the infrastructure. I think the bigger picture for me, what I'm particularly happy is, for a long time we carried industry baggage in the infrastructure space.If no one wants to change, no one wants to rewrite application. As part of the equation, we carried the, like, POSIX baggage, like SAN and NAS. You can't even do [unintelligible 00:32:48] as a Service, NFS as a Service. It's too much of a baggage. All of that is getting thrown out. Like, the cloud players be helped the customers start with a clean slate. I think to me, that's the biggest advantage. And that now we have a clean slate, we can now go on a whole new evolution of the stack, keeping it simpler and everyone can benefit from this change.Corey: Before we wind up calling this an episode, I do have one last question for you. As I mentioned at the start, you're very much open-source, as in legitimate open-source, which means that anyone who wants to can grab an implementation and start running it. How do you, I guess make peace with the fact that the majority of your user base is not paying you? And I guess how do you get people to decide, “You know what? We like the cut of his jib. Let's give him some money.”AB: Mm-hm. Yeah, if I looked at it that way, right, I have both the [unintelligible 00:33:38], right, on the open-source side as well as the business. But I don't see them to be conflicting. If I run as a charity, right, like, I take donation. If you love the product, here is the donation box, then that doesn't work at all, right?I shouldn't take investor money and I shouldn't have a team because I have a job to pay their bills, too. But I actually find open-source to be incredibly beneficial. For me, it's about delivering value to the customer. If you pay me $5, I ought to make you feel $50 worth of value. The same software you would buy from a proprietary vendor, why would—if I'm a customer, same software equal in functionality, if its proprietary, I would actually prefer open-source and pay even more.But why are, really, customers paying me now and what's our view on open-source? I'm actually the free software guy. Free software and open-source are actually not exactly equal, right? We are the purest of the open-source community and we have strong views on what open-source means, right. That's why we call it free software. And free here means freedom, right? Free does not mean gratis, that free of cost. It's actually about freedom and I deeply care about it.For me it's a philosophy and it's a way of life. That's why I don't believe in open core and other models that holding—giving crippleware is not open-source, right? I give you some freedom but not all, right, like, it's it breaks the spirit. So, MinIO is a hundred percent open-source, but it's open-source for the open-source community. We did not take some community-developed code and then added commercial support on top.We built the product, we believed in open-source, we still believe and we will always believe. Because of that, we open-sourced our work. And it's open-source for the open-source community. And as you build applications that—like the AGPL license on the derivative works, they have to be compatible with AGPL because we are the creator. If you cannot open-source, you open-source your application derivative works, you can buy a commercial license from us. We are the creator, we can give you a dual license. That's how the business model works.That way, the open-source community completely benefits. And it's about the software freedom. There are customers, for them, open-source is good thing and they want to pay because it's open-source. There are some customers that they want to pay because they can't open-source their application and derivative works, so they pay. It's a happy medium; that way I actually find open-source to be incredibly beneficial.Open-source gave us that trust, like, more than adoption rate. It's not like free to download and use. More than that, the customers that matter, the community that matters because they can see the code and they can see everything we did, it's not because I said so, marketing and sales, you believe them, whatever they say. You download the product, experience it and fall in love with it, and then when it becomes an important part of your business, that's when they engage with us because they talk about license compatibility and data loss or a data breach, all that becomes important. Open-source isn't—I don't see that to be conflicting for business. It actually is incredibly helpful. And customers see that value in the end.Corey: I really want to thank you for being so generous with your time. If people want to learn more, where should they go?AB: I was on Twitter and now I think I'm spending more time on, maybe, LinkedIn. I think if they—they can send me a request and then we can chat. And I'm always, like, spending time with other entrepreneurs, architects, and engineers, sharing what I learned, what I know, and learning from them. There is also a [community open channel 00:37:04]. And just send me a mail at ab@min.io and I'm always interested in talking to our user base.Corey: And we will, of course, put links to that in the [show notes 00:37:12]. Thank you so much for your time. I appreciate it.AB: It's wonderful to be here.Corey: AB Periasamy, CEO and co-founder of MinIO. I'm Cloud Economist Corey Quinn and this has been a promoted guest episode of Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice that presumably will also include an angry, loud comment that we can access from anywhere because of shared APIs.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

This week Drusilla and Josh cover the 2016 micro-budget horror film, The Alchemist Cookbook. Josh is sick, being lazy is anti-capitalist, How to Do Nothing, They talk a long time about SCREIVI or SCREAM 6. From wiki: “The Alchemist Cookbook is a horror film directed by Joel Potrykus. The film was released on the 7th of October 2016 in New York City. The film stars Ty Hickson as "Sean" and Amari Cheatom as "Cortez". The film was produced by Oscilloscope Laboratories and by producers Andrew D. Corkin, Bryan Reisberg and Ashley Young.[1] "Sean" is an outcast who isolates himself from society to practice alchemy, accompanied by only his cat. As his mental condition deteriorates the line of what is real and what is not becomes blurred, and as his chemistry turns to black magic, he instead summons a demon.[1] The film breaks the conventional boundaries of genre, as elements of a black comedy, horror and a psychological thriller are all incorporated and intertwined.[2] The Alchemist cookbook was released on an alternate release strategy, as a pay what you want film, in both theatres and on BitTorrent bundle.[2] The film was an official selection at the 2016 South by Southwest Film Festival.[1]”  NEXT WEEK: Don't Deliver Us From Evil (1972) Website: http://www.bloodhauspod.com  Twitter: https://twitter.com/BloodhausPod Instagram: https://www.instagram.com/bloodhauspod/ Email: bloodhauspod@gmail.com         Drusilla's art: https://www.sisterhydedesign.com/  Drusilla's Instagram: https://www.instagram.com/hydesister/  Drusilla's Letterboxd: https://letterboxd.com/drew_phillips/         Joshua's website: https://www.joshuaconkel.com/  Joshua's Twitter: https://twitter.com/JoshuaConkel   Joshua's Instagram: https://www.instagram.com/joshua_conkel/  Joshua's Letterboxd: https://letterboxd.com/joshuaconkel

The Big Mates discuss polyfauna, BitTorrent, supergroups, and a whole host of side projects by the members of Radiohead.Adam, Steve, and Lucas continue their deep dive into the career and discography of Radiohead by taking in the various pieces of work undertaken by the constituent parts of Radiohead between 2013 and 2015. They talk about the writing and recording processes, the context surrounding the songs, and offer up analysis, opinions, and thoughts from three differing perspectives on music, from being deeply into analysis and music, to not caring for art or critique, and everything in between.They cover AMOK by Atoms for Peace, Tomorrow's Modern Boxes, the second solo album by Thom Yorke, Philip Selway's second solo album Weatherhouse, and the album Jonny Greenwood created with Shye Ben Tzur and the Rajasthan Express, Junun!Who did what and when and why? Where did Ed go? What on earth is Colin up to? Find out on this episode of What Is Music?Our next episode is out on Monday April 3rd and will begin our deep-dive into A Moon Shaped Pool.Join the conversation on:Twitter: https://twitter.com/whatismusicpodInstagram: https://www.instagram.com/whatismusicpodE-mail: whatismusicpod@gmail.comGet access to more shows, exclusive bonus content, ad-free episodes of this show, and more music discussion by subscribing to our Patreon!Head to patreon.com/whatismusicpod and receive up to two new episodes of our various shows every week (including shows about Manic Street Preachers and monthly themed playlists!), ad-free archives of What Is Music?, and access to our Patron-only Discord server for even more music (and non-music) discussion!Support our show when starting your own podcast!By signing up to Buzzsprout with this link: https://www.buzzsprout.com/?referrer_id=780379Check out our merch!https://whatismusicpod.redbubble.comDonate to our podcast!https://ko-fi.com/whatismusichttp://whatismusic.buzzsprout.com/Support the show

This week's podcast is about network effects, as described by James Courrier and NFX.You can listen to this podcast here, which has the slides and graphics mentioned. Also available at iTunes and Google Podcasts.Here is the link to the TechMoat Consulting.Here is the link to the China Tech Tour.Here is the NFx Network Effects manual.Here are NFX's 16 network effects, in order of strength:Physical (e.g. landline telephones)Protocol (e.g. Ethernet)Personal Utility (e.g. iMessage, WhatsApp)Personal (e.g. Facebook)Market Network (e.g. HoneyBook, AngelList)Marketplace (e.g. eBay, Craigslist)Platform (e.g. Windows, iOS, Android)Asymptotic Marketplace (e.g. Uber, Lyft)Data (e.g. Waze, Yelp!)Tech Performance (e.g. Bittorrent,Skype)Language (e.g. Google, Xerox)Belief (currencies, religions)Bandwagon (e.g. Slack, Apple)Expertise (Figma, Microsoft Excel)Tribal (Apple, Harvard, NY Yankees…)Hub-and-Spoke (TikTok, Medium, Craigslist)——Related articles:3 Types of Network Effects (Asia Tech Strategy – Daily Lesson / Update)Questions for Huawei's CEO, JD & Jingxi, Metcalfe's Law Is Dumb (Asia Tech Strategy)From the Concept Library, concepts for this article are:Network EffectsData Advantages and Network EffectsEmbeddingBandwagon EffectsThis content (articles, podcasts, website info) is not investment, legal or tax advice. The information and opinions from me and any guests may be incorrect. The numbers and information may be wrong. The views expressed may no longer be relevant or accurate. Support the show

One thing to make abundantly clear: Nostr is a protocol. It's a set of rules that servers and clients use to communicate (just like Bitcoin, email or Bittorrent). Nostr is not an app nor a “platform” (like Twitter, Facebook, etc.), but many applications can be built on top of Nostr. In the words of Edward Snowden, “If a platform is a silo, a protocol is a river: no one owns it, and everyone is free to swim.” – Walker V. If you have been hearing about this Nostr thing, and wanted to know what it was and how to find and use it... this is the episode for you. The future is here, it's just not evenly distributed yet. Follow the link below for the great resources shared in the article and to get started with Nostr today! https://www.btctimes.com/news/what-is-nostr-how-does-it-work-why-does-it-matter Check out the Nostrovia podcast if you want to take a deep dive! Walker's Nostr Pubkey: npub1cj8znuztfqkvq89pl8hceph0svvvqk0qay6nydgk9uyq7fhpfsgsqwrz4u Guy's Nostr Pubkey: npub1h8nk2346qezka5cpm8jjh3yl5j88pf4ly2ptu7s6uu55wcfqy0wq36rpev Don't forget to check out our amazing sponsors: • Dive into the Bitcoin only wallet, the cypherpunk calculator, with the NEW Coldcard Q1! A company that has built secure Bitcoin products for nearly a decade. Code BITCOINAUDIBLE gets 9% off everything in your cart! (https://guyswann.com/coldcard) • Gets sats back every time you dump fiat at a store, to pay your bills, everything in your fiat life pays you sats with the Fold Debit Card and FoldApp. 20,000 FREE SATS! at (https://guyswann.com/fold) • The best place to onboard a true Bitcoiner - Stack sats automatically, withdraw automatically, and learn or get help from the best team of Bitcoiners out there with Swan Bitcoin. (https://swanbitcoin.com/guy) -------------------------------------- "Do not let your fire go out, spark by irreplaceable spark in the hopeless swamps of the not-quite, the not-yet, and the not-at-all. Do not let the hero in your soul perish in lonely frustration for the life you deserved and have never been able to reach. The world you desire can be won. It exists.. it is real.. it is possible.. it's yours." — Ayn Rand (Atlas Shrugged) Learn more about your ad choices. Visit megaphone.fm/adchoices

Cuánto cuesta mantener ChatGPT / La fábrica europea de Intel en peligro / Donaciones falsas para Turquía / Teclado de Mordor / Evolución de usuarios de Mastodon / Transmission 4.0 Patrocinador: Estamos todos flipando con el nuevo Galaxy S23, y no es de extrañar. Se viene el móvil con la mejor cámara, por su zoom y su modo Nightography, para capturar tus noches más icónicas sin perder ni un detalle. ¡Di adiós a las fotos con poca luz! — Tienes toda la información en Samsung.es, para aprovecharte de las promociones de lanzamiento. Cuánto cuesta mantener ChatGPT / La fábrica europea de Intel en peligro / Donaciones falsas para Turquía / Teclado de Mordor / Evolución de usuarios de Mastodon / Transmission 4.0

Binny Gill started his career as a programmer after studying CS and Engineering at IIT Kanpur and later UIUC, University of Illinois Urbana-Champaign. He then had an impressive eight-year run as a technology leader at Nutanix, the hyper converged infrastructure company, eventually becoming its CTO for cloud services. In January 2021, Binny left Nutanix to start Kognitos based on a bold vision to make everyone a programmer. In this discussion, we learn about his journey and how generative AI just may change everything.Thanks to Steve Kaplan for the intro to Binny.Listen and learn:Binny's inspiration for starting Kognitos: "...why should humans need to think like machines... when machines can now think like humans?"What makes programming so hard.Why the future of programming is using natural language to describe the features you want.How computing interfaces restrict us from communicating like humans when programming.Why Binny says "generative AI is the new electricity."The most important leadership lesson Binny learned working alongside iconic leaders at IBM.References in this episode:How Petals just became the BitTorrent of LLMsPhil McKinney, former HP CTO, on AI and the Future of WorkThe Kognitos blog

Dr. Jonathan Paprocki Jonathan is a specialist in topological quantum compiling, a Tlon engineer, you can find him on Urbit @  ~Datnut-Pollen Blog can be found at Discordia (http://datnut-pollen.discordja.net/discordja) What is a Kundalini awakening? What do old dusty books say about Kundalini? What is the relationship between Kundalini and topological quantum computers? Can everything be explained by science? How do you deal with the skepticism that other people give you that you used to give them? How much does the flow state come to you when working with quantum computers? Is it very frustrating? What did you learn about Atman after you randomly spoke Atman? Do you think in images? What do you think is the base layer reality now? When did you start studying quantum physics? Who is John Wheeler? What is the participatory universe? He talks about Quantum Bayesianism The idea is that the Universe consists of participants/observers and observations, rules on consistencies of what these interactions guarantee the relationship. What is the study of condensed matter? What is quantum field theory? What is quantum topology? A topologist can't tell the difference between a coffee mug and a donut. Who is Edward Witten, the physicist who received the Field Medal? What are your favorite cognitive hazards? How often do new age thinkers get quantum mechanics right when tying it to spirituality? How important is math to understanding quantum theory? What is quantum theory? Who are the new age mystics who do get quantum theory right? What percentage of quantum theory has to do with the Heisenberg principle? What is the difference between the laws of qualia and the laws of physics? How did you end up at Tlon? What are the differences between Holochain and Urbit? What is a distributed hash table? What is a hash chain? What is naturally inspired computation? What are the aspects of natural physics that are dependent on quantum mechanics? Who is Sam Frank from Urbit? What is the thing that convinces you to pursue incomprehensible projects to the point of comprehensibility? Is doing documentation a good way to learn programming? What is the blog that Jon is writing? What are some differences between writing software for external applications/legacy internet within Urbit versus say writing software in an imperative programming language and navigating Facebook's API? Why did IP addresses not become permanent ways to signify where the message receiver/sender is stable? What does Urbit replace about the current IP address system? What are the differences and similarities between Urbit and Holochain? What is Holochain Scepter? What is Deep Key when it comes to Holochain? How does Urbit enforce global state? What is the data availability problem of the blockchain? Why doesn't adding more blocks make the data processing faster? What is a blockchain? A way to give a clock to distributed system. What is the problem of time in distributed systems? Holochain is a way to run apps on BitTorrent basically (analogy). What percentage of email is spam? What is enforced scarcity? What is the membrane from Holochain? What are all the types of attacks in distributed systems?

A DeFi protocol known as Cryptex offers a permissionless way to speculate on the total crypto market capitalization through its novel index token, TCAP. Now, Cryptex is on the verge of releasing its newest index token, JPEGz, which will allow holders to track the total market cap of NFTs on Ethereum. In this episode of The Scoop, Cryptex co-founder Joe Sticco shares the details behind the upcoming JPEGz NFT index token, including why the index token is a good way to speculate on the future growth of the NFT sector. According to Sticco, the list of NFT collections included in the JPEGz index token is dynamic, which ensures that whenever a hot new NFT project crops up in the future, it will be included in the index. As Sticco explains, “NFTs are literally in their infancy. I look at them now and it's like where crypto was in 2017 — we had that first bust cycle now, and now we're in the ground, and who knows what happens next, but JPEGs is going to be the ability to track the NFT market cap.” Episode 123 of Season 4 of The Scoop was recorded live with The Block's Frank Chaparro and Cryptex co-founder Joe Sticco. Listen below, and subscribe to The Scoop on Apple, Spotify, Google Podcasts, Stitcher or wherever you listen to podcasts. Or click here to watch the full interview on YouTube. Email feedback and revision requests can be sent to podcast@theblockcrypto.com. This episode is brought to you by our sponsors Tron, Ledn About Tron TRON is dedicated to accelerating the decentralization of the internet via blockchain technology and decentralized applications (dApps). Founded in September 2017 by H.E. Justin Sun, the TRON network has continued to deliver impressive achievements since MainNet launch in May 2018. July 2018 also marked the ecosystem integration of BitTorrent, a pioneer in decentralized web3 services boasting over 100 million monthly active users. The TRON network completed full decentralization in December 2021 and is now a community-governed DAO. | TRONDAO | Twitter | Discord | About Ledn Ledn was founded on the unshakeable conviction that digital assets have the power to democratize access to the global economy. We help you to experience the real life benefits of your Bitcoin without having to sell it. Start a savings account, take out a loan, or double your Bitcoin. For more information visit Ledn.io

In this month's 'Market Pulse' episode of The Scoop, The Block Vice President of Research Larry Cermak and Loomdart, a founding member of crypto-native venture firm eGirl Capital, reflect on FTX's collapse and speculate on what comes next for the crypto industry. According to Cermak, former FTX CEO Sam Bankman-Fried (SBF) is remaining in the Bahamas to stay engaged with local regulators: “Sam still has connections in the Bahamas and the government is still standing by him … He just made them look terrible, so they kind of have to go in this direction” During a recent interview, SBF confirmed he is still living in the Bahamas, although he is not communicating with FTX's new CEO John Ray. While FTX's collapse pushed crypto prices to new yearly lows, Loomdart says it is ultimately better for the long-term health of the industry:  “Crypto is such an open and free ecosystem to where if we don't have this type of education by fire it just keeps getting worse … If FTX didn't blow up now and they managed to keep going — I don't think that hole was going to shrink.” Episode 122 of Season 4 of The Scoop was recorded remotely with The Block's Frank Chaparro and Larry Cermak, and Loomdart, co-founder of eGirl Capital. Listen below, and subscribe to The Scoop on Apple, Spotify, Google Podcasts, Stitcher or wherever you listen to podcasts. Or click here to watch the full interview on YouTube. Email feedback and revision requests can be sent to podcast@theblockcrypto.com. This episode is brought to you by our sponsors Tron, Ledn About Tron TRON is dedicated to accelerating the decentralization of the internet via blockchain technology and decentralized applications (dApps). Founded in September 2017 by H.E. Justin Sun, the TRON network has continued to deliver impressive achievements since MainNet launch in May 2018. July 2018 also marked the ecosystem integration of BitTorrent, a pioneer in decentralized web3 services boasting over 100 million monthly active users. The TRON network completed full decentralization in December 2021 and is now a community-governed DAO. | TRONDAO | Twitter | Discord | About Ledn Ledn was founded on the unshakeable conviction that digital assets have the power to democratize access to the global economy. We help you to experience the real life benefits of your Bitcoin without having to sell it. Start a savings account, take out a loan, or double your Bitcoin. For more information visit Ledn.io

In this venture capital-focused segment of The Scoop, CoinShare Chief Strategy Officer Meltem Demirors and Aglaé Ventures Managing Partner Vanessa Grellet explain which crypto verticals are oversaturated, and examine how market dynamics have changed since last year. Although the crypto industry has experienced many boom-and-bust cycles before, Demirors says “2021 will be one of the worst years for crypto venture returns.” Demirors explains that one of the reasons for this is a mismatch in block space supply and demand that was created during the proliferation of competing Layer 1 blockchains last year: “We're seeing that now with all of these overfunded L1s — there's just more block space than there is demand for block space.”Instead of mimicking existing successful projects, Grellet suggests that up-and-coming crypto projects should identify how they plan to differentiate themselves early on: “The question is, if you are a new platform coming into space or a new marketplace, what is distinctive that you bring that will allow for the end user to come to your platform rather than going to the existing incumbents?” Episode 115 of Season 4 of The Scoop was recorded live with The Block's Frank Chaparro, CoinShares CSO Meltem Demirors, and Aglaé Ventures' Managing Partner Vanessa Grellet. Listen below, and subscribe to The Scoop on Apple, Spotify, Google Podcasts, Stitcher or wherever you listen to podcasts. Email feedback and revision requests can be sent to podcast@theblockcrypto.com. This episode is brought to you by our sponsors Tron, Ledn, Athletic Greens About Tron TRON is dedicated to accelerating the decentralization of the internet via blockchain technology and decentralized applications (dApps). Founded in September 2017 by H.E. Justin Sun, the TRON network has continued to deliver impressive achievements since MainNet launch in May 2018. July 2018 also marked the ecosystem integration of BitTorrent, a pioneer in decentralized web3 services boasting over 100 million monthly active users. The TRON network completed full decentralization in December 2021 and is now a community-governed DAO. | TRONDAO | Twitter | Discord | About Ledn Ledn was founded on the unshakeable conviction that digital assets have the power to democratize access to the global economy. We help you to experience the real life benefits of your Bitcoin without having to sell it. Start a savings account, take out a loan, or double your Bitcoin. For more information visit Ledn.io About Athletic Greens Build a Foundation for Better Health. It's time to reclaim your health and arm your immune system with convenient, daily nutrition! Fill nutrient gaps, promote gut health, and support whole-body vitality with AG1. One daily serving delivers a potent blend of 9 health products—a multivitamin, minerals, probiotics, adaptogens and more—working together to help you feel like your healthiest self. For more information visit AthleticGreens.com/Scoop

Part one of this two-part special on Nouriel Roubini's new book, Megathreats, examines how the economy is facing a combination of interconnected phenomena that Roubini believes will result in a severe economic crisis. Roubini discusses some alternative assets that could gain prominence in the event of a global financial meltdown and explains why he prefers gold over bitcoin in such a situation, among other topics in his book Megathreats: Ten Dangerous Trends That Imperil Our Future, And How to Survive Them. According to Roubini, the "weaponization" of the U.S. dollar is making it less attractive as a reserve currency: “We have weaponized the dollar as a tool of national security and foreign policy… If I owe you a billion it is my problem; if I owe you a trillion it is your problem because we could default on those treasuries eventually if there is a conflict…” Similar to USD, Roubini believes other common reserve currencies such as the euro, yen, pound, and Swiss franc can be similarly weaponized, leaving gold as the only widely held reserve currency that remains a viable option: “What's the only other asset that is a liquid asset that can be a reserve currency? … What's the only one that cannot be seized if there are sanctions? It's gold — as long as you keep it in your own vault.” Although ardent bitcoin proponents claim BTC has similar properties, Roubini is dismissive of the largest cryptocurrency's potential to be used legitimately in global commerce.  “Bitcoin and other cryptocurrencies — we know they are not currencies,” Roubini says, “They are not a unit of account, nobody's pricing anything in Bitcoin, they're not scalable means of payment.”  Episode 110 of Season 4 of The Scoop was recorded live with The Block's Frank Chaparro and Roubini Macro Associates Chairman and CEO, Nouriel Roubini. Listen below, and subscribe to The Scoop on Apple, Spotify, Google Podcasts, Stitcher or wherever you listen to podcasts. Email feedback and revision requests can be sent to podcast@theblockcrypto.com. This episode is brought to you by our sponsors Tron, Ledn, Athletic Greens About Tron TRON is dedicated to accelerating the decentralization of the internet via blockchain technology and decentralized applications (dApps). Founded in September 2017 by H.E. Justin Sun, the TRON network has continued to deliver impressive achievements since MainNet launch in May 2018. July 2018 also marked the ecosystem integration of BitTorrent, a pioneer in decentralized web3 services boasting over 100 million monthly active users. The TRON network completed full decentralization in December 2021 and is now a community-governed DAO. | TRONDAO | Twitter | Discord | About Ledn Ledn was founded on the unshakeable conviction that digital assets have the power to democratize access to the global economy. We help you to experience the real life benefits of your Bitcoin without having to sell it. Start a savings account, take out a loan, or double your Bitcoin. For more information visit Ledn.io

About KevinKevin Miller is currently the global General Manager for Amazon Simple Storage Service (S3), an object storage service that offers industry-leading scalability, data availability, security, and performance. Prior to this role, Kevin has had multiple leadership roles within AWS, including as the General Manager for Amazon S3 Glacier, Director of Engineering for AWS Virtual Private Cloud, and engineering leader for AWS Virtual Private Network and AWS Direct Connect. Kevin was also Technical Advisor to the Senior Vice President for AWS Utility Computing. Kevin is a graduate of Carnegie Mellon University with a Bachelor of Science in Computer Science.Links Referenced: snark.cloud/shirt: https://snark.cloud/shirt aws.amazon.com/s3: https://aws.amazon.com/s3 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is brought to us in part by our friends at Datadog. Datadog is a SaaS monitoring and security platform that enables full-stack observability for modern infrastructure and applications at every scale. Datadog enables teams to see everything: dashboarding, alerting, application performance monitoring, infrastructure monitoring, UX monitoring, security monitoring, dog logos, and log management, in one tightly integrated platform. With 600-plus out-of-the-box integrations with technologies including all major cloud providers, databases, and web servers, Datadog allows you to aggregate all your data into one platform for seamless correlation, allowing teams to troubleshoot and collaborate together in one place, preventing downtime and enhancing performance and reliability. Get started with a free 14-day trial by visiting datadoghq.com/screaminginthecloud, and get a free t-shirt after installing the agent.Corey: Managing shards. Maintenance windows. Overprovisioning. ElastiCache bills. I know, I know. It's a spooky season and you're already shaking. It's time for caching to be simpler. Momento Serverless Cache lets you forget the backend to focus on good code and great user experiences. With true autoscaling and a pay-per-use pricing model, it makes caching easy. No matter your cloud provider, get going for free at gomomento.co/screaming. That's GO M-O-M-E-N-T-O dot co slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Right now, as I record this, we have just kicked off our annual charity t-shirt fundraiser. This year's shirt showcases S3 as the eighth wonder of the world. And here to either defend or argue the point—we're not quite sure yet—is Kevin Miller, AWS's vice president and general manager for Amazon S3. Kevin, thank you for agreeing to suffer the slings and arrows that are no doubt going to be interpreted, misinterpreted, et cetera, for the next half hour or so.Kevin: Oh, Corey, thanks for having me. And happy to do that, and really flattered for you to be thinking about S3 in this way. So more than happy to chat with you.Corey: It's absolutely one of those services that is foundational to the cloud. It was the first AWS service that was put into general availability, although the beta folks are going to argue back and forth about no, no, that was SQS instead. I feel like now that Mai-Lan handles both SQS and S3 as part of her portfolio, she is now the final arbiter of that. I'm sure that's an argument for a future day. But it's impossible to imagine cloud without S3.Kevin: I definitely think that's true. It's hard to imagine cloud, actually, with many of our foundational services, including SQS, of course, but we are—yes, we were the first generally available service with S3. And pretty happy with our anniversary being Pi Day, 3/14.Corey: I'm also curious, your own personal trajectory has been not necessarily what folks would expect. You were the general manager of Amazon Glacier, and now you're the general manager and vice president of S3. So, I've got to ask, because there are conflicting reports on this depending upon what angle you look at, are Glacier and S3 the same thing?Kevin: Yes, I was the general manager for S3 Glacier prior to coming over to S3 proper, and the answer is no, they are not the same thing. We certainly have a number of technologies where we're able to use those technologies both on S3 and Glacier, but there are certainly a number of things that are very distinct about Glacier and give us that ability to hit the ultra-low price points that we do for Glacier Deep Archive being as low as $1 per terabyte-month. And so, that definitely—there's a lot of actual ingenuity up and down the stack, from hardware to software, everywhere in between, to really achieve that with Glacier. But then there's other spots where S3 and Glacier have very similar needs, and then, of course, today many customers use Glacier through S3 as a storage class in S3, and so that's a great way to do that. So, there's definitely a lot of shared code, but certainly, when you get into it, there's [unintelligible 00:04:59] to both of them.Corey: I ran a number of obnoxiously detailed financial analyses, and they all came away with, unless you have a very specific very nuanced understanding of your data lifecycle and/or it is less than 30 or 60 days depending upon a variety of different things, the default S3 storage class you should be using for virtually anything is Intelligent Tiering. That is my purely economic analysis of it. Do you agree with that? Disagree with that? And again, I understand that all of these storage classes are like your children, and I am inviting you to tell me which one of them is your favorite, but I'm absolutely prepared to do that.Kevin: Well, we love Intelligent Tiering because it is very simple; customers are able to automatically save money using Intelligent Tiering for data that's not being frequently accessed. And actually, since we launched it a few years ago, we've already saved customers more than $250 million using Intelligent Tiering. So, I would say today, it is our default recommendation in almost every case. I think that the cases where we would recommend another storage class as the primary storage class tend to be specific to the use case where—and particularly for use cases where customers really have a good understanding of the access patterns. And we saw some customers do for their certain dataset, they know that it's going to be heavily accessed for a fixed period of time, or this data is actually for archival, it'll never be accessed, or very rarely if ever access, just maybe in an emergency.And those kinds of use cases, I think actually, customers are probably best to choose one of the specific storage classes where they're, sort of, paying that the lower cost from day one. But again, I would say for the vast majority of cases that we see, the data access patterns are unpredictable and customers like the flexibility of being able to very quickly retrieve the data if they decide they need to use it. But in many cases, they'll save a lot of money as the data is not being accessed, and so, Intelligent Tiering is a great choice for those cases.Corey: I would take it a step further and say that even when customers believe that they are going to be doing a deeper analysis and they have a better understanding of their data flow patterns than Intelligent Tiering would, in practice, I see that they rarely do anything about it. It's one of those things where they're like, “Oh, yeah, we're going to set up our own lifecycle policies real soon now,” whereas, just switch it over to Intelligent Tiering and never think about it again. People's time is worth so much more than the infrastructure they're working on in almost every case. It doesn't seem to make a whole lot of sense unless you have a very intentioned, very urgent reason to go and do that stuff by hand in most cases.Kevin: Yeah, that's right. I think I agree with you, Corey. And certainly, that is the recommendation we lead with customers.Corey: In previous years, our charity t-shirt has focused on other areas of AWS, and one of them was based upon a joke that I've been telling for a while now, which is that the best database in the world is Route 53 and storing TXT records inside of it. I don't know if I ever mentioned this to you or not, but the first iteration of that joke was featuring around S3. The challenge that I had with it is that S3 Select is absolutely a thing where you can query S3 with SQL which I don't see people doing anymore because Athena is the easier, more, shall we say, well-articulated version of all of that. And no, no, that joke doesn't work because it's actually true. You can use S3 as a database. Does that statement fill you with dread? Regret? Am I misunderstanding something? Or are you effectively running a giant subversive database?Kevin: Well, I think that certainly when most customers think about a database, they think about a collection of technology that's applied for given problems, and so I wouldn't count S3 as providing the whole range of functionality that would really make up a database. But I think that certainly a lot of the primitives and S3 Select as a great example of a primitive are available in S3. And we're looking at adding, you know, additional primitives going forward to make it possible to, you know, to build a database around S3. And as you see, other AWS services have done that in many ways. For example, obviously with Amazon Redshift having a lot of capability now to just directly access and use data in S3 and make that a super seamless so that you can then run data warehousing type queries on top of S3 and on top of your other datasets.So, I certainly think it's a great building block. And one other thing I would actually just say that you may not know, Corey, is that one of the things over the last couple of years we've been doing a lot more with S3 is actually working to directly contribute improvements to open-source connector software that uses S3, to make available automatically some of the performance improvements that can be achieved either using both the AWS SDK, and also using things like S3 Select. So, we started with a few of those things with Select; you're going to see more of that coming, most likely. And some of that, again, the idea there as you may not even necessarily know you're using Select, but when we can identify that it will improve performance, we're looking to be able to contribute those kinds of improvements directly—or we are contributing those directly to those open-source packages. So, one thing I would definitely recommend customers and developers do is have a capability of sort of keeping that software up-to-date because although it might seem like those are sort of one-and-done kind of software integrations, there's actually almost continuous improvement now going on, and around things like that capability, and then others we come out with.Corey: What surprised me is just how broadly S3 has been adopted by a wide variety of different clients' software packages out there. Back when I was running production environments in anger, I distinctly remember in one Ubuntu environment, we wound up installing a specific package that was designed to teach apt how to retrieve packages and its updates from S3, which was awesome. I don't see that anymore, just because it seems that it is so easy to do it now, just with the native features that S3 offers, as well as an awful lot of software under the hood has learned to directly recognize S3 as its own thing, and can react accordingly.Kevin: And just do the right thing. Exactly. No, we certainly see a lot of that. So that's, you know—I mean, obviously making that simple for end customers to use and achieve what they're trying to do, that's the whole goal.Corey: It's always odd to me when I'm talking to one of my clients who is looking to understand and optimize their AWS bill to see outliers in either direction when it comes to S3 itself. When they're driving large S3 bills as in a majority of their spend, it's, okay, that is very interesting. Let's dive into that. But almost more interesting to me is when it is effectively not being used at all. When, oh, we're doing everything with EBS volumes or EFS.And again, those are fine services. I don't have any particular problem with them anymore, but the problem I have is that the cloud long ago took what amounts to an economic vote. There's a tax savings for storing data in an object store the way that you—and by extension, most of your competitors—wind up pricing this, versus the idea of on a volume basis where you have to pre-provision things, you don't get any form of durability that extends beyond the availability zone boundary. It just becomes an awful lot of, “Well, you could do it this way. But it gets really expensive really quickly.”It just feels wild to me that there is that level of variance between S3 just sort of raw storage basis, economically, as well as then just the, frankly, ridiculous levels of durability and availability that you offer on top of that. How did you get there? Was the service just mispriced at the beginning? Like oh, we dropped to zero and probably should have put that in there somewhere.Kevin: Well, no, I wouldn't call it mispriced. I think that the S3 came about when we took a—we spent a lot of time looking at the architecture for storage systems, and knowing that we wanted a system that would provide the durability that comes with having three completely independent data centers and the elasticity and capability where, you know, customers don't have to provision the amount of storage they want, they can simply put data and the system keeps growing. And they can also delete data and stop paying for that storage when they're not using it. And so, just all of that investment and sort of looking at that architecture holistically led us down the path to where we are with S3.And we've definitely talked about this. In fact, in Peter's keynote at re:Invent last year, we talked a little bit about how the system is designed under the hood, and one of the thing you realize is that S3 gets a lot of the benefits that we do by just the overall scale. The fact that it is—I think the stat is that at this point more than 10,000 customers have data that's stored on more than a million hard drives in S3. And that's how you get the scale and the capability to do is through massive parallelization. Where customers that are, you know, I would say building more traditional architectures, those are inherently typically much more siloed architectures with a relatively small-scale overall, and it ends up with a lot of resource that's provisioned at small-scale in sort of small chunks with each resource, that you never get to that scale where you can start to take advantage of the some is more than the greater of the parts.And so, I think that's what the recognition was when we started out building S3. And then, of course, we offer that as an API on top of that, where customers can consume whatever they want. That is, I think, where S3, at the scale it operates, is able to do certain things, including on the economics, that are very difficult or even impossible to do at a much smaller scale.Corey: One of the more egregious clown-shoe statements that I hear from time to time has been when people will come to me and say, “We've built a competitor to S3.” And my response is always one of those, “Oh, this should be good.” Because when people say that, they generally tend to be focusing on one or maybe two dimensions that doesn't work for a particular use case as well as it could. “Okay, what was your story around why this should be compared to S3?” “Well, it's an object store. It has full S3 API compatibility.” “Does it really because I have to say, there are times where I'm not entirely convinced that S3 itself has full compatibility with the way that its API has been documented.”And there's an awful lot of magic that goes into this too. “Okay, great. You're running an S3 competitor. Great. How many buildings does it live in?” Like, “Well, we have a problem with the s at the end of that word.” It's, “Okay, great. If it fits on my desk, it is not a viable S3 competitor. If it fits in a single zip code, it is probably not a viable S3 competitor.” Now, can it be an object store? Absolutely. Does it provide a new interface to some existing data someone might have? Sure why not. But I think that, oh, it's S3 compatible, is something that gets tossed around far too lightly by folks who don't really understand what it is that drives S3 and makes it special.Kevin: Yeah, I mean, I would say certainly, there's a number of other implementations of the S3 API, and frankly we're flattered that customers recognize and our competitors and others recognize the simplicity of the API and go about implementing it. But to your point, I think that there's a lot more; it's not just about the API, it's really around everything surrounding S3 from, as you mentioned, the fact that the data in S3 is stored in three independent availability zones, all of which that are separated by kilometers from each other, and the resilience, the automatic failover, and the ability to withstand an unlikely impact to one of those facilities, as well as the scalability, and you know, the fact that we put a lot of time and effort into making sure that the service continues scaling with our customers need. And so, I think there's a lot more that goes into what is S3. And oftentimes just in a straight-up comparison, it's sort of purely based on just the APIs and generally a small set of APIs, in addition to those intangibles around—or not intangibles, but all of the ‘-ilities,' right, the elasticity and the durability, and so forth that I just talked about. In addition to all that also, you know, certainly what we're seeing for customers is as they get into the petabyte and tens of petabytes, hundreds of petabytes scale, their need for the services that we provide to manage that storage, whether it's lifecycle and replication, or things like our batch operations to help update and to maintain all the storage, those become really essential to customers wrapping their arms around it, as well as visibility, things like Storage Lens to understand, what storage do I have? Who's using it? How is it being used?And those are all things that we provide to help customers manage at scale. And certainly, you know, oftentimes when I see claims around S3 compatibility, a lot of those advanced features are nowhere to be seen.Corey: I also want to call out that a few years ago, Mai-Lan got on stage and talked about how, to my recollection, you folks have effectively rebuilt S3 under the hood into I think it was 235 distinct microservices at the time. There will not be a quiz on numbers later, I'm assuming. But what was wild to me about that is having done that for services that are orders of magnitude less complex, it absolutely is like changing the engine on a car without ever slowing down on the highway. Customers didn't know that any of this was happening until she got on stage and announced it. That is wild to me. I would have said before this happened that there was no way that would have been possible except it clearly was. I have to ask, how did you do that in the broad sense?Kevin: Well, it's true. A lot of the underlying infrastructure that's been part of S3, both hardware and software is, you know, you wouldn't—if someone from S3 in 2006 came and looked at the system today, they would probably be very disoriented in terms of understanding what was there because so much of it has changed. To answer your question, the long and short of it is a lot of testing. In fact, a lot of novel testing most recently, particularly with the use of formal logic and what we call automated reasoning. It's also something we've talked a fair bit about in re:Invent.And that is essentially where you prove the correctness of certain algorithms. And we've used that to spot some very interesting, the one-in-a-trillion type cases that S3 scale happens regularly, that you have to be ready for and you have to know how the system reacts, even in all those cases. I mean, I think one of our engineers did some calculations that, you know, the number of potential states for S3, sort of, exceeds the number of atoms in the universe or something so crazy. But yet, using methods like automated reasoning, we can test that state space, we can understand what the system will do, and have a lot of confidence as we begin to swap, you know, pieces of the system.And of course, nothing in S3 scale happens instantly. It's all, you know, I would say that for a typical engineering effort within S3, there's a certain amount of effort, obviously, in making the change or in preparing the new software, writing the new software and testing it, but there's almost an equal amount of time that goes into, okay, and what is the process for migrating from System A to System B, and that happens over a timescale of months, if not years, in some cases. And so, there's just a lot of diligence that goes into not just the new systems, but also the process of, you know, literally, how do I swap that engine on the system. So, you know, it's a lot of really hard working engineers that spent a lot of time working through these details every day.Corey: I still view S3 through the lens of it is one of the easiest ways in the world to wind up building a static web server because you basically stuff the website files into a bucket and then you check a box. So, it feels on some level though, that it is about as accurate as saying that S3 is a database. It can be used or misused or pressed into service in a whole bunch of different use cases. What have you seen from customers that has, I guess, taught you something you didn't expect to learn about your own service?Kevin: Oh, I'd say we have those [laugh]  meetings pretty regularly when customers build their workloads and have unique patterns to it, whether it's the type of data they're retrieving and the access pattern on the data. You know, for example, some customers will make heavy use of our ability to do [ranged gets 00:22:47] on files and [unintelligible 00:22:48] objects. And that's pretty good capability, but that can be one where that's very much dependent on the type of file, right, certain files have structure, as far as you know, a header or footer, and that data is being accessed in a certain order. Oftentimes, those may also be multi-part objects, and so making use of the multi-part features to upload different chunks of a file in parallel. And you know, also certainly when customers get into things like our batch operations capability where they can literally write a Lambda function and do what they want, you know, we've seen some pretty interesting use cases where customers are running large-scale operations across, you know, billions, sometimes tens of billions of objects, and this can be pretty interesting as far as what they're able to do with them.So, for something is sort of what you might—you know, as simple and basics, in some sense, of GET and PUT API, just all the capability around it ends up being pretty interesting as far as how customers apply it and the different workloads they run on it.Corey: So, if you squint hard enough, what I'm hearing you tell me is that I can view all of this as, “Oh, yeah. S3 is also compute.” And it feels like that as a fast-track to getting a question wrong on one of the certification exams. But I have to ask, from your point of view, is S3 storage? And whether it's yes or no, what gets you excited about the space that it's in?Kevin: Yeah well, I would say S3 is not compute, but we have some great compute services that are very well integrated with S3, which excites me as well as we have things like S3 Object Lambda, where we actually handle that integration with Lambda. So, you're writing Lambda functions, we're executing them on the GET path. And so, that's a pretty exciting feature for me. But you know, to sort of take a step back, what excites me is I think that customers around the world, in every industry, are really starting to recognize the value of data and data at large scale. You know, I think that actually many customers in the world have terabytes or more of data that sort of flows through their fingers every day that they don't even realize.And so, as customers realize what data they have, and they can capture and then start to analyze and make ultimately make better business decisions that really help drive their top line or help them reduce costs, improve costs on whether it's manufacturing or, you know, other things that they're doing. That's what really excites me is seeing those customers take the raw capability and then apply it to really just to transform how they not just how their business works, but even how they think about the business. Because in many cases, transformation is not just a technical transformation, it's people and cultural transformation inside these organizations. And that's pretty cool to see as it unfolds.Corey: One of the more interesting things that I've seen customers misunderstand, on some level, has been a number of S3 releases that focus around, “Oh, this is for your data lake.” And I've asked customers about that. “So, what's your data lake strategy?” “Well, we don't have one of those.” “You have, like, eight petabytes and climbing in S3? What do you call that?” It's like, “Oh, yeah, that's just a bunch of buckets we dump things into. Some are logs of our assets and the rest.” It's—Kevin: Right.Corey: Yeah, it feels like no one thinks of themselves as having anything remotely resembling a structured place for all of the data that accumulates at a company.Kevin: Mm-hm.Corey: There is an evolution of people learning that oh, yeah, this is in fact, what it is that we're doing, and this thing that they're talking about does apply to us. But it almost feels like a customer communication challenge, just because, I don't know about you, but with my legacy AWS account, I have dozens of buckets in there that I don't remember what the heck they're for. Fortunately, you folks don't charge by the bucket, so I can smile, nod, remain blissfully ignorant, but it does make me wonder from time to time.Kevin: Yeah, no, I think that what you hear there is actually pretty consistent with what the reality is for a lot of customers, which is in distributed organizations, I think that's bound to happen, you have different teams that are working to solve problems, and they are collecting data to analyze, they're creating result datasets and they're storing those datasets. And then, of course, priorities can shift, and you know, and there's not necessarily the day-to-day management around data that we might think would be expected. I feel [we 00:26:56] sort of drew an architecture on a whiteboard. And so, I think that's the reality we are in. And we will be in, largely forever.I mean, I think that at a smaller-scale, that's been happening for years. So, I think that, one, I think that there's a lot of capability just being in the cloud. At the very least, you can now start to wrap your arms around it, right, where used to be that it wasn't even possible to understand what all that data was because there's no way to centrally inventory it well. In AWS with S3, with inventory reports, you can get a list of all your storage and we are going to continue to add capability to help customers get their arms around what they have, first off; understand how it's being used—that's where things like Storage Lens really play a big role in understanding exactly what data is being accessed and not. We're definitely listening to customers carefully around this, and I think when you think about broader data management story, I think that's a place that we're spending a lot of time thinking right now about how do we help customers get their arms around it, make sure that they know what's the categorization of certain data, do I have some PII lurking here that I need to be very mindful of?And then how do I get to a world where I'm—you know, I won't say that it's ever going to look like the perfect whiteboard picture you might draw on the wall. I don't think that's really ever achievable, but I think certainly getting to a point where customers have a real solid understanding of what data they have and that the right controls are in place around all that data, yeah, I think that's directionally where I see us heading.Corey: As you look around how far the service has come, it feels like, on some level, that there were some, I guess, I don't want to say missteps, but things that you learned as you went along. Like, back when the service was in beta, for example, there was no per-request charge. To my understanding that was changed, in part because people were trying to use it as a file system, and wow, that suddenly caused a tremendous amount of load on some of the underlying systems. You originally launched with a BitTorrent endpoint as an option so that people could download through peer-to-peer approaches for large datasets and turned out that wasn't really the way the internet evolved, either. And I'm curious, if you were to have to somehow build this off from scratch, are there any other significant changes you would make in how the service was presented to customers in how people talked about it in the early days? Effectively given a mulligan, what would you do differently?Kevin: Well, I don't know, Corey, I mean, just given where it's grown to in macro terms, you know, I definitely would be worried taking a mulligan, you know, that I [laugh] would change the sort of the overarching trajectory. Certainly, I think there's a few features here and there where, for whatever reason, it was exciting at the time and really spoke to what customers at the time were thinking, but over time, you know, sort of quickly those needs move to something a little bit different. And, you know, like you said things like the BitTorrent support is one where, at some level, it seems like a great technical architecture for the internet, but certainly not something that we've seen dominate in the way things are done. Instead, you know, we've largely kind of have a world where there's a lot of caching layers, but it still ends up being largely client-server kind of connections. So, I don't think I would do a—I certainly wouldn't do a mulligan on any of the major functionality, and I think, you know, there's a few things in the details where obviously, we've learned what really works in the end. I think we learned that we wanted bucket names to really strictly conform to rules for DNS encoding. So, that was the change that was made at some point. And we would tweak that, but no major changes, certainly.Corey: One subject of some debate while we were designing this year's charity t-shirt—which, incidentally, if you're listening to this, you can pick up for yourself at snark.cloud/shirt—was the is S3 itself dependent upon S3? Because we know that every other service out there is as well, but it is interesting to come up with an idea of, “Oh, yeah. We're going to launch a whole new isolated region of S3 without S3 to lean on.” That feels like it's an almost impossible bootstrapping problem.Kevin: Well, S3 is not dependent on S3 to come up, and it's certainly a critical dependency tree that we look at and we track and make sure that we'd like to have an acyclic graph as we look at dependencies.Corey: That is such a sophisticated way to say what I learned the hard way when I was significantly younger and working in production environments: don't put the DNS servers needed to boot the hypervisor into VMs that require a working hypervisor. It's one of those oh, yeah, in hindsight, that makes perfect sense, but you learn it right after that knowledge really would have been useful.Kevin: Yeah, absolutely. And one of the terms we use for that, as well as is the idea of static stability, or that's one of the techniques that can really help with isolating a dependency is what we call static stability. We actually have an article about that in the Amazon Builder Library, which there's actually a bunch of really good articles in there from very experienced operations-focused engineers in AWS. So, static stability is one of those key techniques, but other techniques—I mean, just pure minimization of dependencies is one. And so, we were very, very thoughtful about that, particularly for that core layer.I mean, you know, when you talk about S3 with 200-plus microservices, or 235-plus microservices, I would say not all of those services are critical for every single request. Certainly, a small subset of those are required for every request, and then other services actually help manage and scale the kind of that inner core of services. And so, we look at dependencies on a service by service basis to really make sure that inner core is as minimized as possible. And then the outer layers can start to take some dependencies once you have that basic functionality up.Corey: I really want to thank you for being as generous with your time as you have been. If people want to learn more about you and about S3 itself, where should they go—after buying a t-shirt, of course.Kevin: Well, certainly buy the t-shirt. First, I love the t-shirts and the charity that you work with to do that. Obviously, for S3, it's aws.amazon.com/s3. And you can actually learn more about me. I have some YouTube videos, so you can search for me on YouTube and kind of get a sense of myself.Corey: We will put links to that into the show notes, of course. Thank you so much for being so generous with your time. I appreciate it.Kevin: Absolutely. Yeah. Glad to spend some time. Thanks for the questions, Corey.Corey: Kevin Miller, vice president and general manager for Amazon S3. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, ignorant comment talking about how your S3 compatible service is going to blow everyone's socks off when it fails.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Most artists want career growth and they want it fast — sometimes to a fault. This is where Curren$y is an outlier. From the jump, Curren$y set out to grow both his career and fanbase slowly but steadily. He successfully did that and it's a reason why he's not only stayed in the rap game for almost twenty years, but is now still earning more money, and at a career point where most of his peers coming up have already fizzled out.Curren$y and his longtime manager, Mousa, joined me on this week's episode to explain how zigging when others zagged contributed to their career longevity. One instance is leaving his hometown Cash Money Records label to create their own, Jet Life. The two have been able to morph the brand into a full-on business empire that now includes apparel, athlete management, products, and more verticals on top of the music label. The duo built Jet Life on the back of touring and being true to their audience. To do so, they turned down more brand partnerships they can remember and even music festival appearances at times — no matter how good the bag was for each. These trend-bucking moves were covered at length in our interview. Here's what we covered:[3:15] New Orleans folks are immune to heat[4:44] Mousa and Curren$y relationship began in 2005[8:49] Growing Jet Life business beyond a record label[11:45] Turning down non-authentic business opportunities[15:59] Emphasizing touring early in Curren$y's career[19:21] Releasing an EP as an NFT[23:52] Curren$y's take on streaming farms[29:47] Macro-view of Jet Life revenue streams[34:47] Touring is cornerstone of Jet Life business[37:08] Performing on own shows vs. music festivals[43:48] Festival money goes to sports car dealership[45:16] Curren$y's partnership with NASCAR (and problems with Coca-Cola)[50:37] What's the secret to a great artist-manager relationship? Listen: Apple Podcasts | Spotify | SoundCloud | Stitcher | Overcast | Amazon | Google Podcasts | Pocket Casts | RSSHost: Dan Runcie, @RuncieDan, trapital.coGuests: Mousa Hamdan & Curren$y, @MOUSA504 & @spitta_andretti  Sponsors: MoonPay is the leader in web3 infrastructure. They have partnered with Timbaland, Snoop Dogg, and many more. To learn more, visit moonpay.com/trapital Enjoy this podcast? Rate and review the podcast here! ratethispodcast.com/trapital Trapital is home for the business of hip-hop. Gain the latest insights from hip-hop's biggest players by reading Trapital's free weekly memo.TRANSCRIPTION[00:00:00] Curren$y: You can always expand and try new things, but if it feels wrong on the core, then you're setting yourself up. We never made a move like that. No matter what deal comes across the table 'cause he's money first. But he'll tell the people, the check writer like, man, just let me talk to bro. Because at the end of the day, he's going to hear me say it's half a million dollars, but he might say it's a boring job and he might not want to do it. [00:00:32] Dan Runcie: Hey, welcome to The Trapital podcast. I'm your host and the founder of Trapital, Dan Runcie. This podcast is your place to gain insights from executives in music, media, entertainment, and more, who are taking hip-hop culture to the next level. [00:00:54] Dan Runcie: Listen, you're going to love today's episode. It is with one of the most successful independent artists in the game and his longtime manager. We got Curren$y and we got Mousa. If you've been following Curren$y's journey for a while, you know that he was originally on No Limit Records 20 years ago. He left the record label. He then went to join Young Money. He was a little early on the Young Money Train, but he ended up leaving the record label before Nicki and before Drake blew up and he started his own. He started Jet Life, and he's been building up his career as an independent artist, and it's been great to see how he has navigated both how he releases music and also how he approached his business overall. And that was a big focus of this episode. We talked about his strategy for releasing music, and Curren$y is someone that is very prolific in terms of the amount of music that he puts out, but it also gets him plenty of opportunities to be able to go on tour, to be able to have several other business ventures that they have through Jet Life and through other areas. We talked about what they're doing in cannabis as well. We talked about the nightclub that they have, the apparel business, and a whole lot more. We also talked about a few partnerships that you may be surprised by, but I still think that fit well within the ethos for what Jet Life is and what Curren$y is trying to build. We even talked about some of the movie deals and opportunities that Curren$y had turned down. I don't want to spoil it. It's a really good one, but this was a really fascinating conversation, is also been great to just see how long these two have stuck together. If you're a big fan of this podcast, these are the type of episodes that you come for. Hope you enjoy it as much as I did. Here's my chat with Curren$y and Mousa. [00:02:41] Dan Runcie: All right. Today we're joined by the duo themselves. We got Curren$y and we got Mousa here, the artist-manager combination. How are you guys doing? [00:02:49] Curren$y: Man, we can't complain. The weather is nice outside and it is been pretty bad out here in Orleans. It's been a hundred degrees and raining every day, but right now it's sunny, 86 degrees, you know what I'm saying? I got long sleeves on, top down, having a good day. I can't complain. [00:03:07] Dan Runcie: See, that's the one thing about folks I know from New Orleans, like it could be 86 degrees and y'all are still in long sleeves. Y'all are still in hoodies. [00:03:15] Curren$y: Well, it is, well, because it is the heat, we're already adjusting. It's just hot in here. So now we've gone more fashion-forward, bro. It's like, just fuck it, bro. Wear what you want to wear because it's still going to be 190 degrees no matter what. So just go for it. I don't really condone that lifestyle unless you have a car. A lot of my younger brothers I see walking up and down the street, and they definitely look like they're about to commit crimes because it's a hundred degrees and they got on the hoodie and I'm, like, weary of, I'm like, hold on, you know what I'm saying, because, fuck, that don't make no sense. You dressed for action. But if you are in the car, you are in the office, you are in the studio. That's where that look really originated. People always tell me, II'm dressed like that forever, but it's been because most of my life has been like tour bus, studio, even when it wasn't me, I was like a little guy on Masterpiece bus. It was 60 degrees, you know what I'm saying? And these big mansions, it's cold as shit. So I just grew acclimated to dressing like that. I think I might have spearheaded that. I honestly, I think that I may have spearheaded that, but what haven't we spearheaded over here, you know?[00:04:25] Dan Runcie: It's true, especially folks at New Orleans, folks like y'all are trendsetters. And one of the things that I feel like sets y'all apart is that you've been doing this for so long, and you've been doing this for so long together. I mean, Mousa, you've been managing Curren$y now for, since '05, right? I know you do 'em before, but you started managing, like, '05, right?[00:04:44] Mousa Hamdan: We're friends before, but definitely since '05, since he joined in with Lil Wayne, with Young Money, Cash Money. So I think that's when he brought me on and asked me to come on as his manager. [00:04:53] Curren$y: Yep. Yeah. [00:04:54] Mousa Hamdan: And you know...[00:04:55] Curren$y: As soon as there was business to manage. [00:04:58] Mousa Hamdan: Right. [00:04:58] Curren$y: You know, right? While I was just slinging t-shirts, like ordering 28 t-shirts on a month, pressing CDs upstairs at my apartment, that was easy to do. When it began to grow and I saw, like, my two homes wasn't going to be enough to handle it, you know, what could I do but reach out to the one homie who I knew forever who don't want to smoke no weed with me, who don't want to get drunk with me, you know what I'm saying? Like, who's just like totally, his high is the business, deals closed and stuff gone successfully is him having a drink, you know what I'm saying? So it worked. It works like that.[00:05:36] Mousa Hamdan: Definitely. I like achieving goals. You know, I'm a goal seeker. And once you achieve one goal, set another one, you know? And that's my inspiration is to see how big we could really take this Jet Life, how, you know, how big deal this will be, and how long we can make it last. I mean, I thought about this morning, I was talking to one of my other homies, I was like, we've been in this game a minute, bro. Like, and he was like, look, I've been home for a little while and y'all been doing this a long time. So I say, yeah, definitely, but we not done, you know. We're nowhere near done. We really just starting, we really starting to grow even more now. [00:06:12] Curren$y: That's crazy to say that, and that's really the truth, to be here in the game. Like, Jet Life, we're like over a decade, and each year it just gets bigger. That's really what you want. It's not a big, hasn't been just one big explosion. It's a slow burn. But it is guaranteed. And we've always grown. A lot of times you see people struggling, like, not to lose ground in the game, you know, and stay relevant. And that's never been a problem with us because we've been blessed to be able to, like, generate or, like, create our own world, you know what I'm saying? And the people who listen to our music or who dress, some people dress only in Jet Life apparel. And it is because they don't give a fuck about nothing else, you know what I'm saying? They've had their time to see what the world had to offer, and they saw that ours was just uncompromised. So they lend themselves to it a hundred percent. And that's been enough to sustain, like, the lifestyle that we have. And the people that support us, they like to pass by the Jet Life store just to see what cars we might have outside. So they continue to support us because now we're going to park more and more shit. Like, they the ones who help us do it, you know? So it's good. It's good. [00:07:27] Mousa Hamdan: It definitely is. It's really a lifestyle, you know? I think it's, you know, from the beginning I remember, Curren$y said in interviews as well as told me directly, like, you know, his vision of seeing how Jet Life and how he wanted to grow. He always said it was like a balloon. And I listened, I heard that, and I was like, he's right. He's like, you could either, you could blow air in it fast and it's going to blow big and then it's going to explode and it's over. Or you could blow in it slow and it's going to slowly blow. [00:07:57] Curren$y: Yeah. Fucking right. [00:07:58] Mousa Hamdan: Then you show the longevity. And that's what we did. We're blowing it slow. [00:08:02] Curren$y: Yep.[00:08:03] Mousa Hamdan: But look at us. We're still here. There's a lot of people that we saw that came before us and during us who we feel like, oh yeah, they got the light quick and they blew up fast. But then what happened? And you know, they're not around no more.[00:08:16] Curren$y: Something explodes, it ceases to exist.[00:08:19] Mousa Hamdan: It's done.[00:08:20] Curren$y: I've never seen anything, you know what I'm saying, explode that still had it ever, you know? [00:08:26] Dan Runcie: Right, oh yeah. You know, and I feel like with y'all, specifically, you're able to see the trajectory. You're able to see everything that you've accomplished, too, because I look at Jet Life, and it started as the imprint for your record label, but now you have your apparel, you also have the other businesses you have. How would you describe the current businesses? What are the current things under Jet Life right now? [00:08:49] Mousa Hamdan: Well, we got, of course, like you said, it started with records, Jet Life Records. And then it went to, we started doing tour merch, which grew into Jet Life Apparel. We were in now Jet Life Athletics. So we started to do deals with managing athletes and growing that brand. Then of course, we've other stuff that's not necessarily labeled Jet Life, but we've opened up a nightclub in New Orleans, so so that's something that's coming. [00:09:16] Curren$y: We got a big footprint in the cannabis community. We got a couple of other startups, like a coffee shop and a cereal bar we're going to launch. We already have two films out, so, I mean, if you want to say Jet Life Films is in existence, that is true. It's so much stuff that we do, but the circle is so tight, like, nobody's going to tell the other one. Like, bro, you realize what we doing because we are still in the midst of doing it. Like, an outside person would have to come in and really show us how many businesses and what's all under the umbrella 'cause we really just wake up and try to, like, just make sure we make something happen, you know, every day. If you want to label it and put a name on it, then, it was news to me, right now just listening to how much stuff we have going on. [00:10:04] Mousa Hamdan: We forgot Starting Line Hobbies.[00:10:06] Curren$y: Yeah, we got hobby shop bro, like that. See? So the more you sit down… [00:10:11] Mousa Hamdan: We forget some of the business. But they exist and they're profitable, right?[00:10:16] Curren$y: He's got an auto body shop, it's still in existence. That's really where a lot of it comes from, his whole foray into it all was being able to survive if one thing fell down. Even though the music was the one that paved the way and drew the attention, the industry is fickle. So you see people like, we see them rise and you think they going to build this whole empire, they end up with a warehouse full of shit. They can't move bobbleheads of themselves. Nobody wants t-shirts, nobody wants home furnishing. Nobody wants it, fucking goes that way, you know what I'm saying? And we've been blessed to like, now we got two or three warehouses, you know what I'm saying? But we're moving the shit, you know. So it's just about staying true and not, we never really tried to do too much, nothing outside of what felt right to us. You can always expand and try new things, but if it feels wrong on the core, then you're setting yourself up. We never made a move like that. No matter what deal comes across the table 'cause he's money first. But he'll tell the people, the check writer like, man, just let me talk to bro. Because at the end of the day, he's going to hear me say it's half a million dollars, but he might say it's a boring job and he might not want to do it even though it's half a million dollars. So he'll just check with me, you know what I'm saying? We probably go and do the 'shit anyway 'cause it's half a million dollars. But he checks with me because in my heart of hearts, I might want to say no, but I got a kid and shit. [00:11:45] Mousa Hamdan: I'll definitely ask him. Do you want to do this though? [00:11:49] Curren$y: Yeah. And I got respect for him for doing that. The fact that he compromised his money mentality that asks me that much, gives me the strength to be able to say, you know what, fuck it, bro, you gave, I'll give. I'm going to come and do this shit, you know what I'm saying? And then lo and behold everybody wins, you know? [00:12:07] Dan Runcie: Yeah. What's an example of something that you have turned down? Like, Mousa, 'cause it sounds like you're the one that's seeing the things and you're thinking about, oh, this is the bag, but is this something that fits with the Jet Life lifestyle?[00:12:18] Curren$y: There's a lot of those, like, TV shit that'll come across, you know what I'm saying? I hope that he knows, I don't care. So he would say, I'm going to jump out in front of you, like, you don't see because these people still come up with more and more ideas. And eventually, they might put, they might table something that we want to pick up. But we've slammed them because it's like, bro, you know, just looking at something where they say, well, he can say it in his own words, but the way they phrase it makes me like, I'll never put this in my own words, I don't want to fucking do it. You know, just fuck it, you know what I'm saying? Or like post, they'll try, you know, they'll pay you for social media stuff just to say you like something or you can't wait for something to fucking hit the theaters. And I'm like, you know what? Fuck no. I don't want to say that. Because as soon as I post this, my fucking true audience is going to say, you know, how much did you get, bro? They'll say shit like that. I don't want to play them like that. [00:13:15] Mousa Hamdan: Yeah. I think we've known each other long enough and I know his answers on some things. Some things I won't even bring to him.[00:13:22] Curren$y: For sure. [00:13:23] Mousa Hamdan: You know, we had some stuff like, you know, I'll be honest with you, like, you know, media companies that come and say, well, you know, let me post this on your page or do this, that, and the others, and it's clickbait. And he was like, nah, bro, I don't want my fans clicking on that. [00:13:37] Curren$y: Yeah, I don't want that. I'm the one who have to answer for this shit.[00:13:41] Mousa Hamdan: I don't care how much it is. And the fans aren't crazy. They'll be like, Curren$y, that shit was clickbait, bro. [00:13:46] Curren$y: They're like, what? Or you had to, bro? Like, I have all that kind of shit. So I'm just like, let's save the company who wants to pay us the embarrassment of when they realized this was not organic and it didn't cross over. Like, now they won't want to spend any money. They may not want to spend money with us later on, on something that might actually work, you know? So it's just better to just say, you know, it is better to protect yourself that way. You end up in the long run, you still make that money. A few times people have double-backed because they realize, you know what, that was kind of lame. I can't believe we asked them to do that shit. And then they come back with something way dope after they've researched me, you know? 'Cause immediately you do a Google search and you are like, all right, cool. We'll get him to do the new weed spray. Let's get him to endorse this new air freshener that kills the weeds, man. Like, bro, the fuck? Like, I'm not even living like that. I'm actually a boss and I don't have to conceal the weed smell in my fucking life, you know what I'm saying? Like, I'm not promoting shit. [00:14:46] Dan Runcie: I'm even come to you with a deal like that though, knowing you. [00:14:50] Mousa Hamdan: Yeah, yeah. They'll bring all type of deals, bro. They'll try and get you out of character if you let them. You know, they'll push the button. [00:14:57] Curren$y: But it feels like trolling a lot of the time. Like, are they trying to see if I would do this, you know what I'm saying?[00:15:03] Dan Runcie: Right. [00:15:03] Mousa Hamdan: I don't think they understand that he's not saying he's true to his lifestyle. He is actually true to it. He's not going to do anything that's going to bend. [00:15:11] Curren$y: It's not about money. We got enough pots on the stove. It's a six-burner stove. And we have pots with food and all of them are cooking, you know what I'm saying? So when somebody comes with the bullshit, it's like, all right, let's just go dip in this, right, quick. You know, like I I've done that with music, when I feel like, it is just sometimes I get a little down on myself just based on the climate of music, you know? And I'll fall back and maybe I'll just come up here and we'll just make a whole collection of clothes at that time, you know? And we were able to keep the lights on and shit through the apparel. If I said fuck it from here on end, you know what I'm saying? But it just so happens, like, I get my win and it is fun again, and I want to do it. You know, so we're lucky as shit.[00:15:59] Dan Runcie: That makes sense, yeah. It's a good position to be in, right? You understand your brand, you understand what makes sense. You're only going to do certain types of deals. And I feel like this goes back to the way that you just go about this industry overall, right? You were early in terms of, let me put out my music and if people get it for free, they may get it for free, but let me go make the money on tour. Let me go make the money with these other business interests. [00:16:24] Curren$y: Yeah, because I mean, it's, shrinkage. It didn't matter how much music, like, what you do, how much you put behind the budget and what the labels do and all this shit. These people were just, our music was being stolen. This was during the time of, like, manufacturing jewel cases and all this shit that the company had to do, so that affected how much money they could give you. And then at the end of the day, everybody had the album a week before any damn, you know? So you can't feed your family like that. But what you can do, and what I did do is, and also when I did that, it was out of necessity. I didn't have no money to pay everybody for beats. But I could download Dr. Dre's instrumental for free. And as long as I don't sell this bitch, he's not coming for me, you know? I'm going to put it out for free. People going to love it. They're going to want me to come and wrap the motherfucker and they're going to pay however much it costs, you know, so that's how we did it. You know, that's just, like utilizing your natural resources. Like, what's growing in the land? Like, what's there? Just looked around and worked off what's growing out of the ground when you don't have the funds to do it. Like, you know, and you're creating business. Like, that's all we've ever done. And the more resources and the more materials we gain, you know, from gaining leverage or going up a level, then we start another joint, you know? Cause we got more to start with, 'Cause we, we did it with zero. So now it's insane. Like, we're just throwing darts at the board, like, fuck it, let's try and start a speedboat racing team tomorrow, you know what I'm saying? Like, fuck, whatever is whatever you want to do. And I've seen people do it. I've seen Master P do it because he had, like, with the bread to try it, you got to go for it. But what you had, but his circle is, was so large at the time with no limit. Like, first crack some ideas, not the best ideas, but you got love for everybody, so you going to roll the dice with everything they come with. You going to try, see, but what's working for us is we don't have that many people, like, around, you know what I'm saying? Like, as far as where the love is, it is right, it is in the room, so we not going to bounce. So if we try each other's ideas, one of 'em going to work 'cause it was just to, you got 19 people in here trying to, you know, tell you what to do and you want to keep everybody happy. You try, you going to end up trying to, like, start a golf cart company and, like, do spacewalks and sell reptiles and wild pets and then just doing everything that they ask you to do. And some of it's not going to work. [00:18:59] Dan Runcie: And I feel like with that, too, is just understanding your brand, understanding what's effective. And I know last year you had released an EP as an NFT, and I know this was the time when a lot of people were first discovering what an NFT is and things like that. What was that like? Because I know that was something that you didn't necessarily need to do to reach your fan base and do everything you wanted to do.[00:19:21] Curren$y: It wasn't to increase the fan base. It was to make our listeners aware that we are in touch with what's going on, and we are going to make sure that you guys aren't left behind as far as having Jet Life representation because we know you wear this shit every day. We know this is all you're listening to. So if the whole world converted to the metaverse, and everybody just wore headsets and live like that, how will you survive if your life is Jet Life? We got to give you something in this shit too. Rather we understand it or not, we have to learn to understand it, to become a part, to take care of y'all out there because it's real, you know? No matter how imaginary it may seem, it's real, you know what I'm saying? It's intangible, but it's a real thing. So we had to be able to provide something for our people 'cause they were there, you know? You look out of touch and, like, not sharp, not able to move, you know, then people wash their hands of you. Other companies won't want to collaborate with us that much because it won't appear that we are in the know, where if you have a big company that's not doing anything in that world, they're like, oh shit, look at Jet Life, well, let's just fuck with them. Let's put some bridge in them because they can handle this for us, blah, blah, and that be our representation 'cause we're far too big to even try to learn and far too big and far too old to even try to learn that shit, you know what I'm saying? So once they saw we did, that makes us look, you know, mobile, you know what I'm saying? [00:20:51] Mousa Hamdan: We have to exist in the future. You know, at the end of the day, we got to do what we have to do to let everybody, like he said, we're in the know, you know, we're aware of what's going on, what's coming, what's worth getting involved with, what's not.[00:21:05] Curren$y: And we going to ride with y'all because if it crashes, all us, then it did it off of us. You know what I'm saying? Fuck it. We going to roll too. [00:21:12] Mousa Hamdan: And even back a long time ago, I don't know if Curren$y even remember this, we did a deal back then with BitTorrent that we released a mixtape on BitTorrent, and it was 'cause the relationship we had with BitTorrent, they wanted to move away from everybody feeling that BitTorrent was a piracy site, and they wanted to like, well what if we give away something that we actually want shared? [00:21:38] Curren$y: Yeah. [00:21:38] Mousa Hamdan: And I remember we did that, I think we had like 156 million shares.[00:21:45] Curren$y: Yeah. [00:21:45] Mousa Hamdan: I told the record label that we were in a deal with at the time and they was like, nah, I got to see that. They didn't believe it. Well, like, what? Don't worry about it. You don't have to believe it. And that's why we're not with y'all now, because y'all don't believe the future. Y'all believing what y'all were taught to believe.[00:22:05] Curren$y: Yep. [00:22:05] Mousa Hamdan: Rather than having your own mind and realizing things change, the world changes. And you just got to be in the mix. You got to know what's going on. You got to get involved where you fit in. [00:22:15] Curren$y: You got to appear agile, man. [00:22:18] Dan Runcie: Stories like that, I feel, is what set y'all apart because if you think back to that time, no one wanted anything to do with BitTorrent or even LimeWire, BearShare, all these places where you could stream music and I get it. It was all the piracy, all the copyright. But at some point, someone asked to be able to say, all right, this is where folks are at. This is how they're getting our music. How could we get our music onto these places? Or how could we just think about it in a different way that isn't just no, don't do that?[00:22:46] Curren$y: Watch it come all the way back to the beginning because we stayed true the entire time, that company that needed to wash his hands and kind of rebirth themselves, needed to stand next to something that was pure the entire time so that they could get some of our life, you feel me? Like, that was the way that shit worked. Their name was so sullied that it was like, okay, as far as music is concerned, people know Jet Life will not falter. They won't fold. They don't go for fucking the dangling carrot. So if we fuck with them, then they would know, like, well, Jet Life wouldn't fuck with us if we were really this pirate fucking factory. So it made everything, you know, legitimate. You know, we saw good in them, so it was cool, yeah. [00:23:34] Dan Runcie: Yeah. It's interesting too, to make me think about the current thing that people are pushing back on, whether it's streaming farms, you know, people trying to drive up these streams and stuff like that. What's your take on that? Because I feel like, for you, something like that's almost irrelevant because you're not in this to, like, sell your music, so you don't care about charts or probably any of that stuff. [00:23:52] Curren$y: I can't blame them because it's not like streams, not like that shit pay you a lot of money, you know? I'm saying it takes a lot of streams to make, like, you know, substantial money. It takes a lot of people. A lot of artists don't even understand, you know what I'm saying? Like, the motherfucker call me like, bro, you did a million streams in the day. Like, so what do you think? I'm going to buy a yacht tonight, like, that was worth $12,000, bro, you know what I'm saying? That was worth 12 grand. I was like, don't trip. So I know they need those machines and shit to try and run those streams up. That could be check fraud. Like, they're trying to fucking, they're riding the clock, like, here man, we did 80 zillion billion streams in Apple music. Here's the paperwork. Fucking pay us, man. It could be that, it could be, we need to fucking this shit up so we could get a deal from some other people, maybe Pepsi Cola will reach out to us because they think we going to bring 'em all this attention and fuck them if we can't. The check's already here. You know, everybody's hustling though. It's not righteous, you know, but none of this shit is righteous. And that's kind of the ceiling that we set on ourselves by trying to, like, be legit, you know, it's not like that, you know what I'm saying? So I don't trip off the stream machines and people with the padded streams or, because I understand why they do it. We're blessed to not have to exist that way. And on the other hand, we do a decent amount of streaming because I put out a good amount of music, so I'm not going to do a million every month on one project like these other dudes, like, dude, some people only got to come out two times a year because that project will stream a million fucking streams a month every month all year. But what I will do is probably drop every month and still make it that way, you know what I'm saying? Or drop every two months, you know, and I'm still making that same bread. We just work harder, you know, because we're not doing a lot of the extra shit. [00:25:56] Mousa Hamdan: It don't hurt that he likes to record and what you're going to do? [00:25:59] Curren$y: Yeah, for sure. [00:26:00] Mousa Hamdan: You going to hold all the music? The music's going to sound old. He was writing about a '96 expedition, right? You got to put it out, bro. Next year, that thing's old. [00:26:11] Curren$y: Yep. [00:26:12] Mousa Hamdan: So at the end of the day, it don't hurt that he likes to record and the fans like to consume the music. They like the new drops. They don't feel like they're oversaturated with his music. They want more. [00:26:23] Curren$y: Yeah. The only time we hear that word is from, like, somebody outside. It's like when I'm doing, like, a press run and the people who had to Google me while we were on the elevator and we get up there to interview me, and that's like some shit they say like, so do you think you know about oversaturation? Like, fuck no, I don't think about oversaturation. I only think about my folks, like, you know what I'm saying? That's you. Y'all don't know. Y'all just tired of saying that Curren$y is coming out again with a project. I'm just tired of saying that. It shows up on y'all fucking thing. You have to mention it. You're just tired of saying this shit. [00:26:58] Dan Runcie: That's them trying to put you into a box. That's them trying to put you into what they know. But like a lot of people that serve their base, you know what they want and you are giving them exactly what they want. [00:27:08] Curren$y: Well, I mean, we interact with and we're around motherfuckers that come to this store all day, sometimes not even, to buy a shirt, like to be like, bro, when is this dropping? Like, you know, to play something for Instagram, when is this coming out? So we got our finger on the pulse of what's keeping us alive. Like, we check our posts often, you know.[00:27:28] Dan Runcie: For sure. Mousa, I want to talk to you a bit about the business of Jet Life and everything you have going on. And I know we talked a little bit about how touring is a big place where you all are getting a lot of the money, but what does the breakdown look like from a high level? Like, how much of the money you all have coming in is from touring compared to the other businesses and then compared to streaming and the music itself? Like, from like a percentage? [00:27:53] Mousa Hamdan: Well, I think, of course, since pandemic, the touring has slowed down. We haven't done anything, but I don't think, for a while, I didn't think the people were ready for a tour, you know, because different cities still had different COVID restrictions and vaccination card restrictions, which would limit the fans of coming to the venues. So it wasn't a good look. I spoke to some other artist manager, who is like, yeah, he's on the road, but he's kind of depressed because shows are not selling out. He feels like he lost it. And it's not that, it's just that the environment wasn't for that. You were going out there too fast looking for the money. The good thing with us was, like he said earlier, that if one thing wasn't doing what we wanted, we had something else that was doing it. So, crazily, the apparel skyrocketed during the pandemic. [00:28:47] Curren$y: And I was the one who thought we needed, I thought we had to stop. [00:28:51] Mousa Hamdan: Yeah. [00:28:51] Curren$y: I was like, nobody is going to buy a fucking hoodie. [00:28:55] Mousa Hamdan: He was like, bro...[00:28:56] Curren$y: There's no toilet paper. There's no fucking lights on in the store. Who the fuck is going to order a shorts, and fuck it, we're selling out of shit. [00:29:05] Mousa Hamdan: Shit was flying. [00:29:06] Curren$y: I was watching the news. There's just one, like, who are these people that are buying? Are they aware that this shit's even happening? Do they know they have nowhere to wear it to? And they're just posting the shit in the crib, in our brand new drop. Like, just fucking kicking it. The love was real, and they kept us alive, bro. I bought like fucking three or four cars while the shit was locked down. You couldn't even, we couldn't even go to dealerships, and I was buying cars because people were buying fucking sweatshirts. I'm sorry. I'm going to go back here. Y'all continue with business talk. [00:29:42] Mousa Hamdan: Definitely.[00:29:43] Curren$y: You know I'm saying? He knows, he knows, he knows. [00:29:47] Mousa Hamdan: So I think when he drops some music, there's a jump in streams, you know what I'm saying? There's a bigger check coming, you know, apparel, same thing. We drop some, a new line or a new drop, it's bam. You know, everybody wants that, and depending on what it is. But, you know, we tend to drop a good little bit of apparel. So I think now apparel and the music kind of coexist, and both have their times, that one makes a little more than the other and vice versa. The other businesses that are fresh starts are creating a revenue. Of course, we don't expect the nightclub business to make the money that the record label makes, but it's an addition. So it is always like our thought of keep putting in the pot. Eventually, that pot will get full or, like he says in the songs, we're trying to fill up a safe. Once we fill that safe up, we just got to get another safe. We're not going to empty that safe. We're going to get another safe. Now we got to fill that one up, you know? So if, you know, at the end of the day, you know, it is Jet Life, we're going to spread our wings, we're going to see what we can put our hands on that will create a revenue and at the same time, sticking to our morals and beliefs of what we feel like Jet Life should stand for. A lot of people don't know, Jet Life, at the beginning, Jets was just an acronym. Just enjoy this shit. So that's what we're doing. We're enjoying it. Or like I tell people, Jet Life has just enjoy this life. So that's where we're at with it, steadily growing, steadily trying to get involved in everything that makes sense. You know, If it doesn't make sense, we leave it alone. So the revenue streams, like I said, it kind of goes back and forth. Apparel definitely is a world of its own now. Apparel is great. You know, we moved from, originally, like you said, with touring. That's when I realized that the apparel was so good because at touring, we were selling so much what I consider tour merch, you know, which is just the name of the show, the city's on the back, a picture of Curren$y on the shirt. You know, all the fans want it. They're like, man, they really love this shit. They're buying it. [00:31:56] Curren$y: That was just a tour shirt. [00:31:58] Mousa Hamdan: So then I was like, well, damn, I'd rather wear our own clothes when I want to go to the nightclub, when I go out to eat, or if I just want to hang around. I don't want to wear a tour shirt all the time, but I want to wear something. [00:32:12] Curren$y: And I didn't want to wear no shirt with my name on it.[00:32:15] Mousa Hamdan: Right. He doesn't want pictures of himself.[00:32:17] Curren$y: I don't want no shirt with me on it, no shirt with my name on it 'cause like, who the fuck am I? You know what I'm saying? Who am I, you know, to even do that? [00:32:27] Dan Runcie: Right. 'Cause that's more like merch, right? And I didn't know that people use merch, but like, no, y'all have a clothing apparel.[00:32:33] Mousa Hamdan: Tour merch. And then we changed it to apparel. Apparel, which you wear on a daily thing, every morning you wake up and you put apparel. You were sleeping in apparel. So we had to reach that. And then every couple of months we just think, what else can we make? What else? 'Cause you know, we started with just t-shirts, you know, then went to hoodies and long sleeves. And then we're like, we got to start getting bottoms and we got to get hats and you know, so now we're, you know, building into accessories and whatever else people may like. And at the same time as well, like I said, we test fitted on ourselves. If it's something that we don't want to wear, I'll always show him stuff like, look, these are some of the new designs. This is some of the stuff that you talked about with me that we created. Now it's on paper. Do you like it? If you like it, let's push the button. Let's go with it. If it's something you would wear. 'Cause at the end of the day, if he doesn't wear it, if I don't wear it, if the other artists on Jet Life don't wear it...[00:33:33] Curren$y: It will sit in the warehouse. [00:33:35] Mousa Hamdan: Why would we expect a fan or a fellow lifer to wear it? They don't want to wear something that you don't even want to wear yourself. So if we don't want to create nothing that we don't like. You know, and that's just, I think our business model with everything we do. We don't want to do anything that we don't agree with. We don't want to do anything that goes against what we stand for. [00:33:58] Dan Runcie: That makes sense. And the point again about the merch, too, I think Curren$y, you had this line in maybe it's an interview, I think you said, but it's like, no one's calling you Sean John merch, right? As you're telling Diddy like, oh yeah, I like your merch. [00:34:12] Curren$y: Right. You know what I'm saying? And just, we have to stand on that, you know? And I think we have for a long time, and it made people change the perception of it, you know. Before, like, just, the fact that we stand behind it like that, it made people buy it who maybe weren't even thinking about it because it made people want to look at it a little more to not like it, you know, like people came in to find like what was wrong. And then it's like, well, shit's just actually, you know, I'm going to buy the shirt, you know what I'm saying? Like, they were coming to point out why it was just merch and it wasn't, you know? [00:34:47] Dan Runcie: Yeah. Yeah. One other question for you, Mousa, about touring itself and just doing live shows. Because of how well the apparel's going right now and how the business overall may have changed since the pandemic, do you ever think that you'll go back to doing the same number of shows that you were doing before the pandemic because of how much success you have with everything else? Do you think it'll scale back a bit at all?[00:35:10] Mousa Hamdan: In my mind, I've always lived thinking never forget where this started from and never forget what created this lane for you to get into. Without the touring, I never knew how much the merch sold. And I noticed that with a lot of artists, there's a lot of artists that don't sell merch, and they don't know the money that they're missing. So without the touring, without the shows that we do, like he said, we put a finger on the pulse of the fans. Well, we'll know who's coming to these shows, you know, and you can see when, all right, well, the shows are getting a little light, so what is it we're doing wrong? There's something that we're missing. Same thing with the apparel. When sales are a little low then I'm like, well, what are we doing that we used to do better? Or what are we missing? Are we getting laid back? Are we feeling like it just is what it is now? But being involved in it like that, I think, keeps us on with whatever else we're doing 'cause it's going to keep telling us, like, this is the pulse of the people. This is what you're doing. So I think we'll always do tours. Maybe we're not, you know, one time we did, I think it was 60 shows in 70 days, which was crazy. [00:36:23] Curren$y: 63. [00:36:24] Mousa Hamdan: Yeah, it was, yeah, crazy. Show every night, Monday, Tuesday, Wednesday, Thursday. And they're like, whoa, when is the break, bro? Like, when are we? So I don't think we'll do that. But we're going to stay out there, you know, as long as the people want to see, and he's got fresh music that he wants to perform. And you know, he's an artist, I think, that feeds off the energy, you know. And if the crowd doesn't have the energy, he's like, why am I here? Why am I performing for these people? They don't really want to see me because the energy is not there. So as long as we're feeling the energy, then I think we're there. Hopefully, I don't see it going down no time soon. You know, we're going to keep doing whatever it is that allows us what the universe puts for us to do, you know? And we're just going to be there. [00:37:08] Dan Runcie: How do you look at doing your own shows versus doing festivals? Do you have preferences? I feel like for an artist like you, your own shows where your people are going to be at, right?[00:37:18] Curren$y: Yeah, bro. This is a whole other show. Don't do it. We love, we love, we love festival checks. If I had to pick, I like, you know, me at the House of Blues. I know exactly that the people who are in there, like, are there for what we going to do, you know what I'm saying? The festival, I've been blessed to be a person that you kind of, you can't get around me in the game, you know what I'm saying? So when you don't fuck with me, people speak out to you. You look stupid, you know what I'm saying? You look crazy. So people put me on shit, like just, no, we got to have him on this festival. We got to put him on this. We got to put him on this, you know what I'm saying? And my core people are there, but they're surrounded by people who are, like, waiting for like the next person to come out and like spit fire, you know what I'm saying? And walk on the crowd, pop, you know, like, I can't do it. I'll never do it, you know? So I'm like, I don't want to put my listeners through it because and they're in there like, shit, man. There's, like, a kid who kept, like, elbowing me, you know what I'm saying? Like, some of my listeners are, like, there's always somebody to put me to the side, like, yo, I'm 51, my nigga, like, this is the shit I'll listen to. So them, them dudes don't, they don't want that. Those ladies, like, who pull me to the side, like, boy, look, you know, I could be... I'm like, Yeah, you don't have time for, you know, for that. So I like to do just my thing. But the festival checks go directly to the sports car dealerships. Like, those are the checks that get you off the lot though. So, you know, you're being a fool not to do it, you know? And that's just business. [00:39:03] Mousa Hamdan: The checks are good, yes, but I think as well...[00:39:08] Curren$y: He makes sense with this. I know what he coming with this, but I'm going to tell you, they're coming to business. He makes sense. [00:39:12] Mousa Hamdan: Sometimes, I honestly, in a lot of things that we do, I always tell him, I think he underestimates his reach, you know, and he's too humble to the point of, nah, bro, like, they're not really here for me and this, that, and the other. Now, I'll be honest with you, we had one festival show. I was a little worried. We got on stage. He wasn't on stage. He was backstage, so he didn't know nothing was going on. I literally walked to the DJ. I said, bro, this is probably our last festival 'cause it was like, there was literally 10 people in front the stage. I said, bro, if he gets out here and there's 10 people out here, he's liable to walk off stage, bro, so listen to me. The intro started and it looked like a rush. Like, I didn't know who. They had about 5,000 people or better rush to the stage. And I like, whoa, that's more like it. Then he came out, he didn't see the dead part. He saw that part. He was like, oh, my people are here. They're here. They showed up. They showed up. I'm like, you just don't know. They really did show up 'cause they wasn't here five minutes ago. [00:40:22] Curren$y: They just showed up.[00:40:23] Mousa Hamdan: Bro, but then that's understanding the festivals. You got six stages. [00:40:28] Curren$y: Yeah. [00:40:29] Mousa Hamdan: They're trying to catch everybody.[00:40:30] Curren$y: I was posting one time, there's a way to do it, you know what I'm saying? As long as you are vocal about what time you go on, your people will navigate through to get there for you, you know what I'm saying? But you also, you got the people who're waiting for somebody else 'cause I'm like, it's a gift and the curse, 'cause, like, I'm, like, the most known unknown. So it's, like they know they can't put me on at fucking one o'clock, you know what I'm saying? So then when you put me on at, like, eight, and then there's, like, whoever the fucking, whoever name was written this big on the flyer, this guy's coming after me, all right. The kids who are waiting for this guy are, like, have been pressed against the barricade for hours, like since 11:00 AM. When I come out there with my low-impact workout, like they're fucking dying, like looking at me and I'm like, I get it. Don't trip. I fuck with this guy's music too. He'll be out here in a minute, and I hope he does a backflip on top of you when he does, you know what I'm saying? Like, that shit kind of fuck with me because I'm delivering a real message. Like, every word I write, like, I mean it. So I really don't want to say it sweating to a person who's like this, like on the barricade, just like, bro, please stop. We get it. You like Chevys, you want us all to get rich? Fucking shut up. We want to fucking rap about drugs. Where's the next guy? So that shit kind of make you not want to do it. But then this guy, fucking, he's also the person who says this like, okay, you also woke some people up to the music you make. Then there's always, like, when I get done, the people who work the festival, the grounds, are, like, bro, I never heard this shit, but this was real music. Like, I couldn't understand nobody else's words, you know what I'm saying? This is fucking good. So I'm like, well, that's cool. I do leave out there with more listeners than I did, you know what I'm saying? It might be 12, it might be 150. He going to count every dollar for each one via stream, so I understand where he comes from with that. But I always like, I say it to myself sometimes, and sometimes when it gets too heavy to me, I say to the people around me, I'm like, bro, I'm actually the only one who fucking have to go and do that shit. Like, I get it. We all here, we all fucking supporting, but they're not looking at you, like, get the fuck out. And you got to do this shit for 45 minutes, you know what I'm saying? Sure. There's some people who are enjoying it, but the motherfucker who's right in front of you is dying, and you have to continue to have a good time. Like, that shit is like being a fucking Disney World animatronic or, being like a Chuck E. Cheese thing. Like, that's a rough time for me for sure. But it works, you know? That's anybody's job.[00:43:27] Dan Runcie: It's a balance, right?[00:43:28] Curren$y: You know, a hundred percent love any fucking gig that you have, any job you have, bro. I'm sure everybody at NBA, that was their dream, to go to the NBA. Some of those days sucked though for those dudes, you know what I'm saying? So it ain't always going to be the shit. The situation overall is one that I wouldn't trade for the world. [00:43:48] Dan Runcie: That makes sense. And you mentioned too that the money that you're getting from the festivals is going to the sports car dealership. Can we talk a little bit about that? How's that business set up and how's that been going? [00:44:01] Curren$y: Oh, well, me and Mousa, we've always been kind of into, like, bringing cars back to life, restoring things, and shit. But I've been holding on 'em. But as of late, we're building a stable of vehicles to kind of release onto the public, but it'll be like a collection, the same way we come out with clothes. There'll be like six vehicles put up for sale at one time that we cultivate and put together. We putting together a BMW, a few vintage sports car that we putting together. We're going to roll 'em all out at one time, you know what I'm saying? So I expect them all to be gone, like, within the week. I expect it to be like shoes. Like, I expect motherfuckers to try it and everybody will blow. You know, everybody try their hands at the shit we do. So another motherfucker with a bigger audience and shit will try to do the same thing, but you know, who cooked that shit up first. [00:44:52] Mousa Hamdan: Okay. They know, They know where they got the idea from.[00:44:56] Curren$y: Yeah, they know, too, so it don't matter. [00:44:59] Dan Runcie: Speaking of cars and trendssetting, I know you got a partnership with NASCAR as well, and I feel like there's another thing, too, where not a lot of hip hop artists are doing those deals, but we are just seeing the way things are trended now. Everyone will be following to that. And you got the Jet Life cup series. All right, let's talk about it. [00:45:16] Curren$y: Yeah, man. Yeah, man. Well, yeah. People of any other nationality other than the original rebel down home boys were not involved in NASCAR and they fucking, they had it that way. They built it that way, executive-wise, it's not like that anymore. Now, you know, doors have been broken down, kicked in, and open-minded. People are now there, and it's made it more accessible for fans. I was shocked when I went that I saw like groups of different people, I don't want to just say black people, just different people in general because the other side of it, the way it was, they weren't picking what nationality or what people they didn't want, they didn't want nothing else, but what the fuck they had, you know? So it's way different now in all aspects. It's not just minorities selling nachos. They driving the cars. They are the ones like turning the wrenches and making sure shit is right. They got headsets on, they out there doing the real thing. And I brought one of my younger homies with me, it blew him away. He's at school for engineering, and he was just, he was nervous for us to even be out there. I made a few small jokes to my friends when we first got there based on the appearance and how it looked. But it really wasn't like that once you got down into the meeting. And I read on social media, like I read a few comments. There were some people who were not excited about our presence. There's some people who weren't into the collaboration. I saw one thing under a video that I was so sad 'cause I was like, I hope my mom don't see it. Because the motherfucker was like, what is he coming to steal? And I was like, damn, if my mama sees that, she'll probably cry, you know what I'm saying? Like, it'll take a minute for me to get her over that shit. But what are you going to do? You know what I'm saying? This shit, you can't blame the behavior 'cause it was taught a long time ago. Like, they didn't pop out like that. That's what that motherfucker told him to do, you know what I'm saying? And what we doing is playing the hand and telling the people who are receptive and the new people, the younger generation, like, it could go this way instead, you know what I'm saying? Like, we were up in all the suites and eating NASCAR food, you know, and actually, I'm going to say this, I was a little bummed with the NASCAR because we couldn't get a Coca-Cola badge on our jersey. We wanted to have it because the race that day was actually Coke Zero, Coke Zero 400, all right. So, when they originally had the design meeting for the package, they included Coca-Cola logo because that was the race, you know, that's when it was coming up. And I think like they did the same thing, like, whoever is involved with the collaborations just did a little brief overview of who I was or what I was about, and they're like, oh, no. So like, that kind of fucked me up. [00:48:17] Mousa Hamdan: They'll be back though. They'll be back. [00:48:19] Curren$y: Yeah. But you know, like, I was like, well this still, you know, shit is still hard, you know? But with time, with time, yeah. And I don't know. And then, and I didn't like the you got gang with you. I heard over there, I'm sorry. [00:48:32] Mousa Hamdan: Oh, yeah.[00:48:32] Dan Runcie: That's from fans or was that from NASCAR? [00:48:35] Curren$y: No, no.[00:48:35] Mousa Hamdan: Coke exec. [00:48:36] Curren$y: Just one of the brass at Coca-Cola. And I drank a lot of Coca-Cola, so I really do need to stop, but for health and maybe for business, because motherfucker was like, to the representative from NASCAR who was showing us to where we were going to go to sit down, like, in the suite. He's like, oh, you got a gang with you. And I was just like, damn, like. I'm sure maybe I'm looking at it with a microscope, you know. [00:49:01] Dan Runcie: But still though, you can't say that, yeah. [00:49:04] Curren$y: I really don't know, I just don't know. I just was on the fence. I thought about it a lot. I think about it. [00:49:09] Dan Runcie: But like, they wouldn't say that if, like, Jason Aldean walked up in there with a group of folks.[00:49:13] Curren$y: You got a lot of people with you, you know I'm saying? It wasn't like he said the gang's all here. If he said the gang's all here, that would not have hit me like that. People say that the gang's all here, that doesn't mean that you got a street gang here. [00:49:29] Mousa Hamdan: Right.[00:49:29] Curren$y: But, whoa, you got a gang with you. [00:49:33] Mousa Hamdan: He could have said, Hey fellas and just kept it moving.[00:49:36] Curren$y: Yeah. [00:49:36] Mousa Hamdan: How y'all doing guys? [00:49:38] Curren$y: Yeah. [00:49:38] Mousa Hamdan: And you didn't have to make conversation with us. You were just passing.[00:49:41] Curren$y: It felt like it was a Chappelle show skit because it could have been where keeping the real goes wrong. Because I was like, half step, like, trying to see if I could make eye contact with one of my friends who felt like maybe that was wrong and I had support in, like, hey man, like, what? But it could have went way south. Like, there could be no more NASCAR 'cause shit if we could, would've did that. You know, we just might not have the Coca-Cola on the next one. Or maybe we will, maybe they're like, oh, shit, man. We didn't mean that. I thought I did say that gang's all here. Let's put a badge on the fucking next jacket, you know, it might work that way. And that's business and that's why we're here talking, you know? And and that's why it it pays to be true to yourself within your business. And if your circle is small, it's easier for you to be honest and not worry about if something sounds stupid or anything because, like, we have a yin and a yang, like, you know what I'm saying? Like, that's what makes it work. [00:50:37] Dan Runcie: Right. And I think that's a good note to close things out. And I want to get your thoughts on this question because as you started with the beginning, y'all have been together for a while, even in this conversation, we can see that chemistry between the two of you, that yin and the yang, you understand each other. What do you think is the secret for having the artist and manager that just stay with each other? 'Cause there's so many times that, either other artists or other managers that have been on this show and they're like, oh, yeah, you know, so and so fired me. [00:51:04] Curren$y: Somebody lied in the beginning.[00:51:07] Mousa Hamdan: Right. [00:51:08] Curren$y: Just like, who fucks up anything. You know, like, just somebody lied in the beginning. The artist was signed to nine different managers. Everybody loaned him $1,500 to help him do something. He's just signing with whoever's going to fucking give him a fucking chain or watch, and he's not being honest. Or there's a fucking, like, a discrepancy on this stack of paperwork or something, and this guy's outsmarting the artist and fucking going to rob and blind, you know what I'm saying? Like, if that happens in the beginning, the intentions are bad in the beginning, then you'll see where it looked good and then it fall into pieces because you find out, you know what I'm saying? Like, fucking, when we read for fucking N.W.A, which is one thing I didn't turn down, I was down to do that. He's coming to me with movie shit. Do you want to read this? Do you want to do this? No, no, no, no. They're like, do you want to be Easy-E? And I was like, no. At first I was like, no, like, they should call his son, you know what I'm saying? But then I end up reading for it, and then they end up getting a guy from New Orleans anyhow. So that was cool. I'm like, damn, I probably could have got that shit. But nonetheless, the part that we were reading, it was after Easy-E's wife went through the paperwork that he did with Jerry Heller and she brought, like, so much shit, attention to him. And when E and Jerry had this talk, it made Jerry cry because he was leaving even though he did so much wrong shit. But in his heart of hearts, he probably didn't think it was wrong 'cause he took dude from nothing. But it was still bad and he couldn't believe how quick Eric was ready to shut the shit down. But it's because he was wrong. Like, once that, there's nothing you could do after that. Once it get like that, it shut down. That shit is heartbreaking. And we never, like, we have not hurt each other like in that aspect, you know what I'm saying? Like, when I came to him, I was like, look, I don't, you know what I'm saying, I ain't signed nothing yet, but this is what's going on. Boom, boom, boom, boom, boom. That's that, you know. I don't know, I'm going to go talk ahead, I'm going to do this and get this and then not say this. You know, he don't fucking pop up and see I'm doing the show in Colorado and, you know what I'm saying, I didn't say anything. I just went, flew out and oh no, I made just 30 grand right quick, you know. That there's just, it's all on the up, bro. So with that, you know, you stay friends, we friends first all. [00:53:28] Mousa Hamdan: That's I think the biggest thing. [00:53:29] Curren$y: Yeah. If he wasn't in my homie, then we wouldn't do business. [00:53:31] Mousa Hamdan: We started as friends, and then we continued to be friends in this. [00:53:36] Curren$y: Yeah. All the way through.[00:53:38] Mousa Hamdan: We're business partners, but we never was just business partners. We was always friends to begin with. [00:53:44] Curren$y: Right. So that make you not be able to do no fucked up business. [00:53:48] Mousa Hamdan: And then we trust each other. [00:53:49] Curren$y: You know what I'm saying? You can't do that to your friend.[00:53:52] Mousa Hamdan: Trust is big, you know. I think he trusts my decisions, I trust his decisions. And then we talk about things, like he said, we were going to always converse about whatever decisions we want make. If there's ever a thought, I think, you know, this may be wrong or whatever, I'm going to consult with him as if he was my manager, you know what I'm saying? So we're going to talk and the trust issue, I always hear that, you know, how, why y'all been together so long? I'm like, if you build a business, who builds a business to separate, right? We build a business together [00:54:24] Curren$y: Who are these people that you're with? Who's in your car? Who the fuck are you riding with in the car? Like, who? That's why. That's why I said, like, having 19 and 30 motherfuckers. Like, now there might be 30 people in this building at a time, and they all could have a Jet Life chain, they all be a part of what's going on. But at the end of the day, you know what I'm saying? When it'll come down, it'll come down. Like, we got to sit down and fucking, you know what I'm saying, and put it together, you know. Everybody respect that because when we come out the room, we come out the with the right answer.

From the archives, another Best of The Bad Crypto Podcast with CZ, CEO of Binance crypto exchange while Joel and Travis are on a break from broadcasting. Does CZ like us, even if we are following each other on Twitter? Find out what it's like to launch a successful crypto exchange in the midst of changing crypto regulations and what considerations were given for user experience and security.  Founded in 2017, Binance is now the world's biggest crypto exchange based on trading volume. Changpeng Zhao, known as CZ in the crypto world, is the CEO responsible for this virtually overnight success. Today we're pleased to welcome CZ to the show for a discussion that covers the gamut. From humble beginnings to crypto world domination, we've got it all. And now, you are gonna get it. In fact, after this interview you'll be saying “Thank you, sirs. May I have another?” It's the “we launched Bad Crypto at the same time you launched Binance, so how come we aren't billionaires” episode #250 of The Bad Crypto Podcast. FULL SHOW NOTES: http://badco.in/250 SUBSCRIBE, RATE, & REVIEW: Apple Podcast: http://badco.in/itunesGoogle Podcasts: http://badco.in/googleSpotify: http://badco.in/spotify FOLLOW US ON SOCIAL MEDIA: Twitter: @badcryptopod - @joelcomm - @teedubyaFacebook: /BadCrypto - /JoelComm - /teedubyawFacebook Mastermind Group: /BadCryptoLinkedIn: /in/joelcomm - /in/teedubyaInstagram: @BadCryptoPodcastEmail: badcryptopodcast[at]gmail[dot]comPhone: SEVEN-OH-8-88FIVE- 90THIRTY TIME STAMPS 00:00 - Intro  09:30 - Feature Guest: CZ at Binance 09:52 - What is the process now for getting a coin listed on Binance?  11:50 - We heard you hate ICOs. Do you not like putting ICOs on there?  12:50 - CZ's background. When you were evaluating the exchange space in 2016-2017, what made you say this was missing from the market and this is where you were going to focus your efforts?  15:40 - Do you trade in about 180 different countries? How do you manage the regulations in those different countries?  17:03 - We are all about teaching regular people about crypto because we want to see mass adoption of this revolutionary technology that is going to change the world. What do you think is going to be the watershed moment for crypto?  18:55 - CZ's Twitter usage 20:58 - The future of Binance 22:38 - What do you think created the mass adoption of Binance?  23:36 - One of the most secure crypto exchanges  24:25 - One of the features I really like is the sweep of small balances, trace amounts of coins and being able to convert them to BNB. Explanation of the use of the native BNB coin. 26:27 - What is your vision for the growth of Binance and its impact on crypto adoption in the next 20 years?  28:34 - What about a Binance app or mobile interface?  29:00 - What about the Binance Decentralized Exchange? How does it work with the centralized exchange?  32:25 - The BitTorrent token sale was a huge success on Binance's launchpad. You Tweeted there were issues on the ICO for the BitTorrent token. Do you really think it would have sold out in under a minute? 35:00 - Binance did an ICO.  35:50 - Speak to the future of crypto; where do you think the world is going and how is blockchain going to change everything?  38:00 - Closing remarks DISCLAIMER:Do your own due diligence and research. Joel Comm and Travis Wright are NOT FINANCIAL ADVISORS. We are sharing our journey with you as we learn more about this crazy little thing called cryptocurrency. We make NO RECOMMENDATIONS. Don't take anything we say as gospel. Do not come to our homes with pitchforks because you lost money by listening to us. We only share with you what we are learning and what we are investing it. We will never "pump or dump" any cryptocurrencies. Take what we say with a grain of salt. You must research this stuff on your own! Just know that we will always strive for RADICAL TRANSPARENCY with any show associations.Support the show: https://badcryptopodcast.comSee omnystudio.com/listener for privacy information.