Podcasts about Observability

About Guang Ming Guang Ming Whitley was elected to Mount Pleasant Town Council in 2017 and resides in Old Mount Pleasant with her husband, four children, and a dog.She earned a B.S. in Chemical Engineering from the University of Southern California and a J.D. from the University of Chicago Law School, where she was a member of Law Review and a moot court semi-finalist. After completing her law degree, Guang Ming taught at the University of Chicago and practiced intellectual property law in Los Angeles. She then retired from active practice to serve as Chief Operating Officer of the Whitley Household. In 2020, she cofounded Lattice Climbers, a company dedicated to teaching soft and life skills to young adults.Guang Ming is also President of the Girls State Alumnae Foundation and attended the American Legion Auxiliary Girls State in 1996, where she was elected governor. She has volunteered with the ALA Girls State program in a variety of capacities since 2000.Links:Lattice Climbers: https://www.latticeclimbers.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Sometimes people like to ask me what this show is really about and my answer has always been, “The business of cloud,” which is intentionally overbroad; really gives me an excuse to talk about anything that strikes my fancy at a given time. A recurring theme has always been, “Where does the next generation of folks working on cloud come from?”That's not strictly bounded to engineers; that goes throughout the entire ecosystem. There are a lot of jobs that are important to the functioning of businesses that don't require a whole bunch of typing into a text editor and being mad about YAML all day long. Today, my guest is Guang Ming Whitley. Guang Ming, thank you for joining me, I'll let you tell the story. Who are you exactly?Guang Ming: Oh, my goodness. That's a tough question. Well, I am someone who has lived my life in a series of segments. I started off as an engineer—a chemical engineer—then went off to law school, taught for a year—Corey: Well, let's interject as well. That is how I got looped into this whole nonsense; you were law school classmates with my spouse. And whatever you're in town, she gets very excited at the chance to see you, and we finally got to meet not that long ago, had a great conversation. It was, “Oh, my God, you need to come on the podcast.” Which is neither here nor there. Please, continue.Guang Ming: So, then I had a segment as a stay-at-home mother. I started having babies and I had a lot of them. I had one daughter, then a son, and then I had identical twin boys. And once I started having them in litters, we decided that it was time to stop. So, four kids in and about a decade as a stay-at-home mom, during which time I wrote some books.And then ran for office back in 2017. And then in 2020, was working with someone just, kind of, over coffee, just having, you know, conversation, and we came up with the idea to start a business, and Lattice Climbers was born out of that.Corey: And Lattice Climbers is what I think we're going to be talking about the most today because there's an entire episode baked into every one of those steps. Maybe not every one of them would fit on a cloud-oriented podcast, but there's a lot of interesting backstory there and it resonates with me because my entire life has been lived in phases as well. And the more I talk to people, the more I start to realize that maybe I'm not that bizarre. People go through stages and they'd love to retcon what the story was at the time and make it all look like there's a common thread and narrative running through, but when we're going through it, it feels—to me at least—like I've been careening from thing to thing to thing without ever really having an end goal in mind. But in hindsight, looking back, it just seems like it was inevitable that I would go from where I was to here. It never feels that way at the time for me.Guang Ming: Well, I think for me, where I've ended up with Lattice Climbers has felt sort of inevitable because one of the through-lines of all my segments that I've gone through is a program called Girls State, and it is one that I have volunteered with. It's sponsored by the American Legion Auxiliary and it's a government simulation program. Over the course of one week, you simulate city, county, and state government. And it's all about civic engagement and education of young women, and empowerment. So, it's such a fantastic program.And I love it, but one of the things that I've seen with this program is, as the young women come through the program, some of them have skills, and some of them don't have skills. And there's elements that are missing, and that's something that I want to try to help with, with Lattice Climbers.Corey: So, what is Lattice Climbers in a nutshell? It's still very early days, which is fine, terrific; the fact that you care enough about a problem that is clearly plaguing not just our industry, but arguably our entire society is worth exploring in-depth. And with the understanding that the narrative may very well shift as times go on what is Lattice Climbers today?Guang Ming: So, Lattice Climbers steps into the gap between formal education and the skills necessary to actually adult at life, to survive in the real world.Corey: That is an area that is of intense interest to me. For listeners who may not have listened to every single episode here, my academic background is checkered, to put it politely. On paper, I have an eighth-grade education and no one can take that away from me. I was expelled from two boarding schools in high school, I wound up getting a diploma from a homeschooling organization that years later I discovered was not accredited, then I failed out of college. But again, no one can take that eighth-grade education away from me.But also look at me. I am a white dude in tech where my failure mode is a board seat and a book deal somewhere, and there are winds of privilege at my back when I do that. What also has been a strong contributing factor is that when I was 12 years old, my dad sat me down and had a long conversation with me about how to handle a job interview, what a job interview was—because when I was 12, I had no idea—and what they're looking to gain from asking you these questions, and why they're asking you the things that they do, what answers they're looking for, and the purpose behind the meeting that you're in. And that more than almost anything else as a single moment in my childhood shaped the reason that I became moderately successful [laugh] in my career, depending on what phase of my career we're talking about. That's stuff is super important and they don't teach it formally in any program I've ever seen. How do you approach it?Guang Ming: So, what we do is we have an intake quiz that assesses your skill gaps, sort of like a self-assessment, and then it gives you a customized curriculum just meant to fill your specific skill gaps. So professionalism, where we cover things like interview skills, behavior at events, table manners, those kinds of things. Financial literacy, we have little mantras like, “Credit cards are not free money,” [laugh] which some people never learned that. And then there's different tracks, so depending on whether or not you're college-bound, or vocational school, or military-bound, you can pick a different track for that and receive two-minute lessons, sort of the gems, distilled down. And there's little animations; we try to keep it as brief and information-packed as possible.Corey: Would it be fair to categorize this as more or less micro-lessons in how to adult?Guang Ming: Exactly. That is exactly what we're trying to do.Corey: I somewhat recently read one of the best stories I've ever heard about teaching students in middle school about financial literacy. And invariably, the financial literacy courses are all sponsored by financial institutions, and that's great. So, what happened was, someone from the bank came in and spoke to the students and then took them all to the bank and had them all open a bank account and deposit $5 into it. Great. A couple of years go by and it earns interest—not much because $5—the bank was then acquired and acquired again and eventually became rolled into Wells Fargo, and had a small balance fee, which then of course wiped out all of these accounts.And I don't think that there is any better lesson in the way the financial system works—in some ways—than that. And yes, that's cynical, but that idea of, if you are sort of toward the bottom, this system is basically stacked against you in a bunch of different ways. Look, I'm not here to rail against capitalism or society as it stands, but understanding that basic concept is foundational to realizing that maybe the credit card company isn't always your friend with your very best interests in mind.Guang Ming: Mm-hm. And we tried to explain that, too, you that when you get a credit limit, that is based on what your ability to pay the minimum balance every month. They don't care if you can pay it off. They care about making that interest off of you, and I think that's something that children and young adults need to understand.Corey: It feels like it ties into the idea of thinking critically. The problem with that is the root of that entire financial literacy anecdote that I came out with just now, is that the financial literacy program was developed and promoted by financial institutions. What I like is that I checked your website very briefly, and given the significant absence of a pile of disclosures at the bottom, I don't believe you're a bank.Guang Ming: We are not a bank, and we are not sponsored by a bank. We want to provide practical real-life advice that is useful, and in digestible chunks.Corey: A while back, before I wound up starting down the path that I'm on now, I basically yelled at people for fun on the internet. I know, imagine that. I was the moderator of two particular subreddits: personal finance—which, great, I spent my 20s in crippling debt; there's no one as passionate about that stuff as someone who has been converted. Great. And the other was the legal advice subreddit, which is probably horrifying to people like you who are actual attorneys.But it turns out that an awful lot of what I was doing in both of those subreddits was giving life advice to people on how to function in society. On the legal side of it, “You can't sue a dog.” “Okay, you are not going to be able to go down to the police station and explain your way out of troubles. Get an attorney.” It's baseline-level stuff.“Oh, you've been given a contract that seems unreasonable, but they'd say that you need you to sign it.” “Yeah. How about don't do that without having someone review it?” It's not actually legal advice. It is how to function in society as an adult, but that's a less catchy subreddit title as it turns out.Guang Ming: Well, it's all about raising your awareness level. So, I have a friend and she tells this story, MBA grad and spent her first six months on the job wearing sneakers every day to work—they were cute, fashionable sneakers, but they were sneakers—and they were not part of appropriate business attire for the work environment she was in because she just was oblivious to that as being an issue. And it took someone who was more senior to finally sit her down and say, “You shouldn't do this. You need to wear appropriate shoes to work.” And she was mortified but learned from that experience.So, what if you never had to have that? What if you never had to have that sit-down conversation with someone correcting you? What if you had a little, sort of, pocket guide that gave you that level of awareness? It's like, “Take a look at your office. See what people are wearing. You can't wear what the CEO is wearing because you're not the CEO”—I mean, unless you are the CEO, then you can wear whatever you want, but if you're just an underling at the company, if you're just starting out, you need to understand what the company culture is and you need to conform to that culture. Unfortunately, that's just, like… the truth of the matter.Corey: The common wisdom is, “Oh, if you don't know how to dress or how to behave in a certain scenario, reach out to one of your mentors and ask them for advice.” Not everyone has one of those things. I get some crap sometimes through it, but one of the big reasons I have open DMs on Twitter is specifically so people can message me and ask me questions about the industry generally, life in general; I'm always willing to talk to folks who are trying to figure things out. That's important. Since a disproportionate number of the listeners to this show do work in tech and the idea of having a dress code is ridiculous, yeah, in a lot of tech culture at t-shirt and jeans is just fine, but in other cases, it's not.And, for example, I'll get on stage wearing a full bespoke three-piece suit and give a talk. And it's fun. It's hilarious. It plays with people's expectations, but it's important to understand I view that more as costuming than I do how I believe someone should necessarily dress in that environment. I am, for better or worse, a very distinctive personality in this space, and using me as a blueprint for someone who is starting out their career is going to lead to disaster.Yes, I'm mouthy and I make fun of big companies because that's my thing. I also got fired an awful lot in—Guang Ming: [laugh].Corey: —my career, and those two things are not entirely unrelated, let's be very clear here. There's a lot that we can learn through observation, but dialing it in and figuring out what the expectations, are important.Guang Ming: Well, I think a lot of young adults—one of the things we focus on, as well, is the importance of mentoring and finding good mentors. And then you being the kind of person that a mentor would want to mentor. Because I think there's a lot of formal mentoring in work environments, and those don't always work as well as the organic relationships. So, we want to be that mentor that you never knew that you needed, the mentor that you wish you always had, to give you all that baseline information so that when you do meet with your substantive mentor, they can truly help you in ways that we cannot with our scalable mentoring micro-lessons.Corey: I have to ask, what is your revenue model? Because if this turns into charging kids money to learning these things, that has a giant exploitative flashing warning sign around it.Guang Ming: So, what we're planning to do is work with school districts and with nonprofits, and do sort of like a B2B model where we pilot with the school district, we pilot with the technical college, and give them an opportunity to add 30 to 50 students, work with the program. And if they find it something valuable, they find that it's a value-add and it's helping their students land jobs and have a better career, I think that then they'll use our program for their full technical school.Corey: I'm done a fair number of mentorships in the course of my career. I helped administer and run the LOPSA 00:13:43—or League Of Professional System Administrators—mentorship program for a couple of years. The reason that I have a career at all is that people did favors for me, and you can never repay that; you can only pay it forward. So, I had a number of people assigned to me through that program and through other areas as well, and what I've learned is that the success of a mentorship is almost entirely on the person seeking guidance: how diligent are they about following up, about going and asking great questions? Because otherwise, if someone comes and says, “Hey, can you mentor me?”—they never frame it quite like that, but that's fine; the terminology is always squishy here.Like, “Hey, can you give me advice on things?” “Sure.” And then they don't ask any questions. Well, if I just butt in with unsolicited advice, that's not helping them in a mentoring capacity; that's being a dude on Twitter. So, I'm trying to figure out the way of solving for that, and I don't know if there is an answer. What's your take?Guang Ming: I think that for many young people, there is a baseline level of information that they need, that almost any mentor can give, but it takes up a lot of time to get to that point. So, for example, I had a young woman reach out to me, and she wanted to get a foot in the door in the legal world and wanted some advice. And I couldn't. It was like, pulling teeth. I couldn't get her to say a word about herself. And our conversation lasted less than five minutes because I couldn't get her to speak about herself.And I almost let it end at that. But then I circled back with her a week later, and called her and said, “You know, I'm going to connect you to someone because I want to help you in your journey. But I need you to think before you get to that conversation about who you are, what you want, where you're going, what's your story. You know, I know just from the person who connected us that you're the first in your family to go to college. Speak to that.” And just really tried to help her understand that she needed to craft a narrative around herself. And I think a lot of young adults don't know how to craft that narrative.Corey: The problem that I see when I look at this systemically is that all of this stuff seems like it's very bespoke. It's [spreading an 00:15:45] opportunity, but it is incumbent upon folks to learn about it for themselves. One of the most foundational memories of my ill-fated academic career was in public school for my first sophomore year of high school, where the US history teacher said, all right. Today, we're not doing our traditional stuff, what I'm about to do is not in the curriculum. Please feel free to complain to your parents and then have them take it to the school board.And what he did was he passed at a flyer where each one of us had different numbers on it, and it was a, “You are a family of x number of people; you made this much money last year.” And then he passed out 1040-EZ forms. And he taught us how to file a tax return in the course of that 45-minute session. And it was, instead of learning a series of whitewashed facts about American history, I was learning how to function as an adult in society. And the fact that he had to do this almost as a subversive thing as opposed to being an accepted part of the curriculum is just mind-boggling to me. I see what you're doing is important and valuable, but it also in some level kind of feels like a band-aid over a massive societal failing. Is that accurate or am I missing something?Guang Ming: No, I think that certain school districts are trying to do this, they're trying to integrate financial literacy into calculus. Some schools will even offer a course, but the course isn't an AP course; it doesn't give you special credit, and so students don't take it, or it's viewed as a less valuable course even though it's probably the most valuable course. And there's also a level of embarrassment. Like, for certain things, we cover personal hygiene. The importance of brushing your teeth every day, and taking a shower, and wearing deodorant.Which is something you wouldn't necessarily think you would need to teach someone, but wait till you're in certain work environments, and that is actually something that people need to know that they're bothering their coworkers by this lack. That can be really embarrassing.With Lattice Climbers, you can do this in the privacy of your own home, you can do it in your bedroom, you can do it wherever you are, and you can get these little lessons and not feel embarrassed. Or sometimes you are afraid to ask a question because you feel dumb asking it. When we did a pilot with 17-to 19-year-olds, the favorite video was actually making an appointment. Just giving tips on how to gather the appropriate documentation you would need to, say for example, make a doctor's appointment, and sample scripts—we have downloadables that go along with—sample scripts of how a conversation would potentially run if you were to call.Corey: The way you describe this and the problem you're solving, I have a hard time seeing this as the business opportunity that becomes a $60 billion company because to do that, you would have to do something that is abjectly terrifying. So apparently, becoming rich beyond the wildest dreams of avarice is not the reason that you're doing this. What made you decide that this was a problem you wanted to address?Guang Ming: So, I am the daughter of an immigrant and a first-generation college student. And there were so many things that my parents just didn't know to teach me. They were very focused on academics and there was no focus on anything outside of book smarts. So, when I had my first college interview, my mom took me to the Fashion for Price Boutique next to the Drug Emporium in the strip mall near our home and bought me an interview suit—we didn't have a ton of money—and the interview suit involved zebra print zippers and a very short skirt. And that is what I wore to my Harvard interview. The one [laugh] school I didn't get into.And not only that, not only was I dressed wholly inappropriately, I also was a deer in the headlights. I had never done a mock interview, I had never done anything that would help prepare me for this situation. And I look back at that 17-year-old and I think, “How can I help her? How can I help people like her who don't have the social or cultural capital to know these things, to know how to move in the world that they want to be in desperately? How do I help them overcome that obstacle?” And that is how Lattice Climbers was born.Corey: The idea of having an experience like that as being necessary to forge this is—it's moving. It's the sort of thing that you hear about other people—you [unintelligible 00:20:04] secondhand cringing from hearing that sort of story, at least I do. And I can definitely understand not wanting other folks to have to go through this. We talk about hilarious interview mistakes that we've made, that we've had candidates make, and in some cases, most of the ones that I like to talk about are the folks who are—let's [unintelligible 00:20:23] here—25 years into their career or so, where they really should know better. Because making fun of some naive kid who'd never been in an interview scenario before is just being shitty, let's be clear.At some point, though, you should learn how to comport yourself in a working environment that makes sense. But without having mentorships or guidance like that, it feels like a lot of people have stories like this. I think what makes your story different than most of them is that you're willing to talk about it in public. Most of us bury those things down the memory hole, I would think.Guang Ming: Yes. I very much own the zebra print story, and it is something that I share when I speak at Girls State. I speak at Girls State just about every year to the young women, and I talk a lot about some of these things that we go over in Lattice Climbers to just try to impart, even in a six-minute speech, some of the key nuggets that I want them to take away with them, as they move through life.Corey: Tell me a little more about Girls State. I've heard the term a couple of times, but know remarkably little about it because, for better or worse, my daughters are still at a point where—I regret this constantly—I have to know entirely too much about the Paw Patrol.Guang Ming: So, Girls Day is a program sponsored by the American Legion Auxiliary. It is a week-long civic engagement program that simulates government over the course of one week. And for California Girls State, it is one girl from each sponsored high school, with about 540 young women—and of course, we've had to be virtual for the past couple of years, but they've done it in a webinar virtual sessions—and the program is all about women empowerment and encouraging civic engagement. And one of the things that has really impacted Lattice Climbers has been my observations in Girls State as a counselor for over 20 years. Because we work with young women from all different backgrounds, whether their parents are migrant workers in Modesto or doctors in Big Sur, there are gaps that these young women have that differ based on their backgrounds.And what I'm hoping to do with Lattice Climbers is fill those gaps and help them avoid these missteps and increase their trajectory as they climb the lattice. And that is one thing that we do is we don't talk about climbing the ladder because a ladder implies that there is one pathway to the top, there's room for only one there. We approach it as you're climbing a lattice: we're all in it together and there are infinite paths to success.Corey: All of these things that you talk about are challenging at the best of times, and these are very clearly not the best of times. One of the reasons that, to date, we at The Duckbill Group have not hired junior folks is because in a full-remote environment—and to be clear, even without the pandemic The Duckbill Group has been full-remote since its inception—I don't know that's necessarily the best way to expose someone new to the workforce. It feels to me like there's not a lot of examples around there. There's a requirement to be a lot more self-directed, and it's likely, for example, that someone will get stuck and spin on something for a while rather than asking for help because they don't want to appear like they don't know what they're doing and inadvertently make things worse. Do you think that remote as we move forward is going to be an increasing burden on folks like this, or—which I'm perfectly willing to accept—am I completely wrong, and that in fact having a full-remote environment like this is in fact a terrific opportunity for folks new to the workforce?Guang Ming: No, I think full-remote is an issue. I think that it takes so much more emotional energy to connect through a video than it does to connect in person. And there's also the lack of organic interactions. There are so many mentorships that develop just from walking down the hall and running into someone over coffee, or at the wat—I mean literally at the watercooler and having the opportunity to chat with someone about something non-work-related that can then evolve into a mentoring relationship. And there is just a lack of that.All these young people entering the work environment, they can wear pajamas all day and lay in bed with their laptop on their laps and work, and they may love that, but I think that if you want to work in a professional office environment, you need to understand appropriate attire, you need to understand appropriate behavior at events. I think that, especially if you're from certain backgrounds and you've never been around an open buffet before, it can be very tempting to just pile that plate as high as you can with crab legs or, you know, shrimp cocktail. And it's not appropriate in that setting. And so we cover those—Corey: Wait. It's not?Guang Ming: [laugh]. Well, it depe—if you're the CEO of the company, Corey, you can do whatever you want.Corey: No, no. That's my business partner. I am just the chief cloud economist because it's not professional to put the word shit-poster on a business card. Or so they tell me.Guang Ming: [laugh].Corey: In my experience, the worst of all worlds, though, is not the full-remote; it's not the in-office; it's the hybrid scenario where you have some people that are in an office together working and then you have folks who are remote, and regardless of what your intentions are, it is almost impossible to avoid having a striated structure where the in-person folks collaborate in different ways and make decisions informally to which remote folks are not privy. And it's not to do with cliques or anything like that, but the watercooler discussions, or, “I'm going to go grab lunch. Do you want to come with me?” Type of engagement stories. And I can't shake the feeling that remote really needs to be all or nothing, at least within the bounds of a team, if not company-wide.Guang Ming: I think that a hybrid version could work if there was a concerted effort to include the remote individuals if there was a scheduled Zoom happy hour. So, one of the things that happened during the COVID times is there's a group that I'm a part of, and we just had a happy hour on Saturday nights. At 8 p.m. everyone just kind of logged on and hung out for a period of time. And it was really good to connect with people in that casual environment; there wasn't always pressure to speak, there wasn't always pressure to perform, it was just being together and having that togetherness. So, I think that in a work environment, you could create opportunities for that. And then also, I think, bringing people into the office for specific meetings and things that are important like that. And then potentially—I don't like assigning mentors, but I think you almost have to assign mentors when you have remote workforce.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: The challenge also becomes one of, great for junior folks, that makes an awful lot of sense. Hire someone with 15 years of experience, and, “Oh, we're going to assign you a mentor here.” And they're like, “Oh, really. So, that's what condescending means. I was always looking for a perfect example.”Guang Ming: [laugh].Corey: That's a delicate balance to strike in my experience.Guang Ming: Oh, very true. Very true. I was thinking more of, like, the young adults starting out in their career because our focus is really early career, as well as young adults in high school.Corey: And that's, I guess, my question for you next is why is that the target age range that you think is best served by this? Now, having, again, spent too much time gazing into the mess that is the Paw Patrol, I understand why preschoolers are not the target market for this, but my approach has generally been targeting folks who are entering the workforce. Although let's be very clear, a large part of that is because I generally don't appreciate the optics of going and hanging out at the local high school trying to talk to kids.Guang Ming: Well, I think that high school is really where it starts. This is the age at which brains are starting to develop a little bit more; they're starting to have more social awareness. This is where beginnings of your network are important. And I think that the sooner that we can convey to young people that they're only as strong as their networks, the better. If they can understand that it's their teachers, their coaches, the parents of their friends are all the beginnings of their network.That's how you get internships, that's how you get a leg up is through these connections because if you're just a resume floating out there, your chances of getting looked at—and we all know how the world works—well, we should all know how the world works, which is it's all about your connections that helps you launch to the next thing.Corey: That's the thing that I think is understated in this is that we wind up telling students a whole bunch of things that are well-intentioned lies. The, “Oh, put your nose to the grindstone and work hard, and one day you will surely be promoted.” Now, I get flack when I say this sometimes from folks who've been at the same company for 15 years and demonstrated growth trajectory internally, but that's the exception, not the rule. Big moves generally look a lot like transitioning between companies.“Oh, you don't want to be a job hopper. It looks bad on the resume.” Yeah, you know who says that? People who don't want you to quit your job because you're unhappy because then they have to backfill you, or people who are trying to recruit you in and want to make sure that you when you show up at this new job, you stay there for a while. It's self-serving.Yeah, there's going to be some questions about it in the interview process, but you should have an answer ready to go for it. It's the interview skills piece of it and make sure that you don't inadvertently torpedo your own candidacy with conversations like that. And this is stuff that I find that is—it's not just the newer generation that we're talking about here; people well into their careers still haven't cracked a lot of these codes, mostly because, for better or worse, it turns out that people aren't nearly as cynical [laugh] about things as I am.Guang Ming: Well, and we also cover things like how to leave a job professionally. Because as we live in a world where you're not going to go work for one company for the next 30 years, or where you shouldn't go work for the same company for 30 years necessarily, but there are stories out there of people just ghosting on the job, ghosting on job interviews, and that burns bridges. And everyone you meet is a potential connection in your network as you climb the lattice, and so you need to preserve those relationships moving forward because you never know who you help out along the way, or who helps you out along the way, you never know how that connection is going to play out later on in life.Corey: That's the trick is that it's talking to people and being friendly with them. And there are ways to do networking properly in my world, and there are ways not to. And, “Oh, I should talk to you because down the road you might be useful to me,” is just cynical and terrible. I hate the pattern.Whereas, I like keeping in touch with people because I find them interesting. My default assumption has always been that I'm going to be talking to someone for longer than either one of us is going to be doing whatever it is we're currently doing, and trying to treat relationships as transactional is a mistake. But that's what networking is often interpreted as.Guang Ming: It's so true. And people can tell. They can tell when you're being fake. They can tell when you're being transactional. They can tell when you just are waiting for the ask.I think it actually is really hard to be genuine and natural for some people that comes across as transactional, and one of the ways that we talked about avoiding that is through just an ongoing relationship. So, you don't only reach out to the person when you have an ask, you reach out to the person quarterly. And you can have a spreadsheet—almost—about it, and of the people that you want to contact and maintain cont—and even if it's just a text message that says, “Hey, this is what I'm up to. Hope all is well with you.” And even if they don't respond, or just it's a one-word answer, you've at least had that touchpoint with them over the course of time.Corey: There's often a criticism levied at folks who are advocating for networking, that it is a lot harder when you're an introvert or when you are neurodivergent, in certain ways. To be clear, I've neurodivergent in ways that do not directly negatively impact my ability to socialize with folks; it just means they think I'm a jerk. But there are folks who definitely have different expressions of different divergences. And that's fine. How do you view the networking aspect for folks who do not work nearly as well interpersonally?Guang Ming: That's so hard because interpersonal skills are something that is so necessary, and I think that unfortunately, there are people who get by one hundred percent on their social skills. Like, their people skills are all they need to move forward in the world. And I think that you have to work at it, and you have to study how to behave in those situations. It's almost like—so for example, my husband is an introvert, but he was also an actor in college. And when he goes into these situations, it's almost like putting on a show.Like you talked about putting on your three-piece suit. There is the extrovert persona that he wears in these environments, and then he takes it off when he gets home. And I think that you almost have to create that persona for yourself. And you can acknowledge that you're neurodivergent, and you can acknowledge that you're an introvert, and I think that's way more acceptable these days than it used to be. And there are lots of people that are in the world that are neurodivergent and are introverts, and so I think it's completely fine to be that way.Corey: I've never had a good answer for folks who ask those questions, just because it is so different from my lived experience that I don't have an answer that's worth listening to, and I try very hard to stay in my lane. I don't ever want that to be interpreted as it's not important because it very much is.So, one last question I have for you is I love, love, love your zebra print suit story, but it's also back when you were applying to school, back in early career, which you are very clearly not now; it's decades old. Do you have any other similar stories from folks that you've been working through, either at Lattice Climbers or through Girls State, that illustrate this in a somewhat more modern era?Guang Ming: Oh, absolutely. So, there was a young woman at Girls State; we were all in a room and they were talking about colleges, the girls were talking about colleges. And this one young woman remains silent during this conversation. And so I approached her. I said, “Well, what about you? What are your plans for college?” And she shared that she wasn't going to go to college because her parents didn't go to college, they didn't have a lot of money, and she just didn't think that college was in the cards for her.And I disabused her of that notion. I told her absolutely not. You're here at Girls State, which means you're the top girl from your high school. You absolutely should go to college, and I told her that there were so many paths to college. You could go to community college and then transfer; you could go to technical college; there's so many different options.And there's so many scholarships out there, especially for low-income individuals. Well, we became friends on social media, and about three years after Girls State—because they attend the summer after their junior years—I received a message from her, sort of, out of the blue, and she let me know that that conversation that we had changed her life because she had gone to community college—she had taken my advice; she had gone to community college and had just been accepted as a transfer student to UC Berkeley. And that story just makes me tear up every time I think about it. And that one conversation had that huge impact on her life, and I'm hoping that through Lattice Climbers and our little lessons, that we can have that kind of impact on young lives, that we can help them avoid these missteps that could have huge impacts on their trajectory, and we could help them increase their trajectory on the lattice.Corey: It's similar in some respects to the folks I talk to who are building products for the cloud industry. It's, “Yes, yes, of course. You're always going to have a story about how it works for you. That's fine. Let's talk about your customers.” Like, “Find me a customer, someone else in the world who has a story like this that really demonstrates the value you provide.”And I love the fact that it is so easy for you to come up with these things off the top of your head, even when you weren't necessarily expecting the question. So, you're onto something. This is a clear problem, and it's not going away anytime soon, and it's largely underserved because there's no opportunity to invest venture capital into it and make a ridiculous return on that investment because there's not money in solving it that I can see—and apparently, most the industry can see—compared to another Twitter for Pets app.Guang Ming: [laugh]. Well, there is not that much money in them there hills because no one owns the problem, and because no one owns the problem, it's very hard to find people willing to pay to solve the problem. But that doesn't mean that the problem isn't there and that doesn't mean that it doesn't need to be solved. And I actually think that companies should have an incentive to do it because it will help with employee retention, it will help with employee performance if they do invest in their workers, and in high school students who, the sooner that they know these things, the better it will be for their long-term careers.Corey: And if nothing else, I think that's the lesson to take away from this for the young folk—the youth, as it were—that this is the single greatest thing I look at and credit my professional trajectory has been in learning to handle expectations in corporate environments. And sure, I have fun with them and I play games with them, but you have to know the rules before you can break them in this context. And there are business meetings in which I assure you, you would question whether it was the same person. And that's what it comes down to, I think, on some level is, if you know how to handle a job interview, you will always be able to find something to put food on the table. Conversely, if you're terrific at any number of different things, but absolutely cannot handle the dynamics of a job interview, you are going to struggle to find work anywhere until you find someone willing to alter their corporate process just in order to bring you aboard.It's a skill that you need to be at least conversant with. And what makes it even worse, as it's a skill that you only really get to practice when you're looking for jobs. I want to thank you for taking the time to speak with me so much about all this stuff. If people want to learn more about what you're up to and how you're approaching it, where can they find you?Guang Ming: So, we are at latticeclimbers.com. And we are currently in waitlist mode, so you can sign up on our waitlist and get more information about when we're ready to launch. We are working with some nonprofits and some school districts on some pilot programs, and we're hoping to have that going, hopefully by the end of the year.Corey: And we will, of course, put a link to that in the [show notes 00:38:07]. Thank you so much for taking the time to speak with me today. I really appreciate it.Guang Ming: Well, thank you so much for having me. I really appreciate the opportunity to share what we're doing with Lattice Climbers, and I just hope, like I said, if I can get one person to not wear zebra print to that Harvard interview, [laugh] then I will view Lattice Climbers as a success.Corey: [laugh]. Excellent. Thank you so much. Once again, Guang Ming Whitley, co-founder of Lattice Climbers. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, along with a rambling comment explaining why we're wrong and that a zebra-print suit for a college interview is in fact a best practice.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About MatthewMatthew Prince is co-founder and CEO of Cloudflare. Cloudflare's mission is to help build a better Internet. Today the company runs one of the world's largest networks, which spans more than 200 cities in over 100 countries. Matthew is a World Economic Forum Technology Pioneer, a member of the Council on Foreign Relations, winner of the 2011 Tech Fellow Award, and serves on the Board of Advisors for the Center for Information Technology and Privacy Law. Matthew holds an MBA from Harvard Business School where he was a George F. Baker Scholar and awarded the Dubilier Prize for Entrepreneurship. He is a member of the Illinois Bar, and earned his J.D. from the University of Chicago and B.A. in English Literature and Computer Science from Trinity College. He's also the co-creator of Project Honey Pot, the largest community of webmasters tracking online fraud and abuse.Links: Cloudflare: https://www.cloudflare.com Blog post: https://blog.cloudflare.com/aws-egregious-egress/ Bandwidth Alliance: https://www.cloudflare.com/bandwidth-alliance/ Announcement of R2: https://blog.cloudflare.com/introducing-r2-object-storage/ Blog.cloudflare.com: https://blog.cloudflare.com Duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Writing ad copy to fit into a 30 second slot is hard, but if anyone can do it the folks at Quali can. Just like their Torque infrastructure automation platform can deliver complex application environments anytime, anywhere, in just seconds instead of hours, days or weeks. Visit Qtorque.io today and learn how you can spin up application environments in about the same amount of time it took you to listen to this ad.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. Today, my guest is someone I feel a certain kinship with, if for no other reason than I spend the bulk of my time antagonizing AWS incredibly publicly. And my guest periodically descends into the gutter with me to do the same sort of things. The difference is that I'm a loudmouth with a Twitter account and Matthew Prince is the co-founder and CEO of Cloudflare, which is, of course, publicly traded. Matthew, thank you for deigning to speak with me today. I really appreciate it.Matthew: Corey, it's my pleasure, and appreciate you having me on.Corey: So, I'm mostly being facetious here, but not entirely, in that you have very publicly and repeatedly called out some of the same things I love calling out, which is AWS's frankly egregious egress pricing. In fact, that was a title of a blog post that you folks put out, and it was so well done I'm ashamed I didn't come up with it myself years ago. But it's something that is resonating with a large number of people in very specific circumstances as far as what their company does. Talk to me a little bit about that. Cloudflare is a CDN company and increasingly looking like something beyond that. Where do you stand on this? What got you on this path?Matthew: I was actually searching through really old emails to find something the other day, and I found a message from all the way back in 2009, so actually even before Michelle and I had come up with a name for Cloudflare. We were really just trying to understand the pricing on public clouds and breaking it all down. How much does the compute cost? How much does storage cost? How much does bandwidth cost?And we kept running the numbers over and over and over again, and the storage and compute costs actually seemed relatively reasonable and you could understand it, but the economics behind the bandwidth just made no sense. It was clear that as bandwidth usage grew and you got scale that your costs eventually effectively went to zero. And I think it was that insight that led to us starting Cloudflare. And the self-service plans at Cloudflare have always been unlimited bandwidth, and from the beginning, we didn't charge for bandwidth. People told us at the time we were crazy to not do that, but I think that that realization, that over time and at scale, bandwidth costs do go to zero is really core to who Cloudflare is.Cloudflare launched a little over 11 years ago now, and as we've watched the various public clouds and AWS in particular just really over that same 11 years not only not follow the natural price of bandwidth down, but really hold their costs steady. At some point, we've got a lot of mutual customers and it's a complaint that we hear from our mutual customers all the time, and we decided that we should do something about it. And so that started four years ago, when we launched the Bandwidth Alliance, and worked with almost all the major public clouds with the exception of Amazon, to say that if someone is sending traffic from a public cloud network to Cloudflare's network, we're not going to charge them for the bandwidth. It's going across a piece of fiber optic cable that yeah, there's some cost to put it in place and maybe there's some maintenance costs associated with it, but there's not—Corey: And the equipment at the end costs money, but it's not cloud cost; it just cost on a per second, every hour of your lifetime basis. It's a capital expense that is amortized across a number of years et cetera, et cetera.Matthew: And it's a fixed cost. It's not a variable cost. You put that fiber optic cable and you use a port on a router on each side. There's cost associated with that, but it's relatively de minimis. And so we said, “If it's not costing us anything and it's not costing a cloud provider anything, why are we charging customers for that?”And I think it's an argument that resonated with almost every other provider that was out there. And so Google discounts traffic when it's sent to us, Microsoft discounts traffic when it's sent to us, and we just announced that Oracle has joined this discounting their traffic, which was already some of the most cost-effective bandwidth from any cloud provider.Corey: Oh, yeah. Oracle's fantastic. As you were announced, I believe today, the fact that they're joining the Bandwidth Alliance is both fascinating and also, on some level, “Okay. It doesn't matter as much because their retail starting cost is 10% of Amazon's.” You have to start pushing an awful lot of traffic relative to what you would do AWS before it starts to show up. It's great to see.Matthew: And the fact that they're taking that down to effectively zero if you're using us is even better, right? And I think it again just illustrates how Amazon's really alone in this at being so egregious in how they do that. And it's, when we've done the math to calculate what their markups are, it's almost 80 times what reasonable assumptions on what their wholesale costs are. And so we really do believe in fighting for our customers and being customer-centric, and this seems like a place where—again, Amazon provides an incredible service and so many things, but the data transfer costs are just completely outrageous. And I'm glad that you're calling them out on it, and I'm glad we're calling them out on it and I think increasingly they look isolated and very anti-customer.Corey: What's interesting to me is that ingress to AWS at all the large public tier-one cloud providers is free. Which has led, I think, to the assumption—real or not—that bandwidth doesn't actually cost anything, whereas going outbound, all I can assume is that one day, some Amazon VP was watching a rerun of Meet the Parents and they got to the line where Ben Stiller says, “Oh, you can milk anything with nipples,” and said, “Holy crap. Our customers all have nipples; we can milk them with egress charges.” And here we are. As much as I think the cloud empowers some amazing stuff, the egress charges are very much an Achilles heel to a point where it starts to look like people won't even consider public cloud for certain workloads based upon that.People talk about how Netflix is a great representation of the ideal AWS customers. Yeah, but they don't stream a single byte to customers from AWS. They have their own CDN called Open Connect that they put all around the internet, specifically for that use case because it would bankrupt them otherwise.Matthew: If you're a small customer, bandwidth does cost something because you have to pay someone to do the work of interconnecting with all of the various networks that are out there. If you start to be, though, a large customer—like a Cloudflare, like an AWS, like an Azure—that is sending serious traffic to the internet, then it starts to actually be in the interest of ISPs to directly interconnect with you, and the costs of your bandwidth over time will approach zero. And that's the just economic reality of how bandwidth pricing works. I think that the confusion, to some extent, comes from all of us having bought our own home internet connection. And I think that the fact that you get more bandwidth up in most internet connections, and you get down, people think that there's some physics, which is associated with that.And there are; that turns out just to be the legacy of the cable system that was really designed to send pictures down to your—Corey: It wasn't really a listening post. Yeah.Matthew: Right. And so they have dedicated less capacity for up and again, in-home network connections, that makes a ton of sense, but that's not how internet connections work globally. In fact, you pay—you get a symmetric connection. And so if they can demonstrate that it's free to take the traffic in, we can't figure out any reason that's not simply about customer lock-in; why you would charge to take data out, but you wouldn't charge to put it in. Because actually cost more from writing data to a disk, it costs more than reading it from a disk.And so by all reasonable accounts, if they were actually charging based on what their costs were, they would charge for ingress but they want to charge for egress. But the approach that we've taken is to say, “For standard bandwidth, we just aren't going to charge for it.” And we do charge for if you use our premium routing services, which is something called Argo, but even then it's relatively cheap compared with what is just standard kind of internet connectivity that's out there. And as we see more of the clouds like Microsoft and Google and Oracle show that this is a place where they can be much more customer-centric and customer-friendly, over time I'm hopeful that will put pressure on Amazon and they will eliminate their egress fees.Corey: People also tend to assume that when I talk about this, that I'm somehow complaining about the level of discounting or whatnot, and they yell at me and say, “Oh, well, you should know by now, Corey, that no one at significant scale pays retail pricing.” “Thanks, professor. I appreciate that, but four years ago, or so I sat down with a startup founder who was sketching out the idea for a live video streaming service and said, ‘There's something wrong with my math because if I built this on AWS—which he knew very well, incidentally—it looks like it would cost me at our scale of where we're hoping to hit $65,000 a minute.'” And I checked and yep, sure enough, his math was not wrong, so he obviously did not build his proof of concept on top of AWS. And the last time I checked, they had raised several 100 million dollars in a bunch of different funding rounds.That is a company now that will not be on AWS because it was never an option. I want to talk as well about your announcement of R2, which is just spectacular. It is—please correct me if I get any of this wrong—it's an object store that lives in your existing distributed-points-of-presence-slash-data-centers-slash-colo-slash-a-bunch-of-computers-in-fancy-warehouse-rooms-with-the-lights-are-always-on-And-it's-always-cold-and-noisy. And people can store data there—Matthew: [crosstalk 00:10:23] aisles it's cold; in the other aisles, it's hot. But yes.Corey: Exactly. But it turns out when you lurk around to the hot aisle, that's not where all the buttons are and the things you're able to plug into, so it's freeze or sweat, and there's never a good answer. But it's an object store that costs a fair bit less than retail pricing for Amazon S3, or most other object stores out there. Which, okay, great. That's always good to see competition in the storage space, but specifically, you're not charging any data transfer costs whatsoever for doing this. First, where did this come from?Matthew: So, we needed it ourselves. I think all of the great products at Cloudflare start with an internal need. If you look at why do we build our zero-trust solutions? It's because we said we needed a security solution that was fast and reliable and secure to protect our employees as they were going out and using the internet.Why did we build Cloudflare Workers? Because we needed a very flexible compute platform where we could build systems ourselves. And that's not unique to us. I mean, why did Amazon build AWS? They built it because they needed those tools in order to continue to grow and expand as quickly as possible.And in fact, I think if you look at the products that Google makes that are really great, it ends up being the ones that Google's employees use themselves. Gmail started as Caribou once upon a time, which was their internal email system. And so we needed an object store and the sometimes belligerent CEO of Cloudflare insisted that our team couldn't use any of the public cloud object stores. And so we had to build it.That was the start of it and we've been using it internally for products over time. It powers, for example, Cloudflare Images, it powers a lot of our streaming video services, and it works great. And at some point, we said, “Can we take this and make it available to everyone?” The question that you've asked on Twitter, and I think a lot of people reasonably ask us, “What's the catch?”Corey: Well, in my defense, I think it's fair. There was an example that I gave of, “Okay, I'm going to go ahead and keep—because it's new, I don't trust new object stores. Great. I'm going to do the same experiment twice, keep one the pure AWS story and the other, I'm just going to add Cloudflare R2 to the mix so that I have to transfer out of AWS once.” For a one gigabyte file that gets shared out for a petabyte's worth of bandwidth, on AWS it costs roughly $52,000 to do that. If I go with the R2 solution, it cost me 13 cents, all of which except for a penny-and-a-half are AWS charges. And that just feels—when you're looking at that big of a gap, it's easy to look at that and think, “Okay, someone is trying to swindle me somewhere. And when you can't spot the sucker, it's probably me. What's the catch?”Matthew: I guess it's not really a catch; it's an explanation. We have been able to drive our bandwidth costs down low enough that in that particular use case, we have to store the file, and that, again, that—there's a hard disk in there and we replicate it to make sure that it's available so it's not just one hard disk, but it's multiple hard disks in various places, but that amortized over time, isn't that big a cost. And then bandwidth is effectively zero. And so if we can do that, then that's great.Maybe a different way of framing the question is like, “Why would we do that?” And I think what we see is that there is an opportunity for customers to be able to use the best of various cloud providers and hook the different parts together. So, people talk about multi-cloud all the time, and for a while, the way that I think people thought about that was you take the exact same workload and you run it in Azure and AWS. That turns out not to be—I mean, maybe some people do that, but it's super rare and it's incredibly hard.Corey: It has been a recurring theme of most things I say where, by default, that is one of the dumbest things I can imagine.Matthew: Yeah, that isn't good. But what people do want to do is they want to say, “Listen, there's some really great services that Amazon provides; we want to use those. And there's some really great services that Azure provides, and we want to use those. And Google's got some great machine learning, and so does IBM. And I want to sort of mix and match the various pieces together.”And the challenge in doing that is the egress fees. If everyone just had a detente and said there's going to be no egress fees for us to be able to hook these various [pits 00:14:48] together, then you would be able to take advantage of a lot of the different technologies and we would actually get stronger applications. And so the vision of what we're trying to build is how can we be the fabric that can stitch the various cloud providers together so that you can do that. And when we looked at that, and we said, “Okay, what's the path to getting there?” The big place where there's the just meatiest cost on egress fees is object stores.And so if you could have a centralized object store, and you can say then from that object go use whatever the best service is at Amazon, go use whatever the best service is at Google, go use whatever the best service is at Azure, that then allows, I think, actually people to take advantage of the cloud in a way which is what people really should mean when they talk about multi-cloud. Which is, there should be competition on the various features themselves, and you should be able to pick and choose the best of all of the different bits. And I think we as consumers then benefit from that. And so when we're looking at how we can strategically enable that future, building an object store was a real key part of that, and that's part of what we're doing. Now, how do we make money off of that? Well, there's a little bit off the storage, and again, even [laugh]—Corey: Well, that is the Amazonian answer there. It's like, “Your margin is my opportunity,” is a famous Bezos quote, and I figure you're sitting there saying, “Ah, it would cost $52,000 to do that in Amazon. Ah, we can make a penny-and-a-half.” That's very Amazonian, you could probably get hired over there with that philosophy.Matthew: Yeah. And this is a commodity service, just [laugh] storing data. If you look across the history of what Cloudflare has done, in 2014, we made encryption free because it's absurd to pay for math, right? I mean, it's just crazy right?Corey: Or to pay for security as a value-add. No, that should be baked into whatever you're doing, in an ideal world.Matthew: Domain registration. Like, it's writing something down in a ledger. It's a commodity; of course it should go to whatever the absolute cost is. On the other hand, there are things that we do that aren't commodities where we are able to better protect people because we see so much traffic, and we've built the machine learning models, and we've done those things, and so we charge for those things. So commodities, we think over time, go to effectively, whatever their cost is, and then the value is in the actual intelligent services that are on top of it.But an object store is a commodity and so we should be trying to drive that pricing down. And in the case of bandwidth, it's effectively free for us. And so if we can be that fabric that connects the different class together, I think that makes sense is a strategy for us and that's why R2 made a ton of sense for us to build and to launch.Corey: There seems to be a lack of ability for lots of folks, at least on the internet to imagine a use case other than theirs. I cheated by being a consultant, I get to borrow other people's use cases at a high degree of turnover. But the question I saw raised was, “Well, how many workloads really do that much egress from static objects that don't change? Doesn't sound like there'd be a whole lot of them.” And it's, “Oh, my sweet summer child. Sure, your app doesn't do a lot of that, but let me introduce it to my friends who are hosting videos on their website, for example, or large images that get accessed a whole bunch of times; things that are written once and then read forever by the internet.”Matthew: And we sit in a position where because of the role that Cloudflare plays where we sit in front of a number of these different cloud providers, we could actually look at the use cases and the data, and then build products in order to solve that. And that's why we started with Workers; that's why we then built the KV store that was on top of that; we built object-store next. And so you can see as we're sort of marching through these things, it is very much being informed by the data that we actually see from real customers. And one of the things that I really like about R2 is in exactly the example that you gave where you can keep everything in S3; you can set R2 in front of it and put it in slurp mode, and effectively it just—as those objects get pulled out, it starts storing them there. And so the migration path is super easy; you don't have to actually change anything about your application and will cut your bills substantially.And so I think that's the right thing to enable a multi-cloud world where, again, it's not you're running the exact same workload in different places, but you get to take advantage of the really great tack that all of these companies are building and use that. And then the companies will compete on building that tech well. So, it's not just about how do I get the data in and then kind of underinvest in all of the different services that I provide. It's how can we make sure that on a service-by-service basis, you actually are having real competition over time. And again, I think that's the right thing for customers, and absolutely R2 might not be the right thing for every use case that's out there, but I think that it wi—enabling more competition is going to make the cloud better for everyone.Corey: Oh, yeah. It's always fun hearing it from Amazonians. It's, “You have a service that talks to satellites in orbit. You really think that's a general-purpose thing that every company out there has to deal with?” No. Well, not yet, anyway.It also just feels to me like their transfer approach is antithetical to almost every other aspect of how they have built their cloud. Amazonians have told me repeatedly—I believe them—that their network is effectively magic. The fact that you can get near line rate between any two points without melting various [unintelligible 00:20:14], which shows that there was significant thought, work, effort, planning, technology, et cetera, put into the network. And I don't dispute that. But if I'm trying to build a workload and put it inside of AWS, I can control how it performs tied to budget; I can have a lot of RAM for things that are memory intensive, or I can have a little RAM; I can have great CPU performance or terrible CPU performance.The challenge with data transfer is it is uniformly great. “I want to get that data over there super quickly.” Yeah, awesome. I'm fine paying a premium for that. But I have this pile of data right here. I want to get it over there, ideally by Tuesday. There's no good way to do that, even with their Snowball—or Snow Family devices—when you fill them with data and send them into AWS, yeah, that's great. Then you just pay for the use of the device.Use them to send data out of AWS, they tack on an additional per-gigabyte fee for getting the data out. You're training as a lawyer, you went to the same law school that my wife did, the University of Chicago, which, oh, interesting stories down that path. But if we look at this, my argument is that the way to do an end-run around this is to sue Amazon for something, and then demand access to the data you have living in their environment during discovery. Make them give it to you for free, though, they'd probably find a way to charge it there, too. It's just a complete lack of vision and lack of awareness because it feels like they're milking a cash cow until it dies.Matthew: Yeah, they probably would charge for it and you'd also have to pay a lot of lawyers. So, I'm not sure that's the cost [crosstalk 00:21:44]—Corey: Its only works above certain volumes, I figure.Matthew: I do think that if your pricing strategy is designed to lock people in to prevent competition, then that does create other challenges. And there are certainly some University of Chicago law professors out there that have spent their careers arguing why antitrust laws don't make any sense, but I think that this is definitely one of those areas where you can see very clearly that customers are actually being harmed by the pricing strategy that's there. And the pricing strategy is not tied in any way to the underlying costs which are associated with that. And so I do think that, especially as you see other providers in the space—like Oracle—taking their bandwidth costs to effectively zero, that's the sort of thing that I think will have regulators start to scratch their heads. If tomorrow, AWS took egress costs to zero, and as a result, R2 was not as advantaged as it is today against them, you know, I think there are a lot of people who would say, “Oh, they showed Cloudflare.” I would do a happy dance because that's the best thing [thing they can do 00:22:52] for our customers.Corey: Our long-term goals, it sounds like, are relatively aligned. People think that I want to see AWS reign ascendant; people also say I want to see them burning and crashing into the sea, and neither one of those are true. What I want is, I want someone in a few years from now to be doing a startup and trying to figure out which cloud provider they should pick, and I want that to be a hard decision. Ideally, if you wind up reducing data transfer fees enough, it doesn't even have to be only one. There are stories that starts to turn into an actual realistic multi-cloud story that isn't, at its face, ridiculous. But right now, you have to pick a horse and ride it, for a variety of reasons. And I don't like that.Matthew: It's entirely egress-based. And again, I think that customers are better off if they are able to pick who is the best service at any time. And that is what encourages innovation. And over time, that's even what's good for the various cloud providers because it's what keeps them being valuable and keeps their customers thinking that they're building something which is magical and that they aren't trapped in the decision that they made, which is when we talk to a lot of the customers today, they feel that way. And it's I think part of why something like R2 and something like the Bandwidth Alliance has gotten so much attention because it really touches a nerve on what's frustrating customers today. And if tomorrow Amazon announced that they were eliminating egress fees and going head-to-head with R2, again, I think that's a wonderful outcome. And one that I think is unlikely, but I would celebrate it if it happened.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: My favorite is people who don't do research on this stuff. They wind up saying, “Oh, yeah. Cloudflare is saying that bandwidth is a fixed cost. Of course not. They must be losing their shirt on this.”You are a publicly-traded company. Your gross margins are 76% or 77%, depending upon whether we're talking about GAAP or non-GAAP. Point being, you are clearly not selling this at a loss and hoping to make it up in volume. That's what a VC-backed company does. Is something that is real and as accurate.I want to, on some level, I guess, low-key apologize because I keep viewing Cloudflare through a lens that is increasingly inaccurate, which is as a CDN. But you've had Cloudflare Workers for a while, effectively Functions as a Service that run at the edge, which has this magic aura around it, that do various things, which is fascinating to me. You're launching R2; it feels like you are in some ways aiming at becoming a cloud provider, but instead of taking the traditional approach of building it from the region's outward, you're building it from the outward in. Is that a fair characterization?Matthew: I think that's right. I think fundamentally what Cloudflare is, is a network. And I remember early on in the pandemic, we did a series of fireside chats with people we thought we could learn from. And so was everyone from Andre Iguodala, the basketball player, to Mark Cuban, the entrepreneur, to we had a [unintelligible 00:25:56] governor and all kinds of things. And we these were just internal on off the record.And I got to do one with Eric Schmidt, the former CEO of Google. And I said, “You know, Eric, one of the things that we struggle with is describing what is Cloudflare.” And without hesitation, he said, “Oh, that's easy. You're the network I plug into and don't have to worry about anything else.” And I think that's better than I could say it, myself, and I think that's what it is that we fundamentally are: we're the network that fits together.Now, it turns out that in the process of being that network and enabling that network, we are going to build things like R2, which start to be an object store and starts to sort of step into some of the cloud provider space. And Workers is really just a way of programming that network in order to do that, but it turns out that there are a bunch of workloads that if you move them into the network itself, make sense—not going to be every workload, but a lot of workloads that makes sense there. And again, I think that you can actually be very bullish on all of the big public cloud providers and bullish on Cloudflare at the same time because what we want to do is enable the ability for people to mix and match, and change, and be the fabric that connects all of those things together. And so over time, if Amazon says, “We're going to drop egress fees,” it may be that R2 isn't a product that exists—I don't think they're going to do that, so I think it's something that is going to be successful for us and get a lot of new users to us—but fundamentally, I think that where the traditional public clouds think of themselves as the place you put data and you process data, I think we think of ourselves as the place you move data. And that's somewhat different.That then translates into it as we're building out the different pieces, where it does feel like we're building from the outside in. And it may be that over time, that put versus move distinction becomes narrower and narrower as we build more and more services like R2, and durable objects, and KV, and we're working on a database, and all those things. And it could be that we converge in a similar place.Corey: One thing I really appreciate about your vision because it is so atypical these days, is that you aren't trying to build the multifunction printer of companies. You are not trying to be all things to all people in every scenario. Which is impossible to do, but companies are still trying their level best to do it. You are staking out the bounds of where you were willing to start and where you're willing to stop, in a variety of different ways. I would be—how do I put it?—surprised if you at some point in the next five years come out with, “And this is our own database that we have built out that directly competes with the following open-source project that we basically have implemented their API and gone down that particular path.” It does not sound like it is in your core wheelhouse at that point. You don't need—to my understanding—to write your own database engine in order to do what you do.Matthew: Maybe. I mean, we actually are kind of working on a database because—Corey: Oh, no, here we go again.Matthew: [laugh]—and yeah—in a couple of different ways. So, the first way is, we want to make sure that if you're using Workers, you can connect to whatever database you want to use anywhere in the world. And that's something that's coming and we'll be there. At the same time, the challenge of distributed computing turns out not to be the computing, it turns out to be the data and figuring out how to—CAP theorem is real, right? Consistency, Availability, and Partition tolerance; you can pick any two out of the three, but you can't get all three.And so you there's always going to be some trade-off that's there. And so we don't see a lot of good examples. There's some really cool companies that are working on things in the space, but we don't see a lot of really good examples of who has built a database that can be run on a distributed workload system, like Cloudflare to it do well. And so our team internally needs that, and so we're trying to figure out how to build it for ourselves, and I would imagine that after we build it for ourselves—if it works the way we expect it will—that that will then be something that we open up.Our motivation and the way we think about products is we need to build the tools for our own team. Our team itself is customer zero, and then some of those things are very specific to us, but every once in a while, when there are functions that makes sense for others, then we'll build them as well. And that does maybe risk being the multifunction printer, but again, I think that because the customer for that starts with ourselves, that's how we think about it. And if there's someone else's making a great tool, we'll use that. But in this case, we don't see anyone that's built a multi-tenant, globally-distributed, ACID-compliant relational database.Corey: I can't let it pass on challenge. Sure they have, and you're running it yourself. DNS: the finest database in the world. You stuff whatever you want to text records, and now you have taken a finely crafted wrench and turned it into a barely acceptable hammer, which is what I love about doing that terrible approach. Yeah, relational is not going to quite work that way. But—Matthew: Yes. That's a fancy key-value store, right? So—and we've had that for a long time. As we're trying to build those things up, the good news is that, again, we've run data at scale for quite some time and proven that we can do it efficiently and reliably.Corey: There's a lot that can be said about building the things you need to deliver your product to customers. And maybe a database is a poor example here, but I don't see that your motivation in this space is to step into something completely outside your areas of expertise solely because there's money to be made over there. Well, yeah, fortune passes everywhere. The question is, which are you best positioned to wind up delivering an actual transformative solution to that space, and what parts of it are just rent-seeking where it's okay, we're going to go and wherever the money is, we're chasing that down.Matthew: Yeah, we're still a for-profit business, and we've been able to grow revenue well, but I think it is that what motivates us and what drives us comes back to our mission, which is how do you help build a better internet? And you can look at every single thing that we've done, and we try to be very long-term-oriented. So, for instance, when we in 2014 made encryption free, the number one reason at the time, when people upgraded for the free version of our service, the paid version of our service is they got encryption for that. And so it was super scary to say, “Hey, we're going to take the biggest feature and give it away for free,” but it was clearly the direction of history and we wanted to be on the right side of history. And we considered it a bug that the internet wasn't built in an encrypted way from the beginning.So, of course, that was going to head that direction. And so I think that we and then subsequently Let's Encrypt, and a bunch of others have said, it's absurd that you're charging for math. And again, I think that's a good example of how we think about products. And we want to continue to disrupt ourselves and take the things that once upon a time were reserved for our customers that spend $10 million-plus with us, and we want to keep pushing those things down because, over time, the real opportunity is if you do right by customers, there will be plenty of ways that you can earn some of their budget. And again, we think that is the long-term winning strategy.Corey: I would agree with this. You're not out there making sneakers and selling them because you see people spend a lot of money on that; you're delivering value for customers. I say this as one of your paying customers. I have zero problem paying you every month like clockwork, and it is the least cloud-like experience because I know exactly what the bill is going to be in advance, which is apparently not how things should be done in this industry, yadda, yadda, yadda. It is a refreshingly delightful experience every time.The few times I've had challenges with the service, it has almost always been a—I'll call it a documentation gap, where the way it was explained in the formal documentation was not how I conceptualize things, which, again, explaining what these complex things are to folks who are not steeped in certain areas of them is always going to be a challenge. But I cannot think back to a single customer service failure I've had with you folks. I can't look back at any point where you have failed me as a customer, which is a strange thing to say, given how incredibly efficient I am at stumbling over weird bugs.Matthew: Terrific to have you as a customer. We are hardly perfect and we make mistakes, but one of the things I think that we try to do and one of the core values of Cloudflare is transparency. If I think about, like, the original sins of tech, a lot of it is this bizarre secrecy which pervades the entire industry. When we make mistakes, we talk about them, and we explain them. When there's an error, we don't throw up a white page; we put up a page that has our logo on it because we want to own it.And that sometimes gets blowback because you're in front of it, but again, I think it's the right thing to do for customers. And it's and I think it's incredibly important. One of the things that's interesting is you mentioned that you know what your bill is going to be. If you go back and look at the history of hosting on the internet, in the early days of internet hosting, it looks a lot like AWS.Corey: Oh, 95th percentile transit billing; go for one five minutes segment over and boom, your bill explodes. Oh, I remember those days. Unkindly.Matthew: And it was super complicated. And then what happened is the hosting world switched from this incredibly complicated billing to much more simplified, predictable, unlimited bandwidth with maybe some asterisks, but largely that was in place. And then it's strange that Amazon came along and then has brought us back to the more complicated world that's out there. I would have predicted that that's a sine wave—Corey: It has to be. I mean—Matthew: —and it's going to go back and forth over time. But I would have predicted that we would be more in the direction of coming back toward simplify, everything included. And again, I think that's how we've priced our things from the beginning. I'm surprised that it has held on as long as it has, but I do think that there's going to be an opportunity for—and I don't think Amazon will be the leader here, but I think there will be an opportunity for one of the big clouds.And again, I think Oracle is probably doing this the best of any of them right now—to say, “How can we go away from that complexity? How can we make bills predictable? How can we not nickel and dime everything, but allow you to actually forecast and budget?” And it just seems like that's the natural arc of history, and we will head back toward that. And, again, I think we've done our part to push that along. And I'm excited that other cloud providers seem to be thinking about that now as well.Corey: Oh, yeah. What I do with fixing AWS bills is the same thing folks were doing in the 70s and 80s with long-distance bills for companies. We're definitely hitting that sine wave. I know that if I were at AWS in a leadership role, I would be actively embarrassed that the company that is delivering a better customer experience around financial things is Oracle of all companies, given their history of audits and surprising people and the rest. It is ridiculous to me.One last topic that I want to cover with you before we call it an episode is, back in college, you had a thesis that you have done an excellent job of effectively eliminating from the internet. And the theme of this, to my understanding, was that the internet is a fad. And I am so aligned with that because I'm someone who has said for years that emerging technologies are fads. I've said it about cloud, about virtualization, about containers. And I just skipped Kubernetes. And now I'm all-in on serverless, which means, of course it's going to fail because I'm always wrong on these things. But tell me about that.Matthew: When I was seven years old in 1980, my grandmother gave me an Apple ][+ computer for Christmas. And I took to it like a just absolute duck to water and did things that made me very popular in junior high school, like going to computer camp. And my mom used to sign up for continuing education classes at the local university in computer science, and basically sneak me in, and I'd do all the homework and all that. And I remember when I got to college, there was a small group of students that would come around and help other students set their computer up, and I had it all set up and was involved. And so, got pretty deeply involved in the computer science program at college.And then I remember there was a group of three other students—so they were four of us—and they wanted to start an online digital magazine. And at the time, this was pre-web, or right in the early days of the web; it was sort of nineteen… ninety-three. And we built it originally on old Apple technology called HyperCard. And we used to email out the old HyperCard stacks. And the HyperCard stacks kept getting bigger and bigger and bigger, and we'd send them out to the school so [laugh] that we—so we kept crashing the mail servers.But the college loved this, so they kept buying bigger and bigger mail servers. But they were—at some point, they said, “This won't scale. You got to switch technologies.” And they introduced us to two different groups. One was a printer company based out in San Francisco that had this technology called PDF. And I was a really big fan of PDF. I thought PDF was the future, it was definitely going to be how everything got published.And then the other was this group of dorky graduate students at the University of Illinois that had this thing called a browser, which was super flaky, and crashed all the time, and didn't work. And so of the four of us, I was the one who voted for PDF and the other three were like, “Actually, I think this HTML thing is going to be a hit.” And we built this. We won an award from Wired—which was only a print magazine at the time—that called us the first online-only weekly publication. And it was such a struggle to get anyone to write for it because browsers sucked and, you know, trying to get students on campus, but no one on campus cared.We would get these emails from the other side of the world, where I remember really clearly is this—in broken English—email from Japan saying, “I love the magazine. Please keep writing more for the magazine.” And I remember thinking at the time, “Why do I care if someone in Japan is reading this if the girl down the hall who I have a crush on isn't?” Which is obviously what motivates dorky college students like myself. And at that same time, you saw all of this internet explosion.I remember the moment when Netscape went public and just blew through all the expectations. And it was right around the time I was getting ready to graduate for college, and I was kind of just burned out on the entire thing. And I thought, “If I can't even get anyone to write for this dopey magazine and yet we're winning awards, like, this stuff has to all just be complete garbage.” And so wrote a thesis on—ehh, it was not a very good [laugh] thesis. It's—but one of the things I said was that largely the internet was a fad, and that if it wasn't, that it had some real risks because if you enabled everyone to connect with whatever their weird interests and hobbies were, that you would very quickly fall to the lowest common denominator. And predicted some things that haven't come true. I thought for sure that you would have both a liberal and conservative search engine. And it's a miracle to this day, I think that doesn't exist.Corey: Now, that you said it, of course, it's going to.Matthew: Well, I don't know I've… [sigh] we'll see. But it is pretty amazing that Google has been able to, again, thread that line and stay largely apolitical. I'm surprised there aren't more national search engines; the fact that it only Russia and China have national search engines and France and Germany don't is just strange to me. It seems like if you're controlling the source of truth and how people find it, that seems like something that governments would try and take over. There are some things that in retrospect, look pretty wise, but there were a lot more things that looked really, really stupid. And so I think at some level, I had to build Cloudflare to atone for that stupidity all those years ago.Corey: There's something to be said for looking back and saying, “Yeah, I had an opinion, and with the light of new information, I am changing my opinion.” For some reason, in some circles, it feels like that gets interpreted as a sign of weakness, but I couldn't disagree more, it's, “Well, I had an opinion based upon what I saw at the time. Turns out, I was wrong, and here we are.” I really wish more people were capable of doing that.Matthew: It's one of the things we test for in hiring. And I think the characteristic that describes people who can do that well is really empathy. The understanding that the experiences that you have lead you to have a unique set of insights, but they also create a unique set of blind spots. And it's rare that you find people that are able to do that. And whenever you do—whenever we do we hire them.Corey: To that end, as far as hiring and similar topics go, if people want to learn more about how you view things, and how you see the world, and what you're releasing—maybe even potentially work with you—where can they find you?Matthew: [laugh]. So, the joke, sometimes, internal at Cloudflare is that Cloudflare is a blogging company that runs this global network just to have something to write about. So, I think we're unlike most corporate blogs, which are—if our corporate blog were typical, we'd have articles on, like, “Here are the top six reasons you need a fast website,” which would just be, you know, shoot me. But instead, I think we write about the things that are going on online and our unique view into them. And we have a core value of transparency, so we talk about that. So, if you're interested in Cloudflare, I'd encourage you to—especially if you're of the sort of geekier variety—to check out blog.cloudflare.com, and I think that's a good place to learn about us. And I still write for that occasionally.Corey: You're one of the only non-AWS corporate blogs that I pay attention to, for that exact reason. It is not, “Oh, yay. More content marketing by folks who just feel the need to hit a quota as opposed to talking about something valuable and interesting.” So, it's appreciated.Matthew: The secret to it was we realized at some point that the purpose of the blog wasn't to attract customers, it was to attract potential employees. And it turns out, if you sort of change that focus, then you talk to people like their peers, and it turns out then that the content that you create is much more authentic. And that turns out to be a great way to attract customers as well.Corey: I want to thank you for taking so much time out of your day to speak with me. I really appreciate it.Matthew: Thanks for all you're doing. And we're very aligned, and keep fighting the good fight. And someday, again, we'll eliminate cloud egress fees, and we can share a beer when we do.Corey: I will absolutely be there for it. Matthew, Prince, CEO, and co-founder of Cloudflare. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a rambling comment explaining that while data packets into a cloud provider are cheap and crappy, the ones being sent to the internet are beautiful, bespoke, unicorn snowflakes, so of course they cost money.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In episode 46 of o11ycast, Liz and Charity are joined by James Deville of Procore Technologies. James shares the insights he's gained guiding o11y teams and the difficulties tech-forward teams face in non-tech sectors.

In episode 46 of o11ycast, Liz and Charity are joined by James Deville of Procore Technologies. James shares the insights he's gained guiding o11y teams and the difficulties tech-forward teams face in non-tech sectors.

Cory is currently a Solutions Engineer at Jeli.io and very well known in the community for his work on Observability. His career in observability began at Twitter where he managed the observability team and then he joined Stripe, where he created and led the observability team, this time around as a Principal Engineer. We talk to him about how he got his start in customer support and the role it played in the later part of his career. We discuss his time at Twitter where there was a power outage in the data center on the day he joined and how once he had to stay up all night dealing with file handle leaks. We also discuss how he created and led the observability team at Stripe as an individual contributor, how one can succeed in a new org, how to navigate information asymmetry in the workplace, what are some effective ways to advocate for your team and how we all are just humans trying to get stuff done.

In this episode, Gerhard is joined by Cyrille Le Clerc, Product Manager Lead on Observability at Elastic, and Oleg Nenashev, Principal Engineer at CloudBees. It all started with Oleg's tweet back in July, in which he was promoting Akihiro Kiuchi's work on Jenkins monitoring with OpenTelemetry. This was done in the context of Google's Summer of Code - a link to Akihiro's demo is in the show notes. As you may remember from episode 20, instrumenting our changelog.com pipeline is on Gerhard's mind, and this conversation helped him clarify a few things. If you are thinking of instrumenting your CI/CD pipeline with OpenTelemetry, this episode is for you.

It does not matter if it runs on your machine.  Your code must run in the production environment and it must do so performantly.  For that, you need tooling to better understand your application’s behavior under different circumstances.  In the earliest days of software development, all we had were logs, which are still around and The post Observability Using Honeycomb.io with Christine Yen appeared first on Software Engineering Daily.

It does not matter if it runs on your machine.  Your code must run in the production environment and it must do so performantly.  For that, you need tooling to better understand your application's behavior under different circumstances.  In the earliest days of software development, all we had were logs, which are still around and The post Observability Using Honeycomb.io with Christine Yen appeared first on Software Engineering Daily.

Java developers love using Spring. But running high performing and scaling Java apps in production takes a little bit more than just compiling your code.In this episode we have Asir Vedamuthu Selvasingh who has been working with Java for 26 years. In the past 25 years Asir (@asirselvasingh) helped Microsoft provide services to their developer community that make it easier to deploy, run and operate Java based applications at scale – nowadays on the Azure Spring Cloud offering.Listen in and learn more about observability when deploying apps on the Azure Cloud, which performance and scaling aspects you have to consider and get a look behind the scenes on how your packaged java app magically becomes available across the globe.Show Links:Asir on Linkedinhttps://www.linkedin.com/in/asir-architect-javaonazure/Asir on Twitterhttps://twitter.com/asirselvasinghMonitoring Spring Boot Apps with Dynatracehttps://docs.microsoft.com/en-us/azure/spring-cloud/how-to-dynatrace-one-agent-monitorObservability on Azure Spring Cloudhttps://www.dynatrace.com/news/blog/dynatrace-extends-observability-to-azure-spring-cloud/

About EdEd Boyajian, President and CEO of EDB, drives the development and execution of EDB's strategic vision and growth strategy in the database industry, steering the company through 47 consecutive quarters of recurring revenue growth. He also led EDB's acquisition of 2ndQuadrant, a deal that brought together the world's top PostgreSQL experts and positioned EDB as the largest dedicated provider of PostgreSQL products and solutions worldwide. A 15+ year veteran of the open source software movement, Ed is a seasoned enterprise software executive who emphasizes that EDB must be a technology-first business in order to lead the open source data management ecosystem. Ed joined EDB in 2008 after serving at Red Hat, where he rose to Vice President and General Manager of North America. While there, he played a central leadership role in the development of the modern business model for bringing open source to enterprises.Links:EDB: https://enterprisedb.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring. Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today's promoted episode is a treasure and a delight. Longtime listeners of this show know that it's not really a database—unless of course, it's Route 53—and of course, I don't solve pronunciation problems with answers that make absolutely everyone hate me. Longtime listeners of the show know that if there's one thing I adore when it comes to databases—you know, other than Route 53—it is solving pronunciation holy wars in such a way that absolutely everyone is furious with me as a result, and today is no exception. My guest is Ed Boyajian, the CEO of EDB, a company that effectively is the driving force behind the Postgres-squeal database. Ed, thank you for joining me.Ed: Hey, Corey.Corey: So, I know that other people pronounce it ‘post-gree,' ‘Postgresql,' ‘Postgres-Q-L,' all kinds of other things. We know it's decidedly not ‘Postgres-squeal,' which is how I go for it. How do you pronounce it?Ed: We say ‘Postgres,' and this is one of the great branding challenges this fantastic open-source project has endured over many years.Corey: So, I want to start at the very beginning because when I say that you folks are the driving force behind Postgres—or Postgres-squeal—I mean it. I've encountered folks from EDB—formerly EnterpriseDB—in the wild in consulting engagements before, and it's great because whenever we found an intractable database problem, back at my hands-on keyboard engineering implementation days, very quickly after you folks got involved, it stopped being a problem, which is kind of the entire point. A lot of companies will get up there and say, “Oh, it's an open-source project,” with an asterisk next to it and 15 other things that follow from it, or, “Now, we're changing our license so the big companies can't compete with us.” Your company's not named after Postgres-squeal and you're also—when you say you have people working on it, we're not talking just one or two folks; your fingerprints are all over the codebase. How do you engage with an open-source project in that sense?Ed: First and foremost, Postgres itself is, as you know, an independent open-source project, a lot like Linux. And that means it's not controlled by a company. I think that's inherently one of Postgres's greatest strengths and assets. With that in mind, it means that a company like EDB—and this started when I came to the company; I came from Red Hat, so I've been in open-source for 20 years—when I came to the company back in 2008, it starts with a commitment and investment in bringing technology leaders in and around Postgres into a business like EDB, to help enterprises and customers. And that dynamic intersection between building the core database in the community and addressing customer needs in a business, at that intersection is where the magic happens. And we've been doing that since I joined EDB in 2008; it was really an explicit focus for the company.Corey: I'd like to explore a little bit, well first and foremost, this story of is there a future for running databases in cloud environments yourself? And I have my own angry, loud opinion on this that I'm sure we'll get to momentarily, but I want to start with yours. Who is writing their own databases in the Year of our Lord 2021, rather than just using whatever managed thing is their cloud provider of choice today is offering for them?Ed: Well, let me give you context, Corey, because I think it matters. We've been bringing enterprise Postgres solutions to companies now, since the inception of the company, which dates back to 2004, and over that trajectory, we've been helping companies as they've done really two things: migrate away, in particular from Oracle, and land on Postgres, and then write new apps. Probably the first ten of the last 13 years since I've been in the company, the focus was in traditional on-prem database transformations that companies were going through. In the last three years, we've really seen an acceleration of that intersection of their traditional deployments and their cloud deployments. Our customers now, who are represented mostly in the Fortune 500 and Global 2000, 40% of our customers report they're deploying EDB's Postgres in the cloud, not in a managed context, but in a traditional EC2 or GCP self-managed cloud deployment.Corey: And that aligns with what I've seen, a fair bit. Years ago, I wound up getting the AWS Cloud Practitioner Certification—did a whole blog post on it—not because it was opening any doors for me, but because it let me get into the certified lounge at re:Invent, and ideally charge a battery and have some mostly crappy coffee. The one question I got wrong was I was honest when I answered, “How long does it take to restore an RDS database from snapshot backup?” Rather than giving the by-the-book answer, which is way shorter than I found in practice a fair bit of the time. And that's the problem I always ran into is that when you're starting out and building something that needs a database, and it needs a relational database that runs in that model so all the no SQL options are not viable for whatever reason, great, RDS is great for getting you started, but there's only so much that you can tune and tweak before you start to run into issues were, for particular workloads as they scale-out, it's no longer a fit for a variety of reasons.And most of the large companies that I work with that are heavily relational-database-driven have either started off or migrated to the idea of, “Oh, we're going to run our own databases on top of EC2 instances,” for a variety of reasons that, again, the cloud providers will say, “Oh, that's not accurate, and they're doing the wrong thing.” But, you know, it takes a certain courage to tell a large-scale customer, “You're doing it wrong.” “Well, why is that?” “Because I have things to sell you,” is kind of a terrible answer. How do you see it? Let's not pick on RDS, necessarily, because all of the cloud providers offered managed database offerings. Where do those make sense and where do they fall down?Ed: Yeah, I think many of our customers who made their first step into cloud picked a single vendor to do it, and we often hear AWS is been that early, early—Corey: Yeah, a five-year head start makes a pretty compelling story.Ed: That's right. And let's remember what these vendors are mostly. They are mostly infrastructure companies, they build massive data centers and set those up, and they do that beautifully well. And they lean on software, but they're not software companies themselves. And I think the early implementation of many of our customers in cloud relied on what I'll call relatively lightweight software offerings from their cloud vendor, including database.They traded convenience, ease of use, an easy on-ramp, and they traded some capability in some depth for that. And it was a good trade, in fact. And for a large number of workloads it may still be a good trade. But our more sophisticated customers, enterprise customers who are running Postgres or databases at scale in their traditional environments have long depended on a very intimate relationship with their database technology vendor. And that relationship is the intersection of their evolving and emerging needs and the actual development of the database capabilities in support of that.And that's the heart of who we are at EDB and what we do with Postgres and the many people we have committed to doing that. And we don't see our customers changing that appetite. So, I think for those customers, they've emerged more aware of the need to have a primary relationship with a database vendor and still be in cloud. And so I think that's how this evolves to see two different kinds of services side-by-side, what they really want is a Database as a Service from the database vendor, which is what we just announced here at Microsoft Ignite event.Corey: So, talk to me a little bit more about that, where it's interesting in 2021 to see a company launching a managed service offering, especially in the database space, when there's been so much pushback in different ways against the large cloud providers—[cough] Amazon—who tend to effectively lose sleep at night over the haunting fear that someone who isn't them is making money, somehow. And they will take whatever is available to them and turn it into a managed service offering. That's always been the fear, so people play games with licenses and the rest. Well, they've been running Postgres offerings for a long time. It is an independent open-source project.I don't think you can wind up forcing a license change through that says everyone except big companies can run this themselves and don't do a managed service with it because that cat is very much out of the bag. How is it that you're taking something to market now and expecting that to fare competitively?Ed: So, I think there's a few things that our customers are clearly telling us they want, and I think this is the most important thing: they want control of their data. And if you step back, Corey, look at it historically, they made a huge trade to big proprietary database companies, companies like Oracle, and they made that trade actually for convenience. They traded data to that database vendor. And we all know the successes Oracle's had, and the sheer extraordinary expense of those technologies. So, it felt like a walled garden.And that's where EDB and Postgres entered to really change that equation. What's interesting is the re-platforming that happened and the transformation to cloud actually had the same, kind of, binding effect; we now moved all that data over to the public cloud vendors, arguably in an even stickier context, and now I think customers are realizing that's created a dimension of inflexibility. It's also created some—as you rightly pointed out—some deficiencies in technical depth, in database, and in software. So, our customers have sorted that out and are kind of coming back to middle. And what they're saying is, “Well, we want all the advantages of an open-source database like a Postgres, but we want control of the data.”And so what control looks like is more the ability to take one version of that software—in our case, we're worrying about Postgres—and deploy the same thing everywhere they go. And that opens the door up for EDB to be their partner as a traditional on-prem partner, in the cloud where they run our Postgres and they manage it themselves, and as their managed service, Postgres Database as a Service Provider, which is what we're doing.Corey: I've been something of a bear on the idea of, “I'm going to build a workload to run everywhere in every cloud provider,” which I get. I think that's generally foolish, and people chasing that, with remarkably few exceptions, are often going after the wrong thing. That said, I'm also a fan of having a path to strategic Exodus, where Google's Cloud Spanner is fascinating, DynamoDB is revelatory, Cosmos DB is a security nightmare, which is neither here nor there, but the idea that I can take a provider's offering that even if it solves a bunch of problems for me, well, if I ever need to move this somewhere else for any reason, I'm re-architecting, my data model and re-architecting the built-in assumptions around how the database acts and behaves, and that is a very heavy lift. We have proof of that from Amazon, who got up on stage and told a story about how much they hate Oracle, and they're migrating everything off of Oracle to Aurora, which they had to build in order to get off of Oracle, and it took them three years to migrate things. And Oracle loves telling that story, too.And it's, you realize you both sound terrible when you tell that story? It's, “This is a massive undertaking that even we struggle with, so you should probably not attempt it.” Well, what I hear from that is good God, don't wind up getting locked into a particular database that is only available from one source. So, if you're all-in on a cloud provider, which I'm a fan of, personally—I don't care which one but pick a cloud provider—having a database that is not only going to work in that environment is just a reasonable step as far as how I view things. Trading up that optionality has got to pay serious dividends, and in many database use cases, I've just don't see it.Ed: Yeah, I think you're bringing up a really important point. So, let's unpack it for a minute.Corey: Please.Ed: Because I think you brought up some really prominent specialty database technologies, and I'm not sure there's ever a way out of that intersection and commitment to a single vendor if you pick their specialty database. But underneath this is exactly one of the things that we've worried about here at EDB, which is to make Postgres a more capable, robust database in its entirety. A Postgres superpower is its ability to run a vast array of workloads. Guess what, it's not sexy. It's not sexy not to be that specialty database, but it's incredibly powerful in the hands of an enterprise who can do more.And that really creates an opportunity, so we're trying to make Postgres apply to a much broader set of workloads, from traditional systems of record, like your ERP systems; systems of analysis, where people are doing lightweight analytic workloads or reporting, you can think in the world of data warehouse; and then systems of engagement, where customers are interacting with a website and have a database on the backend. All areas Postgres has done incredibly well in and we have customer experience with. So, when you separate out that core capability and then you look at it on a broader scale like Postgres, you realize that customers who want to make Postgres strategic, by definition need to be able to deploy it wherever they want to deploy it, and not be gated or bound by one cloud vendor. And all the cloud vendors picked up Postgres offerings, and that's been great for Postgres and great for enterprises. But that corresponding lock-in is what people want to get away from, at this point.Corey: There's something to be said for acknowledging that there is a form of lock-in as far as technology selection goes. If you have a team of folks who are terrific at one database engine and suddenly you're switching over to an entirely different database, well, folks who spent their entire career working on one particular database that's still in widespread use are probably not super thrilled to stick around for that. Having something that can migrate from environment to environment is valuable and important. When you say you're launching this as a database as a service offering, how does that actually work? Is that going to be running in your own cloud environment somewhere and people just make queries across the wire through standard connections to the database like they would something locally? Are you running inside of their account or environment? Is it something else?Ed: So, this is a fully-managed database as a service, just like you'd get from any cloud vendor or DBAAS vendor that you've worked with in the past, just being managed and run by EDB. And with that, you get lot of the goodies that we bring, including our compatibility, and all our deep Postgres expertise, but I think one of the other important attributes is we're going to run that service in our clients' account, which gives them a level of isolation and a level of independence that we think is really important. And as different as that is, it's not heroic; it's exactly what our customers told us they wanted.Corey: There's something to be said for building the thing that your customers have said that they want and make sense for you to build as opposed to, “We're going to build this ridiculous thing and we're sure folks are going to love it.” It's nice to see that shaping up in the proper order. And I've fallen victim to that myself; I think most technologists have to some extent. How big is EDB these days?Ed: So, we have over 650 employees. Now, around the world, we have 6000 customers. And of the 650 employees, about 300 of those are focused on Postgres. A subset of that are 30-odd core team members in the Postgres community, committers in the Postgres community, major contributors, and contributors in the Postgres community. So, we have a density of technical depth that is really unparalleled in Postgres.Corey: You're not, for lack of a better term, pulling an Amazon, insofar as you're, “Well, we have three people working on open-source projects, so we're going to go ahead and claim we're an open-source company,” in other words. Conversely, you're also not going down the path of this is a project that you folks have launched, and it claims to be open-source because we love it when people volunteer for for-profit entities, but we exercise total control over the project. You have a lot of contributors, but you're also still a minority, I think the largest minority, but still a minority of people contributing to Postgres.Ed: That's right. And, look, we're all-in on Postgres, and it's been that way since I got here. As I mentioned earlier, I came from Red Hat where I was—I was at Red Hat for a little over six years, so I've been an open-source now for 20 years. So, my orientation is towards really powerful, independent open-source projects. And I think we'll see Postgres really be the most transformative open-source technology since Linux.I think we'll see that as we look forward. And you're right, though, I think what's powerful about Postgres is it's an independent project, which means it's supported by thousands of contributors who aren't tied to single companies, around the world. And it just makes the software—we develop innovation faster, and I think it makes the software better. Now, EDB plays a big part in there. Roughly, a little less than a third of the last res—actually, the 13 release—were contributions that came from contributors who came from EDB.So, that's not a majority, and that's healthy. But it's a big part of what helps move Postgres along and there aren't—you know, the next set of companies are much, much—next set of combined contributors add up to quite small numbers. But the cloud vendors are virtually non-existent in that contribution.Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: Something else that does strike me as, I guess, strange, just because I've seen so many companies try to navigate this in different ways with varying levels of success. I always encountered EDB—even back when it was EnterpriseDB, which was, given their love of acronyms, I'm still somewhat partial to. I get it; branding, it's a thing—but the folks that I engaged with were always there in a consulting service's capacity, and they were great at this. Is EDB a services company or a product company?Ed: Yeah, we are unashamedly a product technology company. Our business is over 90% of our revenue is annually recurring subscription revenue that comes from technical products, database server, mostly, but then various adjacent capabilities in replication and other areas that we add around the database server itself. So no, we're a database technology company selling a subscription. Now, we help our customers, so we do have a really talented team of consultants who help our customers with their business strategy for Postgres, but also with migrations and all the things they need to do to get Postgres up and running.Corey: And the screaming, “Help, help, help, fix it, fix it, fix it now,” emergencies as well.Ed: I think we have the best Postgres support operation in the world. It is a global 24/7 organization, and I think a lot of what you likely experienced, Corey, came out of our support organization. So, our support guys, these guys aren't just handling lightweight issues. I mean, they wade into the gnarly questions and challenges that customers face. But that's a support business for us. So, that's part and parcel. You get that, it's included with the subscription.Corey: I would not be remembering this for 11 years later, if it hadn't been an absolutely stellar experience—or a horrible experience, for that matter; one or the other. You remember the superlatives, not the middle of the road ones—and if it hadn't been important. And it was. It also noteworthy; with many vendors that are product-focused, their services may have an asterisk next to it because it's either a, “Buy our product and then we'll support it,” or it's, “Ohh, we're going to sell you a whole thing just to get us on the phone.” And as I recall, there wasn't a single aspect of upsell involved in this.It was, “Let's get you back up and running and solve the problem.” Sure, later in time, there were other conversations, as all good businesses will have, but there was no point during those crisis moments where it felt like, “Oh, if you had gone ahead and bought this thing that we sell, this wouldn't happen,” or, “You need to buy this or we won't help you.” I guess that's why I've contextualized you folks as a services company, first and foremost.Ed: Well, I'm glad you have that [laugh] experience because that's our goal. And I think—look, this is an interesting point where customers want us to bring that capability to their managed DBAAS world. Step back again, go back to what I said about the big cloud vendors; they are, at their core, infrastructure companies. I mean, they're really good at that. They're not particularly well-positioned to take your Postgres call, and I don't think they want that call.We're the other guys; we want to help you run your Postgres, at scale, on-prem, in the cloud, fully managed in the cloud, by EDB, and solve those problems at the same time. And I think that's missing in the market today. And we can step back and look at this overall cloud evolution, and I think some might think, “Gee, we're into the mature phase of cloud adoption.” I would tell you, since the Red Sox have done well this year, I think in a nine-inning baseball game—for those of your listeners who follow American baseball—we're in, like, the top of the second inning, maybe. Maybe the bottom of the second inning. So, we've been able to listen and learn from the experiences our customers have had. I think that's an incredible advantage as we now firmly plant ourselves in the cloud DBAAS market alongside our robust Postgres capabilities that you experienced.Corey: The world isn't generating less data, and it's important that we're able to access that in a bunch of different ways. And the last time I really was playing with relational databases, you can view my understanding of it as Excel with a weirder interface, and you're mostly there. One thing that really struck me since the last time I went deep into database-land over in the Postgres-squeal world has been just the sheer variety of native data types that it winds up supporting. The idea of, “Here's some JSON. Take this and store it that way,” or it's GIS data that it can represent, or the idea of having data types that are beyond just string or var or whatever other somewhat limited boolean values or whatnot. Without having just that traditional list, which is of course all there as well. It also seems to have extensively improved its coverage that just can only hint to my small mind about these things and what sort of use cases people are really putting these things into.Ed: Yeah, I think this is one of Postgres' superpowers. And it started with Mike Stonebraker's original development of Postgres as an object-relational database. Mike is an adviser to EDB, which has been incredibly helpful as we've continued to evolve our thinking about what's possible in Postgres. But I think because of that core technology, or that core—because of that core technical capability within Postgres, we have been able to build a whole host of data types. And so now you see Postgres being used not just as the context of a traditional relational database, but we see it used as a time-series database. You pointed out a geospatial database, more and more is a document-oriented database with JSON and JSONB.These are all the things that make Postgres have much more universal appeal, universal appeal to developers—which is worth talking about in the recent StackOverflow developer survey, but we can come back to that—and I think universal applicability for new applications. This is what's bringing Postgres forward faster, unlike many of the specialty database companies that you mentioned earlier.Corey: Now, this is something that you can use for your traditional CRUD app, the my first hello world app that returns something from a database, yeah, that stuff works. But it also, for example, has [cyter 00:25:09] data types, where you can say, give me the results where the IP range contains this address, and it'll do that. Before that, you're trying to solve a whole bunch of very messy things in application logic that's generally awful. The database now does that for you automatically, and there's something—well, it would if I were smart and used it instead of storing it as strings because I make terrible life choices, but for sensible people, it solves a lot of those problems super well. And it's taken the idea of where logic should live in application versus database, and sort of turn a lot of those assumptions I was starting my career with on their head.Ed: Yeah, I think if you look now at the appeal of Postgres to developers, which we've paid a lot of attention to—one of our stated strategies at EDB is to make Postgres easier. That's been true for many years, so a drive for engineering and development here has been that call to action. And if you measure that, over time, we've been contributing—not alone, but contributing to making Postgres more approachable, easier to use, easier to engage with. Some of those things we do just through edb.com, and the way we handle EDB docs is a great example of that, and our developer advocacy and outreach into adjacent communities that care about Postgres. But here's where that's landed us. If you looked at the last Stack Overflow developer survey—the 2021 Stack Overflow developer survey, which I love because I think it's very independent-oriented—and they surveyed, I think this past year was 80,000 developers.Corey: Oh yeah, if Stack Overflow is captured by any particular constituency, it's got to be ‘Big Copy and Paste' that is really behind them. But yeah, other than the cabal of keyboard manufacturers for those copy-and-paste stories, yeah, they're fairly objective when it comes to stuff like this.Ed: And if you look at that survey, Corey, if you just took and summed it because it's helpful to sum it, most used, most loved, and most wanted database: Postgres wins. And I find it fascinating that if you—having been here, in this company for 13 years and watch the evolution from—you know, 13 years ago, Postgres needed help, both in terms of its awareness in the market and some technical capabilities it just lacked, we've come so far. For that to be the new standard for developers, I think, is a remarkable achievement. And I think it's a representation of why Postgres is doing so well in the market that we've long served, in the cloud market that we are now serving, and I think it speaks to what's ahead as a transformational database for the future.Corey: There really is something to be said for a technology as—please don't take this term the wrong way—old. As a relational database, Postgres has been around for a very long time, but it's also not your grandparents' Postgres. It is continuing to evolve. It continues to be there in a bunch of really interesting ways for developers in a variety of different capacities, and it's not the sort of thing that you're only using in, “Legacy environments,” quote-unquote. Instead, it's something that you'll see all over the place. It is rare that I see an environment that doesn't have Postgres in it somewhere these days.Ed: Yeah, I think quite the contrary to the old-school database, which I love that; I love that shade because when you step away from it, you realize, the Postgres community represents the very best of what's possible with open-source. And that's why Postgres continues to accelerate and move forward at the rate that it does. And obviously, we're proud to be a contributor to that, so we don't just watch that outcome happen; we're actually part of creating it. But I also think that when you see all that Postgres has become and where it's going, you really start to understand why the market is adopting open-source.Corey: It's one of those areas where even if some company comes out with something that is amazing and transformatively better, and you should jump into it with both feet and never look back, yeah, it turns out that it takes a long time to move databases, even when they're terrible. And you can lobby an awful lot of accusations at Postgres—or Postgres-squeal—but you can't call it terrible. It's used in enough interesting applications by enough large-scale companies out there—and small as well—that it's very hard to find a reason not to explore it. It's my default relational database when Route 53 loses steam. It just makes sense in a bunch of ways that other things really didn't for me before.Ed: Yeah, and I think we'll continue to see that. And we're just going to keep making Postgres better. And it gets better because of that intersection, as I mentioned, that intimate intersection between enterprise users, and the project, and the community, and the bridge that a company like EDB provides for that. That's why it'll get better faster; the breadth of use of Postgres will keep it accelerating. And I think it's different than many of the specialty databases.Look, I've been in open-source now for 20 years and it's intriguing to me how many new specialty open-source databases have come to market. We tend to forget the amount of roadkill we've had over the course of the past ten years of some of those open-source projects and companies. We certainly are tuned into some of the more prolific ones, even today. And I think again, here again, this is where Postgres shines, and where I think Postgres is a better call for a long-term. Just like Linux was.Corey: I want to thank you for taking so much time out of your day to talk to me about databases, which given my proclivities, is probably like pulling teeth for you. If people want to learn more, where can they find you?Ed: So, come to enterprisedb.com. You still get EnterpriseDB, Corey. Just come to enterprise—Corey: There we go. It's hidden in the URL, right in plain sight.Ed: Come to enterprisedb.com. You can learn all the things you need about the technology, and certainly more that we can do to help you.Corey: And we will, of course, put links to that in the [show notes 00:31:10]. Thank you once again for your time. I really do appreciate it.Ed: Thanks, Corey. My pleasure.Corey: Ed Boyajian, CEO of EDB. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a long angry comment because you are one of the two Amazonian developers working on open-source databases.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In this episode of the Armchair Architect series of the Azure Enablement Show, David Blank-Edelman is joined by Uli Homann, Microsoft Corporative Vice President and Distinguished Architect, and Eric Charran, Chief Architect at Microsoft in a deep-dive conversation into observability of hybrid and multi-cloud applications.✅Resources:Management and Monitoring patterns - Cloud Design Patterns | Microsoft DocsSidecar pattern - Cloud Design Patterns | Microsoft DocsObservability patterns | Microsoft Docs[00:00] Introduction[01:22] From an architectural perspective, what do we need to do to build and run resilient, self-healing applications?[04:22] Is monitoring from within the app itself an anti-pattern? Should sidecar patterns be used instead?[06:21] What's the best method for Observability? Push or Pull?[09:16] Can you use AI to monitor logs and report anomalies?[10:19] Azure uses ML to predict host failure, then proactively removes VMs and takes the host out of the cluster[12:15] Apps and microservices create so many logs; how can you tie them all together to track and understand an event?

Seguindo a série que conversa sobre observabilidade com diferentes players do mercado, o Kubicast traz nesse programa o  Adilson Somensari F., Senior Solutions Engineer da Splunk. Apesar de ser sinônimo de log, a Splunk vem, de dois anos para cá, expandindo seu business, e hoje tem como solução sua plataforma de observabilidade. Para o Adilson, a melhor maneira de convencer Devs e gestores(as) a adotar observabilidade e OpenTelemetry é mostrar que com eles é possível ter controle, acesso a dados, inclusive customizados, e relatórios. É trabalhar às claras! Quanto à feature mais wow da plataforma, ele considera que é o end to end para 100% dos usuários; isso é tem para todo mundo! Como dica matadora, para realmente aproveitar o valor da solução de observabilidade, ele diz: “planeje a gestão de incidentes para receber alertas antes do problema acontecer”! CONTATO - Para falar com o Adilson, escreva para: asomensari@splunk.com.O LINK para acessar a talk do Adilson no evento da Splunk, segue aqui. As RECOMENDAÇÕES do programa são:Adilson:Desafiando o Talento do Geoff Colvin  (livro) Ted Lasso (Apple Tv)Mythic Quest (Apple Tv)João: Cry Macho: O Caminho para Redenção (Estava nos cinemas e agora está na HBO Max) CFP do #KCDBrasil - O prazo para inscrever sua palestra no #KCDBrasil  termina na sexta (29). Não se intimide, sugira a sua ideia e vamos espalhar a palavra no maior e melhor evento de Kubernetes do Brasil. Acesse aqui! O Kubicast é uma produção da Getup, especialista em Kubernetes e apoiadora do projeto UnDistro, uma distribuição para gerenciar múltiplos clusters Kubernetes. #o11y #Observability #OpenTelemetry #DevOps #Kubernetes #Containers #docker 

About ChrisChris Williams is a Enterprise Architect for World Wide Technology — a technology solution and service provider. There he helps customers design the next generation of public, private, and hybrid cloud solutions, specializing in AWS and VMware. His first computer was a Commodore 64, and he's been playing video games ever since.Chris blogs about virtualization, technology, and design at Mistwire. He is an active community leader, co-organizing the AWS Portsmouth User Group, and both hosts and presents on vBrownBag. He is also an active mentor, helping students at the University of New Hampshire through Diversify Thinking—an initiative focused on empowering girls and women to pursue education and careers in STEM.Chris is a certified AWS Hero as well as a VMware vExpert. Fun fact that Chris doesn't want you to know: he has a degree in psychology so you can totally talk to him about your feelings.Links: WWT: https://www.wwt.com/ Twitter: https://twitter.com/mistwire Personal site: https://mistwire.com vBrownBag: https://vbrownbag.com/team/chris-williams/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the things I miss the most from the pre-pandemic times is meeting people at conferences or at various business meetings, not because I like people—far from it—but because we go through a ritual that I am a huge fan of, which is the exchange of business cards. Now, it's not because I'm a collector or anything here, but because I like seeing what people's actual titles are instead of diving into the morass of what we call ourselves on Twitter and whatnot. Today, I have just one of those folks with me. My guest is Chris Williams, who works at WWT, and his business card title is Enterprise Architect, comma AWS Cloud. Chris, welcome.Chris: Hi. Thanks for having me on the show, Corey.Corey: No, thank you for taking the time to speak with me. I have to imagine that the next line in your business card is, “No, I don't work for AWS,” because you know a company has succeeded when they get their name into people's job titles who don't work there.Chris: So, I have a running joke where the next line should actually be cloud therapist. And my degree is actually in psychology, so I was striving to get cloud therapist in there, but they still don't want to let me have it.Corey: Former guest Bobby Allen is now a cloud therapist over at Google Cloud, which is just phenomenal. I don't know what they're doing in a marketing context over there; I just know that they're just blasting them out of the park on a consistent, ongoing basis. It's really nice to see. It's forcing me to up my game a little bit. So, one of the challenges I've always had is, I don't like putting other companies' names into the title.Now, I run the Last Week in AWS newsletter, so yeah, okay, great, there's a little bit of ‘do as I say, not as I do' going on here. Because it feels, on some level, like doing unpaid volunteer work for a $2 trillion company. Speaking of, you are an AWS Community Hero, where you do volunteer work for a $2 trillion company. How'd that come about? What did you do that made you rise to their notice?Chris: That was a brilliant segue. Um—[laugh]—Corey: I do my best.Chris: So I, actually prior to becoming an AWS Community Hero, I do a lot of community work. So, I have run and helped to run four different community-led organizations: the Virtualization Technology User Group of New England; the AWS Portsmouth User Group, now the AWS Boston User Group; I'm a co-host and presenter for vBrownBag; I also do the New England AWS Community Day, which is a conglomeration of all the different user groups in one setting; and various and sundry other things, as well, along the way. Having done all of that, and having had a lot of the SAs and team members come and do speaking presentations for these various and sundry things, I was nominated internally by AWS to become one of their Community Heroes. Like you said, it's basically unpaid volunteer work where I go out and tout the services. I love talking about nerd stuff, so when I started working on AWS technologies, I really enjoyed it, and I just, kind of like, glommed on with other people that did it as well. I'm also a VMware vExpert, which basically use the exact same accolade for VMware. I have not been doing as much VMware stuff in the recent past, but that's kind of how I got into this gig.Corey: One of the things that strikes me as being the right move with respect to these, effectively, community voice accolades is Microsoft got something very right—they've been doing this a long time—they have their MVP program, but they have to re-invite people who have to requalify for it by whatever criteria they are, every year. AWS does not do this with their Heroes program. If you look at their Heroes page, there's a number of folks up there who have been doing interesting things in the cloud years ago, but then fell off the radar for a variety of reasons. In fact, the only way that I'm aware that you can lose Hero status is via getting a job at AWS or one of AWS competitors.Now, the hard part, of course, is well, who is Amazon's competitors? Basically everyone, but it mostly distills down to Microsoft, Google, and Oracle, as best I can tell, for Hero status. How does VMware fall on that spectrum? To be more specific, how does VMware fall on the spectrum of their community engagement program and having to renew, not, “Are they AWS's competitor?” To which the answer is, “Of course.”Chris: So, the renewal process for the VMware vExpert program is an annual re-up process where you fill out the form, list your contribution of the year, what you've done over the previous year, and then put it in for submission to the board of VMware vExperts who then give you the thumbs up or thumbs down. Much like Nero, you know, pass or fail, live or die. And I've been fortunate enough, so my vBrownBag contributions are every week; we have a show that happens every week. It can be either VMware stuff, or cloud in general stuff, or developer-related stuff. We cover the gamut; you know, people that want to come on and talk about whatever they want to talk about, they come on. And by virtue of that, we've had a lot of VMware speakers, we've had a lot of AWS speakers, we've had a lot of Azure speakers. So, I've been fortunate enough to be able to qualify each year with those contributions.Corey: I think that's the right way to go, from my perspective at least. But I want to get into this a little bit because you are an enterprise architect, which is always one of those terms that is super easy to make fun of in a variety of different ways. Your IDE is probably a whiteboard, and at some point when you have to write code, I thought you had a team of people who would be able to do that all for you because your job is to cogitate, and your artifacts are documentation, and the entire value of what you do can only be measured in the grand sweep of time, et cetera, et cetera, et cetera.Chris: [laugh].Corey: But you don't generally get to be a Community Hero for stuff like that, and you don't usually get to be a vExpert on the VMware side, by not having at least technical chops that make people take a second look. What is it you'd say it is you do hear for, lack of a better term?Chris: “What would you say ya, do you here, Bob?” So, I'm not being facetious when I say cloud therapist. There is a lot of working at the eighth layer of the OSI model, the political layer. There's a lot of taking the requirements from the customer and sending them to the engineer. I'm a people person.The easy answer is to say, I do all the things from the TOGAF certification manual: the requirements, risks, assumptions, and constraints; the logical, conceptual, and physical diagrams; the harder answer is the soft skill side of that, is actually being able to communicate with the various levels of the industry, figuring out what the business really wants to do and how to technically solution that and figure out how to talk to the engineers to make that happen. You're right EAs get made fun of all the time, almost as much as consultants get made fun of. And it's a very squishy layer that, you know, depending upon your personality and the personality of the customer that you're dealing with, it can work wonderfully well or it can crash and burn immediately. I know from personal experience that I don't mesh well with financials, but I'm really, really good with, like, medical industry stuff, just the way that the brain works. But ironically, right now I'm working with a financial and we're getting along like a house on fire.Corey: Oh, yeah. I've been saying for a while now that when it comes to cloud, cost and architecture are the same things, and I think that ties back to a lot of different areas. But I want to be very clear here that we talk about, I'm not super deep into the financials, that does not mean you're bad at architecture because working on finance means different things to different folks. I don't think that it is possibly a good architect in the cloud environment and not have a conception of, “Huh, that thing seems really expensive if I do it that way.” That is very different than having the skill of reading a profit and loss statement or understanding various implications of the time value of money calculation that a company uses, or how things get amortized.There are nuances piled on top of nuances in finance, and it's easy to sit here and think that oh, I'm not great at finance means I don't know how money works. That is very rarely true. If you really don't know how money works, you'll go start a cryptocurrency startup.Chris: [laugh]. So, I plugged back to you; I was listening to one of your old shows and I cribbed one of your ideas and totally went with it. So, I just said that there's the logical, conceptual, and physical diagrams of an environment; on one of your shows, you had mentioned a financial diagram for an environment, and I was like, “That's brilliant.” So, now when I go into a customer, I actually do that, too. I take my physical diagram, I strip out all of the IP addresses, and our names, and everything like that, and I plot down how much it's going to cost, like, “This is the value of the EC2 instance,” or, “This is how much this pipe is going to cost if you run this over it.” And they go bananas over it. So, thanks for providing that idea that I mercilessly stole.Corey: Kind of fun on a lot of levels. Part of the challenge is as things get cloudier and it moves away from EC2 instances, ideally the lie we would like to tell ourselves that everything's in an auto-scaling group. Great—Chris: Right.Corey: —stepping beyond that when you start getting into something that's even more intricately tied to a specific user, we're talking about effectively trying to get unit economic measures of every user, every thousand users is going to cost me X dollars to service them on average, on top of a baseline of steady-state spend that is going to increase differently. At that point, talking to finance about predictive models turn into, “Well, this comes down to a question of business modeling.” But conversely, for engineering minds that is exactly what finance is used to figuring out. The problem they have is, “Well, every time we hire a new engineer, we wind up seeing our AWS bill increase.” Funny how that works. Yeah, how do you map that to something that the business understands? That is part of what they do. But it does, I admit, make it much more challenging from a financial map of an environment.Chris: Yeah, especially when the customer or the company is—you know, they've been around for a while, and they're used to just like that large bolus of money at the very beginning of a data center, and they buy the switches, and they buy the servers, and they virtualize them, and they have that set cost that they knew that they had to plunk down at the beginning. And it's a mindset shift. And they're coming around to it, some faster than others. Oddly enough, the startups nowadays are catching on very quickly. I don't deal with a lot of startups, so it takes some finesse.Corey: An interesting inflection that I've seen is that there's an awful lot of enterprises out there that say, “Oh, we're like a startup.” Great. You mean with weird cultural inflections that often distill down to cult of personality, the constant worry about whether you're going to wind up running out of runway before finding product-market fit? And the rooms filled with—Chris: The eighty-hour work weeks? The—[laugh]—Corey: And they're like, “No, no, no, it's like the good parts.” “Oh, so you mean out the upside.” But you don't hear it the other way around where you have a startup that you're interviewing with, “Ha-ha, we're like an enterprise. We have a six-month interview process that takes 18 different stages,” and so on and so forth. However, we do see startups having to mature rapidly, and move up the compliance path as they're dealing with regulated entities and the rest, and wanting to deal with serious customers who have no sense of humor about, “Yeah, we'll figure that part out later as part of an audit document.”So, what we also see, though, is that enterprises are doing things that look a lot more startup-y. If I take a look at the common development environments and tools and techniques that big enterprises use, it looks an awful lot like how startups were doing it five or ten years ago. That is the slow and steady evolution of time. And what startups are doing today becomes enterprise tomorrow, and I can't shake the feeling that there's a sea of vendors out there who, in the event that winds up happening are eventually going to find themselves without a market at all. My model has been that if I go and found a Twitter for Pets style startup tomorrow and in ten years, it has grown to become an S&P 500 component—which is still easier to take seriously than most of what Tesla says—great.During that journey, at what point do I become a given company's customer because if there is no onboarding story for me to become your customer, you're in a long-tail decline phase. That's been my philosophy, but you are a—trademarked term—Enterprise Architect, so please feel free to tell me if I'm missing any of the nuances there, which I'm sure I am because let's face it, nuance is hard; sweeping statements are easy.Chris: As an architect, [laugh] it would be a disservice to not say my favorite catchphrase, it depends. There are so many dependencies to those kinds of sweeping statements. I mean, there's a lot of enterprises that have good process; there are a lot of enterprises that have bad process. And going back to your previous statement of the startup inside the enterprise, I'm hearing a lot of companies nowadays saying, “Oh, well, we've now got this brand new incubator system that we're currently running our little startup inside of. It's got the best of both worlds.”And I'm not going to go through the litany of bad things that you just said about startups, but they'll try to encapsulate that shift that you're talking about where the cheese is moving so quickly now that it's very hard for these companies to know the customer well enough to continue to stay salient and continue to be able to look into that crystal ball to stay relevant in the future. My job as an EA is to try to capture that point in time where what are the requirements today and what are the known detriments that you're going to see in your future that you need to protect against? So, that's kind of my job—other than being a cloud therapist—in a nutshell.Corey: I love the approach. My line has been that I do a lot of marriage counseling between engineering and finance, which is a fun term that also just so happens to be completely accurate.Chris: Absolutely. [laugh]. I'm currently being a marriage counselor right now.Corey: It's an interesting time. So, you had a viral tweet recently that honestly, I'm a bit jealous about. I have had a lot of tweets that have done reasonably well, but I haven't ever had anything go super-viral, where it was just a screenshot of a conversation you had with an AWS recruiter. Now, before we go into this, I want to make a couple of disclaimers here. Before I entered tech myself, I was a technical recruiter, and I can say that these people have hard jobs.There is a constant pressure to perform, it is a sales job that is unlike most others. If you sell someone a pen, great, you can wrap your head around what that's like. But you don't have to worry about the pen deciding it doesn't want to go home with the buyer. So, it becomes a double sale in a lot of weird ways, and there's a constant race to the bottom and there's a lot of competition in the space. It's a numbers game and a lot of folks get in and wash out who have terrible behaviors and terrible patterns, so the whole industry gets tainted—in some respects—like that. A great example of someone who historically has been a terrific example of recruiting done right has been Jill Wohlner. And she's one of the shining beacons of the industry as far as how to do these things in the right way—Chris: Yes.Corey: —but the fact that she is as exceptional as she is is in no small part because there's a lot of random folks coming by. All which is to say that our conversation going forward is not and should not be aimed at smacking around individual recruiters or recruiting as a whole because that is unfair. Now, that disclaimer has been given. Great, what happened?Chris: So, first off, shout out to Jill; she actually used to be a host on vBrownBag. So, hey girl. [laugh]. What happened was—and I have the utmost empathy and sympathy for recruiting; I actually used to have a side gig where I would go around to the local recruiting places around my area here and teach them how to read a cloud resume and how to read a req and try to separate the wheat from the chaff, and to actually have good conversations. This was back when cloud wasn't—this was, like, three or four years ago.And I would go in there and say, “This is how you recruit a cloud person nowadays.” So, I love good recruiters. This one was a weird experience in that—so when a recruiter reaches out to me, what I do is I take an assessment of my current situation: “Am I happy where I'm at right now?” The answer is, “Yes.” And if they ping me, I'll say, “Hey, I'm happy right now, but if you have something that is, you know, a million dollars an hour, taste-testing margaritas on St. John island in the sand, I'm all ears. I'm listening. Conversely, I also am a Community Hero, so I know a ton of people out in the industry. Maybe I can help you out with landing that next person.”Corey: I just want to say for the record, that is absolutely the right answer. And something like that is exactly what I would give, historically. I can't do it now because let's be clear here. I have a number of employees and, “Hey, Corey's out there doing job interviews,” sends a message that isn't good when it comes to how is that company doing anyway. I miss it because I enjoyed the process and I enjoyed the fun, but even when I was perfectly happy, it's, “Well, I'm not actively on the market, but I am interested to have a conversation if you've got something interesting.”Because let's face it, I want to hear what's going on in the market, and if I'm starting to hear a lot of questions about a technology I have been dismissive of, okay, maybe it's time to pay more attention. I have repeatedly been able to hire the people interviewing me in some cases, and sometimes I've gone on interviews just to keep my interview skills sharp and then wound up accepting the job because it turned out they did have something interesting that was compelling to me even though I was reasonably happy at the time. I will always take the meeting; I will always at least have a chat about what they're doing, and I think that doing otherwise is doing yourself a disservice in the long arc of your career.Chris: Right. And that's basically the approach that I take, too. I want to hear what's out there. I am very happy at World Wide right now, so I'm not interested, interested. But again, if they come up with an amazing opportunity, things could happen. So, I implied that in my response to him.I said, “I'm happy right now, thanks for asking, but let's set up the meeting and we can have a chat.” The response was unexpected. [laugh]. The response was basically, “If you're not ready to leave right now, it makes no sense for me to talk to you.” And it was a funny… interaction.I was like, “Huh. That's funny.” I'm going to tweet about that because I thought it was funny—I'm not a jerk, so I'm going to block out all of the names and all of the identifying information and everything—and I threw it up. And the commiseration was so impressive. Not impressive in a good way; impressive in a bad way.Every person that responded was like, “Yes. This has happened to me. Yes, this is”—and honestly, I got a lot of directors from AWS reaching out to me trying to figure out who that person was, apologizing saying that's not our way. And I responded to each and every single one of them. And I was like, “Somebody has already found that person; somebody has already spoken to that person. That being said, look at all of the responses in the timeline. When you tell me personally, that's not the way you do things, I believe that you believe that.”Corey: Yeah, I believe you're being sincere when you say this, however the reality of what the data shows and people's lived experience in the form of anecdotes are worlds apart.Chris: Yeah. And I'm an AWS Hero. [laugh]. That's how I got treated. Not to blow my own horn or anything like that, but if that's happening to me, either A, he didn't look me up and just cold-called me—which is probably the case—and b, if he treats me like that, imagine how he's treating everybody else?Corey: This episode is sponsored in part by something new. Cloud Academy is a training platform built on two primary goals. Having the highest quality content in tech and cloud skills, and building a good community the is rich and full of IT and engineering professionals. You wouldn't think those things go together, but sometimes they do. Its both useful for individuals and large enterprises, but here's what makes it new. I don't use that term lightly. Cloud Academy invites you to showcase just how good your AWS skills are. For the next four weeks you'll have a chance to prove yourself. Compete in four unique lab challenges, where they'll be awarding more than $2000 in cash and prizes. I'm not kidding, first place is a thousand bucks. Pre-register for the first challenge now, one that I picked out myself on Amazon SNS image resizing, by visiting cloudacademy.com/corey. C-O-R-E-Y. That's cloudacademy.com/corey. We're gonna have some fun with this one!Corey: Every once in a while I get some of their sourcers doing outreach to see folks who are somewhat aligned on them via LinkedIn or other things, and, “Oh, okay, yeah; if you look at the things I talked about in various places, I can understand how I might look like a potentially interesting hire.” And they send outreach emails to me, they're always formulaic, and once in a while, I'll tweet a screenshot of them where I redact the person's name, and it was—and there's a comment, like, “Should I tell them?” Because it's fun; it's hilarious. But I want to be clear because that often gets misconstrued; they have done absolutely nothing wrong. You've got to cast a wide net to find talent.I'm surprised I get as few incidents of recruiter outreach as I do. I am not hireable and that's okay, but I don't begrudge people reaching out. I either respond with a, “No thanks,” if it's a particularly good email, or I just hit the archive button and never think about it again. And that's fine, too. But I don't make people feel like a jerk for asking, and that is an engineering behavioral pattern that drives me up a wall.It's, “So, I'm thinking about a job here and I'm wondering if you might be a fit,” and your response is just to set them on fire? Well, guess what an awful lot of those people sending out those emails in the sourcing phase of recruiting are early career, and guess what, they tend to get promoted in the fullness of time. Sometimes they're no longer recruiting at all; sometimes they wind up being hiring managers in different ways or trying to figure out what offer they're going to extend to someone. And if you don't think that people in those roles remember when they're treated poorly as a response to their outreach, I have news for you. Don't do it. Your reputation lingers long after you no longer work there.Chris: Just exactly so. And I feel really bad for that guy.Corey: I do hope that he was not reprimanded because he should not be. It is clearly a systemic problem, and the fact that one person happened to do this in a situation where it went viral does not mean that they are any worse than other folks doing it. It is a teachable opportunity. It is, “I know that you have incredible numbers of roles to hire for, all made all the more urgent by the fact that you're having some significant numbers of departures—clearly—in the industry right now.” So, I get it; you have a hard job. I'm not going to waste your time because I don't even respond to them just because, at AWS particularly, they have hard work to do, and just jawboning with me is not going to be useful for them.Chris: [laugh].Corey: I get it.Chris: And you're trying to hire the same talent too. So.Corey: Exactly. One of the most egregious things I've seen in the course of my career was when that whole multiple accounts opened for Wells Fargo's customers and they wound up firing 3500 people. Yeah, that's not individual tellers doing something unethical. That is a systemic problem, and you clean house at the top because you're not going to convince me that you're hiring that many people who are unethical and setting out to do these things as a matter of course. It means that the incentives are wrong, it means that the way you're measuring things are wrong, and people tend to do things out of fear or because there's now a culture of it. And if you fire individuals for that, you're wrong.Chris: And that was the message that I conveyed to the people that reached out to me and spoke to me. I was like, there is a misaligned KPI, or OKR, or whatever acronym you want to use, that is forcing them to do this churn-and-burn mentality instead of active, compassionate recruiting. I don't know what that term is; I'm very far removed from the recruiting world. But that person isn't doing that because they're a jerk. They're doing that because they have numbers to hit and they've got to grind out as many as humanly possible. And you're going to get bad employees when you do that. That's not a long-term sustainable path. So, that was the conversation that I had with them. Hopefully, it resonated and hits home.Corey: I still remember from ten years ago—and I don't always tell the story, but I absolutely will now—I went up to San Francisco when I lived in Los Angeles; I interviewed with Yammer. I went through the entire process—this was not too long before they got acquired by Microsoft so that gives you some time basis—and I got a job offer. And it was a not ridiculous offer. I was going to think about it, and I [unintelligible 00:24:19], “Great. Thank you. Let me sleep on this for a day or two and I'll get back to you definitely before the end of the week.”Within an hour, I got a response rescinding the offer claiming it had been sent by mistake. Now, I believe that that is true and that they are being sincere with this. I don't know that if it was the wrong person; I don't know if that suddenly they didn't have the req or they had another candidate that suddenly liked better that said no and then came back and said yes, but it's been over a decade now and every time I talk to someone who's considering something in that group, I tell this story. That's the sort of thing that leaves a mark because I have a certain philosophy of I don't ever resign from a job before I wind up making sure everything is solid—things are signed, good to go, the background check clears, et cetera—because I don't want to find myself suddenly without income or employment, especially in that era. And that was fine, but a lot of people don't do that.As soon as the offer comes in, they're like, “I'm going to go take a crap on my boss's desk,” which, let's be clear, I don't recommend. You should write a polite and formulaic resignation letter and then you should email it to your boss, you should not carve it into their door. Do this in a responsible way, and remember that you're going to encounter these people again throughout your career. But if I had done that, I would have had serious problems. And so that points to something systemically awful at a company.I have never in my career as a hiring manager extended an offer and then rescinded it for anything other than we can't come to an agreement on this. To be clear, this is also something I wonder about in the space, when people tell stories about how they get a job offer, they attempt to negotiate the offer, and then it gets withdrawn. There are two ways that goes. One is, “Well if you're not happy with this offer, get out of here.” Yeah, that is a crappy company, but there's also the story of people who don't know how to negotiate effectively, and in turn, they come back with indications that you do not know how to write a business email, you do not know how negotiations work, and suddenly, you're giving them a last-minute opportunity to get out before they hire someone who is going to be something of a wrecking ball in the company, and, “Whew, dodged a bullet on that.”I haven't encountered that scenario myself, but I've seen it from other folks and emails that have been passed around in various channels. So, my position on this is everyone should negotiate offers, but visit fearlesssalarynegotiation.com, it's run by my friend, Josh; he has a whole bunch of free content on his site. Look at it. Read it. It is how to handle this stuff effectively and why things are the way that they are. Follow his advice, and you won't go too far wrong. Again, I have no financial relationship, I just like what he's done a lot and I've been talking to him for years.Chris: Nice. I'll definitely check that out. [laugh].Corey: Another example is developher—that's develop H-E-R dot com. Someone else I've been speaking to who's great at this takes a different perspective on it, and that's fine. There's a lot of advice out there. Just make sure that whoever it is you're talking to about this is in a position to know what they're talking about because there's crap advice that's free. Yeah. How do you figure out the good advice and the bad advice? I'm worried someone out there is actually running Route 53 is a database for God's sake.Chris: That's crazy talk. Who would do that? That's madness.Corey: I can't imagine it.Chris: We're actually in the process of trying to figure out how to do a panel chat on exactly that, like, do a vBrownBag on salary negotiations, get some really good people in the room that can have a conversation around some of the tough questions that come around salary negotiation, what's too much to ask for? What kind of attitude should you go into it with? What kind of process should you have mentally? Is it scrawling in crayon, “No. More money,” and then hitting send? Or is it something a little bit more advanced?Corey: I also want to be clear that as you're building panels and stuff like that—because I got this wrong early on in my public speaking career, to be clear—I built talks aligned with this based on what worked for me—make sure that there are folks on the panel who are not painfully over-represented as you and I are because what works for us and we're considered oh, savvy business people who are great negotiators comes across as entitled, or demanding, or ooh, maybe we shouldn't hire her—and yes, I'm talking about her in a lot of these scenarios—make sure you have a diverse group of folks who can share lived experience and strategies that work because what works for you and me is not universal, I promise.Chris: So, the only requirement to set this panel is that you have to be a not-white guy; not-old-white guy. That's literally the one rule. [laugh].Corey: I like the approach. It's a good way to do it. I don't do manels.Chris: Yes. And it's tough because I'm not going to get into it, but the mental space that you have to be in to be a woman in tech, it's a delicate balance because when I'm approaching somebody, I don't want to slide into their DMs. It's like this, “Hey, I know this other person and they recommended you and I am not a weirdo.” [laugh]. As an old white guy, I have to be very not a weirdo when I'm talking to folks that I'm desperate to get on the show.Because I love having that diverse aspect, just different people from different backgrounds. Which is why we did the entire career series on vBrownBag. We did data science with Ayodele; we did how to get into cybersecurity with Christoph. It was a fantastic series of how to get into IT. This was at the beginning of the pandemic.We wanted to do a series on, okay, there's a lot of people out there that are furloughed right now. How do we get some people on the show that can talk to how to get into a part of IT that they're passionate about? We did a triple series on how to get into game development with Dennis Diack, the founder of Apocalypse Studios. We had a bunch of the other AWS Heroes from serverless, and Lambda, and AI on the show to talk, and it was really fantastic and I think it resonated well with the community.Corey: It takes work to have a group of guests on things like podcasts like this. You've been running vBrownBag for longer than I've been running this, and—Chris: 13 years now.Corey: Yeah. This is I think, coming up on what, four years-ish, maybe three, in that range? The passing of time, especially in a pandemic era, is challenging. And there's always a difference. If I invite a white dude to come on the podcast, the answer is yes before I get the word podcast fully out of my mouth, whereas folks who are not over-represented, they're a little more cautious. First, there's the question of, “Am I a trash bag?” And the answer is, “No.” Well, no, not in the way that you're concerned about other ways—Chris: [laugh]. That you're aware of. [laugh].Corey: Oh, God, yes, but—yeah. And then—and that's part of it, and then very often, there's a second one of, “Well, I don't think I have anything, really, to talk about,” is often a common objection here. And it's, yeah, if I'm inviting you on this show, I promise that's not true. Don't worry about that piece of it. And then it's the standard stuff that just comes with being me, of, “Yeah, I've read your Twitter feed; you got to insult me here?” It's, “No, no, not really the same tone. But great question; throw the”—it goes down to process. But it takes constant work, you can't just put an open call out for guest nominations, and expect that to wind up being representative of our industry. It is representative of our biases, in many respects.Chris: It's a tough needle to thread. Because the show has been around for a long time, it's easier for me now, because the show has been around for 13 years. We actually just recorded our two thousandth and sixtieth episode the other night. And even with that, getting that kind of outreach, [#techtwitter 00:31:32] is wonderful for making new recommendations of people. So, that's been really fun. The rest of Twitter is a hot trash fire, but that's beside the point. So yeah, I don't have a good solution for it. There's no easy answer for it other than to just be empathic, and communicative, and reach people on their level, and have a good show.Corey: And sometimes that's all it takes. The idea behind doing a podcast—despite my constant jokes—it's not out of a love affair of the sound of my own voice. It's about for better or worse, for reasons I don't fully understand, I have a platform. People listen to the show and they care what people have to say. So, my question is, how can I wind up using that platform to tell stories that lift up narratives that are helpful for folks that they can use as inspiration—in my case, as critical warnings of what to avoid—and effectively showcasing some of the best our industry has to offer, in many respects.So, if the guest has a good time and the audience can learn something, and I'm not accidentally perpetuating horrifying things, that's really more than I have any right to ask from a show like this. The fact that it's succeeded is due in no small part to not just an amazing audience, but also guests like you. So, thank you.Chris: Oh no, Thank you. And it is. It's… these kinds of shows are super fun. If it wasn't fun, I wouldn't have done it for as long as I have. I still enjoy chatting with folks and getting new voices.I love that first-time presenter who was, like, super nervous and I spend 15 minutes with them ahead of the show, I say, “Okay, relax. It's just going to be me and you facing each other. We're going to have a good time. You're going to talk about something that you love talking about, and we're going to be nerds and do nerd stuff. This is me and you in front of a water cooler with a whiteboard just being geeks and talking about cool stuff. We're also going to record it and some amount of people is going to see it afterwards.” [laugh].And yeah, that's the part that I love. And then watching somebody like that turn into the keynote speaker at a conference ten years down the road. And I get to say, “Oh, I knew that person when.”Corey: I just want to be remembered by folks who look back fondly at some of the things that we talk about here. I don't even need credit, just yeah. People who see that they've learned things and carry them forward and spread to others, there's so many favors that people have done for us that we can only ever pay forward.Chris: Yeah, exactly. So—and that's actually how I got into vBrownBag. I came to them saying, “Hey, I love the things that you guys have done. I actually passed my VCIX because of watching vBrownBags. What can I do to help contribute back to the community?” And Alistair said, “Funny you should mention that.” [laugh]. And here we are seven years later.Corey: Well, to that end, if people are inspired by what you're saying and they want to hear more about what you have to say or, heaven forbid, follow in your footsteps, where can they find you?Chris: So, you can find me on Twitter; I am at mistwire.com—M-I-S-T-W-I-R-E; if you Google ‘mistwire,' I am the first three pages of hits; so I have a blog; you can find me on vBrownBag. I'm hard to miss on Twitter [laugh] I discourage you from following me there. But yeah, you can hit me up on all of the formats. And if you want to present, I'd love to get you on the show. If you want to learn more about what it takes to become an AWS Hero or if you want to get into that line of work, I highly discourage it. It's a long slog but it's a—yeah, I'd love to talk to you.Corey: And we of course put links to that in the [show notes 00:35:01]. Thank you so much for taking the time to speak with me, Chris. I really appreciate it.Chris: Thank you, Corey. Thanks for having me on.Corey: Chris Williams, Enterprise Architect, comma AWS Cloud at WWT. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me that while you didn't actively enjoy this episode, you are at least open to enjoying future episodes if I have one that might potentially be exciting.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

The infrastructure that networked applications lives on is getting more and more complicated. There was a time when you could serve an application from a single machine on premises. But now, with cloud computing offering painless scaling to meet your demand, your infrastructure becomes abstracted and not really something you have contact with directly. Compound that problem with with architecture spread across dozens, even hundreds of microservices, replicated across multiple data centers in an ever changing cloud, and tracking down the source of system failures becomes something like a murder mystery. Who shot our uptime in the foot? A good observability system helps with that. On this sponsored episode of the Stack Overflow Podcast, we talk with Greg Leffler of Splunk about the keys to instrumenting an observable system and how the OpenTelemetry standard makes observability easier, even if you aren't using Splunk's product. Observability is really an outgrowth of traditional monitoring. You expect that some service or system could break, so you keep an eye on it. But observability applies that monitoring to an entire system and gives you the ability to answer the unexpected questions that come up. It uses three principal ways of viewing system data: logs, traces, and metrics.Metrics are a number and a timestamp that tell you particular details. Traces follow a request through a system. And logs are the causes and effects recorded from a system in motion. Splunk wants to add a fourth one—events—that would track specific user events and browser failures. Observing all that data first means you have to be able to track and extract that data by instrumenting your system to produce it. Greg and his colleagues at Splunk are huge fans of OpenTelemetry. It's an open standard that can extract data for any observability platform. You instrument your application once and never have to worry about it again, even if you need to change your observability platform. Why use an approach that makes it easy for a client to switch vendors? Leffler and Splunk argue that it's not only better for customers, but for Splunk and the observability industry as a whole. If you've instrumented your system with a vendor locked solution, then you may not switch, you may just let your observability program fall by the wayside. That helps exactly no one. As we've seen, people are moving to the cloud at an ever faster pace. That's no surprise; it offers automatic scaling for arbitrary traffic volumes, high availability, and worry-free infrastructure failure recovery. But moving to the cloud can be expensive, and you have to do some work with your application to be able to see everything that's going on inside it. Plenty of people just throw everything into the cloud and let the provider handle it, which is fine until they see the bill.Observability based on an open standard makes it easier for everyone to build a more efficient and robust service in the cloud. Give the episode a listen and let us know what you think in the comments.

The infrastructure that networked applications lives on is getting more and more complicated. There was a time when you could serve an application from a single machine on premises. But now, with cloud computing offering painless scaling to meet your demand, your infrastructure becomes abstracted and not really something you have contact with directly. Compound that problem with with architecture spread across dozens, even hundreds of microservices, replicated across multiple data centers in an ever changing cloud, and tracking down the source of system failures becomes something like a murder mystery. Who shot our uptime in the foot? A good observability system helps with that. On this sponsored episode of the Stack Overflow Podcast, we talk with Greg Leffler of Splunk about the keys to instrumenting an observable system and how the OpenTelemetry standard makes observability easier, even if you aren't using Splunk's product. Observability is really an outgrowth of traditional monitoring. You expect that some service or system could break, so you keep an eye on it. But observability applies that monitoring to an entire system and gives you the ability to answer the unexpected questions that come up. It uses three principal ways of viewing system data: logs, traces, and metrics.Metrics are a number and a timestamp that tell you particular details. Traces follow a request through a system. And logs are the causes and effects recorded from a system in motion. Splunk wants to add a fourth one—events—that would track specific user events and browser failures. Observing all that data first means you have to be able to track and extract that data by instrumenting your system to produce it. Greg and his colleagues at Splunk are huge fans of OpenTelemetry. It's an open standard that can extract data for any observability platform. You instrument your application once and never have to worry about it again, even if you need to change your observability platform. Why use an approach that makes it easy for a client to switch vendors? Leffler and Splunk argue that it's not only better for customers, but for Splunk and the observability industry as a whole. If you've instrumented your system with a vendor locked solution, then you may not switch, you may just let your observability program fall by the wayside. That helps exactly no one. As we've seen, people are moving to the cloud at an ever faster pace. That's no surprise; it offers automatic scaling for arbitrary traffic volumes, high availability, and worry-free infrastructure failure recovery. But moving to the cloud can be expensive, and you have to do some work with your application to be able to see everything that's going on inside it. Plenty of people just throw everything into the cloud and let the provider handle it, which is fine until they see the bill.Observability based on an open standard makes it easier for everyone to build a more efficient and robust service in the cloud. Give the episode a listen and let us know what you think in the comments.

Os sistemas têm apresentado cada vez mais complexidade para serem administrados, por isso a adoção de ferramentas de observabilidade se torna tão relevante e ganha, aqui no Kubicast, uma série especial sobre o assunto. Sejam bem-vindos(as) à Saga de Observabilidade do Kubicast! No episódio da vez, o papo é com o excelentíssimo Igor Simões, Sales Engineer da Dynatrace, que entrega como a feature mais top da sua plataforma a visão de ponta a ponta de todo o processo de suas aplicações. A ferramenta deles também suporta o OpenTelemetry de forma nativa.Além de explicar tudo sobre o recurso da solução, o Igor comenta sobre os vários desafios da observabilidade no dia a dia. CONTATO: Para conversar com o Igor, escreva para:E-mail: igor.simoes@dynatrace.comLinkedIn: /krishnarupa/Twitter: @krishnarupaGitHub: aborigenes O LINK do episódio do Kubicast sobre a Black Friday segue aqui. As RECOMENDAÇÕES do programa são:Igor:Oficina Mãos na Graxa - Ajudar como voluntário(a) na manutenção de bicicletas para doar a crianças no Natal 2021. É um projeto da ONG: Ciclo Ativo. João: Documentário sobre o Schumacher  (Tá na Netflix!)CPF do #KCDBrasil - Se você acompanhou a Kubecon e está com umas ideias para compartilhar com a comunidade Kubernetes, escreva-as aqui até 29/10. O KCD Brasil vai acontecer em 18 de janeiro do ano que vem e a programação do evento sai no início de novembro! O Kubicast é uma produção da Getup, especialista em Kubernetes e apoiadora do projeto UnDistro, uma distribuição para gerenciar múltiplos clusters Kubernetes. #o11y #Observability #OpenTelemetry #DevOps #Kubernetes #Containers #docker 

About ChloeChloe is a Bay Area based Cloud Advocate for Microsoft. Previously, she worked at Sentry.io where she created the award winning Sentry Scouts program (a camp themed meet-up ft. patches, s'mores, giant squirrel costumes, and hot chocolate), and was featured in the Grace Hopper Conference 2018 gallery featuring 15 influential women in STEM by AnitaB.org. Her projects and work with Azure have ranged from fake boyfriend alerts to Mario Kart 'astrology', and have been featured in VICE, The New York Times, as well as SmashMouth's Twitter account. Chloe holds a BA in Drama from San Francisco State University and is a graduate of Hackbright Academy. She prides herself on being a non-traditional background engineer, and is likely one of the only engineers who has played an ogre, crayon, and the back-end of a cow on a professional stage. She hopes to bring more artists into tech, and more engineers into the arts.Links: Twitter: https://twitter.com/ChloeCondon Instagram: https://www.instagram.com/gitforked/ YouTube: https://www.youtube.com/c/ChloeCondonVideos TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Somehow in the years this show has been running, I've only had Chloe Condon on once. In that time, she's over for dinner at my house way more frequently than that, but somehow the stars never align to get us together in front of microphones and have a conversation. First, welcome back to the show, Chloe. You're a senior cloud advocate at Microsoft on the Next Generation Experiences Team. It is great to have you here.Chloe: I'm back, baby. I'm so excited. This is one of my favorite shows to listen to, and it feels great to be a repeat guest, a friend of the pod. [laugh].Corey: Oh, yes indeed. So, something-something cloud, something-something Microsoft, something-something Azure, I don't particularly care, in light of what it is you have going on that you have just clued me in on, and we're going to talk about that to start. You're launching something new called Master Creep Theatre and I have a whole bunch of questions. First and foremost, is it theater or theatre? How is that spelled? Which—the E and the R, what direction does that go in?Chloe: Ohh, I feel like it's going to be the R-E because that makes it very fancy and almost British, you know?Corey: Oh, yes. And the Harlequin mask direction it goes in, that entire aesthetic, I love it. Please tell me what it is. I want to know the story of how it came to be, the sheer joy I get from playing games with language alone guarantee I'm going to listen to whatever this is, but please tell me more.Chloe: Oh, my goodness. Okay, so this is one of those creative projects that's been on my back burner forever where I'm like, someday when I have time, I'm going to put all my time [laugh] and energy into this. So, this originally stemmed from—if you don't follow me on Twitter, oftentimes when I'm not tweeting about '90s nostalgia, or Clippy puns, or Microsoft silly throwback things to Windows 95, I get a lot of weird DMs. On every app, not just Twitter. On Instagram, Twitter, LinkedIn, oh my gosh, what else is there?Corey: And I don't want to be clear here just to make this absolutely crystal clear, “Hey, Chloe, do you want to come back on Screaming in the Cloud again?” Is not one of those weird DMs to which you're referring?Chloe: No, that is a good DM. So, people always ask me, “Why don't you just close your DMs?” Because a lot of high profile people on the internet just won't even have their DMs open.Corey: Oh, I understand that, but I'm the same boat. I would have a lot less nonsense, but at the same time, I want—at least in my case—I want people to be able to reach out to me because the only reason I am what I am is that a bunch of people who had no reason to do it did favors for me—Chloe: Yes.Corey: —and I can't ever repay it, I can only ever pay it forward and that is the cost of doing favors. If I can help someone, I will, and that's hard to do with, “My DMs are closed so hunt down my email address and send me an email,” and I'm bad at email.Chloe: Right. I'm terrible at email as well, and I'm also terrible at DMs [laugh]. So, I think a lot of folks don't understand the volume at which I get messages, which if you're a good friend of mine, if you're someone like Corey or a dear friend like Emily, I will tell you, “Hey, if you actually need to get ahold of me, text me.” And text me a couple times because I probably see it and then I have ADHD, so I won't immediately respond. I think I respond in my head but I don't.But I get anywhere from, I would say, ohh, like, 30 on a low day to 100 on a day where I have a viral tweet about getting into tech with a non-traditional background or something like that. And these DMs that I get are really lovely messages like, “Thank you for the work you do,” or, “I decided to do a cute manicure because the [laugh] manicure you posted,” too, “How do I get into tech? How do I get a job at Microsoft?” All kinds of things. It runs the gamut between, “Where's your shirt from?” Where—[laugh]—“What's your mother's maiden name?”But a lot of the messages that I get—and if you're a woman on the internet with any sort of presence, you know how there's that, like—what's it called in Twitter—the Other Messages feature that's like, “Here's the people you know. Here's the people”—the message requests. For the longest time were just, “Hey,” “Hi,” “Hey dear,” “Hi pretty,” “Hi ma'am,” “Hello,” “Love you,” just really weird stuff. And of course, everyone gets these; these are bots or scammers or whatever they may be—or just creeps, like weird—and always the bio—not always but I [laugh] would say, like, these accounts range from either obviously a bot where it's a million different numbers, an account that says, “Father, husband, lover of Jesus Christ and God.” Which is so [laugh] ironic… I'm like, “Why are you in my DMs?”Corey: A man of God, which is why I'm in your DMs being creepy.Chloe: Exactly. Or—Corey: Just like Christ might have.Chloe: And you would be shocked, Corey, at how many. The thing that I love to say is Twitter is not a dating site. Neither is LinkedIn. Neither is Instagram. I post about my boyfriend all the time, who you've met, and we adore Ty Smith, but I've never received any unsolicited images, knock on wood, but I'm always getting these very bait-y messages like, “Hey, beautiful. I want to take you out.” And you would be shocked at how many of these people are doing it from their professional business account. [laugh]. Like, works at AWS, works at Google; it's like, oh my God. [laugh].Corey: You get this under your name, right? It ties back to it. Meanwhile—again, this is one of those invisible areas of privilege that folks who look like me don't have to deal with. My DM graveyard is usually things like random bot accounts, always starting with, “Hi,” or, “Hey.” If you want to guarantee I never respond to you, that is what you say. I just delete those out of hand because I don't notice or care. It is either a bot, or a scam, or someone who can't articulate what they're actually trying to get from me—Chloe: Exactly.Corey: —and I don't have the time for it. Make your request upfront. Don't ask to ask; just ask.Chloe: I think it's important to note, also, that I get a lot of… different kinds of these messages and they try to respond to everyone. I cannot. If I responded to everybody's messages that I got, I just wouldn't have any time to do my job. But the thing that I always say to people—you know, and managers have told me in the past, my boyfriend has encouraged me to do this, is when people say things like, “Close your DMs,” or, “Just ignore them,” I want to have the same experience that everybody else has on the internet. Now, it's going to be a little different, of course, because I look and act and sound like I do, and of course, podcasts are historically a visual medium, so I'm a five-foot-two, white, bright orange-haired girl; I'm a very quirky individual.Corey: Yes, if you look up ‘quirky,' you're right there under the dictionary definition. And every time—like, when we were first hanging out and you mentioned, “Oh yeah, I used to be in theater.” And it's like, “You know, you didn't even have to tell me that, on some level.” Which is not intended to be an insult. It's just theater folks are a bit of a type, and you are more or less the archetype of what a theatre person is, at least to my frame of reference.Chloe: And not only that, but I did musicals, so you can't see the jazz hands now, but–yeah, my degree is in drama. I come from that space and I just, you know, whenever people say, “Just ignore it,” or, “Close your DMs,” I'm like, I want people to be able to reach out to me; I want to be able to message one-on-one with Corey and whoever, when—as needed, and—Corey: Why should I close my DMs?Chloe: Yeah.Corey: They're the ones who suck. Yeah.Chloe: [laugh]. But over the years, to give people a little bit of context, I've been working in tech a long time—I've been working professionally in the DevRel space for about five or six years now—but I've worked in tech a long time, I worked as a recruiter, an office admin, executive assistant, like, I did all of the other areas of tech, but it wasn't until I got a presence on Twitter—which I've only been on Twitter for I think five years; I haven't been on there that long, actively. And to give some context on that, Twitter is not a social media platform used in the theater space. We just use Instagram and Facebook, really, back in the day, I'm not on Facebook at all these days. So, when I discovered Twitter was cool—and I should also mention my boyfriend, Ty, was working at Twitter at the time and I was like, “Twitter's stupid. Who would go on this—[laugh] who uses this app?”Fast-forward to now, I'm like—Ty's like, “Can you please get off Twitter?” But yeah, I think I've just been saving these screenshots over the last five or so years from everything from my LinkedIn, from all the crazy stuff that I dealt with when people thought I was a Bitcoin influencer to people being creepy. One of the highlights that I recently found when I was going back and trying to find these for this series that I'm doing is there was a guy from Australia, DMed me something like, “Hey, beautiful,” or, “Hey, sexy,” something like that. And I called him out. And I started doing this thing where I would post it on Twitter.I would usually hide their image with a clown emoji or something to make it anonymous, or not to call them out, but in this one I didn't, and this guy was defending himself in the comments, and to me in my DM's saying, “Oh, actually, this was a social experiment and I have all the screenshots of this,” right? So, imagine if you will—so I have conversations ranging from things like that where it's like, “Actually I messaged a bunch of people about that because I'm doing a social experiment on how people respond to, ‘Hey beautiful. I'd love to take you out some time in Silicon Valley.'” just the weirdest stuff right? So, me being the professional performer that I am, was like, these are hilarious.And I kept thinking to myself, anytime I would get these messages, I was like, “Does this work?” If you just go up to someone and say, “Hey”—do people meet this way? And of course, you get people on Twitter who when you tweet something like that, they're like, “Actually, I met my boyfriend in Twitter DMs,” or like, “I met my boyfriend because he slid into my DMs on Instagram,” or whatever. But that's not me. I have a boyfriend. I'm not interested. This is not the time or the place.So, it's been one of those things on the back burner for three or four years that I've just always been saving these images to a folder, thinking, “Okay, when I have the time when I have the space, the creative energy and the bandwidth to do this,” and thankfully for everyone I do now, I'm going to do dramatic readings of these DMs with other people in tech, and show—not even just to make fun of these people, but just to show, like, how would this work? What do you expect the [laugh] outcome to be? So Corey, for example, if you were to come on, like, here's a great example. A year ago—this is 2018; we're in 2021 right now—this guy messaged me in December of 2018, and was like, “Hey,” and then was like, “I would love to be your friend.” And I was like, “Nope,” and I responded, “Nope, nope, nope, nope.” There's a thread of this on Twitter. And then randomly, three weeks ago, just sent me this video to the tune of Enrique Iglesias' “Rhythm Divine” of just images of himself. [laugh]. So like, this comedy [crosstalk 00:10:45]—Corey: Was at least wearing pants?Chloe: He is wearing pants. It's very confusing. It's a picture—a lot of group photos, so I didn't know who he was. But in my mind because, you know, I'm an engineer, I'm trying to think through the end-user experience. I'm like, “What was your plan here?”With all these people I'm like, “So, your plan is just to slide into my DMs and woo me with ‘Hey'?” [laugh]. So, I think it'll be really fun to not only just show and call out this behavior but also take submissions from other people in the industry, even beyond tech, really, because I know anytime I tweet an example of this, I get 20 different women going, “Oh, my gosh, you get these weird messages, too?” And I really want to show, like, A, to men how often this happens because like you said, I think a lot of men say, “Just ignore it.” Or, “I don't get anything like that. You must be asking for it.”And I'm like, “No. This comes to me. These people find us and me and whoever else out there gets these messages,” and I'm just really ready to have a laugh at their expense because I've been laughing for years. [laugh].Corey: Back when I was a teenager, I was working in some fast food style job, and one of my co-workers saw customer, walked over to her, and said, “You're beautiful.” And she smiled and blushed. He leaned in and kissed her.Chloe: Ugh.Corey: And I'm sitting there going what on earth? And my other co-worker leaned over and is like, “You do know that's his girlfriend, right?” And I have to feel like, on some level, that is what happened to an awful lot of these broken men out on the internet, only they didn't have a co-worker to lean over and say, “Yeah, they actually know each other.” Which is why we see all this [unintelligible 00:12:16] behavior of yelling at people on the street as they walk past, or from a passing car. Because they saw someone do a stunt like that once and thought, “If it worked for them, it could work for me. It only has to work once.”And they're trying to turn this into a one day telling the grandkids how they met their grandmother. And, “Yeah, I yelled at her from a construction site, and it was love at first ‘Hey, baby.'” That is what I feel is what's going on. I have never understood it. I look back at my dating history in my early 20s, I look back now I'm like, “Ohh, I was not a great person,” but compared to these stories, I was a goddamn prince.Chloe: Yeah.Corey: It's awful.Chloe: It's really wild. And actually, I have a very vivid memory, this was right bef—uh, not right before the pandemic, but probably in 2019. I was speaking on a lot of conferences and events, and I was at this event in San Jose, and there were not a lot of women there. And somehow this other lovely woman—I can't remember her name right now—found me afterwards, and we were talking and she said, “Oh, my God. I had—this is such a weird event, right?”And I was like, “Yeah, it is kind of a weird vibe here.” And she said, “Ugh, so the weirdest thing happened to me. This guy”—it was her first tech conference ever, first of all, so you know—or I think it was her first tech conference in the Bay Area—and she was like, “Yeah, this guy came to my booth. I've been working this booth over here for this startup that I work at, and he told me he wanted to talk business. And then I ended up meeting him, stupidly, in my hotel lobby bar, and it's a date. Like, this guy is taking me out on a date all of a sudden,” and she was like, “And it took me about two minutes to just to be like, you know what? This is inappropriate. I thought this is going to be a business meeting. I want to go.”And then she shows me her hands, Corey, and she has a wedding ring. And she goes, “I'm not married. I have bought five or six different types of rings on Wish App”—or wish.com, which if you've never purchased from Wish before, it's very, kind of, low priced jewelry and toys and stuff of that nature. And she said, “I have a different wedding ring for every occasion. I've got my beach fake wedding ring. I've got my, we-got-married-with-a-bunch-of-mason-jars-in-the-woods fake wedding ring.”And she said she started wearing these because when she did, she got less creepy guys coming up to her at these events. And I think it's important to note, also, I'm not putting it out there at all that I'm interested in men. If anything, you know, I've been [laugh] with my boyfriend for six years never putting out these signals, and time and time again, when I would travel, I was very, very careful about sharing my location because oftentimes I would be on stage giving a keynote and getting messages while I delivered a technical keynote saying, “I'd love to take you out to dinner later. How long are you in town?” Just really weird, yucky, nasty stuff that—you know, and everyone's like, “You should be flattered.”And I'm like, “No. You don't have to deal with this. It's not like a bunch of women are wolf-whistling you during your keynote and asking what your boob size is.” But that's happening to me, and that's an extra layer that a lot of folks in this industry don't talk about but is happening and it adds up. And as my boyfriend loves to remind me, he's like, “I mean, you could stop tweeting at any time,” which I'm not going to do. But the more followers you get, the more inbound you get. So—Corey: Right. And the hell of it is, it's not a great answer because it's closing off paths of opportunity. Twitter has—Chloe: Absolutely.Corey: —introduced me to clients, introduced me to friends, introduced me to certainly an awful lot of podcast guests, and it informs and shapes a lot of the opinions that I hold on these things. And this is an example of what people mean when they talk about privilege. Where, yeah, “Look at Corey”—I've heard someone say once, and, “Nothing was handed to him.” And you're right, to be clear, I did not—like, no one handed me a microphone and said, “We're going to give you a podcast, now.” I had to build this myself.But let's be clear, I had no headwinds of working against me while I did it. There's the, you still have to do things, but you don't have an entire cacophony of shit heels telling you that you're not good enough in a variety of different ways, to subtly reinforcing your only value is the way that you look. There isn't this whole, whenever you get something wrong and it's a, “Oh, well, that's okay. We all get things wrong.” It's not the, “Girls suck at computers,” trope that we see so often.There's a litany of things that are either supportive that work in my favor, or are absent working against me that is privilege that is invisible until you start looking around and seeing it, and then it becomes impossible not to. I know I've talked about this before on the show, but no one listens to everything and I just want to subtly reinforce that if you're one of those folks who will say things like, “Oh, privilege isn't real,” or, “You can have bigotry against white people, too.” I want to be clear, we are not the same. You are not on my side on any of this, and to be very direct, I don't really care what you have to say.Chloe: Yeah. And I mean, this even comes into play in office culture and dynamics as well because I am always the squeaky wheel in the room on these kind of things, but a great example that I'll give is I know several women in this industry who have had issues when they used to travel for conferences of being stalked, people showing up at their hotel rooms, just really inappropriate stuff, and for that reason, a lot of folks—including myself—wouldn't pick the conference event—like, typically they'll be like, “This is the hotel everyone's staying at.” I would very intentionally stay at a different hotel because I didn't want people knowing where I was staying. But I started to notice once a friend of mine, who had an issue with this [unintelligible 00:17:26], I really like to be private about where I'm staying, and sometimes if you're working at a startup or larger company, they'll say, “Hey, everyone put in this Excel spreadsheet or this Google Doc where everyone's staying and how to contact them, and all this stuff.” And I think it's really important to be mindful of these things.I always say to my friends—I'm not going out too much these days because it's a pandemic—and I've done Twitter threads on this before where I never post my location; you will never see me. I got rid of Swarm a couple [laugh] years ago because people started showing up where I was. I posted photos before, you know, “Hey, at the lake right now.” And people have shown up. Dinners, people have recognized me when I've been out.So, I have an espresso machine right over here that my lovely boyfriend got me for my birthday, and someone commented, “Oh, we're just going to act like we don't see someone's reflection in the”—like, people Zoom in on images. I've read stories from cosplayers online who, they look into the reflection of a woman's glasses and can figure out where they are. So, I think there's this whole level. I'm constantly on alert, especially as a woman in tech. And I have friends here in the Bay Area, who have tweeted a photo at a barbecue, and then someone was like, “Hey, I live in the neighborhood, and I recognize the tree.”First of all, don't do that. Don't ever do that. Even if you think you're a nice, unassuming guy or girl or whatever, don't ever [laugh] do that. But I very intentionally—people get really confused, my friends specifically. They're like, “Wait a second, you're in Hawaii right now? I thought you were in Hawaii three weeks ago.” And I'm like, “I was. I don't want anyone even knowing what island or continent I'm on.”And that's something that I think about a lot. When I post photo—I never post any photos from my window. I don't want people knowing what my view is. People have figured out what neighborhood I live in based on, like, “I know where that graffiti is.” I'm very strategic about all this stuff, and I think there's a lot of stuff that I want to share that I don't share because of privacy issues and concerns about my safety. And also want to say and this is in my thread on online safety as well is, don't call out people's locations if you do recognize the image because then you're doxxing them to everyone like, “Oh”—Corey: I've had a few people do that in response to pictures I've posted before on a house, like, “Oh, I can look at this and see this other thing and then intuit where you are.” And first, I don't have that sense of heightened awareness on this because I still have this perception of myself as no one cares enough to bother, and on the other side, by calling that out in public. It's like, you do not present yourself well at all. In fact, you make yourself look an awful lot like the people that we're warned about. And I just don't get that.I have some of these concerns, especially as my audience has grown, and let's be very clear here, I antagonize trillion-dollar companies for a living. So, first if someone's going to have me killed, they can find where I am. That's pretty easy. It turns out that having me whacked is not even a rounding error on most of these companies' budgets, unfortunately. But also I don't have that level of, I guess, deranged superfan. Yet.But it happens in the fullness of time, as people's audiences continue to grow. It just seems an awful lot like it happens at much lower audience scale for folks who don't look like me. I want to be clear, this is not a request for anyone listening to this, to try and become that person for me, you will get hosed, at minimum. And yes, we press charges here.Chloe: AWSfan89, sliding into your DMs right after this. Yeah, it's also just like—I mean, I don't want to necessarily call out what company this was at, but personally, I've been in situations where I've thrown an event, like a meetup, and I'm like, “Hey, everyone. I'm going to be doing ‘Intro to blah, blah, blah' at this time, at this place.” And three or four guys would show up, none of them with computers. It was a freaking workshop on how to do or deploy something, or work with an API.And when I said, “Great, so why'd you guys come to this session today?” And maybe two have iPads, one just has a notepad, they're like, “Oh, I just wanted to meet you from Twitter.” And it's like, okay, that's a little disrespectful to me because I am taking time out to do this workshop on a very technical thing that I thought people were coming here to learn. And this isn't the Q&A. This is not your meet-and-greet opportunity to meet Chloe Condon, and I don't know why you would, like, I put so much of my life online [laugh] anyway.But yeah, it's very unsettling, and it's happened to me enough. Guys have shown up to my events and given me gifts. I mean, I'm always down for a free shirt or something, but it's one of those things that I'm constantly aware of and I hate that I have to be constantly aware of, but at the end of the day, my safety is the number one priority, and I don't want to get murdered. And I've tweeted this out before, our friend Emily, who's similarly a lady on the internet, who works with my boyfriend Ty over at Uber, we have this joke that's not a joke, where we say, “Hey if I'm murdered, this is who it was.” And we'll just send each other screenshots of creepy things that people either tag us in, or give us feedback on, or people asking what size shirt we are. Just, wiki feed stuff, just really some of the yucky of the yuck out there.And I do think that unless you have a partner, or a family member, or someone close enough to you to let you know about these things—because I don't talk about these things a lot other than my close friends, and maybe calling out a weirdo here and there in public, but I don't share the really yucky stuff. I don't share the people who are asking what neighborhood I live in. I'm not sharing the people who are tagging me, like, [unintelligible 00:22:33], really tagging me in some nasty TikToks, along with some other women out there. There are some really bad actors in this community and it is to the point where Emily and I will be like, “Hey, when you inevitably have to solve my murder, here's the [laugh] five prime suspects.” And that sucks. That's [unintelligible 00:22:48] joke; that isn't a joke, right? I suspect I will either die in an elevator accident or one of my stalkers will find me. [laugh].Corey: It's easy for folks to think, oh, well, this is a Chloe problem because she's loud, she's visible, she's quirky, she's different than most folks, and she brings it all on herself, and this is provably not true. Because if you talk to, effectively, any woman in the world in-depth about this, they all have stories that look awfully similar to this. And let me forestall some of the awful responses I know I'm going to get. And, “Well, none of the women I know have had experiences like this,” let me be very clear, they absolutely have, but for one reason or another, they either don't see the need, or don't see the value, or don't feel safe talking to you about it.Chloe: Yeah, absolutely. And I feel a lot of privilege, I'm very lucky that my boyfriend is a staff engineer at Uber, and I have lots of friends in high places at some of these companies like Reddit that work with safety and security and stuff, but oftentimes, a lot of the stories or insights or even just anecdotes that I will give people on their products are invaluable insights to a lot of these security and safety teams. Like, who amongst us, you know, [laugh] has used a feature and been like, “Wait a second. This is really, really bad, and I don't want to tweet about this because I don't want people to know that they can abuse this feature to stalk or harass or whatever that may be,” but I think a lot about the people who don't have the platform that I have because I have 50k-something followers on Twitter, I have a pretty big online following in general, and I have the platform that I do working at Microsoft, and I can tweet and scream and be loud as I can about this. But I think about the folks who don't have my audience, the people who are constantly getting harassed and bombarded, and I get these DMs all the time from women who say, “Thank you so much for doing a thread on this,” or, “Thank you for talking about this,” because people don't believe them.They're just like, “Oh, just ignore it,” or just, “Oh, it's just one weirdo in his basement, like, in his mom's basement.” And I'm like, “Yeah, but imagine that but times 40 in a week, and think about how that would make you rethink your place and your position in tech and even outside of tech.” Let's think of the people who don't know how this technology works. If you're on Instagram at all, you may notice that literally not only every post, but every Instagram story that has the word COVID in it, has the word vaccine, has anything, and they must be using some sort of cognitive scanning type thing or scanning the images themselves because this is a feature that basically says, hey, this post mentioned COVID in some way. I think if you even use the word mask, it alerts this.And while this is a great feature because we all want accurate information coming out about the pandemic, I'm like, “Wait a minute. So, you're telling me this whole time you could have been doing this for all the weird things that I get into my DMs, and people post?” And, like, it just shows you, yes, this is a global pandemic. Yes, this is something that affects everyone. Yes, it's important we get information out about this, but we can be using these features in much [laugh] more impactful ways that protects people's safety, that protects people's ability to feel safe on a platform.And I think the biggest one for me, and I make a lot of bots; I make a lot of Twitter bots and chatbots, and I've done entire series on this about ethical bot creation, but it's so easy—and I know this firsthand—to make a Twitter account. You can have more than one number, you can do with different emails. And with Instagram, they have this really lovely new feature that if you block someone, it instantly says, “You just blocked so and so. Would you like to block any other future accounts they make?” I mean, seems simple enough, right?Like, anything related—maybe they're doing it by email, or phone number, or maybe it's by IP, but like, that's not being done on a lot of these platforms, and it should be. I think someone mentioned in one of my threads on safety recently that Peloton doesn't have a block user feature. [laugh]. They're probably like, “Well, who's going to harass someone on Peloton?” It would happen to me. If I had a Peloton, [laugh] I assure you someone would find a way to harass me on there.So, I always tell people, if you're working at a company and you're not thinking about safety and harassment tools, you probably don't have anybody LGBTQ+ women, non-binary on your team, first of all, and you need to be thinking about these things, and you need to be making them a priority because if users can interact in some way, they will stalk, harass, they will find some way to misuse it. It seems like one of those weird edge cases where it's like, “Oh, we don't need to put a test in for that feature because no one's ever going to submit, like, just 25 emojis.” But it's the same thing with safety. You're like, who would harass someone on an app about bubblegum? One of my followers were. [laugh].Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: The biggest question that doesn't get asked that needs to be in almost every case is, “Okay. We're building a thing, and it's awesome. And I know it's hard to think like this, but pivot around. Theoretically, what could a jerk do with it?”Chloe: Yes.Corey: When you're designing it, it's all right, how do you account for people that are complete jerks?Chloe: Absolutely.Corey: Even the cloud providers, all of them, when the whole Parler thing hit, everyone's like, “Oh, Amazon is censoring people for freedom of speech.” No, they're actually not. What they're doing is enforcing their terms of service, the same terms of service that every provider that is not trash has. It is not a problem that one company decided they didn't want hate speech on their platform. It was all the companies decided that, except for some very fringe elements. And that's the sort of thing you have to figure out is, it's easy in theory to figure out, oh, anything goes; freedom of speech. Great, well, some forms of speech violate federal law.Chloe: Right.Corey: So, what do you do then? Where do you draw the line? And it's always nuanced and it's always tricky, and the worst people are the folks that love to rules-lawyer around these things. It gets worse than that where these are the same people that will then sit there and make bad faith arguments all the time. And lawyers have a saying that hard cases make bad law.When you have these very nuanced thing, and, “Well, we can't just do it off the cuff. We have to build a policy around this.” This is the problem with most corporate policies across the board. It's like, you don't need a policy that says you're not allowed to harass your colleagues with a stick. What you need to do is fire the jackwagon that made you think you might need a policy that said that.But at scale, that becomes a super-hard thing to do when every enforcement action appears to be bespoke. Because there are elements on the gray areas and the margins where reasonable people can disagree. And that is what sets the policy and that's where the precedent hits, and then you have these giant loopholes where people can basically be given free rein to be the worst humanity has to offer to some of the most vulnerable members of our society.Chloe: And I used to give this talk, I gave it at DockerCon one year and I gave it a couple other places, that was literally called “Diversity is not Equal to Stock Images of Hands.” And the reason I say this is if you Google image search ‘diversity' it's like all of those clip arts of, like, Rainbow hands, things that you would see at Kaiser Permanente where it's like, “We're all in this together,” like, the pandemic, it's all just hands on hands, hands as a Earth, hands as trees, hands as different colors. And people get really annoyed with people like me who are like, “Let's shut up about diversity. Let's just hire who's best for the role.” Here's the thing.My favorite example of this—RIP—is Fleets—remember Fleets? [laugh]—on Twitter, so if they had one gay man in the room for that marketing, engineering—anything—decision, one of them I know would have piped up and said, “Hey, did you know ‘fleets' is a commonly used term for douching enima in the gay community?” Now, I know that because I watch a lot of Ru Paul's Drag Race, and I have worked with the gay community quite a bit in my time in theater. But this is what I mean about making sure. My friend Becca who works in security at safety and things, as well as Andy Tuba over at Reddit, I have a lot of conversations with my friend Becca Rosenthal about this, and that, not to quote Hamilton, but if I must, “We need people in the room where it happens.”So, if you don't have these people in the room if you're a white man being like, “How will our products be abused?” Your guesses may be a little bit accurate but it was probably best to, at minimum, get some test case people in there from different genders, races, backgrounds, like, oh my goodness, get people in that room because what I tend to see is building safety tools, building even product features, or naming things, or designing things that could either be offensive, misused, whatever. So, when people have these arguments about like, “Diversity doesn't matter. We're hiring the best people.” I'm like, “Yeah, but your product's going to be better, and more inclusive, and represent the people who use it at the end of the day because not everybody is you.”And great examples of this include so many apps out there that exists that have one work location, one home location. How many people in the world have more than one job? That's such a privileged view for us, as people in tech, that we can afford to just have one job. Or divorced parents or whatever that may be, for home location, and thinking through these edge cases and thinking through ways that your product can support everyone, if anything, by making your staff or the people that you work with more diverse, you're going to be opening up your product to a much bigger marketable audience. So, I think people will look at me and be like, “Oh, Chloe's a social justice warrior, she's this feminist whatever,” but truly, I'm here saying, “You're missing out on money, dude.” It would behoove you to do this at the end of the day because your users aren't just a copy-paste of some dude in a Patagonia jacket with big headphones on. [laugh]. There are people beyond one demographic using your products and applications.Corey: A consistent drag against Clubhouse since its inception was that it's not an accessible app for a variety of reasons that were—Chloe: It's not an Android. [laugh].Corey: Well, even ignoring the platform stuff, which I get—technical reasons, et cetera, yadda, yadda, great—there is no captioning option. And a lot of their abuse stuff in the early days was horrific, where you would get notifications that a lot of people had this person blocked, but… that's not a helpful dynamic. “Did you talk to anyone? No, of course not. You Hacker News'ed it from first principles and thought this might be a good direction to go in.” This stuff is hard.People specialize in this stuff, and I've always been an advocate of when you're not sure what to do in an area, pay an expert for advice. All these stories about how people reach out to, “Their black friend”—and yes, it's a singular person in many cases—and their black friend gets very tired of doing all the unpaid emotional labor of all of this stuff. Suddenly, it's not that at all if you reach out to someone who is an expert in this and pay them for their expertise. I don't sit here complaining that my clients pay me to solve AWS billing problems. In fact, I actively encourage that behavior. Same model.There are businesses that specialize in this, they know the area, they know the risks, they know the ins and outs of this, and consults with these folks are not break the bank expensive compared to building the damn thing in the first place.Chloe: And here's a great example that literally drove me bananas a couple weeks ago. So, I don't know if you've participated in Twitter Spaces before, but I've done a couple of my first ones recently. Have you done one yet—Corey: Oh yes—Chloe: —Corey?Corey: —extensively. I love that. And again, that's a better answer for me than Clubhouse because I already have the Twitter audience. I don't have to build one from scratch on another platform.Chloe: So, I learned something really fascinating through my boyfriend. And remember, I mentioned earlier, my boyfriend is a staff engineer at Uber. He's been coding since he's been out of the womb, much more experienced than me. And I like to think a lot about, this is accessible to me but how is this accessible to a non-technical person? So, Ty finished up the Twitter Space that he did and he wanted to export the file.Now currently, as the time of this podcast is being recorded, the process to export a Twitter Spaces audio file is a nightmare. And remember, staff engineer at Uber. He had to export his entire Twitter profile, navigate through a file structure that wasn't clearly marked, find the recording out of the multiple Spaces that he had hosted—and I don't think you get these for ones that you've participated in, only ones that you've hosted—download the file, but the file was not a normal WAV file or anything; he had to download an open-source converter to play the file. And in total, it took him about an hour to just get that file for the purposes of having that recording. Now, where my mind goes to is what about some woman who runs a nonprofit in the middle of, you know, Sacramento, and she does a community Twitter Spaces about her flower shop and she wants a recording of that.What's she going to do, hire some third-party? And she wouldn't even know where to go; before I was in tech, I certainly would have just given up and been like, “Well, this is a nightmare. What do I do with this GitHub repo of information?” But these are the kinds of problems that you need to think about. And I think a lot of us and folks who listen to this show probably build APIs or developer tools, but a lot of us do work on products that muggles, non-technical people, work on.And I see these issues happen constantly. I come from this space of being an admin, being someone who wasn't quote-unquote, “A techie,” and a lot of products are just not being thought through from the perspective—like, there would be so much value gained if just one person came in and tested your product who wasn't you. So yeah, there's all of these things that I think we have a very privileged view of, as technical folks, that we don't realize are huge. Not even just barrier to entry; you should just be able to download—and maybe this is a feature that's coming down the pipeline soon, who knows, but the fact that in order for someone to get a recording of their Twitter Spaces is like a multi-hour process for a very, very senior engineer, that's the problem. I'm not really sure how we solve this.I think we just call it out when we see it and try to help different companies make change, which of course, myself and my boyfriend did. We reached out to people at Twitter, and we're like, “This is really difficult and it shouldn't be.” But I have that privilege. I know people at these companies; most people do not.Corey: And in some cases, even when you do, it doesn't move the needle as much as you might wish that it would.Chloe: If it did, I wouldn't be getting DMs anymore from creeps right? [laugh].Corey: Right. Chloe, thank you so much for coming back and talk to me about your latest project. If people want to pay attention to it and see what you're up to. Where can they go? Where can they find you? Where can they learn more? And where can they pointedly not audition to be featured on one of the episodes of Master Creep Theatre?Chloe: [laugh]. So, that's the one caveat, right? I have to kind of close submissions of my own DMs now because now people are just going to be trolling me and sending me weird stuff. You can find me on Twitter—my name—at @chloecondon, C-H-L-O-E-C-O-N-D-O-N. I am on Instagram as @getforked, G-I-T-F-O-R-K-E-D. That's a Good Placepun if you're non-technical; it is an engineering pun if you are. And yeah, I've been doing a lot of fun series with Microsoft Reactor, lots of how to get a career in tech stuff for students, building a lot of really fun AI/ML stuff on there. So, come say hi on one of my many platforms. YouTube, too. That's probably where—Master Creep Theatre is going to be, on YouTube, so definitely follow me on YouTube. And yeah.Corey: And we will, of course, put links to that in the [show notes 00:37:57]. Chloe, thank you so much for taking the time to speak with me. I really appreciate it, as always.Chloe: Thank you. I'll be back for episode three soon, I'm sure. [laugh].Corey: Let's not make it another couple of years until then. Chloe Condon, senior cloud advocate at Microsoft on the Next Generation Experiences Team, also chlo-host of the Master Creep Theatre podcast. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment saying simply, “Hey.”Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About NickNick Heudecker leads market strategy and competitive intelligence at Cribl, the observability pipeline company. Prior to Cribl, Nick spent eight years as an industry analyst at Gartner, covering data and analytics. Before that, he led engineering and product teams at multiple startups, with a bias towards open source software and adoption, and served as a cryptologist in the US Navy. Join Corey and Nick as they discuss the differences between observability and monitoring, why organizations struggle to get value from observability data, why observability requires new data management approaches, how observability pipelines are creating opportunities for SRE and SecOps teams, the balance between budgets and insight, why goats are the world's best mammal, and more.Links: Cribl: https://cribl.io/ Cribl Community: https://cribl.io/community Twitter: https://twitter.com/nheudecker Try Cribl hosted solution: https://cribl.cloud TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted episode is a bit fun because I'm joined by someone that I have a fair bit in common with. Sure, I moonlight sometimes as an analyst because I don't really seem to know what that means, and he spent significant amounts of time as a VP analyst at Gartner. But more importantly than that, a lot of the reason that I am the way that I am is that I spent almost a decade growing up in Maine, and in Maine, there's not a lot to do other than sit inside for the nine months of winter every year and develop personality problems.You've already seen what that looks like with me. Please welcome Nick Heudecker, who presumably will disprove that, but maybe not. He is currently a senior director of market strategy and competitive intelligence at Cribl. Nick, thanks for joining me.Nick: Thanks for having me. Excited to be here.Corey: So, let's start at the very beginning. I like playing with people's titles, and you certainly have a lofty one. ‘competitive intelligence' feels an awful lot like jeopardy. What am I missing?Nick: Well, I'm basically an internal analyst at the company. So, I spend a lot of time looking at the broader market, seeing what trends are happening out there; looking at what kind of thought leadership content that I can create to help people discover Cribl, get interested in the products and services that we offer. So, I'm mostly—you mentioned my time in Maine. I was a cryptologist in the Navy and I spent almost all of my time focused on what the bad guys do. And in this job, I focus on what our potential competitors do in the market. So, I'm very externally focused. Does that help? Does that explain it?Corey: No, it absolutely does. I mean, you folks have been sponsoring our nonsense for which we thank you, but the biggest problem that I have with telling the story of Cribl was that originally—initially it was, from my perspective, “What is this hokey nonsense?” And then I learned and got an answer and then finish the sentence with, “And where can I buy it?” Because it seems that the big competitive threat that you have is something crappy that some rando sysadmin has cobbled together. And I say that as the rando sysadmin, who has cobbled a lot of things like that together. And it's awful. I wasn't aware you folks had direct competitors.Nick: Today we don't. There's a couple that it might be emerging a little bit, but in general, no, it's mostly us, and that's what I analyze every day. Are there other emerging companies in the space? Are there open-source projects? But you're right, most of the things that we compete against are DIY today. Absolutely.Corey: In your previous role, which you were at for a very long time in tech terms—which in a lot of other cases is, “Okay, that doesn't seem that long,” but seven and a half years is a respectable stint at a company. And you were at Gartner doing a number of analyst-like activities. Let's start at the beginning because I assure you, I'm asking this purely for the audience and not because I don't know the answer myself, but what exactly is the purpose of an analyst firm, of which Gartner is the most broadly known and, follow up, why do companies care what Gartner thinks?Nick: Yeah. It's a good question, one that I answer a lot. So, what is the purpose of an analyst firm? The purpose of an analyst firm is to get impartial information about something, whether that is supply chain technology, big data tech, human resource management technologies. And it's often difficult if you're an end-user and you're interested in say, acquiring a new piece of technology, what really works well, what doesn't.And so the analyst firm because in the course of a given year, I would talk to nearly a thousand companies and both end-users and vendors as well as investors about what they're doing, what challenges they're having, and I would distill that down into 30-minute conversations with everyone else. And so we provided impartial information in aggregate to people who just wanted to help. And that's the purpose of an analyst firm. Your second question, why do people care? Well, I didn't get paid by vendors.I got paid by the company that I worked for, and so I got to be Tron; I fought for the users. And because I talk to so many different companies in different geographies, in different industries, and I share that information with my colleagues, they shared with me, we had a very robust understanding of what's actually happening in any technology market. And that's uncommon kind of insight to really have in any kind of industry. So, that's the purpose and that's why people care.Corey: It's easy from the engineering perspective that I used to inhabit to make fun of it. It's oh, it's purely justification when you're making a big decision, so if it goes sideways—because find me a technology project that doesn't eventually go sideways—I want to be able to make sure that I'm not the one that catches heat for it because Gartner said it was good. They have an amazing credibility story going on there, and I used to have that very dismissive perspective. But the more I started talking to folks who are Gartner customers themselves and some of the analyst-style things that I do with a variety of different companies, it's turned into, “No, no. They're after insight.”Because it turns out, from my perspective at least, the more that you are focused on building a product that solves a problem, you sort of lose touch with the broader market because the only people you're really talking to are either in your space or have already acknowledged and been right there and become your customer and have been jaded to see things from your point of view. Getting a more objective viewpoint from an impartial third party does have value.Nick: Absolutely. And I want you to succeed, I want you to be successful, I want to carry on a relationship with all the clients that I would speak with, and so one of the fun things I would always ask is, “Why are you asking me this question now?” Sometimes it would come in, they'd be very innocuous;, “Compare these databases,” or, “Compare these cloud services.” “Well, why are you asking?” And that's when you get to, kind of like, the psychology of it.“Oh, we just hired a new CIO and he or she hates vendor X, so we have to get rid of it.” “Well, all right. Let's figure out how we solve this problem for you.” And so it wasn't always just technology comparisons. Technology is easy, you write a check and you hope for the best.But when you're dealing with large teams and maybe a globally distributed company, it really comes down to culture, and personality, and all the harder factors. And so it was always—those were always the most fun and certainly the most challenging conversations to have.Corey: One challenge that I find in this space is—in my narrow niche of the world where I focus on AWS bills, where things are extraordinarily yes or no, black or white, binary choices—that I talked to companies, like during the pandemic, and they were super happy that, “Oh, yeah. Our infrastructure has auto-scaling and it works super well.” And I look at the bill and the spend graph over time is so flat you could basically play a game of pool on top of it. And I don't believe that I'm talking to people who are lying to me. I truly don't believe that people make that decision, but what they believe versus what is evidenced in reality are not necessarily congruent. How do you disambiguate from the stories that people want to tell about themselves? And what they're actually doing?Nick: You have to unpack it. I think you have to ask a series of questions to figure out what their motivation is. Who else is on the call, as well? I would sometimes drop into a phone call and there would be a dozen people on the line. Those inquiry calls would go the worst because everyone wants to stake a claim, everyone wants to be heard, no one's going to be honest with you or with anyone else on the call.So, you typically need to have a pretty personal conversation about what does this person want to accomplish, what does the company want to accomplish, and what are the factors that are pushing against what those things are? It's like a novel, right? You have a character, the character wants to achieve something, and there are multiple obstacles in that person's way. And so by act five, ideally everything wraps up and it's perfect. And so my job is to get the character out of the tree that is on fire and onto the beach where the person can relax.So, you have to unpack a lot of different questions and answers to figure out, well, are they telling me what their boss wants to hear or are they really looking for help? Sometimes you're successful, sometimes you're not. Not everyone does want to be open and honest. In other cases, you would have a team show up to a call with maybe a junior engineer and they really just want you to tell them that the junior engineer's architecture is not a good idea. And so you do a lot of couples therapy as well. I don't know if this is really answering the question for you, but there are no easy answers. And people are defensive, they have biases, companies overall are risk-averse. I think you know this.Corey: Oh, yeah.Nick: And so it can be difficult to get to the bottom of what their real motivation is.Corey: My approach has always been that if you want serious data, you go talk to Gartner. If you want [anec-data 00:09:48] and some understanding, well, maybe we can have that conversation, but they're empowering different decisions at different levels, and that's fine. To be clear, I do not consider Gartner to be a competitor to what I do in any respect. It turns out that I am not very good at drawing charts in varying shades of blue and positioning things just so with repeatable methodology, and they're not particularly good at having cartoon animals as their mascot that they put into ridiculous situations. We each have our portion of the universe, and that's working out reasonably well.Nick: Well, and there's also something to unpack there as well because I would say that people look at Gartner and they think they have a lot of data. To a certain degree they do, but a lot of it is not quantifiable data. If you look at a firm like IDC, they specialize in—like, they are a data house; that is what they do. And so their view of the world and how they advise their clients is different. So, even within analyst firms, there is differentiation in what approach they take, how consultative they might be with their clients, one versus another. So, there certainly are differences that you could find the more exposure you get into the industry.Corey: For a while, I've been making a recurring joke that Route 53—Amazon's managed DNS service—is in fact a database. And then at some point, I saw a post on Reddit where someone said, “Yeah, I see the joke and it's great, but why should I actually not do this?” At which point I had to jump in and say, “Okay, look. Jokes are all well and good, but as soon as people start taking me seriously, it's very much time to come clean.” Because I think that's the only ethical and responsible thing to do in this ecosystem.Similarly, there was another great joke once upon a time. It was an April Fool's Day prank, and Google put out a paper about this thing they called MapReduce. Hilarious prank that Yahoo fell for hook, line, and sinker, and wound up building Hadoop out of it and we're still paying the price for that, years later. You have a bit of a reputation from your time at Gartner as being—and I quote—“The man who killed Hadoop.” What happened there? What's the story? And I appreciate your finally making clear to the rest of us that it was, in fact, a joke. What happened there?Nick: Well, one of the pieces of research that Gartner puts out every year is this thing called a Hype Cycle. And we've all seen it, it looks like a roller coaster in profile; big mountain goes up really high and then comes down steeply, drops into a valley, and then—Corey: ‘the trough of disillusionment,' as I recall.Nick: Yes, my favorite. And then plateaus out. And one of the profiles on that curve was Hadoop distributions. And after years of taking inquiry calls, and writing documents, and speaking with everybody about what they were doing, we realized that this really isn't taking off like everyone thinks it is. Cluster sizes weren't getting bigger, people were having a lot of challenges with the complexity, people couldn't find skills to run it themselves if they wanted to.And then the cloud providers came in and said, “Well, we'll make a lot of this really simple for you, and we'll get rid of HDFS,” which is—was a good idea, but it didn't really scale well. I think that the challenge of having to acquire computers with compute storage and memory again, and again, and again, and again, just was not sustainable for the majority of enterprises. And so we flagged it as this will be obsolete before plateau. And at that point, we got a lot of hate mail, but it just seemed like the right decision to make, right? Once again, we're Tron; we fight for the users.And that seemed like the right advice and direction to provide to the end-users. And so didn't make a lot of friends, but I think I was long-term right about what happened in the Hadoop space. Certainly, some fragments of it are left over and we're still seeing—you know, Spark is going strong, there's a lot of Hive still around, but Hadoop as this amalgamation of open-source projects, I think is effectively dead.Corey: I sure hope you're right. I think it has a long tail like most things that are there. Legacy is the condescending engineering term for ‘it makes money.' You were at Gartner for almost eight years and then you left to go work at Cribl. What triggered that? What was it that made you decide, “This is great. I've been here a long time. I've obviously made it work for me. I'm going to go work at a startup that apparently, even though it recently raised a $200 million funding round”—congratulations on that, by the way—“It still apparently can't afford to buy a vowel in its name.” That's C-R-I-B-L because, of course, it is. Maybe another consonant, while you're shopping. But okay, great. It's oddly spelled, it is hard to explain in some cases, to folks who are not already feeling pain in that space. What was it that made you decide to sit up and, “All right, this is where I want to be?”Nick: Well, I met the co-founders when I was an analyst. They were working at Splunk and oddly enough—this is going to be an interesting transition compared to the previous thing we talked about—they were working on Hunk, which was, let's use HDFS to store Splunk data. Made a lot of sense, right? It could be much more cost-effective than high-cost infrastructure for Splunk. And so they told me about this; I was interested.And so I met the co-founders and then I reconnected with them after they left and formed Cribl. And I thought the story was really cool because where they're sitting is between sources and destinations of observability data. And they were solving a problem that all of my customers had, but they couldn't resolve. They would try and build it themselves. They would look at—Kafka was a popular choice, but that had some challenges for observability data—works fantastically well for application data.And they were just—had a very pragmatic view of the world that they were inhabiting and the problem that they were looking to solve. And it looked kind of like a no-brainer of a problem to solve. But when you double-click on it, when you really look down and say, “All right, what are the challenges with doing this?” They're really insurmountable for a lot of organizations. So, even though they may try and take a DIY approach, they often run into trouble after just a few weeks because of all the protocols you have to support, all the different data formats, and all the destinations, and role-based access control, and everything else that goes along with it.And so I really liked the team. I thought the product inhabited a unique space in the market—we've already talked about the lack of competitors in the space—and I just felt like the company was on a rocket ship—or is a rocket ship—that basically had unbounded success potential. And so when the opportunity arose to join the team and do a lot of the things I like doing as an analyst—examining the market, talking to people looking at competitive aspects—I jumped at it.Corey: It's nice when you see those opportunities that show up in front of you, and the stars sort of align. It's like, this is not just something that I'm excited about and enthused about, but hey, they can use me. I can add something to where they're going and help them get there better, faster, sooner, et cetera, et cetera.Nick: When you're an analyst, you look at dozens of companies a month and I'd never seen an opportunity that looked like that. Everything kind of looked the same. There's a bunch of data integration companies, there's a bunch of companies with Spark and things like that, but this company was unique; the product was unique, and no one was really recognizing the opportunity. So, it was just a great set of things that all happen at the same time.Corey: It's always fun to see stars align like that. So—Nick: Yeah.Corey: —help me understand in a way that can be articulated to folks who don't have 15 years of grumpy sysadmin experience under their belts, what does Cribl do?Nick: So, Cribl does a couple of things. Our flagship product is called LogStream, and the easiest way to describe that is as an abstraction between sources and destinations of data. And that doesn't sound very interesting, but if you, from your sysadmin background, you're always dealing with events, logs, now there's traces, metrics are also hanging around—Corey: Oh, and of course, the time is never synchronized with anything either, so it's sort of a giant whodunit, mystery, where half the eyewitnesses lie.Nick: Well, there's that. There's a lot of data silos. If you got an agent deployed on a system, it's only going to talk to one destination platform. And you repeat this, maybe a dozen times per server, and you might have 100,000 or 200,000 servers, with all of these different agents running on it, each one locked into one destination. So, you might want to be able to mix and match that data; you can't. You're locked in.One of the things LogStream does is it lets you do that exact mixing and matching. Another thing that this product does, that LogStream does, is it gives you ability to manage that data. And then what I mean by that is, you may want to reduce how much stuff you're sending into a given platform because maybe that platform charges you by your daily ingest rates or some other kind of event-based charges. And so not all that data is valuable, so why pay to store it if it's not going to be valuable? Just dump it or reduce the amount of volume that you've got in that payload, like a Windows XML log.And so that's another aspect that it allows you to do, better management of that stuff. You can redact sensitive fields, you can enrich the data with maybe, say, GeoIPs so you know what kind of data privacy laws you fall under and so on. And so, the story has always been, land the data in your destination platform first, then do all those things. Well, of course, because that's how they charge you; they charge you based on daily ingest. And so now the story is, make those decisions upfront in one place without having to spread this logic all over, and then send the data where you want it to go.So, that's really, that's the core product today, LogStream. We call ourselves an observability pipeline for observability data. The other thing we've got going on is this project called AppScope, and I think this is pretty cool. AppScope is a black box instrumentation tool that basically resides between the application runtime and the kernel and any shared libraries. And so it provides—without you having to go back and instrument code—it instruments the application for you based on every call that it makes and then can send that data through something like LogStream or to another destination.So, you don't have to go back and say, “Well, I'm going to try and find the source code for this 30-year old c++ application.” I can simply run AppScope against the process, and find out exactly what that application is doing for me, and then relay that information to some other destination.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: I have to ask because I love what you're doing, don't get me wrong. The counterargument that always comes up in this type of conversation is, “Who in their right mind looks at the state of the industry today and says, ‘You know what we need? That's right; another observability tool.'” what differentiates what you folks are building from a lot of the existing names in the space? And to be clear, a lot of the existing names in the space are treating observability simply as hipster monitoring. I'm not entirely sure they're wrong, but that's a different fight for a different time.Nick: Yeah. I'm happy to come back and talk about that aspect of it, too. What's different about what we're doing is we don't care where the data goes. We don't have a dog in that fight. We want you to have better control over where it goes and what kind of shape it's in when it gets there.And so I'll give an example. One of our customers wanted to deploy a new SIEM—Security Information Event Management—tool. But they didn't want to have to deploy a couple hundred-thousand new agents to go along with it. They already had the data coming in from another agent, they just couldn't get the data to it. So, they use LogStream to send that data to their new desired platform.Worked great. They were able to go from zero to a brand new platform in just a couple days, versus fighting with rolling out agents and having to update them. Did they conflict with existing agents? How much performance did it impact on the servers, and so on? So, we don't care about the destination. We like everybody. We're agnostic when it comes to where that data goes. And—Corey: Oh, it's not about the destination. It's about the journey. Everyone's been saying it, but you've turned it into a product.Nick: It's very spiritual. So, we [laugh] send, we send your observability data on a spiritual [laugh] journey to its destination, and we can do quite a bit with it on the way.Corey: So, you said you offered to go back as well and visit the, “Oh, it's monitoring, but we're going to call it observability because otherwise we get yelled out on Twitter by Charity Majors.” How do you view that?Nick: Monitoring is the things you already know. Right? You know what questions you want to ask, you get an alert if something goes out of bounds or something goes from green to red. Think about monitoring as a data warehouse. You shape your data, you get it all in just the right condition so you can ask the same question over and over again, over different time domains.That's how I think about monitoring. It's prepackaged, you know exactly what you want to do with it. Observability is more like a data lake. I have no idea what I'm going to do with this stuff. I think there's going to be some signals in here that I can use, and I'm going to go explore that data.So, if monitoring is your known knowns, observability is your unknown unknowns. So, an ideal observability solution gives you an opportunity to discover what those are. Once you discover them. Great. Now, you can talk about how to get them into your monitoring system. So, for me, it's kind of a process of discovery.Corey: Which makes an awful lot of sense. The problem I've always had with the monitoring approach is it falls into this terrible pattern of enumerate the badness. In other words, “Imagine all the ways that this system can fail,” and then build an alerting that lets you know when any of those things happen. And what happens next is inevitable to anyone who's ever dealt with the tricksy devils known as computers, and what happens, of course, is that they find new ways to fail and you generally get to add to the list of things to check for, usually at two o'clock in the morning.Nick: On a Sunday.Corey: Oh, absolutely. It almost doesn't matter when. The real problem is when these things happen, it's, “What day, actually, is it?” And you have to check the calendar to figure out because your third time that week being woken up in the dead of night. It's like an infant but less than endearing.So, that has been the old school approach, and there's unfortunately still an awful lot of, we'll just call it nonsense, in the industry that still does exactly the same thing, except now they call it observability because—hearkening back to earlier in our conversation—there's a certain point in the Gartner Hype Cycle that we are all existing within. What's the deal with that?Nick: Well, I think that there are a lot of entrenched interests in the monitoring space. And so I think you always see this when a new term comes around. Vendors will say, “All right, well, there's a lot of confusion about this. Let me back-fit my product into this term so that I can continue to look like I'm on the leading edge and I'm not going to put any of my revenues in jeopardy.” I know, that's a cynical view, but I've seen it over and over again.And I think that's unfortunate because there's a real opportunity to have a better understanding of your systems, to better understand what's happening in all the containers you're deploying and not tearing down the way that you should, to better understand what's happening in distributed systems. And it's going to be a real missed opportunity if that is what happens. If we just call this ‘Monitoring 2.0' it's going to leave a lot of unrealized potential in the market.Corey: The big problem that I've seen in a lot of different areas is—I'll be direct—consolidation where you have a company that starts to do a thing—and that's great—and then they start doing other things that are tied to it. And in turn, they start, I guess, gathering everything in the ecosystem. If you break down observability into various constituent parts, I—know, I know, the pillars thing is going to upset people; ignore that for now—and if you have an offering that's weak in a particular area, okay, instead of building it organically into the product, or saying, “Yeah, that's not what we do,” there's an instinct to acquire a company or build that functionality out. And it turns out that we're building what feels the lot to me like the SaaS equivalent of multifunction printers: they can print, they can scan, they can fax, and none of those three very well, so it winds up with something that dissatisfies everyone, rather than a best-of-breed solution that has a very clear and narrow starting and stopping point. How do you view that?Nick: Well, what you've described is a compromise, right? A compromise is everyone can work and no one's happy. And I think that's the advantage of where LogStream comes in. The reality is best-of-breed. Most enterprises today have 30 or more different monitoring tools—call them observability tools if you want to—and you will never pry those tools from the dead hands of those sysadmins, DevOps engineers, SREs, et cetera.They all integrate those tools into how they work and their processes. So, we're living in a best-of-breed world. It's like that in data and analytics—my former beat—and it's like that in monitoring and observability. People really gravitate towards the tools they like, they gravitate towards the tools their friends are using. And so you need a way to be able to mix and match that stuff.And just because I want to stay [laugh] on message, that's really where the LogStream story kind of blends in because we do that; we allow you to mix and match all those different pieces.Corey: Joke's on you. I use Nagios and I have no friends. I'm not convinced those two things are entirely unrelated, but here we are. So here's, I guess, the big burning question that a lot of folks—certainly not me, but other undefined folks, ‘lots of people are saying'—so you built something interesting that actually works. I want to be clear on this.I have spoken to customers of yours. They swear by it instead of swearing at it, which happens with other companies. Awesome. You have traction, you're moving forward, things are going great. Here's $200 million is the next part of that story, and on some level, my immediate reaction—which does need updating, let's be clear here—is like, all right.I'm trying to build a product. I can see how I could spend a few million bucks. “Well, what can you do with I don't know, 100 times that?” My easy answer is, “Something monstrous.” I don't believe that is the case here. What is the growth plan? What are you doing that makes having that kind of a war chest a useful and valuable thing to have?Nick: Well, if you speak with the co-founders—and they've been open about this—we view ourselves as a generational company. We're not just building one product. We've been thinking about, how do we deliver on observability as this idea of discovery? What does that take? And it doesn't mean that we're going to be less agnostic to other destinations, we still think there's an incredible amount of value there and that's not going away, but we think there's maybe an interim step that we build out, potentially this idea of an observability data lake where you can explore these environments.Certainly, there's other types of options in the space today. Most of them are SQL-based, which is interesting because the audience that uses monitoring and observability tools couldn't care less about SQL right? They want search, they want regex, and so you've got to have the right tool for that audience. And so we're thinking about what that looks like going forward. We're doubling down on people.Surprisingly, this is a very—like anything else in software, it is people-intensive. And so certainly those are other aspects that we're exploring with the recent investment, but definitely, multiproduct company is our future and continued expansion.Corey: Expansion is always a fun one. It's the idea of, great, are you looking at going deeper into the areas you're already active within, or is it more of a, “Ah, so we've solved the, effectively, log routing problem. That's great. Let's solve other problems, too.” Or is it more of a, I guess, a doubling down and focusing on what's working? And again, that probably sounds judgmental in a way I don't intend it to at all. I just have a hard time contextualizing that level of scale coming from a small company perspective the way that I do.Nick: Yeah. Our plan is to focus more intently on the areas that we're in. We have a huge basis of experience there. We don't want to be all things to all people; that dilutes the message down to nothing, so we want to be very specific in the audiences we talk to, the problems we're trying to solve, and how we try to solve them.Corey: The problem I've always found with a lot of the acquisition, growth thrashing of—let me call it what I think it is: companies in decline trying to strain relevancy, it feels almost like a, “We don't see a growth strategy. So, we're going to try and acquire everything that hold still long enough, at some level, trying to add more revenue to the pile, but also thrashing in the sense of, okay. They're going to teach us how to do things in creative, awesome ways,” but it never works out that way. When you have a 50,000 person company acquiring a 200 person company, invariably the bigger culture is going to dominate. And I don't understand why that mistake seems to continually happen again, and again, and again.And people think I'm effectively alluding to—or whenever the spoken word version of subtweeting is—a particular company or a particular acquisition. I'm absolutely not, there are probably 50 different companies listening right now who thinks, “Oh, God. He's talking about us.” It's the common repeating trend. What is that?Nick: It's hard to say. In some cases, these acquisitions might just be talent. “We need to know how to do X. They know how to do X. Let's do it.” They may have very unique niche technology or software that another company thinks they can more broadly apply.Also, some of these big companies, these may not be board-level or CEO-level decisions. A business unit might decide, “Oh, I like what that company is doing. I'm going to go acquire it.” And so it looks like MegaCorp bought TinyCorp, but it's really, this tiny business unit within MegaCorp bought tiny company. The reality is often different from what it looks like on the outside.So, that's one way. Another is, you know, if they're going to teach us to be more effective with tech or something like that, you're never going to beat culture. You're never going to be the existing culture. If it's 50,000, against 200, obviously we know who wins there. And so I don't know if that's realistic.I don't know if the big companies are genuine when they say that, but it could just be the messaging that they use to make people happy and hopefully retain as many of those new employees for as long as they can. Does that make sense?Corey: No, it makes perfect sense. It's the right answer. It does articulate what is happening there, and I think I keep falling prey to the same failure. And it's hard. It's pernicious, but companies are not monolithic entities.There's no one person at all of these companies each who is making these giant unilateral decisions. It's always some product manager or some particular person who has a vision and a strategy in the department. It is not something that the company board is agreeing on every little decision that gets made. They're distributed entities in many respects.Nick: Absolutely. And that's only getting more pervasive as companies get larger [laugh] through acquisition. So, you're going to see more and more of that, and so it's going to look like we're going to put one label on it, one brand. Often, I think internally, that's the exact opposite of what actually happened, how that decision got made.Corey: Nick, I want to thank you for taking so much time to speak with me about what you're up to over there, how your path has shaped, how you view the world, and also what Cribl does these days. If people want to learn more about what you're up to, how you think about the world, or even possibly going to work at Cribl which, having spoken to a number of people over there, I would endorse it. How do they find you?Nick: Best place to find us is by joining our community: cribl.io/community, and Cribl is spelled C-R-I-B-L. You can certainly reach out there, we've got about 2300 people in our community Slack, so it's a great group. You can also reach out to me on Twitter, I'm @nheudecker, N-H-E-U-D-E-C-K-E-R. Tell me what you thought of the episode; love to hear it. And then beyond that, you can also sign up for our free cloud tier at cribl.cloud. It's a pretty generous one terabyte a day processing, so you can start to send data in and send it wherever you'd like to be.Corey: To be clear, this free as in beer, not free as an AWS free tier?Nick: This is free as in beer.Corey: Excellent. Excellent.Nick: I think I'm getting that right. I think it's free as in beer. And the other thing you can try is our hosted solution on AWS, fully managed cloud at cribl.cloud, we offer a free one terabyte per day processing, so you can start to send data into that environment and send it wherever you'd like to go, in whatever shape that data needs to be in when it gets there.Corey: And we will, of course, put links to that in the [show notes 00:35:21]. Thank you so much for your time today. I really appreciate it.Nick: No, thank you for having me. This was a lot of fun.Corey: Nick Heudecker, senior director, market strategy and competitive intelligence at Cribl. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment explaining that the only real reason a startup should raise a $200 million funding round is to pay that month's AWS bill.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About AbbyWith over twenty years in the tech world, Abby Kearns is a true veteran of the technology industry. Her lengthy career has spanned product marketing, product management and consulting across Fortune 500 companies and startups alike. At Puppet, she leads the vision and direction of the current and future enterprise product portfolio. Prior to joining Puppet, Abby was the CEO of the Cloud Foundry Foundation where she focused on driving the vision for the Foundation as well as  growing the open source project and ecosystem. Her background also includes product management at companies such as Pivotal and Verizon, as well as infrastructure operations spanning companies such as Totality, EDS, and Sabre.Links: Cloud Foundry Foundation: https://www.cloudfoundry.org Puppet: https://puppet.com Twitter: https://twitter.com/ab415 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time, I was deep into the weeds of configuration management, which explains a lot, such as why it seems I don't know happiness in any meaningful sense. Then I wound up progressing into other areas of exploration, like the cloud, and now we know for a fact why happiness isn't a thing for me. My guest today is the former CEO of the Cloud Foundry Foundation and today is the CTO over at a company called Puppet, which we've talked about here from time to time. Abby Kearns, thank you for joining me. I appreciate your taking the time out of your day to suffer my slings and arrows.Abby: Thank you for having me. I have been looking forward to this for weeks.Corey: My stars, it seems like things are slow over there, and I kind of envy you for that. So, help me understand something; you went from this world of cloud-native everything, which is the joy of working with Cloud Foundry, to now working with configuration management. How is that not effectively Benjamin Button-ing your career. It feels like the opposite direction that most quote-unquote, “Digital transformations” like to play with. But I have a sneaking suspicion, there's more to it than I might guess from just looking at the label on the tin.Abby: Beyond I just love enterprise infrastructure? I mean, come on, who doesn't?Corey: Oh, yeah. Everyone loves to talk about digital transformation, reading about books like a Head in the Cloud to my children used to be a fun nightly activity before it was formally classified as child abuse. So yeah, I hear you, but it turns out the rest of the world doesn't necessarily agree with us.Abby: I do not understand it. I have been in enterprise infrastructure my entire career, which has been a really, really long time, back when Unix and Sun machines were still a thing. And I'll be a little biased here; I think that enterprise infrastructure is actually the most fascinating part of technology right now. And why is that? Well, we're in the process of actively rewritten everything that got us here.And we talk about infrastructure and everyone's like, “Yeah, sure, whatever,” but at the end of the day, it's the foundation that everything that you think is cool about technology is built on. And for those of us that really enjoy this space, having a front-row seat at that evolution and the innovation that's happening is really, really exciting and it creates a lot of interesting conversation, debate, evolution of technologies, and innovation. And are they all going to be on the money five, ten years from now? Maybe not, but they're creating an interesting space and discussion and just the work ahead for all of us across the board. And I'm kind of bucketing this pretty broadly, intentionally so because I think at the end of the day, all of us play a role in a bigger piece of pie, and it's so interesting to see how these things start to fit together.Corey: One of the things that I've noticed is that the things that get attention on the keynote stage of, “This is this far future, serverless, machine-learning Kubernetes, dingus nonsense,” great is—Abby: You forgot blockchain. [laugh].Corey: Oh, yeah. Oh, yeah blockchain as well. Like, what other things can we wind up putting into the buzzword thing to wind up guaranteeing that your seed round is at least $200 million? Great. There's that.But when you look at the actual AWS bill—my specialty, of course—and seeing where the money is actually going, it doesn't really look that different, as far as percentages go—even though the numbers are higher—than it did ten years ago, at least in the enterprise world. You're still buying a bunch of EC2 instances, you're still potentially modernizing to some of the managed services like RDS—which is Amazon's reimagining of what a database could be if you still had to manage the finicky bits, but had no control over when and how they worked—and of course, data transfer and disk. These are the basic building blocks of everything in cloud. And despite how much we talk about the super neat stuff, what we're doing is not reflected on the conference stage. So, I tend to view the idea of aspirational architecture as its own little world.There are still seasoned companies out there that are migrating from where they are today into this idea of, well, virtualization, we've just finally got our heads around that. Now, let's talk about this cloud thing; seems like a fad—in 2021. And people take longer to get to where they think they're going or where they intend to go than they plan for, and they get stuck somewhere and instead of a cloud migration, they're now hybrid because they can redefine things and declare victory when they plant that flag, and here we are. I'm not here to make fun of these companies because they're doing important work and these are super hard problems. But increasingly, it seems that the technology is not the thing that's holding them back or even responsible for their outcome so much as it is people.The more I work with tech, the more I realized that everything that's hard becomes people issues. Curious to get your take on that, given your somewhat privileged perspective as having a foot standing very deeply in each world.Abby: Yeah, and that's a super great point. And I also realized I didn't fully answer the first question either. So, I'll tie those two things together.Corey: That's okay, we're going to keep circling around until you get there. It's fine.Abby: It's been a long week, and it's only Wednesday.Corey: All day long, as it turns out.Abby: I have a whole soapbox that I drag around behind me about people and process, and how that's your biggest problem, not technology, and if you don't solve for the people in the process, I don't care what technology you choose to use, isn't going to fix your problem. On the other hand, if you get your people and process right, you can borderline use crayons and paper and get [laugh] really close to what you need to solve for.Corey: I have it on good authority that's known as IBM Cloud. Please continue.Abby: [laugh]. And so I think people and process are at the heart of everything. They're our biggest accelerators with technology and they're our biggest limitation. And you can cloud-native serverless your way into it, but if you do not actually do continuous delivery, if you did not actually automate your responses, if you do not actually set up the cross-functional teams—or sometimes fondly referred to as two-pizza teams—if you don't have those things set up, there isn't any technology that's going to make you deliver software better, faster, cheaper. And so I think I care a lot about the focus on that because I do think it is so important, but it's also—the reason a lot of people don't like to talk about it and deal with it because it's also the hardest.People, culture change, digital transformation, whatever you want to call it, is hard work. There's a reason so many books are written around DevOps. And you mentioned Gene Kim earlier, there's a reason he wrote The Phoenix Project; it's the people-process part is the hardest. And I do think technology should be an enabler and an accelerator, but it really has to pair up nicely with the people part. And you asked your earlier question about my move to Puppet.One of the things that I've learned a lot in running the Cloud Foundry Foundation, running an open-source software foundation, is you could a real good crash course in how teams can collaborate effectively, how teams work together, how decisions get made, the need for that process and that practice. And there was a lot of great context because I had access to so much interesting information. I got to see what all of these large enterprises were doing across the board. And I got to have a literal seat at the table for how a lot of the decisions are getting made around not only the open-source technologies that are going into building the future of our enterprise infrastructure but how a lot of these companies are using and leveraging those technologies. And having that visibility was amazing and transformational for myself.It gave me so much richness and context, which is why I have firmly believed that the people and process part were so crucial for many years. And I decided to go to a company that sold products. [laugh]. You're like, “What? What is she talking about now? Where is this going?”And I say that because running an open-source software foundation is great and it gives you so much information and so much context, but you have no access to customers and no access to products. You have no influence over that. And so when I thought about what I wanted to do next, it's like, I really want to be close to customers, I really want to be close to product, and I really want to be part of something that's solving what I look at over the next five to ten years, our biggest problem area, which is that tweener phase that we're going to be in for many years, which we were just talking about, which is, “I have some stuff on-prem and I have some stuff in a cloud—usually more than one cloud—and I got to figure out how to manage all of that.” And that is a really, really, really hard problem. And so when I looked at what Puppet was trying to do, and the opportunity that existed with a lot of the fantastic work that Puppet has done over the last 12 years around Desired State Configuration management, I'm like, “Okay, there's something here.”Because clearly, that problem doesn't go away because I'm running some stuff in the cloud. So, how do we start to think about this more broadly and expansively across the hybrid estate that is all of these different environments? And who is the most well-positioned to actually drive an innovative product that addresses that? So, that's my long way of addressing both of those things.Corey: No, it's a fair question. Friend of the show, Matt Stratton, is famous for saying that, “You cannot buy DevOps, but I sure would like to sell it to you,” and if you're looking at it from that perspective, Puppet is not far from what that product store look like in some ways. My first encounter with Puppet was back around 2009, 2010 or so, and I was using it in an environment I was working within and thought, “Okay, this is terrible, and it's crap, and obviously, I know what I'm doing far better than this, and the problem is the Puppet's a bad product.” So, I was one of the early developers behind SaltStack, which was a terrific, great way of approaching the problem from a novel perspective, and it wasn't crap; it was awesome. Right up until I saw the first time a customer deployed it and looked at their environment, and it wasn't crap, it was worse because it turns out that you can build a super finely crafted precision instrument that makes a fairly bad hammer, but that's how customers are going to use it anyway.Abby: Well, I mean, [sigh] look, you actually hit something that I think we don't actually talk about, which is how hard all of this shit really is. Automation is hard. Automation for distributed systems at scale is super duper hard. There isn't an easy way to solve that problem. And I feel like I learned a lot working with Cloud Foundry.Cloud Foundry is a Platform as a Service and it sits a layer up, but it had the same challenges in that solving the ability to run cloud-native applications and cloud-native workloads at scale and have that ephemerality to it and that resilience to it, and the things everyone wants but don't recognize how difficult it is, actually, to do that well. And I think the same—you know, that really set me up for the way that I think about the problem, even the layer down which is, running and managing desired state, which at the end of the day is a really fancy way of saying, “Does your environment look like the way you think it should? And if it doesn't, what are you going to do about it?” And it seems like, in this year of—what year are we again? 2021, maybe? I don't know. It feels like the last two years of, sort of, munged together?Corey: Yeah, the passing of time is something it's very hard for me to wrap my head around.Abby: But it feels like, I know some people, particularly those of us that have been in tech a long time are probably like, “Why are we still talking about that? Why is that a thing?” But that is still an incredibly hard problem for most organizations, large and small. So, I tend to spend a lot of time thinking about large enterprises, but in the day, you've got more than 20 servers, you're probably sitting around thinking, “Does my environment actually look the way I think it does? There's a new CVE that just came out. Am I able to address that?”And I think at the end of the day, figuring out how you can solve for that on-prem has been one of the things that Puppet has worked for, and done really, really well the last 12 years. Now, I think the next challenge is okay, how do you extend that out across your now bananas complex estate that is—I got a huge data estate, maybe one or two data centers, I got some stuff in AWS, I got some stuff in GCP, oh yeah, got a little thing over here and Azure, and oh, some guy spun up something on OCI. So, we got a little bit of everything. And oh, my God, the SolarWinds breach happened. Are we impacted? I don't know. What does that mean? [laugh].And I think you start to unravel the little pieces of that and it gets more and more complex. And so I think the problems that I was solving in the early aughts with servers seems trite now because you're like, I can see all of my servers; there's eight of them. Things seem fine. To now, you've got hundreds of thousands of applications and workloads, and some of them are serverless, and they're all over the place. And who has what, and where does it sit?And does it look like the way that I think it needs to so that I can run my business effectively? And I think that's really the power of it, but it's also one of those things that I don't feel like a lot of people like to acknowledge the complexity and the hardness of that because it's not just the technology problem—going back to your other question, how do we work? How do we communicate? What are our processes around dealing with this? And I think there's so much wrapped up in that it becomes almost like, how do you eat an elephant story, right? Yes, one bite at a time, but when you first look at the elephant, you're like, “Holy shit. This is big. What do I need to do?” And that I think is not something we all collectively spend enough time talking about is how hard this stuff is.Corey: One of the biggest challenges I see across the board is this idea of conference-ware style architecture; the greatest lie you ever see is someone talking about their infrastructure in public because peel it back a little bit and everything's messy, everything's disastrous, and everything's a tire fire. And we have this cult in tech—Abby: [laugh].Corey: —it's almost a cult where we have this idea that anything that isn't rewritten completely within the last six months based upon whatever is the hot framework now that is designed to run only in Google Chrome running on the latest generation MacBook Pro on a gigabit internet connection is somehow less than. It's like, “So, what does that piece of crap do?” And the answer is, “Well, a few $100 million a quarter in revenue, so how about you watch your mouth?” Moving those things is delicate; moving those things is fraught, and there are a lot of different stakeholders to the point where one of the lessons I keep learning is, people love to ask me, “What is Amazon's opinion of you?” Turns out that there's no Ted Amazon who works over there who forms a single entity's opinion. It's a bunch of small teams. Some of them like me, some of them can't stand me, far and away the majority don't know who I am. And that is okay. In theory; in practice, I find it completely unforgivable because how dare you? But I understand it's—Abby: You write a memo, right now. [laugh].Corey: Exactly. Companies are people and people are messy, and for better or worse, it is impossible to patch them. So, you have to almost route around them. And that was something that I found that Puppet did very well, coming from the olden days of sysadmin work where we spend time doing management [bump 00:15:53] the systems by hand. Like, oh, I'm going to do a for loop. Once I learned how to script. Before that, I use Cluster SSH and inadvertently blew away a University's entire config file what starts up on boot across their entire FreeBSD server fleet.Abby: You only did it once, so it's fine.Corey: Oh, yeah. I'm never going to screw up again. Well, not like that. In other ways. Absolutely, but at least my errors will be novel.Abby: Yeah. It's learning. We all learn. If you haven't taken something down in production in real-time, you have not lived. And also you [laugh] haven't done tech. [laugh].Corey: Oh, yeah, you either haven't been allowed close enough to anything that's important enough to be able to take down, you're lying to me, or thirdly—and this is possible, too—you're not yet at a point in your career where you're allowed to have access to the breaky parts. And that's fine. I mean, my argument has always been about why I'd be a terrible employee at Google, for example, is if I went in maliciously on day one, I would be hard-pressed to take down google.com for one hour. If I can't have that much impact intentionally going in as a bad actor, it feels like there'd be how much possible upside, positive impact can I have what everyone's ostensibly aligned around the same thing?It's the challenge of big companies. It's gaining buy-in, it's gaining investment in the idea and the direction you're going in. Things always take longer, you have to wind up getting multiple stakeholders on board. My consulting practice is entirely around helping save money on the AWS bill. You'd think it would be the easiest thing in the world to sell, but talking to big companies means a series of different sales conversations with different folks, getting them all on the same page. What we do functionally isn't so much look at the computer parts as it is marriage counseling between engineering and finance. Different languages, different ways of thinking about things, ostensibly the same goals.Abby: I mean, I don't think that's a big company problem. I think that's an every company problem if you have more than, like, five people in your company.Corey: The first few years here, it was just me and I had none of those problems. I had very different problems, but you know—and then we started bringing other people in, it's like, “Oh, yeah, things were great until we hired people. Ugh, mistake. Never do that.” And yeah, it turns out that's not particularly sustainable.Abby: Stakeholder management is hard. And you mentioned something about routing around. Well, you can't actually route around people, unfortunately. You have to get people to buy in, you have to bring people along on the journey. And not everybody is at the same place in the way they think about the work you're doing.And that's true at any company, big or small. I think it just gets harder and more complex as the company gets bigger because it's harder to make the changes you need to make fast enough, but I'd say even at a company the size of Puppet, we have the exact same challenges. You know, are the teams aligned? Are we aligned on the right things? Are we focusing on the right things?Or, do we have the right priorities in our backlog? How are we doing the work that we do? And if you're trying to drive innovation, how fast are we innovating? Are we innovating fast enough? How tight are our feedback loops?It's one of those things where the conversations that you and I have had externally with customers are the same conversations I have internally all the time, too. Let's talk about innovators' dilemma. [laugh]. Let's talk about feedback loop. Let's talk about what does it mean to get tighter feedback loops from customers and the field?And how do you align those things to the priorities in your backlog? And it's one of those never-ending challenges that's messy and complicated. And technology can enable it, but the technology is also messy and hard. And I do love going to conferences and seeing how pretty and easy things could look, and it's definitely a great aspiration for us to all shoot for, but at the end of the day, I think we all have to recognize there's a ton of messiness that goes on behind to make that a reality and to make that really a product and a technology that we can sell and get behind, but also one that we buy in, too, and are able to use. So, I think we as a technology industry, and particularly those of us in the Bay Area, we do a disservice by talking about how easy things are and why—you know, I remember a conversation I had in 2014 where someone asked me if Docker was already passe because everybody was doing containerized applications, and I was like, “Are they? Really? Is that an everyone thing? Or is that just an ‘us' thing?” [laugh].Corey: Well, they talk about it on the conference stages an awful lot, but yeah. New problems that continue to arise. I mean, I look back at my early formative years as someone who could theoretically be brought out in public and it was through a consulting project, where I was a traveling trainer for Puppet back in 2014, 2015, and teaching people who hadn't had exposure before what Puppet was about. And there was a definite experience in some of the people attending class where they were very opposed to the idea. And dig down a little bit, it's not that they had a problem with the software, it's not that they had a problem with any of the technical bits.It's that they made the mistake that so many technologists made—I know I have, repeatedly—of identifying themselves with the technology that they work on. And well, in some cases, yeah, the answer was that they ran a particular script a bunch of times and if you can automate that through something like Puppet or something else, well, what does that mean for them? We see it much larger-scale now with people who are, okay, I'm in the data center working on the storage arrays. When that becomes just an API call or—let's be serious, despite what we see in conference stages—when it becomes clicking buttons in the AWS console, then what does that mean for the future of their career? The tide is rising.And I can't blame them too much for this; you've been doing this for 25 years, you don't necessarily want to throw all that away and start over with a whole new set of concepts and the rest because unlike what Twitter believes, there are a bunch of legitimate paths in this industry that do treat it as a job rather than an all-consuming passion. And I have no negative judgment toward folks who walk down that direction.Abby: Most people do. And I think we have to be realistic. It's not just some. A lot of people do. A lot of people, “This is my nine-to-five job, Monday through Friday, and I'm going to go home and I'm going to spend time with my family.”Or I'm going to dare I say—quietly—have a life outside of technology. You know, but this is my job. And I think we have done a disservice to a lot of those individuals who for better or for worse, they just want to go in and do a job. They want to get their job done to the best of their abilities, and don't necessarily have the time—or if you're a single parent, have the flexibility in your day to go home and spend another five, six hours learning the latest technology, the latest programming language, set up your own demo environment at home, play around with AWS, all of these things that you may not have the opportunity to do. And I think we as an industry have done a disservice to both those individuals, as well in putting up really imaginary gates on who can actually be a technologist, too.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: Gatekeeping, on some level, is just—it's a horrible thing. Something I found relatively early on is that I didn't enjoy communities where that was a thing in a big way. In minor ways, sure, absolutely. I wound up gravitating toward Ubuntu rather than Debian because it turned out that being actively insulted when I asked how to do something wasn't exactly the most welcoming, constructive experience, where they, “Read the manual.” “Yeah, I did that and it was incomplete and contradictory, and that's why I'm here asking you that question, but please continue to be a condescending jackwagon. I appreciate that. It really just reminds me that I'm making good choices with my life.”Abby: Hashtag-RTFM. [laugh].Corey: Exactly. In my case, fine, its water off a duck's back. I can certainly take it given the way that I dish it out, but by the same token, not everyone has a quote-unquote, thick skin, and I further posit that not everyone should have to have one. You should not get used to personal attacks as a prerequisite for working in this space. And I'm very sensitive to the idea that people who are just now exploring the cloud somehow feel that they've missed out on their career, and that so there's somehow not appropriate for this field, or that it's not for them.And no, are you kidding me? You know that overwhelming sense of confusion you get when you look at the AWS console and try and understand what all those services do? Yeah, I had the same impression the first time I saw it and there were 12 services; there's over 200 now. Guess what? I've still got it.And if I am overwhelmed by it, I promise there's no shame in anyone else being overwhelmed by it, too. We're long since past the point where I can talk incredibly convincingly about AWS services that don't exist to AWS employees and not get called out on it because who in the world has that entire Rolodex of services shoved into their heads who isn't me?Abby: I'd say you should put out… a call for anyone that does because I certainly do not memorize the services that are available. I don't know that anyone does. And I think even more broadly, is, remember when the landscape diagram came out from the CNCF a couple of years ago, which it's now, like… it's like a NASCAR logo of every logo known to man—Corey: Oh today, there's over 400 icons on it the last time I saw—I saw that thing come out and I realized, “Wow, I thought I was going to shit-posting,” but no, this thing is incredible. It's, “This is great.” My personal favorite was zooming all the way in finding a couple of logos on in the same box three times, which is just… spot on. I was told later, it's like, “Oh, those represent different projects.” I'm like, “Oh, yeah, must have missed that in the legend somewhere.” [laugh]. It's this monstrous, overdone thing.Abby: But the whole point of it was just, if I am running an IT department, and I'm like, “Here you go. Here's a menu of things to choose,” you're just like, “What do I do with this information? Do I choose one of each? All the above? Where do I go? And then, frankly, how do I make them all work together in my environment?” Because they all serve very different problems and they're tackling different aspects of that problem.And I think I get really annoyed with myself as an industry—like, ourselves as an industry because it's like, “What are we doing here?” We're trying to make it harder for people, not only to use the technology, to be part of it. And I think any efforts we can make to make it easier and more simple or clear, we owe it to ourselves to be able to tell that story. Which now the flip side of that is describing cloud-native in the cloud, and infrastructure and automation is really, really hard to do [laugh] in a way that doesn't use any of those words. And I'm just as guilty of this, of describing things we do and using the same language, and all of a sudden you're looking at it this says the same thing is 7500 other websites. [laugh]. So.Corey: Yep. I joke at RSA's Expo Hall is basically about twelve companies selling different things. Sure, each one has a whole bunch of booths with different logos and different marketing copy, but it's the same fundamental product. Same challenge here. And this is, to me, the future of cloud, this is where it's going, where I want something that will—in my case, I built a custom URL shortener out of DynamoDB, API Gateway, Lambda, et cetera, and I built this thing largely as a proof of concept because I wanted to have experience playing with these tools.And that was great, not but if I'm doing something like that in production, I'm going with Bitly or one of the other services that provide this where someone is going to maintain it full time. Unless it is the core of what I'm doing, I don't want to build it myself from popsicle sticks. And moving up the stack to a world of folks who are trying to solve a business problem and they don't want to deal with the ten prerequisite services to understand the cloud, and then a whole bunch of other things tied together, and the billing, and the flow becomes incredibly problematic to understand—not to mention insecure: because we don't understand it, you don't know what your risk exposure is—people don't want that. They—Abby: Or to manage it.Corey: Yeah.Abby: Just the day-to-day management. Care and feeding, beyond security. [laugh].Corey: People's time is free. So, yeah. For example, do I write my own payroll system? Absolutely not. I have the good sense to pay a turnkey company to handle that for me because mistakes will show.I started my career running email systems. I pay for Google workspaces—or GSuite, or Gmail, or whatever the hell they're calling it this week—because it's not core and central to my business. I want a thing that winds up solving a business problem, and I will pay commensurately to the value that thing delivers, not the individual constituent costs of the components that build it together. Because until you're significantly scaled out and it is the core of what you do, you're spending more on people to run the monstrous thing than you are for the thing itself. That's always the way it works.So, put your innovation where it matters for your business. I posit the for an awful lot of the things we're building, in order to achieve those outcomes, this isn't it.Abby: Agreed. And I am a big believer in if I can use off-the-shelf software, I will because I don't believe in reinventing everything. Now, having said that, and coming off my soapbox for just a hot minute, I will say that a lot of what's happening, and going back to where I started around the enterprise infrastructure, we're reinventing so many things that there is a lot of new things coming up. We've talked about containers, we've talked about Kubernetes, around container scheduling, container orchestration, we haven't even mentioned service mesh, and sidecars, and all of the new ways we're approaching solving some of these older problems. So, there is the need for a broad proliferation of technology until the contraction phase, where it all starts to fundamentally clicks together.And that's really where the interesting parts happen, but it's also where the confusion happens because, “Okay, what do I use? How do I use it? How do these pieces fit together? What happens when this changes? What does this mean?”And by the way, if I'm an enterprise company, I'm a payroll company, what's the one thing I care about? My payroll software. [laugh]. And that's the problem I'm solving for. So, I take a little umbrage sometimes with the frame that every company is a software company because every company is not a software company.Every company can use technology in ways to further their business and more and more frequently, that is delivering their business value through software, but if I'm a payroll company, I care about delivering that payroll capabilities to my customer, and I want to do it as quickly as possible, and I want to leverage technology to help me do that. But my endgame is not that technology; my endgame is delivering value to my customers in real and meaningful ways. And I worry, sometimes, that those two things get conflated together. And one is an enabler of the other; the technology is not the outcome.Corey: And that is borderline heresy for an awful lot of folks out there in the space, I wish that people would wake up a little bit more and realize that you have to build a thing that solves customer pain, ideally, an expensive customer pain, and then they will basically rush to hurl money at you. Now, there are challenges and inflections as you go, and there's a whole bunch of nuances that can span entire fields of endeavor that I am hand-waving over here, and that's fine, but this is the direction I think we're going and this is the dawning awareness that I hope and trust we'll see start to take root in this industry.Abby: I mean, I hope so. I do take comfort in the fact that a lot of the industry leaders I'm starting to see, kind of, equate those two things more closely in the top [track 00:31:20]. Because it's a good forcing function for those of us that are technologists. At the end of the day, what am I doing? I am a product company, I am selling software to someone.So clearly, obviously, I have a vested interest in building the best software out there, but at the end of the day, for me, it's, “Okay, how do I make that truly impactful for customers, and how do I help them solve a problem?” And for me, I'm hyper-focused on automation because I honestly feel like that is the biggest challenge for most companies; it's the hardest thing to solve. It's like getting into your auto-driving car for the first time and letting go the steering wheel and praying to the software gods that that software is actually going to work. But it's the same thing with automation; it's like, “Okay, I have to trust that this is going to manage my environment and manage my infrastructure in a factual way and not put me on CNN because I just shut down entire customer environment,” or if I'm an airline and I've just had a really bad week because I've had technology problems. [laugh]. And so I think we have to really take into consideration that there are real customer problems on the other end of that we have to help solve for.Corey: My biggest problem is the failure mode of this is not when people watch the conference-ware presentations is that they're not going to sit there and think, “Oh, yeah, they're just talking about a nuanced thing that doesn't apply to our constraints, and they're hand-waving over a lot of stuff,” it's that, “Wow, we suck.” And that's not the takeaway anyone should ever have. Even Netflix doesn't operate the way that Netflix says that they do in their conference talks. It's always fun sitting next to someone from the company that's currently presenting and saying something to them, like, “Wow, I wish we did things that way.” And they said, “Yeah, I wish we did, too.”And it's always the case because it's very hard to get on stage and talk for 45 minutes about here's what we completely screwed up on, especially at the large publicly traded companies where it's, “Wait, why did our stock price just dive five perce—oh, my God, what did you say on stage?” People care [laugh] about those things, and I get it; there's a risk factor that I don't have to deal with here.Abby: I wish people would though. It would be so refreshing to hear someone like, “You know what? Ohh, we really messed this up, and let me walk you through what we did.” [laugh]. I think that would be nice.Corey: On some level, giving that talk in enough detail becomes indistinguishable from rage-quitting in public.Abby: [laugh].Corey: I mean, I'm there for it. Don't get me wrong. But I would love to see it.Abby: I don't think it has to be rage-quitting. One of the things that I talk to my team a lot about is the safety to fail. You can't take risk if you're too afraid to fail, right? And I think you can frame failure in a way of, “Hey, this didn't work, but let me walk you through all the amazing things we learned from this. And here's how we used that to take this and make this thing better.”And I think there's a positive way to frame it that's not rage-quitting, but I do think we as an industry gloss over those learnings that you absolutely have to do. You fail; everything does not work the first time perfectly. It is not brilliant out the gate. If you've done an MVP and it's perfect and every customer loves it, well then, you sat on that for way too long. [laugh]. And I think it's just really getting comfortable with this didn't work the first time or the fourth, but look, at time seven, this is where we got and this is what we've learned.Corey: I want to thank you for taking so much time out of your day to wind up speaking to me about things that in many cases are challenging to talk about because it's the things people don't talk about in the real world. If people want to learn more about what you're up to, who you are, et cetera, where can they find you?Abby: They can find me on the Twitters at @ab415. I think that's the best way to start, although I will say that I am not as prolific as you are on Twitter.Corey: That's a good thing.Abby: I'm a half-assed Tweeter. [laugh]. I will own it.Corey: Oh, I put my full ass into it every time, in every way.Abby: [laugh]. I do skim it a lot. I get a lot of my tech news from there. Like, “What are people mad about today?” And—Corey: The daily outrage. Oh, yeah.Abby: The daily outrage. “What's Corey ranting about today? Let's see.” [laugh].Corey: We will, of course, put a link to your Twitter profile in the [show notes 00:35:39]. Thank you so much for taking the time to speak with me. I appreciate it.Abby: Hey, it was my pleasure.Corey: Abby Kearns, CTO at Puppet. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me about the amazing podcast content you create, start to finish, at Netflix.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Charity Majors, CTO and co-founder at honeycomb, grew up in rural Idaho and dropped out of college. This is her unlikely journey from pianist to successful high-tech entrepreneur. She's a pioneer in the monitoring and observability space who  turned her learning at Facebook into a company focused on helping developers find and fix bugs faster. Charity's opinionated, thoughtful, and one of the most outspoken critics of, well, the status quo :).Listen and learn...What motivated Charity to start a career in tech having been a "perennial dropout"Why "ops has a well-deserved reputation for masochism"Why Charity says the "Kool-aid at Facebook is strong and potent" Why it's impossible to troubleshoot software bugs with high cardinality dataHow Charity defines observabilityWhat it means to practice observability-driven development (ODD) and why it should replace test-driven development (TDD)References in this episode:Charity's personal siteCharity on TwitterCharity's manifesto on observabilityThanks to Rachel Chalmers for making this episode happen!

Is observability mission critical at this point? The folks at New Relic think so. And they're not alone, according to their recent whitepaper on observability 90% of respondents believe observability is strategically important to their business. New Relic's CTO Buddy Brewer and SVP Greg Perotto join the podcast to discuss observability best practices, how to avoid data silos, the value of telemetry across the entire software life cycle and the five key insights of New Relic's observability report.New Relic's Observability Forecast: https://newrelic.com/resources/white-papers/2021-observability-forecastJoin our Discord Community ►► discord.gg/devinterruptedOur Website ►► devinterrupted.com/Want to try LinearB?  Book a LinearB Demo and use the "Dev Interrupted Podcast" discount code.Have 60 seconds? Review the show on Apple Podcasts

Para você que caiu aqui de pára-quedas, seja bem-vindo(a) à Saga de Observabilidade, a série que desvenda, a cada episódio, um player de observabilidade para você se aprofundar no assunto e conhecer a plataforma de diferentes vendors do mercado.Nesse episódio especial, que conta com a abertura do Toni, filho mais novo do João Brito, a conversa é com o Ruan Marins, engenheiro de vendas da DATADOG e pianista nas horas vagas! O Ruan define observabilidade como a “resolução automatizada de problemas em um nível inimaginável de escalável”. Indo além, o Kubicast perguntou para ele o quão preparada estaria a plataforma da empresa para usar o OpenTelemetry, se dá para administrar logs diretamente dela, se há uma inteligência por trás que subentende automaticamente cenários para análise preditiva, qual a feature mais amazing da ferramenta deles e quais seriam os principais argumentos para embarcar a liderança na observabilidade.CONTATO - Para falar com o Ruan escreva para:LinkedIn: /ruan-rodrigues-marins/Instagram: @ruanpianoAs RECOMENDAÇÕES do programa estão aqui:Ruan:Kubicast #63 - Kubernetes EssentialsInstagram @ruanpiano Ouvir ChopanJoão:Caminhos da Memória (Estava nos cinemas! Já tá na HBOMax)AVISO! Não perca a chance de subir ao palco do #KCDBrasil para compartilhar sua jornada cloud native com milhares de pessoas apaixonadas por Kubernetes. O CFP do evento se encerra em 29/10. Inscreva-se aqui.O Kubicast é uma produção da Getup, especialista em Kubernetes e apoiadora do projeto UnDistro, uma distribuição para gerenciar múltiplos clusters Kubernetes. #o11y #Observability #OpenTelemetry #DevOps #Kubernetes #Containers #docker  

About EwereCloud, DevOps Engineer, Blogger and AuthorLinks: Infrastructure Monitoring with Amazon CloudWatch: https://www.amazon.com/Infrastructure-Monitoring-Amazon-CloudWatch-infrastructure-ebook/dp/B08YS2PYKJ LinkedIn: https://www.linkedin.com/in/ewere/ Twitter: https://twitter.com/nimboya Medium: https://medium.com/@nimboya My Cloud Series: https://mycloudseries.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I periodically make observations that monitoring cloud resources has changed somewhat since I first got started in the world of monitoring. My experience goes back to the original Call of Duty. That's right: Nagios.When you set instances up, it would theoretically tell you when they were unreachable or certain thresholds didn't work. It was janky but it kind of worked, and that was sort of the best we have. The world has progressed as cloud has become more complicated, as technologies have become more sophisticated, and here today to talk about this is the first AWS Hero from Africa and author of a brand new book, Ewere Diagboya. Thank you for joining me.Ewere: Thanks for the opportunity.Corey: So, you recently published a book on CloudWatch. To my understanding, it is the first such book that goes in-depth with not just how to wind up using it, but how to contextualize it as well. How did it come to be, I guess is my first question?Ewere: Yes, thanks a lot, Corey. The name of the book is Infrastructure Monitoring with Amazon CloudWatch, and the book came to be from the concept of looking at the ecosystem of AWS cloud computing and we saw that a lot of the things around cloud—I mostly talked about—most of this is [unintelligible 00:01:49] compute part of AWS, which is EC2, the containers, and all that, you find books on all those topics. They are all proliferated all over the internet, you know, and videos and all that.But there is a core behind each of these services that no one actually talks about and amplifies, which is the monitoring part, which helps you to understand what is going on with the system. I mean, knowing what is going on with the system helps you to understand failures, helps you to predict issues, helps you to also envisage when a failure is going to happen so that you can remedy it and also [unintelligible 00:02:19], and in some cases, even give you a historical view of the system to help you understand how a system has behaved over a period of time.Corey: One of the articles that I put out that first really put me on AWS's radar, for better or worse, was something that I was commissioned to write for Linux Journal, back when that was a print publication. And I accidentally wound up getting the cover of it with my article, “CloudWatch is of the devil, but I must use it.” And it was a painful problem that people generally found resonated with them because no one felt they really understood CloudWatch; it was incredibly expensive; it didn't really seem like it was at all intuitive, or that there was any good way to opt out of it, it was just simply there, and if you were going to be monitoring your system in a cloud environment—which of course you should be—it was just sort of the cost of doing business that you then have to pay for a third-party tool to wind up using the CloudWatch metrics that it was gathering, and it was just expensive and unpleasant all around. Now, a lot of the criticisms I put about CloudWatch's limitations in those days, about four years ago, have largely been resolved or at least mitigated in different ways. But is CloudWatch still crappy, I guess, is my question?Ewere: Um, yeah. So, at the moment, I think, like you said, CloudWatch has really evolved over time. I personally also had that issue with CloudWatch when I started using CloudWatch; I had the challenge of usability, I had the challenge of proper integration, and I will talk about my first experience with CloudWatch here. So, when I started my infrastructure work, one of the things I was doing a lot was EC2, basically. I mean, everyone always starts with EC2 at the first time.And then we had a downtime. And then my CTO says, “Okay, [Ewere 00:04:00], check what's going on.” And I'm like, “How do I check?” [laugh]. I mean, I had no idea of what to do.And he says, “Okay, there's a tool called CloudWatch. You should be able to monitor.” And I'm like, “Okay.” I dive into CloudWatch, and boom, I'm confused again. And you look at the console, you see, it shows you certain metrics, and yet [people 00:04:18] don't understand what CPU metric talks about, what does network bandwidth talks about?And here I am trying to dig, and dig, and dig deeper, and I still don't get [laugh] a sense of what is actually going on. But what I needed to find out was, I mean, what was wrong with the memory of the system, so I delved into trying to install the CloudWatch agent, get metrics and all that. But the truth of the matter was that I couldn't really solve my problem very well, but I had [unintelligible 00:04:43] of knowing that I don't have memory out of the box; it's something that has to set up differently. And trust me, after then I didn't touch CloudWatch [laugh] again. Because, like you said, it was a problem, it was a bit difficult to work with.But fast forward a couple of years later, I could actually see someone use CloudWatch for a lot of beautiful stuff, you know? It creates beautiful dashboards, creates some very well-aggregated metrics. And also with the aggregated alarms that CloudWatch comes with, [unintelligible 00:05:12] easy for you to avoid what to call incident fatigue. And then also, the dashboards. I mean, there are so many dashboards that simplified to work with, and it makes it easy and straightforward to configure.So, the bootstrapping and the changes and the improvements on CloudWatch over time has made CloudWatch a go-to tool, and most especially the integration with containers and Kubernetes. I mean, CloudWatch is one of the easiest tools to integrate with EKS, Kubernetes, or other container services that run in AWS; it's just, more or less, one or two lines of setup, and here you go with a lot of beautiful, interesting, and insightful metrics that you will not get out of the box, and if you look at other monitoring tools, it takes a lot of time for you to set up, for you to configure, for you to consistently maintain and to give you those consistent metrics you need to know what's going on with your system from time to time.Corey: The problem I always ran into was that the traditional tools that I was used to using in data centers worked pretty well because you didn't have a whole lot of variability on an hour-to-hour basis. Sure, when you installed new servers or brought up new virtual machines, you had to update the monitoring system. But then you started getting into this world of ephemerality with auto-scaling originally, and later containers, and—God help us all—Lambda now, where it becomes this very strange back-and-forth story of, you need to be able to build something that, I guess, is responsive to that. And there's no good way to get access to some of the things that CloudWatch provides, just because we didn't have access into AWS's systems the way that they do. The inverse, though, is that they don't have access into things running inside of the hypervisor; a classic example has always been memory: memory usage is an example of something that hasn't been able to be displayed traditionally without installing some sort of agent inside of it. Is that still the case? Are there better ways of addressing those things now?Ewere: So, that's still the case, I mean, for EC2 instances. So before, now, we had an agent called a CloudWatch agent. Now, there's a new agent called Unified Cloudwatch Agent which is, I mean, a top-notch from CloudWatch agent. So, at the moment, basically, that's what happens on the EC2 layer. But the good thing is when you're working with containers, or more or less Kubernetes kind of applications or systems, everything comes out of the box.So, with containers, we're talking about a [laugh] lot of moving parts. The container themselves with their own CPU, memory, disk, all the metrics, and then the nodes—or the EC2 instance of the virtual machines running behind them—also having their own unique metrics. So, within the container world, these things are just a click of a button. Everything happens at the same time as a single entity, but within the EC2 instance and ecosystem, you still find this there, although the setup process has been a bit easier and much faster. But in the container world, that problem has totally been eliminated.Corey: When you take a look at someone who's just starting to get a glimmer of awareness around what CloudWatch is and how to contextualize it, what are the most common mistakes people make early on?Ewere: I also talked about this in my book, and one of the mistakes people make in terms of CloudWatch, and monitoring in generalities: “What am I trying to figure out?” [laugh]. If you don't have that answer clearly stated, you're going to run into a lot of problems. You need to answer that question of, “What am I trying to figure out?” I mean, monitoring is so broad, monitoring is so large that if you do not have the answer to that question, you're going to get yourself into a lot of trouble, you're going to get yourself into a lot of confusion, and like I said, if you don't understand what you're trying to figure out in the first place, then you're going to get a lot of data, you're going to get a lot of information, and that can get you confused.And I also talked about what I call alarm fatigues or incident fatigues. This happens when you configure so many alarms, so many metrics, and you're getting a lot of alarms hitting and notification services—whether it's Slack, whether it's an email—and it causes fatigue. What happens here is the person who should know what is going on with the system gets a ton of messages and in that scenario can miss something very important because there's so many messages coming in, so many integrations coming in. So, you should be able to optimize appropriately, to be able to, like you said, conceptualize what you're trying to figure out, what problems are you trying to solve? Most times you really don't figure this out for a start, but there are certain bare minimums you need to know about, and that's part of what I talked about in the book.One of the things that I highlighted in the book when I talked about monitoring of different layers is, when you're talking about monitoring of infrastructure, say compute services, such as virtual machines, or EC2 instances, the certain baseline and metrics you need to take note of that are core to the reliability, the scalability, and the efficiency of your system. And if you focus on these things, you can have a baseline starting point before you start going deeper into things like observability and knowing what's going on entirely with your system. So, baseline understanding of—baseline metrics, and baseline of what you need to check in terms of different kinds of services you're trying to monitor is your starting point. And the mistake people make is that they don't have a baseline. So, we do not have a baseline; they just install a monitoring tool, configure a CloudWatch, and they don't know the problem they're trying to solve [laugh] and that can lead to a lot of confusion.Corey: So, what inspired you from, I guess, kicking the tires on CloudWatch—the way that we all do—and being frustrated and confused by it, all the way to the other side of writing a book on it? What was it that got you to that point? Were you an expert on CloudWatch before you started writing the book, or was it, “Well, by the time this book is done, I will certainly know [laugh] more about the service than I did when I started.”Ewere: Yeah, I think it's a double-edged sword. [laugh]. So, it's a combination of the things you just said. So, first of all, I have experienced with other monitoring tools; I have love for reliability and scalability of a system. I started Kubernetes at some of the early times Kubernetes came out, when it was very difficult to deploy, when it was very difficult to set up.Because I'm looking at how I can make systems a little bit more efficient, a little bit more reliable than having to handle a lot of things like auto-scaling, having to go through the process of understanding how to scale. I mean, that's a school of its own that you need to prepare yourself for. So, first of all, I have a love for making sure systems are reliable and efficient, and second of all, I also want to make sure that I know what is going on with my system per time, as much as possible. The level of visibility of a system gives you the level of control and understanding of what your system is doing per time. So, those two things are very core to me.And then thirdly, I had a plan of a streak of books I want to write based on AWS, and just like monitoring is something that is just new. I mean, if you go to the package website, this is the first book on infrastructure monitoring AWS with CloudWatch; it's not a very common topic to talk about. And I have other topics in my head, and I really want to talk about things like networking, and other topics that you really need to go deep inside to be able to appreciate the value of what you see in there with all those scenarios because in this book, every chapter, I created a scenario of what a real-life monitoring system or what you need to do looks like. So, being that I have those premonitions, I know that whenever it came to, you know, to share with the world what I know in monitoring, what I've learned in monitoring, I took a [unintelligible 00:12:26]. And then secondly, as this opportunity for me to start telling the world about the things I learned, and then I also learned while writing the book because there are certain topics in the book that I'm not so much of an expert in things, like big data and all that.I had to also learn; I had to take some time to do more research, to do more understanding. So, I use CloudWatch, okay? I'm kind of good in CloudWatch, and also, I also had to do more learning to be able to disseminate this information. And also, hopefully, X-Ray some parts of monitoring and different services that people do not really pay so much attention into.Corey: What do you find that is still the most, I guess, confusing to you as you take a look across the ecosystem of the entire CloudWatch space? I mean, every time I play with it, I take a look, and I get lost in, “Oh, they have contributor analyses, and logs, and metrics.” And it's confusing, and every time I wind up, I guess, spiraling out of control. What do you find that, after all of this, is a lot easier for you, and what do you find that's a lot more understandable?Ewere: I'm still going to go back to the containers part. I'm sorry, I'm in love containers. [laugh].Corey: No, no, it's fair. Containers are very popular. Everyone loves them. I'm just basically anti-container based upon no better reason than I'm just stubborn and bloody-minded most of the time.Ewere: [laugh]. So, pretty much like I said, I kind of had experience with other monitoring tools. Trust me, if you want to configure proper container monitoring for other tools, trust me, it's going to take you at least a week or two to get it properly, from the dashboards, to the login configurations, to the piping of the data to the proper storage engine. These are things I talked about in the book because I took monitoring from the ground up. I mean, if you've never done monitoring before, when you take my book, you will understand the basic principles of monitoring.And [funny 00:14:15], you know, monitoring has some big data process, like an ETL process: extraction, transformation, and writing of data into an analytic system. So, first of all, you have to battle that. You have to talk about the availability of your storage engine. What are you using? An Elasticsearch? Are you using an InfluxDB? Where do you want to store your data? And then you have to answer the question of how do I visualize the data? What method do I realize this data? What kind of dashboards do I want to use? What methods of representation do I need to represent this data so that it makes sense to whoever I'm sharing this data with. Because in monitoring, you definitely have to share data with either yourself or with someone else, so the way you present the data needs to make sense. I've seen graphs that do not make sense. So, it requires some level of skill. Like I said, I've [unintelligible 00:15:01] where I spent a week or two having to set up dashboards. And then after setting up the dashboard, someone was like, “I don't understand, and we just need, like, two.” And I'm like, “Really?” [laugh]. You know? Because you spend so much time. And secondly, you discover that repeatability of that process is a problem. Because some of these tools are click and drag; some of them don't have JSON configuration. Some do, some don't. So, you discover that scalability of this kind of system becomes a problem. You can't repeat the dashboards: if you make a change to the system, you need to go back to your dashboard, you need to make some changes, you need to update your login, too, you need to make some changes across the layer. So, all these things is a lot of overhead [laugh] that you can cut off when you use things like Container Insights in CloudWatch—which is a feature of CloudWatch. So, for me, that's a part that you can really, really suck out so much juice from in a very short time, quickly and very efficiently. On the flip side, when you talk about monitoring for big data services, and monitoring for a little bit of serverless, there might be a little steepness in the flow of the learning curve there because if you do not have a good foundation in serverless, when you get into [laugh] Lambda Insights in CloudWatch, trust me, you're going to be put off by that; you're going to get a little bit confused. And then there's also multifunction insights at the moment. So, you need to have some very good, solid foundation in some of those topics before you can get in there and understand some of the data and the metrics that CloudWatch is presenting to you. And then lastly, things like big data, too, there are things that monitoring is still being properly fleshed out. Which I think that in the coming months and years to come, they will become more proper and they will become more presentable than they are at the moment.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: The problem I've always had with dashboards is it seems like managers always want them—“More dashboards, more dashboards”—then you check the usage statistics of who's actually been viewing the dashboards and the answer is, no one since you demoed it to the execs eight months ago. But they always claim to want more. How do you square that?I guess, slicing between what people asked for and what they actually use.Ewere: [laugh]. So yeah, one of the interesting things about dashboards in terms of most especially infrastructure monitoring, is the dashboards people really want is a revenue dashboards. Trust me, that's what they want to see; they want to see the money going up, up, up, [laugh] you know? So, when it comes to—Corey: Oh, yes. Up and to the right, then everyone's happy. But CloudWatch tends to give you just very, very granular, low-level metrics of thing—it's hard to turn that into something executives care about.Ewere: Yeah, what people really care about. But my own take on that is, the dashboards are actually for you and your team to watch, to know what's going on from time to time. But what is key is setting up events across very specific and sensitive data. For example, when any kind of sensitive data is flowing across your system and you need to check that out, then you tie a metric to that, and in turn alarm to it. That is actually the most important thing for anybody.I mean, for the dashboards, it's just for you and your team, like I said, for your personal consumption. “Oh, I can see all the RDS connections are getting too high, we need to upgrade.” Oh, we can see that all, the memory, there was a memory spike in the last two hours. I know that's for you and your team to consume; not for the executive team. But what is really good is being able to do things like aggregate data that you can share.I think that is what the executive team would love to see. When you go back to the core principles of DevOps in terms of the DevOps Handbook, you see things like a mean time to recover, and change failure rate, and all that. The most interesting thing is that all these metrics can be measured only by monitoring. You cannot change failure rates if you don't have a monitoring system that tells you when there was a failure. You cannot know your release frequency when you don't have a metric that measures number of deployments you have and is audited in a particular metric or a particular aggregator system.So, we discovered that the four major things you measure in DevOps are all tied back to monitoring and metrics, at minimum, to understand your system from time to time. So, what the executive team actually needs is to get a summary of what's going on. And one of the things I usually do for almost any company I work for is to share some kind of uptime system with them. And that's where CloudWatch Synthetics Canary come in. So, Synthetic Canary is a service that helps you calculate that helps you check for uptime of the system.So, it's a very simple service. It does a ping, but it is so efficient, and it is so powerful. How is it powerful? It does a ping to a system and it gets a feedback. Now, if the status code of your service, it's not 200 or not 300, it considers it downtime.Now, when you aggregate this data within a period of time, say a month or two, you can actually use that data to calculate the uptime of your system. And that uptime [unintelligible 00:19:50] is something you can actually share to your customers and say, “Okay, we have an SLA of 99.9%. We have an SLA of 99.8%.” That data should not be doctored data; it should not be a data you just cook out of your head; it should be based on your system that you have used, worked with, monitored over a period of time so that the information you share with your customers are genuine, they are truthful, and they are something that they can also see for themselves.Hence companies are using [unintelligible 00:20:19] like status page to know what's going on from time to time whenever there is an incident and report back to their customers. So, these are things that executives will be more interested in than just dashboards, [laugh] dashboards, and more dashboards. So, it's more or less not about what they really ask for, but what you know and what you believe you are going to draw value from. I mean, an executive in a meeting with a client and says, “Hey, we got a system that has 99.9% uptime.”He opens the dashboard or he opens the uptime system and say, “You see our uptime? For the past three months, this has been our metric.” Boom. [snaps fingers]. That's it. That's value, instantly. I'm not showing [laugh] the clients and point of graphs, you know? “Can you explain the memory metric?” That's not going to pass the message, send the message forward.Corey: Since your book came out, I believe, if not, certainly by the time it was finished being written and it was in review phase, they came out with Managed Prometheus and Managed Grafana. It looks almost like they're almost trying to do a completely separate standalone monitoring stack of AWS tooling. Is that a misunderstanding of what the tools look like, or is there something to that?Ewere: Yeah. So, I mean by the time those announced at re:Invent, I'm like, “Oh, snap.” I almost told my publisher, “You know what? We need to add three more chapters.” [laugh]. But unfortunately, we're still in review, in preview.I mean, as a Hero, I kind of have some privilege to be able to—a request for that, but I'm like, okay, I think it's going to change the narrative of what the book is talking about. I think I'm going to pause on that and make sure this finishes with the [unintelligible 00:21:52], and then maybe a second edition, I can always attach that. But hey, I think there's trying to be a galvanization between Prometheus, Grafana, and what CloudWatch stands for. Because at the moment, I think it's currently on pre-release, it's not fully GA at the moment, so you can actually use it. So, if you go to Container Insights, you can see that you can still get how Prometheus and Grafana is presenting the data.So, it's more or less a different view of what you're trying to see. It's trying to give you another perspective of how your data is presented. So, you're going to have CloudWatch: it's going to have CloudWatch dashboards, it's going to have CloudWatch metrics, but hey, this different tools, Prometheus, Grafana, and all that, they all have their unique ways of presenting the data. And part of the reason I believe AWS has Prometheus and Grafana there is, I mean, Prometheus is a huge cloud-native open-source monitoring, presentation, analytics tool; it packs a lot of heat, and a lot of people are so used to it. Everybody like, “Why can't I have Prometheus in CloudWatch?”I mean—so instead of CloudWatch just being a simple monitoring tool, [unintelligible 00:22:54] CloudWatch has become an ecosystem of monitoring tool. So, we got—we're not going to see cloud [unintelligible 00:23:00], or just [unintelligible 00:23:00] log, analytics, metrics, dashboards, no. We're going to see it as an ecosystem where we can plug in other services, and then integrate and work together to give us better performance options, and also different perspectives to the data that is being collected.Corey: What do you think is next, as you take a look across the ecosystem, as far as how people are thinking about monitoring and observability in a cloud context? What are they missing? Where's the next evolution lead?Ewere: Yeah, I think the biggest problem with monitoring, which is part of the introduction part of the book, where I talked about the basic types of monitoring—which is proactive and reactive monitoring—is how do we make sure we know before things happen? [laugh]. And one of the things that can help with that is machine learning. There is a small ecosystem that is not so popular at the moment, which talks about how we can do a lot of machine learning in DevOps monitoring observability. And that means looking at historic data and being able to predict on the basic level.Looking at history, [then are 00:24:06] being able to predict. At the moment, there are very few tools that have models running at the back of the data being collected for monitoring and metrics, which could actually revolutionize monitoring and observability as we see it right now. I mean, even the topic of observability is still new at the moment. It's still very integrated. Observability just came into Cloud, I think, like, two years ago, so it's still being matured.But one thing that has been missing is seeing the value AI can bring into monitoring. I mean, this much [unintelligible 00:24:40] practically tell us, “Hey, by 9 p.m. I'm going to go down. I think your CPU or memory is going down. I think I'm line 14 of your code [laugh] is a problem causing the bug. Please, you need to fix it by 2 p.m. so that by 6 p.m., things can run perfectly.” That is going to revolutionize monitoring. That's going to revolutionize observability and bring a whole new level to how we understand and monitor the systems.Corey: I hope you're right. If you take a look right now, I guess, the schism between monitoring and observability—which I consider to be hipster monitoring, but they get mad when I say that—is there a difference? Is it just new phrasing to describe the same concepts, or is there something really new here?Ewere: In my book, I said, monitoring is looking at it from the outside in, observability is looking at it from the inside out. So, what monitoring does not see under, basically, observability sees. So, they are children of the same mom. That's how I put it. One actually needs the other and both of them cannot be separated from each other.What we've been working with is just understanding the system from the surface. When there's an issue, we go to the aggregated results that come out of the issue. Very basic example: you're in a Java application, and we all know Java is very memory intensive, on the very basic layer. And there's a memory issue. Most times, infrastructure is the first hit with the resultant of that.But the problem is not the infrastructure, it's maybe the code. Maybe garbage collection was not well managed; maybe they have a lot of variables in the code that is not used, and they're just filling up unnecessary memory locations; maybe there's a loop that's not properly managed and properly optimized; maybe there's a resource on objects that has been initialized that has not been closed, which will cause a heap in the memory. So, those are the things observability can help you track. Those are the things that we can help you see. Because observability runs from within the system and send metrics out, while basic monitoring is about understanding what is going on on the surface of the system: memory, CPU, pushing out logs to know what's going on and all that.So, on the basic level, observability helps gives you, kind of, a deeper insight into what monitoring is actually telling you. It's just like the result of what happened. I mean, we are told that the symptoms of COVID is coughing, sneezing, and all that. That's monitoring. [laugh].But before we know that you actually have COVID, we need to go for a test, and that's observability. Telling us what is causing the sneezing, what is causing the coughing, what is causing the nausea, all the symptoms that come out of what monitoring is saying. Monitoring is saying, “You have a cough, you have a runny nose, you're sneezing.” That is monitoring. Observability says, “There is a COVID virus in the bloodstream. We need to fix it.” So, that's how both of them act.Corey: I think that is probably the most concise and clear definition I've ever gotten on the topic. If people want to learn more about what you're up to, how you view about these things—and of course, if they want to buy your book, we will include a link to that in the [show notes 00:27:40]—where can they find you?Ewere: I'm on LinkedIn; I'm very active on LinkedIn, and I also shared the LinkedIn link. I'm very active on Twitter, too. I tweet once in a while, but definitely, when you send me a message on Twitter, I'm also going to be very active.I also write blogs on Medium, I write a couple of blogs on Medium, and that was part of why AWS recognized me as a Hero because I talk a lot about different services, I help with comparing services for you so you can choose better. I also talk about setting basic concepts, too; if you just want to get your foot wet into some stuff and you need something very summarized, not AWS documentation per se, something that you can just look at and know what you need to do with the service, I talk about them also in my blogs. So yeah, those are the two basic places I'm in: LinkedIn and Twitter.Corey: And we will, of course, put links to that in the [show notes 00:28:27]. Thank you so much for taking the time to speak with me. I appreciate it.Ewere: Thanks a lot.Corey: Ewere Diagboya, head of cloud at My Cloud Series. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment telling me how many more dashboards you would like me to build that you will never look at.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About TimTim's tech career spans over 20 years through various sectors. Tim's initial journey into tech started as a US Marine. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for largeUnix-based datastores.Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with clients in his current role. Tim is also a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and 3-time Pan American Brazilian Jiu-Jitsu champion in his division.Links: Twitter: https://twitter.com/elchefe The Duckbill Group: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, I have a whole bunch of guests come on up, second time. Now, it's easy to take the naive approach of assuming that it's because it's easier for me to find a guest if I know them and don't have to reach out to brand new people all the time. This is absolutely correct; I'm exceedingly lazy. But I don't have too many folks on a third time, but that changes today.My guest is Tim Banks. I've had him on the show twice before, both times it led to really interesting conversations around a wide variety of things. Since those episodes, Tim has taken the job as a principal cloud economist here at The Duckbill Group. Yes, that is probably the strangest interview process you can imagine, but here we are. Tim, thank you so much for joining me both on the show and in the business.Tim: My pleasure, Corey. It was definitely an interesting interview process, you know, but I was glad to be here. So, I'm happy to be here a third time. I don't know if you get a jacket like you do in Saturday Night Live, if you host, like, a fifth time, but we'll see. Maybe it's a vest. A cool vest would be nice.Corey: We can come up with something.[ effectively, it can be like reverse hangman where you wind up getting a vest and every time you come on after that you get a sleeve, then you get a second sleeve, and then you get a collar, and we can do all kinds of neat stuff.Tim: I actually like that idea a lot.Corey: So, I'm super excited to be able to have this conversation with you because I don't normally talk a lot on this show about what cloud economics is because my guest usually is not as deep into the space as I am, and that's fine; people should never be as deep into this space as I am, in the general sense, unless they work here. Awesome. But I do guest on other shows, and people ask me all kinds of questions about AWS billing and cloud economics, and that's fine, it's great, but they don't ask the questions about the space in the same way that I would and the way that I think about it. So, it's hard for me to interview myself. Now, I'm not saying I won't try it someday, but it's challenging. But today, I get to take the easy path out and talk to you about it. So Tim, what the hell is a principal cloud economist?Tim: So, a principal cloud economist, is a cloud computing expert, both in architecture and practice, who looks at cloud cost in the same way that a lot of folks look at cloud security, or cloud resilience, or cloud performance. So, the same engineering concerns you have about making sure that your API stays up all the time, or to make sure that you don't have people that are able to escape containers or to make sure that you can have super, super low response times, is the same engineering fundamentals that I look at when I'm trying to find a way to reduce your AWS bill.Corey: Okay. When we say cloud cost and cloud economics, the natural picture that leads to mind is, “Oh, I get it. You're an Excel jockey.” And sometimes, yeah, we all kind of play those roles, but what you're talking about is something else entirely. You're talking about engineering expertise.And sure enough, if you look at the job postings we have for roles on the team from time to time, we have not yet hired anyone who does not have an engineering and architecture background. That seems odd to folks who do not spend a lot of time thinking about the AWS bill. I'm told those people are what is known as ‘happy.' But here we are. Why do we care about the engineering aspect of any of this?Tim: Well, I think first and foremost because what we're doing in essence, is still engineering. People aren't putting construction paper up on [laugh] AWS; sometimes they do put recipes up on there, but it still involves working on a computer, and writing code, and deploying it somewhere. So, to have that basic understanding of what it is that folks are doing on the platform, you have to have some engineering experience, first and foremost. Secondly, the fact of the matter is that most cost optimization, in my opinion, can be done on the whiteboard, before anything else, and really I think should be done on the whiteboard before anything else. And so the Excel aspect of it is always reactive. “We have now spent this much. How much was it? Where did it go?” And now we have to figure out where it went.I like to figure out and get a ballpark on how much something is going to cost before I write the first line of code. I want to know, hey, we have a tier here, we're using this kind of storage, it's going to take this kind of instance types. Okay, well, I've got an idea of how much it's going to cost. And I was like, “You know, that's going to be expensive. Before we do anything, is there a way that we can reduce costs there?”And so I'm reverse engineering that on already deployed workloads. Or when customers want to say, “Hey, we were thinking about doing this, and this is our proposed architecture,” I'm going to look at it and say, “Well, if you do this and this and this and this, you can save money.”Corey: So, it sounds like you and I have a bit of a philosophical disagreement in some ways. One of my recurring talking points has always been that, “Oh, by and large, application developers don't need to think overly much about cloud cost. What they need to know generally fits on an index card.” It's, okay, big things cost more than small things; if you turn something on, it will never get turned off and will bill you in perpetuity; data transfer has some weird stuff; and if you store data, you pay for data, like, that level of baseline understanding. When I'm trying to build something out my immediate thought is, great, is this thing possible?Because A, I don't always know that it is, and B, I'm super bad at computers so for me, it may absolutely not be, whereas you're talking about baking cost assessments into the architecture as a day one type of approach, even when sketching ideas out on the whiteboard. I'm curious as to how we diverge there. Can you talk more about your philosophy?Tim: Sure. And the reason I do that is because, as most folks that have an engineering background in cloud infrastructure will tell you, you want to build resilience in, on the whiteboard. You certainly want to build performance in, on the whiteboard, right? And security folks will tell you you want to do security on the whiteboard. Because those things are hard to fix after they're deployed.As soon as they're deployed, without that, you now have technical debt. If you don't consider cost optimization and cost efficiency on the whiteboard, and then you try and do it after it's deployed, you not only have technical debt, you may have actual real debt.Corey: One of the comments I tend to give a lot is that architecture and cost are the same thing in the world of cloud. And I think that we might be in violent agreement, as Liz Fong-Jones is fond of framing it, where I am acutely aware of aspects of cost and that does factor into how I build things on the whiteboard—let's also be very clear, most of the things that I build are very small scale; the largest cost by a landslide is the time I spend building it—in practice, that's an awful lot of environments; people are always more expensive than the AWS environment they're working on. But instead, it's about baking in the assumptions and making sure you're not coming up with something that is going to just be wasteful and horrible out of the gate, and I guess part of that also is the fact that I am at a level of billing understanding that I sort of absorbed these concepts intrinsically. Because to me, there is no difference between cost and architecture in an environment like this. You're right, there's always an inherent trade-off between cost and durability. On the one hand, I don't like that. On the other, it feels like it's been true forever and I don't see a way out of it.Tim: It is inescapable. And it's interesting because you talk about the level of an application developer or something like that, like what is your level of concern, but retroactively, we'll go in for cost optimization houses—and I've done this as far back as when I was working at AWS has a TAM—and I'll ask the question to an application developer or database administrator, and I'm like, “Why do you do this? What do you have a string value for something that could be a Boolean?” And you'll ask, “Well, what difference does that make?” Well, it makes a big difference when you're talking about cycles for CPU.You can reduce your CPU consumption on a database instance by changing a string to a Boolean, you need fewer instances, or you need a less powerful instance, or you need less memory. And now you can run a less expensive instance for your database architecture. Well, maybe for one node it's not that biggest difference, but if you're talking about something that's multi-AZ and multi-node, I mean, that can be a significant amount of savings just by making one simple change.Corey: And that might be the difference right there. I didn't realize that, offhand. It makes sense if you think about it, but just realizing that I've made that mistake on one of my DynamoDB tables. It costs something like seven cents a month right now, so it's not something I'm rushing to optimize, but you're right, expand that out by a factor of a million or so, and we're talking serious money, and then that sort of optimization makes an awful lot of sense. I think that my position on it is that when you're building out something small scale as a demo or a proof of concept, spending time on optimizations like this is not the best use of anyone's time or brain sweat, for lack of a better term. How do you wind up deciding when it's time to focus on stuff like that?Tim: Well, first, I will say that—I daresay that somewhere in the 80% of production workloads are just—were the POC, [laugh] right? Because, like, “It worked for this to get funding, let's run it,” right?Corey: Let they who does not have a DynamoDB table in production with the word ‘test' or ‘dev' in it cast the first stone.Tim: It's certainly not me. So, I understand how some of those decisions get made. And that's why I think it's better to think about it early. Because as I mentioned before, when you start something and say, “Hey, this works for now,” and you don't give consideration to that in the future, or consideration for what it's going to be like in the future, and when you start doing it, you'll paint yourself into corners. That's how you get something like static values put in somewhere, or that's how you get something like, well, “We have to run this instance type because we didn't build in the ability to be more microservice-based or stateless or anything like that.”You've seen people that say, “Hey, we could save you a lot of money if you can move this thing off to a different tier.” And it's like, “Well, that would be an extensive rewrite of code; that'd be very expensive.” I daresay that's the main reason why most AS/400s are still being used right now is because it's too expensive to rewrite the code.Corey: Yeah, and there's no AWS/400 that they can migrate to. Yet. Re:Invent is nigh.Tim: So, I think that's why, even at the very beginning, even if you were saying, “Well, this is something we will do later.” Don't make it impossible for you to do later in your code. Don't make it impossible for you to do later in your architecture. Make things as modular as possible, so that way you can say, “Hey”—later on down the road—“Oh, we can switch this instance type.” Or, “Here's a new managed service that we can maybe save money on doing this.”And you allow yourself to switch things out, or turn different knobs, or change the way you do things, and give yourself more options in the future, whether those options are for resilience, or those options or for security, or those options are for performance, or they're for cost optimizations. If you make binding decisions earlier on, you're going to have debt that's going to build up at some point in the future, and then you're going to have to pay the piper. Sometimes that piper is going to be AWS.Corey: One thing that I think gets lost in a lot of conversations about cloud economics—because I know that it happened to me when I first started this place—where I am planning to basically go out and be the world's leading expert in AWS cost analysis and understanding and optimization. Great. Then I went out into the world and started doing some of my first engagements, and they looked a lot less like far-future cost attribution projections and a lot more like, “What's a reserved instance?” And, “We haven't bought any of those in 18 months.” And, “Oh, yeah, we shut down an entire project six months ago. We should probably delete all the resources, huh?”The stuff that I was preparing for at the high end of the maturity curve are great and useful and terrific to have conversations about in some very nuanced depth, but very often there's a walk before you can run style of conversation where, okay, let's do the easy stuff first before we start writing a whole bunch of bespoke internal stuff that maps your business needs to the AWS bill. How do you, I guess, reconcile those things where you're on the one hand, you see the easy stuff and on the other, you see some of the just the absolutely challenging, very hard, five-years-of-engineering-effort-style problems on the other?Tim: Well, it's interesting because I've seen one customer very recently who has brilliant analyses as to their cost; just well-charted, well-tagged, well-documented, well—you know, everything is diagrammed quite nicely and everything like that, and they're very, very aware of their costs, but they leave test instances running all weekend, you know, and their associated volumes and things like that. And that's a very easy thing to fix. That is a very, very low-hanging fruit. And so sometimes, you just have to look at where they're spending their efforts where sometimes they do spend so much time chasing those hard to do things because they are hard to do and they're exciting in an engineering aspect, and then something as simple as, “Hey, how about we delete these old volumes?” It just isn't there.Or, “How about we switch to your S3 bucket storage type?” Those are easy, low-hanging fruits, and you would be surprised how sometimes they just don't get that. But at the same time, sometimes customers have, like, “Hey, we could knock this thing out, we knock this thing out,” because it's Trusted Advisor. Every AI cost optimization recommendation you can get will tell you these five things to do, no matter who you are or where you are, but they don't do the conceptual things like understanding some of the principles behind cost optimization and cost optimization architecture, and proactive cost optimization versus react with cost optimizations. So, you're doing very conceptual education and conversations with folks rather than the, “Do these five things.” And I've not often found a customer that you have to do both on; it's usually one or the other.Corey: It's funny that you made that specific reference to that example. One of my very first projects—not naming names. Generally, when it comes to things like this, you can tell stories or you can name names; I bias for stories—I was talking to a company who was convinced that their developer environments were incredibly overwrought, expensive, et cetera, and burning money. Okay, great. So, I talked about the idea of turning those things off at night or between test runs, deleting volumes to snapshot, and restore them on a schedule when people come in in the morning because all your developers sit in the same building in the same time zones. Great. They were super on board with the idea, and it was going to be a little bit of work, but all right, this was in the days before the EC2 Instance Scheduler, for example.But first, let's go ahead and do some analysis. This is one of those early engagements that really reinforced my idea of, yeah, before we start going too far down the rabbit hole, let's double-check what's going on in the account. Because periodically you encounter things that surprise people. Like, “What's up with those Australia instances?” “Oh, we don't have anything in that region.” “I believe you're being sincere when you say this, however, the API generally doesn't tell lies.”So, that becomes a, oh, security incident time. But looking at this, they were right; they had some fairly sizable developer instances that were running all the time, but doing some analysis, their developer environment was 3% of their bill at the time and they hadn't bought RIs in a year-and-a-half. And looking at what they were doing, there was so much easier stuff that they could do to generate significant savings without running the potential of turning a developer environment off at night in the middle of an incident or something like that. The risk factor and effort were easier just do the easy stuff, then do another pass and look at the deep stuff. And to be clear, they weren't lying to me; they weren't wrong.Back when they started building this stuff out, their developer environments were significantly large and were a significant portion of their spend. And then they hit product-market fit, and suddenly their production environment had to scale significantly in a short period of time. Which, yay, cloud. It's good at that. Then it just became such a small portion that developer environments weren't really a thing. But the narrative internally doesn't get updated very often because once people learn something, they don't go back to relearn whether or not it's still true. It's a constant mistake; I make it myself frequently.Tim: I think it's interesting, there are things that we really need to put into buckets as far as what's an engineering effort and what's an administrative effort. And when I say ‘administrative effort,' I mean if I can save money with a stroke of a pen, well, that's going to be pretty easy, and that's usually going to be RIs; that's going to be EDPs, or PPAs or something like that, that don't require engineering effort. It just requires administrative effort, I think RIs being the simplest ones. Like, “Oh, all I have to do is go in here and click these things four times and I'm going to save money?” “Well, let's do that.”And it's surprising how often people don't do that. But you still have to understand that, and whether it's RIs or whether it's a savings plan, it's still a commitment of some kind, but if you are willing to make that commitment, you can save money with no engineering effort whatsoever. That's almost free money.Corey: So, much of what we do here comes down to psychology, in many ways, more than it does math. And a lot of times you're right, everything you say is right, but in a large-scale environment, go ahead and click that button to buy the savings plan or the reserved instance, and that's a $20 million purchase. And companies will stall for months trying to run a different series of analyses on this and what if this happens, what if that happens, and I get it because, “Yeah, I'm going to click this button that's going to cost more money than I'll make in my lifetime,” that's a scary thing to do; I get it. But you're going to spend the money, one way or the other, with the provider, and if you believe that number is too high, I get it; I am right there with you. Buy half of them right now and then you can talk about the rest until you get to a point of being comfortable with it.Do it incrementally; it's not all or nothing, you have one shot to make the buy. Take pieces out of it that makes sense. You know you're probably not going to turn off your database cluster that handles all of production in the next year, so go ahead and go for it; it saves some money. Do the thing that makes sense. And that doesn't require deep-dive analytics that requires, on some level, someone who's seen a lot of these before who gets what customers are going through. And honestly, it's empathy in many respects, becomes one of those powerful things that we can apply to our customer accounts.Tim: Absolutely. I mean, people don't understand that decision paralysis, about making those commitments costs you money. You can spend months doing analysis, but those months doing analysis, you're going to spend 30, 40, 50, 60, 70% more on your EC2 instances or other compute than you would otherwise, and that can be quite significant. But it's one of those cases where we talk about psychology around perfect being the enemy of good. You don't have to make the perfect purchase of RIs or savings plans and have that so tuned perfectly that you're going to get one hundred percent utilization and zero—like, you don't have to do that.Just do something. Do a little bit. Like you said, buy half; buy anything; just something, and you're going to save money. And then you can run analysis later on, while you're saving money [laugh] and get a little better and tune it up a little more and get more analysis on and maybe fine-tune it, but you don't actually ever need to have it down to the penny. Like, it never has to be that good.Corey: At some point, one of the value propositions we have for our customers has always been that we tell you when to stop focusing on saving money because there's a theoretical cap of a hundred percent of the cloud bill that you can save, but you can make so much more than that by launching the right feature to the right market a little sooner; focus on that. Be responsible stewards of the money that's invested with you, but by and large, as a general piece of guidance, at some point, stop cutting and go back to doing the thing that makes your company work. It's not all about saving money at all costs for almost all of us. It is for us, but we're sort of a special case.Tim: Well, it's a conversation I often have. It's like, all right, are you trying to save money on AWS or are you trying to save money overall? So, if you're going to spend $400,000 worth of engineering effort to save $10,000 on your AWS bill, that doesn't make no sense. So—[laugh]—Corey: Right. There has to be a strategic reason to do things like that—Tim: Exactly.Corey: —and make sure you understand the value of what you're getting for this. One reason that we wind up charging the way that we do—and we've gotten questions on this for a while—has been that we charge a fixed fee for what we do on engagements. And similarly—people have asked this, but haven't tied the two things together—you talk about cost optimization, but never cost-cutting. Why is that? Is that just a negative term?And the answer has been no, they're aligned. What we do focuses on what is best for the customer. Once that fixed fee is decided upon, every single thing that we say is what we would do if we were in the customer's position. There are times we'll look at what they have going on and say, “Ah, you really should spend more money here for resiliency, or durability,” or, “Okay, that is critical data that's not being backed up. You should consider doing that.”It's why we don't take percentages of things because, at that point, we're not just going with the useful stuff, it's, well we're going to basically throw the entire kitchen sink at you. We had an early customer and I was talking to their AWS account manager about what we were going to be doing and their comment was, “Oh, saving money on AWS bills is great, make sure you check the EBS snapshots.” Yeah, I did that. They were spending 150 bucks a month on EBS snapshots, which is basically nothing. It's one of those stories where if, in the course of an hour-long meeting, I can pay for that entire service, by putting a quarter on the table, I'm probably not going to talk about it barring [laugh] some extenuating circumstances.Focus on the big things, not the things that worked in a different environment with a different account and different constraints. It's hard to context switch like that, but it gets a lot easier when it is basically the entirety of what we do all day.Tim: The difference I draw between cost optimization and cost-cutting is that cost optimization is ensuring that you're not spending money unnecessarily, or that you're maximizing your dollar. And so sometimes we get called in there, and we're just validation for the measures they've already done. Like, “Your team is doing this exactly right. You're doing the things you should be doing. We can nitpick if you want to; we're going to save you $7 a year, but who cares about that? But y'all are doing what you should be doing. This is great. Going forward, you want to look for these things and look for these things and look for these things. We're going to give you some more concepts so that you are cost-optimized in the future.” But it doesn't necessarily mean that we have to cut your bill. Because if you're already spending efficiently, you don't need your bill cut; you're already cost-optimized.Corey: Oh, we're not going to nitpick on that, you're mostly optimized there. It's like, “Yeah, that workload's $140 million a year and rising; please, pick nits.” At which point? “Okay, great.” That's the strategic reason to focus on something. But by and large, it comes down to understanding what the goals of clients are. I think that is widely misunderstood about what we do and how we do it.The first question I always ask when someone does outreach of, “Hey, we'd like to talk about coming in here and doing a consulting engagement with us.” “Great.” I always like to ask the quote-unquote, “Foolish question” of, “Why do you care about the AWS bill?” And occasionally I'll get people who look at me like I have two heads of, “Why wouldn't I care about the AWS bill?” Because there are more important things to care about for the business, almost certainly.Tim: One of the things I try and do, especially when we're talking about cost optimization, especially trying to do something for the right now so they can do things going forward, it's like, you know, all right, so if we cut this much from your bill—if you just do nothing else, but do reserved instances or buy a savings plan, right, you're going to save enough money to hire four engineers. Think about what four engineers would do for your overall business? And that's how I want you to frame it; I want you to look at what cost optimization is going to allow you to do in the future without costing you any more money. Or maybe you save a little more money and you can shift it; instead of paying for your AWS bill, maybe you can train your developers, maybe you can get more developers, maybe you can get some ProServ, maybe you can do whatever, buy newer computers for your people so they can do—whatever it is, right? We're not saying that you no longer have to spend this money, but saying, “You can use this money to do something other than give it to Jeff Bezos.”Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: There was an article recently, as of the time of this recording, where Pinterest discussed what they had disclosed in one of their regulatory filings which was, over the next eight years, they have committed to pay AWS $3.2 billion. And in this article, they have the head of engineering talking to the reporter about how they're thinking about these things, how they're looking at things that are relevant to their business, and they're talking about having a dedicated team that winds up doing a whole bunch of data analysis and running some analytics on all of these things, from piece to piece to piece. And that's great. And I worry, on some level, that other companies are saying, “Oh, Pinterest is doing that. We should, too.” Yeah, for the course of this commitment, a 1% improvement is $32 million, so yeah, at that scale I'm going to hire a team of data scientists, too, look at these things. Your bill is $50,000 a month. Perhaps that's not worth the effort you're going to put into it, barring other things that contribute to it.Tim: It's interesting because we will get folks that will approach us that have small accounts—very small, small spend—and like, “Hey, can you come in and talk to us about this whatever.” And we can say very honestly, “Look, we could, but the amount of money we're going to charge you is going to—it's not going to be worth your while right now. You could probably get by on the automated recommendations, on the things that already out there on the internet that everybody can do to optimize their bill, and then when you grow to a point where now saving 10% is somebody's salary, that's when it, kind of, becomes more critical.” And it's hard to say what point that is in anyone's business, but I can say sometimes, “Hey, you know what? That's not really what you need to focus on.” If you need to save $100 a month on your AWS bill, and that's critical, you've got other concerns that are not your AWS bill.Corey: So, back when you were interviewing to work here, one of the areas of focus that you kept bringing up was the concept of observability, and my response to this was, “Ah, hell. Another one.” Because let's be clear, Mike Julian—my business partner and our CEO—has written a book called Practical Monitoring, and apparently what we learned from this is as soon as you finish writing a book on the topic, you never want to talk about that topic ever again, which yeah, in hindsight makes sense. Why do you care about observability when you're here to look at cloud costs?Tim: Because cloud costs is another metric, just like you would use for performance, or resilience, or security. You do real-time monitoring to see if somebody has compromised the system, you do real-time monitoring to see if you have bad performance, if response times are too slow. You do real-time monitoring to know if something has gone down and then you need to make adjustments, or that the automated responses you have in response to that downtime are working. But cloud costs, you send somebody a report at the end of the month. Can you imagine, if you will—just for a second—if you got a downtime report at the end of month, and then you can react to something that has gone down?Or if you get a security report at the end of the month, and then you can react to the fact that somebody has your root keys? Or if you get [laugh] a report at the end of month, this said, “Hey, the CPU on this one was pegged. You should probably scale up.” That's outrageous to anybody in this industry right now. But why do we accept that for cloud cost?Corey: It's worse than that. There are a number of startups that talk about, “Oh, real-time cloud cost monitoring. Okay, the only way you're going to achieve such a thing is if you build an API shim that interprets everything that you're telling your cloud control plane to do, taking cost metrics out of it, and then passing it on to the actual cloud control plane.” Otherwise, you're talking about it showing up in the billing record in—ideally, eight hours; in practice, several days, or you're talking about the CloudTrail events, which is not holistic but gives you some rough idea, but it's also in some cases, 5 to 20 minutes delayed. There's no real-time way to do this without significant disruption to what's going on in your environment.So, when I hear about, “Oh, we do real-time bill analysis.” Yeah, it feels—to be very direct—you don't know enough about the problem space you're working within to speak intelligently about it because anyone who's played in this space for a while knows exactly how hard it is to get there. Now, I've talked to companies that have built real-time-ish systems that take that shim approach and acts sort of as a metadata sidecar ersatz billing system that tracks all of this so they can wind up intercepting potentially very expensive configuration mistakes. And that's great. That's also a bit beyond for a lot of folks today, but it's where the industry is going. But there is no way to get there today, short of effectively intercepting all of those calls, in a way that is cohesive and makes sense. How do you square that circle given the complete lack of effective tooling?Tim: Honestly, I'm going to point that right back at the cloud provider because they know how much you're spending, real-time. They know exactly how much you spend in real-time. They've figured it out. They have the buckets, they have APIs for it internally. I'm sure they do; it would make no sense for them not to. Without giving anything anyway, I know that when I was at AWS, I knew how much they were spending, almost real-time.Corey: That's impressive. I wish that existed. My never having worked at AWS perspective on it is that they, of course, have the raw data effective immediately, or damn close to it, but the challenge for the billing system is distilling and summarizing and attributing all of that in a reasonable timeframe; it is an exabyte-scale problem. I've talked to folks there who have indicated it is comfortably north of a petabyte in raw data per day. And that was a couple of years ago, so one can only imagine as the footprint has increased, so has all of this.I mean, the billing system is fundamentally magic from the outside. I'm not saying it's good magic, but it is magic, and it's something that is unappreciated, that every customer uses, and is one of those areas that doesn't get the attention it deserves. Because, let's be clear, here, we talk about observability; the bill is still the only thing that AWS offers that gives you a holistic overview of everything running in your account, in one place.Tim: What I think is interesting is that you talk about this, the scale of the problem and that it makes it difficult to solve. At the same time, I can have a conversation with my partner about kitty litter, and then all of a sudden, I'm going to start getting ads about kitty litter within minutes. So, I feel like it's possible to emit cost as a metric like you would CPU or disk. And if I'm going to look at who's going to do that, I'm going to look right back at AWS. The fun part about that, though, is I know from AWS's business model, that if that's something they were to emit, it would also cost you, like, 25 cents per call, and then you would actually, like, triple your cloud costs just trying to figure out how much it costs you.Corey: Only with 16 other billing dimensions because of course it would. And again, I'm talking about stuff, because of how I operate and how I think about this stuff, that is inherently corner case, or [vertex 00:31:39] case in many cases. But for the vast majority of folks, it's not the, “Oh, you have this really weird data transfer paradigm between these two resources,” which yeah, that's a problem that needs to be addressed in an awful lot of cases because data transfer pricing is bonkers, but instead it's the, “Huh. You just spun up a big cluster that's going to cost $20,000 a month.” You probably don't need to wait a full day to flag that.And you also can't put this on the customer in the sense of, “Oh, just set some budget alarms, that's great. That's the first thing you should do in a new AWS account.” “Well, jackhole, I've done an awful lot of first things I'm supposed to do in an AWS account, in my dedicated test account for these sorts of things. It's been four months, I'm not done yet with all of those first things I'm supposed to do.” It's incredibly secure, increasingly expensive, and so far all it runs is a single EC2 instance that is mostly there just so that everything else doesn't error out trying to divide by zero.Tim: There are some things that are built-in. If I stand up an EC2 instance and it goes down, I'm going to get an alert that this instance terminated for some reason. It's just going to show up informationally.Corey: In the console. You're not going to get called about it or paged about it, unless—Tim: Right.Corey: —you have something else in the business that will, like a boss that screams at you two o'clock in the morning. This is why we have very little that's production-facing here.Tim: But if I know that alert exists somewhere in the console, that's easy for me to write a trap for. That's easy for me to write, say hey, I'm going to respond to that because this call is going to come out somewhere; it's going to get emitted somewhere. I can now, as an engineer, write a very easy trap that says, “Hey, pop this in the Slack. Send an alert. Send a page.”So, if I could emit a cost metric, and I could say, “Wow. Somebody has spun up this thing that's going to cost X amount of money. Someone should get paged about this.” Because if they don't page about this and we wait eight hours, that's my month's salary. And you would do that if your database server went down; you would do that if someone rooted that database server; you would do that if the database server was [bogging 00:33:48] you to scale up another one. So, why can't you do that if that database server was all of sudden costing you way more than you had calculated?Corey: And there's a lot of nuance here because what you're talking about makes perfect sense for smaller-scale accounts, but even some of the very large accounts where we're talking hundreds of millions a year in spend, you can set compromised keys up on GitHub, put them in Payspin, whatever, and then people start spinning up Bitcoin miners everywhere. Great. It takes a long time to materially move the needle on that level of spend; it gets lost in the background noise. I lose my mind when I wind up leaving a managed NAT gateway running and it cost me 70 bucks a month in my $5 a month test account. Yeah, but you realize you could basically buy an island and it gets lost in the AWS bill at some of the high watermarks for some of these larger accounts.“Oh, someone spun up a cluster that's going to cost $400,000 a year?” Yeah, do I need to re-explain to you what a data science team does? They light money on fire in return for questionable returns, as a general rule. You knew that when you hired them; leave them alone. Whereas someone in their developer account does this, yeah, you kind of want to flag that immediately.It always comes down to rules and context. But I'd love to have some templates ready to go of, “I'm a starving student, please alert me anytime it looks like I might possibly exceed the free tier,” or better yet, “Don't let me, and if I do, it's on you and you eat the cost.” Conversely, it's, “Yeah, this is a Netflix sub-account or whatnot. Maybe don't bother me for anything whatsoever because freedom and responsibility is how we roll.” I imagine that's what they do internally on a lot of their cloud costing stuff because freedom and responsibility is ingrained in their culture. It's great. It's the freedom from having to think about cloud bills and the responsibility for paying it, of the cloud bill.Tim: Yeah, we will get internally alerted if things are [laugh] up too long, and then we will actually get paged, and then our manager would get paged, [laugh] and it would go up the line. If you leave something that's running too expensive, too long. So, there is a system there for it.Corey: Oh, yeah. The internal AWS systems for employees are probably my least favorite AWS service, full stop. And I've seen things posted about it; I believe it's called Isengard, for spinning up internal accounts and the rest—there's a separate one, I think, called Conduit, but I digress—that you spin something up, and apparently if it doesn't wind up—I don't need you to comment on this because you worked there and confidentiality is super important, but to my understanding it's, great, it has a whole bunch of formalized stuff like that and it solves for a whole lot of nifty features that bias for the way that AWS focuses on accounts and how they've view security and the rest. And, “Oh, well, we couldn't possibly ship this to customers because it's not how they operate.” And that's great.My problem with this internal provisioning system is it isolates and insulates AWS employees from the real pain of working with multiple accounts as a customer. You don't have to deal with the provisioning process of Control Tower or whatnot; you have your own internal thing. Eat your own dog food, gargle your own champagne, whatever it takes to wind up getting exposure to the pain that hits customers and suddenly you'll see those things improve. I find that the best way to improve a product is to make the people building it live with the painful parts.Tim: I think it's interesting that the stance is, “Well, it's not how the customers operate, and we wouldn't want the customers to have to deal with this.” But at the same time, you have to open up, like, 100 accounts if you need more than a certain number of S3 buckets. So, they are very comfortable with burdening the customer with a lot of constraints, and they say, “Well, constraints drive innovation.” Certainly, this is a constraint that you could at least offer and let the customers innovate around that.Corey: And at least define who the customer is. Because yeah, “I'm a Netflix sub-account is one story,” “I'm a regulated bank,” is another story, and, “I'm a student in my dorm room, trying to learn how this whole cloud thing works,” is another story. From risk tolerance, from a data protection story, from a billing surprise story, from a, “I'm trying to learn what the hell this is, and all these other service offerings you keep talking to me about confuse the hell out of me; please streamline the experience.” There's a whole universe of options and opportunity that isn't being addressed here.Tim: Well, I will say it very simply like this: we're talking about a multi-trillion dollar company versus someone who, if their AWS bill is too high, they don't pay rent; maybe they don't eat; maybe they have other issues, they don't—medical bill doesn't get paid; child care doesn't get paid. And if you're going to tell me that this multi-trillion dollar company can't solve for that so that doesn't happen to that person and tells them, “Well, if you come in afterwards, after your bill gets there, maybe we can do something about it, but in the meantime, suffer through this.” That's not ethical. Full stop.Corey: There are a lot of things that AWS gets right, and I want to be clear that I'm not sitting here trying to cast blame and say that everything they're doing is terrible. I feel like every time I talk about billing in any depth, I have to throw this disclaimer in. Ninety to ninety-five percent of what they do is awesome. It's just the missing piece that is incredibly painful for customers, and that's what I spend most of my time focusing on. It should not be interpreted to think that I hate the company.I just want them to do better than they are, and what they're doing now is pretty decent in most respects. I just want to fix the painful parts. Tim, thank you for joining me for a third time here. I'm certain I'll have you back in the somewhat near future to talk about more aspects of this, but until then, where can people find you slash retain your services?Tim: Well, you can find me on Twitter at @elchefe. If you want to retain my services for which you would be very, very happy to have, you can go to duckbillgroup.com and fill out a little questionnaire, and I will magically appear after an exchange of goods and services.Corey: Make sure to reference Tim by name just so that we can make our sales team facepalm because they know what's coming next. Tim, thank you so much for your time; it's appreciated.Tim: Thank you so much, Corey. I loved it.Corey: Principal cloud economist here at The Duckbill Group, Tim Banks. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, wait at least eight hours—possibly as many as 48 to 72—and then leave a comment explaining what you didn't like.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

This week Brian speaks with Manjunath Venkatram, Founder and CEO of Thought Data ThoughtData helps enterprise companies to reduce downtime and poor performance of IT application services/infrastructure with unified IT monitoring solutions. ThoughtData's Enterprise360 software solution provides unified and actionable observability to detect and resolve network, application, and infrastructure performance issues. The unified Solution approach helps enterprise businesses to significantly reduce cost, tool clutter, maintenance, manual data correlation and mean time to respond (MTTR) to critical IT issues. For more information, visit https://thoughtdata.com/ If you have the next big idea, apply to the Expert Dojo Accelerator: www.expertdojo.com

Join Jared Kells (Senior Developer at Easy Agile), Angad Sethi (Developer at Easy Agile), Jess Belliveau (Staff Infrastructure Engineer at Apptio) and Jordan Simonovski (Observability Engineer at Atlassian). In this roundtable style conversation, we let the developers take over. Sharing their insights and experience working with observability, leaving you with some key takeaways and some quick wins. Links mentioned in the episode belowhttps://www.youtube.com/watch?v=HiDqrqa34Ls&t=279shttps://docs.honeycomb.io/getting-started/learning-about-observability/https://docs.datadoghq.com/monitors/service_level_objectives/

A série que conversa com vendors de observabilidade recebe, em seu quarto episódio, o Willian Valerio, que desde a faculdade sonhava em criar uma empresa com o nome de Apoena, que no tupi-guarani significa: aquele que enxerga longe.Anos depois, essa visão deu certo e hoje o Willian é co-fundador da Appoena, consultoria focada em observabilidade!Para o nosso convidado, monitoração é ser reativo a algo que você conhece, como métricas; e  observabilidade é lidar com aquilo que você não conhece! Além disso, ele comenta que a ferramenta que correlaciona sinais está, hoje em dia, ajudando muito mais os SRE do que os Devs e Ops. Entenda o porquê disso e muito mais no Kubicast #78!CONTATO - Para falar com o Willian, escreva para: E-mail: contato@appoena.ioTwitter: @_valeriowillianLinkedIn: valeriowillianO LINK comentado nesse programa é:Tweet da Sheimy: aqui As RECOMENDAÇÕES para a audiência são:Willian:1984 do George Orwell (livro)Entenda os fundamentos do seu ambiente! (Essa é uma dica matadora!)João: Space Jam (Está no HBO Max)LEMBRETE! Se você tem vontade de contribuir para a Comunidade Kubernetes, mas não sabe como começar, participe do Kubernetes Contributor Summit do #KCDBrasil em 17/1/22.  O Kubicast é uma produção da Getup, especialista em Kubernetes e apoiadora do projeto UnDistro, uma distribuição para gerenciar múltiplos clusters Kubernetes. #o11y #Observability #DevOps #Kubernetes #Containers #docker  

On this episode of Break I.T. Down, we break down Full-Stack Observability for non-tech listeners and techies alike.  We would like to thank Cisco Webex for sponsoring this episode.

About JasonJason is now the Managing Director at Redpoint Ventures.Links: GitHub: https://github.com/ @jasoncwarner: https://twitter.com/jasoncwarner GitHub: https://github.com/jasoncwarner Jasoncwarner/ama: https://github.com/jasoncwarner/ama TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Jason Warner, the Chief Technology Officer at GifHub, although he pronounces it differently. Jason, welcome to the show.Jason: Thanks, Corey. Good to be here.Corey: So, GitHub—as you insist on pronouncing it—is one of those companies that's been around for a long time. In fact, I went to a training conducted by one of your early folks, Scott Chacon, who taught how Git works over the course of a couple of days, and honestly, I left more confused than I did when I entered. It's like, “Oh, this is super awful. Good thing I'll never need to know this because I'm not really a developer.” And I'm still not really a developer and I still don't really know how Git works, but here we are.And it's now over a decade later; you folks have been acquired by Microsoft, and you are sort of the one-stop-shop, from the de facto perspective of, “I'm going to go share some code with people on the internet. I'll use GitHub to do it.” Because, you know, copying and pasting and emailing Microsoft Word documents around isn't ideal.Jason: That is right. And I think that a bunch of things that you mentioned there, played into, you know, GitHub's early and sustained success. But my God, do you remember the old days when people had to email tar files around or drop them in weird spots?Corey: What the hell do you mean, by, “Old days?” It still blows my mind that the Linux kernel is managed by—they use Git, obviously. Linus Torvalds did write Git once upon a time—and it has the user interface you would expect for that. And the way that they collaborate is not through GitHub or anything like that. No, they use Git to generate patches, which they then email to the mailing list. Which sounds like I'm making it up, like, “Oh, well, yeah, tell another one, but maybe involve a fax machine this time.” But no, that is actually what they do.Jason: It blew my mind when I saw that, too, by the way. And you realize, too, that workflows are workflows, and people will build interesting workflows to solve their use case. Now, obviously, anyone that you would be talking to in 2021, if you walked in and said, “Yeah, install Git. Let's set up an email server and start mailing patches to each other and we're going to do it this way.” They would just kind of politely—or maybe impolitely—show you out of the room, and rightfully [laugh] so. But it works for one of the most important software projects in history: Linux.Corey: Yeah, and it works almost in spite of itself to some extent. You've come a long way as a company because initially, it was, “Oh, there's this amazing, decentralized version control system. How do we make it better? I know, we're going to take off the decentralized part of it and give it a central point that everything can go through.” And collaboratively, it works well, but I think that viewing GitHub as a system that is used to sell free Git repositories to people is rather dramatically missing the point. It feels like it's grown significantly beyond just code repository hosting. Tell me more about that.Jason: Absolutely. I remember talking to a bunch of folks right around when I was joining GitHub, and you know, there was still talk about GitHub as, you know, GitHub for lawyers, or GitHub for doctors, or what could you do in a different way? And you know, social coding as an aspect, and maybe turning into a social network with a resume. And all those things are true to a percentage standpoint. But what GitHub should be in the world is the world's most important software development platform, end-to-end software development platform.We obviously have grown a bunch since me joining in that way which we launched dependency management packages, Actions with built-in CI, we've got some deployment mechanisms, we got advanced security underneath it, we've Codespaces in beta and alpha on top of it now. But if you think about GitHub as, join, share, and see other people's code, that's evolution one. If you see it as world's largest, maybe most developed software development platform, that's evolution two, and in my mind, its natural place where it should be, given what it has done already in the world, is become the world's most important software company. I don't mean the most profitable. I just mean the most important.Corey: I would agree. I had a blog post that went up somewhat recently about the future of cloud being Microsoft's to lose. And it's not because Azure is the best cloud platform out there, with respect, and I don't need you to argue the point. It is very clearly not. It is not like other clouds, but I can see a path to where it could become far better than it is.But if I'm out there and I'm just learning how to write code—because I make terrible life choices—and I go to a boot camp or I follow a tutorial online or I take a course somewhere, I'm going to be writing code probably using VS Code, the open-source editor that you folks launched after the acquisition. And it was pretty clear that Atom wasn't quite where the world was going. Great. Then I'm going to host it on GitHub, which is a natural evolution. Then you take a look at things like GitHub Actions that build in CI/CD pipelines natively.All that's missing is a ‘Deploy to Azure' button that is the next logical step, and you're mostly there for an awful lot of use cases. But you can't add that button until Azure itself gets better. Done right, this has the potential to leave, effectively, every other cloud provider in the dust because no one can touch this.Jason: One hundred percent. I mean, the obvious thing that any other cloud should be looking at with us—or should have been before the acquisition, looking at us was, “Oh, no, they could jump over us. They could stop our funnel.” And I used internal metrics when I was talking to them about partnership that led to the sale, which was I showed them more about their running business than they knew about themselves. I can tell them where they were stacked-ranked against each other, based on the ingress and egress of all the data on GitHub, you know, and various reactions to that in those meetings was pretty astounding.And just with that data alone, it should tell you what GitHub would be capable of and what Azure would be capable of in the combination of those two things. I mean, you did mention the ‘Deploy to Azure' button; this has been a topic, obviously, pre and post-acquisition, which is, “When is that coming?” And it was the one hard rule I set during the acquisition was, there will be no ‘Deploy to Azure' button. Azure has to earn the right to get things deployed to, in my opinion. And I think that goes to what you're saying is, if we put a ‘Deploy to Azure' button on top of this and Azure is not ready for that, or is going to fail, ultimately, that looks bad for all of us. But if it earned the right and it gets better, and it becomes one of those, then, you know, people will choose it, and that is, to me, what we're after.Corey: You have to choose the moment because if you do it too soon, you'll set the entire initiative back five years. Do it too late, and you get leapfrogged. There's a golden window somewhere and finding it is going to be hard. And I think it's pretty clear that the other hyperscalers in this space are learning, or have learned, that the next 10 years of cloud or 15 years of cloud or whatever they want to call it, and the new customers that are going to come are not the same as the customers that have built the first half of the business. And they're trying to wrap their heads around that because a lot of where the growth is going to come from is established blue chips that are used to thinking in very enterprise terms.And people think I'm making fun of them when I say this, but Microsoft has 40 years' experience apologizing to enterprises for computer failures. And that is fundamentally what cloud is. It's about talking computers to business executives because as much as we talk about builders, that is not the person at an established company with an existing IT estate, who gets to determine where $50 million a year in cloud-spend is going to go.Jason: It's [laugh] very, [laugh] very true. I mean, we've entered a different spot with cloud computing in the bell curve of adoption, and if you think that they will choose the best technology every time, well, history of computing is littered with better technologies that have failed because the distribution was better on one side. As you mentioned, Microsoft has 40 years, and I wager that Microsoft has the best sales organizations and the best enterprise accounts and, you know, all that sort of stuff, blah, blah, blah, on that side of the world than anyone in the industry. They can sell to enterprises better than almost anyone in the industry. And the other hyperscalers—there's a reason why [TK 00:08:34] is running Google Cloud right now. And Amazon, classically, has been very, very bad assigned to the enterprises. They just happened to be the first mover.Corey: In the early days, it was easy. You'd have an Amazon salesperson roll up to a company, and the exec would say, “Great, why should we consider running things on AWS?” And the answer was, “Oh, I'm sorry, wrong conversation. Right now you have 80 different accounts scattered throughout your org. I'm just here to help you unify them, get some visibility into it, and possibly give you a discount along the way.” And it was a different conversation. Shadow IT was the sole driver of cloud adoption for a long time. That is no longer true. It has to go in the front door, and that is a fundamental shift in how you go to market.Jason: One hundred percent true, and it's why I think that Microsoft has been so successful with Azure, in the last, let's call it five years in that, is that the early adopters in the second wave are doing that; they're all enterprise IT, enterprise dev shops who are buying from the top down. Now, there is still the bottoms-up adoption that going to be happening, and obviously, bottom-up adoption will happen still going forward, but we've entered the phase where that's not the primary or sole mechanism I should say. The sole mechanism of buying in. We have tops-down selling still—or now.Corey: When Microsoft announced it was acquiring GitHub, there was a universal reaction of, “Oh, shit.” Because it's Microsoft; of course they're going to ruin GitHub. Is there a second option? No, unless they find a way to ruin it twice. And none of it came to pass.It is uniformly excellent, and there's a strong argument that could be made by folks who are unaware of what happened—I'm one of them, so maybe I'm right, maybe I'm wrong—that GitHub had a positive effect on Microsoft more than Microsoft had an effect on GitHub. I don't know if that's true or not, but I could believe it based upon what I've seen.Jason: Obviously, the skepticism was well deserved at the time of acquisition, let's just be honest with it, particularly given what Microsoft's history had been for about 15—well, 20 years before, previous to Satya joining. And I was one of those people in the late '90s who would write ‘M$' in various forums. I was 18 or 19 years old, and just got into—Corey: Oh, hating Microsoft was my entire personality.Jason: [laugh]. And it was, honestly, well-deserved, right? Like, they had anti-competitive practices and they did some nefarious things. And you know, I talked about Bill Gates as an example. Bill Gates is, I mean, I don't actually know how old he is, but I'm going to guess he's late '50s, early '60s, but he's basically in the redemption phase of his life for his early years.And Microsoft is making up for Ballmer years, and later Gates years, and things of that nature. So, it was well-deserved skepticism, and particularly for a mid-career to older-career crowd who have really grown to hate Microsoft over that time. But what I would say is, obviously, it's different under Satya, and Scott, and Amy Hood, and people like that. And all we really telling people is give us a chance on this one. And I mean, all of us. The people who were running GitHub at the time, including myself and, you know, let Scott and Satya prove that they are who they say they are.Corey: It's one of those things where there's nothing you could have said that would have changed the opinion of the world. It was, just wait and see. And I think we have. It's now, I daresay, gotten to a point where Microsoft announces that they're acquiring some other beloved company, then people, I think, would extend a lot more credit than they did back then.Jason: I have to give Microsoft a ton of credit, too, on this one for the way in which they handled acquisitions, like us and others. And the reason why I think it's been so successful is also the reason why I think so many others die post-acquisition, which is that Microsoft has basically—I'll say this, and I know I won't get fired because it feels like it's true. Microsoft is essentially a PE holding company at this point. It is acquired a whole bunch of companies and lets them run independent. You know, we got LinkedIn, you got Minecraft, Xbox is its own division, but it's effectively its own company inside of it.Azure is run that way. GitHub's got a CEO still. I call it the archipelago model. Microsoft's the landmass underneath the water that binds them all, and finance, and HR, and a couple of other things, but for the most part, we manifest our own product roadmap still. We're not told what to go do. And I think that's why it's successful. If we're going to functionally integrate GitHub into Microsoft, it would have died very quickly.Corey: You clearly don't mix the streams. I mean, your gaming division writes a lot of interesting games and a lot of interesting gaming platforms. And, like, one of the most popularly played puzzle games in the world is a Microsoft property, and that is, of course, logging into a Microsoft account correctly. And I keep waiting for that to bleed into GitHub, but it doesn't. GitHub is a terrific SAML provider, it is stupidly easy to log in, it's great.And at some level, I wish that would bleed into other aspects, but you can't have everything. Tell me what it's like to go through an acquisition from a C-level position. Because having been through an acquisition before, the process looks a lot like a surprise all-hands meeting one day after the markets close and, “Listen up, idiots.” And [laugh] there we go. I have to imagine with someone in your position, it's a slightly different experience.Jason: It's definitely very different for all C-levels. And then myself in particular, as the primary driver of the acquisition, obviously, I had very privy inside knowledge. And so, from my position, I knew what was happening the entire time as the primary driver from the inside. But even so, it's still disconcerting to a degree because, in many ways, you don't think you're going to be able to pull it off. Like, you know, I remember the months, and the nights, and the weekends, and the weekend nights, and all the weeks I spent on the road trying to get all the puzzle pieces lined up for the Googles, or the Microsofts, or the eventually AWSs, the VMwares, the IBMs of the world to take seriously, just from a product perspective, which I knew would lead to, obviously, acquisition conversations.And then, once you get the call from the board that says, “It's done. We signed the letter of intent,” you basically are like, “Oh. Oh, crap. Okay, hang on a second. I actually didn't—I don't actually believe in my heart of hearts that I was going to actually be able to pull that off.” And so now, you probably didn't plan out—or at least I didn't. I was like, “Shit if we actually pulled this off what comes next?” And I didn't have that what comes next, which is odd for me. I usually have some sort of a loose plan in place. I just didn't. I wasn't really ready for that.Corey: It's got to be a weird discussion, too, when you start looking at shopping a company around to be sold, especially one at the scale of GitHub because you're at such a high level of visibility in the entire environment, where—it's the idea of would anyone even want to buy us? And then, duh, of course they would. And you look the hyperscalers, for example. You have, well, you could sell it to Amazon and they could pull another Cloud9, where they shove it behind the IAM login process, fail to update the thing meaningfully over a period of years, to a point where even now, a significant portion of the audience listening to this is going to wonder if it's a service I just made up; it sounds like something they might have done, but Cloud9 sounds way too inspired for an AWS service name, so maybe not. And—which it is real. You could go sell to Google, which is going to be awesome until some executive changes roles, and then it's going to be deprecated in short order.Or then there's Microsoft, which is the wild card. It's, well, it's Microsoft. I mean, people aren't really excited about it, but okay. And I don't think that's true anymore at all. And maybe I'm not being fair to all the hyperscalers there. I mean, I'm basically insulting everyone, which is kind of my shtick, but it really does seem that Microsoft was far and away the best acquirer possible because it has been transformative. My question—if you can answer it—is, how the hell did you see that beforehand? It's only obvious—even knowing what I know now—in hindsight.Jason: So, Microsoft was a target for me going into it, and the reason why was I thought that they were in the best overall position. There was enough humility on one side, enough hubris on another, enough market awareness, probably, organizational awareness to, kind of, pull it off. There's too much hubris on one side of the fence with some of the other acquirers, and they would try to hug us too deeply, or integrate us too quickly, or things of that nature. And I think it just takes a deep understanding of who the players are and who the egos involved are. And I think egos has actually played more into acquisitions than people will ever admit.What I saw was, based on the initial partnership conversations, we were developing something that we never launched before GitHub Actions called GitHub Launch. The primary reason we were building that was GitHub launches a five, six-year journey, and it's got many, many different phases, which will keep launching over the next couple of years. The first one we never brought to market was a partnership between all of the clouds. And it served a specific purpose. One, it allowed me to get into the room with the highest level executive at every one of those companies.Two allow me to have a deep economic conversation with them at a partnership level. And three, it allowed me to show those executives that we knew what GitHub's value was in the world, and really flip the tables around and say, “We know what we're worth. We know what our value is in the world. We know where we sit from a product influence perspective. If you want to be part of this, we'll allow it.” Not, “Please come work with us.” It was more of a, “We'll allow you to be part of this conversation.”And I wanted to see how people reacted to that. You know how Amazon reacted that told me a lot about how they view the world, and how Google reacted to that showed me exactly where they viewed it. And I remember walking out of the Google conversation, feeling a very specific way based upon the reaction. And you know, when I talked to Microsoft, got a very different feel and it, kind of, confirmed a couple of things. And then when I had my very first conversation with Nat, who have known for a while before that, I realized, like, yep, okay, this is the one. Drive hard at this.Corey: If you could do it all again, would you change anything meaningful about how you approached it?Jason: You know, I think I got very lucky doing a couple of things. I was very intentional aspects of—you know, I tried to serendipitously show up, where Diane Greene was at one point, or a serendipitously show up where Satya or Scott Guthrie was, and obviously, that was all intentional. But I never sold a company like this before. The partnership and the product that we were building was obviously very intentional. I think if I were to go through the sale, again, I would probably have tried to orchestrate at least one more year independent.And it's not—for no other reason alone than what we were building was very special. And the world sees it now, but I wish that the people who built it inside GitHub got full credit for it. And I think that part of that credit gets diffused to saying, “Microsoft fixed GitHub,” and I want the people inside GitHub to have gotten a lot more of that credit. Microsoft obviously made us much better, but that was not specific to Microsoft because we're run independent; it was bringing Nat in and helping us that got a lot of that stuff done. Nat did a great job at those things. But a lot of that was already in play with some incredible engineers, product people, and in particular our sales team and finance team inside of GitHub already.Corey: When you take a look across the landscape of the fact that GitHub has become for a certain subset of relatively sad types of which I'm definitely one a household name, what do you think the biggest misconception about the company is?Jason: I still think the biggest misconception of us is that we're a code host. Every time I talk to the RedMonk folks, they get what we're building and what we're trying to be in the world, but people still think of us as SourceForge-plus-plus in many ways. And obviously, that may have been our past, but that's definitely not where we are now and, for certain, obviously, not our future. So, I think that's one. I do think that people still, to this day, think of GitLab as one of our main competitors, and I never have ever saw GitLab as a competitor.I think it just has an unfortunate naming convention, as well as, you know, PRs, and MRs, and Git and all that sort of stuff. But we take very different views of the world in how we're approaching things. And then maybe the last thing would be that what we're doing at the scale that we're doing it as is kind of easy. When I think that—you know, when you're serving almost every developer in the world at this point at the scale at which we're doing it, we've got some scale issues that people just probably will never thankfully encounter for themselves.Corey: Well, everyone on Hacker News believes that they will, as soon as they put up their hello world blog, so Kubernetes is the only way to do anything now. So, I'm told.Jason: It's quite interesting because I think that everything breaks at scale, as we all know about from the [hyperclouds 00:20:54]. As we've learned, things are breaking every day. And I think that when you get advice, either operational, technical, or managerial advice from people who are running 10 person, 50 person companies, or X-size sophisticated systems, it doesn't apply. But for whatever reason, I don't know why, but people feel inclined to give that feedback to engineers at GitHub directly, saying, “If you just…” and in many [laugh] ways, you're just like, “Well, I think that we'll have that conversation at some point, you know, but we got a 100-plus-million repos and 65 million developers using us on a daily basis.” It's a very different world.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense.Corey: One of the things that I really appreciate personally because, you know, when you see something that company does, it's nice to just thank people from time to time, so I'm inviting the entire company on the podcast one by one, at some point, to wind up thanking them all individually for it, but Codespaces is one of those things that I think is transformative for me. Back in the before times, and ideally the after times, whenever I travel the only computer I brought with me for a few years now has been an iPad or an iPad Pro. And trying to get an editor on that thing that works reasonably well has been like pulling teeth, my default answer has just been to remote into an EC2 instance and use vim like I have for the last 20 years. But Code is really winning me over. Having to play with code-server and other things like that for a while was obnoxious, fraught, and difficult.And finally, we got to a point where Codespaces was launched, and oh, it works on an iPad. This is actually really slick. I like this. And it was the thing that I was looking for but was trying to have to monkey patch together myself from components. And that's transformative.It feels like we're going back in many ways—at least in my model—to the days of thin clients where all the heavy lifting was done centrally on big computers, and the things that sat on people's desks were mostly just, effectively, relatively simple keyboard, mouse, screen. Things go back and forth and I'm sure we'll have super powerful things in our pockets again soon, but I like the interaction model; it solves for an awful lot of problems and that's one of the things that, at least from my perspective, that the world may not have fully wrapped it head around yet.Jason: Great observation. Before the acquisition, we were experimenting with a couple of different editors, that we wanted to do online editors. And same thing; we were experimenting with some Action CI stuff, and it just didn't make sense for us to build it; it would have been too hard, there have been too many moving parts, and then post-acquisition, we really love what the VS Code team was building over there, and you could see it; it was just going to work. And we had this one person, well, not one person. There was a bunch of people inside of GitHub that do this, but this one person at the highest level who's just obsessed with make this work on my iPad.He's the head of product design, his name's Max, he's an ex-Heroku person as well, and he was just obsessed with it. And he said, “If it works on my iPad, it's got a chance to succeed. If it doesn't work on my iPad, I'm never going to use this thing.” And the first time we booted up Codespaces—or he booted it up on the weekend, working on it. Came back and just, “Yep. This is going to be the one. Now, we got to work on those, the sanding the stones and those fine edges and stuff.”But it really does unlock a lot for us because, you know, again, we want to become the software developer platform for everyone in the world, you got to go end-to-end, and you got to have an opinion on certain things, and you got to enable certain functionality. You mentioned Cloud9 before with Amazon. It was one of the most confounding acquisitions I've ever seen. When they bought it I was at Heroku and I thought, I thought at that moment that Amazon was going to own the next 50 years of development because I thought they saw the same thing a lot of us at Heroku saw, and with the Cloud9 acquisition, what they were going to do was just going to stomp on all of us in the space. And then when it didn't happen, we just thought maybe, you know, okay, maybe something else changed. Maybe we were wrong about that assumption, too. But I think that we're on to it still. I think that it just has to do with the way you approach it and, you know, how you design it.Corey: Sorry, you just said something that took me aback for a second. Wait, you mean software can be designed? It's not this emergent property of people building thing on top of thing? There's actually a grand plan behind all these things? I've only half kidding, on some level, where if you take a look at any modern software product that is deployed into the world, it seems impossible for even small aspects of it to have been part of the initial founding design. But as a counterargument, it would almost have to be for a lot of these things. How do you square that circle?Jason: I think you have to, just like anything on spectrums and timelines, you have to flex at various times for various things. So, if you think about it from a very, very simple construct of time, you just have to think of time horizons. So, I have an opinion about what GitHub should look like in 10 years—vaguely—in five years much more firmly, and then very, very concretely, for the next year, as an example. So, a lot of the features you might see might be more emergent, but a lot of long-term work togetherness has to be loosely tied together with some string. Now, that string will be tightened over time, but it loosely has to see its way through.And the way I describe this to folks is that you don't wake up one day and say, “I'm going on vacation,” and literally just throw a finger on the map. You have to have some sort of vague idea, like, “Hey, I want to have a beach vacation,” or, “I want to have an adventure vacation.” And then you can kind of pick a destination and say, “I'm going to Hawaii,” or, “I'm going to San Diego.” And if you're standing on the East Coast knowing you're going to San Diego, you basically know that you have to just start marching west, or driving west, or whatever. And now, you don't have to have the route mapped out just yet, but you know that hey, if I'm going due southeast, I'm off course, so how do I reorient to make sure I'm still going in the right direction?That's basically what I think about as high-level, as scale design. And it's not unfair to say that a lot of the stuff is not designed today. Amazon is very famous for not designing anything; they design a singular service. But there's no cohesiveness to what Amazon—or AWS specifically, I should say, in this case—has put out there. And maybe that's not what their strategy is. I don't know the internal workings of them, but it's very clear.Corey: Well, oh, yeah. When I first started working in the AWS space and looking through the console, it like, “What is this? It feels like every service's interface was designed by a different team, but that would—oh…” and then the light bulb went on. Yeah. You ship your culture.Jason: It's exactly it. It works for them, but I think if you're going to try to do something very, very, very different, you know, it's going to look a certain way. So, intentional design, I think, is part of what makes GitHub and other products like it special. And if you think about it, you have to have an end-to-end view, and then you can build verticals up and down inside of that. But it has to work on the horizontal, still.And then if you hire really smart people to build the verticals, you get those done. So, a good example of this is that I have a very strong opinion about the horizontal workflow nature of GitHub should look like in five years. I have a very loose opinion about what the matrix build system of Actions looks like. Because we have very, very smart people who are working on that specific problem, so long as that maps back and snaps into the horizontal workflows. And that's how it can work together.Corey: So, when you look at someone who is, I don't know, the CTO of a wildly renowned company that is basically still catering primarily to developers slash engineers, but let's be honest, geeks, it's natural to think that, oh, they must be the alpha geek. That doesn't really apply to you from everything I've been able to uncover. Am I just not digging deeply enough, or are you in fact, a terrible nerd?Jason: [laugh]. I am. I'm a terrible nerd. I am a very terrible nerd. I feel very lucky, obviously, to be in the position I'm in right now, in many ways, and people call me up and exactly that.It's like, “Hey, you must be king of the geeks.” And I'm like, “[laugh], ah, funny story here.” But um, you know, I joke that I'm not actually supposed to be in tech in first place, the way I grew up, and where I did, and how, I wasn't supposed to be here. And so, it's serendipitous that I am in tech. And then turns out I had an aptitude for distributed systems, and complex, you know, human systems as well. But when people dig in and they start talking about topics, I'm confounded. I never liked Star Wars, I never like Star Trek. Never got an anime, board games, I don't play video games—Corey: You are going to get letters.Jason: [laugh]. When I was at Canonical, oh, my goodness, the stuff I tried to hide about myself, and, like, learn, like, so who's this Boba Fett dude. And, you know, at some point, obviously, you don't have to pretend anymore, but you know, people still assume a bunch stuff because, quote, “Nerd” quote, “Geek” culture type of stuff. But you know, some interesting facts that people end up being surprised by with me is that, you know, I was very short in high school and I grew in college, so I decided that I wanted to take advantage of my newfound height and athleticism as you grow into your body. So, I started playing basketball, but I obsessed over it.I love getting good at something. So, I'd wake up at four o'clock in the morning, and go shoot baskets, and do drills for hours. Well, I got really good at it one point, and I end up playing in a Pro-Am basketball game with ex-NBA Harlem Globetrotter legends. And that's just not something you hear about in most engineering circles. You might expect that out of a salesperson or a marketing person who played pro ball—or amateur ball somewhere, or college ball or something like that. But not someone who ends up running the most important software company—from a technical perspective—in the world.Corey: It's weird. People counterintuitively think that, on some level, that code is the answer to all things. And that, oh, all this human interaction stuff, all the discussions, all the systems thinking, you have to fit a certain profile to do that, and anyone outside of that is, eh, they're not as valuable. They can get ignored. And we see that manifesting itself in different ways.And even if we take a look at people whose profess otherwise, we take a look at folks who are fresh out of a boot camp and don't understand much about the business world yet; they have transformed their lives—maybe they're fresh out of college, maybe didn't even go to college—and 18 weeks later, they are signing up for six-figure jobs. Meanwhile, you take a look at virtually any other business function, in order to have a relatively comparable degree of earning potential, it takes years of experience and being very focused on a whole bunch of other things. There's a massive distortion around technical roles, and that's a strange and difficult thing to wrap my head around. But as you're talking about it, it goes both ways, too. It's the idea of, “Oh, I'll become technical than branch into other things.” It sounded like you started off instead with a non-technical direction and then sort of adopted that from other sides. Is that right, or am I misremembering exactly how the story unfolds?Jason: No, that's about right. People say, “Hey, when did I start programming?” And it's very in vogue, I think, for a lot of people to say, “I started programming at three years old,” or five years old, or whatever, and got my first computer. I literally didn't get my first computer until I was 18-years-old. And I started programming when I got to a high school co-op with IBM at 17.It was Lotus Notes programming at the time. Had no exposure to it before. What I did, though, in college was IBM told me at the time, they said, “If you get a computer science degree will guarantee you a job.” Which for a kid who grew up the way I grew up, that is manna from heaven type of deal. Like, “You'll guarantee me a job inside where don't have to dig ditches all day or lay asphalt? Oh, my goodness. What's computer science? I'll go figure it out.”And when I got to school, what I realized was I was really far behind. Everyone was that ubergeek type of thing. So, what I did is I tried to hack the system, and what I said was, “What is a topic that nobody else has an advantage on from me?” And so I basically picked the internet because the internet was so new in the mid-'90s that most people were still not fully up to speed on it. And then the underpinnings in the internet, which basically become distributed systems, that's where I started to focus.And because no one had a real advantage, I just, you know, could catch up pretty quickly. But once I got into computers, it turned out that I was probably a very average developer, maybe even below average, but it was the system's thinking that I stood out on. And you know, large-scale distributed systems or architectures were very good for me. And then, you know, that applies not, like, directly, but it applies decently well to human systems. It's just, you know, different types of inputs and outputs. But if you think about organizations at scale, they're barely just really, really, really complex and kind of irrational distributed systems.Corey: Jason, thank you so much for taking the time to speak with me today. If people want to learn more about who you are, what you're up to, how you think about the world, where can they find you?Jason: Twitter's probably the best place at this point. Just @jasoncwarner on Twitter. I'm very unimaginative. My name is my GitHub handle. It's my Twitter username. And that's the best place that I, kind of, interact with folks these days. I do an AMA on GitHub. So, if you ever want to ask me anything, just kind of go to jasoncwarner/ama on GitHub and drop a question in one of the issues and I'll get to answering that. Yeah, those are the best spots.Corey: And we will, of course, include links to those things in the [show notes 00:33:52]. Thank you so much for taking the time to speak with me today. I really appreciate it.Jason: Thanks, Corey. It's been fun.Corey: Jason Warner, Chief Technology Officer at GitHub. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review in your podcast platform of choice anyway, along with a comment that includes a patch.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About LaurenLauren Hasson is the Founder of DevelopHer, an award-winning career development platform that has empowered thousands of women in tech to get ahead, stand out, and earn more in their careers. She also works full-time on the frontlines of tech herself. By day, she is an accomplished software engineer at a leading Silicon Valley payments company where she is the architect of their voice payment system and messaging capabilities and is chiefly responsible for all of application security.Through DevelopHer, she's partnered with top tech companies like Google, Dell, Intuit, Armor, and more and has worked with top universities including Indiana and Tufts to bridge the gender gap in leadership, opportunity, and pay in tech for good. Additionally, she was invited to the United Nations to collaborate on the global EQUALS initiative to bridge the global gender divide in technology. Sought after across the globe for her insight and passionate voice, Lauren has started a movement that inspires women around the world to seek an understanding of their true value and to learn and continually grow.  Her work has been featured by industry-leading publications like IEEE Women in Engineering Magazine and Thrive Global and her ground-breaking platform has been recognized with fourteen prestigious awards for entrepreneurship, product innovation, diversity and leadership including the Women in IT Awards Silicon Valley Diversity Initiative of the Year Award, three Female Executive of the Year Awards, and recognition as a Finalist for the United Nations WSIS Stakeholder Prize.Links: DevelopHer: https://developher.com The DevelopHer Playbook: https://www.amazon.com/DevelopHer-Playbook-Simple-Advocate-Yourself-ebook/dp/B08SQM4P5J TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You could build you go ahead and build your own coding and mapping notification system, but it takes time, and it sucks! Alternately, consider Courier, who is sponsoring this episode. They make it easy. You can call a single send API for all of your notifications and channels. You can control the complexity around routing, retries, and deliverability and simplify your notification sequences with automation rules. Visit courier.com today and get started for free. If you wind up talking to them, tell them I sent you and watch them wince—because everyone does when you bring up my name. Thats the glorious part of being me. Once again, you could build your own notification system but why on god's flat earth would you do that?Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring. Corey: You could build you go ahead and build your own coding and mapping notification system, but it takes time, and it sucks! Alternately, consider Courier, who is sponsoring this episode. They make it easy. You can call a single send API for all of your notifications and channels. You can control the complexity around routing, retries, and deliverability and simplify your notification sequences with automation rules. Visit courier.com today and get started for free. If you wind up talking to them, tell them I sent you and watch them wince—because everyone does when you bring up my name. Thats the glorious part of being me. Once again, you could build your own notification system but why on god's flat earth would you do that?Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A somewhat recurring theme of this show has been the business of cloud, and that touches on a lot of different things. One thing I've generally cognizant of not doing is talking to folks who don't look like me and asking them questions like, “Oh, that's great, but let's ignore everything that you're doing, and instead talk about what it's like not to be a cis-gendered white dude in tech,” because that's crappy. Today, we're sort of deviating from that because my guest is Lauren Hasson, the founder of DevelopHer, which is a career development platform that empowers women in tech to get ahead. Lauren, thanks for joining me.Lauren: Thanks so much for having me, Corey.Corey: So, you're the founder of DevelopHer, and that is ‘develop-her' as in ‘she'. I'm not going to be as distinct on that pronunciation, so if you think I'm saying ‘developer' and it doesn't make intellectual sense, listener, that's what's going on. But you're also a speaker, you're an author, and you work on the front lines of tech yourself. That's a lot of stuff. What's your story?Lauren: Yeah, I do. So, I'm not only the founder-developer, but I'm just like many of your listeners: I work on the front lines of tech myself. I work remotely from my home in Dallas for a Silicon Valley payments company, where I'm the architect of our voice payment system, and I up until recently was chiefly responsible for all of application security. Yeah, and I do keep busy.Corey: It certainly seems like it. Let's go back to, I guess, the headline item here. You are the founder of DevelopHer, and one thing that always drives me a little nutty is when people take a glance at what I do and then try and tell the story, and then effectively mess the whole thing up. What is DevelopHer?Lauren: So, DevelopHer is what I wish I had ten years ago—or actually nine years ago. It's an empowerment platform that helps individual women—men, too—get ahead in their careers, earn more, and stand out. And part of my story, you know, I have the degrees from undergrad in electrical engineering and computer science, but I went a completely different direction after graduating. And at the end of the Great Recession, I found myself with no job with no technical skills, and I mean, no job prospects, at all. It was really, really bad, ugly crying on my couch bad, Corey.And I took a number of steps to get ahead and really relearn my tech skills, and I only got one offer to give myself a chance. It was a 90-days to prove myself, to get ahead, and teach myself iOS. And I remember it was one of the most terrifying things I've ever done. And within two years, I not only managed to survive that 90-day period and keep that job, but I had completely managed to thrive. My work had been featured in Apple's iOS7 keynote, I'd won the company-wide award at a national agency four times, I had won the SXSW international Hackathon, twice in a row.And then probably the pinnacle of it all is I was one of 100 tech innovators worldwide invited to attend the [UKG 00:03:41] Innovation Conference. And they flew me there on a private 747 jet, and it was just unreal. And so I founded DevelopHer because I needed this ten years ago, when I was at rock bottom, to figure out how to get ahead: how do I get into my career; how do I stand out? And of course, you know there's more to the story, but I also found out I was underpaid after achieving all of that, that a male peer was paid exactly what I was paid, with no credentials, despite all of the awards that I won. And I went out and learned to negotiate, and tripled my salary in two years, and turned around and said, “I'm going to teach other women—and men, too—how to get real change in their own life.”Corey: I love hearing stories where people discover that they're underpaid. I mean, it's a bittersweet moment because on the one hand it's, “Wait, you mean they've been taking advantage of me?” And you feel bad for people, but at the same time, you're sort of watching the blindfold fall away from their eyes of, “Yeah, but it's been this way, and now you know about it. And now you're in a position to potentially do something about it.” I gave a talk at a tech conference a few years back called “Weasel your Way to the Top: How to Handle a Job Interview” and it was a fun talk.I really enjoyed it, but what I discovered was after I'd given it I got some very direct feedback of, “That's a great talk and you give a lot of really useful advice. What if I don't look like you?” And I realized, “Oh, my God, I built this out of things that worked for me and I unconsciously built all of my own biases and all of my own privilege into that talk.” At which point I immediately stopped giving it until I could relaunch it as a separate talk with a friend of mine, Sonia Gupta, who does not look like me. And between the two of us, it became a much stronger, much better talk.Lauren: It's good that you understand what you were bringing to the table and how you can appeal to an even larger audience. And what I've done is really said, “Here's my experience as a woman in tech, and here's what's worked for me.” And what's been surprising is men have said, “Yeah, that's what I did.” Except for I put a woman in tech spin on it and… I mean, I knew it worked for me; I have more than quintupled my base salary—just my base salary alone—in nine years. And the results that women are getting from my programming—I had one woman who earned $80,000 more in a single negotiation, which tells me, one, she was really underpaid, but she didn't just get one offer at $80,000 more; she got at least two. I mean, that changed her life.And I think the lowest I've heard is, like, $30,000 difference change. I mean, this is, this is life-changing for a lot of women. And the scary thing is that it's not just, say it's $50,000 a year. Well, over ten years, that's half a million dollars. Over 20 years, that's a million, and that's not even interest and inflation and compounding going into that. So, that's a huge difference.Corey: It absolutely is. It's one of those things that continues to set people further and further back. One thing that I think California got very right is they've outlawed recently asking what someone's previous compensation was because, “Oh, we don't want to give someone too big of a raise,” is a way you perpetuate the systemic inequality. And that's something that I wish more employers would do.Lauren: It's huge. I know the women and proponents who had moved that forward; some of them are personal friends of mine, and it's huge. And that's actually something that I trained specifically for is how to handle difficult questions like, “How much are you currently making?” Which you can't legally get asked in California, although it still happens, so how do you handle it if you still get asked and you don't want to rule yourself out? Or even worse—which they still can ask—which is, “How much do you want to make?”And a lot of times, people get asked that before they know anything about the job. And they basically, if you give an answer upfront, you're negotiating against yourself. And so I tackle tough things like that head-on. And I'm very much an engineer at heart, so for me, it's very methodical; I prepare scripts in advance to handle the pushback that I'm going to get, to handle the difficult questions. Without a doubt, I know all of my numbers, and that's where I'm getting real results for women is by taking the methodical approach to it.Corey: So, I spent my 20s in crippling credit card debt, and I was extremely mercenary, as a result. This wasn't because of some grand lost vision or something. Nope. I had terrible financial habits. So, every decision I made in that period of my life was extraordinarily mercenary. I would leave jobs I enjoyed for a job I couldn't stand because it paid $10,000 more.And the thing that I picked up from all of this, especially now having been on the other side of that running a company myself, is I'm not suggesting at any point that people should make career decisions based upon where they can make the most money, but that should factor in. One thing we do here at The Duckbill Group, in every job posting we put up is we post the salary range for the position. And I want to be clear here, it is less than anyone here could make at one of the big tech unicorns or a very hot startup that's growing meteorically, and we're upfront about that. We know that if money is the thing you're after and that is the driving force behind what you're going for, great; I don't fault you for that.This might not be the best role for you and that's perfectly okay. I get it. But you absolutely should know what your market worth is so you can make that decision from a place of being informed, rather than being naive and later discovering that you were taken advantage of.Lauren: So, I want to unpack just a couple things. There's just so many gold nuggets in that. Number one, for any employer listening out there, that is such a great best practice, to post the range. You're going to attract the right candidates when you post the right range. The last thing you want is to get to the end of the process to find out that, hey, you guys were totally off, and all the time invested could have been avoided if you'd had some sort of expectation set, upfront.That said, that's actually where I start with my negotiation training. A lot of people think I start with the money and that it's all about the money. That's not where I start. The very first thing I train women, and the men who've taken it, too, on the course is, figure out what success looks like to you. And not just the number success, but what does your life look like? What does your lifestyle look like? What does it feel like? What kinds of things do you do? What kinds of things do you value?Money is one of those components, but it's not all. And here's the reason I did that: because at a certain point in my life, I only got out at—broke even out of debt, you know, within the last five years. That's how underpaid I was at the time. But then once I started climbing out of debt, I started realizing it's not all about money. And that's actually how I ended up in my dream position.I mean, I'm living out how I define success today. Could I be making a lot more money at a big tech unicorn? Yeah, I could. But I also have this incredible lifestyle; it's sustainable. I get on apps like Blind and other internet forums, and I hear just horror stories of people burning out and the toxic cultures they work with. I don't have that at all. I have something that I could easily do for the next 50 years of my life if I live that long.But it's not by accident that I'm in the role that I'm in right now. I actually took the time to figure out what success looks like to me, and so when this opportunity came along—and I was looking at it alongside other opportunities that honestly paid more, I recognized this opportunity for what it was because I'd put in the work up front to figure out what success looks like to me. And so that's why what you guys are saying, “Hey, it's a lifestyle that you guys are supporting and mission that you're joining that's so important.” And you need to know that and do that work up front.Corey: That's I think what it really comes down to is understanding that in many cases… in fact, I'm going to take that back—in all cases, there's an inherent adversarial nature to the discussions you have about compensation with your employer or your prospective employer. And I say ‘adversarial' not antagonistic because you are misaligned as far as the ultimate purpose of the conversation. I'm not going to paint myself as some saint here and say that, oh, I'm on the side of every person I'm negotiating against, trying to get them to take a salary that's less than they deserve. Because, first, although I view myself that I'm not in that position, you have to take that on faith from me, and I think that is too far of a bridge to cross. So, take even what I'm saying now from the position as someone who has a vested interest in the outcomes of that negotiation.I mean, we're not one of those unicorn startups; we can't outbid Netflix and we wouldn't even try to. We're one of those old-fashioned businesses that has taken no investment and we fund ourselves through the magic of revenue and profitability, which means we don't have a SoftBank-sized [laugh] war chest sitting in the bank that we can use to just hurl ridiculous money at people and see who pans out. Hiring has to be intentional and thoughtful because we're a very small team. And if you're looking for something that doesn't align with that, great; I certainly don't blame you. That isn't this, and that's okay, I'm not trying to hire everyone.And if it's not going to work out, why wouldn't we say that upfront to avoid trying to get to all the way at the end of a very expensive interview process—both in terms of time and investment and emotionally—only to figure out that we're worlds apart on comp, and it's never going to work.Lauren: A hundred percent agree. I mean, I've been through it on both ends, both as someone who is being hired and also as a hiring manager, and I understand it. And you need to find alignment, and that's what negotiation is all about is finding an alignment, finding something where everyone feels like they're winning in the situation. And I'm a big proponent—and this is going to go so counterculture—I think a lot of people overlook a lot of opportunities that are just golden nuggets. I think there's a lot of idol worship of the big tech companies.And don't get me wrong; I'm sure they pay really well, great opportunity for your career, but I think people are overlooking a lot of really great career opportunities to get experience, and responsibility, and have good pay and lifestyle. And I'm a big proponent and looking for those golden nuggets rather than shooting for one of the big tech unicorns.Corey: And other people are going to have a very different perspective on that, and that is absolutely okay. So, tell me a little bit more about what it is that DevelopHer does and how you go about doing it because it's one thing to say, “Oh, we help women figure out that they are being underpaid,” but there's a whole lot of questions that opens up because great. How do you do that?Lauren: I do a number of things. So, it's not all about pay either. Part of it's building your value, building your confidence, standing out, getting ahead. DevelopHer started, actually, as a podcast. Funny story; I wanted to solve the problem of, we need more technical women as visible leaders out there, and I said, “Where are the architects? Where are the CTOs? Where are the CSOs?”And I didn't think anyone would care about me. I mean, I'm not Sheryl Sandberg; I'm not [laugh] the CEO of Facebook. Who's going to listen to me? And then I was actually surprised when people cared about my own story, about coming back from being underpaid and then getting back into tech and figuring out how to stand out in such a short amount of time. And other women were saying, “Well, how did you do it?”And it wasn't just women; it was men, too, saying, “Hey, I also don't know how to effectively advocate for myself.” And then it was companies saying, “Hey, can you come in and help us build our internal bench, recruit more women to come work for us, and build our own women leaders?” And then I've started working with universities to help bridge the gap before it even starts. I partnered with major universities to license my program and train them, not only how do you negotiate for what you're worth, for your first salary, but also how do you come in and immediately make an impact and accelerate your career growth? And then, of course, I work with individual women.I've talked about I have a salary negotiation course that's won a couple awards for the work, the results that it's getting, but then I just recently wrote a book because I wanted to reach women and men at scale and help them really get ahead. And this was literally my playbook. It's called The DevelopHer Playbook. And it's, how did I break into tech? And then once I was in tech, how did I get ahead so quickly? And it's not rocket science. And that's what I'm working on is training other people do it. And look, I'm still learning; I'm still paving my own path forward in tech, myself.Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part.Corey: I feel like no one really has a great plan for, “Oh, where are you going next in tech? Do you have this whole thing charted out?” “Of course not. I'm doing this fly by night, seat of my pants, if I'm being perfectly honest with you.” And it's hard to know where to go next.What's interesting to me is that you talk about helping people individually—generally women—through your program, but you also work directly with companies. And when you're talking about things like salary negotiation, I think a natural question that flows from that is, are there aspects of what you wind up talking to individuals about versus what you do when talking to companies that are in opposition to each other?Lauren: Yeah, so that's a great question. So, the answer is there are some progressive companies that have brought me in to do salary negotiation training. Complete candor, most companies aren't interested. It's my Zero-To-Hero DevelopHer Playbook program which is, how do you get ahead? How do you build your value, become an asset at the company?So, it's less focused on pay, but more how do you become more valuable, and get ahead and add more value to the company? And that's where I work with the individuals and the companies on that front.Corey: It does seem like it would be a difficult sell, in most enterprise scenarios, to get a company to pay someone to come in to teach their staff how to more effectively [laugh] negotiate their next raise. I love the vision.Lauren: It has happened. I also thought it was crazy, but it has happened. But no, most of my corporate clients say, “We not only want to encourage more women into tech, but we already have a lot of women who are already in our ranks, and we want to encourage them to really feel like they're empowered and to stand out and reach the next levels.” And that's my sweet spot for corporate.Corey: Somewhat recently, I was asked on a Twitter Spaces—which is like Clubhouse but somehow different and strange—did I think that the privilege that I brought to what I do had enabled me to do these things, being white, being a man, being cis-gendered—speaking English as my primary language was an interesting one that I hadn't heard contextualized like that before—and whether that had advantaged me as I went through these things? And I think it's impossible to say anything other than absolutely because it's easy to, on some level, take a step back and think, “Well, I've built this company, and this media platform, and the rest. And that wasn't given to me; I had to build it.” And that's absolutely true. I did have to build it, and it wasn't given to me.But as I was building it, the winds were at my back not against me. I was not surrounded by people who are telling me I couldn't do it. Every misstep I made wasn't questioned as, well, you sure you should be doing this thing that you're not really doing? It was very much a fail-forward. And if you think that applies to everyone, then you are grievously mistaken.Lauren: I think that's a healthy perspective, which is why I consider you one of developers in my strongest allies, the fact that you're willing to look at yourself and go, “What advantages did I have? And how might I need to adapt my messaging or my advice so that it's applicable to even more people?” But it's also something I've experienced myself. I mean, I set out to help women in tech because I'm in women in tech myself. And I was surprised by a couple of things.Number one, I was surprised that men were [laugh] asking me for advice as well. And individuals and medicine, and finance, and law, in business not even related to tech, but what I'm really proud of that I didn't set out to build because I didn't feel qualified, but I'm really glad that I've been able to serve is that there were three populations that I've been really able to serve, especially at the university level. Number one, international students who, you mentioned yourself, English might not be their first language, but they're not familiar with the US hiring and advancement and pay process, and I help normalize that. And that's something that I myself in the benefit of, having been born here in the US. People who, where English isn't their first language; you think it's hard enough to answer, “Why do you think you should be promoted?”Or, “How much do you think you should make for this role? What do you want?” In your first language? Try answering it in your third, right? And then when I'm really proud of is, especially at the university level, I've been really able to help students where they're first-generation college students, where they don't have a professional mentor within their immediate family.And providing them a roadmap—or actually, the playbook to how to get ahead and then how to advocate for yourself. And these were things that I didn't feel qualified to help, but these are the individuals who've ended up coming and utilizing my program, and finding a lot of benefit from that. And it made me realize that I'm doing something bigger than I even set out to do, and that is very meaningful to me.Corey: You mentioned that you give guidance on salary negotiation and career advancement to not just women, but also men, and not just people who are in tech, but people who are in other business areas as well. How does what you're advising people to do shift—if at all—from folks who are women working in tech?Lauren: So, that's the key is it really doesn't shift. What I'm teaching are fundamentals and, spoiler alert, I teach grounding yourself in data, and knowing your data, and taking the emotion out of the process, whether you're trying to get ahead, to stand out, to earn more. And I teach fundamentals, which is five-point process.Number one, you got to figure out what success looks like to you. I talked a little bit about that earlier, but it's foundational. I mean, I start with that because that alone changed my life. I would still be pursuing success today and not have reached it, but I'm living out how I defined success because I started there.Then you got to really know your worth. Absolutely without a doubt, know how much you're worth. And for me, this was transformational. I mean, eye-opening. Like you said earlier, the blindfold coming off. When I saw for a fact how much employers paid other people with my skill sets, it was a game-changer for me. And so I—without a shadow of a doubt, I use four different strategies, multiple resources in each strategy to know comprehensively how much I'm worth.And then I teach knowing your numbers. It's not an emotional thing; it's very much scientific, so I talked about knowing your key numbers, your target, your ask, and your walk away, and those are all very dependent on your employment and financial situation, so it's different from person to person. And then I talk about—and this is a little different than what other people teach—is I talk about finding leverage, what you uniquely bring to the table, or identifying companies where you uniquely add value, where you can either lock in an offer or negotiate a premium.And then I prepare. I prepare. Just like you prepare for an interview, I prepare for a negotiation, and if I'm asking for the right amount of money, I am going to be prepared for pushback and I want to be able to handle that, and I don't want to just know it on the fly; I want to have scripts and questions prepared to handle that pushback. I want to be prepared to answer some of the most difficult questions that you're going—get asked, like we talked about earlier.And then the final step is I practice over and over and over again, just like a sporting event. I am ready to go into action and get a great thing. So, those are the fundamentals. I've marketed to women in tech because I'm a woman in tech and we don't have enough women in tech, and women are 82 cents on the dollar in tech, but what I found is that doctors were using the same methodology. I wasn't marketing it to them. Lawyers, business people, finance people were using it because I was teaching such fundamentals.Corey: Taking it one step further, if someone is listening to this and starting to get a glimmering of the sense that they're not where they could be career-wise, either in terms of compensation, advancement, et cetera, what advice would you have for them as far as things to focus on first? Not to effectively extract the entire content of your course into podcast form, but where do they start?Lauren: Yeah. So, you start by investing in yourself and investing in the change that you want. And that first investment might be figuring out how much you're worth, you know, doing that research to figure out how much you're worth. And then going out and learning the skills. And look, I have a course, I have a book that you can use to get ahead; if I'm not the right fit, there are a ton of resources out there. The trick is to find the best fit for you.And my only regret as I look back over the last 10, 15 years of my career is that I didn't invest in myself sooner and that I didn't go out and figure out how much I was worth, and that I—when they said, “Well, you're just not there yet,” when I asked for more money, that I believed them. And that was on me that I didn't go out and go, “I wonder how much I'm worth?” And do the research. And then, I regret not hiring a career coach earlier. I wish I'd gotten back into tech sooner.And I wish that I had learned to negotiate and advocate for myself sooner. But my knack, Corey—and I believe things happen to me for a reason—is my special skills is I take things that were meant not necessarily intentionally to harm me, but things that hurt me, I learned from them, I turn it around in the best way possible, and then I teach and I create programs to help uplift other people. And that's my special skill set; that's sort of my mission and purpose in life, and now I'm just trying to really exploit it and make this into a big movement that impacts millions of lives.Corey: So, what's next for you? You've built this platform, you've put yourself out there, you've clearly made a dent in the direction that you're heading in. What's next?Lauren: [laugh]. I am looking to scale. I'm just like any company; I've really focused on delivering value proof of concept. What a lot of people don't realize is not only did I build DevelopHer in quote, “my spare time,” but I did this without any outside investors. I funded it at all myself, built it on my own sweat equity—Corey: [laugh]. That one resonates.Lauren: Yeah. [laugh]. I know you know what that feels like. And so for me, I'm focused on scale: bringing in more corporate partners; bringing in more university clients, to scale and bridge the gap before it even starts; and scaling and reaching more women and men and anyone who wants to figure out how to get ahead, stand out, and earn more. And so the next year, two years are really focused on scale.Corey: If people want to learn more about what you do, how you do it, or potentially look at improving their own situations, where can they find you?Lauren: I am online. Go to developher.com. I have resources for individuals; I have a book, which is a great, cost-effective way to learn a lot.I have an award-winning negotiation course that helps you go out and earn what you're truly worth, and I have a membership to connect with me and other like-minded individuals. If you're a company leader, I work with companies all the time to train their women—and men, too—to get ahead and build their value. And then also, I work with universities as well to help bridge the gender wage gap before it starts, and builds future leaders.Corey: And we will, of course, include links to that in the [show notes 00:27:55]. Thank you so much for taking the time to speak with me today. I really appreciate it.Lauren: Corey, thank you so much for having me, and I really mean it. You know, Corey is a strong ally. We connected, and I am glad to count you as not only my own ally but an ally of DevelopHer.Corey: Well, thank you. That's incredibly touching to hear. I appreciate it.Lauren: I mean it.Corey: Thank you. Sometimes all you can say to a sincere compliment is, “Thank you.” Arguing it is an insult, and I'm not that bold. [laugh].Lauren: That's actually really good advice that I give women is, so many times, we cut down our own compliments. And so that's a great example right there, and it is not just women who sometimes I have a challenge with it; men, too. When someone gives you a compliment, just say, “Thank you.”Corey: Good advice for any age, in any era. Lauren Hasson, founder of DevelopHer, speaker, author, frontline engineer some days. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and an insulting comment telling me that my company is never going to succeed if I don't attempt to outbid Netflix.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Jonan Scheffler talks to Founder and CEO of Krunch Data, Jordan Chung, about Krunch being a tool for content creators to understand what their audiences are finding engaging. They discuss regional salaries and the belief in paying people for the work they perform, not paying them based on where they live, how there are more opportunities to work in Developer Relations now more than ever, that half of working in DevRel is convincing people that DevRel should even exist, and soon every company will soon *need* a Chief Developer Relations Officer (CDRO)! Jordan also gives solid advice for newbies in the field: follow your curiosity and be open-minded.Should you find a burning need to share your thoughts or rants about the show, please spray them at devrel@newrelic.com. While you're going to all the trouble of shipping us some bytes, please consider taking a moment to let us know what you'd like to hear on the show in the future. Despite the all-caps flaming you will receive in response, please know that we are sincerely interested in your feedback; we aim to appease. Follow us on the Twitters: @ObservyMcObserv.

In this week's episode of the SD Times "What the Dev?" podcast, editor-in-chief David Rubinstein discusses the acquisition of observability company Lightstep by ServiceNow and what that means for the observability and SRE space. His guest is Austin Parker, lead developer advocate at Lightstep who will also be talking about the OpenTelemetry project. 

Dando sequência à série sobre observabilidade em arquiteturas distribuídas, o Kubicast apresenta o ilustríssimo Bruno Pereira, CEO e co-founder da Elvenworks.A plataforma de observabilidade da Elvenworks foi criada sob a perspectiva de entregar resultados com a menor complexidade possível. Nela dá para priorizar métricas que realmente podem impactar o ciclo de vida do produto, como a correlação entre latência, tráfego e erro entre outros. A solução roda em SaaS e tem em seu roadmap plano para ser aparelhada ao OpenTelemetry.Ouça o episódio para conferir o que mais a Elvenworks agrega à observabilidade de sistemas distribuídos.CONTATO DA ELVENWORKS - Para entrar em contato com o Bruno, envie mensagem para: contact@elven.works ou @blpsilva no Twitter.As RECOMENDAÇÕES do programa seguem aqui:Bruno:Buscar equilíbrio entre a vida pessoal e profissional, praticando um esporte, por exemplo!Não se comparar aos outrosCobra Kai (Tem na Netflix)João:Nove Desconhecidos (Está no Amazon Prime)#KCDBRASIL - O maior e mais importante evento de Kubernetes do Brasil está com o CFP aberto. Mande até 29/10 sua sugestão de palestra para espalhar a palavra para nossa comunidade! O Kubicast é uma produção da Getup, especialista em Kubernetes e apoiadora do projeto UnDistro, uma distribuição para gerenciar múltiplos clusters Kubernetes. #o11y #Observability #DevOps #Kubernetes #Containers #docker 

Boris joins Adam to discuss building and running his SaaS businesses on AWS and the biggest pain points in the serverless developer experience.

Forrest Knight joined us this week to discuss customer challenges around application delivery and how Lightstep solves it via Observability. And, check out this excellent overview of Lightstep by Forrest.  See omnystudio.com/listener for privacy information.

“Software telemetry is what you use to figure out what your production systems are doing. It's all about shortening that feedback loop between the user experience and the engineers who are writing the user experience." Jamie Riedesel is a Staff Engineer at Dropbox working on the HelloSign product and also the author of “Software Telemetry”. In this episode, Jamie shared an overview of software telemetry and explained why it is important for us to understand how our production systems are behaving by using those telemetry data. She also explained different software telemetry types, concepts such as observability and cardinality, and shared some software telemetry best practices. In the second part of our conversation, Jamie opened up and shared her own personal experience dealing with toxic work environments. She emphasized the importance of self-awareness and psychological safety, as well as went through the five key dynamics to a successful team based on Google's re:Work blog post. Listen out for: Career Journey - [00:05:15] Software Telemetry - [00:07:22] Knowing Your Production System - [00:12:13] Types of Software Telemetry - [00:16:45] High Cardinality - [00:22:34] Observability & Buzzwords - [00:27:08] In-House vs. SaaS - [00:30:04] Some Telemetry Best Practices - [00:32:35] Toxic Workplace - [00:38:45] Identifying Your Toxicity - [00:44:18] Psychological Safety - [00:49:02] Identifying a Person's Baggage - [00:53:52] Who is On The Team Matters Less - [00:58:09] 3 Tech Lead Wisdom - [01:01:49] _____ Jamie Riedesel's Bio Jamie Riedesel has over twenty years of experience in the tech industry, and has spent her time as a System Administrator, Systems Engineer, DevOps Engineer, and Platform Engineer. She is currently a Staff Engineer at Dropbox, working on their HelloSign product. Jamie's blog at sysadmin1138.net has been there since 2004 and survived the apocalypse of Google Reader shutting down. Jamie is the author of “Software Telemetry” through Manning Publications, and also has a deep interest in reforming team cultures to be less toxic. Follow Jamie: Blog – https://sysadmin1138.net/mt/blog/ Twitter – https://twitter.com/sysadm1138 LinkedIn – https://www.linkedin.com/in/jamie-riedesel-983773b Our Sponsor Are you looking for a new cool swag? Tech Lead Journal now offers you some swags that you can purchase online. These swags are printed on-demand based on your preference, and will be delivered safely to you all over the world where shipping is available. Check out all the cool swags by visiting https://techleadjournal.dev/shop. Like this episode? Subscribe on your favorite podcast app and submit your feedback. Follow @techleadjournal on LinkedIn, Twitter, and Instagram. Pledge your support by becoming a patron. For more info about the episode (including quotes and transcript), visit techleadjournal.dev/episodes/57.

If you need to learn how Prometheus, OpenTelemetry, Loki, FluentD, FluentBit .. help you with your observability requirements in the cloud native and non-cloud native space but you don't have hours or days to dig into the details yourself then you have a new place to go to get educated within 20-30 minutes: Is it Observable is a new educational YouTube channel by Henrik Rexed, Cloud Native Advocate at Dynatrace.In this episode we have Henrik explain the motivation of creating Is it Observable, how to best use the videos and the tutorials on GitHub to educate yourself and gave a glance on upcoming episodes that will also include guests from various tool and platform vendors.Make sure to subscribe to his channel, check out the tutorials on GitHub and give him feedback on twitter (@hrexed) or LinkedInShow LinksIs It Observable YouTube Channel:https://www.youtube.com/channel/UCRiin4u8YZGlVQRZp7qOOawHenrik Rexed on Linkedinhttps://www.linkedin.com/in/hrexed/Is It Observable Github Repohttps://github.com/isItObservableHenrik Rexed on Twitterhttps://twitter.com/Hrexed

O volume dois da série de players de observabilidade apresenta Ricardo Ferreira, developer advocate da Elastic, empresa que tem sua solução de observabilidade integrada e Open Source, baseada no Elasticsearch e Kibana.Numa conversa profunda sobre a prática de avaliar o comportamento de sistemas distribuídos cada vez mais complexos, falamos sobre o conceito de observabilidade x monitoração, prós e contras das instrumentações Black box e White box, suporte para OpenTelemetry e as principais features do produto da Elastic. Siga com o play para conferir na íntegra o novo episódio da Saga de Observabilidade:CONTATO - Para falar com o Ricardo, envie sua mensagem para: Telegram: https://t.me/riferreiTwitter: https://twitter.com/riferreiEmail: riferrei@riferrei.comYouTube: https://videos.riferrei.comBlog: https://riferrei.comOs LINKS comentados nesse Kubicast são:OpenTelemetry com JAVAOpenTelemetry com GolangAs RECOMENDAÇÕES do programa seguem aqui:Ricardo:Restaurante Bolão em Belo Horizonte - MGPão na chapa Mind Hunter (Tem na Netflix)João:Esquadrão dos Confeiteiros (Tem na Netflix)AVISO! O CFP do #kcdbrasil está aberto e vai até 29/10. Apresente sua ideia para espalhar a palavra no maior evento de Kubernetes do Brasil. O Kubicast é uma produção da Getup, especialista em Kubernetes e apoiadora do projeto UnDistro, uma distribuição para gerenciar múltiplos clusters Kubernetes. #o11y #Observability #DevOps #Kubernetes #Containers #docker 

A serviço da comunidade Open Source, o Kubicast apresenta a Saga de Observabilidade, uma série extraordinária que, a cada episódio, desvenda um player do mercado de soluções de observabilidade para sistemas distribuídos.A obra estreia com Fernando Sanches da Instana, plataforma de observabilidade da IBM que - através da coleta de métricas, logs e traces de forma de forma instantânea, segura e automatizada - dá contexto para tomada de decisões. A ferramenta tem ainda inteligência para identificar incidentes, com base em estudos prévios de laboratório, correlacionar eventos, descobrir a raiz do problema, configurar e comunicar alertas em plataformas de mensagens e outras features.  Ouça esse e os próximos Kubicast para mergulhar a fundo no universo da observabilidade e tirar suas próprias conclusões!INSTANA - Para entrar em contato com o Fernando, escreva para: fernando.sanches@instana.com. Para testar a ferramenta, acesse aqui (playground da Instana com guia das funcionalidades).As RECOMENDAÇÕES do programa seguem abaixo:Fernando:Big Data in Practice: How 45 Successful Companies Used Big Data Analytics to Deliver Extraordinary Results (Livro do Bernard Marr)Live-action da Mulan (Disney+)João:A Arte da Autodefesa (Netflix)ATENÇÃO: o #kcdbrasil está de volta!O maior e mais importante evento de Kubernetes do Brasil já está com data marcada: 18/1/22. O CFP está aberto para você sugerir o assunto que quer compartilhar com a comunidade. Acessa lá! O Kubicast é uma produção da Getup, especialista em Kubernetes e apoiadora do projeto UnDistro, uma distribuição para gerenciar múltiplos clusters Kubernetes.#Observability #DevOps #Kubernetes #Containers #docker  #o11y

Is your application observable? In this episode, Henrik Rexed, a Cloud-Native Advocate at Dynatrace, shares how you and your teams can implement better observability into your systems. Discover how he transitioned from performance testing to SRE, the difference between APM and Observability, Kubernetes, Telemetry, his new YouTube “Is it Observable” channel, and more. Listen up. Never miss another episode. Join the Guild and get alerted every time a new episode is release + get members-only bonus content: https://testguild.com/newsletter/

All-star trio Charity Majors, Liz Fong-Jones, and George Miranda join us this week to talk all things observability, observability engineering, and their new book on the subject.

Observability has been a topic I've heard more and more about the past few years. But what is it really, and how does it apply to testing? In this episode, Abby Bangser, a Site Reliability Engineer (SRE) at Duffel, breaks it down for you. Discover what observability & telemetry is, how to use observability in debugging, how observability is different from monitoring, and much more. Listen up! ** Never miss another episode join the Guild and receive weekly FREE automation awesomeness: https://testguild.com/newsletter/  

This episode features Josh Benamrum, who is the co-founder of Databand. Databand is a company that helps engineering teams achieve better observability and control over their tech stack.Top 3 Value Bombs: When observing our data we should be looking at our data and pipelinesDon't wait till the board meeting for an incorrect metric to make DQ a priorityHaving clear SLA's on just what data quality means across the organization is essential 

In this episode of the Virtual Coffee with Ashish edition, we spoke with Colby Funnel (Linkedin - @Colby) is a Development Manager at Atlassian(@Atlassian). Episode ShowNotes, Links and Transcript on Cloud Security Podcast: www.cloudsecuritypodcast.tv Host Twitter: Ashish Rajan (@hashishrajan) Guest Linkedin: Colby Funnel (Linkedin - @Colby) Podcast Twitter - Cloud Security Podcast (@CloudSecPod) If you want to watch videos of this LIVE STREAMED episode and past episodes - Check out our YouTube Channel: - Cloud Security Podcast: https://www.youtube.com/c/cloudsecuritypodcast?sub_confirmation=1 Cloud Security Academy: www.cloudsecuritypodcast.tv/cloud-security-academy

On today's Heavy Networking, we drill into VMware's vRealize Network Insight (vRNI) to learn how it provides end-to-end monitoring, how it uses flow records and other data sources, and its architecture. We'll also discuss modeling/digital twin capabilities, and applying vRNI to security, troubleshooting, and other use cases. VMware is our sponsor. The post Heavy Networking 593: Network Observability With VMware vRealize Network Insight (Sponsored) appeared first on Packet Pushers.

On today's Heavy Networking, we drill into VMware's vRealize Network Insight (vRNI) to learn how it provides end-to-end monitoring, how it uses flow records and other data sources, and its architecture. We'll also discuss modeling/digital twin capabilities, and applying vRNI to security, troubleshooting, and other use cases. VMware is our sponsor. The post Heavy Networking 593: Network Observability With VMware vRealize Network Insight (Sponsored) appeared first on Packet Pushers.

On today's Heavy Networking, we drill into VMware's vRealize Network Insight (vRNI) to learn how it provides end-to-end monitoring, how it uses flow records and other data sources, and its architecture. We'll also discuss modeling/digital twin capabilities, and applying vRNI to security, troubleshooting, and other use cases. VMware is our sponsor. The post Heavy Networking 593: Network Observability With VMware vRealize Network Insight (Sponsored) appeared first on Packet Pushers.

In this episode of the Virtual Coffee with Ashish edition, we spoke with Ted Young (@tedsuo) is a contributor along with AWS, Google Cloud, Microsoft Azure in the Observability eco-system. He is also the Director of Developer Education at LightStep (@LightStepHQ) Episode ShowNotes, Links and Transcript on Cloud Security Podcast: www.cloudsecuritypodcast.tv Cloud Security Academy: www.cloudsecuritypodcast.tv/cloud-security-academy Host Twitter: @hashishrajan Guest Linkedin: Ted Young (@tedsuo) Podcast Twitter - @CloudSecPod If you want to watch videos of this LIVE STREAMED episode and past episodes - Check out our YouTube Channel: - Cloud Security Podcast: https://www.youtube.com/c/cloudsecuritypodcast?sub_confirmation=1

Tune in as Co-founder and CTO of Honeycomb, an observability platform, Charity Majors joins Chris to drop some knowlege bombs such as: Thinking of observability as being about the unknown unknowns: Allowing for high cardinality, high dimensionality, ad hoc queries at any point in time. Comparing instrumentation to a muscle: It's a habit that needs to be developed and fostered. Sincere continuous deployment: 15 minutes or bust. And bunches more, since y'all know you hear her name come up at least once during every other episode! Honeycomb.io (https://www.honeycomb.io/) o11ycast (https://www.heavybit.com/library/podcasts/o11ycast/) Charity's blog (charity.wtf) (https://charity.wtf/) Charity on twitter (https://twitter.com/mipsytipsy) Charity's post on cost of not doing continuous deployment (https://charity.wtf/2021/02/19/how-much-is-your-fear-costing-you/) Charity's post - The Engineer Manager Pendulum (https://charity.wtf/2017/05/11/the-engineer-manager-pendulum/) Transcript: CHRIS: Hello, and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Chris Toomey. And this week Steph is taking a quick break, but while she's away, I was joined by a special guest, Charity Majors. Now, folks who've been listening to the show lately will know I've been mentioning one idea or another from Charity almost every episode these days. Charity's work spans from the deeply technical through to the deeply human. And across all of it, she brings such a wealth of experience in pragmatism while consistently providing grounded, actionable advice about how we can improve all aspects of our work. And to give a bit more context for those who aren't as familiar with Charity's work, she is the co-founder and CTO of Honeycomb, which is an observability platform that we talk about more in the episode. Charity is also a prolific blogger, tweeter and speaker, and general leaver of digital breadcrumbs for the rest of us to hopefully follow. And Charity is also one of the hosts of the o11ycast podcast. That's observability, o11y podcast. And in fact, in the intro to the first o11ycast episode, Charity provides a beautiful summary of her approach to the varied work that we do. Quote, "I'm someone who's always been drawn to where the beautiful theory of computing meets the awkward, messy reality of actually trying to do things." And that quote rang so deeply true to me when I heard it and really encompassed what I see across the variety of work that Charity has shared with us. And frankly, I've been so impressed with the quality and quantity of wonderful content that Charity has shared over the years. I was really just thrilled to get the chance to sit down and talk with her directly. So without further ado, here's our conversation with Charity Majors. Thanks so much for joining us today, Charity. CHARITY: Thanks for having me. It's great to be here. CHRIS: As I've mentioned on many an episode, I've been following your work for a while now. And at this point, I would say that just about every Bike Shed episode has a reference to you and some piece of work that you have put out into the world, whether it be a tweet or a blog post, or a conference talk or something. So I'm so grateful for all the work that you put out into the world and for taking the time to chat with us today. CHARITY: That's so exciting. Yay. I feel right at home here then. [chuckles] CHRIS: Fantastic. Well, I want to dive in. I think it's sort of the core of some of the conversation that we'll be having, which is around instrumentation and observability, and observability as a newer, noveler form of how we think about this space. But to give a bit of context, I was hoping you might be able to give just the quick summary for anyone who might not be as familiar with observability as a concept and what that means now, and Honeycomb as a product and how it offers affordances around observability and pushes that envelope forward. CHARITY: Yeah, I think of the observability as being about the unknown unknowns. For a long time, all of the complexity was really bound up in the app. You had the load balancer, you had the app, and the database. And all the complexity you could just attach to a debugger and step through it if you had to. But then we kind of blew up the app, the monolith, and now it's in services scattered to the winds, and you can't just trace it. And so observability is a way of passing that context along hop by hop so that you can actually slice and dice in real-time. And the hardest problem is not usually debugging the code. It's finding out wherein the system is the code that you need to debug. And observability, if you accept my definition, which is it's about unknown unknowns, that you should be able to ask any question of your systems, understand any internal state just by observing it from the outside, well, then a lot of things proceed from that, in my opinion. Like, you need to be able to handle high cardinality, high dimensionality. You need to be able to string together a lot of these high cardinality dimensions. You need to... any kind of schema or indexing scheme in advance is verboten because you don't know what questions you're going to need to ask. And so there's a lot that flows from that definition; arbitrarily-wide structured data blobs is the source of truth, et cetera. But at its heart, it's just about the concepts, that our problems are getting harder and harder. We don't get paged to go, "Oh, that again? Oh, that again?" CHRIS: [chuckles] CHARITY: Ideally, we fix those things. But we still get paged. What the hell is this? It's about allowing engineers, empowering them in a reasonable amount of time to be in constant conversation with that code that's out there in the world because most problems honestly we never get paged about. They're too subtle until they snowball, and they pick up other problems. It's like a hairball under your couch until it gets so big and so impacting that it actually does alert someone. And then you just start picking up the rock and be like, oh God, what's that? Well, we've never understood this. And that's why ops has such a reputation for masochism. [chuckles] CHRIS: Absolutely. There are so many little pieces in what you just said that really deeply resonate with me, although there is one facet of some of the way that you talk about observability that I find interesting. I'm someone who likes to cling to the perhaps unrealistic these days ideal of a monolith of what if we were to just keep everything in the same place and all the data lived together in one database, and I could have foreign keys, and consistencies, and asset compliance? CHARITY: Which you should do for as long as you possibly can. You should never impose more complexity on yourself than you absolutely need to. And I would say that it's never not better to have observability than the older paradigms of monitoring and so forth. Some of Honeycomb's biggest and best customers still use monoliths. But they still find it really valuable to be able to apply the principles. I think that it's the microservices revolution, if you will, that forced this set of changes. It was inevitable. The steps that I started talking about, like, somebody would have because the older way just became untenable when you started adopting containerization and a lot of these things that made everything suddenly a high cardinality including the number of applications you have. But it's never not better to have high cardinality tools and to be able to instrument your code for spans and tracing. Tracing is still valuable even in the monolith. CHRIS: Yeah. As I've observed and started to play around with Honeycomb, that's definitely what I've seen is I'm almost exclusively working in the context of monoliths and, like I said, clinging to them for as long as I possibly can, which isn't going to be forever. CHARITY: It's true. [chuckles] CHRIS: I recognize that truth, but already I see the value. And so Honeycomb is a platform that you've built that allows for this high cardinality, high dimensionality ad hoc queries at any point in time. And so the idea that I can come into the tool and say, "Huh, I've got a new novel problem today." I don't need to re-instrument my code. I can just ask a new question, and the system will responsively be able to answer that question, ideally. And that feels like it holds true in a monolith all the more so, like you said, in an SOA architecture. But even in my safe little playground of everything is in the same space, I still don't know how everything's working all the time if we're being honest. So being able to answer those questions feels meaningful. CHARITY: Totally. I think that one way of thinking about the SOA or microservices is that it pushes a lot of what was in the operations realm into a realm of development, and suddenly you're responsible for a lot more of the operating of your services, things like retries and backoffs, and load distribution, and thundering herds, and all these things that ops traditionally took care of. Well, now you have to think about them. So you need some ops tools, too. What I like about...of course I like everything about Honeycomb because we designed it for this problem. But it speaks in the language of variables, and endpoints, and functions, and not in the low-level language of proc IPv6 timeouts and stuff where I feel like ops has also traditionally been the translation layer between software engineers and their actual code in production. And it's time to start giving software engineers those tools in their own language. CHRIS: Yeah. I love that. And I'm very happy to have Honeycomb as part of an instrumentation stack, which actually shifts me to the next question, which as I look at Honeycomb, very quickly the first time I saw it, I was like, oh okay, this makes sense. I want this in the world. CHARITY: Oh, I like you. [laughs] Not all people are like you. CHRIS: It might have been my second or third look, but it was definitely...once I got it, I was like, oh yes, I absolutely want that. But now, the question that I have is I typically will have a collection of tools that exist in this space. And there's a weird Venn diagram overlap of well, there's logging, and there's error tracking, and there are APM performance tools, and there's metrics, dashboards. And my sense is that Honeycomb perhaps can or an observability tool more generally can subsume a bunch of those, but it's not clear to me exactly. I think I probably still want logging. I think I still want error tracking as a discreet service tool that I'm using but maybe not APM and maybe not metrics as a distinct thing. Maybe I can infer those from a tool like Honeycomb. But I'm wondering what's the current thought on that? CHARITY: Well, part of what you're seeing is just observability tooling is very new, and we haven't had time to grow up. And here I'm like, officially, we play very nicely with all other vendors, and none of us would ever try to compete or take away from each other's faces. But I do think that ultimately, logging pretty much the only real use case for it is security stuff, the security archiving, just keep every log light. It's gotten cheap enough, but it's not actually useful for debugging or understanding your system, not really. It's useful for compliance. It's useful for proving that you did something in the past. Most logs are just a pile of trash, but they can be useful trash. And I understand people's emotional want to hold onto them for a while, and there's nothing wrong with that. There's nothing wrong with keeping some trash around for a while, while you make it...[laughs] Sorry, not to totally slam on logs, but they are trash. CHRIS: I love the analogies that we're going for. [laughs] CHARITY: But the thing about observability is I do think the kind of center of the world is these arbitrarily-wide structured data blobs from what you can infer logs, from which you can infer metrics, from which you can roll-up. So I do think that well metrics are the right first tool for understanding infrastructure. Like if you're Amazon and you're responsible for all this hardware and stuff, you should be asking yourself, is my service healthy? But if you're someone who's writing and shipping code on top of that service you care about, can my request complete? What is my user's experience? And that's observability's territory. So I think that ultimately, I do think metrics, logs, and traces all get subsumed under the observability umbrella and performance management, too, if the tools get built correctly. There will still be use cases. They will just get smaller, for logs, for standalone metrics tools. Honeycomb just launched our metrics product. Metrics is like a 30-year-old piece of technology. Prometheus and Datadog are going to be the last best metrics tools ever built. We have wrung the water out of this laundry. [chuckles ] But we aren't trying to compete with that. What we are trying to do is give people an on-ramp into Honeycomb. They've got decades' worth of stuff. They've been corralling metrics, structuring them. You rely on them. You don't want to give them up. So yeah, let's feed them in. Let's give them an overlay. And number two, the more interesting use case for me is when you're a software engineer who's writing and shipping code, you do care about did the memory usage just triple, or is the CPU completely buzzing after I shipped my last change? But there's really only like three or four of those metrics that you really care about as system metrics. The rest are mostly legacy. CHRIS: I like the idea that aspirationally, Honeycomb is moving towards a place where given sufficient input data, given this arbitrarily-wide data blob with high cardinality, et cetera, that we can infer basically all of those others from it. But also speaking to also observability is somewhat new, and so we got to build a lot of product to get there and that idea that there is perhaps a space right now where you might be bringing together a few of these tools. But if there is a future world in which I can have one of these tools that just handles everything and tells me about my code and directs me to the line of code that I incorrectly instrumented, that would be wonderful. Happy to do the work in the interim to cobble it together from the pieces. CHARITY: The place in the meantime that we're at where all of these big vendors are acquiring other vendors and trying to put together...they're like, we have three pillars. Coincidentally, we have three products to sell you. It's like, it's not good for the users because when you're...like, you're sitting in the middle here. You've got your metrics dashboard. It's telling you that there's a problem. Okay, if you can't slice and dice and figure out what it is, you have to jump over into logs and visually correlate based on the times and hope no timestamps are wrong and try and find the thing. And then, oh, okay, so you want to trace it. So you've got to copy over and try and find that in your tracing product and hope that that would get sampled in. It's not good. You can't follow the question from the beginning. I have a problem to the end. I have a solution and back. And it's not linear. You're going to be following a trail; then you're going to need to back up, then you're going to find another trail. And then you're going to want us to zoom out and see who else is impacted. And you really can't back your way into that with different products. You have to start with the arbitrarily-wide structured data blob. What does confuse me is I know that New Relic is built on this. New Relic has these. And we almost didn't start Honeycomb because we were just like, edit data, and New Relic is going to figure it out. Here we are like six years later, and they still haven't fcking figured it out. [laughs] But like Datadog, they aren't based on that arbitrarily-wide structure, so they are really...and I know that they're trying to get...all of these big vendors are trying to get to where Honeycomb sits technically faster than we can grow up and become a business. CHRIS: The race is on. CHARITY: Yeah. It's fun. CHRIS: One of the related things that I've seen you talk about a few times is the idea that instrumentation is a muscle. It's a habit that needs to be developed and fostered, and that rings very true to me. At the same time, a lot of my instrumentation work has been more in a reactive space. If we're being completely honest, something went wrong; we can't figure it out from the information that we have available, so then we go in, and we add a new logging line. We wrap the code in some way. And so I'm wondering if you can talk a little bit more about that. What does that look like in practice or perhaps some examples or something? But how can we tease that apart and understand that a little bit better? Because it sounds wonderful to me. CHARITY: I think of instrumenting a lot like commenting your code. It's a way of thinking to the future and reverse engineering; what am I going to care about? What is someone else going to care about? And I really do think of commenting as just a less true version of instrumentation, honestly. It's you talking about what you think the code should be doing, but you've left production out of the loops. You don't know what the code is doing. [chuckles] But ideally, they're kind of the same muscle. It's why you're writing your code. You've just developed a monitoring thread almost in your brain. It's like, yeah, this is going to be valuable. Oh, this is going to be valuable. And so I do think that it's on vendors to make sure that we do as much for you as possible. And this, honestly, is the long winding journey to Honeycomb finding product-market fit, which took almost three and a half, four years. And for a long time, I was like, it's not magic. You have to understand your code. You have to blah, blah, blah, which is true. But also, we need to walk closer to the user. We need to make it easier. We need to do the beeline, which will initialize the event, pre-populate it with a bunch of stuff, create the framework so that all you have to do as a user is just printf now and then just stuff this in the blob, vendors making it as easy as possible, as automated as possible. We have more to do. We really should be pre-populating it with all of the language internals and all of the stuff about the environment. We'll just be glad to tap that well. But there's something that we can't do for you, which is understand what you're trying to do and what is important. Honestly, here's a story from the past. The reason that New Relic was so big, they hit the ground, and they super hockey-sticked everything was because they dovetailed with the rise of Ruby and Rails because Ruby allows for so much fcking monkey patching. Every web app looks the same. You can just be like, we assume all this crap, and so we could make it just like magic for you. You just install this library. Boom, you're off to the races. Well, try as you might, I want to say a type language like Go, you can't do that stuff with. You can't make it as magical. You have to think a lot more about how you're structuring things for better or for worse, which is why their growth slowed because those languages just aren't so popular anymore. So it's trade-offs all the way down. Yes, everybody should be an expert in forecasting the future and understanding all the subtle things that you don't know you're going to know, but you're super are going to want to know. But as you've discovered, most of your learning comes from being in the trenches, which is why it's so good for devs to be on call and be close to their code and be in this constant conversation with it because you develop a sixth sense. I can't tell you exactly why I know it's going to be a problem, but I'm just going to wrap it because I'm pretty sure it is. CHRIS: There was a tiny bit that I was hoping that you would have some very specific like, oh, you just do X, Y, and Z. I kind of knew that wasn't going to be the answer, but it also represents something that I so appreciate about your thinking and the work that you put out into the world, which is it's realistic. Sometimes you're like, you know what? There's going to be some tacit knowledge involved here. You got to put in the work. You got to learn the thing, and that's just true sometimes. And so I appreciate your willingness to be like yeah, you know what you got to do? You got to do the work. And then after that, you'll know...and so there's sort of a virtuous cycle that can happen here. There is a feature, as far as I understand it, of Honeycomb, too if I can briefly hype up your products slightly but the idea that you can observe the series of questions that another developer asks. So if they were in a debugging session, you can see like, oh, they asked this, and then they asked this, and then they filtered on that. CHARITY: It's like your Bash history but for debugging. [chuckles]. CHRIS: I want this for everything. CHARITY: Right? CHRIS: Let's have a shared hive mind of the developers on a team, both in terms of our observability tool but also just kind of everything. CHARITY: What did you do? CHRIS: Yeah, what did you do, and why? What were you thinking? I saw you went down a road there, but then you stopped and backed up, and you went a different way. That's interesting to me. CHARITY: This is why we keep trying to build things into the product that will incentivize people to write texts about what they're doing, whether it's retroactively applying tags or writing a breadcrumb to yourself. Why was this meaningful? As you're putting it in your bookmarks, why are you putting it in your bookmarks? Collaboration is just as much about collaborating with your past self and your future self as it is with the rest of your team. I don't remember why the fck I did that two years ago. I don't know. I don't know why I did that two months ago. But the more you can leave breadcrumbs for yourself and then surface that to the team, you're right; it's transformational. I wanted this so selfishly because I have never been that person on the team who loves graphs. I hate graphs. I don't think visually very well at all. I've been working with my friend, Ben Harts, off and on for like 10, 12 years now. He's always the person I've hired repeatedly. He's always the person who comes in and makes the graphs. And then I look over his shoulder, and I bookmark them. I can be up all night making the perfect dashboard. And then I'm like, great, mine. [chuckles] So there's room in the world for both of us. But the point is that not all of us should have to go through that effort. [chuckles] We should be able to learn from each other. Only one person should ever have to have to craft the perfect query, and then the rest of the team should be able to effortlessly piggyback on it. CHRIS: Yeah, absolutely. And again, I want that but for everything. I dream of a future in which that's true. CHARITY: And so much of debugging is this wandering path where you go down the wrong place, and you need to be able to zoom back to all right; where did I first know that I had a beat on it? CHRIS: There's a corollary that I see to pair programming where one of the things that I find so valuable is, what Google query do you type in when you hit that wall? When you're like, oh, this isn't working as I'm thinking, and then you type something and I'm like, whoa, wait, I wouldn't have even thought to ask that question of the internet. CHARITY: Oh, I love that. That's fantastic. CHRIS: But now you've productized that, and I love that. So thank you for building that thing in the world. CHARITY: Excellent. CHRIS: Shifting gears slightly, one of the other themes that you really pushed for in the world is the idea of continuous deployment and not like yeah, you should ship your code pretty quickly after you merge it, but true, sincere continuous deployment. CHARITY: 15 minutes or bust. CHRIS: 15 minutes of bust, test in production. There are some really wonderful if we're being honest, scary themes that you talk about. I love the ideas that you're putting out there, but they're probably the things that I look at, and I'm like, ooh, that seems like a whole thing right there. CHARITY: It assumes a lot. Let's put it that way. It assumes a lot. CHRIS: It definitely does that. I desperately want to get to that world. I want to get to the place where there's that confidence. And similarly, there's a theme that you've talked about around Friday deploy freezes and why that's not a good thing. And the empathy for humans that part's good, but maybe we're applying it in the wrong way if we say we're not allowed to deploy code on Friday. Because it's like yeah, deploying code is terrifying and scary. No, let's solve that problem. But I wonder if you can talk a little bit about that. How do you get there? How do you get to the place where continuous deployment is a realistic outcome for you? CHARITY: Yeah, that's a very good question. There are no easy answers, unfortunately. And the answer is always going to depend on where are you starting from? Are you starting from a clean slate? Are you starting...a lot of the advice that I give sounds like Looney Tunes to someone who's coming from enterprise because they're just like, "You don't understand the constraints that I am operating under." And I'm like, "Yeah, you're right. I'm not of your world. That probably shows." [chuckles] So I think the easiest way, though, is always when you're starting a new project that what you do on day one would be to set up your CI/CD and deploy it to prod before you've even started building. My favorite analogy to that is to like...you know the myth about Alexander the Great and his horse how when he was a little boy he would pick it up every day before he had breakfast? And so, by the time he was an adult, he could pick up his horse because he picked it up every day, and it was never hard. When you start deploying that way, it's never hard. When you're just like, okay, anytime this gets above 10 minutes, we're going to put in a couple of hours of work, and it's never hard. It's just the easiest thing in the world. And everything's easier because you get to watch what you're doing and in real-time, and you develop that muscle of I'm merging it to main. I'm going to go look at it in a couple of minutes. And you don't feel done in your gut until you've looked at it. And that's doing it on easy mode. And you can do this in a hybrid way. Even if you have like, well, I'm paying for a deploy. Nobody is saying you have to sign up for a long, painful deploy process when you got to spin up a new project. And I've seen it gain momentum. If you start something that's clearly the new way, everybody sees how fast this team is executing. Everybody wants a piece of it. And so you start learning from the way that you are able to do it in your unique environment. You're the best evangelist to the rest of your team members because you know the subtleties. You know the problems. So that's the easy answer is start fresh. [laughs] CHRIS: [laughs] That makes sense. I do, again, I appreciate the pragmatism or the realism of the way that you approach a lot of the topics. CHARITY: Another answer, though, it's just that the engineering work involved in taking a deployed pipeline down from hours, days, to 15 minutes it's just engineering work. It is just labor. It can be done. The political problems are the hard ones. I mean, in the past, sometimes our deploy probably would get up to two or three hours, and we were just like, oh God, this is not…put in the work. You just start instrumenting your pipeline, and you start looking at where the tests are taking time. And it will pay dividends every bit of time that you pay down, which is why I really see these long…our own pipelines is it's a vacuum of engineering leadership that they've allowed it to happen because there's nothing fancy about it. You just put in some work. CHRIS: Yeah, the solvability of the technical challenge feels very true, but what you're saying of it's people problems which again, that's always true of the tech stuff. CHARITY: It is people problems, but I also hate it when people are just like, oh, it's people problems. That means mysterious and unsolvable. Now, most of the time, when you see this, it's a lack of collective confidence in themselves. They see this as being as just for the elite engineers, or only ex-Googlers are allowed to do this or something. Or they go to conferences, and they hear about it, and they're just like, God, I wish I was allowed to do that, or I wish we could do this. But the thing is that engineers have more power than they realize. We build these companies. They wouldn't exist if it's not for us. We have all the power if we just choose to use it. I know that a lot of these people who I've talked to that were just like, "Oh, I wish we…" I'm like, "Have you ever lobbied for it?" And they're like, "No, I just know we could, or that's someone else's decision." I'm not going to promise you that you can get whatever you want. But I promise you that if you start speaking up if you start talking to your colleagues and being like, "Wouldn't it be nice?" And they start speaking up...if a quarter of the engineers want something in the company, it gets done. [chuckles] CHRIS: That definitely feels true. And to the topic of actually lobbying for this and having the hard conversations internally and working on the people problems, you have done, I think, a really fantastic job of providing actual benchmarks in terms of timing and what does this look like as a practice and what are the multitasks? CHARITY: It's so expensive. It's so costly to organizations. And it's the easy answer for any engineering leader to be like, "Well, we need to hire." That is the laziest answer in the world. You probably don't. You probably just need to fix your CI/CD system and then bask in the resources that you suddenly freed up. [chuckles] CHRIS: You have a wonderful blog post that really I think does such a good job of highlighting the cost that you're talking about there, the human costs for every slowdown in your deploy process, it has this downstream ramification. And having that as sort of a piece, a bargaining chip in the conversation of here's a voice that is saying a very clear thing about this cost of not doing this work, which granted, it's always trade-offs. Everything is an optimization. But here is a way to actually measure the cost of not going with this approach. And again, I appreciate you're putting that out there in the world so that the rest of us can be like, "Look, on the internet, it says so." CHARITY: [chuckles] Exactly. I'm happy to be the internet for you. But it's so true because other people in your business don't want you to suffer too, either. They don't want everything to get slow. They just aren't equipped to understand the cost of this slowness the way that engineers are. And I feel like sometimes this is...it's like we're always lamenting like, why does product get to own all the engineering cycles? Where aren't we allowed to do all this other stuff? I promise you're allowed to. You just have to make the case because the case is righteous and justified. But you have to explain to them the cost that it's incurring your organization in terms of your ability to execute and in terms of your ability to hire and retain people. You just have to explain those costs. And engineers are just like, "Well, we only say it once, right?" Well, that's not how you win arguments. You have to say it. You'll probably lose. And you say it again, and you'll probably lose. You say it a third. And you will win eventually because you control all of the creative labor of the technical organization. So just make the fcking case. [chuckles] I don't know. I make it sound simple; it's not. CHRIS: I love the sound bite of the cause is righteous, and that is the kernel of the thing here, which is like, just to be clear, this is a virtuous path that you were going down, battle for it, work towards it, absolutely. So I think a related topic here, so continuous deployment is one of those things that you want to get to and a practice that you want to evolve to. But in exploring some of your other work, one of the things that I was exposed to is the DORA metrics, which is not something that I hadn't seen before. But for anyone who's unfamiliar, the DORA metrics is a set of four key metrics to track developer and team productivity, so their deployment frequency, lead time for changes, change failure rate and the time to restore the service. And they are deeply interesting because frankly, I have for a long time felt like developer productivity was not really a quantifiable thing. CHARITY: It's not, yeah. CHRIS: Individual developer productivity I still feel like this is a bad thing. Don't do that. But team productivity these metrics actually are like oh, actually, as I look at those, those seem like the good ones. We should do that. I'm wondering, what does that look like in practice when you see that actually employed within an organization? What are the feedback loops, and how does this appear in the world? CHARITY: Yeah. We all owe a huge debt of gratitude to Jez Humble, Gene Kim, and Nicole, who worked on this for years and got this out into the world, just putting some actual research behind the stories that we were telling ourselves about productivity. And people who haven't read Accelerate...a lot of people are always asking me, do we have any stories? Do you have any research? Do you have any proof or something? I just always point to the book Accelerate. That's where it all comes from. Yeah, it's true because it's such a noisy world. When you're an engineering org, and there's just so much going on, and there's so much stuff that bugs you personally, and some of the stuff that you have true beliefs about. And it's hard to just cut through the noise. And I feel like that's the great gift of the DORA metrics. If you start focusing on one of them, you will lift your org out of poverty, and the others will get better too. And it provides just this wonderful focus point for teams that aren't sure where they stand or aren't sure how to get better because it can be so mystifying. When you're in the trenches, and you're just like, why does everything feel so hard? Why is it that we thought this would take two days, and here it is two months later, and we can't ship anything? And it feels like the more we ship, the farther behind we get. These are the beacon of hope. It's like, you pay attention to these, your lives will get better. You can dig yourself out of this ditch. That's certainly been true for the teams that I've been on. And high-performing teams, I think we all have this idea in our heads that high-performing teams are ones where the great engineers join when in fact, those great engineers could join your team, and they wouldn't get any more done than you are. Because most of our productivity is defined not by the data structures and algorithms that you know but by these social-technical systems that we swim in every day, it's the water around us. It's the friction involved in getting that code to production. If it takes the magical engineer from Google 24 hours to get their code changed out, well, they're not a member of a high-performing team either. You mentioned earlier all these people are out there who haven't experienced a world like this don't live in a world like this. And in my experience, they often lack a lot of confidence because they don't think they're that good, or they don't think that they can have nice things. And the DORA metrics that's your ticket to a better life. It's like go to college and graduate because it kicks off these virtuous feedback loops, these cascading cycles of things getting better for everyone and people getting more excited and energized. And they just don't get burned out by shipping too much code. They get burned out by not being able to ship code. And if you're a leader in any type of organization, and I don't just mean manager, I mean any type of senior engineer or manager or whatever, then it's part of your job to pay attention to these metrics, lobby for them, track them, track them on your own if you must, and try to make them better because every engineering team has two customers or two...whatever. I'm blanking on the word. But it's your customers and your engineering team. You're responsible to both of them. And I've never seen one of those sets deliriously happy and the other set miserable. They tend to rise and fall in tandem. CHRIS: I'm just nodding along for anyone in the audience who can't see what my head's doing. But I love so much all of the things that you're saying and, again, the passion and conviction that you bring to this conversation because these are amorphous, hard to pin down ideas. But I appreciate the North Star that you're setting across all of these different things that as I'm reading, I'm like yeah, that sounds true. I want that. Those things are the things that I want. But interestingly, one of the other threads that I see weaving through a lot of your work is obviously we've talked a bunch about just deeply technical topics thus far, but also a lot of your work spans across to the interpersonal. And frankly, even dividing in that way is not representative of the world because it's a Venn diagram mishmash of some days it's technical, some days it's personal, some days it's both. But one of the things that you've talked about is the engineer manager pendulum which I find super interesting. I think every engineer at some point has that question, that internal oh, do I want to go engineer track or manager track? And this distinct idea or the idea that management is a promotion and any other movement would be different, and you have wonderful things to say about that. The other thing that you've pointed out is that former managers can often make great engineers after the fact because of the earned empathy that they have now from looking at things from a slightly different angle. CHARITY: Amazing engineers. CHRIS: But I'd love to hear a little bit more of your thoughts on that because I think it's such an important space, and I've definitely previously operated under I'm an engineer, and then I guess I got to be a manager, and then I guess I don't know where I go from there, but it's this very linear path. And you shook that worldview of mine, and again, I appreciate that shaking. But yeah, I'd love to hear a little bit more about that. CHARITY: The best people that I've ever worked with have been engineers who had been managers for a while and then went back to engineering, and it's not just empathy, although there's a lot of that too. It's also a deeper understanding of the business and the reason that we do things. So much of being a powerful engineer is choosing the right work to work on so that you get a lot done very efficiently and quickly, and you don't spend a lot of time just foundering, which you've mastered, and you know the basic technical principles. And how do you get better? A lot of it is just getting better at identifying what to do and what not to do because we have to not do so much more than we can ever do in order to move forward. I wrote a blog post as a present for a friend of mine who was a director of engineering at the time, and he was suffering. He was just miserable, and he kept thinking about going back to engineering, just kind of dragging his...because he wasn't in an org where that was really celebrated or anything. When you've been there from the beginning, you built the organization; you're like a senior director and everything. It feels like a long way to fall. And I wrote that post for him. And he did end up going on to be an engineer after that. And he was so much happier. But I think he was surprised at how he didn't fall at all. He actually probably had...I think the engineers had a higher opinion of him afterwards when he was one of them again. And he still had this vaunted voice because he could speak to how the system had been there since the beginning. And he basically got to look around and look out farther than the engineers who were heads down every day and go, "This is going to bite us. I'm going to take a small team. We're going to do this forward-looking security product." I don't want to identify details, but that for me really just kind of cinched...It was like the more we can strip hierarchy out of these discussions; the healthier everyone's going to be because we're just monkey brains. And the monkey brain in our skull hates losing hierarchy, hates losing power or stance or anything. And I think that the thing that you learn after you've been a manager is a lot of it is just the wizard behind the curtain, the idea that you have more power as a manager. You have more of some types of power, and you have a lot less of other types. And you're just as constrained as the engineers but in other ways. And the path moving forward is not to dominate people or be above them but to combine your powers for good and self-sort to find a place that actually gives you the most joy. CHRIS: It's a wonderful philosophy. And actually, a thing that you said in there really stuck out to me, which was you wrote that blog post as a gift to someone, and that is such a kind thing to do. And it also, again, reflects what I see in your work overall. You're really clearly leaving a trail of breadcrumbs behind you to help other folks that are traversing a similar path by questioning aspects of it. Or how do we do this well? Why is everyone sad, and why is it bad? And so again, I so appreciate all of that work that you've done. CHARITY: I think that that comes from my lifetime in the trenches of operations. [chuckles] Ops is notorious for the pain that we bring upon ourselves and try to solve. But I would just like to add a pitch out there for other ops engineers of the world and our colleagues. I was fortunate enough to rise up through the ranks in organizations that really respected operations. We always felt we ruled the roost. We felt like we were way above all the other developers. We got to say what went into production and what didn't. And I feel like ultimately...if you have to have an imbalance of power, I think that's slightly healthier than the developers ruling the roost. Ultimately, there shouldn't necessarily be any imbalance of power. But I just want to pitch it; this whole no-ops thing really got my goat for a while there because operations is just the engineering workaround delivering value to users. I think the second wave of DevOps is now about okay, software engineers; it's your turn. It's time to learn to write operable software. And so I just wanted to throw in my hat in the ring for all the ops people out there. You're just as good. You're just as good as anyone else. [chuckles] CHRIS: I mean, it's sort of a theme that I've seen in your writing of everybody's doing good, important work and breaking down hierarchy and just collaboratively moving in the same directions and trying to choose the right North Stars to aim towards. And yeah, it's all fantastic. And so with that, I think we probably reached a perfect spot to wrap up. But Charity, if folks want to keep up with more of your work online, where are the best places to find you? CHARITY: My blog post is at charity.wtf, and I'm @mipsytipsy on Twitter, and of course Honeycomb.io and our blog. CHRIS: We will include links to all of that and many of the blog posts, and other podcasts interviews that you've been on, and a bunch of just various things that I collected as I was preparing for this episode because, again, you've produced such a wealth of information on the internet that I want to point as many folks as possible towards those things. But yeah, thank you so much for taking the time. CHARITY: My pleasure. CHRIS: The show notes for this episode can be found at bikeshed.fm. STEPH: This show is produced and edited by Mandy Moore. CHRIS: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review on iTunes,; you as it really helps other folks find the show. STEPH: If you have any feedback for this or any of our other episodes, you can reach us @bikeshed or reach me on Twitter @SViccari. CHRIS: And I'm @christoomey. STEPH: Or you can reach us at hosts@bikeshed.fm via email. CHRIS: Thanks so much for listening to The Bike Shed, and we'll see you next week. All: Bye. Announcer: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success._

FreeBSD Performance Observability, Advance!BSD thoughts 1/2, Lumina Desktop Maintainership Change, How to Handle Secrets on the Command Line, Like NetBSD DragonFlyBSD Now Has "COVID", and more. NOTES This episode of BSDNow is brought to you by Tarsnap (https://www.tarsnap.com/bsdnow) Headlines FreeBSD Performance Observability (https://klarasystems.com/articles/freebsd-performance-observability/) Advance!BSD – thoughts on a not-for-profit project to support *BSD (1/2) (https://eerielinux.wordpress.com/2021/06/20/advancebsd-thoughts-on-a-not-for-profit-project-to-support-bsd-1-2/) News Roundup Maintainership Change :: Lumina Desktop Environment (https://lumina-desktop.org/post/2021-06-23/) Study the past if you would define the Future (https://lumina-desktop.org/post/2021-07-01/) How to Handle Secrets on the Command Line (https://smallstep.com/blog/command-line-secrets/) Following NetBSD, DragonFlyBSD Now Has "COVID" (https://www.phoronix.com/scan.php?page=news_item&px=DragonFlyBSD-COVID) Tarsnap This weeks episode of BSDNow was sponsored by our friends at Tarsnap, the only secure online backup you can trust your data to. Even paranoids need backups. Feedback/Questions Jim - freebsd kde (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/412/feedback/Jim%20-%20freebsd%20kde.md) michal - zfs question (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/412/feedback/michal%20-%20zfs%20question.md) tim - lumina and snapshots (https://github.com/BSDNow/bsdnow.tv/blob/master/episodes/412/feedback/tim%20-%20lumina%20and%20snapshots.md) *** Send questions, comments, show ideas/topics, or stories you want mentioned on the show to feedback@bsdnow.tv (mailto:feedback@bsdnow.tv) ***