POPULARITY
On this episode, my guests are Jesse Mann (editor-director) and Tyson Sadler (director), the brains behind the documentary The Last Tourist. Jesse is both a picture editor and director whose professional work has spanned commercial, tv and film projects. The Last Tourist is her second feature film as editor. Her first film, as both editor and director, Material Success, screened internationally and won the Audience Choice at the Canadian Film Festival and Best Film at the Canadian Film Festival (2012). Most recently, she both directed and edited the online horror mini-series “The Confinement” (2021). She is a member of the Directors Guild of Canada and an associate member of the Canadian Cinema Editors. Jesse has an B.F.A. from Ryerson University's School of Image Arts.Tyson Sadler is an explorer, traveller and award-winning filmmaker. He has directed video content and documentaries for The New York Times, Associated Press, and The Huffington Post. His films have been screened at festivals around the world including Tribeca Film Festival, South by Southwest, The Edinburgh Fringe Festival, and the Toronto International Film Festival. An early pioneer i virtual reality documentary, he has worked tirelessly to use cutting edge technology to tell stories that build empathy around causes such as climate change, forced migration, and human rights.Show Notes:The Film's InceptionIndustry PolarityRegulating TravelAddressing the Root ProblemsGreenwashing Animal and Child Welfare TourismHow the Wealth Gap Increased During COVIDWhat was Left on the Cutting Room FloorHomework:WATCH THE FILM HEREThe Last Tourist Film Website | Instagram | FacebookThe Last Tourist TrailerJesse Mann's Website | InstagramTyson Sadler's Website | InstagramTranscript:Chris: Welcome Jesse and Tyson to the End of Tourism Podcast. Jesse: Thanks Chris for having us. Tyson: Thanks for having us. Chris: Thank you so much for joining me today to talk about your documentary film, The Last Tourist. So, it was released in 2022 to great reception.And as you might imagine, many of my listeners have pointed me in its direction. And after watching the film, I found myself really grateful that people finally approached these themes in the medium of documentary filmmaking and with what seemed to be a budget to do justice to what those themes confront. And so first I'd like to ask you two how The Last Tourist got started and what the inspiration behind the film's creation was and how did you decide to write a treatment for it? Tyson: Ultimately, you know, I think the inspiration for The Last Tourist came from a combination of personal shared experience and a shared passion for travel and particularly responsible travel.You know, in early 2018, I was approached [00:01:00] by the executive producers to write a treatment, for a short film around responsible tourism in the country of Peru. And through, you know, some follow up conversations, we sort of quickly realized that we had an opportunity because, a large scale documentary, on the subject of responsible tourism just didn't exist yet. I mean, we have wonderful films out there which challenge our conversations with our relationship with climate change and our relationship with our food source like An Inconvenient Truth or like Food Inc., but we didn't at that time yet have a documentary which really challenged our perceptions of the global impact of the travel and tourism industry and so over conversations with with our team and the producers we quickly realized that we had a unique opportunity to make "An Inconvenient Truth" for the travel industry and in early 2018, we seized on that opportunity to explore the positive and negative impacts of tourism on destinations.Jesse: And I think just to add to what Tyson was saying, I think originally [00:02:00] it started off as a short project and yourself and the executive producers who brought us together kind of pushed for it to become the large scale project that had ended up being filmed in 15 different countries or 16 with 400 hours of footage.So, sometimes you don't know, especially myself as a co-writer and editor, when you come on to a project, in the initial stages, you think, "Oh, you know, this is wonderful. It sounds like a great project." I had been an avid traveler, lived in a few different countries over 20 years and I thought, yeah, this is a story that needs to be told, but I will say in no way in the beginning, did I ever think it was going to be, and I think Tyson didn't either, was it going to be such a huge project. And you comment, Chris, on the expansiveness of the subject and the different kind of facets of the tourism [00:03:00] industry we were able to look at.And really I have to say that I'm happy that we got to touch on all those points and to the chagrin of my personal sleep and Tyson's as well, but it started off small, I have to say that, and it grew into something tremendous that I think we're very proud of.Tyson: It really was a natural progression of our desire to raise awareness about responsible tourism and its consequences. We had our world premiere, honestly, I think it was about two years ago now at the Vancouver International Film Festival, two years ago. And a little over two years ago, a little over two years ago.And the film is still doing a festival run. Just last weekend, Jesse was at the the Innsbruck Film Festival and it's still been getting a wonderful response, great conversations around it. And we're streaming on platforms around the world, you know, Crave in Canada, Hulu in the U.S., Amazon Prime in other countries. I was just in French Polynesia and was able to find our film on Amazon Prime, which was really delightful [00:04:00] to see that. Jesse: Yeah, it makes us happy as filmmakers to know that the story that we spent a lot of time trying to create in a way that we thought would connect with audiences worldwide is actually getting out there. And so it's really nice to be speaking to you on your podcast as well and kind of extend that out to potentially more people who haven't seen the film or some who just want to talk more about the topics. Chris: And I wanted to ask you two, given the fact that the film was released, you know, still very much in a pandemic during the, the COVID 19 times what the reactions were given the fact that tourism had ground to a halt in that time you know, I received a lot of Mail regarding your film, like, ah, you have to check this out.You have to watch this, right? And so a lot of people really excited about the project, about the film. But then I guess I'm also curious about[00:05:00] if there was much of an industry backlash in regards to the degree of sincerity perhaps around which the film exhibited these kind of deep and sometimes dire consequences that visit themselves on places and people in the name of tourism.Tyson: Yeah. I think the reactions to the film have been in my experience, almost entirely positive. But people don't come to me with criticism, they come to me with congratulations, but I think there's a lot of individuals in the tourism industry and sponsors that have really welcomed our film, The Last Tourist, as a necessary and eye opening piece of work.It sparked a lot of interesting conversations and prompted the industry to, in many ways, I think, reevaluate some of their practices. You Jesse: If we back it up to when we were just when we were meeting with all of the different professionals that we interviewed across the different facets of the tourism industry, I mean, when you and I were writing the story, there were so [00:06:00] many different conflicting opinions on how to solve one issue that we still come across that sometimes when we meet those industry experts outside of, let's say, film audiences.Because When we were writing it, we had to kind of decide which side of the coin we were going to follow through our story with, and whether it's a topic of let's say regulation, that was a topic that Tyson and I had a lot of conversations about when we were making the film because the industry in itself is kind of very polarized in terms of regulation.There's some strong proponents within the tourism industry who want regulation. And then there's some who are very against it. And Tyson and I had a lot of conversations about that. And I think we still do when we meet certain industry professionals out there and I think that's a really important topic as well.[00:07:00] Now that you are bringing up, post covid and the pandemic where we saw kind of what happens when things are shut down. You know, we see both the positive and the negative. I mean we mentioned it in the film, but almost every single person that we interviewed in our film lost their means of income during that time. Gone.And at the same time, the world experienced kind of you know, a refuge from, you know, airplane emissions and damage to destinations. And, you know, these were calculable things like we could see that this was a definite impact. So, I think there's these kind of topics and these conversations are where we see a lot of polarity.And I think that we tried our best to focus on the people and the developing nations in our story. But, there are a lot [00:08:00] of stories that were left on the cutting room floor and so it is good to discuss them after in a podcast like this. I know that's kind of a long winded interjection, but I do believe that we do have some polarity for sure Tyson and I have come across, but just not in terms of what Tyson is talking about, in terms of not not so much with the audiences It is more a bit in the industry. Tyson: I'll share an observation. During the process of creating this film, we interviewed literally dozens of travel experts, you know from academics to tour operators across the industry.It's universally recognized that responsible tourism and sustainable tourism is a good thing. But then when we dive a little bit deeper I found a very fractured kind of industry because everybody believes that they know how to do responsible tourism, right. And everybody else, often, isn't executing responsible tourism correctly.And so, on a foundational level, sure, we all believe in responsible tourism, but I think, the approach at how we get there isn't universally accepted, throughout the industry, and I think [00:09:00] that that's, on a basic level, we kind of explore those themes in the film as well as we interview people who kind of take different approaches to how to engage responsibly with tourists and host communities.Jesse: And we had to make some hard decisions, obviously based on the research that was coming in on what side we wanted to promote in the story. But sometimes I think it's fair to say, Tyson, if you agree with me that we did try to leave it a bit more open for audiences to try to make their own decision based on what we had learned, because it's not always an easy answer for every situation.It's quite nuanced, I think. Tyson: Absolutely. Chris: And you mentioned Jesse, regulation. In terms of the people you interviewed and your research, generally speaking, what does that look like, or what might that look like in particular locales or within the industry?Jesse: Well, I think we've seen quite a bit of it post pandemic because where we thought there was going to be [00:10:00] an ease back into traveling that has not happened. All the reports are coming back in that tourism is back up in droves and a lot of the same problems, if not more are back with travel again.And so we've seen places like Machu Picchu has started regulating the amount of people that are able to visit annually. Worldwide, there's different popular tourist destinations who have decided to limit the amount of tourism and tourists that come to certain destinations.You know, I think when it comes to destinations, we just have to be really careful that it doesn't become a tax that is elitist because that happens a lot with regulation across the board. You've seen it with cigarettes, you've seen it with airfare as well. You're paying a higher price for travel. And although I do believe personally and Tyson, you [00:11:00] can speak to this as well, but because we haven't discussed this recently, but when you put a strong tariff or tax on travel, I think it is important, but I think it has to be calculated so that it's not elitist because if we only have people traveling in the world who can afford it, if it becomes so unaffordable, this is totally against the core value of travel, which is to unite all people of all different socioeconomic statuses and really connect with different cultures and people, and if it's just suddenly becomes for people who can afford a really overpriced flight, I think that will completely take away the value of the core value of travel.But I do strongly believe that, and this I think is a very controversial thing to say. I do believe that things have to be regulated and there has to be something done, especially about the amount of flights that are happening on this planet. Because, you know, for instance, I don't know the statistics for Canada, but I was looking at it the other day for[00:12:00] the UK.And I think something like 70 percent of all flights in the UK are taken by 15 percent of the population, and I don't doubt that that's very similar for a lot of the Western world without knowing all the statistics specifically, so I think that that has to be really taken into consideration when putting out regulation in terms of flights or destinations and such.Chris: Yeah, I've thought about it quite a bit as well, and the elitization of travel, and this understanding that, well, we remove subsidies, for example, for air travel or other things, that we'll just have rich people flying around the world, which is more often than not the case already. Most flights are undertaken for business travel and that's a huge thing that I think most people don't know about, but instead of getting people with corporate backed funds or just rich people flying around the world, we're obviously looking for people with strong principles and good behavior to be the ones [00:13:00] traveling.But then how do you measure that? How do you quantify that in a way that honors the kinds of worlds we want to live in? And it's not just this like, oh, it's impossible. Sure. And we could take that understanding that to quantify such a thing would be next to impossible, but then it could also open up these kind of doors of imagination where in like what kind of infrastructures would we want to put in place that were not maybe Government sanctioned or not maybe top down, but actually from the bottom up, creating these kind of international or intercultural bonds and practices between people that would allow not just for the kinds of people who locals in quote unquote "destinations" want to receive to be able to travel, but then also to create the conditions whereby that kind of, for lack of a better word, behavior or principles could be deepened in the world.I mean, it'd be extremely difficult to, [00:14:00] to manage, but How would you quantify that? How would you measure that? Jesse: Yeah, and I think that's such a, such a complicated issue and I love what you're saying though. I think it's absolutely wonderful. And I do think it's possible to move in that direction.I'm not sure I have the answer for that and maybe Tyson wants to speak to it, but I also want to say that something that I think could be part of the solution is also promoting train travel as well. And I know that that kind of that stops at domestic. But at the same time, you know, I think governments should be allocating a lot of funds to push train travel to create and develop train travel and to also subsidize train travel more because it's just a wonderful way to reduce emissions and to actually continue allowing people to travel. Tyson: Yeah. There are some countries here in Europe, Jesse, and I think Europe is just far beyond a lot of the rest of the world. Canada, I think we're still living in the dark ages, you know, when it comes to long [00:15:00] distance train travel.Jesse: That's a shame. It hasn't been made a priority. Tyson: It just doesn't exist.Chris: Well, I'm curious also for you two how your own travels in your lives might have come to bear on your production or pre production and research in the film. Maybe you could tell us a little bit of how you've seen the world over the course of your life and your travels and if that came to bear at all in the making of the film.Tyson: Personally for me, I traveled extensively, for 20 years prior to embarking on this film. I worked in the travel industry. I worked for an airline for many years. I was also a journalist and I've done work for several large publications, which allowed me to travel to very, some very remote and exotic destinations around the world.This travel really exposed me to the beauty and the wonder of the world. But also the environmental and the cultural challenges that a lot of these destinations face. You know, the experiences, with travel for both work and for pleasure, I think they served as a [00:16:00] catalyst, for me just to delve deeper into this topic.And so when I was approached about directing and writing a film on this topic, I don't think there could have been a better fit. Like it just, everything just kind of fell into place and I just felt like I was really well poised because prior to that I had traveled to 70 different countries in a number of different capacities.I had personally engaged in a number of behaviors that we are critical of in the film in terms of animal interaction, you know, interaction with other cultures. And my travel through two decades was really an eye opening and learning experience for me. And what this film really is, I think for me and in many points in the film is a reflection of my past travels and things that have been illuminated in my mind in terms of what are some important aspects that we really need to address if we want to be more conscious and responsible travelers.Jesse: And I think from my end I surely [00:17:00] had not traveled to as many destinations as Tyson. But I did indulge my wanderlust in my twenties and thirties. I lived in a few different countries, Holland, South Africa, Germany, Canada. And I think that the absolute best aspect for me personally of travel is meeting the people.I absolutely love meeting people in different cultures and seeing the differences and the similarities and learning the wisdom as much as I possibly can in such a moment of my life, when I'm in a destination, if it grants me such a gift. And I think from that, I still do have such a deep, deep gratitude to my privilege and being able to travel in and meet such people.I've made such beautiful friendships over the course of 20 years and many I still keep in touch with across the various places I've lived. And I think that when you come back, [00:18:00] perhaps from living in different cultures in different countries and you come back to your home, the place you were raised in and where you were born and where you were raised culturally, and you can note the vast differences, and it's not a negative thing, but you start to pick up on the ways in which many people can see different cultures or different countries in a way that is completely disconnected from the actual truth and and this knowledge often comes with privilege.I was privileged enough to be able to travel to these destinations to meet certain people and to dissuade certain prejudices, but not everyone can. And so part of this film, I have to say, ironically, is that if you can't travel, this film is also a wonderful film to watch because it's something that you can meet these people [00:19:00] in these destinations.You know, there's so many places that I've been to also that are so many places I have not been to that were featured in the film as well. And with documentary filmmaking and making, and especially as an editor, I really do feel like I get to know these people so well, and I don't think that's just exclusive to myself.I think audiences, hopefully, if I've done it correctly, if Tyson and I have, have made the film correctly, that these people's hearts have come through in the film. And, this is something that Tyson and I really worked hard on. And I'm sorry if I'm deviating from the question a bit, but it does connect very deeply to my love of travel is is people and also my love of editing people and so it was something we worked really hard to focus on is how do we give these people a podium. They have their voice there. They're saying their truths. How do we give them a podium [00:20:00] and as the white filmmakers who are coming into different countries, how do we take a step back and really let them preach their wisdom? And so it was a balance for us really in finding the right voices, obviously speaking to the right professionals and speaking to professionals within different cultures that you see featured in the film and really making sure that their knowledge, their voice, their direction was giving us direction on how we edited and formulated the story and stories of everyone that we met in the travels of the production as well. Chris: And on that note, I'm curious for each of you, what was the most important topic or theme that came up in the film for you personally?Tyson: For me, maybe it's an echo of Jesse's sentiments there, but really it comes down to the impact on the communities that so graciously host us when we when we do travel. Travel you know has you know has an encyclopedia [00:21:00] of challenges that come with traveling responsibly unless we address the people. And I think we address this and the the film addresses this, but we can't begin to you know to start focusing on the environment unless we first focus on the communities who are there and helping them emerge from poverty. We can't begin to focus on how we can solve the problem of poaching in the African savannah unless we first address the root causes of why it's there and how it affects the host communities and how poverty affects people's decisions to be involved in this trade. Jesse: That was something that Jane Goodall was was really adamant about speaking in the film about as well Tyson: Yeah, and so to answer your question Chris, I mean ultimately it comes down to the communities and the people, the humanity of it all. First, we need to address the human aspect and then we can address the secondary aspects of what it means to be a responsible traveler.[00:22:00] Jesse: So for me, personally, the most incredible voices in the film that when they came in, I just thought, "oh, wow," were definitely Lek from Elephant Nature Park and one of our contacts in Kenya Judy Kefergona, who ended up being one of the main speakers and narrators throughout a huge portion of our film.These two women were just heroes. Like just with Judy, with the words that she was able to express for the people working in not just only the tourism industry in Kenya, but the people of Kenya who were vulnerable to unethical tourism, and then speaking to so many different subjects that were connected to so many different stories worldwide with such eloquence and such power and such knowledge [00:23:00] and encourage really, I think. She spoke about these topics. I was just enamored, just blown away and so grateful for her coming onto our film and Lek Chailart is just a modern day hero. She is the tiniest woman with the biggest heart who when I saw her story in our film against the backdrop of these beautiful giants, these elephants that she has rescued. And essentially she's an elephant whisperer in the truest sense of the word. And it's just such a magical thing to see.Listeners won't be able to understand this. I'm getting goosebumps when I just talk about her and this was the kind of thing that when Tyson and I were looking through footage and looking through stories that we were looking for, "Oh, that's the moment, ah! the moment. I got the goosebumps. That's it. It's there. And we found it in the stories, not just these two women, everyone you see featured in the story and some that were left on the cutting room floor, unfortunately there that gave us beautiful insight that we still used in crafting the story, that just wasn't able to [00:24:00] make it into the actual film for timing constraints.But really, I would have to say that these two, it's not specifically the stories, although both of them touch on really important stories, including child welfare and animal welfare, which to me, those two stories were really close to my heart. And we felt really strongly that they needed a really important part in the film.But it was really these two women who I just felt so grateful for in the film. Chris: I mean, the, the chapter on animal welfare was quite devastating for me in part because it had reminded me that in my early twenties, I had visited Thailand and I had gone to an elephant sanctuary. Not a zoo but a sanctuary, and had the opportunity to ride an elephant for a short time, and I felt really uncomfortable probably because I was on top of an elephant, just the kind of immediate awkwardness of such a thing, having never seen one in person, having never [00:25:00] experienced that before, but also kind of like, "what am I doing here?" And so, that part of the film really kind of opened up for me why I felt that perhaps existential discomfort, not just about being on top of an animal of that size, but in the context of the dynamic, you around how that was happening and why that was happening and not having the context for it so long ago.And of course, this is one of the things that we touch on in the episodes, in the interviews, in the podcast, is how can we come to understand these things when our visits are so short in these places, when we are only in a place for just a very brief time and there's really no context for the history and the culture and the political dynamics that surround these things.And then, most travelers, most tourists just end up leaving and the consequence of one's presence on the scene is kind of forgotten, at least by the tourist or traveler. Yeah, so thank you for [00:26:00] for that. Jesse: That question did come up actually at the recent screening of the film in Innsbruck at the Nature Film Festival, whereas someone in the audience asked very specifically, even though it would be great to stay at a destination for a long time, most people are, saving up for short travel when they have time off work. And they need to take this kind of tour because they can only afford this one. And what do you say to those people who want to travel better, but feel like they're kind of stuck or don't know where to go. But I would say in short, I think that touches on an issue that we have in the modern world with a lot of things. We are all overtired, overworked, and we don't have time to be ethical, we don't have time to do the research, we don't have time to investigate, if anything is against our core values, and we don't have time to and I get that, you know, it's not easy for everybody.And if you do have the [00:27:00] time and you are able to really do a deep dive then you're very privileged, because you have the time, which just means that you have the money. So we did try to give a lot of smaller tips at the end of the film in terms of how you can do smaller acts of, of kindness and of conscious traveling, to travel local, to put money into local economies to make sure that you're not requesting your sheets to be washed every day, making sure that you're traveling using reusable, to make sure that you're and I guess this goes into a little bit more of the extensive questioning is to ask the destination that you're traveling to what their policies are in connection with the locals, in connections with the environment. Unfortunately, there is a lot of greenwashing in tourism. And you know, that's in all industries right now worldwide. So it's not going to be so easy for someone who isn't an investigative [00:28:00] journalist to really find out the truth behind it all the time, but we can do our best.And there are quite a few links on the website thelasttouristfilm.com where you can see some of the different organizations that we spoke with that have lists upon lists of different collaborating companies that you can look to that can can show you places that are not greenwashing or that are working with locals, but back to the idea of taking the time and it is about taking the time, whether it's an hour even, just to take the time and do a bit of research and this especially comes back to the topic of child welfare and people who are still, en masse, volunteering worldwide to orphanages and orphanages where children have existing family members and that these orphanages have become of monetary value to developing nations [00:29:00] because they make money and it's really easy for us to pass judgment on places that do this, and it's definitely wrong, but if we as travelers are going there and saying with our travel dollars, this is something we want to invest in, then people in developing nations are going to say, this is how I make money.And it's the same with animal welfare. You pay for an animal sanctuary that you haven't done the research on. They're going to keep perpetuating that cycle. It's going to be the same with child welfare. So, if you do want to be ethical, you have to fit in a little bit of research in your travel time. And that's it. And I know it's not so easy for everybody.Tyson: Yeah, I'd like to expand a little bit on a theme that sort of both of you have talked about. Jessie, you had mentioned greenwashing. Chris, you had mentioned that you had been to an elephant sanctuary in Thailand. One of the most memorable takeaways that I had from this film is just the massive amount of greenwashing [00:30:00] and deception that exists within the industry.I came to know a lot of companies that appeared to just be normal travel companies doing the same thing that has been going on in the travel industry but marketing "green" although nothing really changes. It's just their marketing strategy that's changed. If we look at, for example, in Thailand, there are a number of places that call themselves an elephant sanctuary, an elephant retirement home, an elephant rescue center. And we can't rely on those names anymore to know that the service provider is giving us an ethical experience. We really need to ask deeper questions. It's the trend in the industry now to use this type of language, " sustainable," "responsible," "eco," all of these buzzwords.And I've just come to find these completely meaningless, in those terms. We really need to ask some deeper, more challenging questions of these experiences and the tour operators and the service providers to actually know whether what they're doing is ethical or not because it is very easy to call yourself a [00:31:00] sanctuary, when in fact, it's the complete opposite. If it truly was an elephant sanctuary, we wouldn't be able to ride the elephants, in that place and they would be providing them a life of dignity free from exploitation. And it's the same with children. Calling these places orphanages ,I think it's a misnomer. It's incorrect. You know, 70 percent of children who live in these quote unquote "orphanages" actually have at least one living parent.But it's all under the guise of trying to gain sympathy from the traveler. "Oh, come see an elephant at a retirement home. Come see a child at an orphanage." It's just a piece of marketing that doesn't reflect the truth. We have a lot of companies doing the same unethical practices they've been doing for decades.However, they've just really changed their marketing to appear more green.Chris: Yeah, it seems to be a never ending cycle where responsible isn't good enough, that now we got to be regenerative. So many of these words just end up becoming, marketing tools. Tyson: I'm hearing that word a [00:32:00] lot more these days as well. Chris: Yeah, so how do we proceed, not just with a degree of dedication towards research and, and and planning, but also deep discipline as to how these words are unfortunately, as you said, becoming kind of meaningless in their significance..I want to take a little turn with you both to ask about what happened as the film was going through post production and its release because most of the footage that you have is quote unquote pre pandemic and the COVID 19 pandemic hit, obviously, in 2020 and from what I understand your team was entering into the post production process and, of course, tourism ground to a halt almost completely, worldwide. And throughout the film, there are people that you two interview that contend with the consequences and context of tourism and look [00:33:00] to a more honorable path that it might take. But I'm curious in regards to the people that you did interview who had found a degree of success and perhaps within a more small scale, a more honorable way of doing tourism that those flights, those trains, the reception of people in their villages went from a hundred to zero, basically overnight.So much of the dire consequences of tourism revolve around or end up as exile, local people can't afford to produce food anymore in their places. And the education systems kind of move them towards getting jobs in the next city or even in other countries, and I'm curious in the context of the film and I guess the treatment that you put forward, you know, never perhaps thinking that something like this could happen. What was the fallout among the people that you interviewed in regards to their understandings of tourism and if it was [00:34:00] still this kind of for lack of a better word, golden goose or calf or sacred cow that they could rely on for the rest of their lives.What did you hear kind of in the ether as you were doing that post production? Tyson: I think it substantially expanded the wealth gap. I think during COVID the people that were hit the hardest are the people in the tourism industry that often are paid the least.Depending on what their jobs were, they were the ones who were suffering layoffs and they were the ones whose businesses couldn't afford to sustain themselves. And so I think, for the most vulnerable populations, the pandemic was absolutely devastating.People couldn't afford to put food on the table and pay rents, not just, I think, in developing countries, but even in the developed world. Airlines were laying off massive amounts of employees. Hotels were cutting staff. This was a global challenge that affected everyone. However, at that same time, we did hear stories that, you know, CEOs of major airlines were taking million dollar bonuses[00:35:00] for cutting costs. And that was an observation of mine, through that experience, you know, that the people who needed tourism the most for daily sustenance and to put bread on the table were the ones who were absolutely the most affected and the people kind of at the top of the tourism industry were still fine and they were also taking bonuses, which really bothered me just because the wealth gap just seemed to get further and further apart through that. But we do know, with the stories, that were featured in our film... there's a wonderful lodge in Ecuador, a homestay that we look at and they lost their income during that time.The elephant sanctuary in Thailand, the one that does do great work, not one that greenwashes an elephant sanctuary but they lost a lot of revenue where it was very challenging to feed the elephants and to house the elephants because they didn't have that revenue coming in to support the project. Jesse: We were hearing in in Africa, right?Some of the rangers [00:36:00] were who were placed in defense of the elephants and animals and rhinos and such were just gone. They just weren't there anymore at that time because if they're not getting paid, they can't stay there. And so poaching also went up in those areas as well.But you know, I think that in terms of what the positive was, I think a lot of people were hoping that post COVID that there was going to be hope for change, for renewal for doing things differently. And I think this was in when we were in post that this was maybe, potential to show people how things can be different by showing kind of the polar opposite and the effect and also showing how intrinsically connected people's livelihoods are to this industry and how vulnerable they are to massive change like this. I mean, the Dominican Republic that has [00:37:00] almost 90 percent of their GDP related to tourism? Tyson: Island nations are the ones who are most affected by tourism. You know, the Bahamas, the Dominican Republic. The Caribbean islands specifically. I don't know if it's as high as 90%, but the vast majority of their GDP from island nations comes from the travel and tourism industry, and they were certainly hit the hardest during the pandemic. Jesse: And so I think that we can see from that, I think the hope for us during post production was, okay, you know, this is a horrific blow to the people most vulnerable in this industry, but if we can get this message out after the pandemic, perhaps this could be something that could really change. And to be fair, change is slow sometimes and I mean, it's wishful thinking that post pandemic right after that suddenly everyone was going to become ethical travelers but I see so much potential just in the discussions online the people that we're [00:38:00] meeting at different screenings, the interest that we're getting worldwide, just to have these conversations and see that there is an interest in change.And when you talk about moving change, it does often come from the people, and I think, across the world over the past few decades, we've all been seeing again, I think our power as people, as citizens, as individuals, and the power that we have to come together over certain issues that we feel need radical change and even if it is slow change, I do see the inklings of that change happening within the tourism industry and I think it's really positive.Tyson: You still have a lot of work ahead of us. Chris: Amen. I think that's really, really important and, and perhaps fits properly inside of the context of the dominant culture, at least of North America or the quote unquote West, wherein, the pandemic also produced a deepening of the [00:39:00] culture of "everything now."Suddenly it was like, "okay, well, I can't go out here, so everything has to be deliverable, and at the tip of your fingers, right? And what might come with that is this notion that, we also expect social change to happen overnight. Right? And that it might be overshadowed by this kind of dominant culture of wanting everything now and also the unwillingness to do the necessary work, which is sometimes generational.Right? Not just a week or a month or a year, but generations.In that regard, the themes of the film are extremely broad and you go into a lot of detail and depth with each. So I'm very grateful for that, but I'm also curious what might've gotten left out.What might, one day end up on a director's cut of the film. Jesse: Oh, so much. Tyson: Ask the editor. Jesse: Oh. Well, I mean, you had so many stories that you brought [00:40:00] to the table that were beautiful, beautiful stories. Forgive me. I don't remember his name and you'll remember Tyson.But one of the. Tyson: I know who you're talking about. Jesse: Oh, do you? One of the men you interviewed at the UN had a really beautiful story to tell. And then we also had a really beautiful story from Costa Rica. An animal welfare and environmental story. Oh, my goodness. There were so many stories.Tyson: We had a wonderful story from Cusco Peru with an organization that was putting the first female porters on the Inca Trail and really fighting for gender equality on the Inca Trail.Jesse: What was that company's name again, Tyson? Well, the reason why I bring it up is because it's so unique that I just think, "oh, we should definitely highlight that to listeners" because they were taking female porters and they were reorganizing the whole industry based on their [00:41:00] precedent because they were treating porters with dignity, with safe conditions, with, valuable wages and this is something that's just not done across the industry on the Inca Trail with the male porters and there was some really, really horrific footage that we came across of the way these porters were living just not too far from where the tourists were sleeping comfortably in their tents. And this is a wide practice across the industry and Miguel from this company Evolution Treks and the gallon, do you remember his full name, tyson? Tyson: Miguel Angel Gongora Jesse: from Evolution Treks, yeah, a wonderful, wonderful man who was really, really passionate about changing the industry. And we spoke with many of the female porters, and yeah, that was a story that we were really, really sad to lose that [00:42:00] story. Tyson: Yeah. Yeah. With only 90 minutes, you know, we yeah, really had to make some challenging decisions on what to be included in the film and I really feel that a 90 minute film can scratch the surface, it can spark some interest in a number of these themes, but certainly, and maybe that's next steps for us, but I certainly think that what this the subject matter, deserves is a series, you know, cause we could dive much deeper.Jesse: Tyson and I have been talking about it. Tyson: Yeah, we, we can dive much deeper into each of these subjects and create a 90 minute doc on each of them. And so, at some point in the future, we'll be making some pitches and writing some additional treatments into how we can make a six part series and expand on the themes in this film, you know, from the environment to gender inequality to animal welfare and everything deserves a lot more time, but hopefully, what we've done with this film is just been able to spark a number of conversations and inspire people to go and do some additional research [00:43:00] into how these themes impact these communities.Chris: Yeah, well, thank you both so much for that. I have a lot more questions, but maybe that'll also be saved for a sequel. Jesse: Yeah, we'd love to speak to you again, Chris, if you'd like, in the future. Chris: That'd be great. Before we finish, I'd like to ask you what might be next for you two in your respective filmmaking and writing lives?Tyson: My world currently revolves around photographing wildlife particularly large wildlife. Yeah, I've been spending a lot of time traveling recently and in the ocean. I mentioned to you, I mean, tomorrow I'm hopping on a flight and I'm heading to Norway to go photograph large marine mammals in Norway. So that's that's where my life is taking me right now is in pursuit of wildlife filmmaking and photography. Jesse: Although I wish I could, I could say similarly, I right now my life is a little bit calmer. I had a baby 20 months ago. So, I've been on an extended maternity leave [00:44:00] and I'm currently just in development of projects for the future.I finished up this project in full right before my child was born. And I still have a few projects that were finished up recently, but as of the moment I'm completely in development and yeah, I really look forward to developing something with Tyson in the future in regards to what he was talking about a series on travel and on a lot of stories that we were so passionate and so blessed to come across, but that we just didn't have the ability to give the podium to in our film, but yeah, so I'm really excited for the future and just coming from this festival that I was at recently, I'm just so invigorated with the energy of the audiences and the passion to travel better and to demand better travel from the industry and from travelers as a whole. So this just really gives me a lot of hope. Chris: Well, congratulations on your [00:45:00] motherhood. Thank you. And having a little one in your life and this work and Tyson and the opportunity to be able to travel as you do, and to try to honor the lives of those beautiful four leggeds and no leggeds and the tailed and finned ones. So finally, how can our listeners watch The Last Tourist? Are there any screenings coming up? Jesse: Definitely on the website, thelasttouristfilm.com, right when you go to the webpage, there's a whole list of all the different avenues you can watch the film.But maybe Tyson could elaborate more. Tyson: Absolutely. Yeah, you can connect with us on Instagram at Last Tourist Film, as well. We're just kind of wrapping up the fall festival season before we do some additional screenings next spring. We just wrapped up in Germany and Iceland.I think streaming is the best way to find us, in Canada we're streaming on Crave. In the United States, we're streaming on Hulu. Delta Airlines, Emirates Airlines. In other countries around the world, you can also find us, I [00:46:00] know in France, in French Polynesia, Hong Kong on Amazon Prime. And there's a number of other markets that the film will be opening in very shortly which is really exciting.It's not available everywhere, unfortunately, around the world, but if you have a desire to see it, you know, please get in touch with us and we'll do our best to make sure that you have an opportunity to see the film. You can host a private screening. There's a number of ways that you can see this film, but please let's keep the conversation going.Let's get in touch. I'd love to hear from a number of the listeners, and let's find a way that we can allow everyone to see this film. Jesse: And what's been really exciting is we've got a lot of interest from schools. So, educational screenings have been happening all across the world, and that's just been amazing.Those are the best for us, I think. And I would just say, if any of the listeners have any further questions for us, we're always open to taking questions and chatting. Personally, Tyson and I can both be reached at our Instagram. So you can I think Tyson's is @TysonSadler, and mine is [00:47:00] @JesseMann, two S's, two N's. So if you want to leave that for your listeners, they're welcome to contact us. And anyone who's listening can contact us directly that way, who want to host a screening or find out ways to watch. Chris: Absolutely. Yeah, I'll make sure that all the websites and handles are available for our listeners at theendoftourism. com. And on behalf of them on behalf of myself, I'd like to thank you both for joining me today. Your film is incredibly inspirational, necessary, and deeply important for these times and I don't doubt that our listeners think the same. So, I wish you the most beautiful paths ahead on your travels with your families.And ...We get the opportunity to speak again sometime. Tyson: Thank you, Chris. This has really been a meaningful conversation. Thank you. Jesse: Yeah. Thank you so much for giving us this space and I wish you all the same. Chris: My pleasure. Get full access to ⌘ Chris Christou ⌘ at chrischristou.substack.com/subscribe
Li Sumpter:So welcome back to another episode of Future Memory. My guest today is Jesse Hagopian. He is a Seattle-based educator and the author of the upcoming Teach Truth: The Attack on Critical Race Theory and the Struggle for Antiracist Education. Hagopian is an organizer with the Zinn Education Project and co-editor of the books Black Lives Matter at School: An Uprising for Educational Justice and Teaching for Black Lives. Welcome, Jesse.Jesse Hagopian:Oh, thanks so much for having me. Good to be with you. Li:Thank you for joining us. Well, I want to get started with some questions about your own education and how you got started. I was curious about what your own early education and high school experiences were like. As a youth, what ways did you relate to or even resist to your own classroom curricula? Jesse:I was very alienated from school growing up. I felt like it didn't really speak to me. I didn't feel like I was intelligent. I can remember very clearly a parent-teacher conference in third grade where the teacher brought us out into the hallway with me and my mom, and she took out my standardized testing scores and there was a blue line that ran through the middle that was the average, and then there was the dot far below that line that represented my reading scores.And I knew from that day forward until about halfway through college, I knew that I was not smart, and I had the test scores to prove it to you. And school just felt like a place that reinforced over and over again that I was not worthy, that I was not intelligent. And there was very little that we studied that was about helping me understand myself, my identity, my place in the world as a Black, mixed-race kid.And really, it was just a fraught experience, and I took quite a bit to get over that. I was sure I was going to fail out of college, that I wasn't smart enough to go to college. And I think that it was finally the experience of a couple of professors in college that showed that education could be more than just eliminating wrong answer choices at faster rates than other children, that it could be about understanding the problems in our world and how we can collectively solve those problems.And then I realized I did have something to contribute. Then I realized that I did have some perspectives on what oppression looks like and how it feels and what we might need to do to get out of it, and I was hungry to learn about the systems that are set up in our society to reproduce inequality. And that was a real change for me. But growing up, my mom would tell me, "You're good with kids. I think you're going to be a teacher." And I said, "That's the last thing I'm going to be."Li:Oh, really?Jesse:School is just so arduous, and why would I want to come back? And then she was right. I came back to my own high school. I came back to Garfield High School, where I graduated, and I taught there for over a decade now. Li:I think that's an amazing story, coming full circle to teach back where you got your first experiences in the classroom. And going back to that, I was wondering if you had any standout memories, like I did, with the actual content. You were saying you didn't relate to it so much, but I remember very clearly a moment with my mother coming to the school when I had a moment in the classroom around Tom Sawyer, Huckleberry Finn, things like that. Do you have any standout memories of content that really either made you feel excluded or exploited or any of these things that really stuck with you? Jesse:For sure. I mean, there are many experiences that I think shaped my approach to education throughout the years. I mean, one of my firsts is from kindergarten. I remember very clearly one of the boys called me the N-word. And I didn't really know what it meant, but I knew it was directed at me and not the other kids. So I went and told the teacher, but there was parent-teacher conferences going on and parents were coming through, prospective parents, to look at the school, and the teacher got just beet red in front of the parents and was very embarrassed that I had said this, and said, "Oh, yeah. We'll deal with that," and just sort of pushed it aside and never came back to it.And the message that I got was that I had done something wrong, like I had disrupted the education process and that it was wrong for me to have done that because nothing was taken care of. And that's something that still sits with me and I think guides a lot of my approach to how to handle situations in the classroom. And I can remember the first time I had a Black teacher and that I began to learn about Black history in sixth grade, an incredible educator named Faith Davis, taught us about ancient Egypt. And it was the first thing I really got excited about learning, and I was amazed by all these accomplishments that Black people had done.And then after that class, it just sort of disappeared for a long time, and I never learned about anything else that Black people had done, and it made me wonder, "Is that why I score so poorly on these tests? Because I'm Black? Because I don't see other people like me in the advanced classes? And maybe those aren't for us. Maybe it has something to do innately with my race." And that's such a disempowering feeling, and I wanted to ensure that no other kids had to go through that kind of humiliation. Li:No, that's a great point that you bring up because I think we had similar experiences. I was actually recently going through some old photos at my mom's house, and I came across my elementary school class photo, the classic one, everyone's lined up, shortest to tallest kind of thing. And there I was, the only Black child in a class of 25 white students. And I think at that young, innocent age, I didn't really understand what I was up against, and today's youth and teachers are facing so many challenges in the classroom today, things that I don't think either of us could have really imagined.And so, as I was exploring the amazing tools and campaigns that you've been authoring and spearheading, like Teaching for Black Lives, Black Lives Matter at School, and the Zinn Education platform of so many resources, I think, "What would my early school experience have been like if these tools were available?" Right?And I'm wondering, would you have thought the same thing? Because when I think about these amazing tools that are being offered, I just imagine, and we're not even talking about the digital stuff. I'm just talking about the things around critical race theory, these ideas, just about things that are showing a representation of Black folks. Like you said, even just having a Black teacher and what that meant for you. So even thinking about, what if the tools that you are all creating today were actually in your classroom back at Garfield when you were youth? Jesse:Oh, wow. That would've been incredible. I mean, at the Zinn Education Project, we have scores of free downloadable people's history lessons that center Black history and struggles against structural racism. And these lessons tell history from the perspective of people who have been marginalized, who have been pushed out of the centers of power. We look at the founding of America from the perspective of those who have been enslaved, not those who were doing the enslaving. We look at American history through the eyes of those who are organizing multiracial struggles for racial and social justice, not the ones that are trying to maintain segregation and hoarding wealth in the hands of the few.And I would've just lit up to be able to have a teacher say that your family's history matters, that struggles that your family went through shaped this country, and whatever semblance of democracy that we're able to hold onto in this country is the result of the Black freedom struggle and the result of multiracial struggles for social justice. Instead, we got the message in American government class that democracy is something that's handed down from those in power and those on high.I can remember, at Garfield High School, my American government teacher assigned a research project, and I did a project about J. Edgar Hoover, the FBI director. And it was the only paper I think I ever really tried on in high school. I was very disengaged from school and didn't see any point in it, but this research project captured my imagination because I learned about some really despicable things that someone in power had done.I couldn't believe that J. Edgar Hoover had led a campaign against the Black freedom movement, had targeted Martin Luther King, someone who we're all supposed to revere, and yet our government was wiretapping and even trying to get him to commit suicide and some pretty despicable things. And I poured myself into the research and I wrote the best paper I had done up until that point, and she gave me a C with the notes that the claims I was making were unsubstantiated. Li:Wow. Jesse:And it's clear that she just didn't agree, that she didn't want to hear that a white man in power had misused it. And that was a strong message I got that some ideas are off-limits, and it doesn't matter how hard you work. If you go against what makes a white teacher comfortable, then there are consequences for that.And after that, I really didn't want to try anymore. I didn't feel like my opinions mattered, and I would've loved to have a teacher help me understand how we can live in a society that calls itself the freest nation on earth, and yet was based on enslavement of Black people and genocide of Native people, continued with Jim Crow segregation to where up through my dad's generation couldn't vote if you were Black.And then in our own generation, we have mass incarceration. And how is it that racism continues to change in focus and character, but is a constant in American society? And I wasn't able to learn that until much later, and I would've loved to have some of the resources that the Zinn Education Project provides today. Li:Yes, you and me both. Jesse:Yeah. Li:And that brings me to my next question about one of your ongoing campaigns is Black Lives Matter at School. And this year, the 2023 Creative Writing Challenge prompt was, "How can a school community support you in being unapologetically Black?" How might the young Jesse have answered that same question? Jesse:Wow. Well, the young Jesse would've been scared to answer that question. Li:Really? Say more. Jesse:I think that because I was so worried about what it meant to be Black and what that meant about my intelligence, that being unapologetically Black was very foreign for me for far too long. It was hard to come to loving my blackness, and it was a long road to get there. And I'm just so glad that the Black Lives Matter at School movement exists, because so many children like me who are scared to embrace their blackness because they're afraid that it could make them labeled as lesser, not as beautiful, not as deserving of love, not as deserving of care, and everything that all of our kids deserve.Now, these students are celebrated in our Week of Action that happens the first week of February every year, and also on our Year of Purpose. So every month, we're revisiting the principles of the Black Lives Matter Global Network and we're highlighting different aspects of the Black freedom struggle. And this would've been transformative in my life, helped me come to love my blackness much earlier. And I hope that for many thousands of kids across this country, they are having that experience. Li:I love that answer. Thank you. So Garfield High School in Seattle is where you actually attended school as a youth and were also a teacher for over a decade. It's the place where your role as an activist also took root. So history was made here, not just for you as an individual, but really locally and then nationally. So why do you think this was happening at Garfield? Why Garfield High School? And what's the culture and social climate of this school that made it such fertile ground to spark local protests and now national change? Jesse:Yeah. I love that question because I bleed purple and I'm a Bulldog to the core. Garfield is a special place to me, and I think the history of the school is a lot of the reason why it was a fertile ground recently for social change. Garfield High School is the school that the founder of the Seattle chapter of the Black Panther Party graduated from in 1968, Aaron Dixon. Li:Wow. Jesse:It's the site where Stokely Carmichael came to speak as the Black Power movement was rising. And before that, Martin Luther King came and spoke at Garfield High School in his only visit to Seattle. It's the heart of the Central District, which was the Black neighborhood in Seattle that was redlined so that Black people could only live in that area. And for that reason, it developed a culture of resistance, and it's an important part of the Black freedom struggle throughout Seattle's history.And I think that in recent years, we've been able to revive some of that legacy in some of the struggles we've participated in. In 2013, we had a historic boycott of the MAP test, the Measures of Academic Progress test. And this was one of the myriad of high-stakes standardized tests that the kids had to take, and studies show that the average student in K-12 education now take 113 standardized tests. We used to take one in elementary, one in middle school, maybe a couple in high school, and now they're taking standardized tests just constantly.And this was a particularly egregious test that wasn't aligned to our standards. And finally, one educator at Garfield, Mallory Clarke, said she wasn't going to administer this test anymore, and she contacted me and wanted to know if I could help, and we began organizing the entire faculty at Garfield. And we called a meeting in the library and we asked everybody, "Is anybody getting useful information out of this test that's helping them with creating their curriculum?" And nobody found this test useful.And then Mallory said she wasn't going to give the test anymore, and who would join her? And we took a vote, and it was unanimous. Everybody said they were going to refuse to administer the test. And so, we organized a press conference in Mr. Gish's room, and we invited the media to come learn why we were going to refuse to give the standardized test, and one of the reasons is because of the legacy of standardized testing based in eugenics. Right? Li:Mm-hmm. Jesse:Standardized testing was created by open white supremacists. A man named Carl Brigham created the SAT exam out of Princeton University, and he was also the author of a book called The Study in American Intelligence, which was one of the Bibles of the eugenics movement. And the book concludes by lamenting that American intelligence is on the decline because we have more Black people than Europe does, and he fears that intermixing of the races will degrade the intelligence of Americans. And so, he created the SAT exam as a gatekeeper.And lo and behold, these tests prove that white native-born men were smarter than everybody else. Right? Well, they designed the test to show that, and then they get the feedback that they were looking for, and that's why people like W.E.B. Du Bois, Horace Mann Bond were some of the first opponents of these bogus IQ standardized testings that started to be grafted onto the public schools at the behest of the eugenics movement.And we knew this history. I'd read Wayne Au's book, Unequal By Design, that explained the racist history of standardized testing, and then we saw it playing out in our own school. We saw how English language learners would get low scores and it would make them feel deficient and unintelligent. But it wasn't measuring their intelligence. It was just measuring their proximity to white dominant culture, the English language, and not their intelligence. And we had so many examples of the way these tests were abusing kids, and we refused to do it. And the school district threatened the faculty of Garfield High School with a 10-day suspension without pay for the tested subject teachers in reading and math, and even our testing coordinator refused to administer the test. Jesse:Kris McBride was an amazing advocate for the MAP test boycott. And even the first-year teachers, who didn't have any tenure protections, none of them backed down. And at the end of the school year, not only did they not suspend any of the teachers because of the overwhelming solidarity we received from thousands of educators and parents and students, not only around the country but around the world, who had heard about our boycott, at the end of the year, they actually suspended the test instead and got rid of the MAP test for all of Seattle's high schools, and it was just a resounding victory. Li:Yeah. That's a triumph. That's a triumph for sure. Jesse:Yeah. Right? Li:And I was watching some of the news coverage, and it was just, like you said, quite a victory to have that test obliterated, really, just removed completely from the system, and also then making way for this idea of multiple literacies and ways of learning that are more just and equitable for all students. And I love to see that, like you said, it begins just with one person. Shout out to Mallory and everyone who followed that one teacher. And like you said, that's all it takes, but then just to see the students really take lead in their own way was a beautiful thing. Jesse:Yeah. Yeah. It was cool that the students, when they knew we weren't going to administer the test, they sent administrators in to try to get the students to march them off to the computer labs to take the test, and some of them just staged to sit in in their own classroom, refused to get up and leave, and then the ones that went just clicked the button on the computer through very quickly so the score was invalidated.So the BSU supported us and the student government supported us, and it was an incredible solidarity that emerged in this struggle. And it wasn't about not wanting assessment. I think as you said, we wanted more authentic forms of assessment, ones that could actually help us understand what our students knew. And we started doing much more performance-based assessments. Li:Right. Jesse:When you get your PhD, they don't want you to eliminate wrong answer choices at faster rates. They want to know, can you think? Can you create? Li:Right. Are you a critical thinker? Jesse:Right. Yeah. Can you critically think? Can you make a thesis and back it up with evidence? And so, that's what we began doing. We wanted to have kids develop a thesis. And it might not be at the PhD level, but it'll be at a developmentally appropriate level for them, and then back it up with evidence and then present that evidence to the class or to other teachers and administrators and defend their position, and that, I think, was a real victory for all of our students for authentic assessment. Li:And went down at Garfield. Jesse:Yeah. No doubt. No doubt. Li:So another question I got for you. Part of the work of Monument Lab is to engage community in the current state of monuments and public memory in this country and beyond. Have you made any connections to this parallel movement to take down monuments that stand as symbols that continue to uphold oppressive systems and then honor the same false histories that you and your comrades are fighting in the classroom? Jesse:Yeah. Definitely. I think one of my favorite assignments I ever gave my students at Garfield was to research the debate over monuments around the country and think about, "How do we decide as a society who to honor, and who should be honored, and who shouldn't be?" And all the students got a big chunk of clay and they created their own monument to replace one that they thought was inappropriate. And so, many chose Confederate monuments or monuments to any slaveholders, including the hallowed Founding Fathers, that many of my students didn't hold in reverence given that they could have been owned by George Washington.And so, at the University of Washington, we have that statue of George Washington. Some people wanted to replace that with a statue of Aaron Dixon, who graduated from Garfield High School, founded the Black Panther Party, went to the University of Washington, and they felt far better represented our community as somebody who started the Free Breakfast Program in Seattle and who founded a free medical clinic that's still open to this day, just a few blocks away from Garfield High School, where many of our students receive free medical care to this day. Li:Oh, that's amazing. Jesse:So creating themselves some beautiful monuments to really honor the people that have made their lives better rather than just powerful people who imposed their will on our society. And I just think it was such an incredible moment in the 2020 uprising when all across the country, people said, "We are no longer going to honor slaveholders and perpetrators of genocide." It was incredible to see them dump the statue of Columbus into the Bay in Baltimore and teach the whole country a lesson, a history lesson about the genocidal attack of Columbus on Native people and how we need to find better heroes. Li:I like that. Find better heroes. You've dedicated a bunch of your recent efforts to resisting House Bills 1807 and 1886 introduced by state Republican Representative Jim Walsh. As you put it in your article that I read, these bills are designed to mandate educators lie to Washington students about structural racism and sexism, essentially forcing educators to teach a false, alternative history of the United States. Can you break down the basic proposals of these bills and their connection to, say, recent book bans, critical race theory, and resources like The 1619 Project? Jesse:For sure. Many people imagine that the attack on critical race theory is mostly in red states or it's just a product of the South. But instead, people should know that actually the attack on critical race theory originated from Christopher Rufo, who ran for city council in Seattle, and he is still a resident in Washington state, and that every state in the nation, except for California, has had a proposed bill that would require educators to lie to students about structural racism or sexism or heterosexism.And even in California, the one state that hasn't had a proposed bill, they have many local school districts that have one of these educational gag order policies in place that seek to coerce educators to lie to students about American history, about Black history, about queer history. And Washington state is one of the many states that has had proposed bills by Republican legislators that are trying to deceive students. They were so frightened of the 2020 uprising and all the questions that young people were asking about our deeply unequitable society that instead of working to try to eliminate that inequality, they just want to ban people from understanding where it comes from.So in my state, last year, they proposed House Bill 1886 that would make it illegal to teach about structural racism. And I found it deeply ironic that the House bill was numbered 1886, because that was the same year as a mob of white people in Seattle rounded up hundreds of Chinese people and forced them into wagons and hauled them to Seattle docks where they were placed on ships and illegally deported. And the chief of police helped this riotous white mob illegally, Police Chief William Murphy, and he never had faced any penalty for it. He was acquitted, even though this racist attack on Chinese people was carried out. Right?And our students have the right to learn about this. They should know that this happened in our city, and too many don't grow up learning the reality of that anti-Chinese attack. And then when hate crimes skyrocketed in our own era in the wake of the COVID-19 pandemic, you saw hate crimes increase by several hundred percent against Asian Americans, and people wonder why. There's a long history of this Yellow Peril narrative in American society that has labeled Asian Americans and Chinese Americans as the other, as dangerous, as dirty, and our students need to learn about that if they're going to overcome those racial divisions today. Li:And what would the passing of these bills mean for the next generation of youth and their futures, and their education? What's the status of these bills now? Jesse:Well, thankfully, the bill in Washington state did not pass, but they are proliferating around the country. 18 states have already passed bills that seek to coerce educators into lying about structural racism, denying the fact that our country was built on structural racism, of enslavement of Black people, and genocide of Native people, and the exploitation of labor of immigrants, hyper-exploitation of Chinese labor on the railroads and Latinx labor in farms, and they want to hide this history.And you saw it in Florida when they banned the AP African American Studies course. In Virginia, they're trying to rework the state standards to hide the legacy of structural racism and the contributions of Black people, and they are trying to send us back to the era of the 1940s and '50s during the second Red Scare known as the McCarthy era. In the McCarthy era, hundreds of teachers, thousands of teachers around the country were fired after having been labeled communist.And then the Red Scare had the overlapping Lavender Scare, which was the attack on LGBTQ people, and that was especially intense against educators, and Florida had a particularly pernicious attack on queer educators. They had the Johns Committee there that would interrogate teachers about their sex lives and then fire them, remove their teaching certificate so they could never teach again. And this is what people like Governor Ron DeSantis in Florida are trying to revive with the Don't Say Gay bill that has outlawed any discussions of LGBTQ people for the younger grades, and also his so-called Stop W.O.K.E. Act that imposes anti-truth laws on Black history.And in Florida now, it is a third-degree felony for an educator to be caught with the wrong book about Black people or about queer people in their classroom. You can get five years in jail and a $5,000 fine for having the wrong book. Thousands of books are being banned all over the country, and they are rapidly trying to bring us back to that Red Scare, Lavender Scare era where they could just label you a communist or today label you a critical race theorist and push you out of the classroom.So we're at a crossroads right now, where everybody has to decide, "Are we going to build a multiracial struggle to create a true democracy? Or are we going to submit to this fearmongering and this racial hatred and allow them to turn back the clock?" And I hope that people will value social justice enough to join our struggle. Li:I'm just blown away by all the things you're saying, and it's really powerful because I come from a family of educators. Both my father and my mother are educators. My brother and myself are both educators. So I see it not as a job, but like a vocation. And it really sounds like you and the folks that you're in community with, in solidarity with in Seattle and beyond are really making amazing strides and asking such critical questions that could determine the future of our country. Jesse:No doubt. Li:For me and so many other educators, Paulo Freire's Pedagogy of the Oppressed and bell hooks' Teaching to Transgress were defining transformative works that greatly impacted my trajectory in the world. And I wanted to know, can you share what books or even creative works that inspired the path that got you where you are today? Jesse:Yeah. I love that question. Definitely those two books are at the top. Li:Oh, you like those books? Aren't they at the top? Jesse:I love those books. Yes. Li:I love them. Jesse:Yes. Li:I mean, and I'm sure you reread them because I'm always rereading those books. Jesse:Sure. Yes. I'm quoting them in the book I'm writing right now. So much of what I'm doing would not be possible without the theoretical framework that bell hooks gave us and that Paulo Freire gave us to understand how to use dialogic pedagogy to engage your students in a conversation, and educating isn't about filling their heads with what you know, the banking model of education, as Paulo Freire put it, right? Li:Right. Jesse:It's about learning from your students. Li:Right. That relationship between this... I learned so much from my students, especially now that I'm getting older. Jesse:Yeah. No doubt. Li:You got to stay in the know with the youth. Jesse:Hey, the students created the greatest lesson plan of my lifetime when they organized the uprising of 2020. That was mostly young BIPOC folks that organized that uprising and taught the nation what structural racism is and taught many of their teachers that they needed to learn something about it and they needed to begin teaching about it. Right? That's where this whole backlash to critical race theory started.And I think that all of us in the struggle would do well to join in study groups around books that can help deepen our understanding of history and theory that will help us in these struggles to come. There are so many books that I could cite that have been pivotal to my understanding of the struggle. I mean, working at the Zinn Education Project, Howard Zinn's People's History of the United States has been really important. Li:Yes. Jesse:So I think reframing who the subjects of history are and... Li:And the authors of history, right? Jesse:Yeah. Exactly. Exactly. I think that Jarvis Givens book, Fugitive Pedagogy, should be read by all educators. Li:Yes. I'm familiar, very familiar with that project, and it is super inspiring. Yes. Jesse:Yeah. I mean, that book is just a key that unlocks the truth about why we're in the situation we're in right now, where they're trying to outlaw education. Li:And all the overlapping systems, because you talked about that, like these intersecting oppressions and overlapping systems of oppression that are really creating something that it feels like it's impenetrable, but people are making strides. Jesse:Yeah. No doubt. And I would just say that the book, Fugitive Pedagogy, just gives you that history of how Black education has always been a fugitive project. It's always been a challenge to the power structure. It's always been verboten. And starting in 1740 were the first anti-literacy laws in South Carolina banning Black people from learning to read and write.Li:How about that? Right. Jesse:Why was that? Because in 1739, the Stono Rebellion happened. A man named Jemmy helped lead an uprising of enslaved people, and he marched with a banner that read "Liberty" as they collected more enslaved people along the way during their uprising, and this terrified the enslavers. And they not only wanted to kill all the people that were trying to get their freedom, they wanted to kill the idea of freedom. They wanted to kill the ability of Black people to ever write the word liberty again.And so, they imposed these laws to ban Black people from learning to read and write. And today's racists aren't so bold as to ban the ability for people to learn to read and write, but they do want to ban the ability to read the world, as Paulo Freire put it. They don't want us to be racially literate. They don't want us to understand how systems of power and oppression are maintained. And so, they're banning ideas now in the classroom. And once you understand the long history of the attacks on Black education, you can understand why it's happening again today. Li:And even through the digital divide, right? This idea of being disconnected from these resources that are so much a part of education today that Black and brown communities don't always have really makes a difference in the education that they receive and how they learn as well. Jesse:No doubt. I mean, that was emphasized during the pandemic, right?Li:Exactly. So much was amplified during the pandemic, especially that digital divide. Jesse:No doubt. No doubt. Li:So, Jesse, I want to think about the future and speculate. In the best-case scenario, maybe a utopian future for education in the United States. Teachers often have to draft a wish list for what they want, the resources, the needs they have for their classrooms as the academic year comes around. So thinking about what you would want, the three essentials that would be on your wish list for the classroom of the future.Jesse:Yeah. I love this question, because too often, images of the future are all about dystopias. Those are the movies and books we get, and there's not enough freedom dreaming about what's possible. Li:I love that. Shout out to Robin D. Kelley. Jesse:No doubt. Another essential book to read. Li:Yes. Jesse:So I think in the classroom of the future that provides a liberatory education for our youth, the first thing I think we might see is the breakdown of subjects and getting rid of these artificial divisions between the different academic disciplines. And so, school would look very different. Instead of going to math class in the first period and then language arts and then social studies, you might have a class called Should Coal Trains be Used in Seattle? Right? They were just debating whether we should allow coal trains to come through our city.So it would be based on a real problem that exists in your society, and then you would use math and science and language arts and social studies to attack this problem. You would want to learn about the science of climate change and the math that helps you understand the changing climate. Right? We would want to learn the history of coal extraction in this country, the toll it's taken on working people who are minors and the toll it's taken on the environment.We would want to use language arts to write speeches, to deliver your opinion to the city council about this. So we would have problem-posing pedagogy, as Paulo Freire put it, where the courses would be organized around things that the kids care about that impact their lives, and then we would use the academic disciplines in service of that.I think in addition to that, my second requirement for this liberatory classroom would be about wraparound services, so that when kids come to school, they also get healthcare. They also get tutoring services, dental care, mental health care, food for their families. And schools could be really the hubs of community where people have their needs taken care of and are invested in to support not just the students, but their families as well.And lastly, I think schools would be flooded with resources, so that instead of wasting trillions of dollars on the Pentagon so that the United States can go bomb countries all over the world and kill children and their families, we would take that money and flood it into the school system so that kids have all the state-of-the-art resources they need, from the digital equipment, recording equipment, music, art supplies, to funding the school nurse, to the auditoriums, and the music halls. I mean, you can imagine that the richest country on earth could have incredible resources for their kids if we valued education, if we valued our young people.Instead, so many schools in America today are falling apart. The first school I ever taught in in Washington, D.C., an elementary school, I had a hole in the ceiling of my classroom, and it just rained into my classroom and destroyed the first project that I ever assigned the students, their research project, and they never even got to present the projects. Li:No way. Jesse:And our kids deserve better than that. Li:Oh, they definitely deserve better than that. Right? Oh my gosh. Jesse:We're in a society where 81 billionaires have the same amount of wealth as the bottom half of humanity, and that wealth divide means that our kids go to schools that are falling apart, and we would transform that in a future society that's worthy of our kids. Li:Most definitely. And if I can, I wanted to add a fourth thing, because I remember something you said about performance-based assessment. Jesse:Oh, yeah. Li:And I think that would- Jesse:I should put that in. Li:... definitely be essential, right? Make sure you get that one in. But last but not least, my final question to you is, what's next for Zinn Education? And more specifically, what is next for Jesse Hagopian? Jesse:Oh, thank you. Well, I'm really excited about the June 10th National Day of Action. The Zinn Education Project has partnered with Black Lives Matter at School and the African American Policy Forum to organize the Teach Truth Day of Action on June 10th, and I hope everybody will join us on that day of action in organizing an event in your community. This is the third annual Teach Truth Day of Action, and the past ones have been incredible.People have organized historical walking tours in their community to highlight examples of the Black freedom struggle and sites that were important in the Black freedom struggle in their own communities or sites of oppression and racial injustice that students have the right to learn about in their own communities. Some people went to sites where Japanese people were rounded up and incarcerated during World War II. Some people in Memphis, Tennessee went to a site right on their school grounds where there was a race riot and many Black people were killed.In Seattle, we went by the clinic that the Black Panther Party started and gave that history and highlighted how, if the bill passed to deny teachers the right to teach about structural racism, we couldn't even teach about the origins of the health clinic in our own community. And so, there'll be many creative protests that happen on June 10th, 2023, and I'm excited to say we have more cosponsors than ever before.The National Education Association is supporting now, and many other grassroots organizations from across the country. So I expect hundreds of teachers and educators will turn out to protest these anti-truth laws, and I'll be right there with them all helping to organize it and learning from the educators and organizers, who are putting these events on, and hopefully helping to tell their story in the new book that I hope to be finishing very soon about this- Li:You're going to finish it. You're going to finish. This month, man. Jesse:Thank you. Li:This is your month. Jesse:I need that encouragement. Li:You got this. Jesse:I hope I finish it on this month. Li:Believe me. When I was so close to finishing my dissertation, everyone kept asking me, "Are you done yet? Are you done yet?" So I know, because I could see you cringe when I asked you that in the beginning. All I can say is, look, I mean, I'm just so grateful to have this conversation with you today. Thank you for joining me. And I also got to say, I'm sorry to say, Jesse, your mother was right. I think this was your calling. I think this might have been what you were set on this planet to do. Jesse:It feels that way now. Thank you so much. Li:Yes, indeed. So this is Monument Lab, Future Memory. Thank you to my guest, Jesse Hagopian. Jesse:Hey, I really appreciate you having me on. I just felt your warm spirit come across and brighten my day. Really great to be with you. Li:My pleasure.
About Jesse Jesse Vincent is the cofounder and CTO of Keyboardio, where he designs and manufactures high-quality ergonomic mechanical keyboards. In previous lives, he served as the COO of VaccinateCA, volunteered as the project lead for the Perl programming language, created both the leading open source issue tracking system RT: Request tracker and K-9 Mail for Android.Links: Keyboardio: https://keyboard.io Obra: https://twitter.com/obra TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You could build you go ahead and build your own coding and mapping notification system, but it takes time, and it sucks! Alternately, consider Courier, who is sponsoring this episode. They make it easy. You can call a single send API for all of your notifications and channels. You can control the complexity around routing, retries, and deliverability and simplify your notification sequences with automation rules. Visit courier.com today and get started for free. If you wind up talking to them, tell them I sent you and watch them wince—because everyone does when you bring up my name. Thats the glorious part of being me. Once again, you could build your own notification system but why on god's flat earth would you do that?Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. As you folks are well aware by now, this show is at least ostensibly about the business of cloud. And that's intentionally overbroad. You can fly a boat through it, which means it's at least wider than the Suez Canal.And that's all well and good, but what do all of these cloud services have in common? That's right, we interact with them via typing on keyboards. My guest today is Jesse Vincent, who is the founder of Keyboardio and creator of the Model 01 heirloom-grade keyboard, which is sitting on my desk that sometimes I use, sometimes it haunts me. Jesse, thank you for joining me.Jesse: Hey, thanks so much for having me, Corey.Corey: So, mechanical keyboards are one of those divisive things that, back in the before times when we were all sitting in offices, it was an express form of passive aggression, where, “I don't like the people around me, and I'm going to show it to them with things that can't really complain about. So, what is the loudest keyboard I can get?” Style stuff. And some folks love them, some folks can't stand them. And most folks to be perfectly blunt, do not seem to care.Jesse: So, it's not actually about them being loud, or it doesn't have to be. Mechanical keyboards can be dead silent; they can be as quiet as anything else. There's absolutely a subculture that is into things that are as loud as they possibly can be; you know, sounds like there's a cannon going off on somebody's desk. But you can also get absolutely silent mechanical switches that are more dampened than your average keyboard. For many, many people, it's about comfort, it is about the key feel.A keyboard is supposed to have a certain feeling and these flat rectangles that feel like you're typing on glass, they don't have that feeling and they're not good for your fingers. And it's been fascinating over the past five or six years to watch this explosion in interest in good keyboards again.Corey: I learned to first use a computer back on an old IBM 286 in the '80s. And this obviously had a Model M—or damn close to it—style buckling spring keyboard. It was loud and I'm nostalgic about the whole thing. True story I've never told on this podcast before; I was a difficult child when I was five years old, and I was annoyed because my parents went out of the house and my brother was getting more attention than I was. I poured a bucket of water into the keyboard.And to this day, I'm surprised my father didn't murder me after that. And we wound up after having a completely sealing rubber gasket on top of this thing. Because this was the '80s; keyboards were not one of those, “Oh, I'm going to run down to the store and pick up another one for $20.” This was at least a $200 whoops-a-doozy. And let's just say that it didn't endear me to my parents that week.Jesse: That's funny because that keyboard is one that actually probably would have dried out just fine. Not like the Microsoft Naturals that I used to carry in the mid-'90s. Those white slightly curved ones. That was my introduction to ergonomic keyboards and they had a fatal flaw as many mid-'90s Microsoft products did. In this case, they melted in the rain; the circuit traces inside were literally wiped away by water. If a cup of water got in that keyboard, it was gone.Corey: Everyone has a story involving keyboard and liquids at some point, or they are the most careful people that are absolutely not my people whatsoever because everyone I hang out with is inherently careless. And over time I used other keyboards as I went through my life and never had strong opinions on them, and then I got to play with a mechanical keyboard had brought all that time rushing back to me of, “Oh, yeah.” And my immediate thought is, “Oh, this is great. I wonder if I could pour water into it? No, no.”And I started getting back into playing with them and got what I thought was the peak model keyboard from Das Keyboards which, there was the black keyboard with no writing on it at all. And I learned I don't type nearly as well as I thought I did in those days. And okay. That thing sat around gathering dust and I started getting a couple more and a couple more, and it turns out if you keep acquiring mechanical keyboards, you can turn an interest into a problem but you can also power your way through to the other side and become a collector. And I started building my own for a while and I still have at least a dozen of them in various states of assembly here.It was sort of a fun hobby that I got into, and for me at least it was, why do I want to build a keyboard myself? Is it, do I believe intrinsically that I can build a better keyboard than I can buy? Absolutely not. But everything else I do in my entire career as an engineer until that point had been about making the bytes on the screen go light up in different patterns. That was it.This was something that I had built that I could touch with my hands and was still related to the thing that I did, and was somewhat more forgiving than other things that I could have gotten into, like you know, woodworking with table saws that don't realize my arm it just lopped off.Jesse: Oh, you can burn yourself pretty good with a soldering iron.Corey: Oh, absolutely I can.Jesse: But yeah, no, I got into this in a similar-sounding story. I had bad wrists throughout my career. I was a programmer and a programming manager and CEO. And my wrist hurts all the time, and I'd been through pretty much every ergonomic keyboard out there. If you seen the one where you stick your fingers into little wells, and each finger you can press back forth, left, right, and down, the ones that looked like they were basically a pair of flat capacitive surfaces from a company that later got bought by Apple and turned into the iPads touch technology, Microsoft keyboards, everything. And nothing quite felt right.A cloud startup I had been working on cratered one summer. Long story short, the thing went under for kind of sad reasons and I swore I was going to take a year off to screw around and figure out what the next thing was going to be. And at some point, I noticed there were people on the internet building their own keyboards. This was not anything I had ever done before. When I started soldering, I did figure out that I must have soldered before because it smelled familiar, but this was supposed to be a one-month project to build myself a single keyboard.And I saw that people on the internet were doing it, I figured, eh, how hard could it be? Just one of those things that Perl hackers are apt to say. Little did I know. It's now, I want to say something like eight years later, and my one-month project to build one keyboard has failed thousands and thousands and thousands of times over as we've shipped thousands of keyboards to, oh God, it's like 75 or 78 countries.Corey: And it's great. It's well made. The Model 01 that I got was part of an early Kickstarter batch. My wife signed me up for it—because she knew I was into this sort of thing—as a birthday gift. And then roughly a year later, if memory serves, it showed up and that was fine.Again, it's Kickstarter is one of those, this might just be an aspirational gift. We don't know. And—because, Kickstarter—but it was fun. And I use it. It's great.I like a lot of the programmability aspects of it. There are challenges. I'm not used to using ergonomic keyboards, and the columnar layout is offset to a point where I miss things all the time. And if you're used to typing rapidly, in things like chats, or Twitter or whatnot, were rapid responses valuable, it's frustrating trying to learn how a new keyboard layout works.Jesse: Absolutely. So, we got some advice very early on from one of the research scientists who helped Microsoft with their design for their natural keyboards, and one of the things that he told us was, “You will probably only ever get one chance to make a keyboard; almost every company that makes a keyboard fails, and so you should take one of the sort of accepted designs and make a small improvement to help push the industry forward. You don't want to go do something radical and have nobody like it.”Corey: That's very reasonable advice and also boring. Why bother?Jesse: Well, we walked away from that with a very different take, which was, if we're only going to get one chance of this, we're going to do the thing we want to make.Corey: Yeah.Jesse: And so we did a bunch of stuff that we got told might be difficult to do or impossible. We designed our own keycaps from scratch. We milled the enclosure out of hardwood. When we started, we didn't know where we were manufacturing, but we did specify that the wood was going to be Canadian maple because it grows like a weed, and as you know, not in danger of being made extinct. But when you're manufacturing in southern China and you're manufacturing with Canadian maple, that comes on a boat from North America.Corey: There's something to be said for the globalization supply chain as we see things shipped back and forth and back and forth, and it seems ridiculous but the economics are there it's—Jesse: Oh, my God. Now, this year.Corey: Yeah [laugh], there's that.Jesse: Supply chains are… how obscenity-friendly is this podcast? [laugh].Corey: Oh, we can censor anything that's too far out. Knock yourself out.Jesse: Because what I would ordinarily say is the supply chains are [BLEEP].Corey: Yep, they are.Jesse: Yeah. This time around, we gave customers the—for the Model 100, which is our new keyboard that the Kickstarter just finished up for—we gave customers the choice of that nice Canadian maple or walnut. We got our quotes in advance. You know, our supplier confirmed wood was no problem a few months in advance. And then the night before the campaign launched, our wood supplier got in touch and said, “So, there are no walnut planks that are wide enough to be had in all of southern China. There are some supply chain issues due to the global container shortage. We don't know what we're going to be able to do. Maybe you could accept it if we did butcher block style walnut and glued planks together.”They made samples and then a week later, instead of FedExing us the samples, I got a set of photographs with a whole bunch of sad faces and crying face emojis saying, “Well, we tried. We know there's no way that this would be acceptable to your customers.” We asked, “So, where's this walnut supposed to be coming from that you can't get it?” They're like, “It's been sitting on the docks at the origin since March. It's being forested in Kentucky in the United States.”Corey: The thing that surprised me the most about the original model on Kickstarter campaign was how much went wrong across the board. I kept reading your updates. It was interesting, at some point, it was like, okay, this is clearly a Ponzi scheme. That's the name of the keyboard: ‘The Ponzi', where there's going to be increasingly outlandish excuses.Jesse: I don't think a Ponzi scheme would be the right aspersion to be casting.Corey: There's that more pedestrian scam-style thing. We could go with that.Jesse: We have a lot of friends who've been in industry longer than us, and every time we brought one of the problems that our factory seemed to be having to them, they said, “Oh, yeah, that's the thing that absolutely happens.”Corey: Yeah, it was just you kept hitting every single one of these, and I was increasingly angry on your behalf, reading these things about, “Oh, yeah. Just one of your factory reps just blatantly ripped you off, and this was expected to be normal in some cases, and it's like”—and you didn't even once threatened to burn the factory now, which I thought was impressive.Jesse: No, nobody threatened to burn the factory down, but one of the factories did have a fire.Corey: Which we can neither confirm nor deny—I kid, I kid, I kid.Jesse: Yeah, yeah, yeah. But so what our friends who had been in industry longer that said, it was like, “Jesse, but, you know, nobody has all the problems.” And eventually, we figured out what was going on, and it was that our factory's director of overseas sales was a con artist grifter who had been scamming both sides. She'd been lying to us and lying to the factory, and making up stories to make her the only trusted person to each side, and she'd just been embezzling huge sums of money.Corey: You hear these stories, but you never think it's going to be something that happens to you. Was this your first outing with manufacturing a physical product?Jesse: This was our first physical product.Corey: But I'm curious about it; are you effectively following the trope of a software person who thinks, “Ah, I could do hardware? How hard could it be? I could ship code around the world seconds, so hardware will be just a little bit slower.” How close to that trope are you?Jesse: So, when we went into the manufacturing side, we knew that we knew nothing, and we knew that it was fraught with peril. And we gave ourselves an awful lot of padding on timing, which we then blew through for all sorts of reasons. And we ran through a hardware incubator that helped us vet our plans, we were working with companies on the ground that helped startups work with factories. And honestly, if it hadn't been for this one individual, yes we would have had problems, but it wouldn't have been anything of the same scale. As far as we can tell, almost everything bad that happened had a grain of truth in it, it's just that… you know, a competent grifter can spin a tiny thing into a giant thing.And nobody in China suspected her, and nobody in China believed that this could possibly be happening because the penalties if she got caught were ten years in a Chinese prison for an amount of money that effectively would be a down payment on an apartment instead of the price of a full apartment or fully fleeing the country.Corey: It seems like that would be enough of a deterrent, but apparently not.Jesse: Apparently not. So, we ended up retaining counsel and talking to friends who had been working in southern China for 15 years for about who they might recommend for a lawyer. We ended up retaining a Chinese lawyer. Her name's [Una 00:13:36]; she's fantastic.Corey: Referrals available upon request.Jesse: Oh, yeah. No, absolutely. I'm happy to send her all kinds of business. She looked at the contract we had with the factory, she's like, “This is a Western contract. This isn't going to help you in the Chinese courts. What we need to do is we need to walk into the factory and negotiate a new agreement that is in Chinese, written by a Chinese lawyer, and get them to sign it.”And part of that agreement was getting them to take full joint responsibility for everything. And she walked in with me to the factory. She dressed down: t-shirt and jeans. They initially thought she was my translator, and she made a point of saying, “Look, I'm Jesse's counsel. I'm not your lawyer. I do not represent your interests.”And three-party negotiations with the factory: the factory's then former salesperson, and us. And she negotiated a new agreement. And I had a long list of all the things that we needed to have in our contract, like all the things that we really cared about. Get to the end of the day and she hands it to me and she's like, “What do you think?” And I read it through and my first thought is that none of the ten points that we need in this agreement are there.And then I realized that they are there, they're just very subtle. And everybody signs it. The factory takes full joint responsibility for everything that was done by their now former salesperson. We go outside; we get into the cab, and she turns to me—and she's not a native speaker of English, but she is fluent—and she's like, how do you think that went, Jesse? I'm like, I think that went pretty well. And she's like, “Yes. I get my job satisfaction out of adverse negotiation, and the factory effectively didn't believe in lawyers.”Corey: No, no. I've seen them. They exist. I married one of them.Jesse: Oh, yeah. As it turned out, they also didn't really believe in the court system and they didn't believe in not pissing off judges. Nothing could help us recover the time we lost; we did end up recovering all of our tooling, we ended up recovering all of our product that they were holding, all with the assistance of the Chinese courts. It was astonishing because we went into this whole thing knowing that there was no chance that a Chinese court would find for a small Western startup with no business presence in China against a local factory, and I think our goal was that they would get a black mark on their corporate social credit report so that nobody else would do business with this factory that won't give the customer back their tooling. And… it turns out that, no, the courts just helped us.Corey: It's nice when things work the way they're supposed to, on some level.Jesse: It is.Corey: And then you solve your production problems, you shipped it out. I use it, I take it out periodically.Jesse: We'd shipped every customer order well before this.Corey: Oh, okay. This was after you had already done the initial pre-orders. This was as you were ongoing—Jesse: Yeah, there were keycaps we owed people, which were—Corey: Oh, okay.Jesse: Effectively the free gift we promised aways in for being late on shipping.Corey: That's what that was for. It showed up one day and I wondered what the story behind that was. But yeah, it was—Jesse: Yeah.Corey: They're great.Jesse: Yeah. You know, and then there was a story in The Verge of, this Kickstarter alleges that—da, da, da, da, da. We're like, “I understand that AOL's lawyers make you say ‘alleges,' but no, this really happened, and also, we really had shipped everything that we owed to customers long before all this went down.”Corey: Yeah. This is something doesn't happen in the software world, generally speaking. I don't have to operate under the even remote possibility that my CI/CD system is lying to me about what it's doing. I can generally believe things that show up in computers—you would think—but there are—Jesse: You would think. I mean—Corey: There a lot of [unintelligible 00:17:19] exceptions to that, but generally, you can believe it.Jesse: In software, you sometimes we'll work with contractors or contract agencies who will make commitments and then not follow through on those commitments, or not deliver the thing they promised. It does sometimes happen.Corey: Indeed.Jesse: Yeah, no, the thing I miss the most from software is that if there is a defect, the cost of shipping an update is nil and the speed at which you can ship an update is instantly.Corey: You would think it would be nil, but then we look at AWS data transfer pricing and there's a giant screaming caveat on that. It's you think that moving bytes would cost nothing. Yeah.Jesse: [unintelligible 00:17:53] compared to international shipping costs for physical goods, AWS transfer rates are incredibly competitive.Corey: No, no, to get to that stage, you need to add an [unintelligible 00:18:02] NAT gateway with their data processing fee.Jesse: [laugh].Corey: But yeah, it's a different universe. It's a different problem, a different scale of speed, a different type of customer, too, on some levels. So, after you've gotten the Model 01's issues sorted out, you launched a second keyboard. The ‘a-TREE-us', if I'm pronouncing that correctly. Or ‘A-tree-us'.Jesse: So Phil, who designed it, pronounces is ‘A-tree-us', so we pronounce it A-tree-us. And so, this is a super minimalist keyboard designed to take with you everywhere, and it was something where Phil Hagelberg, who is a software developer of some repute for a bunch of things, he had designed this sort of initially for his own use and then had started selling kits. So, laser-cut plywood enclosures, hand-built circuit boards, you just stick a little development board in the middle of it, spend some time soldering, and you're good to go. And he and I were internet buddies; he had apparently gotten his start from some of my early blog posts. And one day, he sent me a note asking if I would review his updated circuit board design because he was doing a revision.I looked at his updated circuit board design and then offered to just make him a new circuit board design because it was going to be pretty straightforward to do something that's going to be a little more reliable and a lot more cost-effective. We did that and we talked a little more, and I said, “Would you be interested in having us just make this thing in a factory and sell it with a warranty and send you a royalty?” And he said, but it's GPL. You don't have to send me a royalty.Corey: I appreciate that I am not compelled to do it. However—yeah.Jesse: Yeah, exactly. It's like, “No. We would like to support people who create things and work with you on it.”Corey: That's important. We periodically have guest authors writing blog posts on Last Week in AWS. Every single one of them is paid for what they do, sometimes there for various reasons that they can't or won't accept it and we donate it to a charity of their choice, but we do not expect people to volunteer for a profit-bearing entity, in some respects.Jesse: Yeah.Corey: Now, open-source is a whole separate universe that I still maintain that is rapidly becoming a, “Would you like to volunteer for a trillion-dollar company in your weekend hours?” Usually not, but there's always an argument.Jesse: Oh, yeah. We have a bunch of open-source contributors to our open-source firmware and we contribute stuff back upstream to other projects, and it is a related but slightly different thing. So, Phil said yes; we said yes. And then we designed and made this thing. We launched an ultra-portable keyboard designed to take with you everywhere.It came with a travel case that had a belt loop, and basically a spring-loaded holster for your keyboard if you want to nerd out like that. All of the Kickstarter video and all the photography sort of showed how nice it looked in a cafe. And we launched it, like, the week the first lockdowns hit, in the spring of 2019.Corey: I have to say I skipped that one entirely. One of the things that I wound up doing—keyboard-wise—when I started this company four years ago and change, now was, I wound up getting a fairly large desk, and it's 72 inches or something like that. And I want a big keyboard with a numpad—yeah, that's right, big spender here—because I don't need a tiny little keyboard. I find that the layer-shifting on anything that's below a full-size keyboard is a little on the irritating side. And this goes beyond. It is—it requires significant—Jesse: Oh, yeah. It's—Corey: Rewiring of your brain, on some level.Jesse: And there are ergonomic reasons why some people find it to be better and more comfortable. There's less reaching and twisting. But it is a very different typing experience and it's absolutely not for everybody. Nothing we've made so far is intended to be a mass-market product. When we launched the Model 01, we were nervous that we would make something that was too popular because we knew that if we had to fulfill 50,000 of them, we'd just be screwed. We knew how little we knew.But the Atreus, when we launched it on Kickstarter, we didn't know if we were going to have to cancel the campaign because no one was going to want their travel keyboard at the beginning of a pandemic, but it did real well. I don't remember the exact timing and numbers, but we hit the campaign goal, I want to say early on the first day, possibly within minutes, possibly within hours—it's been a while now; I don't remember exactly—ultimately, we sold, like, 2600 of them on Kickstarter and have done additional production runs. We have a distributor in Japan, and a distributor in the US, and a distributor in the UK, now. And we also sell them ourselves directly online, from keyboard.io.So, this is one of the other fascinating logistics things, is that we ship globally through Hong Kong. Which, before the pandemic was actually pretty pleasant. Inexpensive shipping globally has gotten kind of nuts because most discount carriers, the way they operated historically is, they would buy cargo space on commercial flights. Commercial international flights don't happen so much.Corey: Yes, suddenly, that becomes a harder thing to find.Jesse: Early on, we had a couple of shipping providers that were in the super-slow, maybe up to two weeks to get your thing somewhere by air taking, I want to say we had things that didn't get there for three months. They would get from Hong Kong to Singapore in three days; they would enter a warehouse, and then we had to start asking questions about, “Hey, it's been eight weeks. What's going on?” And they're like, “Oh, it's still in queue for a flight to Europe. There just aren't any.”Corey: It seems like that becomes a hard problem.Jesse: It becomes a hard problem. It started to get a little better, and now it's starting to get a little worse again. Carriers that used to be ultra-reliable are now sketchy. We have FedEx losing packages, which is just nuts. USPS shipments, we see things that are transiting from Hong Kong, landing at O'Hare, going through a sorting center in Chicago, and just vanishing for weeks at a time, in Chicago.Corey: I don't pretend to understand how this stuff works. It's magic to me; like, it is magic, on some level, that I can order toilet paper on the internet, it gets delivered to my house for less money than it costs me to go to the store and buy it. It feels like there's some serious negative externalities in there. But we don't want to look too closely at those because we might feel bad about things.Jesse: There's all kinds of fascinating stuff for us. So, shipping stuff, especially by air, there are two different ways that the shipping weight can get calculated. It can either get calculated based on the weight on a scale, or it can get calculated using a formula based on the dimensions. And so bulky things are treated as weighing an awful lot. I'm told that Amazon's logistics teams started doing this fascinating thing where ultra-dense, super-heavy shipments they pushed on to FedEx and UPS, whereas the ultra-light stuff that saved on jet fuel, they shoved onto their own planes.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: I want to follow up because it seems like, okay, pandemic shipping is a challenge; you clearly are doing well. You still have them in stock and are selling them as best I'm aware, correct?Jesse: Yes.Corey: Yeah. I may have to pick one up one of these days just so I can put it on the curiosity keyboard shelf and kick it around and see how it works. And then you recently concluded a third keyboard Kickstarter, in this case. And—Jesse: Yeah.Corey: —this is not your positioning; this is my positioning of what I'm picking up of, “Hey, remember that Model 01 keyboard we sold you that you love and we talked about and it's amazing? Yeah, turns out that's crap. Here's the better version of it.” Correct that misapprehension, please. [laugh].Jesse: Sure. So, it absolutely is not crap, but we've been out of stock in the Model 01 for a couple of years now. And we see them going used for as much or sometimes more than we used to charge for them new. It went out of stock because of the shenanigans with that first factory. And shortly before we launched the Atreus, we'd been planning to bring back an updated version of the Model 01; we've even gotten to the point of, like, designing the circuit boards and starting to update the tooling, the injection molding tooling, and then COVID, Atreus, life, everything.And so it took us a little longer to get there. But there is a larger total addressable market for a keyboard like the Model 01 than the total number that we ever sold. There are certainly people who had Model 01s who want replacements, want extras, want another one on another desk. There are also plenty of people who wanted a Model 01 and never got one.Corey: Here's my question for you, with all three of these keyboards because they're a different layout, let's be clear. Some more so than others, but even the columnar layout is strange here. Once upon a time, I had a week in which I wasn't doing much, and I figured, ah, I'll Dvorak—which is a different keyboard layout—and it's not that it's hard; it's that it's rewiring a whole bunch of muscle memory. The problem I ran into was not that it was impossible to do, by any stretch, but because of what I was doing—in those days help desk and IT support—I was having to do things on other people's computers, so it was a constant context switching back and forth between different layouts.Jesse: Yeah.Corey: Do you see that being a challenge with layouts like this, or is it more natural than that?Jesse: So, what we found is that it is easier to switch between an ergonomic layout and a traditional layout, like a columnar layout, and what's often called a row-stagger layout—which is what your normal keyboard looks like—than it is to switch between Dvorak and Qwerty on a traditional keyboard. Or the absolute bane of my existence is switching between a ThinkPad and a MacBook. They are super close; they are not the same.Corey: Right. You can't get an ergonomic keyboard layout inside of a laptop. I mean, looking at the four years of being gaslit by Apple, it's clear you can barely get a keyboard into a MacBook for a while. It's, “Oh, it's a piece of crap, but you're using it wro”—yeah. I'm not a fan of their entire approach to keyboards and care very than what Apple has to say about anything even slightly keyboard-related, but that's just me being bitter.Jesse: As far as I can tell, large chunks of Apple's engineering organization felt the same way that you did. Their new ones are actually decent again.Corey: Yes, that's what I've heard. And I will get one at some point, but I also have a problem where, “Oh, yeah, you know that $3,000 laptop with a crappy keyboard, you can't use for anything? Great. The solution is to give us 3000 more dollars, and then we'll sell you one that's good.” And it's, I feel like I don't want to reward the behavior.Jesse: I hear you. I ditched Mac OS for a number of years. I live the dream: Linux on the desktop. And it didn't hurt me a lot—printing worked fine, scanning worked fine, projectors were fine—but when I was reaching for things like Photoshop, and Lightroom, and my mechanical CAD software, it was the bad kind of funny.Corey: I have to be careful, now for the first time in my life I'm not updating to new operating systems early on, just because of things like the audio stuff I have plugged into my nonsense and the media nonsense that I do. It used to be that great, my computer only really needs to be a web browser and a terminal and I'm good. And worst case, I can make do with just the web browser because there are embedded a terminal into a web page options out there. Yeah, now it turns out that actually have a production workflow. Who knew?Jesse: Yep. That's the point where I started thinking about having separate machines for different things. [laugh].Corey: Yeah, I'm rapidly hitting that point. Yeah, I do want to get into having fun with keyboards, on some level, but it's the constant changing of what you're using. And then, of course, there's the other side of it where, in normal years, I spent an awful lot of time traveling and as much fun as having a holster-mounted belt keyboard would be, in many cases, it does not align with the meetings that I tend to be in.Jesse: Of course.Corey: It's, “Oh, great. You're the CFO of a Fortune 500. Great, let me pair my mini keyboard that looks like something from the bowels of your engineering department's reject pile.” Like, what is this? It's one of those things that doesn't send the right message in some cases. And let's be honest; I'm good at losing things.Jesse: This is a pretty mini keyboard, but I hear you.Corey: Or I could lose it, along with my keys. It will be great.Jesse: Yeah. There are a bunch of things I've wanted to do around reasonable keyboards for tablets.Corey: Yes, please do.Jesse: Yeah. We actually started looking at one point at a fruit company in Cupertino's requirements around being able to do dock-connector connected keyboards for their tablets, and… it's nuts. You can't actually do ergonomic keyboards that way, it would have to be Bluetooth.Corey: Yeah. When I travel on the road these days, or at least—well, ‘these days' being two years ago—the only computer I'd take is an iPad. And that was great; it works super well for a lot of my use cases. There's still something there, and even going forward, I'm going to be spending a lot more time at home. I have young kids now, and I want to be here to watch them grow up.And my lifestyle and use cases have changed for the last year and a half. I've had an iMac. I've never had one of those before. It's big screen real estate; things are great. And I'm looking to see whether it's time to make a full-on keyboard evolution if I can just force myself over the learning curve, here. But here's the question you might not be prepared to answer yet. What's next? Do you have plans on the backburner for additional keyboards beyond what you've done?Jesse: Oh, yeah. We have, like, three more designs that are effectively in the can. Not quite ready for production, but if this were a video podcast, I'd be pulling out and waving circuit boards at you. One of the things that we've been playing with is what is called in the trade a symmetric staggered keyboard where the right half is absolutely bog-standard normal layout like you'd expect, and the left side is a mirror of that. And so it is a much more gentle introduction to an ergonomic-style keyboard.Corey: Okay, I can almost wrap my head around that.Jesse: Because if you put your hands on your keyboard and you feel the angles that you have to move on your right side, you'll see that your fingers move basically straight back and forth. On the left side, it's very different unless you're holding your hand at a crazy, crazy angle.Corey: Yeah.Jesse: And so it's basically giving you that same comfort on the right side and also making the left side comfy. It's not a weird butterfly-shaped keyboard; it is still a rectangle, but it is just that little bit better. We're not the first people who have done this. Our first prototype of this thing was, like, 2006, something like that. But it was a one-off, like, “I wonder if I would like this.” And we were actually planning to do that one next after the Model 01 when the Atreus popped up, and that was a much faster, simpler, straighter-forward thing to bring to production.Corey: The one thing I want from a keyboard—and I haven't found one yet; maybe it exists, maybe I have to build it myself—but I want to do the standard mechanical keyboard—I don't even particularly care about the layout because it all passes through a microcontroller on the device itself. Great. And those things are programmable as you've demonstrated; you've already done an awful lot of open-source work that winds up being easily used to control keyboards. And I love it, and it's great, but I also want to embed a speaker—a small one—into the keyboard so I can configure it that every time I press a key, it doesn't just make a clack, it also makes a noise. And I want to be able to—ideally—have it be different keys make different noises sometimes. And the reason being is that when we eventually go back to offices, I don't want there to be any question about who is the most obnoxious typist in the office; I will—Jesse: [laugh].Corey: —win that competition. That is what I want from a keyboard. It's called the I-Don't-Want-Anyone-Within-Fifty-Feet-Of-Me keyboard. And I don't quite know how to go about building that yet, but I have some ideas.Jesse: So, there's absolutely stuff out there. There is prior art out there.Corey: Oh, wonderful.Jesse: One of the other options for you is solenoids.Corey: Oh, those are fun.Jesse: So, a solenoid is—there is a steel bar, an electromagnet, and a tube of magnetic material so that you can go kachunk every time you press a key.Corey: It feels functionally like a typewriter to my understanding.Jesse: I mean, it can make it feel like a typewriter. The haptic engine in an iPhone or a Magic Trackpad is not exactly a solenoid but might give you the vaguest idea of what you're talking about.Corey: Yeah, I don't think I'm going to be able to quite afford 104 iPhones to salvage all of their haptic engines so that I can then wind up hooking each one up to a different key but, you know, I am sure someone enterprising come up with it.Jesse: Yeah. So, you only need a couple of solenoids and you trigger them slightly differently depending on which key is getting hit, and you'll get your kachunk-kachunk-kachunk-kachunk-kachunk.Corey: Yeah, like spacebar for example. Great. Or you can always play a game with it, too, like, the mystery key: whenever someone types in the hits the mystery key, the thing shrieks its head off and scares the heck out of them. Especially if you set it to keys that aren't commonly used, but ever so frequently, make everyone in the office jumpy and nervous.Jesse: This will be perfect for Zoom.Corey: Oh, absolutely, it would. In fact, one thing I want to do soon if this pandemic continues much longer, is then to upgrade my audio setup here so I can have a second microphone pointed directly into my keyboard so that people who are listening at a meeting with me can hear me typing as we go. I might be a terrible colleague. One wonders.Jesse: You might be a terrible colleague, but you might be a wonderful colleague. Who knows?Corey: It all depends on the interests we have. I want to thank you for taking the time to walk me through the evolution of Keyboardio. If people want to learn more, or even perhaps buy one of these things, where can they do that?Jesse: They can do that at keyboard.io.Corey: And hence the name. Thank you so much for taking the time to speak with me about all this. I really appreciate it.Jesse: Cool. Thanks so much for having me. I had fun.Corey: I did, too. Jesse Vincent—obra on Twitter, and of course, the CTO of Keyboardio. I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment, but before typing it, switch your keyboard to Dvorak.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About JesseJesse is a seasoned operations engineer with a deep passion for understanding complex technical and organizational systems. He's spent his career helping Engineering teams achieve their business goals by improving how they interact with their technical systems, and with each other. He's currently a Cloud Economist with Duckbill Group, guiding organizations along their journey of cloud cost optimization and management.Links: The Duckbill Group: https://www.duckbillgroup.com/ Jesse's Twitter: https://twitter.com/jesse_derose AWS Morning Brief: https://www.lastweekinaws.com/podcast/aws-morning-brief/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Your company might be stuck in the middle of a DevOps revolution without even realizing it. Lucky you! Does your company culture discourage risk? Are you willing to admit it? Does your team have clear responsibilities? Depends on who you ask. Are you struggling to get buy in on DevOps practices? Well, download the 2021 State of DevOps report brought to you annually by Puppet since 2011 to explore the trends and blockers keeping evolution firms stuck in the middle of their DevOps evolution. Because they fail to evolve or die like dinosaurs. The significance of organizational buy in, and oh it is significant indeed, and why team identities and interaction models matter. Not to mention weither the use of automation and the cloud translate to DevOps success. All that and more awaits you. Visit: www.puppet.com to download your copy of the report now!Corey: Up next we've got the latest hits from Veem. Its climbing charts everywhere and soon its going to climb right into your heart. Here it is!Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Jesse DeRose, my colleague, and cloud economist at The Duckbill Group. Jesse, thank you for joining me, even though when I asked you, it isn't exactly like you felt you had much of a choice.Jesse: [laugh]. I appreciate being on this podcast with you. I think you've had an opportunity to talk to a few other folks from our organization so far, so I'm just happy to be included. I'm hoping that I get the Members Only jacket after this recording.Corey: Oh, absolutely. The swag that goes out to guests is secret and wonderful, all at the same time. So, let's start at the very beginning. It turns out that despite the prevalent narrative that we put out there, are clouds' economist did not just spring fully-formed from the forehead of some God, and appear—fully-formed—ready to slice AWS bills to ribbons. There's a process and there's always an origin story. Where do you come from? Where were you before this?Jesse: So, my background is mixed. I started with a management of information systems degree, which is inherently interdisciplinary. You're linking, sort of, the importance of both technical systems and business goals and outcomes into a single degree. And when I graduated with this degree, I went out into the working world and said, “Okay, I'm here. I'm ready. Here's my degree. Here's what I'm interested in. Let's do this.”And nobody knew what to do with me, Corey. Nobody knew what management of information systems was. They simultaneously said, “You know, you don't have a computer science or a computer engineering degree, so we don't believe that you've got programmer chops.” Especially since this is the golden age of boot camps and quickstart programming books, and this movement that anybody can be a developer, which makes it even stranger that I'm not spending my weekends learning every programming language under the sun and automating the little tasks.Corey: Oh, absolutely. In fact, I got the exact same feedback; I don't have a degree, and, “Oh, you couldn't possibly be a good programmer because you don't have the degree, you didn't go to the right schools, you didn't go to a boot camp. And when I asked you to write some sample code to demonstrate how to program, you just started crying instead.” Because yeah, it turns out, I'm actually not a good programmer, I just, sort of, brute force my way through it. And so far, so good.Unfortunately, people aren't paying me to program these days, for excellent reason. In fact, I strongly suspect some people are paying me not to. But yeah, now it's funny to laugh about, but back when you're getting started, and going out in that space, in the world of operations, which we both came up in, and looking at it through a lens of the SRE movement, suddenly, “Hey, you used to be really, really good at all these Linux things and working on systems and keeping them up. Great. Now, learn to code.” And that was a big lift, at least for me.Jesse: Absolutely. I struggled with that so much because every company that I interviewed with, every company that I even talked to, just assumed that I had some kind of programming experience and didn't want to talk to me if I didn't have programming experience. And to me, looking back, I think I really look at it like I may not have the programming chops to be the software engineer that is going to write the code for you, day in and day out, but I am the person who knows enough that I can have the conversation with the software engineers. I can be the SRE, I can be the ops person that has the conversation with your software engineers and knows the things that they're talking about, but also knows when to get out of their way and let them be the expert and do what they do best.Corey: There's something to be said for valuing expertise in areas that are—how to put this—not the thing that you think you're looking for. I mean, back when I was getting jobs, before The Duckbill Group and I would be the first ops hire into a team of developers—which happened a few times—the process was always the same, where you'd have a bunch of developers asking what they thought were ops questions, or just giving up on that entirely and trying to figure out how decent have an ops person I would be by how badly I programmed.Or, “Oh, okay, cool. You're an ops person. Great. Can you invert a binary tree on a whiteboard?” It's, “No, but I can invert a rack in your data center. I'll go rage-flip the rack. Why not?” And it takes time. You have to guide those interviews and those conversations. But it's always weird.] interviews are always weird because you're being judged on a skill set that only matters when you're interviewing for a job.Jesse: Yes. This is one of the things that I struggled with the most because I knew that most of the people who were interviewing me were either business people themselves—so they assumed that because their engineering team thought a certain way and acted a certain way that I should act that way—specifically, too—the same way as everybody else on the team. Or they were software engineers themselves, so they said, “Okay, I know how to invert a binary tree”—to your point, Corey—“So, do how to invert a binary tree. If you know how to do that, then sure, you can be part of my team because you know how to do these things and think about these things the same way I do.” Whereas because I was coming from this operations space, I knew other things that were equally as important but weren't part of the conversations that they were used to having, day-to-day.So, they didn't understand that just because I didn't have the same engineering chops as them, that I didn't have important information to share and wasn't able to stand on my own two feet in other ways. And that was one of the things that I really struggled with when I was starting out in the industry because I was thinking to myself, well I have such passion to be part of these conversations, to have that conversation between the business side of the organization and the engineering side of the organization, from an ops perspective, from a business perspective, from a technical perspective, and if I can't convince these people of my own volition, of my own passion for the good of the company, maybe data will help. Maybe there's something that I can find from a scientific research perspective. Maybe there's something that—I'm sure somebody else has already researched this topic or found the same problems that I have in this space, and maybe they're already talking about it, and maybe I can ride their coattails, so to speak, or follow in their footsteps and use the information that other people are talking about the industry to help me not just land these jobs, but ultimately better sell myself and help these companies move forward.Corey: That is fundamentally an encapsulation of what I believe the ops role to be of, make things better, and move them forward. But man, do we get stuck in an awful lot of weird and strange places. And interviewing itself is a skill. Giving an interview, very often, it's a, “I know a bunch of things that are trivia, but I know them. And if I know them, everyone must know them, therefore, if you don't know them, you must be bad at things.” And it turns out—for better or worse—being able to memorize the documentation and spit out answers is not indicative of whether someone is a good ops person or a terrible one.Jesse: Absolutely. I think that is one of the biggest problems that I have faced and one of the biggest problems in the interviewing space today because it's not just about, can you regurgitate this information, but it's about how do you think? How do you look at problems? How do you communicate to the rest of your team, and within the rest of the organization? Those technical skills are ultimately important because you do need to understand some amount of technical information to have those conversations, but the soft skills are also super, super important to be able to communicate effectively, to be able to think collaboratively, and help everybody, not just yourself, but help the team build that shared purpose and move forward together.Corey: We're talking right now, so far, about traditional ops roles. Then we have what we do here, which is beyond the rest of all of that, where all of what we just said is necessary but not sufficient. Then it comes down to great, okay. So, you understand how systems work together; we found that, for what we do and how we do it, you need to be a competent ops person as a fundamental tenet.Otherwise, learning what all these AWS services do will occupy you for the next three years. So okay, we start off there. Then on top of that is, okay, there are consulting skills that it turns out are possible to teach, but incredibly challenging and time-consuming because a lot of them boil down to, can you be in a meeting with stakeholders of various levels? Can you deliver bad news in a way that they don't hate you? Because they don't really want to pay you just say yes to whatever they think.And can you do that in such a way without, you know, actively insulting them, which sounds like a strange thing until you realize, oh, wait, that's right. I do that, too. So ooh, yeah. Corey is going to have that problem, isn't he? Yeah. And that's part of the beautiful part about this place is that finally, I'm able to hire people like you.You were the first cloud economist here, which meant suddenly I didn't have to do it all myself and my mouth slowly stopped getting me in trouble in consulting engagements, so I could spend more time having my mouth getting me in trouble on Twitter.Jesse: Yeah, I have to tell you, Corey, when I originally spoke with you and Mike about this role, I had just taken another operations position with a tech startup, and I was about two months into the role. And Mike sat down with me for coffee one morning and said, “Hey, we're thinking about doing this thing. Are you interested?” And I said, “Yes, but I just started this other operations gig. I can't up and leave them; I really care about the team, I really care about the company. And it would look really bad on me if I just, you know, two months left.”Because—unless they were a really, really awful employer, which they weren't. So, I said, “Sure. I'm interested in doing some kind of part-time work.” And that's ultimately where I started with you and your business partner, Mike. And I have to admit to when Mike originally approached me and said, “Hey, this is what we are thinking about; this is what we're doing,” I didn't really think twice about the opportunity because I wanted to work with you and Mike again.But the way that Mike described the work, just didn't stick with me. It didn't resonate with me, it was more about, “Hey, I would love to work with Mike and Corey again,” than, “Oh, my God. This sounds like the dream role that I want to be a part of.” And then, when I came back to Mike, probably, I don't know, a month or two later, after I had started working part-time with both of you, I said, you know, “Mike, I don't think I really made myself clear. I want to make sure that I help you understand, ultimately, the things that I want to do are having these conversations, being that bridge with the business side, and being able to talk tech with the tech side, and being able to talk business, and make sure that both sides of the conversation are aligned.”And he just looked at me and said, “What do you think we're doing? What do you think we sold you on?” And it was that aha moment where I thought, “Oh, my god, yes.” I had already said yes; I was already working with both of you part-time, but that was the moment that really solidified it for me of, this is what I've kind of been moving towards. I've been wanting to be that person that can speak both languages and have a conversation with both sides of the table, and speak to multiple different audiences, and now I'm finally getting the chance to do that. I'm getting the chance to grow both skill sets, which I think is extremely rare in a lot of the smaller tech spaces that we see today.Corey: You've hit on one of the secrets of The Duckbill Group if I can be so grandiose as to claim that. And it's true because we take a look at people that we bring in, and things that they're good at, and things that we do—the things we do publicly and the things that we do, sort of, behind the scenes and there's no reason we don't talk about them publicly, but there's no real reason for us to do so. Easy example, and what I want to talk to you about next is, you are deep into improving understanding of complex systems, both technical—okay, great people expect that—and organizational, which sometimes throws people for a loop. And it sounds like a weird thing to focus on here because we fix the AWS bill. We do not bill ourselves as management consultants, we do not bill ourselves as coming in and we will restructure your organization because that sounds patently ridiculous, and no one in their right mind is going to buy that thing.I wouldn't buy it, at least not for me. My God, there are large consultancies that specialize in these sorts of things. I don't know how they do it because I certainly don't. We're not here to sell that, though. Fixing the AWS bill—I mean actually fixing it. Fixing the business problem tied to it mandates an understanding of those complex systems. And your expertise and interest in that area is incredibly helpful here. Tell me more about it.Jesse: This is one of the things that I've been really fascinated by ever since I joined Duckbill Group. I think everybody in Duckbill Group has a superpower or has a really passionate hobby to some extent, which makes each of us really interesting, unique individuals that can focus in different areas of a client's bill or a client's pain points when it comes to cloud cost management and help, and find the parts that are frustrating, find the levers that can be moved, and point them out and say, “Okay, this is ultimately where you want to pull this lever or not pull this lever to make these changes.” And to your point, Corey, the one that is most interesting and passionate for me is that organizational development space. It is really understanding, not just the small things that we can do today to help you save money on your AWS bill, but how we, and collectively how our clients can think about costs long term to save money on their AWS bill. And I know that sounds really, really broad, and that's part of why I think that there is a lot of nuance in this space, to your point about other organizations or other vendors that are providing these consulting services, and I think is also something that is also difficult to sell, which is why it's not our expertise in terms of what we are on the cover trying to sell to any of our clients.But I definitely think that there is opportunity to have some of those conversations within each of our clients' spaces to talk about some of the pain points that we see that may ultimately lead to better cost management practices long term, things that ultimately might help the engineering teams communicate better with finance on a long term basis, help the finance team and any of the leadership team more collaboratively talk with the engineering teams about understanding how much money is the product costing us? How much can we continue to spend on this product? Or how much can we discount one of our products for our customers before we are losing shares, losing money? What are the fine lines that we understand, based on how much money we're ultimately spending on these features, on these products, that will help us make better data-driven decisions about other parts of the company?Corey: I really love installing, upgrading, and fixing security agents in my cloud estate! Why do I say that? Because I sell things, because I sell things for a company that deploys an agent, there's no other reason. Because let's face it. Agents can be a real headache. Well, now Orca Security gives you a single tool that detects basically every risk in your cloud environment -- and that's as easy to install and maintain as a smartphone app. It is agentless, or my intro would've gotten me into trouble here, but it can still see deep into your AWS workloads, while guaranteeing 100% coverage. With Orca Security, there are no overlooked assets, no DevOps headaches, and believe me you will hear from those people if you cause them headaches. and no performance hits on live environments. Connect your first cloud account in minutes and see for yourself at orca.security. Thats “Orca” as in whale, “dot” security as in that things you company claims to care about but doesn't until right after it really should have.Corey: And I want to call out that this is something that we are comparatively enthusiastic amateurs around. It's valuable; it's important; it's an awful lot of deep work, but I'm not sure that we go more than three working days without referencing Dr. Nicole Forsgren's work, internally, as we think about these things. So, if you're hearing this, and you think that okay, AWS bill, fine, whatever. We really want to talk about organizational challenge and improvement, oh, my God, talk to Dr. Forsgren. Holy crap. She's been on this show, at least I think, three times now, and every time I feel like I'm lucky to get her. Most weeks, you know, I'm stuck with people like you. My God, Jesse.Jesse: [laugh].Corey: But no, her work is seminal in this space. And in seriousness, every time I start to question the value of expertise, I look at how deep she goes on all of these things and the level both of understanding that's baked into this, and the amount of sheer work that it takes for her to take all of that very deep, penetrating analysis, and make it accessible and understandable. But every time I look at her work, I come away more impressed than I started, and that wasn't a low bar, to begin with.Jesse: Yes. And this gets back to my earlier comment about, I just want to be here to help. And in a lot of cases, when I was starting out, folks didn't know what to do with me because I didn't have data, I didn't have any information. But we have folks in the industry like Dr. Nicole Forsgren, and other folks who are doing the research, who are knowledgeable in this space, who are putting in the effort to run these studies, to analyze this data, to share the results—and to your comment, Corey—to share the results in a way that makes sense to everybody, that's easy to read, it's approachable, it's understandable.And I am so thankful to have folks in the industry who are doing that work because that is not my expertise. But that means that I get to say to the folks that I'm working with—internally and with our clients—“Hey, don't take my word for it. There are other folks who have done research, and here's what the data says, and here's how we can help you apply this work, or you can apply this work within your own organization.”Corey: And this is the challenge in some cases, too, where there's a lot of organizational theory, and that is being advanced heavily, and in ways that makes teams more effective is super helpful. The challenge, of course, is that sitting here and talking about the theoretical layout of teams and how to improve functioning as an organization is all well and good, but we're brought in by our clients to help them with their AWS billing situation, so at some point the conversation has to evolve beyond, “Okay, so here's what you could do in theory, in a vacuum, assuming spherical cows, et cetera, et cetera.” And their response is, “That's great. You actually going to fix the bill or just pontificate for a while here?” So, for better or worse, we don't really get to sit there and have deep organizational conversations at length with our clients, just because that's not the problem we're there to solve. Everyone's busy and we want to make sure that we're respectful of their time.Jesse: Yeah, one of the things that I've learned through my time with Duckbill Group and through other similar roles in the past, is that I may have a strong passion, I may have this strong guiding light in my head, but it's not the same guiding light that our clients or our customers have. And that's fine because we don't need to necessarily have the same goal in sight, but that means that I, to best serve our clients or best serve our customers, need to make sure that I am aligning, that Duckbill Group is aligning with the clients that we're working with, with the organizations that we're working with. So, I may be thinking to myself, “Oh, my gosh, I would love to come in with this long list of organizational development practices and share a million different things,” but that's not ultimately what they need. Maybe that's something that they're going to want long-term, but it's not what we're here for today. And it's more important to help serve the client that is in front of me that is asking for things now, today, than try to educate them on quote-unquote, “What their problem is,” and then sell them on a solution.Corey: The thing that I think gets lost as well, whenever I start talking in-depth about what we do on Twitter, for example, is generally from other engineers whose response is, “Okay, yeah, sounds great. You come in and say that it will save a bunch of money before you rearchitect our application. But that's an awful lot of engineering time, so I bet engineers most hate you.” And the honest answer is, “Yeah. We know that you would save some significant money if you rearchitected your application, but we also pay attention to organizational dynamics, and we know you're not going to do it because there's no business justification for doing it. So, we're not going to bring it up, other than, possibly, in passing, just so people are aware of the relative benefit if they want to bake that in.”But we don't go in and suggest nonsense that is abhorrent. We all started as engineers ourselves. We are sensitive to engineering time, both in terms of what engineers enjoy working on—which is more important than people think—as well as the sheer cost of engineering time. People get concerned about the AWS bill, but it's invariably pale compared to the cost of the people working on the AWS infrastructure. You want to optimize the right things, and then you want to get back to doing what your company does, not continue to iterate forward and spend thousands of dollars to save tens of dollars.Jesse: Yeah, it's an extremely difficult balance to find. And it's really important to think about, is the ROI on this change worth the change? How much money am I ultimately going to save for the amount of engineering effort that I'm going to put in? Because we don't want to run in and tell your engineering teams, “Rearchitect everything,” if it's going to maybe save them tens of dollars. We want to make it very clear that here are the different levers that you can pull to affect change within your AWS bill, to optimize your AWS bill.It's up to you which ones you want to pull. We are just giving you that guiding path, and then you have the option to say, “Yes, I want to spend the engineering effort to get this kind of ROI.” Or, “You know what? I don't think that's the priority for us right now.” One of the things that I've noticed with a number of our clients is that balance of, do we focus more time on building new features which brings in new customers, or do we spend time on the existing infrastructure and making changes to the existing workloads that we have?And it's this delicate balance of internal work—the things that you have put on the backburner over time that you ultimately say, “Well, I'll come back to that,” versus the new things that are ultimately going to bring in new customers, bring in new users, potentially bring in more money. Because both are important, but there needs to be a delicate balance of both, and I feel like that's one of the biggest challenges that we face when managing an AWS bill and trying to optimize, and organize, and better manage cloud costs.Corey: And that's fundamentally what it comes down to what is the best outcome for the client? And the answer to ‘what is best?' Varies based upon their constraints, what they're focused on, what they're trying to achieve. You could look at the end result of one of our analyses for a customer, and take issue with it in a vacuum of but they're spending all this money on things that they shouldn't be doing, or don't need to be. Why didn't you suggest this, or this, or this, or this?And the answer is because, based upon our conversations, we knew that they weren't going to do it, and suggesting things that we know they're not going to do is one of the best ways to erode trust. We're there to deliver an outcome. We're not naive software that is just running a pile of tools on an Amazon bill and saying, “Here you go. Have fun.” We're not the billing equivalent of a Nessus scan that someone slaps their logo on top of, drops off, then it's 700 pages long. “Have a good one. Check, please.” It doesn't work. Not well, anyway. It doesn't drive to lasting change.Jesse: Yeah. And I think that's ultimately part of where we come in best because we ultimately want to be that bridge between the engineering teams, and finance, and leadership, and essentially the business side of the business. We want to give both sides the information that they need to be able to speak effectively and collaboratively with the other side. We want to make sure that the finance side understands enough tech that they can work with the engineering teams to guide them in terms of what goals are important for finance, in terms of managing budget, in terms of forecasting spend. And then from the flip side, we want to make sure that engineering understands that the business collaboratively wants to manage these things, and also help build the organization, and engineering has great, great potential to do little things, take little steps to help the organization get the data that they need to make these decisions.And that scratches that itch for me. That scratches that itch of how can I really be both sides of the conversation? How can I flex the business lingo and also flex the tech lingo, maybe not in the same conversation but with the same client over time, to really help both sides understand that both sides are important, and both sides need to understand each other in order to help the business succeed?Corey: And that's ultimately what it comes down to. Jesse, thanks for taking the time to speak with me. If people want to hear more about what you have to say, and how you like to say it, where can they find you, other than go into The Duckbill Group and get me a consulting engagement underway?Jesse: So, there's two places that folks can find me. The first is on Twitter at @jesse_derose. And then the second is our other podcast, the AWS Morning Brief podcasts, on the mornings where we don't get to hear your melodious voice, Corey.My colleague, Pete Cheslock and I are talking about all of the interesting things that we've seen on AWS, from client-specific situations to things that we've seen on the job in previous organizations that we use to work at, to new features that AWS is releasing. There's a whole slew of interesting things that we get to talk about from a more practical perspective. Because there's so many releases coming out day-to-day that you're obviously helping us stay on top of, Corey, but there's so many other things that we want to make sure that we are talking about from the real-world applicable perspective.Corey: And we will, of course, put links to these things in the [show notes 00:27:48], but you should already be aware of most of them, at least the ones that are on the company side. Jesse, thank you so much for taking the time to speak with me.Jesse: Thank you for having me.Corey: Jesse DeRose, cloud economist here at The Duckbill Group. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment telling me that organizational dynamics really aren't that hard and you could solve it better than Dr. Forsgren does, in a weekend.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.This has been a HumblePod production. Stay humble.
Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways that we've seen AWS used and abused in the wild. Today, we're going to be talking about the relationship between cost optimization work and investing in reservations or private pricing with AWS. This is kind of a situation conversation. Let's say you've got three months left on your EDP, or maybe your spend is reaching the point where you're starting to think about investing in, or signing an EDP. But you've also got some cost optimization opportunities that you want to work on. How do you prioritize those two ideas?Tim: I think when we're talking about this, first it's important to talk about what goes into an EDP, like, what it is and what it involves. So, EDP for AWS is Enterprise Discount Program, and what it involves is you making a monetary commitment to AWS to spend a certain amount over a certain amount of time. So, a three year EDP, you're going to spend X amount in one year, X amount the next year, and X amount the third year for a total of whatever you decide on. So, you know, AWS typically going to want 20% year-over-year growth, so you're going to say—you're going to spend a million dollars, and then a million dollars plus 20% is something like $1.2 million; then, you know, 20% of that and so forth and so on.And then so your total commit will be somewhere around, like, $3.6, $3.7 million, we'll say, right? Once you signed the EDP, that's how much you're going to get billed for, minimum. So, it's important to cost optimize before you make that commitment because if AWS is expecting you and you're on the hook to make 20% year-over-year growth, but then you optimize and you save 20% of your bill, it won't matter because you're still going to owe AWS the same amount of money even if you cost-optimize.Jesse: Yeah, I want to take a step back and talk about EDP—as we mentioned, Enterprise Discount Program—also has—there's a couple other flavors that give you a variety of different types of discounts. EDP generally focuses on a cross-service discount for a certain annual commit, but there are also private pricing agreements or private pricing addendums, and other private pricing, generally speaking, offered by AWS. All of those basically expect some amount of either spend on a yearly basis or some amount of usage on a yearly basis, in exchange for discounts on that usage. And really, that is something that, broadly speaking, we do recommend you focus on, we do recommend that you invest in those reservations, but it is important to think about that—I agree—I would say after cost optimization work.Amy: The thing is that AWS also provides discounts that are commandment required, that you don't need an EDP for, namely in reservations and savings plans. So, you would similarly be on the hook if you decide, “I have this much traffic, and I want to savings plan or reservation for it.” And then suddenly you don't have that requirement anymore, but you still have to make up that commitment.Tim: I'll say, I think too, that also matters when you're looking at things like reservations. If you're going to reserve instances, you're going to get an idea of how many you're specifically going to need, so that way you're not reserving too many, and then you optimize, you downsize, and all of a sudden, now you have all these reservations that you're not going to use.Jesse: One thing to also call out: when renewing an EDP, or private pricing, or when entering into a new agreement for any kind of private pricing with AWS, they will generally look at the last six months of your usage—either broadly speaking if it's an EDP, or specifically within a specific AWS service if it's private pricing for a specific service—and they will double, basically, that spend over the last six months and expect you to continue spending that. So, if you spent a high amount of money over the last six months, they're going to expect that kind of trend to continue, and if you enter into an agreement with that 12-month spend, essentially, going forward, and then make cost optimization changes, you're ultimately going to be on the hook for this higher level of spending you're not spending any more. So, if you focus on that cost optimization work first, it will ultimately give you the opportunity to approach AWS with a lower commit level, which may ultimately mean a lower tier of percentage discount, but ultimately, then you're not on the hook for spend that you wouldn't otherwise be spending.Tim: I think one of the main things people see, too, is when they've looked at, like, oh, what's the low hanging fruit for me to get lower the cost? They'll think, “Oh, well, I can do EDP,” because AWS is going to want you to sign on; they would love to have that guaranteed money, right? And a lot of times, that's going to be a much easier thing to do, organizationally, than the work of cost optimization because almost always, that involves engineering hours, it involves planning, it involves some changes that are going to have to be made that's probably going to be harder than just signing a contract. But again, it's super necessary because you really need to know, have eyes open, when you're going to go, and figure out what you're going to commit, whether it's private pricing agreement, or an EDP, or reservations. You want to go in there and at least decide what you want to do, what it should look like, get as optimized and as lean as you can, then make your commitments. And then once you get to an EDP, that's when you're going to want to do your reservation or savings plans purchases and things like that, so you do that with a discount across those.Jesse: Yeah, that's another important thing to point out: focus on the cost optimization work first. Get your architecture, your workloads, as optimized as possible, or as optimized as you can within the given timeframe, then focus on the investment because then you'll be able to have a much better idea of what your growth is going to look like year-over-year for an EDP or any kind of private pricing. And then after that, purchase any reservations, like reserved instances or savings plans because ultimately, then you get not only the discount from the EDP that you just signed, but any upfront payments that you make, or partial upfront payments that you make for those reservations applied towards your first year EDP. So ultimately, not only are you getting a discount on that, but you are also able to put money towards that first-year commit; you're essentially giving yourself a little bit more wiggle room by purchasing reservations after you've signed an EDP.Tim: And another way to game that system is if you know that you're going to be undertaking some projects, especially that you want to get discounts around, and you're going to need to utilize software or service or anything like that involves an AWS partner on the AWS marketplace, you're going to want to do that after you sign your EDP, too, because even though you may not get a discount on it, that money will still count towards your commit.Corey: I really love installing, upgrading, and fixing security agents in my cloud estate. Why do I say that? Because I sell things for a company that deploys an agent. There's no other reason. Because let's face it; agents can be a real headache. Well, Orca Security now gives you a single tool to detect basically every risk in your cloud environment that's as easy to install and maintain as a smartphone app. It is agentless—or my intro would have gotten me in trouble here—but it can still see deep into your AWS workloads while guaranteeing 100% coverage. With Orca Security there are no overlooked assets, no DevOps headaches—and believe me, you will hear from those people if you cause them headaches—and no performance hits on live environment. Connect your first cloud account in minutes and see for yourself at orca dot security. That's orca—as in whale—dot security as in that thing your company claims to care about but doesn't until right after it really should have.Tim: It is important to talk about the future goals for your company, from a financial perspective, both at an architectural level but also at a strategic level, so you can make good quality decisions. And, you know, to toot our own horn, that's a lot of where our expertise comes in, where we can say, “These are the order you're going to do these things in, and these are what you should prioritize.” I mean, everyone knows that in the end, the net result should still be the same. You're going to have to do the engineering and architecture work to optimize; you're going to have to do the administrative stuff to sign these agreements to get discounts, but you need to know what to prioritize and what's going to be most important, and sometimes you don't have the insight on that. And that's where if you don't, get someone in there to help you figure out what's what, what's going to give you the best, most bang for your buck, but also what's going to make the most sense for you going forward, six months, a year, two years, three years, and so forth and so on. So, it is okay to not know these things. Nobody's an expert on everything, but it behooves you to rely on the people who are experts when it's a blind spot for you.Jesse: I think that's a really good point that you make, Tim. One of the things that we see in a number of organizations that we work with is essentially a disconnect between the folks who are—well, two disconnects really: one between the folks who are doing the work day-to-day, and another between the folks who are purchasing reservations. But that also a disconnect between the people who are purchasing the reservations and potentially the people who are purchasing or investing in some kind of Enterprise Discount Program or private pricing. And to Tim's point, it's really important to get all of those people in a conversation together, get everybody in a room together, so to speak, to make sure that everybody understands what everybody else is doing so that finance and engineering and product and leadership all understand together that the cost optimization work is going on, that reservations are being purchased, that we're having a conversation about investing in some kind of private pricing with AWS. So collaboratively, collectively, everybody can make a decision together, make a data-driven decision together, that's going to ultimately help everybody, essentially, win and accomplish their goals.Amy: Speaking of collaboration, we often talk about having a good relationship with your AWS account manager, and this is one of those places that having a good rapport really works in your favor because if you are in a lot of communications with your account manager, and you know each other well, and you have a good working relationship, and they are good at their job, then they'll know that you are using XYZ service, and you're using at a high volume, they will be able to tell you, it's like, “Hey, you hit a threshold. Let's see if we can get you some extra discounts.” They'll be the ones who can actually know what those discount programs are and be able to facilitate them.Jesse: All right, well, that will do it for us this week, folks. If you've got questions you'd like us to answer please go to lastweekinaws.com/QA; fill out the form and we'll answer those questions on a future episode of the show. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us how you would cost-optimize your organization.Announcer: This has been a HumblePod production. Stay humble.
TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we've seen AWS used and abused in the wild. Today, we're going to be talking about AWS, an open-source software. Now, that's kind of a broad topic, but there have been some specific, recent events I'll say, over the last year maybe or maybe even less, related to AWS and open-source software that really got us talking, and I wanted to have a deeper conversation with both of you on this topic.Tim: Well, you should probably start by going over some of the things that you're mentioning, when you say ‘some of these things,' what are those things, Jesse?Jesse: Yeah. So, I think the best place to start is what constitutes open-source software. And specifically, I think, not just what constitutes open-source software, but how does that differ from an open-source company?Tim: So, open-source software can be anything: Linux kernel, bash, anything like that, any Python functioning module. If you make a piece of software, whatever it is, and you license it with one of the various open-source licenses, or your own open-source license or whatever, it's something that the community kind of owns. So, when they get big, they have maintainers, everything like that, but at its essence, it's a piece of software that you can freely download and use, and then you're free to modify it as you need, and then it's up to the specifics of the license to whether you're required to send those modifications back, to include them, or to whatever. But the essence is that it's a piece of software that's free for me to use and free for me to modify under it's license.Jesse: And one of the other things I want to add to that is, correct me if I'm wrong here, but isn't a lot of open-source software is very community-owned, so there's a lot of focus on folks from the community that is using this software giving back not because they need to under the licensing, necessarily, but because they want to continue using this and making it better over time.Amy: I think one of the issues is that becomes a very opinionated kind of statement where there are a lot of people in the open-source community who feel that if you're going to use something and make changes to better suit what your needs are, that you should be able to submit those changes back to the community, or back to whoever owns the base of the software. But that said, it's like the community edition of MySQL before Microsoft bought it, where the assumption was that there's essentially a candidate of it that anyone can use without the expectation of submitting it back.Jesse: So, that's a broad definition of open-source software, but how does open-source software, broadly speaking, differ from an open-source company? I'm thinking specifically there is the open-source software of Elasticsearch, for example, or I should say, previously the open-source software of Elasticsearch that was owned by the open-source company, Elastic. So, what does that relationship look like? How does an open-source company like that differ from the open-source software itself?Tim: So, there are typically a couple of ways. Usually, a company that is the owner of an open-source product still has some kind of retention of the IP in their various licenses that they can do that with, but essentially—and this is in the words of one of the founders of Elastic—that they're benevolent dictators over the software. And so they allow folks to contribute, but they don't have to. And most of those open-source software companies will have a commercial version of that software that has other features that are not available, packages with support or some of the things like that, some kind of value-added thing that you're going to wind up paying for. The best way to describe—like you said—there's the company Elastic and then the product Elasticsearch.I relate back to before: there was Red Hat Linux, which was open-source, and then the company Red Hat. And I remember when they went public and everyone was shocked that a company can make profit off of something they gave away for free. But while the core of the software itself was free, the support was not free, nor was the add-on features that enterprises wanted. And so that tends to be kind of what the business model is, is that you create the software, it's open-source for a while to get a big user base, and then when it gets adopted by enterprises or people that really would pay for support or for other features, that's when the license tends to change, or there's a fork between the open-source version and then the commercial version.Jesse: And it definitely sounds like there can be benefits to an open-source company essentially charging for not just the open-source software, but these extra benefits like supports and additional features because I know I've traced multiple code bugs back to a piece of open-source software that there's a PR or an issue that has been sitting open for months, if not longer because the community just doesn't have the time to look into the issue, doesn't have the time to work on the issue, they are managing it on their own, separate as a side job, separate from their day-to-day work. Whereas if that is a bug that I'm tracing back to a feature in an open-source piece of software, or I should say software that I am paying for through an open-source company, I have a much clearer support path to a resolution to resolving that issue.Tim: And I think what the end up doing is then you see it more like a traditional core software model, like, you know, a la Oracle, or something like that where you pay for the software essentially, but it comes packaged with these things that you get because of it, and then there's a support contract on top of it, and then there's hosting or cloud, whatever it is, on top of that, now, but you would still end up paying for the software and then support as part of the same deal. But as you know, these are for-profit companies. People get paid for them; they are publicly traded; they sell this software; they sell this product, whether it's the services or the hosting, for profit. That is not open-source software. So, if company X that makes software X, goes under, they are acting like the software would then go under as if the software doesn't belong to the community.So, a business that goes after a business is always going to be fair play; I believe they call it capitalism. But when you talk about going after open-source software, you're looking at what Microsoft was doing in the '90s and early 2000s, with Linux and other open-source challenges to the Windows and the other paid commercial enterprise software market. When folks started using Linux and servers because it was free, customizable, and they could do pretty much everything they wanted to or version of it that they were using commercial Unices for, or even replacing Windows for, you didn't really see the commercial Unices going after it because that very specialized use cases; the user had specialized hardware. What folks were doing, they're buying Wintel machines and putting Linux on them, they were getting them without Windows licenses, or trial licenses, throwing Linux on it. And Microsoft really went after open-source; they really went after open-source.They were calling it insecure, they were calling it flash in the pan, saying it would never happen. They ran a good marketing campaign for a long time against open-source software so that people would not use it and would instead use their closed-source software. That is going after open-source, not going after quote-unquote, “Open-source companies.”Jesse: Yeah, I think that's ultimately what I want to dive into next, which is, there's been a lot of buzz about AWS going after open-source, being a risk to open-source software, specifically, with the release of AWS Managed Services for software like Elasticsearch, for example, Kubernetes, Prometheus vs. Other open-source packages that you can now run as a managed service in AWS. There's a lot of concern that AWS is basically a risk to all of these pieces of open-source software, but that doesn't necessarily seem to be the case, based on what we're talking about. One of the things that I want to dive into really specifically here is this licensing idea. Is it important to end-users? How would they know about what license they're using, or if the license changes?Tim: I'll let Amy dig in on it because she's probably the expert of three of them, but I will say one case in point, I remember where licensing did become very important was Java. JDK licenses, when Oracle started cornering the market on enclosing all the licenses, you had to use different types of Javas. So, you had to get, like, open JDK; you couldn't use Sun, Oracle Java, or whatever it was. And so that became a heavy lift of replacing packages and making sure all that stuff was in compliance, and while tracking packages, replacing them, doing all the necessary things because if you're running Java, you're probably running it in production. Why you would, I don't know, but there are those things that you would have to do in order to be able to just replace a package. The impact of the license, even if it doesn't cost a dime for usage, it still matters, and in real dollars and real engineering time.Amy: Even free licensing will cost you money if you do it wrong. The reason why I love talking about licensing is because I used to work for the government—Jesse: [laugh].Amy: —and if you think a large company like Amazon or Microsoft loves doing anything to rattle the cage of smaller businesses, it's not nearly as much as they love doing it to the government. So, any company that has a government-specific license, and the government is not using it, they will get sued and fined for a bunch of money, which sounds like a conflict between a super-large company and the government and who the hell cares about that, but this also translates the way they handle licensing for end-users and for smaller companies. So, for the most part for the end-user, you're going to look at what is generally sent to you to use any piece of licensing, the EULA, the End-User License Agreement, and you're just going to say, “Yeah, fine, this thing is 20 pages long; I'm not going to read this, it's fine.” And for most end-users, that is actually, you're good to go because they're not going to be coming after small, single-person users. What these licenses do is restrict the way larger organizations—be it the government or mid to larger companies—actually use their software, so that—this is a little dating—someone does not buy a single disk that does not report home, and then install that one disk on 20 computers, which is a thing that everyone has seen done if they've been in the industry long enough.Jesse: Yeah.Amy: Yeah. And it means things like licensing inventory is important, to the single you're using this license at home and you install Adobe on three computers, you would think it's not… would not hurt their value very much, but they also make it so that you can't even do that anymore. So, in purchased software, it makes a big deal for end-users; if it's just something free like being able to use some community SQL workbench just to mess around with stuff at home or on personal projects, you're usually going to be okay.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like HubSpot, Klarna, Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Jesse: Yeah, this is a really big issue. There's so much complexity in this space because Tim, like you said, there's some amount of capitalism here of AWS competing with open-source companies; there's business opportunities to change licensing, which can be a good thing for a company or it could be a terrible thing for a company's user base. There's lots of complexity to this issue. And I mean, in the amount of time that we've been talking, we've only really scratched the surface. I think there's so much more to this space to talk about.Tim: There really is, and there's a lot of history that we really need to cover to really paint an accurate picture. I think back when web hosting first became a thing, and everyone was running LAMP stacks and nobody was saying, “Oh, no, using cPanel is going to kill Apache.” That wasn't a thing because, yeah, it was a for-profit company that was using open-source software to make money and yet Apache still lived, and [unintelligible 00:15:00] still lived; MySQL still made it; PHP was still around. So, to say that utilizing open-source software to provide a service, to provide a paid service, is going to kill the open-source softwares, at best it's misrepresentation and omits a lot of things. So, yeah, there's a lot of stuff we can dig into, a lot of things we can cover.And the topic is broad, and so this is why it's important for us to talk about it, I think, in the context of AWS and the AWS, kind of, ecosystem is that when you see companies with big crocodile tears, saying, “Oh, yeah, AWS is trying to kill open-source,” it's like, “No, they're not trying to kill open-source.” They may be trying to go after your company, but they aren't the same.Jesse: And it feels to me like that is part of the way that the business world works. And I'm not saying that it's a great part of the way the business world works, but how can you differentiate your company in such a way that you still retain your user base if AWS releases a competing product? I'm not thrilled with the fact that AWS is releasing all these products that are competing with open-source companies, but I'm also not going to say that it's not beneficial, in some ways, for AWS customers. So, I see both sides of the coin here and I don't have a clear idea of what the best path forward is.Amy: As much as I hate the market demands it type of argument, a lot of the libraries, and open-source software, and all of these other things that AWS has successfully gone after, they've gone after ones that weren't entirely easy to use in the first place. Things like Kubernetes, and Prometheus, and MongoDB, and Elastic. These are not simple solutions to begin with, so if they didn't do it, there are a lot of other management companies that will help you deal with these very specific products. The only difference is, one of them is AWS.Jesse: [laugh]. One of them is a multibillion-dollar company.Amy: Oh, they've all got money, man.Jesse: [laugh].Amy: I mean, let's be real. At our pay grade, the difference between a multimillion-dollar and a billion-dollar company, I don't think affects you at your level at all.Jesse: No.Amy: I'm not seeing any of that difference. I am not. [laugh].Tim: Yeah, I definitely think if you all want us to dig into more of this—and we could do a lot more—let us know. If there are things you think we're wrong on, or things that you think we need to dig deeper on, yeah, we'd love to do that. Because this is a complex and nuanced topic that does have a lot of information that should be discussed so that folks can have a clear view of what the picture looks like.Jesse: Well, that'll do it for us this week, folks. If you've got questions you'd like us to answer please go to lastweekinaws.com/QA, fill out the form and we'll answer those questions on a future episode of the show.If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us your thoughts on this conversation, on AWS versus open-source software versus open-source companies.Announcer: This has been a HumblePod production. Stay humble.
TranscriptCorey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we've seen AWS used and abused in the wild with a healthy dose of complaining about AWS for good measure. Today, we're going to be talking about a recent addition to the AWS family: AWS Application Cost Profiler.Tim: But hold on for a second, Jesse, because AWS Application Cost Profiler we can get to; that's rather unremarkable. I really want to talk about how impressed I am with AWS InfiniDash. I've been benchmarking this thing, and it is fan… tastic. It's so good. And we could probably talk about for a while, but suffice to say that I am far more impressed with AWS InfiniDash than I am with AWS Application Cost Profiler.Jesse: You know, that's fair. And I feel like InfiniDash should absolutely get credit where credit is due. I want to make sure that everybody can really understand the full breadth of everything that InfiniDash is able to accomplish. So, I want to make sure that we do get to that; maybe in a future episode, we can touch on that one. But for right now, I have lots of feelings about AWS Application Cost Profiler, and what better place to share those feelings than with two of my favorite people, Amy and Tim, and then all of you listeners who are listening in to this podcast. I can't wait to dive into this. But I think we should probably start with, what is AWS Application Cost Profiler?Amy: It is [unintelligible 00:01:54] in a trench coat.Jesse: [laugh].Amy: Which is the way AWS likes to solve problems sometimes. And in this case, it's talking about separating billing costs by tenants by service, which is certainly a lot of things that people have problems with.Jesse: That is a lot of buzzwords.Amy: A lot of words there.Jesse: Yeah. Looking at the documentation, the sales page, “AWS Application Cost Profiler is a managed service that helps us separate your AWS billing and costs by the tenants of your service.” That has a lot of buzzwords.Tim: Well, to be fair, that's also a majority of the documentation about service.Jesse: Yeah, that is fair. That is a lot of what we saw, and I think we'll dive into that with documentation in a minute. But I do want to call out before we dive into our thoughts on this service—because we did kick the tires on this service and we want to share what our experience was like, but I do want to call out that this problem that AWS Application Cost Profiler is trying to solve. This idea of cost allocation of shared resources, it is a real, valid problem and it is one that is difficult to solve.Amy: And we've had clients that have had this very explicit problem and our findings have been that it's very difficult to accurately splice usage and spend against what's essentially consumption-based metrics—which is how much a user or request is using all the way along your pipeline—if they're not using dedicated resources.Jesse: Yeah, when we talk about cost allocation, generally speaking, we talk about cost allocation from the perspective of tagging resources, broadly speaking, and moving resources into linked accounts and separating spend by linked accounts, or allocating spend by linked accounts. But if you've got a shared compute cluster, a shared database, any kind of shared resources where multiple tenants are using that infrastructure, slapping one tag on it isn't going to solve the issue. Even putting all of those shared resources in a single linked account isn't going to solve that issue. So, the problem of cost allocation for shared resource is real; it is a valid problem. So, let's talk specifically about AWS Application Cost Profiler as a solution for this problem. As I mentioned, we kicked the tires on this solution earlier this week and we have some thoughts to share.Tim: I think one of the main things around this AWS Application Profiler like I said, there's some problems that can be solved there, there's some insights that people really want to gain here, but the problem is people don't want to do a lot more work or rewrite their observability stack to do it. The problem is, that's exactly what AWS Cost Profiler seems to be doing or seems to want you to do. It doesn't get data from, I think it only gets data from certain EC2 services, and it's just, it's doing things that you can already do in other tools to do aggregation. And if I'm going to do all the work to rewrite that stack, to be able to use the Profiler, am I going to want to spend that time doing something else? I mean, that kind of comes to the bottom line about it.Jesse: Yeah, the biggest thing that I ran into, or that I experienced when we were setting up the Cost Profiler, is that documentation basically said, “Okay, configure Cost Profiler and then submit your data.” And [unintelligible 00:05:54] stop, like wait, what? Wait, what do you mean, ‘submit data?' And it said, “Okay, well now that you've got Cost Profiler as a service running, you need to upload all of the data that Cost Profiler is going to profile for you.” It boggles my mind.Tim: And it has to be in this format, and it has to have these specific fields. And so if you're not already emitting data in that format with those fields, now you have to go back and do that. And it's not really solving any problems, but it offers to create more problems.Amy: And also, if you're going to have to go through the work of instrumenting and managing all that data anyway, you could send it anywhere you wanted to. You could send it to your own database to your own visualization. You don't need Profiler after that.Jesse: Yeah, I think that's a really good point, Amy. AWS Cost Profiler assumes that you already have this data somewhere. And if not, it explicitly says—in its documentation it says, to generate reports you need to submit tenant usage data of your software applications that use shared AWS resources. So, it explicitly expects you to already have this data. And if you are going to be looking for a solution that is going to help you allocate the cost of shared resources and you already have this data somewhere else, there are better solutions out there than AWS Application Cost Profiler. As Amy said, you can send that data anywhere. AWS Application Cost Profiler probably isn't going to be the first place that you think of because it probably doesn't have as many features as other solutions.Amy: If you were going to instrument things to that level, and let's say you were using third-party services, you could normalize your own data and build out your own solution, or you can send it to a better data and analytics service. There are more mature solutions out there that require you to do less work.Corey: This episode is sponsored in part by ChaosSearch. You could run Elastic Search or Elastic Cloud or Open Search, as they're calling it now, or a self hosted out stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for App performance monitoring, cyber security. If you're using ElasticSearch consider not running ElasticSearch. They're also available now on the AWS market place, if you prefer not to go direct and have half of whatever you pay them count toward your EDP commitment. Discover what companies like, Klarna, Equifax, Armor Security and Blackboard already have. To learn more visit chaossearch.io and tell them I sent you just so you can see them facepalm yet again.Jesse: I feel like I'd missed something, broadly speaking. I get that this is a preview, I get that this is a step on the road for this solution, and I'm hoping that ultimately AWS Application Cost Profiler can automatically pull data from resources. And also, not just from EC2 compute resources, but from other shared services as well. I would love this service to be able to automatically dynamically pull this data from multiple AWS services that I already use. But this just feels like a very minimal first step to me.Tim: And let's be honest; AWS has a history of putting out services before they're ready for primetime, even if they're GA—Jesse: Yeah.Tim: —but this seems so un-useful that I'm not sure how it made it past the six-pager or the press release. It's disappointing for a GA service from AWS.Amy: What would you both like to see, other than it just being… more natively picked up by other services?Tim: I would like to see either a UI for creating the data tables that you're going to need, or a plugin that you can automatically put with those EC2 resources: an agent you can run, or a sidecar, or a collector that you just enable to gather that data automatically. Because right now, it's not really useful at all. What it's doing is basically the same thing you can do in an Excel spreadsheet. And that's being very, very honest.Jesse: Yeah, I think that's a really good point that ultimately, a lot of this data is not streamlined and that's ultimately the thing that is the most frustrating for me right now. It is asking a lot of the customer in terms of engineering time, in terms of design work, in terms of implementation details, and I would love AWS to iterate on this service by providing that dynamically, making it easier to onboard and use this service.Amy: Personally, what I would like is some either use case, or demonstration, or tutorial that shows how to track consumption costs using non-compute resources like Kinesis especially, because you're shoving a lot of things in there and you just need to be able to track these things and have that show up in some sort of visualization that's like Cost Explorer. Or even have that wired directly to Cost Explorer so that you can, from Cost Explorer, drill down to a request and be able to see what it is actually doing, and what it's actually costing. I want a lot of things.Jesse: [laugh]. But honestly, I think that's why we're here, you know? I want to make these services better. I want people to use the services. I want people to be able to allocate costs of shared resources. But it is still a hard problem to solve, and no one solution has quite solved it cleanly and easily yet.You know what? Amy, to get back to your question, that's ultimately what I would love to see, not just specifically with an AWS Application Cost Profiler necessarily, but I would love to see better native tools in AWS to help break out the cost of shared resources, to help break out and measure how tenants are using shared resources in AWS, natively. More so than this solution.Amy: I would love that. It would make so many things so much easier.Jesse: Mm-hm. I'm definitely going to be adding that to my AWS wishlist for a future episode.Tim: How many terabytes is your AWS wishlist right now?Jesse: Oh… it is long. I, unfortunately, have made so many additions to my AWS wishlist that are qualitative things—more so than quantitative things—that just aren't going to happen.Amy: You become that kid at Christmas that, they get onto Santa's lap in the mall, and it's a roller page that just hops off the platform, and just goes down the wall, and all the other kids are staring at you and ready to punch you in the face when you get off. [laugh].Jesse: [laugh]. All right, well that'll do it for us this week, folks. If you've got questions you'd like us to answer please go to lastweekinaws.com/QA, fill out the form and we'd be happy to answer that question on a future episode. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us how you allocate the costs of shared resources.Announcer: This has been a HumblePod production. Stay humble.
LinksPete and Jesse Talk Account ManagersTranscriptCorey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we've seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we're going to be talking about, really, a couple things; building your relationship with AWS, really. This stems from one of the questions that we got from a listener from a previous event. The question is, “How do the different companies that we've worked with work with AWS? Is the primary point of contact for AWS at a company usually the CTO, the VP of engineering, an architect, an ops person, a program manager, or somebody from finance, a [unintelligible 00:01:00] trainer? Who ultimately owns that relationship with AWS?”And so we're going to talk about that today. I think there's a lot of really great content in this space. Pete and I, back in the day, recorded an episode talking about building your relationship with your account manager, and with your TAM, and with AWS in general. I'll link that in the show notes. That's a great precursor to this conversation. But I think there's a lot of great opportunities to build your relationship and build rapport with AWS, as you work with AWS and as you put more things on the platform.Amy: I think one of the things we always say right off the bat is that you should introduce yourself and make a good relationship with your account manager and your technical account manager, just because they're the ones who, if you need help, they're going to be the ones to help you.Jesse: Yeah, I think one of the things that we should also take a step back and add is that if you are listening to this and you're saying to yourself, “I don't have an account manager,” that's actually wrong; you do have an account manager. Anybody who's running workloads on AWS has an account manager. Your account manager might not have reached out to you yet because usually speaking, account managers don't reach out unless they see that you're spending a certain amount of money. They usually don't start a conversation with you unless you specifically are spending a certain amount of money, have reached a certain threshold, and then they want to start talking to you about opportunities to continue using AWS, opportunities to save money, invest in AWS. But you definitely have an account manager and you should definitely start building that rapport with them as soon as possible.Amy: First question. How do you actually engage your account manager?Tim: So, there's a couple ways to do it. If you have reached a certain spend threshold where your account manager will reach out to you, it's real simple: you just reply back to them. And it kind of depends. The question most people are going to have is, “Well, why do I need to reach out to my account manager? If I just have, like, a demo account, if I'm just using free tier stuff.”You probably don't ever need to reach out to your account manager, so what are the things, typical things that people need to reach out to their account manager for? Well, typically because they want to grow and want to see what kind of discounts are offered for growth, and I want to see what I can do. Now, you can open a support ticket, you can open a billing ticket, but what will end up happening is once you reach a spend threshold, your account manager will reach out to you because they want to talk to you about what programs they have, they want to see how they can help you grow your account, they want to see what things they can do for you because for them, that means you're going to spend more money. Most account managers within a little bit of time of you opening your account and reaching a lower spend threshold, they're going to send you an email and say, “Hey, this is my name, this is how you reach me,” et cetera, et cetera. And they'll send you some emails with links to webinars or other events and things like that, and you can typically reply back to those and you'll be able to get your account manager sometimes as well. But like I said, the easiest way to get a hold of your account manager or find out who it is, is to start increasing your spend on AWS.Jesse: So, then if you're a small company, maybe a startup or maybe just a student's using AWS for the first time, likely that point of contact within a company is going to be you. From a startup perspective, maybe you are the lead engineer, maybe you are the VP of engineering, maybe you are the sole engineer in the company. We have seen most organizations that we talk to have a relationship with AWS, or build that relationship or own that relationship with AWS at a engineering management or senior leadership level. Engineering management seems to be the sweet spot because usually, senior leadership has a larger view of things on their plate than just AWS so they're focused on larger business moves for the company, but the engineering manager normally has enough context and knowledge of all of the day-to-day specifics of how engineering teams are using AWS to really be involved in that conversation with your account manager, with your technical account manager, or with your solutions architect, or whatever set of folks you have from AWS's side for an account team. And I think that's another thing that we should point out as well, which is, you will always have an account manager; you won't always have a technical account manager.The technical account manager generally comes in once you have signed an enterprise discount program agreement. So, generally speaking, that is one of the perks that comes with an EDP, but obviously, there are other components to the EDP to be mindful of as well.Tim: So, let me clarify that. You get a technical account manager when you sign up for enterprise support. You don't have to have an EDPs to have enterprise support, but when you sign up for enterprise support, you automatically get a technical account manager.Jesse: And, Tim, if you could share with everybody, what kind of things can you expect from a technical account manager?Tim: So, a technical account manager, I mean, they will do—like, all TAMs everywhere pretty much can liaise with support to escalate tickets or investigate them and see what's going on with them, try and, kind of, white-glove them into where they need to be. AWS TAM's, they also have the same—or a lot of the same access to the backend. Not your data because no one at AWS actually has access to your data or inside your systems, but they have access to the backend so they can see API calls, they can see logs, and they can see other things like that to get insight into what's going on in your system and so they can do analytics. They have insight to your billing, they can see your Cost Explorer, they can see what your contract spends are, they can see all the line items in your bills, they have access to the roadmaps, they have access to the services and the service teams so that if you need to talk to someone at a particular service team, they can arrange that meeting for you. If you need to talk to specialists SAs, they can arrange those meetings for you.With a TAM, you—and if you have enterprise support, and they're looking you for an EDP, you can have what's called an EBC or an Executive Briefing Council, where they, in non-pandemic times, they will bring you to Seattle, put you up for a couple of days and you'll have a couple of days of meetings with service teams to go over, kind of like, what the roadmap looks like, what your strategy for working with those teams are or working with those services are. And you can get good steps on how to utilize these services, whether it's going to be some more deep dives on-site, or whether it's going to be some key roadmap items that the service team is going to prioritize and other things like that. And the EBC is actually pretty neat, but you know, you have to be larger spender to get access to those. Another thing that a TAM can do is they can actually enter items on the roadmap for you. They have access to and can provide you access to betas, or pilot programs, or private releases for various services.You'll have access to a weekly email that include what launches are pending, or what releases are pending over the next week or two weeks. You'll have access to quarterly or monthly business reviews where you get access to see what your spend looks like, what your spending trends are, support ticket trends, you know, usage and analytics, and things like that. So, a TAM can be quite useful. They can do quite a lot for you, especially in the realm of cloud economics. That said, every TAM has their specialty.I mean, depending on how many customers they have, the level of engagement you may get. And, you know, some TAMs are super, super, really good at the financial aspects, some are better at the technical aspects. So, to be fair because the TAM org is so large at AWS, you don't always have the same experience with all your TAMs, and the level of depth to which they can dive is going to vary somewhat.Corey: This episode is sponsored in part by ChaosSearch. You could run Elastic Search or Elastic Cloud or Open Search, as they're calling it now, or a self hosted out stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for App performance monitoring, cyber security. If you're using ElasticSearch consider not running ElasticSearch. They're also available now on the AWS market place, if you prefer not to go direct and have half of whatever you pay them count toward your EDP commitment. Discover what companies like, Klarna, Equifax, Armor Security and Blackboard already have. To learn more visit chaossearch.io and tell them I sent you just so you can see them facepalm yet again. Amy: So, let's say we got the best TAM—even though he technically works for us now—when trying to envision what our relationship with the world's best TAM is going to be—and I just imagine that as a nice little block text on a white mug—what is that relationship going to look like? How are we going to engage with them? And even, how often should we talk to them?Jesse: I used to work for an organization that had, I believe, quarterly meetings with our account manager and our TAM, and every time we met with them, it felt like this high stakes poker game where we didn't want to show our cards and they didn't want to show their cards, but then nobody really was able to do anything productive together. And I have to say that is the exact opposite of how to engage your account manager and your TAM.Tim: Yeah, that doesn't sound great.Jesse: No, it was not great. I do not recommend that. You want to have an open, honest conversation about your roadmap, about what you want to do with AWS.Amy: They're not getting that mug.Tim: No, no.Jesse: [laugh].Tim: So, if you have a super-engaged TAM—and I will use my own experience as a TAM at AWS—that we had office hours, routinely, bi-weekly. One customer I had, I would have onsite office hours at their offices in LA, and I would have virtual office hours in offices in London. And those office hours, sometimes I'd have—we—that—we would use those to bring in, whether it was specialist SAs, whether we go over roadmap items, or tickets, or something like that, or we do architectural reviews, or cost reviews, we would schedule quarterly business reviews aside from that, typically sometimes the same day or on the same group of days, but there was typically be different than office hours. I was in their Slack channel so they needed to ping me on something that's not a ticket but a question, we could have conversations in there. A couple of their higher points of contact there had my phone number, so they would call me if something was going on. They would page me—because AWS TAMS have pagers—if they had a major issue, or, like, an outage or something [unintelligible 00:11:05] that would affect them.Jesse: I'm sorry, I just have to ask really quick. Are we talking, like, old school level pager?Tim: No, no, no. Like on your phone, like PagerDuty.Jesse: Okay, okay. I was really excited for a minute there because I kind of miss those old-school pagers.Tim: Let me say, it was like PagerDuty; it wasn't actual PagerDuty because AWS did not actually use PagerDuty. They had something internal, but PagerDuty was the closest analog.Amy: Internal PagerDuty as a Service.Tim: Something like that.Jesse: Oh, no.Amy: So, you know, if you have a very engaged TAM, you would have regular, several times a week, contact if not daily, right? Additionally, the account team will also meet internally to go over strategy, go over issues, and action items, and things like that once or twice a week. Some accounts have multiple TAM, in which case then, you know, the touchpoints are even more often.Jesse: I feel like there's so much opportunity for engagement with your AWS account team, your account manager, your TAM. It's not entirely up to you to build that relationship, but it is a relationship; it definitely requires investment and energy from both sides.Tim: And I would say in the context of who's working with a TAM, ideally, the larger contact paths you have at an org with your TAM, the better off it's going to be. So, you don't want your TAM or account team to only talk to the VP of engineering, or the DevOps manager, or the lead architect; you want them to be able to talk to your devs, and your junior devs, and your finance people, and your CTO, and other folks like that, and pretty much anyone who's a stakeholder because they can have various conversations, and they can bring concerns around. If they're talking about junior devs, your TAM can actually help them how to use CloudFormation, and how to use a AWS CLI, or do a workshop on the basics of using Kubernetes, or something like that. Whereas if you're going to have a conversation with the VP of engineering, they're going to talk about strategies, they're going to talk about roadmap items, they're going to talk about how things can affect the company, they're going to talk about EDPs and things like that. So ideally, in a successful relationship with your TAM, your TAM is going to have several people in your org are going to have that TAM's contact information and will talk with them regularly.Jesse: One of the clients that we worked with actually brought us in for a number of conversations, and brought their TAM in as part of those conversations, too. And I have to say, having the TAM involved in those conversations was fantastic because as much as I love the deep, insightful work that we do, there were certain things about AWS's roadmap that we just don't have visibility into sometimes. And the TAM had that visibility and was able to be part of those conversations on multiple different levels. The TAM was able to communicate to multiple audiences about both roadmap items from a product perspective, from a finance perspective, from an engineering architecture perspective; it was really great to have them involved in the conversation and share insights that were beneficial for multiple parties in that meeting.Tim: And oftentimes, too, involving your TAM when you do have this one thing in your bill you can't figure out, saying, “We've looked and this spend is here, but we don't know exactly why it is.” Your TAM can go back and look at the logs, or go back and look at some of the things that were spun up at the specific time and say, “Oh, here was the problem. It was when you deploy this new AMI, it caused your CPU hours to go way, way up so you had to spin up more instances.” Or a great one was a few years back when Datadog changed its API calls and a lot of people's CloudWatch costs went through the roof. And then several TAMs had to through and figure out, it was this specific call and this is how you fix that and give that guidance back to their customers to reduce their spend. So, being able to have that backend access is very, very useful, even when you are working with an optimization group like ourselves or other folks, to say, “Hey, we've noticed these things. These are the line items we want to get some insight into.” I mean, your TAM can definitely be a good partner in that.Jesse: All right, folks, well, that'll do it for us this week. If you've got questions that you'd like us to answer, please go to lastweekinaws.com/QA. Fill out the form; we'd be happy to answer those on a future show. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review. Give it a five-star rating on your podcast platform of choice and tell us, did Tim pronounce the shortening of ‘Amazon Machine Image' correctly as ‘ah-mi' or should he have said ‘A-M-I?'Amy: I heard it and I wasn't going to say it. [laugh].Jesse: [laugh].Amy: I was just going to wait for someone to send him the t-shirt.Tim: Just to note, if you put beans in your chili, you can keep your comments to yourself.Jesse: [laugh].Amy: You're just going to keep fighting about everything today, is all I'm—[laugh].Jesse: [laugh]. Oh, no.Announcer: This has been a HumblePod production. Stay humble.
Links: https://www.duckbillgroup.com/blog/aws-cost-allocation-guide-tagging-best-practices/ https://www.duckbillgroup.com/blog/aws-cost-allocation-guide-identifying-your-costs/ TranscriptCorey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com. Jesse: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we've seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we're actually going to talk about a very specific listener question that we didn't get to last week, but really, we had so many thoughts on this topic that we wanted to break it out into its own episode. So, today we're going to be talking about tagging, and the importance of tagging, and how tagging can be used. And when I say tagging, specifically we're talking about user-defined cost allocation tags. The original question that I'll read off was from [Aaron 00:00:58].Aaron asks, “Is tagging over-recommended as a cost reporting mechanism? I recently took on managing my company's AWS bill and when talking to AWS and reading third-party blog posts about cost management, a solid tagging strategy is often extolled this step zero for understanding AWS costs. Based on what I know about AWS so far, this approach seems like it may work for some aspects of cost management, but does not seem to be a sound strategy for more formal cost reporting, like budgeting or calculating total spend for a given product or cost center. To me, these activities require complete or near-complete accuracy the tags just don't seem to be able to provide since there are some costs like data transfer that aren't tagged, and the fact that the tags are not retroactive—” that's a big one that I can say is super frustrating for me. “Is there something I'm missing here? Is there in fact, a way to use these tags to ensure that 100% of an AWS account's costs are in fact attributed back to a specific cost center accurately? It seems drastically simpler to embrace a multi-account strategy where each account is simply billed to whatever cost center makes sense to the organization.” So, Amy and Tim, again, the main question here is, is tagging over-recommended as a cost reporting mechanism?Tim: The simple answer is no, it is not over-recommended. And the question makes a lot of good points around some of the heartaches and some the problems that come with tagging, specifically about tags not being retroactive, but, if you're going to make changes to reflect changes in the past, I mean, you know, I don't really have a good answer for that, if we're being honest. But if we're talking about going forward, tracking costs from this point forward, tagging is going to be a much more concise solution than using multi-account strategy. That said, there are a lot of reasons you should use multi-account strategy and tagging together. Multi-account strategy and tagging strategies should definitely be an ‘and' situation, not an ‘or' situation. That's like pizza or steak. No. It's both pizza and steak.And I feel like because there are a number of non-cost reasons to use multiple accounts, especially in AWS, the biggest concern of which are service limits, right? Service limits, as you know, are done by account by region, so, if I have a service limit of S3 buckets that I can create—and I think that the hard limit is, like, one thousand—once I need that one thousandth and one S3 bucket, I have to create another account. That account can still be production, it can still be for all the same things that I've used for anything else, but I had to add another account so I can spin up S3 buckets. So, how do I track those, what those buckets are for, what those costs are going to be? I'm going to track those with tags.And I'm going to track those tags from the payer account, or from up in the organization. So, as you set up multiple accounts, you can have—even if they're all production, they still need to be tagged. Even if they're all dev, they still need to be tagged. If you're using the account vending machine style stuff from Control Tower where you spin up a sandbox account, you run some stuff, and then you throw it away, tagging is going to be the best way to track those costs, not just the fact that this account is named a certain thing. Names are arbitrary; they don't really reflect necessarily what they're going to be for, accounts can come and go.So, I don't necessarily like the use of name. Plus, sometimes it's hard to do that if you're doing, like, [unintelligible 00:04:21] various countries and things like that, various languages. Different things can impart different meanings. Tags also still probably use language problems, but they are arbitrary values. You know you're going to try and lump these all together; that's all that matters.So, I definitely think that, if we're using tagging, tagging is going to let you be more concise with your costs, it's going to let you apply costs across different accounts more readily, it's going to let you apply costs across different cloud providers, especially if you use one of the CMP tools like CloudHealth, or Cloudcheckr, or something like that and you run production workloads from a single cost center across multiple clouds, you're going to want to tag those in those tools, so, that way, you can keep a consistent track and more concise tracking of costs, versus just using account names. Account names after a while is going to just become unmanageable when it comes to tracking costs.Amy: I totally agree. And one of the big things that I harp on, especially on this podcast, is that if you're worried that it's not going to be as explicit as other billing methods, you will still at least have that data. You will still know per resource—if it's properly tagged—who it's supposed to be charged to and who owns it. You would make that decision on an architectural level, you should also make it for your bill, just to make sure that if you ever need that information in the future, you can go get it. You're not going to get it—since they don't happen retroactively, then you may as well do it as early as possible.Jesse: Yeah. It's super frustrating that a lot of this information is not available retroactively. And while I understand the technical limitations to that, I can't harp enough why starting to tag resources early is super, super critical to understanding that spend, and using that tagging setup, that tagging policy, to better understand your spend in a number of different ways. But again, I also want to call out that, like, I've been saying everything about tagging related to spend, there are other ways that tags can be beneficial to your organization. I've seen organizations where security needs to know, are all of the containers that were running patched to a certain level?Are all of the AMIs that we're running patched to a certain level? Tags can do that; tags can help you understand what resources are using a certain AMI version, or a certain container version, or other security pieces that are important for security to know and be able to understand that all of these resources are patched to the latest available version of whatever we're looking at. One of the things that we talk about a lot in this podcast is having conversations with other teams because I feel like cloud cost management is not just an engineering responsibility. It's a responsibility of finance, and product, and security, and IT because there's all sorts of different groups that may ultimately be using the cloud. And that's kind of important for everybody to be on the same page in terms of how you're using the cloud. And so it's not just about tagging so, you can know the cost of something, but tagging so that you can know all these other important things like security, like product details, like maybe IT details, all these other different use cases for different departments that are also involved in cloud usage.Corey: This episode is sponsored in part by ChaosSearch. You could run Elastic Search or Elastic Cloud or Open Search, as they're calling it now, or a self hosted out stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for App performance monitoring, cyber security. If you're using ElasticSearch consider not running ElasticSearch. They're also available now on the AWS market place, if you prefer not to go direct and have half of whatever you pay them count toward your EDP commitment. Discover what companies like Hubspot, Clarna, Equifax, Armor Security and Blackboard already have. To learn more visit chaossearch.io and tell them I sent you just so you can see them facepalm yet again.Tim: Yeah. I think there's this idea that comes, I think, from very legacy data center operations where you're going to use an account name to, kind of, specify what it does and where it comes from in the same way that you would use, like, a host naming scheme to define what a computer is and what it does and things like that. And I think that can be practical, but sometimes it's often short-sighted, especially as an organization grows, and you create more accounts, and you bubble up other accounts [unintelligible 00:08:21] accounts. It comes time to sign the EDP and you need to have a master payer account, you acquire some other accounts and things like that, and then all of a sudden, whatever naming schemes they used is now integrated into what your naming scheme is. And that becomes, maybe, unmanageable.So, I've always preferred to have account names. I mean, if you need to have it specified, understand it's going to just be for humans to, really quick, find it, but I'm just as content to have an account name be a UUID and then have some other kind of method for looking at what it does or assigning billing to it. Because in the end, like I said, I prefer to use tagged resources to define what they are and where they go. They are obviously going to be exceptions made for things that are, like, dev, test, UAT, or something like that, where [unintelligible 00:09:06] are different, but we're still talking about changes on an account, and then you make the changes on the account as you need. And then if it's for production, then obviously those accounts can be tagged as production. They don't have to necessarily be named production.Amy: Right, and I think, security boundaries and resource permissions aside, if you're just looking at trying to track costs to a resource, an account ID is really just one piece of information as opposed to tags, where you can just overload it with as much information as you need.Jesse: Absolutely. Now, one other thing that I do want to talk about is we're talking about a lot of good use cases for tags. We should also talk about some of the not-so-good use cases for tags, or some of the not-so-great best practices for tags that we have seen. Amy, I know specifically you had some examples that you want to talk about.Amy: Yes. [laugh]. So, this comes from having to do data normalization, back in the day. First thing you never want to do when developing your tag strategy is you want it to just determine things like casing, or whether or not you're allowed to use spaces because I've seen in different places, not just on resource tagging, but also the way information is meta-keyed, where they have their key name identical to a completely different key name, like you have ‘product owner' except ‘product owner' is capitalized in one instance and not capitalized in another instance, and these are considered to be different things within the system. Whether or not that's your intention, they will show up as different things on some visualizations. On other visualizations, they will get normalized and turned into the same thing. So, it really depends on what it is that you want your reports to look like and what you want these resources to be able to tell you.Jesse: Yeah, that's a really great point that one of the things that we haven't potentially touched on for this episode, and is covered in a number of other podcast episodes and blog posts in general is a good tagging strategy is equally as important as tagging coverage. Knowing that the tags should all be uppercase or all lowercase, or use these types of characters and not those types of characters is equally as important as making sure that those tags are applied accurately across all of your resources. So, as you are talking about tagging, as you are thinking about tagging, even in the multi-account situation, it is important to think about, what are the best practices? What are the standards that you want for your tagging? And again, this may not be a conversation that you have in a silo by yourself; this may be a conversation that you have with a number of other teams because there may be a number of other teams that need certain information from tags and need to use certain letters or special characters. And you need to incorporate all of that; you need to include all of that in the tagging policy that you create.Tim: I think it's also important, though too, that with most analytics tools, even if it's just, you know, Cost Explorer within AWS Console, you can still aggregate those tags together, especially if you're doing costs, you can absolutely aggregate multiple cases and things like that. CloudHealth, I know you can select multiples or anything that matches a pattern regardless of case and do it that way. So, it is possible to work around those mistakes. It's not a, “Oh, we didn't have our tagging schema set up correctly, so, throw your hands up and give up.” It's just something else you have to consider, and hopefully, you can normalize going forward.Jesse: Yeah, absolutely.Amy: And really, the other thing is to make sure that the tags that you choose makes sense for what you're doing. So, if you are tagging the environment and that is the only tag that you put on a resource, then just know that when you start pulling things up in Cost Explorer or Cost and Usage Report, that's the only thing you're going to see. So, you're only going to see things split up between your production account and your dev account; you're not going to be able to see what service is actually costing you more money, or what storage, as associated to a team, has suddenly decided to grow beyond the usual predictive usage patterns.Jesse: Yeah, we have some recommendations we can make if you are just getting started on your tagging journey, and I will make sure that information is shared in the [show notes 00:13:53]. But ultimately, again, it becomes a strategy conversation. It becomes a question of what are you trying to accomplish? What are the goals that you're trying to accomplish? What is the information that you want out of tagging? Because that's ultimately going to drive what you tag and why you tag.All right, that'll do it for us this week, folks. If you've got questions you'd like us to answer, please go to lastweekinaws.com/QA, fill out the form and we'd be happy to answer your question on a future episode. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review. Give it a five-star rating on your podcast platform of choice and tell us what are the most important things that you focus on in your tagging strategies? What are the things that you tag for your company?Announcer: This has been a HumblePod production. Stay humble.
TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways that we've seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today on the show, we are going to be talking about AWS re:Invent. Now, I know that most of you know what re:Invent is, but I just would love to set the playing field level for everybody really quick. Amy, Tim, what is AWS re:Invent.Tim: AWS re:Invent is AWS's week-long corporate conference. It's not really a user conference; it's certainly not, like, a community conference, but it's a week-long sales pitch in the desert. It's like the worst version of a corporate Burning Man you could ever imagine because they even have a concert.Jesse: It is in Las Vegas. Now, I personally have mixed feelings about going to Las Vegas in general, but this adds so much to the conference in general because it's not just in a single conference venue that's centrally located near the hotels. Is it is across the strip—Amy: It's the entire strip.Jesse: It's the entire strip. So—Amy: They block every hotel and they buy every piece of ad space.Jesse: Yes. There is no escaping AWS re:Invent for the entire week that you're there. And sometimes that's a good thing because you do want to be involved in what's going on, but other times, it is a lot.Tim: So, I'm trying to figure out which LP that ‘buy the entire Las Vegas trip' covers because it's certainly not be frugal.Amy: No. [laugh].Jesse: No, not at all. But we do have new information. We decided to do this episode specifically because new information was just released about re:Invent for this year. Amy, what is that information? What do we know?Amy: They've decided, in having to go virtual last year, due to some kind of horrible global crisis, to return in person to the world's most densely packed tourist spot, Las Vegas, and host this huge event from November 29th to December 3rd—that's right after Thanksgiving—and just, what do they say? Return to normal. Return to normal.Tim: That way everybody can get exposed to COVID before they go home for the holidays.Jesse: [laugh].well, you at least get one holiday in, if you celebrate or recognize Thanksgiving, and then you get to bring everything back after that.Amy: Yeah, people bring enough things back from Vegas. I'm not sure we'd have to find more reasons. [laugh].Tim: [laugh].Jesse: I know that there's that great marketing tactic of, “What happens in Vegas stays in Vegas,” but—Tim: That's not what they say at the clinic.Jesse: Nope. Mm-mm. Now, I will say, I know that almost every conference event was completely virtual last year due to the pandemic, and this year, a lot of conferences are still trying to straddle that line between what's acceptable, can we do maybe smaller events in person, some kind of a hybrid online/in-person thing. I have mixed feelings on this. I appreciate that I can still attend AWS re:Invent from home this year digitally, I can still watch a lot of the main keynote events and a lot of the other information that is being shared, but I don't know, it's always hard because if you do a hybrid event, you're automatically going to miss out on any of that in-person socializing and networking.Tim: Well. So, I think it's interesting. AWS re:Invent suffers from the same issue that pretty much all other conferences suffer from is that there's not really value-add in the talks, at least for attending.Jesse: Yeah.Amy: If you're going to be able to see those talks afterwards if the announcements are going to be publicized afterwards which, that is true in both cases, then what's the point of spending the money, and the time, and the possible exposure to go watch them in person? So, then the other thing is, “Well, we want to go for some of the training seminars,” or some of these other things. Well, those are also offered online, often. Or, like, copies of them online. These are the same kinds of tutorials like that that you can have your TAM or SA run if you're an AWS customer currently; that's what they're doing there.The other thing is, too, those in-person sessions get filled up so quickly that there's no guarantee [unintelligible 00:05:08] anyways. And that's one of the complaints they've had about re:Invent in the past is that you can't get into any of the sessions. And so, you couple all that along with most of the reason going being—if it's not the talks and is not the sessions, it's the hallway track. And then you got to kind of wonder, is the hallway track going to be valuable this year because if it's hybrid, what percent of the people that you would normally talk to you are going to be there and what percentage aren't? And so there's a lot of calculus that's got to go into it this year.Jesse: I've always struggled with any vendor-sponsored event, all the talks feel either like a sales pitch, or they feel like a use case that just doesn't fit for me. And that may just be where I'm at in my professional journey; there's definitely reasons to go if you want to see some of these talks or see some of this information live, or be the first person to talk about it. Or even the people who are going to be the news sources for everybody else who want to be the first person to talk about, “Oh, we attended, and we saw these things and were live-tweeting the entire conference.” If that's your shtick, I fully support that, but I always struggle going to any kind of vendor conference because I just feel like the value that I get from the talks, from training if I go to training, just doesn't feel like enough for me, personally.Amy: So, I've done some of the AWS-led training when Summit was in Chicago, a couple years ago, and I'll be honest, you lose a lot in these large AWS-led trainings because these classes, it's not going to be like the ones that you would sign up for even being hosted either by your company or by your local user group chapter where you will have at max 100 people. You have well over that. You have an entire conference room full of people, and they're asking questions that are across the level of expertise for that topic. I went for one of the certification training seminars and straight-up 15 minutes was spent talking about what a region is. And given that's page one of any training material, that was a waste of $300.Jesse: Yeah.Tim: I think you run into the problem because it is, in fact, I mean, let's be honest, it's a multi-day sales pitch. It's not a user conference, it's not user-generated content. It's cherry-picked by the powers-that-be at AWS, the service groups. Is a big push for account executives to encourage high-level or high-spend accounts to participate in those so they get logo recognition. And so that becomes more of the issue than the actual cool user stories.And that's fine if you're using it literally just a sales conference because it's very compelling sales material, your account executive will go there and try and close deals, or close bigger deals, or sign EDPs or something like that, but from an engineering standpoint, from a technical standpoint, it's remarkably uncompelling.Jesse: Yeah, I think that's one other thing to call out, which is, there is definitely this networking opportunity that we talked about from a hallway track perspective, but there's also a networking and business opportunity to meet with your account manager, or your TAM, or your SA in person and have conversations about whatever things you want to talk about; about future architecture, or about closing an EDP—or I should say, about an EDP because the account manager will try to close that EDP with you—and then basically use that as next steps for what you want to do with AWS. But again, all of those things can be done without flying you to Las Vegas and being amongst all these other people.Tim: I mean, let's not take away, there's a certain synergy that happens when you have face-to-face contact with folks, and a lot of these conversations you have in hallways are super, super organic. And so I think that's indicative of conferences as a whole. One of the things that we learned in the pandemic is that, yeah, you can have talks where people just, like, look at a screen and watch talks, and a lot of conferences have done that. But that's not why people want to go to the conference; they want to go to the conference to talk to people and see people. And if you want to have a conference where people talk to people and see people, and that's the whole point of doing it, then the business model behind that looks dramatically different, and the content behind that looks dramatically different.You just have a bunch of birds-of-feather sessions or a bunch of breakout sessions. You do a keynote at the beginning, you do a keynote at the end, and then you just let people mingle, and maybe you have some led topics, but you don't generate content; you shut up and you let the people innovate.Jesse: I also want to add to that. It is one thing to have a conference that is in one venue where everybody is going to be gathered in the same space, creating conversation, or creating easy opportunities—Amy: Five miles worth of content isn't exciting for you?Jesse: Yeah. So, in Las Vegas because the entire conference is spread across the entire strip, you're going to have opportunities to network across the entire strip basically, and sometimes that means you're going to only spend time networking with the people who are in the same hotel as you at the time of the track that you are waiting for, or the time of the event that you are waiting for. It is unlikely that you are going to run all around the strip just to be able to network with everybody that you run into.Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself; make one of them go away. To learn more visit lumigo.io.Amy: The other issue I have, not just with re:Invent, but this is really any larger conference or conferences that rely on the kind of content where it is a person speaking at you and you don't get to meet these people, is that without any level of Q&A or interactivity—and this is true especially for AWS-led events—is that it is no different than watching someone on video. You can go to these talks, and you can perhaps have conversations with people as they filter out of the room, but there's no way you're going to be able to talk to that person who was delivering that content, unless you can track them down amongst the sea of people in re:Invent or [unintelligible 00:12:16] in Las Vegas.Tim: What typically has to happen is that after someone has given a compelling talk and you really want to talk to them, you have to go and talk to your account manager; your account manager will then set up a meeting that will happen at a later time where you're going to all call in over Chime, and then you will quote-unquote, “Meet” that person virtually. And if that's the case, you could have just stayed home and watched [laugh] the talk online, and then done the same thing.Amy: Conferences need more Chime. That's what [laugh] the problem is.Jesse: [laugh]. I think my eye just started twitching a little bit as soon as you said that, Amy.Amy: I'm glad. So, then why would people go? There's the hallway track, but is that worth the heavy price tag of going to Vegas? A lot of us live in areas where there is either going to be an AWS Summit or there are AWS user groups. What do you get from going to a larger event such as re:Invent and having that level of communication that you can't get from those smaller groups?Tim: I mean, the importance of networking cannot be overstated. It is extremely important, whether it's for laying groundwork for future deals, laying groundwork for future collaborations. I've been at conferences where a hallway track, just folks meeting up in the hallway and having a really organic discussion turned into a product within three months. So, those kinds of things are important. And, unfortunately or unfortunately, they do happen better quite often, when people are in-person and they've had a chance to talk, maybe even a couple of drinks or whatever.So, I mean, people ink deals, they shake hands, they get, you know, a lot of work done when it comes to maintaining and managing relationships, and to some people, that is worth it. But I do think that you have to be very, kind of, eyes-open about going into this. It's like, you're not going to go in there to get a lot of technical insight, you're not going to go in there to talk to a whole bunch of people unless you really have a relationship or establish some kind of rapport with them beforehand. Because just to go up and blindly like, “Hey, I'm going to grab you in the hallway, and this is who I am,” that's not always great, especially nowadays, when people are, kind of, already averse to, you know, talking to strangers, sometimes.Jesse: I've always struggled with talking to strangers in general at conferences because I'm predominantly introverted, so if I don't have an open introduction to someone through a mutual third party or mutual friend, it's just not going to happen. And I've gotten better at that over the years as I go to conferences, but it's going to be especially tough now in cases where folks are not just averse to, I don't want to say strangers, but averse to physical contact and adverse to people just, kind of, approaching them out of the blue. It's tough. I want to be more mindful of that and I want to be better, but it's hard, especially in cases where you're in a crowd of hundreds of people or, you know, thousands of people across the strip, that it just gets overwhelming really quickly for some folks.Amy: I do want to loop this round, if anything, just for a poll for Twitter. Do not close an EDP in Vegas. You're probably not of the right mind [laugh] and have the right people to do that. Wait till you get back to work. Please. That's just me. [laugh].Jesse: I would also like to add—we talk about why people go; I think that there's definitely a solid contingent of folks who attend re:Invent because it is the one time a year that the company sanctions them getting away from their family for a couple of days, getting away from, you know, the day-to-day routine of whatever work is going on for a couple days, and go to Vegas. Now, I know that the company is not going to sponsor them drinking every night, or gambling, or whatnot, but they're likely going to be doing those things anyhow, so it is this company-sanctioned opportunity to just go experience, you know, something different; go take a vacation, basically, for a couple days.Amy: Corporate Burning Man.Tim: Corporate Burning Man, exactly. A vacation in Vegas.Amy: I am not a fan of ever working in Vegas. If I'm on the clock, I cannot be in Vegas, not because I'm prone to excessive behavior when I'm on my own, but more that I cannot be productive in that much noise and that much flaky internet. It drives me absolutely batty, and I'm only going to be, as far as implementations, so productive in a crowd that large.Tim: I will say this, especially in regards to Vegas, there are other places you can go, other places that need the money more. AWS wants to rent a city, rent a city that needed the money. Put that money where it could be to used, where it really makes a difference. I don't know if Vegas is the right place for that, if I'm being honest, especially after all we've learned and dealt with in 2020. And so that's why in 2021, yeah, no for me, continuing to have re:Invent in Vegas is very, very tone-deaf.Jesse: I still think, Amy, you and I just need to—actually sorry, all three of us should attend and basically keep a running Waldorf and Statler commentary through the entire conference. I don't know if we can get that little, you know, opera booth that's kind of up and away from all the action, but if we can get something like that and do some sports commentary—ohh, maybe on the expo hall—Amy: That would be great. That would be great if we don't get banned. [laugh].Tim: I think what would be even more fun is to give a MST3K—Jesse: Ohhh.Tim: —treatment of the keynotes afterwards, you know what I mean?Jesse: Yeah.Amy: Yes.Jesse: I mean, Amy and I had also talked about playing some Dungeons and Dragons while we were there, and I feel like if we can find some, I'm going to say, tech-themed RPG—I realize that is a broad category, and everybody's going to spam me afterwards for this, but—Amy: I got that. Don't worry about it.Jesse: Yeah, I'm on board. I feel like anything that we can do to create a roleplaying game out of this conference, I'm down.Tim: I'm still waiting for you to explain to the audience in general who Waldorf and Statler were?Jesse: Oh, yes, that is fair. Okay. Waldorf and Statler are two characters from the old-school Muppets Show, which is amazing and delightful. It's on Disney+; I highly recommend it. They are basically—Amy: They're two grumpy old muppets, and they have been roasting people since the 70s. That is—that's all it is. [laugh].Tim: All they do is they sit up in the upper booth and they throw shade, and I love it.Amy: Yes. And they just show up in random parts in different movies. They'll be, like, on a park bench, and there'll be a serious moment, and then they'll just start talking crap for no reason. And it's great.Jesse: They're the best. They're absolutely fantastic. I adore them. I hope to be them one day.Amy: One day.Tim: Really, both of them? I don't, I don't know how that's going to work.Jesse: I am hoping to clone myself. One of me is going to have fabulous hair and one of me is going to be balding. Probably the clone is going to be balding; sorry about it, future me. But—Amy: [laugh].Tim: Well, I mean, and have just a magnificent chin, right?Jesse: Yes, yes, that's the trade-off. Losing the hair up top but absolutely fantastic chin.Tim: Here's what I want to see. I want to see the listeners submit things that you think should be on the re:Invent bingo cards.Amy: Ohh, yes.Jesse: Yes.Amy: I would love to see that.Jesse: So, for those of you listening, you've got two options for submitting things that you'd like to be on the re:Invent bingo cards. The ideal option is going to lastweekataws.com/QA. Fill out the form and let us know what you think should be on the bingo card. You can also respond to the social media post that will be posted for this content, and we can take a look at that as well. But that'll be a little bit harder for us to follow because I'm unfortunately not like Corey. I can't absorb all of Twitter in a day; it takes me a longer time to read all that content.Jesse: If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review. Give it a five-star rating on your podcast platform of choice and tell us what you think about AWS re:Invent.Announcer: This has been a HumblePod production. Stay humble.
TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Tim: And I'm Tim Banks.Jesse: This is the podcast within a podcast where we talk about all the ways we've seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today is a very special episode for two reasons. First, we're going to be talking about all the things that you want to talk about. That's right, it's time for another Q&A session. Get hyped.Amy: And second as is Duckbill's customary hazing ritual, we're putting a new Duckbill Group Cloud Economist Tim Banks through the wringer to answer some of your pressing questions about cloud costs and AWS. And he has pretty much the best hobbies.Tim: [laugh].Jesse: Absolutely.Tim: You know, I choke people for fun.Jesse: [laugh]. I don't even know where to begin with that. I—you know—Amy: It's the best LinkedIn bio, that's [laugh] where you begin with that.Tim: Yeah, I will change it right after this, I promise. But no, I think it's funny, we were talking about Jiu-Jitsu as a hobby, but my other hobby is I like to cook a lot, and I'm an avid, avid chili purist. And we were in a meeting earlier and Amy mentioned something about a bowl of sweet chili. And, dear listeners, let me tell you, I was aghast.Amy: It's more of a sweet stewed meat than it is, like, some kind of, like, meat candy. It is not a meat candy. Filipinos make very sweet stews because we cannot handle chili, and honestly, we shouldn't be able to handle anything that's caramelized or has sugar in it, but we try to anyway. [laugh].Tim: But this sounds interesting, but I don't know that I would categorize it as chili, especially if it has beans in it.Jesse: It has beans. We put beans in everything.Tim: Oh, then it can't be chili.Jesse: Are you a purist that your chili cannot have beans in it?Tim: Well, no. Chili doesn't have beans in it.Amy: Filipino food has beans in it. Our desserts have beans in it. [laugh].Jesse: We are going to pivot, we're going to hard pivot this episode to just talk about the basis of what a chili recipe consists of. Sorry, listeners, no cost discussions today.Tim: Well, I mean, it's a short list: a chili contains meat and it contains heat.Jesse: [laugh].Tim: That's it. No tomatoes, no beans, no corn, or spaghetti, or whatever people put in it.Amy: Okay, obviously the solution is that we do some kind of cook-off where Tim and Pete cook for everybody, and we pull in Pete as a special quote-unquote, outside consultant, and I just eat a lot of food, and I'm cool with that. [laugh].Jesse: I agree to this.Tim: Pete is afraid of me, so I'm pretty sure he's going to pick my chili.Jesse: [laugh].Amy: I could see him doing that. But also, I just like eating food.Tim: No, no, it's great. We should definitely do a chili cook-off. But yeah, I am willing to entertain any questions about, you know, chili, and I'm willing to defend my stance with facts and the truth. So…Amy: If you have some meat—or [sheet 00:03:19]—related questions, please get into our DMs on Twitter.Jesse: [laugh]. All right. Well, thank you to everyone who submitted their listener questions. We've picked a few that we would like to talk about here today. I will kick us off with the first question.This first question says, “Long-time listener first-time caller. As a solo developer, I'm really interested in using some of AWS's services. Recently, I came across AWS's Copilot, and it looks like a potentially great solution for deployment of a basic architecture for a SaaS-type product that I'm developing. I'm concerned that messing around with Copilot might lead to an accidental large bill that I can't afford as a solo dev. So, I was wondering, do you have a particular [bizing 00:04:04] availability approach when dealing with a new AWS service, ideally, specific steps or places to start with tracking billing? And then specifically for Copilot, how could I set it up so it can trip off billing alarms if my setup goes over a certain threshold? Is there a way to keep track of cost from the beginning?”Tim: AWS has some basic billing alerts in there. They are always going to be kind of reactive.Jesse: Yes.Amy: They can detect some trends, but as a solo developer, what you're going to get is notification that the previous day's spending was pretty high. And then you'll be able to trend it out over that way. As far as asking if there's a proactive way to predict what the cost of your particular architecture is going to be, the easy answer is going to be no. Not one that's not going to be cost-prohibitive to purchase a sole developer.Jesse: Yeah, I definitely recommend setting up those reactive billing alerts. They're not going to solve all of your use cases here, but they're definitely better than nothing. And the one that I definitely am thinking of that I would recommend turning on is the Cost Explorer Cost Anomaly Detector because that actually looks at your spend based on a specific service, a specific AWS cost category, a specific user-defined cost allocation tag. And it'll tell you if there is a spike in spend. Now, if your spend is just continuing to grow steadily, Cost Anomaly Detector isn't going to give you all the information you want.It's only going to look for those anomalous spikes where all of a sudden, you turned something on that you meant to turn off, and left it on. But it's still something that's going to start giving you some feedback and information over time that may help you keep an eye on your billing usage and your spend.Amy: Another thing we highly recommend is to have a thorough tagging strategy, especially if you're using a service to deploy resources. Because you want to make sure that all of your resources, you know what they do and you know who they get charged to. And Copilot does allow you to do resource tagging within it, and then from there should be able to convert them to cost allocation tags so you can see them in your console.Jesse: Awesome. Well, our next question is from Rob. Rob asks, “How do I stay HIPAA compliant, but keep my savings down? Do I really need VPC Flow Logs on? Could we talk in general about the security options in AWS and their cost impact? My security team wants everything on but it would cost us ten times our actual AWS bill.”Rob, we have actually seen this from a number of clients. It is a tough conversation to have because the person in charge of the bill wants to make sure that spend is down, but security may need certain security measures in place, product may need certain measures in place for service level agreements or service level objectives, and there's absolutely a need to find that balance between cost optimization and all of these compliance needs.Tim: Yeah, I think it's also really important to thoroughly understand what the compliance requirements are. Fairly certain for HIPAA that you may not have to have VPC Flow Logs specifically enabled. The language is something like, ‘logging of visitors to the site' or something like that. So, you need to be very clear and concise about what you actually need, and remember, for compliance, typically it's just a box check. It's not going to be a how much or what percent; it's going to be, “Do you have this or do you not?”And so if the HIPAA compliance changes where you absolutely have to have VPC Flow Logging turned on, then there's not going to be a way around that in order to maintain your compliance. But if the language is not specifically requiring that, then you don't have to, and that's going to become something you have to square with your security team. There are ways to do those kinds of logging on other things depending on what your application stack looks like, but that's definitely a conversation you're going to want to have, either with your security team, with your product architects, or maybe even outside or third-party consultant.Jesse: Another thing to think about here is, how much is each of these features in AWS costing you? How much are these security regulations, the SLA architecture choices, how much are each of those things costing you in AWS? Because that is ultimately part of the conversation, too. You can go back to security, or product, or whoever and say, “I understand that this is a business requirement. This is how much it's costing the business.”And that doesn't mean that they have to change it, but that is now additional information that everybody has to collaboratively decide, “Okay, is it worthwhile for us to have this restriction, have this compliance component at this cost?” And again, as Tim was mentioning, if it is something that needs to be set up for compliance purposes, for audit purposes, then there's not really a lot you can do. It's kind of a, I don't want to say sunk cost, but it is a cost that you need to understand that is required for that feature. But if it's not something that is required for audit purposes, if it's not something that just needs to be, like, a checkbox, maybe there's an opportunity here if the cost is so high that you can change the feature in a way that brings the cost down a little bit but still gives security, or product, or whoever else the reassurances that they need.Tim: I think the other very important thing to remember is that you are not required to run your application in AWS.Jesse: Yeah.Tim: You can run it on-premise, you can run at a different cloud provider. If it's going to be cost-prohibitive to run at AWS and you can't get the cost down to a manageable level, through, kind of, normal cost reduction methods of EDPs, or your pricing agreement, remember you can always put that on bare metal somewhere and then you will be able to have the logging for free. Now, mind you, you're going to have to spend money elsewhere to get that done, but you're going to have to look and see what the overall cost is going to be. It may, in fact, be much less expensive to host that on metal, or at a different provider than it would be at AWS.Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn't translate well to cloud or multi-cloud environments, and that's not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.Jesse: Our next question is from Trevor Shaffer. He says, “Loving these Friday from the field episodes and the costing”—thank you—“I'm in that world right now, so all of this hits home for me. One topic not covered with the cost categorization, which I'm tasked with, is how to separate base costs versus usage costs. Case in point, we're driving towards cost metrics based on users and prices go up as users go up. All of that makes sense, but there's always that base load required to serve quote-unquote, ‘no users.'“The ALP instance hours, versus the LCU hour, minimum number of EC2 instances for high availability, things like that. Currently, you can't tag those differently, so I think I'm just doomed here and my hopes will be dashed. For us, our base costs are about 25% of our bill. Looking for tricks on how to do this one well. You can get close with a lot of scripting and time, teasing out each item manually.” Trevor, you can, and I also think that is definitely going to be a pain point if you start scripting some of these things. That sounds like a lot of effort that may give you some useful information, but I don't know if it's going to give you all of the information that you want.Tim: Well, it's also a lot of effort, and it's also room for error. It won't take but a simple error in anything that you write where these costs can then be calculated incorrectly. So, that's something to consider as well: is it worth the overall costs of engineering time, and maintenance, and everything like that, to write these scripts? These are decisions that engineers groups have to make all the time. That said, I do think that this is, for me I think, one of the larger problems that you see with AWS billing is that it is difficult to differentiate something that should be reasonably difficult to differentiate.If I get my cell phone bill, I know exactly how much it's going to cost us to have the line, and then I can see exactly how much it's going to cost me for the minutes. The usage cost is very easily separated from—I'm sorry, the base cost is very easily separated from the usage cost. It's not always that way with AWS, I do think that's something that they could fix.Jesse: Yeah, one thing that I've been thinking of is, I don't want to just recommend turning things on and measuring, but I'm thinking about this from the same perspective that you would think about getting a baseline for any kind of monitoring service: as you turn on a metric or as you start introducing a new metric before you start building alerts for that metric, you need to let that metric run for a certain amount of time to see what the baseline number, usage amount, whatever, looks like before you can start setting alerts. I'm thinking about that same thing here. I know that's a tougher thing to do when this is actually cost involved when it's actually costing you money to leave something on and just watch what usage looks like over time, but that is something that will give you the closest idea of what base costs look like. And one of the things to think about, again, is if the base costs are unwieldy for you or not worthwhile for you in terms of the way the architecture is built, is there either a different way that you can build the architecture that is maybe more ephemeral that will make it cost less when there are no users active? Is there a different cloud provider that you can deploy these resources to that is going to ultimately cost you less when you have no users active?Tim: I think too, though, that when you have these discussions with engineering teams and they're looking at what their priorities are going to be and what the engineering cost is going to be, oftentimes, they're going to want metrics on how much is this costing us—how much would it cost otherwise? What is our base cost, what's our usage cost?—so that you can make a case and justify it with numbers. So, you may think that it is better to run this somewhere else or to re-architect your infrastructure around this, but you're going to have to have some data to back it up. And if this is what you need to gather that data, then yeah, it is definitely a pain point.Amy: I agree. I think this is one of those cases where—and I am also loath to just leave things on for the sake of it, but especially as you onboard new architectures and new applications, this should be done at that stage when you start standing things up and finalizing that architecture. Once you know the kind of architecture you want and you're pushing things to production, find out what that baseline is, have it be part of that process, and have it be a cost of that process. And finally, “As someone new to AWS and wanting to become a software DevOps insert-buzzword-here engineer”—I'm a buzzword engineer—“We've been creating projects in Amplify, Elastic Beanstalk, and other services. I keep the good ones alive and have done a pretty good job of killing things off when I don't need it. What are your thoughts on free managed services in general when it comes to cost transparencies with less than five months left on my free year? Is it a bad idea to use them as someone who is just job hunting? I'm willing to spend a little per month, but don't want to be here with a giant bill.”So, chances are if you're learning a new technology or a new service, unless you run into that pitfall where you're going to get a big bill as a surprise and you've been pretty diligent about turning your services off, your bill is not going to rise that much higher. That said, there have been a lot of instances, on Twitter especially, popping up where they are getting very large bills. If you're not using them and you're not actively learning on them, I would just turn them off so you don't forget later. We've also talked about this in our build versus buy, where that is the good thing about having as a managed service is if you don't need it anymore and you're not learning or using them, you can just turn them off. And if you have less than half a year on your first free year, there are plenty of services that have a relatively free tier or a really cheap tier at the start, so if you want to go back and learn on them later, you still could.Tim: I think too, Amy, it's also important to reflect, at least for this person, that if they're in an environment where they're trying to learn something if maintaining infrastructure is not the main core of what they're trying to learn, then I wouldn't do it. The reason that they have these managed services is to allow engineering teams to be more focused on the things that they want to do as far as development versus the things they have to do around infrastructure management. If you don't have an operations team or an infrastructure team, then maintaining the infrastructure on your own sometimes can become unwieldy to the point that you're not really even learning the thing you wanted to learn; now you're learning how to manage Elasticsearch.Amy: Yeah.Jesse: Absolutely. I think that's one of the most critical things to think about here. These managed services give you the opportunity to use all these services without managing the infrastructure overhead. And to me, there may be a little bit extra costs involved for that, but to me that cost is worth the freedom to not worry about managing the infrastructure, to be able to just spin up a cluster of something and play with it. And then when you're done, obviously, make sure you turn it off, but you don't have to worry about the infrastructure unless you're specifically going to be looking for work where you do need to manage that infrastructure, and that's a separate question entirely.Amy: Yeah. I'm not an infrastructure engineer, so anytime I'm not using infrastructure, and I'm not using a service, I just—I make sure everything's turned off. Deleting stacks is very cathartic for me, just letting everything—just watching it all float away into the sunset does a lot for me, just knowing that it's not one more thing I'm going to have to watch over because it's not a thing I like doing or want to do. So yeah, if that's not what you want to do, then don't leave them on and just clean up after yourself, I suppose. [laugh].Tim: I'll even say that even if you're an infrastructure engineer, which is my background, that you can test your automation of building and all this, you know, building a cluster, deploying things like that, and then tear it down and get rid of it. You don't have to leave it up forever. If you're load testing an application, that's a whole different thing, but that's probably not what you're doing if you're concerned about the free tier costs. So yeah, if you're learning Terraform, you can absolutely deploy a cluster or something and just tear it back out as soon as you're done. If you're learning how to manage whatever it is, build it, test it, make sure it runs, and then tear it back down.Jesse: All righty, folks, that's going to do it for us this week. If you've got questions you would like us to answer, please go to lastweekinaws.com/QA, fill out the form and we'd be happy to answer those on a future episode of the show. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us whether you prefer sweet chili or spicy chili.Announcer: This has been a HumblePod production. Stay humble.
TranscriptCorey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Jesse: This is the podcast within a podcast where we talk about all the ways that we have seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. I feel like it's just kind of always necessary. There always has to be just that little bit of something extra; it's the spice that really makes the dish. Today we're going to be talking about the ‘Cloud Cost Management Team Starter Kit.' Now, in a previous episode, we talked about the ‘Cloud Cost Management Starter Kit,' which was a little bit more generalized, and one of the things that we talked about, ultimately, was building a team that is responsible for some of this work, some of this cloud cost management work.So today, we're going to take that one step further; we're going to talk about all of the things that your cloud cost management team should ultimately be responsible for, what it should look like, how you might want to start building that team within your organization. So, I'm going to kick us off. I think one of the first things that is so, so critical for any team that is going to be doing any work is buy-in at the executive leadership level. You need to make sure that engineering leadership, the C-suite leadership has your back in everything that you're doing. You need to make sure that the work that you're doing has been signed off at the highest level so that that leadership can help empower you to do your work.Amy: And we've referenced this before, and really, every time we talk about things like what makes a successful project is that as the one executing that project, you probably need the authority and actionable goals in order to do that, and the leadership is going to be the ones to lay that out for you.Jesse: Absolutely. If you don't have the backing of leadership, whether it is your boss, whether it is the C-suite, whether it's a VP suite, you're not going to get other people to listen to what you have to say; you're not going to get other people to, broadly speaking, generally speaking, care about the work that you're trying to do, the work that you're trying to incentivize and empower other people in the organization to do.Amy: And that kind of leads us into the next portion of it where you need to know what the responsibilities are and have that clear delineation so that you understand the things that is expected of you, what the engineering teams, what they're expected to do, and product teams, and finance teams. Everyone has to have a pretty much fenced-in idea of what they're allowed to do and what they are expected to deliver, just like in any project.Jesse: Absolutely. It's so critical for me to understand what I'm responsible for, you to understand what you're responsible for. I can't tell you how many times I've been in a meeting where somebody will say something generally like, “We should do X,” and then everyone nods and goes, “Oh, yeah, yeah, yeah. We should do X.” And then everybody leaves the meeting and thinks that somebody else is responsible for it, and nobody's been clearly assigned that work, or nobody knows that work is ultimately their responsibility.Amy: And if you don't assign it, people are going to assume that this is going to be a thing that if they have time to, they'll get to it. And we harp on it enough that whenever work is not prioritized, it is automatically deprioritized. That's just the way task lists shake out, especially at the end of sprint meetings.Jesse: Absolutely. And I think that's one of the other things that's so important, too, is that it's not just about assigning the work, but it's about making sure that everybody who is involved in the conversation, everybody who's involved in the work agrees on what those boundaries are, agrees on who is responsible for what actions, more specifically speaking from a task responsibility perspective. Because at the end of the day, I want my team, whether that is my individual team or a cross-functional team, to all be bought into who's responsible for what parts of the project. We all need to be on the same page in terms of, “Yes, this is my responsibility. This part of the work is my responsibility. I will take ownership over this,” so that we can all help each other.Get that project goal together. One of the other big ideas that is so critical to starting a cloud cost management team is identifying and socializing your business KPI metrics. Now, this is something that some engineering teams already think about day-to-day. They might have ideas of service-level agreements, metrics, maybe service-level objective metrics, but there might be other business metrics that indirectly—or directly—relate to engineering work. It could be number of users using your SaaS platform, it could be number of API requests, it could be the amount of storage that customers are storing on your platform. You want to identify what these metrics are, and start measuring your cloud spend against these metrics.Amy: And as far as cost optimization projects go, the KPIs may not line up directly against how many servers you're standing up, or how many users are coming through. They'll be very indicative because you are spending money per user and per resource, but perhaps your business goals are different. Maybe you're not looking at trying to save money, but better understand where that money is going.Jesse: Absolutely. It's not just about how many instances are running per hour, it's not just about how many servers are running per hour, or how many users per server. It's really about understanding what are the core driving indicators of your business? What are the things that ultimately influence and impact how your workloads, and servers, and API functions, and everything, flow and grow and change over time?Amy: These metrics also can be influenced by things that are not architecturally specific, like savings plans, or the saving you would get through reservations, or some other contractual deal you get from your provider.Jesse: Yeah, that's one of the hard things, too, that we always hear from our clients. There is this idea that they think that they are spending a certain amount of money because they're getting discounts from savings plans, or from reserved instances or from an enterprise discount program, and maybe their usage is a lot higher than that, but because they get these discounts, they think that they're actually using a lot less than they actually are. And while this is not something we're talking about specifically or directly in this conversation, it is something to be mindful of because there definitely can be a difference between your usage and your overall spend if your company is investing in things like savings plans, and reserved instances, and discounts through either a private pricing addendum or an enterprise discount program.Amy: Yeah. Really, the bottom line with that is you want to be aware of what your business's goals are—and this goes back into buy-in, this goes back to leadership—that having a fully contextualized understanding of what it is that they want to do will help you make the right decisions and define your metrics in a way that basically helps you try to set your goals.Jesse: Absolutely. And all of this comes together in policies and best practices. All of this can come together in a way where you, your team, your cloud cost management team can put all of these ideas and all these things that everybody is agreeing to, into writing. Make sure that everybody is bought in and then write it down; make it an artifact and say, “Okay, after this meeting, we've agreed that the way that we are going to handle cross-availability zone traffic is like this,” or, “The way that we are going to handle scaling is like that,” or, “The best practices that we want for storage is this.” Put all this down in writing.Make sure that there are best practices being created. There's a number of clients that I've worked with before that have seen multiple different teams using the same service but using it in different ways. And maybe one of them has encryption and compression enabled and they've got this really tight turnaround time for their services, and another team doesn't. They're using a lot more data transfer because they aren't focusing on compression, for example. And this is an opportunity for a best practice to get everybody on the same page and say, “Okay. If you're going to use this particular type of service, you need to have compression enabled, you need to architect your services to focus on talking to other services in the same region, in the same availability zone to ultimately try to cut down on data transfer costs, or on storage costs, or other things.”Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself; make one of them go away. To learn more, visit lumigo.io.Amy: That brings up a really good point that I've noticed when I was actually coding day-to-day that each project and each team is ultimately different because you're building different things and you're building it with different people. So, it's entirely possible that your KPIs may be different between teams.Jesse: Yeah.Amy: But you're not going to know that unless all the other stuff that we mentioned. And it's perfectly fine if your KPIs are different between teams, or if your practices have to be modified to better work with what your goals are. And also that best practices, just like everything else in the cloud, can change. If the cloud architecture backend can change once every five minutes, you have to be able to be flexible and say, “These cost management rulings that we made two quarters ago, two years ago, they made sense then. And just like anything else, things evolved, we scaled, and our needs have changed, so we have to review these.”This is why before, we also mentioned is, like, maybe reviewing it as part of your cloud cost analysis once a quarter because things change all the time. AWS changes all the time. That's not a thing that I've complained about tons of times in different places, with a lot of recorded evidence.Jesse: Amy, why can I see this giant—your eye is just twitching, just this giant throbbing vein on your forehead right now?Amy: [laugh]. I have to start recording these podcasts with some kind of blood pressure monitor, and we can see, as soon as I say the word, “AWS starts changing stuff,” and just watch that skyrocket.Jesse: That is the quote. When our audio engineers post this, that is the quote to use to highlight this episode on social media.Amy: [laugh]. Yes, absolutely.Jesse: And I think to really bring this back around, all of these ideas, all of these things that we're talking about aren't just about saying we're going to do the one thing and we're going to do it that way for all of eternity, like Amy said. Things change over time, and that's fine. That's perfectly normal. So, your best practices should change over time, too. Maybe one of the things that you write down as part of your best practices is that you're going to review your best practices, maybe once a quarter, once every six months, every year, maybe once every, whatever time period works best for your team, the way that your workloads work, the way that your team works, the pace at which your team works, make sure that you're actively reviewing this information because all of us have seen documentation that is written once and is immediately out of date, and nobody ever touches it ever again. And that's not what these best practices are about.Amy: If you want to make sure that teams are bought in, show that you care, that you are aware that this stuff that they work on evolves and changes with them. If you want them to care about cloud cost management policies, it's hard enough to say much less hard enough to get buy-in. You want them to know that you're doing it with an awareness of what they are doing and then why they are doing it. You don't want to go in and say, “We're making widespread changes. We do not care what happens to your infrastructure because of it.” You want to go, “Because you are running things this way, it has to look like this because the cost is going to look like this.”Jesse: Absolutely having data to back up your decisions really, really helps in every decision you're making. It shows that you're making data-driven decisions; there is a why behind what you're doing. And it helps other people understand what you're doing as well.Well, if you've got a question you'd like us to answer on air, please go to lastweekinaws.com/QA. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review. Give it a five-star rating on your podcast platform of choice and tell us what your ideal starter kit would include.Announcer: This has been a HumblePod production. Stay humble.
Links:The cloud economist starter kit: https://www.lastweekinaws.com/podcast/aws-morning-brief/cloud-cost-management-starter-kit-2/ TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I'm Jesse DeRose.Amy: I'm Amy Negrette.Jesse: This is the podcast within the podcast where we like to talk about all the ways we've seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we're going to be talking about balancing cost optimization work against feature work.Amy: Buckle up everyone. I've got a lot of thoughts about this. Just kidding. It's just the one: don't.Jesse: You heard it here first, folks. Don't. Amy Negrette just says, “Don't.”Amy: Don't. [laugh].Jesse: So Amy, does that mean, don't balance the work?Amy: More like don't choose. It's always hard to make the argument to take an engineer off of feature work. This goes for all sorts of support tasks like updates and documentation, and as a group, we figured out that trying to put those off until an engineer has time to do it is not going to be a thing that becomes prioritized, it eventually gets deprioritized, and no one looks at it. And that's why DocOps is the thing. It's a process that now gets handled as part of and in parallel with software development.Jesse: Yeah, I've had so many conversations in previous companies that I've worked for, where they basically said, “Well, we don't have time to write documentation.” Or they will say, “The code is the documentation.” And, to their credit, there are a lot of places where the code is very cleanly documented, but if somebody is coming into this information for the first time and they don't have technical knowledge or they don't have deep expertise in what you're looking at, they need documentation that is clear, understandable, and approachable. And it is so difficult to find that balance to actually make sure that that work is part of everything that you do.Amy: And I think what the industry has decided is that if you make it a requirement for pull requests that if you're going to make a change, you have to document that change somewhere, and that change if it has any kind of user impact, it will be displayed alongside it. That's the only way to make it a priority with software. And cost optimization has to be treated in a similar respect.Jesse: Yeah, so let's talk about cost optimization as a process. To start, let's talk about when to do it. Is this something that we do a little bit all the time, or do we do it after everything's already done?Amy: I know I just cited CostOps as a good model for this, even though that's literally what we cannot do. We can't treat cost optimization as something we do a little bit along the way because, again, speaking as an engineer, if I'm allowed to over-optimize or over-engineer something, I'm going to take that opportunity to do that.Jesse: Absolutely.Amy: And if we're going to do project-wide cost optimization, we need to know what usage patterns are, we need to have a full user and business context on how any system is used. So, if we do a little at each step, you get stuck in that micro-optimization cycle and you're never actually going to understand what the impact of those optimizations were. Or if you spent too much time on one part over-optimizing another part.Jesse: It's also really hard if this is a brand new workload that you've never run in the cloud before. You don't necessarily know what the usage is going to be for this workload. Maybe you have an idea of usage patterns based on some modeling that you've done or based on other workloads that you're running, but as a whole, if this is a brand new workload, you may be surprised when you deploy it and find out that it is using twice the amount of resources that you expected, or half the amount of resources that you expected, or that it is using resources and cycles that you didn't expect.Amy: Yeah. We've all been in the situation, or at least if you work with—especially with consumer software—that, you're going to run into a situation where the bunch of users are going to do things that you don't expect to happen within your application, causing the traffic patterns that you predicted to move against the model. To put it kindly. [laugh].Jesse: Yeah. So, generally speaking, what we've seen work the best is making time for cost optimization work maybe a cycle every quarter, to do some analysis work: to look at your dashboards, look at whatever tooling you're using, whatever metrics you're collecting, to see what kind of cost optimization opportunities are available to you and to your teams.Amy: So, that comes down to who's actually doing this work. Are we going to assign a dedicated engineer to it in order to ensure it gets done? Anyone with the free cycles to do it?Jesse: See, this is the one that I always love and hate because it's that idea of if it's everyone's responsibility, it's no one's responsibility. And I really want everybody to be part of the conversation when it comes to cost optimization and cloud cost management work, but in truth, that's not the reality; that's not the way to get this work started. Never depend on free cycles because if you're just waiting for somebody to have a free cycle, they're never going to do any work. They're never going to prioritize cost optimization work until it becomes a big problem because that work is just going to be deprioritized constantly. There's a number of companies that I worked for in the past who did hackathons, maybe once a quarter or once every year, and those hackathons were super, super fun for a lot of teams, but there was a couple individuals who always picked up feature work as part of the hackathon, thinking, “Oh, well, I didn't get a chance to work on this because my cycles were focused on something else, so now I'll get a chance to do this.” No, that's not what a hackathon is about.Amy: You don't hack on your own task list. That's not how anything works.Jesse: Exactly. So instead, rather than just relying on somebody to have a free cycle, kind of putting it out there and waiting for somebody to pick up this work, there should be a senior engineer or architect with knowledge of how the system works, to periodically dedicate a sprint to do this analysis work. And when we say knowing how the system works, we're really talking about that business context that we've talked about many, many times before. A lot of the cloud cost management tooling out there will make a ton of recommendations for you based on things like right-sizing opportunities, reservation investments, but those tools don't have the business context that you and your teams do. So, those tools don't know those resources that are sitting idle in us-west-2 are actually your disaster recovery site, and you actually kind of need those—even though they're not taking any work right now, you need those to keep your SLAs in check in case something goes down with your primary site.Or maybe security expects resources to be set up in a certain way that requires higher latency times based on end-to-end encryption. There's lots of different business context opportunities that a lot of cloud cost management tools don't have, and that's something that anybody who is looking at cloud cost optimization work should have and needs to have those conversations with other teams. Whoever does this cloud cost optimization work, or whoever makes the cloud cost optimization recommendations to other teams needs to know the business context of those teams' workloads so that the recommendations they make are actually actionable.Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself; make one of them go away. To learn more visit lumigo.io.Amy: And they should also have the authority to do this work. It's easy to deliver a team a list of suggestions saying, “Oh, I've noticed our utilization is really low on this one instance. We shouldn't possibly move it,” or what have you. And because they're not the ones making the full architectural decisions, or leading that team, or in charge of that inventory, they actually don't have the authority to tell anyone to do anything. So, whoever gets tasked with this really needs to be an architect on that team—if you're going to go with this embedded resource type of person—where they have that authority to make that decision and to act on it and move things.Jesse: Yeah. It's really important that teams stay accountable to the resources that they're running. And some teams don't know any of the resources that they're running; they, kind of, deploy into the cloud as a black box. And that is a perfectly fine business model for some organizations, but then they also need to understand that if the senior engineer or architect who is focused on cloud cost optimization work for this group says, “Hey, we need to tweak some of these workloads or configurations to better optimize these workloads,” the teams need to be willing to have that conversation and be a part of that conversation. So, we've talked about a couple different ideas of who this person might be that does this work. This could be a DevOps team that attaches a dedicated resource to doing this analysis work, to making these recommendations, and then delegates the cost optimization work to the engineering teams, or it could be a dedicated cloud economist or cloud economist team who does this work.Amy: We did touch on having someone in DevOps do this, just because they have a very broad view and the authority to issue tasks to engineering teams because if they see an application or an architecture, where resources are being—or are hitting their utilization cap, or if they realize there are applications that need more or less resources, they're able to do those types of investigations. Maybe someone on that team can take up this work and have a more infrastructure-minded view on the entire account, see what's going on on the account and make those suggestions that way.Jesse: Absolutely. It's so important. Or if there is a dedicated cloud economist or maybe a cloud economist team that is able to make these recommendations, that has the authority to make these recommendations, maybe that's the direction your group should go.Amy: If only we spent an entire podcast talking about this.Jesse: [laugh]. Huh, if only we spent an entire podcast talking about how to build a cloud cost team and talk about how to get started as a cloud economist. Hmm…Amy: Please check out the cloud economist starter kit that we all have already published.Jesse: Yes, several weeks ago. We'll post the episode link in the [show notes 00:12:38] again. So, Amy, we've talked about when to do this work, who should do this work. What I want to know is how do these teams come together to have these conversations together? I'm thinking about best practices here. I'm thinking about how do teams start building best practices around this work so that each team isn't working in a silo doing their own cost optimization work?Amy: If you're lucky, someone in your company has already done this work. [laugh]. And you can just steal their work.Jesse: Absolutely.Amy: Or borrow. Or collaborate. Whatever word you want to use.Jesse: [laugh].Amy: See if you can see how the project went, how they structured it. Maybe they ran into a process issue like they weren't able to get the kind of access they needed without jumping through a whole bunch of red tape and hoops. That's a good thing to know going into one of these projects, just being able to see the resources that you're going to be looking at, and making sure you have access to them.Jesse: Absolutely. This is part of why we also harp so much on open and clear communication across teams about the cloud cost management work that you're doing. If you are trying to solve a problem, it's likely that another team in the organization is also trying to solve that same problem, or ideally has already solved that problem, and then they can help you solve the problem. They can explain to you how they solved the problem so that you can solve it faster so you don't have to waste engineering cycles, trying to reinvent the wheel essentially. It's a really, really great opportunity to build these best practices, to have these conversations together, maybe to build communities of practice within the organization, depending on how large your organization is, around the best ways to use these different tools and resources within the organization.Jesse: Well, that will do it for us this week. If you've got questions that you would like us to answer on an upcoming episode, go to lastweekinaws.com/QA. If you've enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us how you integrate cost as a component of your engineering work.Announcer: This has been a HumblePod production. Stay humble.
Links:Ted Talk: The Science of Productive ConflictTranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure. Today, we’re going to talk about the people side of technical projects, especially people who might introduce roadblocks for completing technical projects. Now, you might be thinking to yourself, “Jesse, Amy, that sounds like it is not about AWS.” But let me assure you that any project involving AWS is going to involve multiple different personalities approaching the project from different angles, who ultimately all have the same solution in mind, but have different ideas about how to get that problem done, or different ideas about what’s the right thing that ultimately get done to begin with. So today, we want to talk about that: we want to dive into how can you have really rewarding conversations with those folks? How can you better engage people who are intentionally or unintentionally difficult?Amy: We want to be very clear that we are not trying to come after anyone. Every time that I’ve gotten an engagement, it isn’t because someone means to be difficult, but maybe there’s a project timeline, maybe there’s something else getting in the way of them being able to fully be present for that specific part of the engagement, and maybe that’s just what’s causing friction, causing speed bumps. And we’re all well aware of this. Jobs are hard, and especially this sort of work can be difficult. So, first of all, we totally understand, and this is just more about how to get everyone moving in the same direction at the same pace.Jesse: Yeah, absolutely. I mean, especially with the pandemic going on right now, everybody’s doing remote work, some people have never actually met their teammates in person and they’re expected to work together efficiently, and quickly, and easily. It’s hard.Amy: It also doesn’t help that when we do come in, we come in under the context of a cost optimization project, or some other efficiency-type title. And that sounds a little like the Bobs from Office Space, which I bring up a lot, especially during internal meetings. And it makes it sound like we’re going to come in to shake a bunch of things up and look for inefficiencies where there aren’t any, which is truly not the case. And it can cause a lot of insecurity, especially about how someone thinks that they’re doing their job, or that their job is somehow going to be impacted by what our suggestions are. It may not just be us, but it may be another migration consultation suite, where someone’s coming in to change the architecture that they’ve worked on for a long time, and that can put a lot of people in a state of unease.Jesse: And I think it’s also important to note that it’s not just about an external party coming in like Duckbill Group or another external, third-party consulting service, or technical group. It could be an internal separate team. It could be your internal cloud cost management team that is starting conversations with development teams saying, “Hey, I want to better understand how you’re using AWS. I want to understand some of these cost optimization opportunities.” Even in situations like that where all of these conversations are internal within the company, even within teams, there are still multiple different personalities, multiple different people approaching the problem from different angles, and it’s still really, really important to make sure that you approach them collaboratively.Amy: And ultimately, we wanted to be clear that what we’re going to be talking about is helping people think differently into a growth mindset, and being able to do this work without anyone feeling shame or embarrassment.Jesse: Yeah. Growth mindset is so critical. It’s something that I love to talk about ad nauseum, and so I won’t dive into it too deeply here, but—Amy: That’s another episode.Jesse: [Laugh]. Exactly. Growth mindset is so important for folks in technology teams, especially in today’s technology era where there’s just so much constantly innovating. There’s so much new constantly going on around you, to new technologies, new teams, new ideas, new ways of doing things, new processes, new tools; it’s really important to be open-minded to learning those different things. You don’t have to use every single one of them, but be open-minded to different people approaching problems from different perspectives and different angles.Amy: Having to face all of this uncertainty will cause some to not be the most cooperative when they have to start reacting into these situations, whether it is an internal change that’s happening, or if it’s an external consulting group; they can start coming back and taking a various sort of stance, and just like being back in middle school, sometimes standing up to a bully is simply how you have to succeed because it’s not about dominating, it’s about compromise and trying to find out what you’re trying to do and find that common ground.Jesse: Yeah, absolutely. I think that’s the most important part here because when we talk about working with other personalities that are different than yours and having conflict, it’s not about dealing with them; it’s not about overcoming them from the perspective of winning the argument, so to speak. It’s about how do you compromise? How do you effectively find that common ground and move forward together? And sometimes it’s just about sharing context, it’s just about sharing that mental model that you have that might be different than the mental model that this other person has, or maybe the other team has.Like for example, some teams that we’ve talked to can’t make a cost optimization change due to security, or legal, or product SLA restrictions, but maybe the person who’s coming in from the cloud cost management team or cloud cost management side doesn’t know that because they aren’t as familiar with the product.Amy: But it can also just be a staffing issue. These projects take work, and if an engineering team is already stressed and stretched to the edge, they’re not going to have the resources, and they don’t want to be the ones to say we simply don’t have the manpower to do this.Jesse: Yeah, absolutely. And it’s so, so important to be able to identify those bottlenecks or identify those constraints. Because ultimately, if you give a team that already has a ton of things on their roadmap new work and say, “Hey, cost optimization is important.” They’re most likely going to ask you, “Okay, where does this fit into all the other things that are already on our roadmap?” We’ve talked to a lot of companies who struggle with that balance of prioritizing new feature developments with cost optimization work.And there definitely needs to be that healthy balance between the two because new feature work is obviously important for the business to grow, but cost optimization work is also important within these processes as they go through more and more agile sprints, build more and more things to make sure that each team understands what are their opportunities to really optimize their spend as they’re building new architecture.Amy: We’ve all seen that bug board where everything is a P1 bug.Jesse: Yes. The thing that’s coming to mind for me is if it’s everyone’s responsibility it’s nobody’s responsibility. And in the same way, if everything is a priority one problem that needs to be fixed, then essentially nothing as a priority one problem that needs to be fixed, nothing is going to get done because the teams are going to get completely burned out with context-switching constantly between one priority and the next, rather than being able to actually focus on each piece of work as it comes up.Corey: This episode is sponsored in part by our friends at Lumigo. If you’ve built anything from serverless, you know that if there’s one thing that can be said universally about these applications, it’s that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You’ve created more problems for yourself; make one of them go away. To learn more visit lumigo.io.Jesse: So, I was thinking about some of the best ways that we’ve seen this kind of work handled within organizations, some of the best ways I’ve seen this handled at previous companies that I’ve worked at, some of the best ways that we’ve handled this with some of the clients we’ve worked with. I actually ended up listening to a really great podcast episode about the science of productive conflict, and I’ll link that in the [show notes 00:10:18]. It basically broke down three types of conflicts; you have task conflicts, relationship conflicts, and status conflicts.Task conflicts are your disagreements about a problem, a solution, or decision. Maybe I think that the best way to implement this new feature is in Python. You may think the best way to implement this new feature is in Golang, or Ruby, or something else. That kind of disagreement is a task conflict.The next one is relationship conflicts where you’ve talked about differences in personalities or values. So, maybe I come from a world where product is the most important thing in everything that I do, so I always want to make sure that we focus on feature things first; I always want to make sure that I am doing what is asked of me. I always want to make sure that I am a yes man, so to speak. Whereas maybe you come from an environment or a space where you are more open to pushing back, having collaborative conversations. And it’s just different mental models of how we have been raised in the world, how we view the world, and it’s really, really difficult for us to get out of those different models, so it’s a harder type of conflict to have, relationship conflicts.And then the third one is status conflicts where we disagree about where we fit into this hierarchy that we’re in together. Basically, who’s in charge? Who gets to decide what gets done?Task conflicts and status conflicts can be productive; relationship conflicts are generally not productive because like I mentioned, relationship conflicts really come from people with different mental models, different views of the world and it’s unlikely that you’re going to change someone’s fundamental values. But with that said, having a conversation with the other person, establishing psychological safety, giving them that space to say, “Hey, I want to know more about how you view this problem, so that I can also share how I view this problem, and we can better understand each other,” that’s going to make a world of difference in helping both sides better understand each other, and then also find the right solution, better opportunities for maybe a cost optimization team to learn that, maybe, there are these restrictions that mean that they can’t apply all the savings opportunities that they want to, and that’s fine; that’s a business decision that the business and the organization needs to make. But it helps both teams understand that so that they can have more constructive conversations about where, where are the opportunities in areas that we can make cost optimization improvements? We can have good technical conversations together?Well, if you’ve got questions you’d like us to answer go to lastweekinaws.com/QA. If you’ve enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us how you try to form those more collaborative, psychologically safe conversations when you run into conflicts in your organization. Thanks again.Announcer: This has been a HumblePod production. Stay humble.
“There are a million paths to motherhood and there is no right way or wrong way. It is just your way.”Jesse Truelove with @movewithtruelove joins us on the podcast today and drops some serious truth bombs about birth, fitness, social media, and motherhood. We couldn't be happier to have her with us and for you to hear the important messages she has to share! Our mission at The VBAC Link is to empower ALL women in their birthing choices, whatever they may be. We applaud all women for choosing the birth path that is best for THEM. There is no right or wrong way to give birth! Additional linksInstagram Live with Jesse and TVLBirth Words: Language For a Better Birth PodcastVBAC vs Repeat C-Section BlogMove with Truelove: Jesse's websiteAB Rehab courseMove Your BumpFull transcriptNote: All transcripts are edited to correct grammar, false starts, and filler words. Meagan: Happy Wednesday, everybody. You are listening to The VBAC Link. This is Meagan. Julie is on with us and we have a special guest. Her name is Jesse Truelove. We are excited to talk about her episode today because this is something that is actually a first for The VBAC Link. We were just chatting a little bit before. We were like, “Why haven't we had this happen?” because this is totally what we talk about, what we teach about in our courses, and this episode is where she had an emergency C-section, ended up recovering from a really difficult delivery and had very little support. And really, for her next child, when all was said and done, she decided to have a repeat C-section.And I love this. I love this because as The VBAC Link, as Meagan and Julie-- yes, I am talking for you, Julie. We are all about everybody making the best choice for them. That may not be a VBAC. Some people may not choose to have a VBAC. One of my best friends has had three C-sections. The first one was unexpected, two were scheduled. I love and support her in that even though that is not what I chose to do. I chose to VBAC. I support her in that and I am so grateful that she had that opportunity.So we are really excited to dive into this episode today and hear her story. She has done so much good in the world. After her births, she decided to dedicate her focus to helping moms recover fully from their pregnancy and delivery. We are going to talk all about that in the end because she has got some pretty cool stuff that she is in charge of.Review of the WeekMeagan: But of course, we have a Review of the Week before we dive right into it.Julie: All right, yes. The Review of the Week. This is kind of a long one, but I really feel like it goes in line with this episode. I'm going to tell you about how I shamelessly stalked Jesse after this. But this review was actually an email. We got an email from Christina T. We really love getting emails from people who we have helped along the way, and so if you want to reach out to us through Messenger, or email, or Instagram Messages, or leave a review on Apple Podcasts, or Google, or Facebook, or wherever you want. We reply to all of the email messages and DM's that we do get. We love talking to people and hearing their stories. This is a long one, so bear with me, but I really like it. She says: “I wanted to tell you about my repeat C-section. On the podcast, I have heard that term so many times. ‘Hold space.' For me, that's what you ladies have been preparing me for these nine months without me knowing it. For holding space. “For nine months, I prepped for the VBAC of my dreams. I listened to every podcast, was a member of the Facebook group, did ‘all of the things to prepare for the birth I thought I so desperately needed. I was ready to have an unmedicated birth and to roar my baby into this world. At 32 weeks, we found out baby was breech. I had been going to the chiropractor twice a week and felt confident he would flip. He did not flip. I then started to kick on my Spinning Babies®, got acupuncture, did everything online I could come up with and he still would not flip. “Around this time was the week you guys had your repeat C-section podcast and for me, it was a sign to start mentally preparing for what might not be. I spent the next few weeks switching my mindset from feeling sorry for myself and switching it to feeling strong. It was during those weeks I thought to myself, ‘What am I missing? All of this education and research can't have been for nothing.' “That's when it clicked for me. I had been preparing to hold space for what won't be and I have the tools to do that. We scheduled my C-section and when March 8th came, I was as ready as I was going to be. The anesthesiologist came to the room and I knew it was going to be a wonderful experience. Prior to this baby, we had suffered two losses in a row. We needed to have a D&C for the second loss and all day, different staff members asked me to state in my own words what procedure I was having. Each time I would break into tears as it was a very difficult thing.“My doctor came into pre-op and instead of asking the same questions, he simply said, ‘I am sorry you are here for this procedure. Are you doing okay?' The same kind, genuine man was going to be my anesthesiologist now. When I first walked into the OR, my stomach dropped. It was bright, cold, and sterile just like last time. My doctor must have sensed my heightened awareness and said, ‘Your baby is coming.' And with that, I was ready. “We were able to drop the drapes, watch our son come into the world, and have skin-to-skin right away. It was night-and-day from our first experience. It was joyful and blissful, and I left a feeling like a mom and not a patient. I left feeling confident and like a bad-ass. I left holding space for my experience and for our story, and I will be forever grateful for that gift from you ladies. “Birth doesn't always go as planned. Sometimes it is better.”And now, that gives me chills at the end. “Birth doesn't always go as planned. Sometimes it's better.”Meagan: I love that. Sometimes it's better.Julie: I'm going to make that into an image.Meagan: Yes. Let's make that an image and quote her because that same thing. I just got the chills.Julie: I know. I have goosebumps.Meagan: I was like, “Yes.” Sometimes it doesn't go as planned, but it's better. Yeah. I feel like I can connect to that even with my second C-section. I didn't want that second C-section, but guess what? It was such an amazing experience and it healed my first birth experience.Julie: Yeah. I love that.Meagan: I love it. Awesome.Julie: All right. Well, let me tell a little bit of a funny story and then I promise I won't take up much more time. Jesse, we did an Instagram Live with her on her Instagram page.Jesse: It's still there.Julie: Yeah, @movewithtruelove, right? That's what it's called? It was really fun. This was a long time ago. We followed her and I just love her content. She has great content. Super fun. Her reels are amazing. I just love seeing her bright, beautiful face as I scroll through our feed and everything. I just have really enjoyed following along with her Instagram. I knew that she had a Cesarean and that she was pregnant again. I don't even know this whole story, but at some point along the way, Jesse decided that she was going to have a repeat Cesarean instead of attempting a VBAC. I was like, “Oh my gosh. Okay, this is really cool. I need to follow along and see how this all ends up,” because we haven't ever had a story on our podcast about somebody who had decided to have a repeat Cesarean.Meagan: Even though there are tons of people out there.Julie: Even though there are lots of people that do all the time.Meagan: Tons, yeah.Julie: All of our C-section stories--Meagan: In fact, a large chunk of people, the majority do.Julie: Yes. Yes, and we have had three VBAC stories where people have tried for a TOLAC and then ended in a repeat Cesarean, but we have never had somebody that has decided during their pregnancy that a Cesarean was the right way for them to go. And so I was so excited. I am like, “Oh my gosh. I need to follow up with this. I need to make sure she's on my radar,” and then she announced her cute little birth story was on a little reel on her Instagram page. You need to go find it. It is the cutest thing ever. I knew that she had her baby and she had a C-section, and then I was like, “Oh my gosh. I want her to share her story on the podcast, but I don't know if she will be open to it,” because I know, from what I picked up from the reels, it wasn't an easy decision to make. I didn't want to overstep my boundaries. She is super cool and way bigger than us on Instagram. I feel so small and tiny, but one day, I am like, “Okay. I'm just going to reach out, and I'm just going to say how much I love her, and how excited I am that she had a really good birth experience and that I would love to share her story on the podcast if she would like to.” And she said, “Yes.”Jesse: Oh my gosh. I only got good vibes from our last conversation and I literally wanted you guys to do my VBAC. That's what I wanted.Julie: Yeah.Jesse: It was a big change for me. It was definitely a mind shift, but it wasn't as big of a deal as I realized it was once I put it out there on social media.Julie: Well, and I'm sure you got a lot of support. I don't know. Maybe there were some people, but I mean social media. As we are getting bigger, there are people that argue with us and disagree with us all the time. But I just love that you were confident in your decision. You just radiated that security and that confidence. I think that that's what we all want going to birth is just being confident and feeling supported. That's the vibe I got from you when you were talking about it and sharing your birth story and things like that.Julie: Without taking up too much more time, first of all, thank you so much. I'm so excited to have you share your story and to listen to you share about that decision that you made. Then we are going to talk a little bit afterwards about when it might be a good idea to choose a repeat C-section, and then Jesse is going to share some of her really awesome resources.Meagan: Awesome stuff. Yeah.Jesse's storyJulie: Yes. She has lots of really cool stuff. So hang in there with us because this is a really really good story and you want to hear what she has to say at the end. All right, Jesse, you've got it.Jesse: Okay, thanks so much, guys, for having me. I am actually really, really honored to be on your podcast. I had such a good experience chatting with you guys. It felt so, so natural the first time we did it, so it was a very easy “yes” for me.So I guess I will start out with my first C-section which really has paved the way for everything that I am doing now. Really, everything about motherhood has shaped everything that I do now. With my first pregnancy, I was working out really hard. I have been a personal trainer since 2014 specializing in women's fitness. I have taught in multiple states boot camp, and circuit classes, and personal training for abs, and all kinds of stuff, but it wasn't until my own delivery really, because I had a fairly easy pregnancy in respects to working out. I was very active, and so I thought in my head, I had this very clear picture of what my birth story was going to look like. It just seemed really, really easy to picture what it was going to be. I thought if anybody could, this is a really weird thought to have, but if anybody could have a natural birth and if anybody could do it successfully-- it didn't even cross my mind that a C-section could be in my future. Meanwhile, I am the oldest of six kids and my mom has had four kids via C-section. So pretty interesting that I never thought it was going to cross my path.I went Into my delivery, like I said, with a very clear picture of what my birth story was going to look like and it quite literally went the exact opposite. I was in labor for 26 hours. I was bleeding. I got a uterine infection. I had a fever. My placenta was failing. Heart rate was dropping for the baby. They broke my water. I got Pitocin. It was literally my worst nightmare and I was so, so exhausted by the time that my doctor-- well, I couldn't have my midwife anymore who had been with me the entire time. By the end of the 26 hours, I had the OB come in. He was checking me for dilation and I just wasn't dilating. They gave me Pitocin and they broke my water. I think I got to maybe an 8 or a 9 and it just wasn't happening. And then contractions slowed down and I really was so exhausted. He came in. He was checking me for dilation and he asked me to push. I was just so out of it. I didn't even know how to push. I feel like if it's such a medical experience, you go in and you are hooked up to these monitors, it just doesn't feel natural. It feels really really medical.Julie: Yeah.Jesse: Especially when doctors are coming in and nurses are coming in checking your blood pressure, and you've got those monitors, and beeping, and honking, and all of the stuff going on. You don't even know what's actually happening with your own body and then I had an epidural by that point as well, and so you know how an epidural feels. You could kind of feel the contractions coming and going but it's not even close. So your doctor asks you to push and you have no idea what he's talking about. Push what? Most women don't even know what the pelvic floor is and I was one of those women. Before my pregnancy, I was lifting and pushing some heavy weight. I was deadlifting 255. I am a five-foot person. That is a lot of weight for a tiny person and I didn't have the mindset of function.I had one picture of what fit looked like, one picture of what strength looked like, and it was not anything of what strength really is. Motherhood taught me that. He asked me to push and I didn't know what he was talking about. He goes, “Oh. You can't do this. You are going to need a C-section.”Julie: What?Jesse: “It's going to be C-sections from now on.”Meagan: Whoa.Jesse: I laid there completely drugged out of my mind. I look at my husband and tears are just welling up in my face. I could cry just thinking about it. My husband is-- you know, we think that we are the strongest people we know. We love each other so much. He wanted to take that whole experience on himself and it was killing him that he couldn't. He put on a brave face. It is hard for husbands too. They go through this with you and it is so much. He looked at me and he was like, “It's going to be okay. We just have to do what we have to do.” In my head, I am already thinking of the future. I'm like, “What do you mean? All my next pregnancies are going to be a C-section?” I feel like it was the worst possible time to lay that on me and it's just the standard of care. It's just not there for moms. So the OR doctor left. He was assembling his team and the anesthesiologist had to be called in. We are in the middle of nowhere out here in the Oklahoma Panhandle so they have to call the team in. They're not just there waiting for you if something goes wrong. So we are waiting for the team. I am laying there and all I could think was, “I have totally and completely failed before I even got to start. My body has totally failed me and I don't even know. I am supposed to be able to do this. This is what I was made to do and he just told me I can't now and I'm never going to be able to.”That stuck with me forever. And then, we are getting ready for the C-section. I have never had any kind of surgery ever. Never, ever, ever, and so I am terrified. I was already terrified to give birth and now I am terrified to go be cut open. And so they had me hunched over-- oh no, this was after I had my epidural. So they changed it over to where they just have to keep pumping the epidural in, so it wasn't a spinal tap. They put me onto the new table. They strapped me down to the table, which was another thing I didn't expect to happen either. It is all such a mental hurdle to get over. You're like, “Wait. They don't tell you any of this going in.”Julie: Yeah.Jesse: But that just all adds to the mental trauma of, “You have no control in this. We are strapping you down. You are numb. You can't have your baby the way you wanted to. You are getting wheeled out of the room you just got comfortable in. Your husband has to wait out here.” It was hard. It was so hard.And so, anyways. We get rolled into the OR room. Just like your mama said previously in that story you were sharing, it was cold. It was sterile. A bunch of people that you don't know are in there. Your husband is not in there. You get placed onto the table. You get strapped down. You get the blue sheet at your neck and it's an emergency situation because the heart rate is dropping. I had a fever of 103 because they had checked me so many times.Meagan: Wow. Wow.Jesse: I had gotten an infection literally while I was there within 24 hours in my uterus and that's why I was on antibiotics for probably-- I was in the hospital for five days. I was on antibiotics the whole time which, you know. Antibiotics, just the additional medicine. I'm not one to even pop an Advil for a headache, so it was just a lot.So they are doing the surgery. It was all very fast. It was a ton of pressure. A ton of pressure, relief when they took the baby out. They sewed me up pretty quick. Again, I had no idea what was happening afterwards, so they took Radley out and I could hear her screaming. In that moment, it is all totally worth it. It is all worth it because even at the end of your emergency, traumatic, C-section, you still get a baby. The baby that you have been dreaming about for nine months, the baby I've been dreaming about-- for me, it was since I was a teenager. Me and my husband have been together for 12 years, since high school, and we had been talking about our kids forever. So in that moment, I am like, “Oh. I could do it again. I could do it 1000 times the same exact way. I would go through everything the same because at the end of your really shitty experience if it is shitty-- which mine was-- it's still worth it.So they brought the baby over to Shane. Shane just put her on my face because that's the only skin I had accessible. They didn't tell me about skin-to-skin. They didn't drop the curtain. I didn't get to see anything and I felt like I missed out. I felt like I got gypped in my birthing experience, which I did. I think that's okay for moms to feel. I feel like there's so much judgment around feeling what you feel about your birth. I feel like a lot of moms, and myself included-- I didn't talk about my first C-section for a long time and I didn't feel comfortable in sharing the fact that I wasn't just grateful for my baby being alive, but I was really pissed. I was mad about my C-section. I was mad about how I was spoken to. I was mad that I felt like my body had failed me.Nobody made that feel valid for me. Nobody asked how I was really doing. Because after pregnancy, I feel like a lot of moms can attest to this, it stops being about you and it goes all to the baby. You get one check-up postpartum and then that's it. At my six-week check-up, they didn't even check me internally. I got the magic check at six weeks like, “You are good to go to back to whatever you were doing before you got pregnant,” and so I went back to doing those things. I realized quickly that my idea of strength and my idea of being fit was so terribly wrong.I had never felt like more of a stranger in my own body. I got home with this new baby that was beautiful and perfect and I had a brutal recovery from being in labor for so long, and the infection, and all of that, and then recovering from a C-section and major surgery. They don't really explain that to you in the respects that you should be getting rehab postpartum to be rebuilding connections with those muscles, and movement patterns, and muscle recruitment patterns, and all of those things. And so I went into my recovery pretty blind. I realized quickly that moms don't get much support at all through anything postpartum.And then I got really educated and I built courses for moms to rehab their own bodies postpartum, everything that they need. I realized that moms don't need a six-pack, even though if you want one, it's totally possible. What we need to be able to do is sneeze without peeing our pants and nobody could have told me how to do that. There wasn't that information really out there. It was really just Kegels and if Kegels don't work, get a surgery. Another surgery. There's just a lot of room for improvement out there for the standard of care for moms and that's what I get to do now. And so I love that.And then in my interim between my next pregnancy, I had a lot of focus in my mind and talking with my family, and talking to you guys that I wanted to have a VBAC, that I was confident that was just a one-off thing, and that I was going to be able to have a VBAC, and it was going to be successful, and that's what I was going to try for, and all of those things. And then once I actually got pregnant, we did experience one loss after my first baby and we had a miscarriage. I got pregnant pretty quickly afterwards. I was discussing with my husband what we are going to do because you just don't know what you're going to do until you are there.So once I was pregnant and thinking about where we were going to deliver, who was going to do the surgery because the OB that did my C-section the first time didn't live here anymore, which probably was better because this doctor that I got this time is just amazing. And then after I met him, I felt a lot more comfortable with choosing another C-section. After going over my options with him-- which I think is super important. If you want to choose a C-section just because you want to choose one, you have those reasons in your mind about why that is a better choice for you and that should be okay. Those should be valid reasons. But I did ask him some things trying to get his medical opinion on what was the best choice for me even though in my head after I was sitting in the hospital, I was like, “I really don't want to be stuck in the same position that I was last time. I don't want to be in labor for 26 hours to get stuck again and to have to go through another labor, the C-section, and a surgery, and then have to go home and take care of two babies.” And that, in my head, was really important to me to still be able to do everything I needed to do and not have such a tough recovery because I remember my recovery being so, so hard from basically going through two deliveries. The 26 hours of labor and feeling all of that, and then going through my emergency C-section.So when I was talking to him, he basically gave me some options. He did mention uterine rupture. I know the odds are very low, but like I said, we are in the middle of nowhere. He personally had seen some uterine ruptures happen and you just don't know that they are happening until baby's heart rate is dropping and for us, that risk of not having a team on staff because the hospital is so small, that risk of having to call a team just wasn't worth it to me. I had to weigh the options and weigh the risks between a repeat Cesarean, which there are risks and the risks of trying to labor and then ending up in the same position that I was in last time.And so we ended up choosing a repeat Cesarean and I felt really comfortable with that knowing that I was going to be scheduled, knowing that my mom was going to be in town, and being able to watch my other daughter. That was really important to me. My daughter's experience through us being gone because we have never left her with anybody before. I just had a ton of stress surrounding that. Not to say that the second C-section didn't bring me a lot of stress too. I don't know why I had this irrational fear, and moms are really good at this, that I was going to die. I had this irrational fear that I was going to go into surgery and not come out for my toddler.You don't have that fear going into the first one of not getting back to somebody, so that was really hard for me. I was shaking like a leaf laying on the table going in for our scheduled C-section for River, my second daughter. It's funny because just like the mom that you mentioned earlier in the beginning of this show, you can get really lucky with the staff that you have for your experience and I totally lucked out. My anesthesiologist felt like family. It's funny to say because you meet them, and they come in and tell you the risks and stuff for the spinal block and all that, and they talk to you, and you are like, “Gosh, I am so scared but for some reason, you just are calming those nerves.” I think it is so important to have that type of support team. You can just tell this guy had daughters. I went into the OR room. They were in there. I'm leaning over my nurse's shoulder and she is just holding me. They are putting in the spinal tap and they laid me down. Again, you're in there without your husband. The lights are all on, cold, sterile, they are strapping you down, and I just had this fear that I was going to die.They are working and they get her out. I hear her screaming and again, it was the most beautiful noise and sound I had ever heard. They bring her over and in that moment, it was just peace. It was so much peace and again, it was just all worth it. They clean her up and they sewed me up. My doctor was really, really careful with my C-section scar which was really important to me too for adhesions and making sure that everything was sewed up very carefully. They took dad and baby out. I went to recovery and it was probably 30 minutes. I feel like that's something they don't tell moms either that you are going to be in recovery by yourself which is always a little bit sad. It was for me. I had both situations where I was in the recovery room by myself, but the feelings that I had surrounding my second C-section were not failure, were not that my body had failed. I had so much power in the choice that I had to be able to know my outcome and it did heal me from my first experience as well because I mourned that delivery. I mourned that experience that I felt like I missed out on. Even though I didn't get to have a vaginal delivery, I didn't get to have that experience, I still got to experience something beautiful and everybody came out okay.Birth words matterThat's not always the most important thing to bring up to a mom too. She is proud of her baby. She is proud that her baby is happy and healthy, but it is also really, really important to let her feel everything else. I feel like that's what this world needs more of. Instead of-- well, I didn't realize how much judgment there was around choosing a repeat C-section until I put it out there. And I was, oh my gosh. I got shamed so hard by a doula. She told me that I was saying I was too good for labor.Julie: Oh my gosh.Jesse: That labor was beneath me.Meagan: Whoa.Jesse: I thought that this was such a dangerous position for that lady to be in as a women's birth support person and her personal feelings around C-sections to a person that she does not know. She is a mother herself and to shame a mom-- I literally was trying to rationalize in my head during my own time of choosing a C-section that this was really the best thing for me and for my family. We don't have family here in town that can come and take care of our house and our baby, so we needed our family to fly in. That was a really big thing for us. We needed to make sure our toddler was taken care of. That alone could have been my only reason to choose a C-section and it should be okay because I do have other kids to take care of.Regardless, I have my own mental health to take care of and my first experience was crippling. There's nothing like looking at your body after you have this beautiful baby in your arms and thinking, “Gosh. I feel like a failure. I feel like my body failed. I feel like I can't do the one thing a woman is supposed to be able to do. I don't recognize the body that I am in.” I am supposed to be this strong fitness person. I put all this pressure on myself to bounce back. I didn't know how to do that. There's so much pressure. It is so much pressure from not only ourselves because moms do that. Women do that. But especially with social media, it can be a blessing, but it can definitely be a curse with how easy it is to access people and access very vulnerable parts of people's lives. This whole C-section conversation that I have become so passionate about talking about really stemmed from that doula's comment to me because I didn't realize that women were getting so much shame from this. It wasn't until I put my own experience out there that women were like, “You know, yes. That is me. I got shamed by my own mom. I got shamed by my best friend.” People telling me that I didn't really birth my baby. That stings as a mom. Our words quite literally carry weight to the people that we say them to whether it's a stranger, or a friend, or somebody on social media, or somebody in your family, sister-in-law, mother-in-law, all the things. There just has to be more support all around because at the end of the day, we are just moms trying to get home with our healthy baby.Julie: Yes.Jesse: And there's no shame in that no matter how you do it.Meagan: Right. We are honestly-- yeah. A healthy baby and we are wanting to stay healthy, but we are also wanting a good experience for us.Jesse: Yeah.Meagan: When I was choosing to do VBAC, I also got the backlash of, “Just schedule a C-section. Why would you do that? Why would you chance it?” I mean, it goes both ways.Jesse: You just can't win.Meagan: Literally, it doesn't have to be about birth. Vaccines, masks, no masks for COVID--Jesse: Totally.Meagan: No matter what it is, it is this battle. I wish so badly to this day that people could just take a step back, take a deep breath and say, “Okay. I know where I stand. I support where you stand whether it is the same or not.”Jesse: Yeah, totally.Meagan: Because if we could just be validated in our own choice and not be questioned and put down for making the choice that we feel is best, this world would be so much happier and less battley and snarky, right? I chose to VBAC. You chose not to VBAC. Do I think you are any less of a person? Do I think you birthed your baby any less? Hell no. That's the answer.Julie: Meagan just said “hell” on the podcast. It's the first time.Jesse: I was going to say something worse, but I didn't know if that was acceptable.Meagan: I know.Jesse: Yeah, I know. It can get a little raw in motherhood, okay guys? It's pretty crazy. There are a million paths to motherhood and there is no right way or wrong way. It is just your way.Meagan: Exactly.Julie: I love that. There are a million paths to motherhood. Hold on. I need to make some notes for some social media posts later. Hold on.Jesse: Write it down, Julie. It's pretty crazy. We are the largest population. There are so much more that connects us than what divides us. We let those, they are really just subtle and big differences, but we let them divide us. If moms would just come together, like you said, it would just be so much nicer because there is already so much pressure from other moms. We say that we want to support. We said we want to be validated, but it is usually moms that are so vicious to other moms and it's mind-blowing to me.Meagan: Which breaks my heart.Jesse: Yeah. It really is heartbreaking because we have all got the same goal and that is to raise these little people with love. Everybody has a different way to either get to motherhood, because that is a journey all in itself and one to be very cognizant of, again, with the comments. When you are going to have another baby, you really don't know the lengths that either a couple is going to be having to have a second baby or even to have their first baby.Meagan: Exactly, yeah.Jesse: Even a birth experience can really change the way if you want to have another baby or not, and recovery. If you don't feel supported for the first one, it's going to be really hard to feel supported in the second one unless something changes. I think that's what we in this field are trying to do. We are trying to change that. We are trying to change and raise the standard of care for moms because we are the population that raises the next generation, that keeps the world going. If you help moms, you help the future and that's what we are trying to do. That's what every mom deserves.Meagan: Wow. Yeah. I'm going to roll back even to what your provider said. He was like, “Oh, you're going to have to have a C-section because you're not doing this and you're always going to have to have a C-section.” My doctor said, “You were going to for sure rupture. I'm glad you didn't do it.” Right? Those things stick with us and they impact us, and they do impact the decisions that we make and the ways that we view things. This is totally not a sponsor of ours, but Sarah Pixton has a podcast called Birth Words. It is called Birth Words: Language For a Better Birth and just like you have been talking about, these words stick with us. Even though you, as a person, may be sharing something with someone based off of how you feel or whatever, remember that that might stick with them and impact them substantially in either a positive or a negative way. And so when we as people are speaking, we need to be aware of what we are saying, how we are saying it, who we are saying it to and be respectful of everybody's decisions, and choices, and views, and like you said, what they have been through. There are a lot of people with infertility stuff who don't even talk about it because they don't feel like they can.Jesse: Yeah. Everything pretty much about motherhood, I feel like, if you don't feel like you're going to be supported, you're not going to share, and then that creates a situation where you are not going to reach out to the right people and get the support that you need--Meagan: Right.Jesse: -- which creates more unsupported moms, which creates more chances of not getting the experience you wanted because you didn't have the knowledge, and the information, and the support that you needed to possibly have the situation that you wanted to happen actually happen for you. I feel like like you said with the words that he said to me, that pushed me into this prideful position of, “Okay. I am for sure having a VBAC next time.” You know? Like, “I'm going to show you I can have a VBAC.” And then once I got there, I was like, “I don't know.”Meagan: That's actually not what I want.Jesse: That's not actually-- I don't really want to try this again because of this, this, and this. Part of that was the mental space it put me in, the mental position that I put my husband in. It's just, and now we have another baby to take care of, so that was something else to take into consideration. So yeah. Your words carry weight, so be mindful of them even if you are a stranger.Meagan: Yeah. And remember there doesn't always have to be a “because”.Jesse: Yeah.Julie: Yes.Meagan: You don't always have to say, “Well, because this.“Julie: And you don't have to justify it.Meagan: “Well, because I want it because this or I don't want to because of this.” There doesn't have to be a “because”. Because is because. Do you know what I mean?Jesse: Right.Meagan: That is the reason and it's okay and you're confident in that. So yeah. I feel like a lot of people feel like they have to defend their reasons and it's like, no. You shouldn't have to defend your reason. If you're making a choice that is best for you, that's all that matters. That's all that matters.Jesse: Yeah. If you make a mom question her decisions, question her parenthood, question her ability or reasons as a mom, that's not support. That's not advice. It's just mean. It's just mean and it's unwanted, unsolicited, not advice, but just unsolicited speaking.Meagan: I love it.Jesse: Yeah. Moms need support.Maternal mental healthJulie: Yeah. It's important. Well and not only that, but I want to take it off on a little tangent. We have a serious maternal mental health crisis in our country, especially for women one to two years postpartum. When we create this environment where women are scared to share their feelings because they feel judged and they see these comments flying around on social media or wherever about which way is the right way to give birth or all those different things, it makes them more afraid to show their feelings. It also makes them feel like their feelings are wrong. When you are in a mental state where you feel guilty about your thoughts and you feel like you can't share them without being judged, then they are going to sit inside your head and your mind, and fester, and grow.It can lead to really long-term complications. I think we all know that mental health affects our physical health as well, and so we are having moms that are literally getting sick because they are not supported in their decisions. I am sure you have seen this too, Jesse. You have a big social media presence. Our social media presence is getting bigger all the time. I find the bigger that we get the more we get these people that seem like they just want to argue with whatever we say, or they take one little thing and pick it apart, right? I know I have texted Meagan a couple of times because I have anxiety over here. I'm working on it, but sometimes some things really upset me. I will text Meagan and I will be like, “What do you think about this?” Meagan calms me down and lets me know that there are 500 positive comments and one negative one.Jesse: Right.Julie: But I really still even get worked up about those types of things. It's something that I have to actively work on and I'm still learning coping strategies and stuff for, but my anxiety did not start until after I had my VBAC baby which is really interesting because I had what I would describe as a perfect, textbook VBAC. It was a perfect birth and everything was as I wanted it to be, and I had really severe postpartum depression and postpartum anxiety. It was my worst mental health after any of my pregnancies. I had this beautiful birth, but I had all of these expectations I set myself up as a mother based on what other people thought and what I perceived to be the right way to be a mom. And because I could not fit this mold that I had set for myself and because I felt like I didn't have a group of people I could talk to because all of my people were in this little mold, or so I thought, right? I kept it inside and it really did a lot of damage to my mental health. I know I'm not the only one that feels like that. That's why we share stories from all different types of people in all different types of births, in all different types of decisions, from all over the world even, because we want people to know that they are not alone. It's always okay to share your feelings. It is always okay to get help and it's never okay to judge somebody else based on their decision. It's never okay.Jesse: Yeah. I can totally relate to the comment stuff. I honestly can't even go to my comments. I've got a really strong group of moms here and, like you said, it's usually not even from people that are following you. They're not even coming to your page because they like your content.Julie: Oh yeah. They see one post.Jesse: Yeah. They're not coming because they like you. They're coming because you're triggering them. Because the people that I do see, I end up blocking them because I don't want anything negative that they say to affect a mom on my page.Julie: We do that too.Jesse: Or to read the comments or see it, so I will block them. But they are not even following me and I find it so intriguing, like, “Why are you here?”Julie: Yes.Jesse: Why are you here? What are you looking to get? It's usually because just they don't feel supported in their choices in their life, so they are attacking you for your choices or whatever. But yeah. The comment section is a dangerous place to be especially if the post has been up for a while. So every once in a while, I will go back and then I'm like, “Why did I do that?” Because you know? It's just toxic. But I think if I had one piece of advice for a new mom, it would be just to not read every book, not ask everybody what they did because moms have an intuition and they just need to feed into that a little bit more. Just trust your gut. If you have got a stomachache about something, it's your second brain trying to be like, “Your first brain is not listening to you. I am going to make you throw up over it and I'm telling you, something is wrong here. Something is not right.” If it doesn't sit well with you, the mom, you get to be the deciding factor and if somebody is giving you advice and it doesn't feel quite right, it's not good advice for you.Meagan: For you. Yes, exactly.Jesse: For you, yeah. And that's exactly it. Just because you're getting advice from your mom, just because you're getting advice from your best friend who also has a kid, you're probably also going to get advice from your friend that doesn't have a kid that's imagining how they would be as a parent-- which that always is great-- just because you're getting all that advice, it doesn't mean it is the right advice for you, and your situation, and your kids, and you as a mom. So my advice is to take what you need and ditch what you don't. If you like something that somebody is doing if you like something that your mom told you, or your friend, or your grandma, or whoever, take it. And if the other parts of it don't align with your vision of how you see yourself being a mom, or how you plan on birthing kids or all of the things, just ditch that. It is going to feel so much better if you have that confidence to just put your foot down and be like, “No. I am the mom. This is how it's going to be.” I feel like we need to support moms in those decisions also. Because for some reason, we feel like, “I don't want to be rude, so I'm just going to let this happen.” But it's either your feelings or their feelings and I don't think the mom should have to sacrifice her feelings around her kids, around her birth experience, around everything else because, at the end of the day, it's your experience. It's your journey. It looks a little bit different for everybody else.Julie: And if you see some content on social media that you don't agree with, just scroll along. It's okay.Jesse: You do not have to comment. You don't. I promise. I promise you don't have to leave your opinion.Julie: Well, we hate to cut it off short. I feel like this is something we could talk about forever. But I did want to say that sometimes it is a hard thing to decide whether to try for a vaginal birth after a Cesarean or to schedule a repeat Cesarean. Neither choice is a bad choice, but we did write a blog about how to decide if an elective C-section is best for you and your baby. Jesse, you said, “Don't read all the books,” but you can read this one blog. We are going to drop a link for that blog in our bio or you can just go to our website, to thevbaclink.com, and just search for “elective C-section” and it will be a link there. It will list medical reasons for repeat C-sections and also reasons why people might choose to have an elective Cesarean. There is no right answer for you and like Jesse said, there are a million ways to motherhood.Meagan: There's no wrong answer.Move with TrueloveJulie: There's no wrong answer. But before we go, we do want Jesse to share a little bit more about some of the stuff that she does and how you can find and follow her, because we know you love her by now just as much as we do.Jesse: Aw, thanks, guys.Meagan: Well, and I think that this information that you are going to share is so powerful because it really isn't paid attention to enough. So tell us all about what you have got your toes and hands dipped into.Jesse: Okay, perfect. Yeah, so I have got my own page. It's Move with Truelove. You guys got plugged into that at the beginning of this, but I am also very deeply connected to Nancy Anderson and if you are in the prenatal/postpartum field or realm or if you are a mom, you need to know what we are doing over here. So I am the Program Director and Head Coach for the Move Your Bump app and that is our prenatal and postnatal fitness and nutrition app. We've got over 400 on-demand coached follow-along workouts of every style, every fitness level. We have got multiple, multiple expert coaches that always have your bump in mind. We focus on minimizing excessive diastasis which is the ab separation, which became this really hot topic on social media, but there are about a thousand ways you can get it. We really focus on the prevention of that during pregnancy to help you have a more comfortable pregnancy, but more importantly, we are training you for labor because it is the hardest workout of your life. Whether you are a vaginal delivery mom or a C-section mom, it is going to be a fight and you want to make sure that you are training properly.We also prepare you for a faster recovery and we do that through the Move Your Bump app through workouts, through breathwork routines, through focusing on posture, troubleshooting issues that we see most commonly in prenatal mamas. We also have challenges throughout our app with private Facebook group communities, with thousands of other moms throughout the world that are connected through their sharing bump pics and checking in for progress, and then also the ability to win prizes there. So it is super fun.Meagan: And there's something starting on the third, right? You guys are doing a challenge starting on the third. Is that right?Jesse: Yes. Yeah. We have a bump challenge.Julie: The third of what?Meagan: It's the third of May.Jesse: Of May.Julie: Well, this episode isn't going to air until June.Meagan: But do you do those often?Jesse: We do. We do them every single month, the first Monday of every month. We do multiple challenges. We have a Before Your Bump challenge, which is our trying-to-conceive challenge and we have got all kinds of information on stress management, fertility, hormones, nutrition to optimize your fertility, as well as fitness. We want to make sure that we are staying nice and strong and focusing on the things that are most important to not only support a pregnancy but also recover from your delivery.We also do through the Birth Recovery Center, which is our umbrella company that is going to be having multiple courses including things like sleep support, lactation consultants, mental health, all kinds of stuff that moms are going to need through their journey. So the hero product there is the AB Rehab course. That's our 12-week postpartum recovery course. It helps you to recover and heal fully from your delivery whether you are a C-section mom or a vaginal mom, whether you delivered 10 weeks, 10 months, or 10 years ago. This program is basically for everybody with a pelvic floor, so that's everybody.We focus on the foundational work, on the reasons why diastasis happens in the first place. Pregnancy is definitely something to recover from as well as your delivery because so much is changing in such a short amount of time. Birth is basically a lot of trauma to the body which takes a lot of rehab and a lot of specific coaching, specific protocols to make sure that we are addressing everything that happens during pregnancy and during delivery. So we will focus on postural imbalances, posture habits that happen during pregnancy, and then even pre-pregnancy, probably a lot of these muscle imbalances you have had before you were even pregnant. Pregnancy just has a really good job of showing us where our weaknesses are because we are trying to support a big bump growing out in front of us and our balance changes. Pressure changes, so imbalance, and muscle weaknesses, and compensations tend to really present themselves in the forefront during pregnancy and then stick around postpartum if we are not actively working to correct them. So through the 12-week course, we focus on posture. We focus on breathing patterns which are huge. We focus on the pelvic floor, not only function but the ability to connect with the pelvic floor and lower abs, which we really don't have a lot of access to during pregnancy if you are not actively working to keep that connection. That's why a lot of moms are left with a lower belly pooch or feeling like you can never really turn on your lower abs. That's muscle dysfunction.We have thousands of moms. We probably have about 5-8000 moms join our course every single month, every single challenge. We coach all of them through video communication, through our course, through video assessments. We have a whole entire team of course specialists that help customize the 12-week course to specific needs because like we said, there are a million roads to motherhood, there are a million roads to recovery too and that can't be done with a one-size-fits-all course, which is why we are so passionate about the ability to customize this course with troubleshooting videos if you have a hypertonic or hypotonic pelvic floor or whether you have developed prolapse. There's a lot of things that are happening postpartum and we address a lot of them. We also have an in-house DPT on our staff who can work to coach you through some more difficult things that need a higher level of expertise. So we really are just helping to raise the standard of care for moms postpartum. Moms deserve all of the support and it really does take a village to do that, and so that is why we have got our hands in all the cookie jars with lactation, and sleep, and birth path, and mental health, and of course, our 12-week AB Rehab, as well as nutrition. We have a little bit of everything and we are working to be the top providers for that since we are already doing that with AB Rehab.Meagan: Love it.Jesse: Yeah. We are really proud of it.Meagan: Oh my gosh. So much goodness in there. So awesome. Oh, well thank you so much for sharing your story, and your wisdom, and, obviously, that awesome course, courses. So we really appreciate you. We love your face and we just can't wait to continue our relationship.Jesse: Right back at ya. Thank you so much for having me.Julie: Absolutely. Thank you.ClosingWould you like to be a guest on the podcast? Head over to thevbaclink.com/share and submit your story. For all things VBAC, including online and in-person VBAC classes, The VBAC Link blog, and Julie and Meagan's bios, head over to thevbaclink.com. Congratulations on starting your journey of learning and discovery with The VBAC Link.Advertising Inquiries: https://redcircle.com/brandsPrivacy & Opt-Out: https://redcircle.com/privacy
Transcript Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Hello, and welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild. With a healthy dose of complaining about AWS for good measure. Today, we’re going to be talking about build versus buy. I feel like this is really kind of a classic engineering conversation. Amy, what is the build versus buy idea?Amy: It’s really the idea of whether you decide to use a managed service or SaaS product versus rolling your own and building yourself. It’s very easy to do these days with a few watches on YouTube, maybe some blog articles. You can also do repairs on my house, which is why I always have to get repairs done on my house. [laugh].Jesse: [laugh]. Yeah, I feel like as much as I love the world of HGTV and the DIY network, I think I can do more than I actually can and I feel like it’s probably a lot safer to just let a professional take the reins. I mean, there’s so many certification programs that teach you how to build and manage your own engineering things, your own distributed databases, your own Kubernetes clusters, your own streaming data platform, and it’s really great to understand the fundamental building blocks of these systems, it’s really great to understand how they work so that ultimately if you are consuming from them or managing them, that you understand the ins and the outs of the system. But the question becomes, do you really need to be the one that’s managing that system? Do you really need to be the one spending your time managing that system on top of writing code for your microservices, on top of managing the architecture, the application, all of the components of your service that are critical?Amy: So, I guess what we really want to decide is, in what use cases is it okay to build something from scratch, and when is it okay to, essentially, just go to the market and look for something that’s made already?Jesse: Yeah. And I think that’s the main question that a lot of folks ask: what is the defining line? What are the questions they should think about as they are choosing to build versus buy?Amy: I think if you want to really look at building a product, and really from the ground up—you have this product in mind and you want to do all the architecture, control it end-to-end—unless this is your core product feature or you’re going to package it for either internal or public release, you almost always—you don’t want to build this yourself because someone has probably built it in a way that’s not going to cause your engineers time or money. Unless it is going to directly make you money, then yes. If this is tied to your product income and your product revenue, please build it yourself. It avoids a lot of licensing issues, you do get to control how it works, how you want it to work. But that said a lot of products, just a bunch of assassins in a trench coat anyway, so—Jesse: [laugh].Amy: —it really depends on what’s important to you.Jesse: Yeah, I feel like this is one of the biggest pitfalls that I see in a lot of organizations where they think about how they want to build out an architecture and they choose that a solution like, stateful distributed service is going to be the right thing that they want. And one of the developers says, “Oh, that’s easy. I can build that in a weekend.” And then they go off and build it, and then they’re stuck managing that system for all of eternity when that’s not the primary purpose of the team that they’re working on, that’s not the primary purpose of the product that they’re working on. So, if you’re going to build something that is directly related to your product, directly related to your business use case, directly related to how your company is making money, something that is absolutely your bread and butter, you definitely want to build that rather than buying that off the shelf.Because building it will give you that great opportunity to focus on controlling all the ins and outs of the system, understanding all the parts of the system, finding the flexibility when you need flexibility, really fine-tuning and honing all the parts of the system in the way that you need it to work so that ultimately your organization is getting the best bang for their buck out of the system, whereas in a lot of cases, you’re not going to get the same level of flexibility from an off the shelf solution.Amy: And especially if you’re going to go in and planning to build your own supporting product, make sure—and I said this before, I’ll say it again—you do check the licenses of any libraries and any SaaS products you use to build it because I reinvented the wheel plenty of times in my career, specifically because I worked in a place where the licensing we were allowed to use would not allow us to use very specific products.Jesse: Yeah. That’s such a critical business risk and something that I think not every engineer is fully aware of. And to be clear, I don’t think that’s the engineer’s fault. I think that’s part of best practices that every organization can get better at to make sure that everybody understands, what are our limitations on using modules, using open-source solutions from the internet? How can we make sure that we ultimately aren’t creating additional unnecessary business risk?Amy: When do we go shopping?Jesse: [laugh]. Yeah, let’s go shopping. Let’s say you’ve decided that the piece of software that you want is not part of your bread and butter, like we were saying. If it’s not part of your organization’s primary product, primary use case, don’t waste engineering time building it for yourself, pay a vendor or a subject matter expert to build it for you—or to manage it for you, even—and then call it a day. It is absolutely worth those trade-offs. The additional cost of paying somebody else to manage it for you is absolutely worthwhile because you then get the opportunity to stay focused on the things that are most important to your team and your business.Corey: If your mean time to WTF for a security alert is more than a minute, it’s time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you’re building a secure business on AWS with compliance requirements, you don’t really have time to choose between antivirus or firewall companies to help you secure your stack. That’s why Lacework is built from the ground up for the cloud: low effort, high visibility, and detection. To learn more, visit lacework.com.Amy: You also don’t end up trapped by having to make sure the product is appropriately upgraded or patched. And then you also have that nice little space of liability, saying we just bought this off the shelf. They said it was safe, and we trusted them. [laugh].Jesse: Yeah, again, business risk conversations, there is absolutely that opportunity for third-party liability rather than internal liability in some of those security risks. I also feel like it’s important to add that AWS, for example, has tons of managed services that give you ease of use by removing that administrative overhead. Yes, we’re primarily focused on AWS, obviously, this is an AWS focused podcast, there is definitely going to be a best and worst use case for these products so I’m not saying that you should go out and start using these all immediately without thinking about your overall goal and use case, but in a lot of cases, again, if the solution that you want is not something that you need to manage yourself, that you need to focus on building and running yourself, give it to AWS, they have tons of these managed solutions available to you built into the ecosystem.Amy: And that’s true of all the large cloud providers. They have managed services to make the things that you do not have the staffing to be an expert in and do all that work for you. And it’s not as if you are locked into these solutions. When you buy into either a SaaS product or a managed service, you can migrate off if you feel like you can build it better, and you actually have spent the time in R&D, and you spent the time building out a minimum viable product, and you know that this use case works for you, and then you can either clear out overhead or fees, and you can actually come in under what you’re spending right now, then make that move. But do it after you already know what it is you want.Jesse: Yeah, I think that’s a really great use case example, Amy. One other thing that I want to talk about is that this build versus buy a conversation so far has been focused on your organization thinking about if they want to build something internally, or if they want to buy it from a third-party vendor. But this conversation can also happen internally, in a single organization, between teams. I’m thinking about some organizations that I’ve worked for where I’ve seen one team build and manage a central platform solution, like a central CI/CD pipeline that every other team is going to be using and consuming from. But then, one team decides that the CI/CD platform that everybody’s using doesn’t really do all the things that they want it to do, so they decide to go off and build their own CI/CD platform internally for their team instead, rather than working with the team that is actually owning this sort of centralized CI/CD platform to make sure that everybody is getting the benefits of these additional features, these additional solutions, these additional bug fixes that the team was asking for.Amy: It’s really hard when you can’t see the forest for all the silos.Jesse: Yes. Absolutely. It is so, so critical to think about building these feedback loops into your internal tools. Because if your customers are internal to your organization, they’re going to want to provide that feedback in some capacity to help you understand when the service that you’re building is fantastic and when the service that you’re building is awful. And it’s so, so critical to make sure that you have those easy feedback loops so that you can continue to iterate on the things that you choose to build internally and hone them and make them better.If you’ve got questions that you’d like us to answer go to lastweekinaws.com/QA. If you’ve enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us the criteria you think about when considering whether you should build or buy.Announcer: This has been a HumblePod production. Stay humble.
TranscriptCorey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure because I mean, who doesn’t love to complain about AWS? I feel like that’s always a good thing that we can talk about, no matter the topic. Today, we’re going to be talking about the ‘cloud cost management starter kit.’ So, the starter kit seems to be a big fad that’s going around. If you’re listening to this episode, you’re probably thinking, “It’s already done. It’s over.”But I still want to talk about it. I think that this is a really relevant topic because I think a lot of companies are trying to get started, get their hands started in cloud cost management. So, I think this would be a great thing for us to talk about: what’s in our cloud cost management starter kit?Amy: And it really will help answer that question that I get asked a lot on: what is even a cloud economist, and what do you do?Jesse: Yeah, I mean, given the current timeframe, I haven’t gone to any parties recently to talk about what I do, but I do feel like anytime I try to explain to somebody what I do, there’s always that moment of, “Okay. Yes, I work with computers, and we’ll just leave it at that.”Amy: It’s easier to just think about it as we look at receipts, and we kind of figure things out. But when you try to get into the nuts and bolts of it, it’s a very esoteric idea that we’re trying to explain. And no, I don’t know why this is a real job. And yet it is.Jesse: This is one of the things that always fascinates me. I absolutely love the work that I do, and I definitely think that it is important work that needs to be done for any organization, to work on their cloud cost management best practices, but it also boggles my mind that AWS, Azure, GCP, haven’t figured out how to bake this in more clearly and easily to all of their workflows and all their services. It still boggles my mind that this is something that exists as—Amy: As a thing we have to do.Jesse: As a thing we have to do. Yeah, absolutely.Amy: Well, the good news is, they’re going to change their practices once every six weeks, and we’ll have a new thing to figure it out. [laugh].Jesse: [laugh]. So, let’s get started with the first item on our cloud cost management starter kit. This one is something that Amy is definitely passionate about; I am definitely passionate about, as well. Amy, what is it?Amy: Turn on your CUR. Turn on your CUr. If you don’t know what it is, just Google AWS CUR. Turn it on. It will save you a headache, and it will save anyone you bring in to help you [laugh] [unintelligible 00:02:59] a huge headache. And it keeps us from having to yell at people, even though that’s the thing that if you pay us to do it, we will totally do it for you.Jesse: If you take nothing away from this episode, go check out the AWS Cost and Usage Report—otherwise known as CUR—turn it on for your accounts, ideally enable it in Parquet format because that’s going to allow you to get all that sweet, sweet data in an optimized manner, living in your S3 bucket. It is a godsend. It gives you all the data from Cost Explorer, and then some. It allows you to do all sorts of really interesting business intelligence analytics on your billing data. It’s absolutely fantastic.Amy: It’s like getting all of those juicy infrastructure metrics, except getting that with a dollar sign attached to it so you know what you actually doing with that money.Jesse: Yeah, this definitely is, like, the first step towards doing any kind of showback models, or chargeback models, or even unit economics to figuring out where your spend is going. The Cost and Usage Report is going to be a huge first step in that direction.Amy: Now, the reason why we yell at people about this—or at least I do—is because AWS will only show you the data from the time that it is turned on. They do have it for historical periods, but if you enable it at a specific point, all of your reports are going to start there. So, if you’re looking to do forecasting, or you want to be able to know what your usage is going to be looking like from this point on, turn it on as early as possible.Jesse: Absolutely. If you are listening to this now and you don’t have the CUR enabled, definitely go pause this episode, enable it now, and come back and listen to the rest of the episode because the sooner you have the CUR enabled, the sooner you’ll be able to get those sweet, sweet metrics for all of your—Amy: And it’s free.Jesse: [laugh]. Yeah, that’s even the more important part. It’s free. There’s going to be a little bit of data storage costs if you send this data to S3, but overall, the amount of money that you spend on that storage is going to be optimized because you’re saving that CUR data in Parquet format. It’s absolutely worthwhile.All right, so number two; the second item on our cloud cost management starter kit, is getting to know your AWS account manager and account team. This one, I feel like a lot of people don’t actually know that they have an AWS account manager. But let me tell you now: if you have an AWS account, you have an AWS account manager. Even if they haven’t reached out to you before they do exist, you have access to them, and you should absolutely start building a rapport with them.Amy: Anytime you are paying for a support plan, you also have an account manager. This isn’t just true for AWS; I would be very surprised for any service that charged you for support but did not give you an account manager.Jesse: So, for those of you who aren’t familiar with your account manager, they are generally somebody who will be able to help you navigate some of the more complex parts of AWS, especially when you have any kind of questions about your bill or about technical things using AWS. They will help you navigate those resources and make sure that your questions are getting to the teams that can actually answer them, and then make sure that those questions are actually getting answered. They are the best champion for you within AWS.If you have more than a certain threshold of spend on AWS, if you’re paying for enterprise support, you likely also have a dedicated technical account manager as well, who will be basically your point person for any technical questions. They are a great resource for any technical questions, making sure that your technical questions are answered, making sure that any concerns that you have are addressed, and that they get to the right teams. They can give you some guidance on possibly how to set up new features, new architecture within AWS. They can give you some great, great guidance about the best ways to use AWS to accomplish whatever your use case is. So, in the cases where you’ve got a dedicated technical account manager as well, get to know them because again, they are going to be your champion. They are here to help you. Both your account manager and your technical account manager want to make sure that you are happy with AWS and continue to use AWS.Amy: And the thing to know about the account manager is, like, if you ever run into that situation where, oh, something was left on erroneously and we ended up with a spike, or this is how I was understanding the service to work and it didn’t work that way, and now I have some weird spend, but I turned it off immediately, if you ever want to get a refund or a credit or anything, these are the people to talk to; they’re the ones who are going to help you out.Jesse: Yeah, that’s a great point. It’s like, whenever you call into any kind of customer support center, if you treat the person who answers the phone with kindness, they are generally more likely to help you solve your problem, or generally more likely to go out of their way to help you solve your problem. Whereas if you just call in and yell at them, they have no interest in helping you. So—Amy: You’ll never see that refund.Jesse: Exactly. So, the more that you can create that rapport with your account manager—and your technical account manager if you have one—the better chances that they will fight for you internally to go above and beyond to make sure that you can get a refund if you accidentally left something running, or make sure that any billing issues are taken care of extremely fast because they ultimately have already built that rapport with you. They care about you and the way that you care about them and the way that you care about continuing to use AWS.Amy: There’s another note about the technical managers where if you are very open with them on what your architecture plans are—“We’re going to move into this type of EKS deployment. This is the kind of traffic we think we’re going to run, and we think it’s going to be shaped this way”—they’ll help you out and build that in most efficient way possible because they also don’t want the resources out there either being overutilized or just being run poorly. They’ll help you out in trying to figure out the best way of building that. They’ll also—if AWS launches a new program and you spent a lot of money on AWS, maybe there’s a preview program that they think will help you solve a very edge case kind of issue that you didn’t think you had before.Jesse: Absolutely.Amy: Yeah. So, it’s a great way to get these paths and get these relationships because it helps both parties out.Corey: This episode is sponsored in part by VM Ware. Because lets face it, the past year hasn’t been kind to our AWS bills or, honestly, any cloud bills. The pandemic had a bunch of impacts. It forced us to move workloads to the cloud sooner than we would otherwise. We saw strange patterns such as user traffic drops off but infrastructure spend doesn't. What do you do about it? Well, the CloudLive 2021 Virtual Conference is your chance to connect with people wrestling with the same type of thing. Be they practitioners, vendors in the space, leaders of thought—ahem, ahem. And get some behind the scenes look into the various ways different companies are handling this. Hosted by Cloudhealth by VM Ware on May 20th the CloudLive 2021 Conference will be 100% virtual and 100% free to attend. So you really have no excuses for missing out on this opportunity to deal with people who care about cloud bills. Vist cloudlive.com/corey to learn more and save your virtual seat today. Thats cloud l-i-v-e.com/corey c-o-r-e-y. Drop the “e,” we’re all in trouble. My thanks for VM Ware for sponsoring this ridiculous episode.Jesse: So, the third item on our cloud cost management starter kit is identifying all of your contracts. Now, I know you’re probably thinking, “Well, wait. I’ve just got my AWS bill, what else should I be thinking about?” There’s other contracts that you might have with AWS. Now, you as the engineer may not know this, but there may be other agreements that your company has entered into with AWS: you might have an enterprise discount program agreement, you might have a private pricing addendum agreement, you might have an acceleration program—migration program—agreement. There’s multiple different contracts that your company might have with AWS, and you definitely want to make sure that you know about all of them.Amy: If you’re ever in charge of an architecture, you’re going to want to know not just what your costs are at the end of the day, but also what they are before all your discounts because those discounts can maybe camouflage a heavy usage if you’re also getting that usage covered by refunds and discounts.Jesse: Absolutely, totally agreed. Yeah, it’s really, really important to understand, not just your net spend at the end of the day, but your actual usage spend. And that’s a big one that I think a lot of people don’t think about regularly and is definitely important to think about when you’re looking at cloud cost management best practices and understanding how much your architecture is actually costing you on a team-by-team or product-by-product basis.Amy: Also, make sure if you’re doing reservations that you know when those reservations and savings plans ent—Jesse: Yes.Amy: —because you don’t want to have to answer the question, “Why did all of your costs go up when you actually have made no changes in your infrastructure?”Jesse: Yeah. Half the battle here is knowing that these contracts and reservations exist; the other half of the battle is knowing when they expire so that you can start having proactive conversations with teams about their usage patterns to make sure that they’re actually fully utilizing the reservations, and fully utilizing these discounts, and that they’re going to continue utilizing those discounts, continue utilizing those reservations so that you could ultimately end up purchasing the right reservations going forward, or ultimately end up renegotiating at the correct discount amount or commitment amount so that you are getting the best discount for how much money you’re actually spending.So, the last item on our cloud cost management starter kit is thinking about the non-technical parts of projects. Amy, when you think about the non-technical parts of projects, what do you think about?Amy: Non-technical always makes you think of people and process. So, this would be the leadership making the decisions on what those cost initiatives are. Maybe they want to push this down to the team lead level: it would include that. Or maybe they want to push it down to the engineering level, or the individual contributor level. There are some companies that are small enough that an engineer can be completely cognizant and responsible for the spend that they make.Jesse: Yeah. I think that this is a really, really critical item to include in our starter kit because leadership needs to be bought into and back whatever work is being done, whatever cloud cost management work is being done. But also teams need to be empowered to make the changes that they want to make, make the changes that will ultimately provide those cloud cost management optimization opportunities and better cost visibility across teams. So, does everybody know what their teams are empowered to do, what their teams are capable of? Does everybody know what their teams are responsible for on the flip side? Do they ultimately know that they are responsible for managing their own spend, or do they think that the spend belongs to somebody else? Also, do they understand which resources are part of their budget or part of their spend?Amy: It’s the idea that ownership of—whether it’s a bill, whether it’s a resource—comes down to communication, and level setting. Do we know who owns this? Do we know who’s paying for it? Do they know the information in the same way? Is there someone who’s outside who can figure out this information for themselves? Just making sure that it’s done in a clear enough way that everyone knows what’s going on.Jesse: Absolutely. Well, that will do it for us this week. Those are our four main items for our cloud cost management starter kits. If you’ve got questions you’d like us to answer, please go to lastweekinaws.com/QA, fill out the fields and submit your questions.If you’ve enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us, what would you put in your ideal starter kit?Announcer: This has been a HumblePod production. Stay humble.
TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Negrette.Jesse: This is the podcast within a podcast where we talk about all the ways we’ve seen AWS used and abused in the wild, with a healthy dose of complaining about AWS for good measure because I mean, who doesn’t love to complain about AWS? I feel like that’s always a good thing that we can talk about, no matter the topic. Today, we’re going to be talking about the ‘cloud cost management starter kit.’ So, the starter kit seems to be a big fad that’s going around. If you’re listening to this episode, you’re probably thinking, “It’s already done. It’s over.”But I still want to talk about it. I think that this is a really relevant topic because I think a lot of companies are trying to get started, get their hands started in cloud cost management. So, I think this would be a great thing for us to talk about: what’s in our cloud cost management starter kit?Amy: And it really will help answer that question that I get asked a lot on: what is even a cloud economist, and what do you do?Jesse: Yeah, I mean, given the current timeframe, I haven’t gone to any parties recently to talk about what I do, but I do feel like anytime I try to explain to somebody what I do, there’s always that moment of, “Okay. Yes, I work with computers, and we’ll just leave it at that.”Amy: It’s easier to just think about it as we look at receipts, and we kind of figure things out. But when you try to get into the nuts and bolts of it, it’s a very esoteric idea that we’re trying to explain. And no, I don’t know why this is a real job. And yet it is.Jesse: This is one of the things that always fascinates me. I absolutely love the work that I do, and I definitely think that it is important work that needs to be done for any organization, to work on their cloud cost management best practices, but it also boggles my mind that AWS, Azure, GCP, haven’t figured out how to bake this in more clearly and easily to all of their workflows and all their services. It still boggles my mind that this is something that exists as—Amy: As a thing we have to do.Jesse: As a thing we have to do. Yeah, absolutely.Amy: Well, the good news is, they’re going to change their practices once every six weeks, and we’ll have a new thing to figure it out. [laugh].Jesse: [laugh]. So, let’s get started with the first item on our cloud cost management starter kit. This one is something that Amy is definitely passionate about; I am definitely passionate about, as well. Amy, what is it?Amy: Turn on your CUR. Turn on your CUr. If you don’t know what it is, just Google AWS CUR. Turn it on. It will save you a headache, and it will save anyone you bring in to help you [laugh] [unintelligible 00:02:59] a huge headache. And it keeps us from having to yell at people, even though that’s the thing that if you pay us to do it, we will totally do it for you.Jesse: If you take nothing away from this episode, go check out the AWS Cost and Usage Report—otherwise known as CUR—turn it on for your accounts, ideally enable it in Parquet format because that’s going to allow you to get all that sweet, sweet data in an optimized manner, living in your S3 bucket. It is a godsend. It gives you all the data from Cost Explorer, and then some. It allows you to do all sorts of really interesting business intelligence analytics on your billing data. It’s absolutely fantastic.Amy: It’s like getting all of those juicy infrastructure metrics, except getting that with a dollar sign attached to it so you know what you actually doing with that money.Jesse: Yeah, this definitely is, like, the first step towards doing any kind of showback models, or chargeback models, or even unit economics to figuring out where your spend is going. The Cost and Usage Report is going to be a huge first step in that direction.Amy: Now, the reason why we yell at people about this—or at least I do—is because AWS will only show you the data from the time that it is turned on. They do have it for historical periods, but if you enable it at a specific point, all of your reports are going to start there. So, if you’re looking to do forecasting, or you want to be able to know what your usage is going to be looking like from this point on, turn it on as early as possible.Jesse: Absolutely. If you are listening to this now and you don’t have the CUR enabled, definitely go pause this episode, enable it now, and come back and listen to the rest of the episode because the sooner you have the CUR enabled, the sooner you’ll be able to get those sweet, sweet metrics for all of your—Amy: And it’s free.Jesse: [laugh]. Yeah, that’s even the more important part. It’s free. There’s going to be a little bit of data storage costs if you send this data to S3, but overall, the amount of money that you spend on that storage is going to be optimized because you’re saving that CUR data in Parquet format. It’s absolutely worthwhile.All right, so number two; the second item on our cloud cost management starter kit, is getting to know your AWS account manager and account team. This one, I feel like a lot of people don’t actually know that they have an AWS account manager. But let me tell you now: if you have an AWS account, you have an AWS account manager. Even if they haven’t reached out to you before they do exist, you have access to them, and you should absolutely start building a rapport with them.Amy: Anytime you are paying for a support plan, you also have an account manager. This isn’t just true for AWS; I would be very surprised for any service that charged you for support but did not give you an account manager.Jesse: So, for those of you who aren’t familiar with your account manager, they are generally somebody who will be able to help you navigate some of the more complex parts of AWS, especially when you have any kind of questions about your bill or about technical things using AWS. They will help you navigate those resources and make sure that your questions are getting to the teams that can actually answer them, and then make sure that those questions are actually getting answered. They are the best champion for you within AWS.If you have more than a certain threshold of spend on AWS, if you’re paying for enterprise support, you likely also have a dedicated technical account manager as well, who will be basically your point person for any technical questions. They are a great resource for any technical questions, making sure that your technical questions are answered, making sure that any concerns that you have are addressed, and that they get to the right teams. They can give you some guidance on possibly how to set up new features, new architecture within AWS. They can give you some great, great guidance about the best ways to use AWS to accomplish whatever your use case is. So, in the cases where you’ve got a dedicated technical account manager as well, get to know them because again, they are going to be your champion. They are here to help you. Both your account manager and your technical account manager want to make sure that you are happy with AWS and continue to use AWS.Amy: And the thing to know about the account manager is, like, if you ever run into that situation where, oh, something was left on erroneously and we ended up with a spike, or this is how I was understanding the service to work and it didn’t work that way, and now I have some weird spend, but I turned it off immediately, if you ever want to get a refund or a credit or anything, these are the people to talk to; they’re the ones who are going to help you out.Jesse: Yeah, that’s a great point. It’s like, whenever you call into any kind of customer support center, if you treat the person who answers the phone with kindness, they are generally more likely to help you solve your problem, or generally more likely to go out of their way to help you solve your problem. Whereas if you just call in and yell at them, they have no interest in helping you. So—Amy: You’ll never see that refund.Jesse: Exactly. So, the more that you can create that rapport with your account manager—and your technical account manager if you have one—the better chances that they will fight for you internally to go above and beyond to make sure that you can get a refund if you accidentally left something running, or make sure that any billing issues are taken care of extremely fast because they ultimately have already built that rapport with you. They care about you and the way that you care about them and the way that you care about continuing to use AWS.Amy: There’s another note about the technical managers where if you are very open with them on what your architecture plans are—“We’re going to move into this type of EKS deployment. This is the kind of traffic we think we’re going to run, and we think it’s going to be shaped this way”—they’ll help you out and build that in most efficient way possible because they also don’t want the resources out there either being overutilized or just being run poorly. They’ll help you out in trying to figure out the best way of building that. They’ll also—if AWS launches a new program and you spent a lot of money on AWS, maybe there’s a preview program that they think will help you solve a very edge case kind of issue that you didn’t think you had before.Jesse: Absolutely.Amy: Yeah. So, it’s a great way to get these paths and get these relationships because it helps both parties out.Corey: This episode is sponsored in part by VM Ware. Because lets face it, the past year hasn’t been kind to our AWS bills or, honestly, any cloud bills. The pandemic had a bunch of impacts. It forced us to move workloads to the cloud sooner than we would otherwise. We saw strange patterns such as user traffic drops off but infrastructure spend doesn't. What do you do about it? Well, the CloudLive 2021 Virtual Conference is your chance to connect with people wrestling with the same type of thing. Be they practitioners, vendors in the space, leaders of thought—ahem, ahem. And get some behind the scenes look into the various ways different companies are handling this. Hosted by Cloudhealth by VM Ware on May 20th the CloudLive 2021 Conference will be 100% virtual and 100% free to attend. So you really have no excuses for missing out on this opportunity to deal with people who care about cloud bills. Vist cloudlive.com/corey to learn more and save your virtual seat today. Thats cloud l-i-v-e.com/corey c-o-r-e-y. Drop the “e,” we’re all in trouble. My thanks for VM Ware for sponsoring this ridiculous episode.Jesse: So, the third item on our cloud cost management starter kit is identifying all of your contracts. Now, I know you’re probably thinking, “Well, wait. I’ve just got my AWS bill, what else should I be thinking about?” There’s other contracts that you might have with AWS. Now, you as the engineer may not know this, but there may be other agreements that your company has entered into with AWS: you might have an enterprise discount program agreement, you might have a private pricing addendum agreement, you might have an acceleration program—migration program—agreement. There’s multiple different contracts that your company might have with AWS, and you definitely want to make sure that you know about all of them.Amy: If you’re ever in charge of an architecture, you’re going to want to know not just what your costs are at the end of the day, but also what they are before all your discounts because those discounts can maybe camouflage a heavy usage if you’re also getting that usage covered by refunds and discounts.Jesse: Absolutely, totally agreed. Yeah, it’s really, really important to understand, not just your net spend at the end of the day, but your actual usage spend. And that’s a big one that I think a lot of people don’t think about regularly and is definitely important to think about when you’re looking at cloud cost management best practices and understanding how much your architecture is actually costing you on a team-by-team or product-by-product basis.Amy: Also, make sure if you’re doing reservations that you know when those reservations and savings plans ent—Jesse: Yes.Amy: —because you don’t want to have to answer the question, “Why did all of your costs go up when you actually have made no changes in your infrastructure?”Jesse: Yeah. Half the battle here is knowing that these contracts and reservations exist; the other half of the battle is knowing when they expire so that you can start having proactive conversations with teams about their usage patterns to make sure that they’re actually fully utilizing the reservations, and fully utilizing these discounts, and that they’re going to continue utilizing those discounts, continue utilizing those reservations so that you could ultimately end up purchasing the right reservations going forward, or ultimately end up renegotiating at the correct discount amount or commitment amount so that you are getting the best discount for how much money you’re actually spending.So, the last item on our cloud cost management starter kit is thinking about the non-technical parts of projects. Amy, when you think about the non-technical parts of projects, what do you think about?Amy: Non-technical always makes you think of people and process. So, this would be the leadership making the decisions on what those cost initiatives are. Maybe they want to push this down to the team lead level: it would include that. Or maybe they want to push it down to the engineering level, or the individual contributor level. There are some companies that are small enough that an engineer can be completely cognizant and responsible for the spend that they make.Jesse: Yeah. I think that this is a really, really critical item to include in our starter kit because leadership needs to be bought into and back whatever work is being done, whatever cloud cost management work is being done. But also teams need to be empowered to make the changes that they want to make, make the changes that will ultimately provide those cloud cost management optimization opportunities and better cost visibility across teams. So, does everybody know what their teams are empowered to do, what their teams are capable of? Does everybody know what their teams are responsible for on the flip side? Do they ultimately know that they are responsible for managing their own spend, or do they think that the spend belongs to somebody else? Also, do they understand which resources are part of their budget or part of their spend?Amy: It’s the idea that ownership of—whether it’s a bill, whether it’s a resource—comes down to communication, and level setting. Do we know who owns this? Do we know who’s paying for it? Do they know the information in the same way? Is there someone who’s outside who can figure out this information for themselves? Just making sure that it’s done in a clear enough way that everyone knows what’s going on.Jesse: Absolutely. Well, that will do it for us this week. Those are our four main items for our cloud cost management starter kits. If you’ve got questions you’d like us to answer, please go to lastweekinaws.com/QA, fill out the fields and submit your questions.If you’ve enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us, what would you put in your ideal starter kit?Announcer: This has been a HumblePod production. Stay humble.
TranscriptCorey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.Jesse: Today, on a very special episode of AWS Morning Brief: Fridays From the Field, we say our goodbyes to Pete Cheslock.Amy: Oh, no. Did the ops bus finally get him?Jesse: No. Wait, what? What? No. No, he’s not—Amy: You know, the ops bus, the one that takes out all of the ops people, which is why you need data recovery plans.Jesse: [laugh]. I mean, I have plans for other reasons, but no. No, Pete, Pete’s not dead. He’s just—I mean, he’s dead to me, but he’s just not going to be here anymore.Amy: Only on the inside.Jesse: Welcome to AWS Morning Brief: Fridays From the Field. I’m Jesse DeRose.Amy: I’m Amy Arumbulo Negrette.Pete: I am Pete Cheslock. I’m here for one last, beautiful, glorious time.Jesse: I feel like this is going to be like Breakfast Club but in the data center server room.Pete: Yeah. A little bit. I think so. We will all sit cross-legged on the floor in a circle, share our thoughts and feelings. And maybe some sushi. There were sushi in that movie. And that was, like, really advanced back then in the ’80s.Jesse: Yeah, I like that. So Pete, you want to give us a little bit of background about why you will be moving on from this podcast?Pete: Moving on to a whole new world. Yes. Sadly, I am not dead. The ops bus did not get me, and I was not eaten by my smoker, my meat smoker.Jesse: [laugh]. Although at this point, it’s probably overdue.Pete: You know, the odds of all three of those are pretty high out, to be really perfectly honest, given this pandemic and everything else going on in this world.Amy: Isn’t that how it works? You eventually become the smoked meat.Pete: Yeah, yeah.Jesse: [laugh].Pete: All the time. You know, you are what you eat. And if you eat junk and whatnot—so I eat smoked meats, eventually, I’m just going to become, you know, smoked meats, I guess. But no, I am moving on from The Duckbill Group. Just bittersweet is the best word I can come up with. Very sad, but also very excited.I’m moving on to a new role at a new company that was just kind of an opportunity that I couldn’t pass up. And I’m really excited for something new, but really sad because I don’t get to work with two of my three favorite cloud economists, Jesse, and Amy. Yeah, Corey is one, too, and yes, it’s fun to work with him. But it’s also fun to rag on him a little bit as well.Jesse: I’m pretty sure you still have the opportunity to rag on him no matter where you go.Pete: Yeah, that’s true. I mean, we’re Twitter connected. So, I can just slide into his DMs as needed. Yeah.Amy: And really, what else is Twitter for—Pete: Exactly.Jesse: [laugh].Amy: —than roasting former coworkers and bosses?Pete: Yeah, I expect a constant stream of Twitter DMs every time you find something, some little fun nugget that I’ve left behind.Jesse: I feel like that’s appropriate. So today, Pete, I have two questions for you now that you will be moving on from Duckbill Group, moving on from this podcast, I want to know, looking back at your time here working with Duckbill Group, what did you learn? What are the things that surprised you, that you didn’t expect? And what would you say to somebody who wanted to start working in this space, maybe start a career in cloud economics on their own?Pete: Yeah, so this kind of feels like an exit interview a little bit.Jesse: [laugh]. And a very public exit interview at that. So, make sure that we bleep all the swear words.Pete: I think it’s in Duckbill fashion to do a public—a very public-facing exit interview, right? That is Duckbill in a nutshell.Jesse: I think the only thing more public is if Corey asks you to hold the exit interview on Twitter.Amy: Exactly.Pete: [laugh]. I mean, we might have to do that, now. I like that idea. Yeah, so I think those are great questions, and I love the opportunity to talk about it. Because Duckbill is a fantastic company, and coming into Duckbill last year was totally by luck.Not really—no, not—luck is maybe not the right word. But I had been doing some consulting on my own, and the pandemic and some other forces caused a bunch of my consulting work to dry up really quickly. And I was sitting at home and I’m like, “Wow, I should get a real job.” And I saw a tweet from Mike on Twitter that was like, “Oh, we’re growing The Duckbill Group.” And Mike and Corey and I have known each other for such a long time.We’ve always said it’d be great to work together at some point in the future, but it’s so hard [laugh] to do. You know, to kind of work with your friends, and timing, and circumstance, and schedule, and everything else. And so when I saw that, I was like, wow, like that might be a lot of fun working with that crew. And I’ve got a lot of experience in AWS and I’ve—my title at one of my previous companies was Captain COGS—for Cost Of Goods Sold—because I was so diligent with the Amazon bill. So, it’s kind of one of those things where I felt like I could be useful and helpful to the organization, and talking with Mike and Corey, it just made a ton of sense.And so, it was a lot of fun to come on board. So, but then once you’re kind of in, and you start doing this type of work—and you know, Amy and Jesse, you’ve both experienced this—I think no matter how much knowledge you have of Amazon, very, very quickly, you realize that you actually don’t know as much as you really think you did, right?Jesse: Yeah.Pete: Because it’s so—there’s just so much.Amy: And it changes once every five minutes.Pete: [laugh].Jesse: Oh, yeah.Amy: Literally if you—well, just keep an eye on that changelog, you can watch your day get ruined as time goes on.Jesse: [laugh].Pete: [laugh]. It’s—yeah, it’s a real-time day ruining. And that’s the new. It’s like Amazon Kinesis: It’s all real-time.Jesse: [laugh].Pete: Yeah, it’s so true. And I think the reason behind it is, you know, one of the first things I kind of realized is that when you are working inside of a business and you’re trying to understand, like, an Amazon service, you don’t necessarily go that deep because you’ve got a real job and other stuff to do. And when you’re finally, like—let’s say you’re in Cost Explorer; this is actually my favorite one because learning this took us a while. The documents aren’t very good. But in Cost Explorer, there’s a dropdown box that can show you your charges in different ways: unblended view, blended view, amortized view—if I’m saying that word really incorrectly—net-amortized view, net-unblended view. Like, what do all these mean?Most people just are like, unblended, move on with their lives. But at some point, you kind of need to know and answer that question, and then understand the impact, and all those things, and spending more hours than I care to count trying to correlate the bill and Cost Explorer to look the same. Something that simple, why is that so hard? You know, it’s things like that.Amy: Why is that so hard? I do not understand it. It is exhausting. [laugh].Jesse: It drives me absolutely crazy, and it’s something that in previous roles, you could just say, “Well, this isn’t my responsibility, so I’m not going to worry about it.” But now we’ve got clients who are asking us these questions because it is our responsibility and we do need to worry about it.Pete: Yeah, exactly. So, I think that’s just, kind of, one example. Now, there was a ton that I learned. I mean, just in how discounts might be applied when you look at charges in an account whether if you have an enterprise discount program, or private pricing in some way. I think one of my favorite ones—and this is actually something that catches a lot of people up—is especially in Cost Explorer, there’s kind of two ways that you can view a charge.So, let’s say you’re looking at S3, and you are trying to find your usage by the usage type. Like, I want to compare standard storage to maybe data transfer or something like that. And you go and group by usage type, and they’ll show you, “Hey, for your S3 for this month or day or whatever, you’ll have some spend associated storage and data transfer,” and you’re like, “That’s neat.” And then you say to yourself, “Now, I want to look at it by API.” And maybe you’ll see, wow, there’s a ton of spend associated with GETs or PUTs.And you’ll think that that is actually a request charge. And it’s totally not. It’s like, when you group by API, it’s the API that started the charge, not the charge itself. So, you could have a PUT that started the charge, but the charge itself is actually storage. It’s the little things like that, where you might glance at it in your account and go, “Oh, okay.” But then when you actually need to get down to the per penny on spend and share it with a client, you go even further down the rabbit hole.Jesse: Because why would all of the billing information across different sources be accurate?Amy: And also, why would things be named the same between the bill, and Cost Explorer, and the curve? Having those names be the same, that would just make it too easy, and just streamline the process too much, and be too logical. No, let’s work for it. We have to work for it. It’s a pillar of excellence; we have to work for it.Pete: [laugh]. Exactly. So yeah, I think it’s those types of things that you just start seeing the edge cases. But because of, kind of, the work we do, we keep going. We’re not just, “Oh, wow. Haha, silly Amazon.”But then we keep diving in deeper and deeper to figure out the why. And the reason for that really just comes down to the fact that we’ll need to communicate that in some effective way to the client to get them to understand it. And actually, that kind of leads me to the other thing that I think is probably the most important skill of being a cloud economist, of being in finops, is your ability to write long-form writing, being able to write clear, concise information explaining why the spend is what it is, explaining all of these edge cases, all these interesting parts of cloud cost management, and being able to write that down in such a way that anyone could read it; like a CFO could understand how the charges are happening, just like a head of engineering, who has maybe more impact to the spend.Jesse: Being able to communicate, the differences between different AWS services, between different billing modes, to different audiences is so critical to the work that we do because we’re ultimately going to be working with different people with different backgrounds at every single client that we work with. So, we need to be able to speak the language of different audiences.Amy: And it’s really different, how different C Suites, different departments, their goals are going to be different, too, because they have requirements that they have to fulfill. Finance is very concerned about the literal cost of things, while engineering is—they understand that their architecture comes at a price, and so long as they have the budget for it, they’re cool with it. And you just have to align what those goals are, and have that translate as like, into the document as, “They built it this way for this reason, which was fine at that stage. But as you grow, you need to make sure that it also fulfills these other external expectations.”Corey: Let’s be honest—the past year has been a nightmare for cloud financial management. The pandemic forced us to move workloads to the cloud sooner than anticipated, and we all know what that means—surprises on the cloud bill and headaches for anyone trying to figure out what caused them. The CloudLIVE 2021 virtual conference is your chance to connect with FinOps and cloud financial management practitioners and get a behind-the-scenes look into proven strategies that have helped organizations like yours adapt to the realities of the past year. Hosted by CloudHealth by VMware on May 20th, the CloudLIVE 2021 conference will be 100% virtual and 100% free to attend, so you have no excuses for missing out on this opportunity to connect with the cloud management community. Visit cloudlive.com/corey to learn more and save your virtual seat today. That’s cloud-l-i-v-e.com/corey to register.Pete: Yeah, that’s exactly right. I mean, it’s just—and can you imagine, you have some knowledge you want to share around something as complex as the Amazon bill. I mean we ask for a PDF of your bill when you start working with Duckbill Group. That could be hundreds of pages long, and you’re trying to distill that down into something that, really, anyone can understand. It’s a true superpower to be able to write long-form content like that really well.And I never used to like writing. I was never—never really enjoyed it that much and over the last year, that muscle that you’re working out, now, the ability to write many, many pages around this type of content, just it comes so much more easily. So, I think that’s another big aspect, right? The more you work on it, obviously the easier it gets.Jesse: I don’t know about you, but now that I have focused more on flexing that writing and communication muscle, I’ve noticed it more in both everyone that I work with day-to-day with Duckbill Group and also in my daily life, just watching how people communicate with each other, and how effectively people communicate with each other; it’s both amazing and nerve-wracking all at the same time.Pete: [laugh]. I know. And even—not to say that whenever we sit down to write our reports that we give to our clients, we don’t go through the wave of emotions between the back and forth of, like, “I don’t know what to write,” and then, “Oh, I know of a lot of stuff to write. Let me just get something down.” And then you can’t stop writing. It’s just—it’s this emotional roller coaster that I feel like no matter how many times we need to write a lot of detailed information down, everyone always goes through.Amy: And we really do have a highly collaborative process here, too, where we’re all in the same document, writing, and the person who owns any given report will always have the same stage at the end when all of the sections are filled out, where they go to one of the other people on the team and go, “Every word I put down is absolute garbage. Please help me trim it down, take it out. I don’t even care anymore. Just look at it and tell me that I wrote down words that are in some kind of human language.” [laugh].Jesse: [laugh].Pete: [laugh]. Oh, the plight of the writer. It’s, like, the imposter syndrome that affects the writer. It’s like, “Okay. I wrote a bunch of stuff. I think it’s terrible.” And then you sleep on it, you come back the next day, and you’re like, “Actually, this is pretty good.” [laugh].Amy: I explained concepts. It was fine. I didn’t use a single comma for three pages, but it’s probably fine. [laugh].Jesse: [laugh].Pete: You can take one of mine. Usually, all of my draft documents are commas and M-dashes, just all over the place. Yeah, so I think that’s honestly a big superpower. And I think the last two things that—this is actually something that I’ve looked for in people that I’ve wanted to work with, and people I was hiring, and I see it here as well as these, kind of, two concepts of intellectual curiosity and aptitude to learn, where if you have a base knowledge around Amazon and you have those other attributes—that curiosity and truly enjoying learning—you can accelerate your ability to understand this so incredibly quickly because there’s such a wealth of information out there, and there’s so many documents, there’s so much stuff. It just requires someone who really cares enough to dive in and really want to understand.That’s something that I think we’ve seen here is that the folks who are most successful are just—they want to know why, and they’re not satisfied until they can explain it in a simple way to someone else. That’s the key, right? The attribute of a true expert is someone who can explain something very difficult in a simple way. And I think that’s something that would be critical if you were joining Duckbill, if you were building your own finops or cloud finance team, it is so complex. It’s the intersection of technical architecture and cost, and it touches almost the entire business. So, I think those are some other attributes that I think are just incredibly helpful.Jesse: We’re also usually not entirely satisfied until we’ve either opened a support case with AWS, responded to one of their feedback icons in the AWS documentation—the public AWS documentation—or trolled somebody on Twitter saying, “Shame on you, AWS, for writing documentation that doesn’t make sense.”Amy: It’ll be fine. Someone in your mentions will go, “Did you check the region?” And you would have, and then it’ll still be wrong.Jesse: [laugh].Amy: And it’ll be fine. [laugh]. Eventually, we’ll fix it.Pete: That one—Jesse: Too soon.Pete: —that one still hurts, when we—oh, I’m just like, “Why do the numbers not line up?” And then someone was like—Amy: It's a thing I check for, even if it’s like, “It’s a global resource.” I don’t care. Just tell me. Just tell me it’s fine. [laugh].Pete: “Are you in the right region?” Like—“Dammit, no, I’m not. Oh.” [laugh]. Yeah, that happens to the best of us.Amy: I did, unfortunately, burn so many hours, I think it was last week trying to find out where someone had put their resources. It’s like, “Oh, not us-west-2. It’s us-west-1. Of course.” [laugh].Jesse: So, annoying. Well, I would just like to say, Pete, it has been a joy and a pleasure working with you, it has been a joy and a pleasure complaining about AWS with you, on this podcast, so thank you for your time. That sounded really… really, really standoffish. I didn’t mean it quite as bad as it came off there. [laugh].Pete: Well, you know, I think we need to thank Corey for having a child and thus needing to offload some of his podcast duties over to us, and then the fact that we just never gave him the podcast back, and we just took it over.Jesse: Well, if you’ve got questions that you’d like us to answer, you can go to lastweekinaws.com/QA. And if you’ve enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review, give it a five-star rating on your podcast platform of choice and tell us what qualities you’re looking for when building out your cloud finance team.Pete: Thanks for coming in.Announcer: This has been a HumblePod production. Stay humble.
Links: Cloud FinOps: https://www.amazon.com/Cloud-FinOps-Collaborative-Real-Time-Management/dp/1492054623 FinOps Foundation: https://www.Finops.org/ AWS cost management blog: https://aws.amazon.com/blogs/aws-cost-management/ Mastering AWS Cost Optimization: https://www.amazon.com/Mastering-AWS-Cost-Optimization-operational/dp/965572803X TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I am Pete Cheslock.Jesse: I’m Jesse DeRose.Pete: Wow, we’re back again. And guess what? We have even more questions. I am… I am… I don’t even know. I have so many emotions right now that are conflicting between a pandemic and non-pandemic that I just—I’m just so happy. I’m just so happy that you listen, all of you out there, all you wonderful humans out there are listening. But more importantly, you are going into lastweekinaws.com/QA and you’re sending us some really great questions.Jesse: Yeah.Pete: And we’re going to answer some more questions today. We’re having so much fun with this, that we’re just going to keep the good times rolling. So, if you also want to keep these good times rolling, send us your questions, and we’ll just—yeah, we’ll just roll with it. Right, Jesse?Jesse: Absolutely. We’re happy to answer more questions on air, happy to let you pick our brains.Pete: All right. Well, we got a couple more questions. Let’s kick it off, Jesse.Jesse: Yeah. So, the first question today is from Barry. Thank you, Barry. “New friend of the pod here.” Always happy to have friends of the pod. Although I do feel like that starts to get, like, Children of the Corn, kind of. I think we started that, and I also am excited about it, and also upset with myself for starting that.Pete: That’s all right. Friend of the pod. Friend of the pod.Jesse: “New friend of the pod here. I work in strategic sourcing and procurement and I was curious if there are any ways that you recommend to get up to speed with managing cloud spend. This is usually closely monitored by finance or different groups in product, but I can see a significant potential value for a sourcing professional to help, also.” And that’s from Barry, thank you, Barry.Pete: Well, I’m struggling not to laugh. “This is usually closely monitored by finance or different groups in product.”Jesse: Yeah…Pete: But I mean, let’s be honest, it’s not monitored by anyone. It’s just running up a meter in a taxi going 100 miles an hour.Jesse: Yeah, that’s the hardest part. I want everybody to be involved in the cloud cost management practice, but there’s that same idea of if it’s everyone’s responsibility, it’s no one’s responsibility. And so this usually ends up at a point where you’ve got the CFO walking over to the head of engineering saying, “Why did the spend go up?” And that’s never a good conversation to have.Pete: No, never a good one. Well, Barry because you’re a friend of the pod, we will answer this question for you. And honestly, I think it’s a great question, which is, we actually have been working with a lot of larger enterprises and these enterprises still have their classic sourcing and procurement teams. That’s not an expertise that is going away anytime soon, but like most teams within the company that are adopting cloud, it’s obviously going to evolve as people are moving away from, kind of, capital intensive purchases and into, honestly, more complex, multi-year OpEx style purchases, with cloud services and all the different vendors that come with it. It’s going to just get a lot harder.I mean, it’s probably already a lot harder for those types of teams. And so there’s a bunch of places I think that you can go that can help level up your skills around cloud spend. And I would say the first place that I personally got to dive in a little bit more—I mean, my history has been using Amazon cloud and being a person who cared about how much my company spent on it, but when you—joining Duckbill, you need to dive into other areas around the FinOps world. And the book, the O’Reilly book, Cloud FinOps is actually a really great resource.Yeah, I think it’s really well written and there’s a lot of great chapters within there that you can kind of pick and choose based on what you’re most interested in learning about. If you’re trying to learn more about unit economics, or you’re trying to learn more about how to monitor and track things like that, it’s a great book to dive into, and becomes a really great reference that you can leverage as you’re trying to level up this expertise within yourself or your team.Jesse: It’s a really, really great resource. The other thing to think about is any kind of collaborative social spaces where you can be with like-minded individuals who also care about cloud costs. Now, there’s a number of meetups that exist under the FinOps title that may be worth looking into. Obviously, we’re recording this during the pandemic so I don’t recommend doing those in person. But as you are able to, there may be opportunities for in-person meetups and smaller local groups focusing on cloud cost management strategies together. But also check out the FinOps Foundation. They have a Slack space that I would love to tell you more about, but unfortunately, we’re not allowed to join. So—Pete: Yep.Jesse: —I can’t really say more about it than that. I would hope that you’re allowed to join, but they have some strict guidelines. So, I mean, the worst that can happen is they say no; it’s definitely worth signing up.Pete: Yeah, and they have to us. [laugh].Jesse: Yeah.Pete: I think when you get into the FinOps Foundation, you should angrily say that we should have more FinOps experts in here like the great Jesse DeRose should be a member of this one because right now, he’s just framed his rejection notice from there, and—Jesse: Oh, yeah.Pete: —while it looks beautiful on the wall, while I’m on a Zoom with him, I want more for you, Jesse.Jesse: I want more for me, too. I’m not going to lie.Pete: So, I don’t know this might sound a little ridiculous that I’m going to say something nice about AWS, but they have a fantastic cost management blog. This is a really fantastic resource, really incredible resource, with a lot more content more recently. They seem to be doing some great work on the recruiting side and bringing on some real fantastic experts around cost management.I mean, just recently within the past few months they talk about unit economics: How to select a unit metric that might support your business, talking more about unit metrics in practice. They start at the basics, too. I mean, obviously, we deal a lot in unit economics and unit metrics; they will start you off with something very basic and say, “Well, what even is this thing?” And talk to you more about cost reporting using AWS organizations for some of this. It’s a really fantastic resource.It’s all free, too, which is—it’s weird to say that something from AWS is free. So, anytime that you can find a free resource from Amazon, I say, highly recommend it. But there are a lot of blogs on the AWS site, but again, the Cost Management Blog, great resource. I read it religiously; I think what they’re writing is some of, really, the best content on the blog in general.Jesse: There’s one other book that I want to recommend called Mastering AWS Cost Optimization and we’ll throw links to all these in the [show notes 00:07:30], but I, unfortunately, have not read this book yet, so I can’t give strong recommendations for it, but it is very similar in style and vein to the Cloud FinOps book that we just mentioned, so might be another great resource to pick up to give you some spot learning of different components of the cloud cost management workflow and style.Pete: Awesome. Yeah, definitely agree. I’d love to see, again, more content out here. There’s a lot of stuff that exists. And even A Cloud Guru has come up with cost management training sessions.Again, we’d like to see more and more of this. I’d love to see more of this come from Amazon. I’d love to see—you know, they have a certification path in all these different areas; I’d love to see more of that in the cost management world because I think it’s going to become more complex, and having that knowledge, there is so much knowledge, it’s spread so far across AWS, helping more people get up to speed on it will be just critical for businesses who want to better understand what their spend is doing. So, really great question, Barry, friend of the pod. We should get some pins for that, right? Friend of the pod pins?Jesse: Oh, yeah.Pete: And yeah, really great question. Really appreciate you sending it and hopefully that helps you. And if not, guess what? You can go to lastweekinaws.com/QA, and just ask us a follow-up question, Barry. Because you’re a friend of the pod. So, we’ll hopefully hear from you again soon.Jesse: Thanks, Barry.Pete: Thanks.Announcer: If your mean time to WTF for a security alert is more than a minute, it’s time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you’re building a secure business on AWS with compliance requirements, you don’t really have time to choose between antivirus or firewall companies to help you secure your stack. That’s why Lacework is built from the ground up for the cloud: Low effort, high visibility, and detection. To learn more, visit lacework.com.Pete: All right, we have one more question. Jesse, what is it?Jesse: “All right, most tech execs I speak with have already chosen a destination hyperscaler of choice. They ask me to take them there. I can either print out a map they can follow, procedural style, or I can be their Uber driver. I could be declarative. I prefer the latter for flexibility reasons, but having said that, where does one actually start?Do you start with Infrastructure as a Service and some RDS to rid them of that pesky expensive Oracle bill? Do we start with a greenfield? I mean, having a massive legacy footprint, it takes a while to move things over, and integrating becomes a costly affair. There’s definitely a chicken and egg scenario here. How do I ultimately find the best path forward?” That question is from Marsellus Wallace? Thank you, Marsellus.Pete: Great question. And I’m not just saying that. I guess I have a question. Or at least, maybe we have different answers based on what this really looks like. Is this a legacy data center migration?The solution here is basically lift-and-shift. Do it quickly. And most importantly, don’t forget to refactor and clean up after you shut down your old data center. Don’t leave old technical debt behind. And, yeah, you’re going to spend a lot, you’re going to look at your bill and go, “Holy hell, what just happened here?”But it’s not going to stay that way. That’s probably—if you do it right—the highest your bill is going to be because lift-and-shift means basically just moving compute from one location to another. And if you’re—as we spoken about probably a million times, Jesse and I, if you just run everything on EC2 like a data center, it’s the most expensive way to do the cloud stuff. So, you’re going to then refactor and bring in ephemerality and tiering of data and all those fun things that we talk about. Now, is this a hybrid cloud world?That’s a little bit different because that means you’re not technically going to get rid of, maybe, physical locations or physical data centers, so where do you start? It’s my personal opinion—and Jesse has his own opinion, too, and guess what it’s our podcast and we’re going to tell it like it is.Jesse: [laugh].Pete: [laugh]. You know, my belief is, starting with storage is honestly a great way to get into cloud. Specifically S3. Maybe even your corporate file systems, using a tool like FSX. It’s honestly why many businesses start their cloud journey, by moving corporate email and file systems into the cloud.I mean, as a former Microsoft Exchange administrator, I am thoroughly happy that you don’t have to manage that, really, anymore and you can push that in the cloud. So, I think storage is honestly a great way to get started within there: Get S3 going, move your file systems in there, move your email in there if you haven’t yet. That’s a really great way to do it. Now, the next one that I would move probably just as aggressively into and, Marsellus, you mentioned it: RDS, right? “Should we move into RDS, get rid of expensive Oracle bills?”Yeah, anytime you can pay ol’ Uncle Larry less money is better in my mindset. Databases are, again, another really great way of getting into AWS. They work so well, RDS is just such a great service, but don’t forget about DMS, the database migration service. This is the most underrated cloud service that Amazon has in there, it will help you migrate your workloads into RDS, into Amazon Aurora. But one thing I do want to call out before you start migrating data in there, talk to your account manager—you have one even if you don’t think you have one—before starting anything, and have them help you identify if there are any current programs that exist to help you migrate that data in.Again, Amazon will incentivize you to do it, they will provide you credits, like map credits or other investment credits, maybe even professional services that can help you migrate this data from an on-premise Oracle into AWS, I think you will be very pleasantly surprised with how aggressive that they can be to help you get into there. The last thing that I would say is another great thing to move in our data projects. So, let’s say you want to do a greenfield one, greenfield type of project into Amazon, data projects are a really great way to move in there. I’m talking things like EMR, Databricks, Qubole, you get to take advantage of Spot Fleets with EMR, but also Databricks and Qubole can manage Spot infrastructure and really take advantage of cloud ephemerality. So if, like I said, you started by pushing all your data into S3, you’re already halfway there on a really solid data engineering project, and now you get to leverage a lot of these other ancillary services like Glue, Glue DataBrew, Athena, Redshift.I mean, once the data is in S3, you have a lot of flexibility. So, that’s my personal opinion on where to get started there. But Jesse, I know you always have a different take on these, so where do you think that they should start?Jesse: Yeah, I think all of the recommendations you just made are really, really great options. I always like to look at this from the perspective of the theory side or the strategy side. What ultimately do these tech execs want to accomplish? Is it getting out of data centers? Is it better cost visibility?Is it optimizing spend? Is it better opportunity to move fast, get new R&D things that you can’t get in a data center? What do these tech execs ultimately want to accomplish? And ask them. Start by asking them.Prioritize the work that they want to accomplish first, and work with teams to change their behaviors to accomplish their goals. One of the biggest themes that we see in the space moving from data centers into cloud providers or even just growing within a given cloud provider is cost visibility. Do teams know why their spend is what it is? Do they know why it went up or down month-over-month? Can they tell you the influences and the drivers that cause their spend to go up or down?Can they specifically call out which teams or product usage increased or decreased, and what ultimately led to your spending changing? Make sure that every team has an architecture diagram and they can explain how they use AWS, how data moves from one service to another, both within their product and to other products. Because there’s definitely going to be sharp edges with data transfer between accounts. We’ve seen this happen to a number of clients before; I’ve gotten bit by this bullet. So, talk to your teams, or talk to your tech executives and have those tech executives talk to their teams to understand what do they ultimately want to accomplish?Can they tie all of what they’re trying to accomplish back to business metrics? Maybe a spike in user logins generated more usage? If you’re a photo storage company, did a world event prompt a lot of users to upload photos prompting higher storage costs? Are you able to pull out these specific insights? That’s ultimately the big question here. Can you boil it down to a business KPI that changed, that ultimately impacted your AWS spend?Pete: I think this is a scenario of where you get started. Why not both? Just maybe do both of these things that we’re saying.Jesse: Yeah.Pete: And honestly, I think you’ll end up in a pretty great place. So, let us know how that works out, Marsellus, and thank you for the question. Again, you also can send us your questions, and we will maybe answer these on a future episode; lastweekinaws.com/QA, drop a question in there, put your name, or not or a fake name, or even a joke. That’s fine, too. I don’t know what the text limit is on the name, Jesse. Can you put a joke there? I don’t know. You know what? Test that out for us. It’s not slash QA for nothing. So, give that a little QA, or a question and answer and [unintelligible 00:17:29]. All right. Well, thanks, Jesse, for helping me out answering more questions.Jesse: Thanks, everybody for the awesome questions.Pete: If you enjoyed this podcast, please go to lastweekinaws.com/review, give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review and give it a five-star rating on your podcast platform of choice and tell us, what would be the last thing that you would move to AWS? It’s QuickSight, isn’t it?Jesse: [laugh].Pete: Thanks, everyone. Bye-bye.Announcer: This has been a HumblePod production. Stay humble.
Links:Unconventional Guide: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/ TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. We’re back again, my name is Pete Cheslock.Jesse: I’m Jesse DeRose. So, happy to be back in the studio after our whirlwind tour of the Unconventional Guide that I feel like we’ve been on for roughly as long as the pandemic’s been going on at this point; probably a little bit less. But lots of really great content there that we were happy to talk about, and I’m happy to be moving on to some other topics.Pete: Yeah, absolutely. And the topics, we actually get to move on to some of our favorite topics, which are answering your questions. And it turns out, Jesse, there’s more than two people that listen to us. There’s a lot of you; there are dozens of you out there, and we love it.Jesse: You like me. You really like me.Pete: So, great. So, great to see. We’ve been getting tons of fantastic questions, a few of which we’re going to answer right now. You can also have your question answered by going over to the lastweekinaws.com/QA and enter in your question there. You can enter in your name, or you can leave it blank, or you could just put something funny there. Anything works. We’re happy to dive in deeper on any particular topic, again, whether it’s about this recent Unconventional Guide series or just something you’re curious about in your day-to-day in your cost management life.Jesse: Today’s questions are really great because they ultimately get at the practical side of all of our recommendations. Because I feel like every single time I subscribe to one of those self-help books or blogs and I read all these really great short, sweet tidbits, I think to myself, “This is perfect. I’ll go apply this to everything in my life.” But then doing the actual work part is so much harder. Where do you even start with that first step once you’ve got the big picture grand idea? So, today we’ve got some really, really great questions, focusing on the best ways to get started on your cloud cost management journey. So, let’s start off with these questions.First question is, “Could you cover some practical approaches to applying some of your Cost Management Guide? A lot of your suggestions sound simple on paper, but in practice, they become quite complicated.” So, true. Absolutely, absolutely a concern. “I’ve had some success pulling in a small group of subject matter experts together for short periods of time focusing on low risk, easy things to do. How have you approached actually doing this? What meetings do you set up? What do you take for notes? How do you document your savings? How do you find new opportunities?” That’s from Brian O. Brian O., That’s a really, really great question.The other one that I want to add to this: “We’re a big AWS shop, and I’ve spent some time inside the AWS beast in the past, and I still struggle with multi-account multi-region data transfer in general, but specifically analyzing cost and usage. There are examples specifically like if data transfer out goes up $25,000 last month, how do you attribute that? How do you know where to apply that? How do you know what ultimately prompted that spend? Love how you work through these types of challenges. What is relatively easy at a single account level gets exponentially more complex with every account and region we function in.” So, true. And that’s from Todd. Thank you, Todd. In both cases, absolutely true.There’s this really great idea of we can give you the really short and sweet things to think about, but taking those first steps for practically applying these ideas is tough, and it needs to scale over time. And not every practice does.Pete: Yeah, these are great questions. I, kind of, am remembering that meme that was around for a while, which was, how to draw an owl. “First, draw two circles, and then, you know, you draw the rest of the owl.”Jesse: Yeah.Pete: And honestly, oftentimes, some of the stuff even that we say, Jesse, feels that way, and it doesn’t intend to come across that way. It’s just, we could bore you all on a multi-hour long recording of some of these topics. I mean, we do this with our clients, and our clients pay for this pleasure [laugh] for us to put them to sleep with our soft tones of the cloud cost management world. But I think the reality is that it is complex and there are probably unlikely to be quick wins in a lot of these places. One thing that we found is honestly, monitoring, visibility, I think all the cool kids are calling it observability now—Jesse: [laugh].Pete: —you know, I can’t believe I’m going to say this, but CloudWatch is actually probably one of the best cloud cost reduction tools that exist out there. There are so many services within AWS that you’re probably using today, that by default, report data to CloudWatch. And those statistics are potentially a huge place to identify resources that are over-provisioned and underused, idle resources, things like that. I can’t tell you how many times that I will go into a client account, and one of the first places I go to is—after Cost Explorer—is probably CloudWatch. So, monitoring spend and monitoring what’s happening there is kind of a great way to get started on that cloud cost idea because you’re getting charged for everything that happens, so knowing what’s happening, and knowing how it’s changing over time is a great way to start understanding and reducing it.Jesse: Yeah. And I think AWS is probably also using some of those CloudWatch metrics in their optimization recommendations that they make within their own optimization tooling. And it’s probably just not clearly defined or clearly outlined for AWS customers to be able to use the same metrics. So, I feel like if my Compute Optimizer could quickly load or link to a graph that showed me low CPU utilization across a number of instances, that’s a really handy way for me to start using more of CloudWatch’s metrics.Pete: Yeah, I think Compute Optimizer is honestly, criminally underused out there. I don’t know why. Then honestly, one of the other complaints is like, “Well, you can’t get memory statistics unless you have a CloudWatch Agent.” Yes. So honestly, install the CloudWatch agent; have it report up, the, like, one or two memory metrics that Compute Optimizer needs to make a recommendation and the cost will more than pay for itself.And now you can even output those statistics to S3 and do some fun programmatic stuff with it. Put those outputs in front of the engineers that own those resources and be like, “Hey, yo. This thing says, change your i3.24xl. Could you move it to something a little bit more useful, like a t3.small?”Jesse: And these are just some practical applications for some of the specific metrics we’re talking about, but this is a practice that you might want to turn into a process, you might want to turn into an ongoing amount of work. And in a lot of cases, we’ve seen this start as one engineer who’s really interested in understanding AWS, really passionate about the bill, maybe isn’t in a leadership or management role so maybe they don’t have a direct business requirement to optimize their spend, but they’re really, really interested in this work and they grow into a role where they are taking on more and more of this work. And that’s not scalable; that engineer is going to get burned out very, very quickly if they have a day-to-day role that is focused in development and doing all of this optimization work, cost cloud management work on the side. We generally recommend at least one dedicated individual who starts building these dashboards, starts looking at some of these metrics, starts these conversations with teams, and ultimately grow that into a full team.Pete: Right. I think that’s the biggest thing that we’re seeing in the industry is actual cloud finance teams coming into existence at companies. It’s such a critical role and it’s sad to see when people are like, “Arg, spend is out of control. We’re doubling year over year on spend and no one really seems to know why.” And honestly, it’s because no one cares about it. You don’t have any ownership on it. And, you know, we see it a lot, right? It’s like, “Well, everyone owns the Amazon bill.” That’s code for, “No one owns the Amazon bill.”Jesse: Yeah.Pete: But these cloud finance teams, and even the term cloud economist, as silly as it is, it’s centered in reality, which is we create financial models to understand spend and we dive into those numbers to make the usage makes sense to folks like CFOs inside of companies. Yeah, there’s a couple of ways that we have seen some of this done at scale. In one case using kind of active monitoring, and actively monitoring the spend based on really granular budgets, and reporting it as such. So, maybe you’re breaking these budgets out to be product-specific, or team specific, or business unit, or things like that, and then basically reaching out to these engineering teams. Because you are actively monitoring the spend on a recurring basis, you can reach out to those teams, when their spend goes over a given threshold that you’ve put in place, or when you, maybe, find some optimization opportunities.You’re probably thinking to yourself, “Wow, I don’t have the time for that.” Yeah, but you need to create the time or you need to create the team for this. The companies that we work with who have a dedicated team around this are the ones that do the best. In some cases, we’ve seen having a Dedicated Cloud finance team causes the bill to actually decrease over time, which, you’re thinking to yourself, “Wow. An Amazon bill that goes down? We so rarely see that.”Even for us, our clients come to us, we help them find optimizations. They’ll make those optimizations, but then they replace that spend with other investments. Usually, it’s new projects and new spend. But actually seeing the bill go down because of a dedicated effort of a team is still, again, amazing to see. The other side is we’ve seen maybe more of a passive monitoring or something around the background of things where you have a cloud platform team that provides abstractions and guardrails to the user.So, you’re not really trying to actively stand in the way of users and what they’re able to do and reaching out to them in an ongoing way, but you’re abstracting away the kind of complexity of the cloud and letting them basically live in a safe space that you are controlling for them. And that’s another way that you can kind of build in some of this cloud financial knowledge where teams can get that visibility into what they’re spending and know, is this too high? Is it going out of a boundary? Is there a number that I need to keep inside of? I think these are important things that level of visibility around cost and that team’s actual charges get people to start thinking, “Well, hold on a second. We’re above budget.” Even though maybe it’s not a real budget, “We’re above a spend by 20%. We need to bring that down.” And you give them the tools they need and the dashboards to effect that change on their own.Jesse: This idea of passive monitoring is really all about making the right thing the easy thing to do. If you, as a member of the cloud platform team or as a member of leadership who cares about cloud spend, wants to make sure that teams are managing their spend in some capacity, maybe not actively directly, but at some level make sure that there are these guide rails in place that keep them within the boundaries of what they’re ultimately able to do, or what you ultimately want them to work on. This makes it a lot easier for them to not spin up an i3 instance that they don’t ultimately need; it makes it a lot easier for them to not deploy resources that are missing tags. Put as many guardrails in place that keep the teams independently able to work within the space that they are building, and developing, and functioning, but ultimately gives them the opportunity to continue being independent and really thrive within whatever work they’re doing.Pete: Yeah, the next thing that we recommend to everyone. And actually, we recommend it before even engaging with The Duckbill Group, you’ll get an onboarding document of things to do, and the thing we always recommend is turn on the Cost and Usage Report. If you’re listening to this and you’re like, “What’s the Cost and Usage Report?” Well, boy are you going to have a fun learning because it is a highly granular usage report of everything that you’ve ever done within Amazon, and it’s extremely powerful. The downside is that it can be hard to navigate; it takes a little time to learn.But go turn it on; the cost is minimal; it’s the cost of storing this data in S3. Preferably when you turn this on, turn it on into Parquet format because it’ll allow you to query it with tools like Athena, or Tableau, or Looker or—God forbid—SageMaker. And this tool, this Cost and Usage Report, lets you dive in at an extremely granular level, down to the resource visibility—per hour, per resource visibility. And it’s something you have to enable, but again, highly recommend it to enable that resource level usage. Because now you can go and find out, well, for SageMaker I’m seeing a growth in spend.Well, which resource is it within SageMaker? You can break that down really granularly. So, Cost and Usage Report is another place that, again, if you’re not using this today, if you don’t have at least a SageMaker dashboard, which costs basically nothing—a couple of dollars a month—pointed at your Cost and Usage Report, you’re missing out on some really great ways to understand the changes in spend over time.Announcer: If your mean time to WTF for a security alert is more than a minute, it’s time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you’re building a secure business on AWS with compliance requirements, you don’t really have time to choose between antivirus or firewall companies to help you secure your stack. That’s why Lacework is built from the ground up for the Cloud: low effort, high visibility, and detection. To learn more, visit lacework.com.Jesse: Another couple of really great options are the AWS Cost Anomaly Detection service and AWS Budgets. Both are free, which is absolutely fantastic. I highly recommend checking them out. AWS Cost Anomaly Detection, once enabled, will actually look for anomalies in your spend across different AWS services, across different cost attribution tags, across different cost categories. There’s a lot of opportunities here for you to see anomalous spend and act on it.This can be shared with teams as soon as the anomaly occurs, through Slack notification or an email, or maybe you get email notifications on a weekly basis, or a monthly basis, or some kind of recurring basis, for all of the anomalies that you saw within a given time period. We recorded an episode about Cost Anomaly Detection a while back; highly recommend checking that episode out. It’s got a lot of really great features and recommendations for getting started.The other one I mentioned is AWS Budgets. Again, if you’re not really sure where to start, try creating some budgets for your teams. Maybe look at the last six months of spend for each team, maybe look at spend across different tags, or team units, or business units, whatever makes the most sense for the way that your organization is set up, and create some budgets for those groups. These budgets could be for specific AWS services if you are a single team running within a single AWS account, it could be as complex as multiple business units across multiple different accounts across different parts of the organization. There’s lots of great opportunities here for you to start to better understand your spend, get better visibility into your cloud spend.Pete: Yeah, absolutely. I think all of those are great tools that can really help you. And, Jesse, I know we’ve talked about this before. Even just monitoring your tagging, not like, “Oh, are we tagging 50% of our resources?” But you want to monitor for your untagged by spend. So, if 95% of your spend is tagged, you’re crushing it. That’s amazing. But that may only be 50% of your things.So, I guess, care less about how many of your resources are tagged—because some of them just can’t be tagged, or are tagged in a painful way—but focus more on the money aspect of it. And that will lead you into the ability to start creating some governance strategies. And that term governance, it just—Jesse: Oof.Pete: —makes me feel gross. Yeah. Oh god, terrible word. But the [laugh] sad state of the world is, that’s what most companies we talk to need; they just don’t have it. When the companies that we talk to who are like, “Our spend is going up, and we’re not sure why.” Or, “How do we get our engineers to care about cost savings?” And things like that. You know, having a governance strategy, a way to react to those changes in spend in a, hopefully, automated way, is critical to helping control that spend.Jesse: This really gets to the heart of why is cloud cost management important? It could be important for different reasons for different parts of the organization. Account structure, tagging, all of these different things can be important for different parts of the organization for different reasons. And that’s fine. The important thing is to socialize those reasons why to all the different parts of the company so that everybody understands what’s at stake.Everybody understands how they can collaborate and create these best practices together. This really dives into the idea of behaviors and systems. I know it sounds a little bit not within the vein of engineering work, and finance, and cloud cost management, but what kind of behaviors do you ultimately want to see within your teams? What kind of actions do you want to see your engineers taking? Do you want them to start thinking about cost in all of their architecture discussions?Do you want them to review the budgets that you’ve created for them every month? Every week? During stand-up meetings? What kind of things do you ultimately want to see them doing on a regular basis that maybe they aren’t doing right now, that maybe would ultimately help the company succeed with all of this cloud cost management work that you’re creating? And again, going back to the idea of making the right thing the easy thing to do, how can you improve the existing technical systems that you have within your organization to make the right thing the easy thing to do?How can you change your CI/CD pipelines? How can you change the tools that you’re using for cost visibility, like Looker, or Tableau, or SageMaker, or something else, such that teams can quickly and easily self-service the information that they need to make their decisions to go about their days, go about their work more easily?Pete: So, Jesse, you’re saying that it’s a mixture of software and culture? Kind of sounds like DevOps a little bit, doesn’t it? [laugh].Jesse: Yeah. Yeah, it kind of does.Pete: Yeah, it kind of does. So, you know, I think all of that is to say, it’s hard work, it’s not going to come easy, but how would we get started? Like, when we enter into an engagement with one of our clients, we’re coming in from total outsiders and we’re trying to navigate through a company with complex communication structures, and maybe teams that are entrenched in different ways. How do we get started? Well, we dive in; we start with big numbers, right?What are your top ten places your money goes, just by service? I’ll answer it for you. It’s probably EC2, S3, RDS, and then dealer’s choice for the last ones, maybe data transfer, maybe Lambda, if you’re really weird. And if Lambda is in your top five, you should absolutely give us a call because that should not be the case. [laugh].But start with those big numbers, understand where the money is going. But then go to the next level in. Okay, within EC2, where is your spend going? Or the dastardly EC2 ‘other cost’ category; okay’s the money going? Is it in regional data transfer, which is also what’s called cross-AZ data transfer? Is it in your NAT gateways? Why?That’s the next question. Why is the spend high in that area? You may not be able to understand because it may not be tagged—we find that a lot—but start asking questions. And that’s what we do: We start reaching out to technical folks within the company. We’d say, “Hey, we see you’ve got a high amount of usage on EMR, but they’re all clusters that are running 24/7. They’re not scaling up and down as the jobs are happening. Who knows more about EMR?” And we just start asking questions. And we’re asking them, “Well, are you doing anything on the cost optimization side? Have you tried to do anything cost optimization-wise to reduce it, and you haven’t been able to? How does this infrastructure scale? Does it scale linearly with the number of users? Does it scale in a different way? Who are the consumers?”And then you kind of even go another level down to see, do you find anything that just looks odd? I saw on one account for a client we were working with, VPC costs were just extremely high, much higher than I’ve ever seen before. What was interesting is that the cost was not data-transfer related; it was the pure number of endpoints that they had created that that cost far outweighed any other costs to data transfer; there was just a piece of technical debt that they were aware of, but the structure of their multi-accounts, they just couldn’t do anything about it. But again, you’re looking for things like that. And you know that you are doing a good job if, essentially, you can get to the end of this process—which could take months and it could take years depending on your scale—is if you can answer this question that if your customers, or users, or consumers of the applications on your cloud service if they increased by 200%, 500%, 1,000%. What would happen to your cloud spend? How would it change? That’s the end game you’re trying to get to. That’s the unit economics, the unit economic model and forecasting, and now you’re a superhero because now you can answer that question that not a lot of people are able to answer with their cloud usage.Jesse: I also want to add that, as you’re asking questions, you’re going to find teams that specifically will tell you, “We created this infrastructure in this way because security told us to,” or, “Because our business requirements say that we have an SLA that means we need to keep data for this amount of time at this level of availability.” And that’s totally fine. That doesn’t mean that you need to necessarily change those requirements. But now you might have a dollar amount for those business decisions. Now, you might ultimately be able to say, okay, our product SLA may say that we need to keep data for 90 days, but keeping data for 90 days, that business decision is costing us hundreds of thousands of dollars every month because of the sheer volume of data that we now have to keep. Is that something that we ultimately are okay with? And are we okay spending that much money every month to keep this business decision, or do we need to revisit that business decision? And that’s only something that you and your teams can decide for yourselves.Pete: Awesome. These are great questions. You could also send us a question lastweekinaws.com/QA. We would love to spend some time diving into it and just helping you out and helping you get through your day. That’s what we’re here for.If you enjoyed this podcast, please go to lastweekinaws.com/review and give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review. Give it a five-star review on your podcast platform of choice, and tell us what is your favorite EC2 instance to turn off for your engineers.Jesse: [laugh].Pete: Thanks, everyone.Announcer: This has been a HumblePod production. Stay humble.
Links: Unconventional Guide to AWS Cost Management: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/ Migrate from Oracle to Amazon Aurora: https://aws.amazon.com/getting-started/hands-on/migrate-oracle-to-amazon-aurora/ TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I am Pete Cheslock.Jesse: I’m Jesse DeRose.Pete: We’re coming at you again with some more listener questions from the Unconventional Guide to AWS Cost Management. I’m excited. People are listening to us, Jesse.Jesse: This is fantastic. I’m really excited that we have one fan. I’ve always wanted one fan.Pete: Well, two fans now. Maybe even more because we keep getting questions. And you can also be one of our Friends of the Pod by going to lastweekinaws.com/QA. And you can give us some feedback, you can give us a question and, like, will totally answer it because we like Friends of the Pod.Jesse: We may or may not enter you into a raffle to get a Members Only jacket that’s branded with ‘Friends with the Pod.’Pete: We should get some pins made, maybe.Jesse: Ohh…Pete: I think that's a good idea.Jesse: Yeah.Pete: So, what are we answering today, or attempting to answer for our listener, Jesse?Jesse: So today, we’ve got a really great question from [Godwin 00:01:20]. Thank you, Godwin, Godwin writes, “I truly believe that the system that I support is, like, a data hoarder. We do a lot of data ingestion, we recently did a lift-and-shift of the system to AWS, we use an Oracle database. The question is, how do I segregate the data and start thinking about moving it out of traditional relational databases and into other types of databases? Presently, our method is all types of data goes into a quote-unquote, ‘all-purpose database,’ and the database is growing quite fast. Where should I get started?”Pete: Well, I just want to commend you for a lift-and-shift into Amazon. That’s a Herculean feat, no matter what you’re lifting and shifting over. Hopefully, you have maybe started to decommission those original data centers and you don’t just have more data in twice as many locations.Jesse: [laugh]. But I also want to call out well done for thinking about not just the lift-and-shift, but the next step. I feel like that’s the thing that a lot of people forget about. They think about the lift-and-shift, and then they go, “Awesome. We’re hybrid. We’re in AWS, now. We’re in our data center. We’re good. Case closed.” And they forget that there’s a lot more work to do to modernize all those workloads in AWS, once you’ve lifted and shifted. And this is part of that conversation.Pete: Yeah, that’s a really good point because I know we’ve talked about this in the past, the lift-and-shift shot clock: when you don’t start migrating, start modernizing those applications to take advantage of things that are more cloud-native, the technical debt is really going to start piling up, and the folks that are going to manage that are going to get more burnt out, and it really is going to end poorly. So, the fact you’re starting to think about this now is a great thing. Also, what is available to you now that you’re on AWS is huge compared to a traditional data center.Jesse: Yeah.Pete: And that’s not just talking about the—I don’t even know if I’ve ever counted how many different databases exist on Amazon. I mean, they have a database for, at this point, every type of data. I mean, is there a type of data that they’re going to create, just so that they can create a database to put it into?Jesse: Wouldn’t surprise me at this point.Pete: They’ll find a way [laugh] to come up with that charge on your bill. But when it comes to Oracle, specifically Oracle databases, there’s obviously a big problem in not only the cost of the engine, running the database on a RDS or something to that effect, but you have licensing costs that are added into it as well. Maybe you have a bring-your-own-license or maybe you’re just using the off-the-shelf, but the off-the-shelf, kind of, ‘retail on-demand pricing’ RDS—I’m using air quotes for all these things, but you can’t see that—they will just have the licensing costs baked in as well. So, you’re paying for it—kind of—either way.Jesse: And I think this is something also to think about that we’ll dive into in a minute, but one of the things that a lot of people forget about when they move into AWS says that you’re not just paying for data sitting on a piece of hardware in a data center that’s depreciating, now. You’re paying for storage, you’re paying for I/O costs, you’re paying for data transfer, to Pete’s point, you’re also paying for some of the license as well, potentially. So, there’s lots of different costs associated with keeping an Oracle Database running in AWS. So, that’s actually probably the best place to start thinking about this next step about where to get started. Think about the usage patterns of your data.And this may be something that you need to involve engineering, maybe involve product for if they’re part of these conversations for storage of your product or your feature sets. Think about what are the usage patterns of your data?Pete: Yeah, exactly. Now, you may say to yourself, “Well, we’re on Oracle”—and I’m sure people listening are like, “Well, that’s your problem. You should just move off of Oracle.” And since you can’t go back in time and undo that decision—and the reality is, it probably was a good decision at the time. There’s a lot of businesses, including Amazon, who ran all of their systems on Oracle.And then migrated off of them. Understanding the usage patterns, what type of data is going into Oracle, I think is a big one. Because if you can understand the access patterns of the types of data that are going in, that can help you start peeling off where that data should go. Now, let’s say you’re just pushing all new data created. And we don’t even know what your data is, so we’re going to take some wild assumptions here on what you could possibly do—but more so just giving you homework, really—thinking about the type of data going in, right?If you’re just—“I’m pushing all of my data into this database because someday we might need to query it.” That’s actually a situation where you really want to start thinking of leveraging more of a data warehouse-style approach to it, where you have a large amount of data being created, you don’t know if you’re going to need to query it in the future, but you might want to glean some value out of that. Using S3, which is now available to you outside of your data center world, is going to be super valuable to just very cheaply shove data into S3, to be able to go back in later time. And then you can use things like Athena to ad hoc query that data, or leverage a lot of the ingestion services that exist to suck that data into other databases. But thinking about what’s being created, when it is going into places is a big first step to start understanding, well, how quickly does this data need to come back?Can the query be measured in many seconds? Can it be done ad hoc, like in Athena? Does it need to be measured in milliseconds? What’s the replication that needs to happen? Is this very valuable data that we need to have multiple backups on?Is it queried more than it’s created? Maybe you need to have multiple replica reader databases that are there. So, all these types of things of really understanding just what’s there to begin with, and it’s probably going to be in talking to a lot of engineering teams.Jesse: Yeah, you can think about this project in the same way that you might move from a monolith to a microservice architecture. So, if you’re moving from a monolith to a microservice architecture, you might start peeling away pieces of the monolith, one at a time. Pieces that can easily be turned into microservices that stand on their own within the cloud, even if they’re running on the same underlying infrastructure as the monolith itself within AWS. And then, as you can pull those pieces away, then start thinking about does this need to be in a relational database? Does this need to have the same amount of uptime and availability as the resources that are sitting in my Oracle Database right now?All those things that Pete just mentioned, start thinking about all of those components to figure out where best to pull off the individual components of data, and ultimately put them in different places within AWS. And to be clear, there’s lots of great guides on the internet that talk about moving from your Oracle database into, gosh, just about any database of choice. AWS even has specific instructions for this, and we’ll throw a link in the [show notes 00:09:02].They really, really want you to move this data to RDS Aurora. They go through painstaking detail to talk about using the AWS schema conversion tool to convert your schema over; they talk about the AWS database migration service to migrate the data over, and then they talk about performing post-migration activities such as running SQL queries for validating the object types, object count, things like that. I think that a lot of folks actually don’t know that the database migration service exists, and it’s something worth calling out as a really powerful tool.Pete: Yeah, the Amazon DMS service is honestly I think, a super-underrated service that people just don’t know about. It has the ability to replicate data from both on-premises databases to Amazon databases but also databases already running on Amazon. You could replicate from a database running on EC2 into Aurora. You could replicate that into S3—you know, replicate data into S3 that way, bringing things into sync—replicate that data into S3, and then maybe use it for other purposes. It can replicate data from DocumentDB into other sources.So, they’re clearly doing a big investment in there. And to Jesse’s point, yeah, Amazon really wants this data. So, talk to your account manager as you’re testing out some of these services. Do a small proof of concept, maybe, to see how well it works, if you can understand the queries, or you can point your application over at an Aurora database with some of this data migrated in; that’s a great way to understand how well this could work for your organization. But as Jesse mentioned, they do want that data in Aurora.So, if it turns out that you’re looking at your—you know, migrate some data in there, and it’s starting to work, and you’re kind of getting a feel for the engineering effort to migrate there, stop. Talk to your account manager before you spend any more money on Aurora because it’s very likely that they can put together a program—if a program doesn’t already exist—to incentivize you to move that data over; they can give you subject matter expertise; they can provide you credits to help you migrate that data over. Don’t feel like you have to do this on your own. You have an account team; you should definitely reach out to them, and they will provide you a lot of help to get that data in there. They’ve done it for many of their other clients, and they’re happy to do it for you because they know that, long term, when you move that data to Aurora, it’s going to be very sticky in Aurora.You’re probably not going to move off of there. It’s a long game for them; that’s how they play it. So, check out those services; that could be a really great way to help you get rid of your Oracle addiction.Jesse: Yeah, and if you’re able to, as we talked about earlier, if you’re able to identify workloads that don’t need to run in a relational database, or don’t need to run in, maybe, a database at all, for that matter, stick that data in S3. Call it a day. Put them on lifecycle management policies or different storage tiers, and use Athena for ad hoc queries, or maybe Redshift if you’re doing more data warehouse-style tasks. But if that data doesn’t need to live in a relational database, there are many cheaper options for that data.Pete: Exactly. But one last point I will make is don’t shove it into MongoDB just because you want to have schema-less, or—Jesse: Please.Pete: —think about what you’re going to use it for, think about what the data access patterns because there is a right place for your data. Don’t just jump into no-SQL just ‘cause because you’ll probably end up with a bigger problem. In the long run.Corey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com.Pete: So Jesse, I’m looking at our list of questions. And it turns out, we have another question.Jesse: Ohh.Pete: Two questions came in.Jesse: You like me, you really like me!Pete: It’s so great. Again, you can also send us a question, lastweekinaws.com/QA. You can go there, drop in a question and feel free to put your name. Or not; you can be anonymous, it’s totally fine. We’ll happily answer your question either way. So Jesse, who is our next question from? What is this one about?Jesse: This one’s from [Joseph 00:13:19]. They write in, “Hey, folks. Love the show. Longtime listener, first-time caller.” Thank you. “I would love to know how people manage their costs in AWS Batch. Jobs themselves can’t be tagged for cost allocation, which makes things a bit complicated.” Lord Almighty, yes, it does. “How best should I see if the jobs are right-sized? Are they over-provisioned in terms of memory or compute? What’s the best way to see if EC2 is my better choice, versus Fargate, versus other options? How can I tell if the batch-managed cluster itself is under-utilized?”Pete: Oof. This is a loaded question with a lot of variables.Jesse: Yeah. And so we’re going to break it down because there’s definitely a couple questions here. But I want to start off with what AWS Batch is, just really quick to make sure everybody’s on the same page here. AWS Batch, effectively, is a managed service in AWS that schedules it and runs your batch computing jobs on top of AWS compute resources. Effectively, it does a lot of the heavy lifting configuration for you so you can just focus on analyzing the results of those queries.Pete: Yeah, exactly. And Batch supports a really wide variety of tooling that can operate this, and that’s why it’s hard for us to give, specifically, how you might optimize this, but I think some of the optimizations actually mirror a lot of the optimizations we’ve done with optimizing EMR clusters and things of that nature, where you’re running these distributed jobs. And you want to make sure that if you’re running straight off of EC2 instances, then you want to make sure that they are essentially maxed out. If the CPU is anything less than 100% for an on-demand instance, then there’s wasted, or there’s opportunity for improvement. And so making sure that your jobs are sized appropriately and balancing out memory and CPU so that, effectively, you’re using all of the memory and all of the CPU, that’s a real basic first step.But honestly, a lot of folks kind of miss out on that. They just kind of run a job and go off and do their own thing. They never really go back and look at those graphs. You can go to CloudWatch, they’re all going to be there for you.Jesse: Yeah. And to this point, there’s always an opportunity to make these workloads more ephemeral. If you have the opportunity to make it more ephemeral, please, please, please, please, absolutely do so. Unless your batch job needs to run 24/7. We’ve seen that in a few cases where they have, essentially, clusters that are running 24/7, but they’re not actually utilized regularly; the workloads are only scheduled for a short amount of time.So, if you don’t need those batch jobs running 24/7, please, by all means, move to more ephemeral resources, like Fargate. Fargate on Spot, Spot Instances in general, or even Lambda, which AWS Batch now supports as well.Pete: Yeah, it has some step function support, which is pretty interesting. Yeah, this is a great opportunity to aggressively—aggressively—leverage Spots, if you’re not currently today. The reality is that check out Fargate on Spot if you don’t need, like, a custom operating system, you don’t need a custom EBS volume size. If you do, then EC2 on Spot is probably the best option that you really have. But really do not want to be running anything on on-demand instances. Even on-demand instances with a really good savings plan, you’re still leaving money on the table because Spot Instances are going to be a lot cheaper than even the best savings plan that’s out there.Jesse: And I think that’s a good point, too, Pete, which is if you do need to run these workloads on-demand, 24/7, think about if you can get away with using Spot Instances. If you can’t get away with using Spot Instances, at least purchase a savings plan if you don’t do anything else. If you take nothing else away from this, at least make sure that you have some kind of savings plan in place for these resources so that you’re not paying on-demand costs 24/7. But in most cases, you can likely make them more ephemeral, which is going to save you a lot more money in the long run.Pete: Yeah, exactly. That’s the name of the game. I mean, when we talk to folks on Amazon, the more ephemeral you can make your application—the more you can have it handle interruption—the less expensive it will be to operate. And that goes from everywhere from Spot Instances and how they’re priced, right? If you just get a normal Spot Instance, it will have a really aggressive discount on it if you need zero time in advance before interruption.So, if that instance can just go in at any second, then you’ll get the best discount on that Spot Instance. But if your app needs a little time, or runs for a defined period of time—let’s say your app runs for one hour—you can get a defined duration Spot of one hour, you’ll get a great discount still and you’ll only pay for however long you use it, but you will get that resource for one whole hour, and then you’ll lose it. If that’s still too aggressive, there’s configurable options up to six hours. Again, less discount, but more stability in that resource. So, that’s the trade-off you make when you move over to Spot Instances.Jesse: So, I also want to make sure that we get to the second part of this question, which is about attributing cost to your AWS Batch workloads. According to the AWS Batch documentation, you can tag AWS Batch compute environments, jobs, job definitions, and job queues, but you can’t propagate those tags to the underlying resources that actually run those jobs. Which to me, kind of just defeats the point.Pete: Yeah. [sigh]. Hashtag AWS wishlist here. You know, again, continuing to expand out tagging support for things that don’t support it. I know we’ve seen kind of weird inconsistencies, and just even, like, tagging ECS jobs and where you have to tag them for they’re to apply.So, I know it’s a hard problem, but obviously, it’s something that should be continually worked out on because, yeah, if you’re trying to attribute these costs, you’re left with the only option to run them in separate Amazon accounts, which solves this problem, but again, depending on your organization, could increase just the management overhead of those. But that is the ultimate way. I mean, that is the one way to ensure 100% of costs are encapsulated to a service is to have them run in a dedicated account. The downside being is that if you have a series of different jobs running across a different, maybe, business units, then obviously that’s going to break down super quick.Jesse: Yeah, and it’s also worth calling out that if there’s any batch jobs that need to send data to different places—maybe the batch job belongs to product A, but it needs to send data to product B—there’s going to be some amount of data transfer either across regionally or across accounts in order to share that data, depending on how your organization, how your products are set up. So, keep in mind that there are potentially some minor charges that may appear with this, but ultimately, if you’re talking about the best ways to really attribute costs for your AWS Batch workloads, linked accounts is the way to go.Pete: Yeah. If you need attribution down to the penny—some of our clients absolutely do. For invoicing purposes, they need attribution for business unit down to the penny. And if you’re an organization that needs that, then the only way to get that, effectively, is segmented accounts. So, keep that in mind.Again, until Amazon comes out with the ability to get a little bit more flexible tagging, but also, too, feel free to yell at your account manager—I mean, ask them nicely. They are people, too. But, you know, let them know that you want this. Amazon builds what the customers want, and if you don’t tell them that you want it, they’re not going to prioritize it. I’m not saying if you tell them, you’re going to get it in a couple of months, but you’re never going to get it if you don’t say anything. So, definitely let people know when there’s something that doesn’t work the way you expect it to.Jesse: Absolutely.Pete: Awesome. Wow. Two questions. I feel it’s like Christmas. Except—Jesse: [laugh].Pete: —it’s Christmas in almost springtime. It’s great. Well, again, you, too, can join us by being a Friend of the Pod, which Jesse really loves that one for some reason. [laugh].Jesse: Yeah. Don’t know why, but it’s going to be stuck in my brain.Pete: Exactly. You too can be a Friend of the Pod by going to lastweekinaws.com/QA and you can send us a question. We would love to spend some time in a future episode, answering them for you.If you’ve enjoyed this podcast, please go to lastweekinaws.com/review. Give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review and give it a five-star rating on your podcast platform of choice and tell us why you want to be a Friend of the Pod. Thank you.Announcer: This has been a HumblePod production. Stay humble.
Links:Unconventional Guide to AWS Cost Management:https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/ TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I am Pete Cheslock.Jesse: I’m Jesse DeRose.Pete: We’re back again. And we’re here. We made it, Jesse.Jesse: I was worried. This was a journey. Thank you, everybody, for coming on this journey with us.Pete: It was quite an experience going through the Unconventional Guide to AWS Cost Savings. We’ve made it. I just can’t believe we’re here.Jesse: Yeah.Pete: So, what are we talking about today for the culmination of our magnum opus of cost savings optimizations?Jesse: This is a fun one. And I know I keep saying that this is my favorite about everyone, but I have to admit that this one, this topic today probably is my absolute favorite. This one I get really nerdy over. Today, we’re talking about how to predict your future and make your CFO happy. No—spoiler alert—there are not any crystal balls involved in this one. There’s no stock market conversations.This is talking about how you can use all of the different things that we’ve talked about throughout the course of this Unconventional Guide to really bring it all together into a couple ideas that will help you better understand your cloud costs, and really better understand your business, I think.Pete: Yeah. All of the things we talked about really lead up to this one, which is the clients of ours that are the most mature, who are incredibly optimized in their Amazon usage, are the ones who have adopted a majority of these specific items. They all lead to this last one, that ability to predict your future usage based on something that’s happening internally, or if a salesperson comes to you and says, “Hey, we’re about to close this deal, but I need to discount our service.” People are going to start wanting to know well, what is the cheapest that you could sell your service for and still have a positive gross margin?Jesse: Yeah. So, if you’ve done a lot of the things that we’ve talked about in the last couple episodes—I apologize, I know homework’s not the best for a podcast—but if you’ve had the opportunity to work on some of those things, you should have a ton of valuable insights into your spend. We’re talking about tagging, and showback models in particular, maybe even a chargeback model. But you can ultimately use all of this data to better understand what is your forecasted spend is going to look like with a new potential customer coming onto the platform? Or if you get into the topic that we’re going to talk about today, which is mostly unit economics, you can really understand how much can I discount my service and still make a profit, like Pete mentioned?Pete: Yeah, I mean, imagine there’s a global pandemic that happens, and it causes your usage to spike by 500% within the course of a month. How did your spend change? Do you know where it changed? And did it change in ways that you were expecting it to? Like, my databases grew by a lot, and this other thing didn’t grow by very much.Like, that would be expected. But also another thing that—a question that we actually like to ask a lot of our clients, if your sales just doubled overnight, okay would your spend change? Where are the places that are most expensive to operate your service? And again, this is kind of generic. I’ve worked in a lot of SaaS services, so I always think of sales, but just think of whether you’re using the cloud for a SaaS service that you provide and sell, like, B2C, things like that, or B2B, you still have users.They might be internal users. Well, what if your users doubled overnight? What if half the company was using your internal service and now the whole company is? How does that change your usage?Jesse: And it’s also important to think about not just your AWS usage, but all of the other services that you use that support your overall business model: things like monitoring and observability tools, logging vendors, maybe third-party sim tools. All of these are affecting your overall total infrastructure cost and are all part of this conversation. So, it’s really important to start thinking about those architecture diagrams. Remember, when we said, way, way back at the beginning of this conversation, to overlay costs on top of your architecture diagram, understanding that, understanding what parts of your product or what parts of your architecture are the most expensive will really help you understand what’s going to change?Pete: Yeah, let’s say you’ve got a six-figure bill to Datadog or one of the big log management vendors out there, but inside of that bill, is that all just evenly spread across the whole business? What if your log vendor was—the entire spend was all by one service that some developer left the debug logging enabled for? You know, you’d want a way of understanding that maybe that spend was concentrated in maybe a non-production aspect of your account. Because then again, that wouldn’t grow, right? That wouldn’t affect your growth in your sales the same way as if maybe all of your services were equally sending logs of a certain volume over.So, all of those extra services, they all add up, and we see it more and more, as more of our clients start adopting more than just Amazon services: they might be adopting a Snowflake, they might be adopting third-party services running databases running in other services, or EMR type workloads that are not on EMR, and they’re running on Qubole or things like that. There’s just a lot of these services that more and more people are consuming from that fall outside of just the AWS invoice.Jesse: And this also gets back to not just architecture diagrams, but also tagging and showback models, cost visibility, really understanding where your spend is going. And this is fantastic to understand where your spend is going, but finance is probably going to want something a little bit more than this. It’s not just about how much are we spending, or where are we spending it, and maybe it’s not even a finance question. Maybe this is a sales conversation, assuming that you’re a SaaS company. Maybe this is, as Pete mentioned before, “Hey, we want to understand where can we provide discounts? What services can we ultimately discount to negotiate getting new customers on the platform?”Pete: So, Jesse, we hear a lot of these terms a lot, and I’d love a ‘explain like I am a five-year-old’ version of it, but we hear a chargeback. And we hear showback, and honestly, I’ve never worked at those massive companies where you might implement these things, but can you give us just a real quick—for all the listeners out there, when we say showback, what does that mean? And when we say chargeback, what does that mean?Jesse: So, a showback model essentially takes all of your cloud costs, all of your total infrastructure spend, your AWS spend, all the third-party spend, and it shows every team, every product, every microservice, maybe, depending—or maybe even business unit, depending on how your organization is split up—it shows each one of those units, how much they are actually spending, how much they’re actually using these different cloud vendors. So, this is where tagging comes in super handy because if you’ve tagged all of your taggable resources, and properly attributed all of your cloud costs with tagging and linked accounts, you have a very clear idea of who’s spending what. You know very clearly, maybe 70 to 80% of your total infrastructure spend is related to one particular product because all the cost is attributed to one particular product. And maybe that’s something you didn’t know before. Maybe now you know okay, maybe that product needs to be a little bit more expensive so that we can make sure that we are making money off of it, or profiting off of it, whereas other services can be discounted because they’re not as expensive.Whereas in a chargeback model, you are ultimately not just showing each of these teams, hey, here’s how much you spent on AWS and Datadog usage and all these other vendors every month, you’re actually charging them for that usage. You’re actually pulling their cloud costs from their budget.Pete: Yeah. They might actually have a budget of money. It’s all—if you want to really explain like I’m five, it’d be like, I give my child their—they get $1 for all of the tasks that they do throughout the week, I don’t actually give them the money because I usually have to subtract out their, like, Roblox spend of the week [laugh] and things like that. It’s all virtual, but at the end of the day, you know, we’re kind of virtually giving this business unit some money, and then, kind of, virtually charging them for their services within.Jesse: Yeah. And this is mature. We don’t see a lot of companies doing this. This is hard because you have to take other steps first to get here. And so this is why we harp so much on cost attribution through tagging and through linked accounts.This is why we harp so much on cost visibility and overlaying those cloud costs on your architecture diagrams to understand all of this data to lead to this point, which is understanding, where, how much is my primary product actually costing us? How much is my secondary product actually costing us? Or maybe how much is this business unit costing us in terms of total infrastructure spend?Pete: Yeah, I mean, I can kind of share my history with this at previous companies is that, again, eventually someone in the financial department is going to say, “What was our cost for Amazon?” They specifically will want to know the production cost because that figures into a term called gross margin, which you often hear at SaaS businesses. Gross margin is basically you take all the revenue that came in and you subtract away what it took to support that revenue. And mostly, that is just the Amazon bill and these other vendors, perhaps, and you end up with a percentage. And hopefully, it’s a positive percentage.It means you’re theoretically making money at a gross, I mean, obviously, before you pay salaries and all those other items, but that being kind of beside the point for now, that number, you’re probably going to get asked for. So, you wouldn’t want to give like your straight Amazon bill, like, “Oh, well, we spent $100,000 last month,” because some of that spend was probably in research and development; it was probably in a development account or a QA account. You really just want your product spent. So, at a previous company, the first step we took was break out our spend via production and development, just two criteria. Now, for us because we started with just a handful of accounts—this was before a lot of accounts were more prevalent, before organizations—before it was easy to handle a lot of accounts—we had a Prod and a Dev. Super easy. Look at Prod, look at Dev. There’s the two bills.But then as time went on, we needed to get more granular. We were running some development workloads, testing out new databases at scale in kind of a hidden dark deployed mode, in production. Well, we want to subtract that spend from there. And that requires tagging. I mean, that’s why we really harped on tagging for a couple of episodes because tagging is the only way you’re going to be able to do that.Now, we see more often a lot of our clients do maybe an account per product, or account per business unit. Those are, again, really effective ways to corral your spend to make it really easy to break it out and add it up. It’s really just trying to break it down to the most reasonable spend unit possible that you can then play around with and adjust. Mostly to go back to your CFO when they asked you, “Hey, I need to know this specific answer.” You’ve got it hopefully somewhat available.Jesse: Okay, so this is where we’re going to start talking about unit economics. And hopefully, your eyes will not glaze over. I want to make sure that—this is important, this is really actually beneficial. It’s not just a specific economic thing that you learned back in Econ 101. This is actually going to be useful and beneficial for your organization.So, unit economics describes your product in terms of revenues and costs in relation to a unit KPI—that’s where the ‘unit’ term comes from in ‘unit economics’—that tracks closely with customer demand. So, that’s a really gross definition, I know, and I apologize.Pete: You know, and we can even extend that a little bit further and give some good examples. Like, maybe if you are a website that provides eLearning services, your unit might be the number of daily active users or thousands of daily active users, right, could be a thing. That could be a unit that you’re selling. I actually worked at a SaaS company where we sold a piece of software that would run per server, and we broke our unit down to the servers—the things that we sold—down to that level.Jesse: Yeah, if you are in the airline industry, for example, your unit would probably be every passenger. How many passengers are you able to sell tickets to on every plane? What do those costs look like?Corey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com.Pete: And you don’t need just, like, one unit. Maybe you have one unit for your whole platform like the whole gross production spend breaks down into one specific unit, you could do that. But you could also have units at a per-service level because maybe it’s like you’re processing a lot of documents. I worked for an email archiving company, forever ago, and our unit was the amount of emails that were indexed and archived so we could figure out, we might have one customer who just didn’t generate a lot of emails, but they had tons of users. Well, one of our units was the volume of emails that we were indexing and archiving for that customer, whereas on the flip side, if maybe our spend was driven more by user count, and not document count, maybe that’s what we want our unit to be, is per user.Jesse: Yeah. It’s really important to call out that you might have a single easy-to-define unit; you might have a more complex relationship that’s weighted with a couple different factors of different components of the architecture. But ultimately, your unit KPI and how you break out your costs to support your customers will be unique to your overall business.Pete: Exactly. And this is where you’re only going to find this answer out with a lot of conversations, internally. It could come to you pretty easily, you know, just based on how your business is. But I think for a lot of folks using Amazon, especially if you’re just in a specific business unit inside of a broader business, it could be a little bit more challenging to figure out. But what you’re really trying to do is just figure out, when X changes, our spend changes, and we spend more or we spend less. Try to solve for X. That’s really what you’re trying to do.Jesse: Okay. So, now we’ve covered the unit KPI part of this conversation. Awesome. So, we’re done, Pete, right? We just take our AWS bill and then—Pete: Yeah.Jesse: —divide it by the unit and we’re done.Pete: So, easy. I’ve got my unit. I’ve got my bill. I got an iPhone that can do a calculator. Good to go.Jesse: [laugh]. Good. We’re done. Well, wait. What about if I have multiple AWS accounts? Wait, what if I have multiple different products?Pete: Yeah, that’s… I mean, I kind of calculator. I mean, I might be here all day, but…Jesse: [laugh]. I’ve got a whiteboard. We’ve got some time.Pete: Yeah, we got time. That’s a great point, though. Again, what if you do have things that are just spread all over the place? What if you’ve got two different products, two different services inside the same account? Because of course, you would. That’s a super normal thing. I’m not even saying that sarcastically. That’s a super normal thing.Jesse: Absolutely.Pete: Well, how do you handle this? How do you handle shared services?Jesse: Yeah. I mean I—Pete: We could go on for too long on that one, but these are questions you really want to start asking.Jesse: Yeah. And remember that you’re potentially going to have different unit KPIs for different products, for different business units. That’s fine. That’s expected. But make sure that you are measuring appropriately for each of those.The incoming costs, the incoming revenues, and costs for each of these isn’t going to change. That’s coming from your tagged usage and your linked accounts, but maybe the unit that you’re dividing that spend by is going to change, and that’s fine. This is where a spreadsheet comes in super, super handy. I love my Excel spreadsheets for this. Very, very easy to just bring in all of the bill data across different accounts, and really clearly attribute this spend is for the service, or the spend is for this product, or the spend is for this business unit, and then divide that by the unit that we have in question to get your actual unit KPI, to get your unit economic metric.Pete: Yeah, and this is where the superpower comes in. Once you have this number, now you can better understand and make product-level decisions. Again, whether you’re a SaaS product with a product you’re selling to external customers or building an internal tool, your product is the thing that the internal users consume. Your product decisions can now be driven by this. I mean, I have recollections of conversations with product teams, where they would talk about certain services internally, how they wanted to expand and do all the stuff with it.And I said, “Well, right now that one service represents one-third of our total spend, right? Our gross margin, that is one-third. But we looked at the users, and it’s only being used by one percent.” When you have these big numbers and saying, “Wow, the company spends a third of their money on something one percent of people use,” then maybe that’s not the place we want to be investing product decisions into. Maybe it is, but you don’t know enough to have that conversation unless you have this data.Jesse: Absolutely. I think there’s one other small caveat that we haven’t touched on that I do want to call out, and this comes back to your conversation about tagging. We have noticed that a lot of teams want to tag to a certain extent, and then start building their showback models immediately, which is great that you’ve got investment, you’ve got energy, you really want to get to that showback model, get to that chargeback model, that unit economic model space. But if your usage or cloud usage is not thoroughly tagged and accurately tagged, your resulting data is not going to be accurate either. So, we think about this in terms of a cost margin or a cost error.So, for example, if your production spend or your production usage is only 60% tagged, that means you’ve got 40% error in that data that’s coming in; your cloud spend for production has 40% error margin, which is huge.Pete: Yeah, exactly. Track your untagged spend, as well as your tagged spend. I mean, make sure you have a story for the things that are not tagged. That includes things like data transfer and things that maybe are not as taggable within AWS. That’s an important aspect of this that you’ll want to make sure you’re at least not forgetting about.Even if you can’t tag it, you don’t have a solution for it, make sure it’s in the back of your head that this is maybe not as accurate of a forecast because we’re just taking data transfer and dividing it by product versus actually looking at which product uses the most to transfer.Jesse: Yeah, and this is a tough concept, so don’t feel bad if you listen to this episode, again, don’t feel bad if you go download the Unconventional Guide from the Duckbill Group website—we’ll have the link in the [show notes 00:20:12]—this one is a tough concept because it brings in a lot of other moving parts to ultimately get at this one unifying really, really important idea. This is one that we see a lot of clients and potential clients struggle with. So, if you’re taking some time to understand this concept, you’re not alone.Pete: Exactly. This is the goal of all of the previous work, and this is something that you would measure in just a multi-year commitment in most businesses. And the larger the business is, the longer that work is going to take because it’s hard, there are a lot of moving pieces, and so many things need to be done in advance of all of this. And again, realize you’re not doing this work in a vacuum. There’s things that are moving and shifting as it’s all happening. So, don’t beat yourself up if you’re looking at this and thinking to yourself, “This is just a huge task. I’m never going to get this done.” It’s just not something that’s going to happen overnight.All right, well, hey, if you’ve enjoyed this podcast, if you’ve enjoyed this series, please go to lastweekinaws.com/reviewand give it a five-star rating on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review give it a five-star rating, but then tell us what’s the next series you want to have if you didn’t like this one.Also, don’t forget, you can give us your feedback and any questions that we’ll be continuing to answer in future episodes lastweekinaws.com/QA. You don’t need to put your name, can be totally anonymous. Give us your question. We’d love to dive into some of those topics.And finally, you can download our Unconventional Guide, the whole PDF of everything we’ve talked about at the Duckbill Group website. We’ll include that link in our [show notes 00:21:51] and you can head over there. Thanks again.Announcer: This has been a HumblePod production. Stay humble.
Jesse Thorn (@JesseThorn) is the owner of the podcast network, Maximum Fun and host of the podcasts, Bullseye, Jordan, Jesse, Go! and The Turnaround. The Turnaround with Jesse Thorn Interviewing with Aaron Lammer To offer your own advice, call Zak @ 844-935-BEST TRANSCRIPT: ZAK: It's The Best Advice Show where every weekday, someone I like offers one morsel of advice. Ideally, something you can start practicing today. On this episode, I'm gonna talk interviewing with one of my favorite interviewers. JESSE: I'm Jesse Thorn and I'm owner of the podcast network, Maximum Fun and also among other things the host of the NPR show, Bullseye. ZAK: Jesse' gonna talk about interviewing an iconic interviewer on his podcast about interviewing. And for those of you keeping tracking at home, this is my second interview with an interviewer about interviewing. The first one was with Longform podcast host, Aaron Lammer. There's a link to that in the show notes. Ok, here's Jesse talking about an essential piece of interview advice you can try at home, even if you don't have a talk show. JESSE: Like, the thing that I think about all the time that I learned from...I did a show called The Turnaround where I interviewed famous interviewers and I just did it because I never went to journalism school or had a mentor or anything. I just was like, doing my college radio show until I was 40. And, we sort of were surprised that Larry King said yes to coming on the show. Like, basically, we just made a list of every famous we could think of, sent out one email to each of them and saw who said yes. Cause it was a low-budget show. ZAK: But you had for people that haven't heard it, you had Ira Glass, Brook Gladstone, Larry King, did you get Terry Gross? JESSE:Yeah, that was the first time I talked to Terry Gross and I was very gratified. I'm a huge fan. But Larry King I was not a huge fan of, may his rest in peace. I wasn't against him or anything, I just never had cable tv as a kid so I never saw him, you know? And I never listened to overnight talk radio. But, I went into his house and he had this big house in Beverly Hills. ZAK: Did someone answer the door or was it him? JESSE: Yes, his assistant just a really sweet, obviously, intensely competent man. His assistant offered me a bottle of water and I was like, what is this gonna be like? And he sat us down in Larry King's trophy room which was like, the trophies were like structural to the room. There were so many trophies and prizes and pictures of him with Hank Aaron or whatever and he came in and he just Larry King right away. The moment he walked in the room, I understood why he was Larry King. Cause you're like, oh, this guy is the most engaged person ever. He locked eyes on me. He was completely present with me and the question that he said he was really proud of when I asked him about this, he said, one time a pilot came on my show... LARRY: And I said when you're going down the runway do you know it's gonna take off? And he said I never think about it. Yes, it will take off but it may not stay up. An engine could go. Birds can fly into the plane. But if I'm going 160 MPH down that runway, it has to take off. Now it may take off for five feet and crash but it will lift off the ground. But he never thinks about it. JESSE: And to me, that's like the perfect question because Larry King doesn't care that it makes him look dumb or makes him look like he doesn't know about pilots. ZAK: Yeah, there's narrative in that question. JESSE: Yeah, and it's like go so much emotional content, you know? ZAK: I spend my entire life not trying to sound dumb. Trying not to sound dumb. Not trying to sound dumb. See? And so, to know that one of the keys to really engaging and asking good questions is to not worry about sounding dumb. This is the work.
Jesse James Garret is a renowned leader in the user experience design field. He's a co-founder of the influential UX consultancy Adaptive Path and author of The Elements of User Experience. These days, Jesse coaches UX design leaders. In this conversation, we discuss the relationship between leadership and information architecture. Listen to the show Download episode 58 Show notes Jesse James Garrett's website @jjg on Twitter Jesse James Garrett on LinkedIn The Elements of User Experience: User-Centered Design for the Web and Beyond, 2nd Edition by Jesse James Garrett Peter Merholz Finding Our Way podcast MacGuffin Concept map Mind maps Some show notes may include Amazon affiliate links. I get a small commission for purchases made through these links. Read the transcript Jorge: Jesse, welcome to the show. Jesse: It's good to be here. Thank you. Jorge: Well, it's my pleasure and honor to have you on the show as a guest. I don't imagine that there are too many folks in the audience who don't know you, but for those who don't, would you please tell us about yourself? About Jesse Jesse: Sure! I'm Jesse James Garrett. I have been working as a professional in the user experience field for, 20 years or so now. If I am known to you at all, I am probably known to you, dear listener, from my book, The Elements of User Experience, which was published in 2002, or the work of my company Adaptive Path, which I co-founded in 2001 and was a part of through its acquisition by Capital One in 2014. I now work as an independent leadership coach working with leaders of UX design teams. Jorge: And as we're recording this, I believe that the founding of Adaptive Path happened 20 years ago. Jesse: Yeah! Yeah, yesterday was the 20th anniversary of the launch, a fun milestone to reflect on. Jorge: Well, the influence that the work has had both in Adaptive Path and The Elements of User Experience is palpable in the field. I occasionally still run into people who bring that diagram — "The Elements of UX" — bring it up so many years later, and it's an artifact that has proven long-lived. And I'm wondering if you have any thoughts on why that might be. On the longevity of The Elements of UX Jesse: It's a little bit of a mystery because Elements seems to have an enduring appeal to people that other similar models don't seem to have that kind of traction. I think that part of it is that I tried with the model to capture — as much as possible — to capture the things that I thought were less likely to change. Although I put the date really prominently at the top of the document when I first published it, in part because I was expecting to update it. I was thoroughly expecting there to be multiple versions and each one would have a date stamp and there would be iterations and evolutions of the model. But then when people started using it and getting really attached to it, I changed my mind about it and felt like I should really leave well enough alone and not tinker with it too much. I've made some little adjustments to the language that I use in the model over time, but the model itself has stayed the same. And I think the fact that people keep picking it up and putting it into practice is surprising to me as it is to anybody, I think. Jorge: Apart from minor tweaks to the language, do you feel like the model stands up overall? Even today? Jesse: Well, I do really think that if it didn't people wouldn't be using it if it didn't produce some sort of positive result... It may not be the positive result I intended. Yeah, I mean, there's a lot more to be said than what is encapsulated in that model. It is intended to provide a basic level framework and obviously there's a lot more complexity to what it takes to actually get those things done. And there is a lot of nuance to how these issues play out. So yeah, in some ways it's not my call as to whether or not the model is still relevant. It's like, it's up to other people as far as I can see. Jorge: Well, that's as good enough as any test for relevance, right? Whether people are using it or not. And for any listeners who might not know what we're talking about, this is a model that describes the work of user experience as happening in... would it be fair to call it five distinct layers? Jesse: Yeah. I call them 'planes' in the book, but it's a visualization. It's this sort of layer cake, sort of visualization of all of the considerations that go into UX work. Jorge: And they range from strategy at the lowest plane, scope, structure, skeleton and surface, which is the stuff that we see when we interact with a product that has been designed. Information architecture and leadership Now, I asked you to come on the show, not to talk about The Elements of User Experience, but because you and your fellow Adaptive Path co-founder and our mutual friend, Peter Merholz recently wrapped up what I'm describing as the first season of your podcast, Finding Our Way, and you and Peter had a conversation in that final episode where you synthesized the things you'd learned in the course of that first season. And you made a statement, you said, and I'm going to quote you back to you now, which is always nerve wracking! You said, "I think leaders are of necessity, orchestrators of systems. And systems instantiate knowledge as information architecture within them. So, the IA that gets embedded and coded, baked into your systems, becomes the way that the organization understands the world. And so, it is on the leader to imbue, infuse, enrich that IA with as complex and nuanced and understanding as they possibly can." There's a lot there... Jesse: I believe that statement. So, so that's a good test. Jorge: That's great. But I feel like there's a lot there to unpack and I wanted to talk about it with you. The context of the podcast, Finding Our Way, is about design leadership, but this strikes me as a statement that might apply to leaders in any field. Jesse: I believe that's true. I believe that any leader, anyone who gives direction to people in an organization, is on some level a steward of the organization's understanding of the problems that the team is trying to solve. And that understanding — when that gets systematized -information architecture is systematized understanding; it takes the associations between ideas that give meaning to human endeavor, human behavior, the world, and makes that concrete in ways that systems can then use. So that knowledge, that insight, can be scaled. And a lot of organizations run into trouble when their information architectures internally don't match the nuances and the complexities of the problems that they're trying to solve. Either problems that they're trying to solve for users or problems that they're trying to solve as a business. Businesses are often getting caught flat footed by market trends that they didn't see coming because they weren't paying attention to the right signals. Because those signals weren't part of the fabric of their understanding of the problem that they were facing. So yes, absolutely. We were talking about the context of design leadership specifically because that's what the mission of that show is. But yeah, I completely agree with you. It is something that I think is a part of what leaders do for organizations is give shape to the ways that organizations, hold onto the ephemeral meaning that otherwise just lives in the heads of the people in the organization. Jorge: Now, this is something that has been happening for way longer than we've had the phrase 'information architecture' and I'm wondering if there are any practices, tools perhaps, that have been around for a while that might point to this function of leadership, as a going concern for leaders. Jesse: It's an interesting question because honestly, a lot of the sensitivity to this stuff, when you're talking about what data does an organization collect, what systems does an organization put in place to make sense of the data that it has collected — this kind of stuff often ends up being the domain of like IT and business analytics and people who do some serious number crunching, which is fine and great. And, in the case of a lot of organizations... I've done a lot of work with financial services organizations. Insurance companies are fascinating in this respect because the actuarial tables rule all, in that business. And the keepers of the actuarial tables really are, expressing a point of view about what constitutes risk in the world. Jorge: And that is a formal structure of information that is stewarded by someone in the org, right? Jesse: Yeah. It's the foundation of the business. If your actuarial tables, as an insurance company, don't reflect the reality of things, then you're a bad insurance company, because you're likely to take on risks that you shouldn't. Jorge: What this implies for folks who are either in positions of leadership or aspiring to be in such positions, is that A) they need to embrace systems thinking, right? A systemic perspective of the work. And the other is that it would behoove them to look for the structures that best articulate the core of the business somehow. And there are formal information structures in a lot of organizations. You've pointed out that in the case of insurance, they're very manifest, but what you're saying there resonates for me in other fields as well. Jesse: Yeah. It's definitely something that I saw in my consulting career across, a lot of different kinds of organizations. I feel like every organization has its own sort of arbiter of truth, internally. I think one thing that we've been doing for a long time as UX practitioners, or at least, one thing that we often did as UX consultants was encourage the leaders that we were working with to step into storytelling as a tool to be able to make their case for what they wanted to do from a design perspective. Storytelling is a sense-making activity. It's a way of giving people an understanding of the world. It's very similar to information architecture in that way. So, for leaders of any stripe, whether you're leading a design team or whether you're leading any other kind of team, to take a step back and ask myself, "Where am I the sense-maker for the organization? Where am I the one who is interpreting and giving meaning to information?" And sometimes that is happening largely in Slack or emails to the team or other kinds of communications, and sometimes that's happening in the context of more formal data structures like you and I have been talking about. So, if the leader is noticing and attending to sense-making as a core part of the value that they bring to the organization as a leader, then they can look across their communications and the various pools of data that they may be responsible for tending and to interpret what they're doing in terms of creating more robust and more nuanced and more accurate information structures. Jorge: I'm hearing two things there. One is that leaders need to have the wherewithal to understand the organization, its context, its goals, its way of being in the world — understand it in some kind of systematic way. And the other thing I'm hearing is that they also need to be able to reflect that understanding back to the organization — through things like stories — in ways that affect how the team understands what they're doing, basically. Jesse: Yes. It gives meaning to the team's activities by placing those activities in a larger frame — a larger context. IA as MacGuffin Jorge: In my experience in interacting with teams and organizations and their leadership, I get the sense that these two functions — the "let's first structure the environment for ourselves, and then, let's think about how we share that structure with others" — they're happening, to greater or lesser degrees, in different organizations. But they're happening somewhat informally. Like, I haven't seen too many processes to say, "let's now draw up the information architecture for what we're doing here." Usually, when people talk about information architecture, it happens in the context of redesigning the website or making changes to the navigation structure of our apps or what have you. And in some ways, those projects end up being kind of MacGuffin for these deeper conversations that need to happen. And I'm wondering if there's a way to overcome that gap where we do information architecture more explicitly in service of having the organization understand itself better, or the team understand itself better and its role. Jesse: Yes. I have done work like that in the guise of process work, that engaging with a team, trying to understand what the different elements of the team are, what each element of the team is intended to accomplish, how those pieces are supposed to work together. In order to engineer any kind of a process like that, that has to be rooted in an understanding at a conceptual level of what are the factors that go into play in producing whatever the team is there to produce. Or achieving whatever the team is attempting to achieve. And how are you making sure that all those factors are accounted for? And how are you setting priorities among those things? These are all decisions that inform the process work, but that's not the process work. That's the IA work that underlies the process work. Jorge: Is this more of a top-down or a bottom-up effort? Jesse: I think of it as being more of a top-down effort, just because I am... I've been thinking a lot about stewardship as one of the elements of leadership that we don't really talk about. Which is that you have a group of people and a set of resources in your care as a leader. And that creates certain obligations from my perspective, on you as a leader, to ensure that you pass those things along to the next leader in the healthiest possible state that you can. And that means looking out for your team. It also means looking out for your processes. It also means looking out for your systems. And it also means looking out for that deep, underlying understanding that drives all of those things. I mean, where are leaders doing that information architecture work right now? I'd say they're doing it every time they structure a document that presents to their executive leadership what they want to try to accomplish with their work. Jorge: What that hints at — to me at least — is the fact that this storytelling function that you were talking about earlier — the part that has to do with sharing with the rest of the organization, the understanding that we have of our own understanding — that act of telling the story influences the understanding. It's like the two are related, right? Jesse: Yeah. Jorge: There's a feedback cycle happening, where you put it out there, you say, "well, this is how we see things." And maybe your peers and other groups might say, "no, it's not like that at all. From our perspective, it looks like this!" And that tweaks your own architecture, no? Jesse: Yeah, I mean when we talk about cross-functional collaboration, what we're often talking about is the process of aligning the differing information architectures. The differing models understanding of the problem that these cross-functional teams have. That the design team has one understanding of a problem, technology team has a different understanding of the problem, business folks have a third different understanding of the problem. These things need to be reconciled in order for those teams to move toward a common goal together. So, we don't end up with the design team is designing a car, but the engineering team is building a submarine while the business folks have sold to the senior leadership that we're building an airplane! Jorge: This is such important work, and it strikes me — just in hearing you describe it — that it's something that happens often as a side effect of other initiatives. It's not like you set out to explicitly build that understanding and compare the delta with the understanding of that other org. It's more that both of you are tasked with collaborating on something and the process of collaboration is what surfaces these distinctions. Jesse: It forces it! Yeah. You're not really doing it as a separate explicit step because it's part of everything you have to do as a leader, in a lot of ways. Leadership as a design problem Jorge: It feels to me like we're talking kind of in the abstract when we talk about these understandings. And when we say that somebody is presenting to their colleagues, what might come to mind is something like a slide deck, right? And folks tend to gravitate towards things that they can see and understand. And the slide deck might be the manifestation of this understanding, but it's not... it might not be its purest expression. And I'm thinking of things like concept maps, where we map out our understanding of a domain, just not even for sharing with others, but to understand it ourselves. And I'm wondering if in the process of stewarding this understanding of who we are, what we do, what our role is, how we're structured, what our processes are... I'm wondering if there are artifacts that could embody that kind of abstract understanding? Jesse: I think so much of it depends on the leader. And I feel like what you're reaching for, or suggesting, is a mode of leadership that is really kind of an IA-centered or an IA-driven leader. And that's a very interesting idea to me. I haven't met one. You know, I would say I have met some leaders who, because of their experience with collaborative ideation processes, are used to getting their ideas out in a way that is still abstract. You talked about concept maps. That's a great example. Mind mapping is a tool that I've seen business leaders use. That is definitely an information architecture tool. You're doing an IA process when you're engaging with mind mapping. But they wouldn't necessarily think of that as IA work. And they don't necessarily make it central to how they analyze problems or make decisions. The people that I've worked with who have been those kinds of leadership roles tend to be a little bit more constrained and not have formal tools for getting their ideas out. They just communicate. And they do it in the context of structuring and organizing their communications. And a lot of times, that is what is foisted upon them by the communications culture of the organization. I have worked with organizations where there were such strong cultural... Taboos around what you could and could not do in the context of a slide deck. Where, you know, like I had worked with an organization, for example, where if you had anything that was going to the board of directors, the Deck had to follow a very specific structure and format. And if your idea needed more than three to five basic sections to express that idea, your idea was not ready for the board of directors. Because they were consuming so much content from across a very large organization, they needed everything encapsulated and summarized and standardized so that they could make the decisions that they had to make. But what that forced on the entire organization was a communication style that drove out nuance. Drove out conversation. Drove out a lot of what you're talking about, which is that moment to moment flexibility in the decision-making process that you know, for a lot of decisions is utterly necessary. Jorge: Yeah, it comes back to this notion of top-down versus bottom-up, right? Because the implication there is that there is a level of nuance that is inappropriate for folks at this level. And that's a questionable stance, I think. Jesse: Yeah. Jorge: So, you advise leaders, you advise folks who are stepping into leadership. How would someone who is either in a leadership position or looking to get into leadership, how could they develop these particular muscles? Jesse: The way that I talk to folks about design leadership, who have come from a design background -that is to say they've been doing design work — is that leadership is just another design problem. And you're working with different materials and you're working toward different outcomes and you're having to follow different principles, but the task is the same task. It is a creative problem-solving task. It is a systems-thinking task, as a leader. So, looking at the ways that you're already doing that systems-thinking, the ways in which you already doing that architecture for yourself in the work that you're already doing, and those will be your strengths. And those will be the pillars that you can lean on that are going to support your work as a leader going forward. They will evolve and they will not look like what they looked like when you were doing content inventories or task flows or whatever other artifacts you might've been working on as a designer. But the skill set that you're building is the same skill set. Jorge: So, it's in you, you just have to recognize it as such, and build into it. Which is kind of what we've been talking about, right? Getting the sensitivity to read the environment and articulate it in a structured way. Jesse: And also, to remain true to your own perspective. You know, I see a lot of people who step into leadership for the first time, and they start trying to emulate what they've seen of other leaders. Which is a totally natural thing to do. It makes total sense. However, every effective leader leads from their own strengths and recognizes that those strengths are going to be different from the strengths of the people around them, and leverages that difference. And leaders who try to emulate modes of leadership that don't suit their natural abilities, they struggle. And they create a lot of hardship for themselves that they don't need to have if they could just believe that they already had the power. Because I believe they do. Closing Jorge: Well, that strikes me as a fabulous place for us to wrap this conversation. It's an empowering exhortation to folks to be themselves and develop their own powers. Thank you so much for that, Jesse. Where can folks follow up with you? Jesse: You can find me on Twitter. I'm @jjg. I'm also on LinkedIn from time to time these days. You can find our podcast, Finding Our Way, at findingourway.design, and you can find out more about my coaching practice at jessejamesgarrett.com. Jorge: Well, thank you so much, Jesse. It's been a real treat having you on the show. Jesse: Thanks, Jorge! It's been fun.
On this episode of the Quiet Light podcast, we have the opportunity to speak with Jesse Kaufman, the CEO and founder of Shipping Tree. Though Amazon sellers often use that company's fulfillment services, some people engage a third party. 3PL's can do everything from start to finish or they can merely be used as a prep center. Regardless of how you use a 3PL, there are ways to optimize your expenses. Tune in to hear our discussion about how to negotiate with a 3PL. Topics: The typical Shipping Tree client. Deciphering quotes from 3PLs. The best integration models for 3PLs. How using a 3PL can save money. Commerce zones. Different types of Amazon seller accounts. Resources: Shipping Tree Jesse@shippingtree.co Quiet Light Podcast@quietlightbrokerage.com Transcription: Mark: So within the world of Amazon FBA, a lot of sellers rely on Amazon's fulfillment services and simply ship all the product over there but there are other sellers who utilize a 3PL either to fulfill the product and do everything from beginning to end and there are also those that use it just as a prep center before sending it off to Amazon in a way to try and save on some of the fees. And I think we can all agree Amazon's fees for fulfillment are pretty high compared with a lot of other solutions out there. Joe, I know you had somebody on who owns a 3PL and you guys talked a lot about how to negotiate the rates with that 3PL and how you can optimize some of your expenses by using a 3PL as opposed to just sending everything carte blanche over to Amazon. Joe: Yeah, these are my favorite kind of podcast guests when they go on and they talk about everything that they do and give it all away for free on podcasts like this. He's not pitching their services. He's just like, if you're negotiating with a 3PL look for this, don't do this, throw that contract away, if you have recurring revenue shipments, this is how you save on your shipping cost. If you have a 3PL located in Southern California, here's the benefit; monetary benefit by way of example of shipping from Ohio and things of that nature. It was fascinating. We've had a lot of people over the years say hey can you recommend any 3PLs and that was the point of having this person on knowing that he would give it all away for free. I think it's going to be very helpful for those that currently have 3PL, very helpful for those that ship exclusively through FBA because it's convenient, and some of the benefits of having a 3PL for kitting, for doing so fulfill Prime to avoid what happened during the pandemic where there were delays from Amazon shipping because of shipping medical supplies first; all sorts of different things that I think will really help the current e-commerce business owners and those that want to buy improve their bottom line and improve their customer experience as well. Mark: Yeah, I think this is all about control, right? I think the pandemic is a great example. Those that were 100% reliant on Amazon often saw; many of those guys saw delays and disruptions in their supply chains and also their ability to fulfill orders. Those that were using 3PLs didn't because they had that outlet for everything. So this is an interesting topic and this is where a lot of ROI is made in acquisitions, is learning how to optimize the expense profile and especially on that Amazon side so I'm excited for this one. Joe: Me too and just as a teaser it gives away one example where I, based upon the numbers you gave me, probably added a million dollars in value to the company. Obviously a very large company but if it adds $10,000 or $100,000 in value just by doing little things that make a difference, it really adds up to the overall value so let's go listen. Joe: Hey folks Joe Valley here from Quiet Light Brokerage and Quiet Light Podcast. Thanks for joining us again. Today we're going to talk about 3PLs, how to save money on shipping, all sorts of different things in that regard. And today, we've got Jesse Kaufman from Shipping Tree. Jesse, welcome to the Quiet Light Podcast. Jesse: Thanks for having me. Joe: Good to have you here. As I said earlier, we don't do fancy introductions, so I don't have a big bio on you. No one knows you better than you so why don't you tell everybody listening who you are and what you do? Jesse: Yes, my name is Jesse and I'm the CEO and founder of Shipping Tree. A 3PL based in Los Angeles with facilities across the country. I'm Canadian and got my start in the fashion distribution business and quickly realized that the 3PL world wasn't where it should be, at least in North America, and that's why started Shipping Tree. Joe: And is your typical client an e-commerce client with lots of different SKUs like from your fashion background? Jesse: Yes, our typical client now are e-commerce direct to consumer-focused companies in the CPG supplement cosmetics space, actually. Joe: Wow. Okay, so lots of people picking, packing, and shipping. That's great. Jesse: Yeah. Joe: Okay, so let's jump into it. A lot of people; I've worked with 3PLs myself, I had a nutritional supplement company that I sold a decade ago if you can believe that; almost a decade ago, before I joined the Quiet Light team and I don't know if I negotiated the greatest deal with my 3PL because he was a friend of mine. Jesse: Impossible, yeah. Joe: We did recurring revenue shipments and the owner was a friend of mine and because of that probably either I got an amazing deal or I got a terrible deal; probably nothing in between. Jesse: You'll never know. Joe: I'll never know. No. And I was just going to go on the craziest side there but people do not need to hear that history. Let's talk about, first and foremost, what's the best approach to reaching out to a 3PL and not just simply accepting the boilerplate prices that you give or should they or is there a way that you can professionally negotiate so it's a really healthy deal for both parties? Jesse: Yeah, totally. So, I think most important in your 3PL search is kind of put as many feelers out there as you can, get your internal data together, and organize before you put out those feelers so you could give those prospective 3PLs the data they need to give you pricing quickly. Joe: What kind of data are we talking about? Jesse: It's really like shipment data. So like a pretty basic Shopify export of your orders that includes the dimensions and the units in the order. That should give any 3PL the ability to quote you really accurately. Then once you start getting those quotes back right away, it'll be pretty evident. Some 3PLs their quotes will have 30 line items. Others like mine and some of our closer competitors will have more in the neighborhood of three to five line items. So right away, all those 3PLs with 30 line items of potential charges throw those proposals in the garbage. There's no use even negotiating with those guys. The other ones with simple line items, three to five, maybe up to 10, those are the ones you want to focus on because, in my opinion, those are the ones that have the most merchant focused approach to the way they do business. And then areas where you can negotiate with 3PLs, in my experience, would be the initial processing fee on an order. So typically speaking, the most labor-intensive and expensive part of the work that we do are the individual picks. So 3PLs are rarely going to have margins to negotiate on the pick fees for your orders. Joe: And the pick is literally someone walking around and picking your product off the shelf and putting it on the proper conveyor belt to have the label put on. Jesse: Exactly. Joe: Okay. Jesse: Yeah, so you want to negotiate on the larger items on that list. So things like storage, processing fees, get rid of any minimums and stuff and kind of like frame your business as one that's even if you're just starting, it's ready to scale you're a smart team, you're going to scale it quickly, get rid of those minimums, focus on things like storage, processing, packaging, and you could kind of dwindle those down a little bit. Joe: Are there startup fees in most cases with 3PLs that I have to pay you $5,000 for the pleasure of doing business with you and that's just the setup fee and then it's going to be a monthly pack and ship fee? Jesse: If that comes across your desk, throw it in the garbage. Joe: Just throw it in the trash, okay. Jesse: Yeah, throw it in the trash. If you have really complex integration needs like an ERP system like NetSuite and a ton of different marketplaces, then there might be; you could expect some sort of integration fee and tech fees for that. But if you're just running like run of the mill, Amazon, Shopify, Walmart.com, maybe an accounting system; like all of that should be out of the box with the 3PL that you work with. Joe: Can you just dumb down what an integration fee is? Jesse: Yeah, so you're going to want your 3PL to plugin with whatever systems are running your business on the shopping cart side or the marketplace side of things and so you that you don't leak your sales channel. You want the 3PL to plug into there so data flows back automatically, your team has very little to do, that really is going to take the weight of shipping and fulfillment off your plate. And some companies charge for these integrations really like a setup fee, which isn't right because for Shopify, for example, we've built the integration already. We enter a couple of lines of code and the integration is done in five to 10 minutes so why would we charge you $500 for that? It's just not right. Joe: Good markup, $500 for five minutes of work. I like that. Jesse: I do like that markup, but we don't do it. Joe: Not if you want to keep the customer long term, I suppose, right? Jesse: Yeah. So, we've built out; and you want to find a 3PL that owns their tech stack. So what I mean is they kind of own their platform and they own the integrations. So we've built out these integrations so we've done the work upfront already and it's ready so we could just deploy it for the merchant. Joe: That makes a lot of sense because that's probably where the $500 charge comes from, is because they're using somebody else's software that somebody else is charging them and they're passing it on to the product owner. Jesse: Exactly, yeah. Joe: Is there a particular; I know that within Shopify, within the different websites platforms, there are different integrations for processing shipping. Is there a favorite integration that most 3PLs are comfortable with? And I cannot think for the life of me of a single one of them right now and I've used them before in the past but is there any particular integration that people like in terms of that processing of the order and having it ready to be shipped just to be shared with a 3PL or am I a little off track here? Jesse: A little off track. A little, so that's like if you just had a regular Shopify store, you would actually install the Shipping Tree app in your Shopify store. Joe: Okay, so you've got your app that you would install. Jesse: Yeah. Joe: Okay. Jesse: But you're talking about a product like Ship Station. Joe: Yes, that's the one I was trying to think of. Thank you. All right. Jesse: So Ship Station is great. We integrate with them also. Ship Station is great if you're selling on a ton of marketplaces like Etsy, Groupon; like if you're really marketplace heavy grand Ship Station is great because it brings all that in one place and then that's just one integration for us to run and manage. Joe: Okay, for people that are selling on Amazon is the largest marketplace and some of their own Shopify sales as well is there a benefit to using a 3PL to store inventory before shipping it off to Amazon, and do you provide those types of services? Jesse: Yeah, totally. So we do that a lot for our customers. We kind of run in parallel to Amazon like the verticals and the brands we serve and everyone needs to work with Amazon these days especially in CPG and cosmetics and supplements and stuff. So, yeah, our storage rates are generally cheaper than Amazon and more flexible. Joe: You can probably do kitting that Amazon's not doing, right? Jesse: Yeah, so we could help prep your stuff to go to Amazon. So if your factory isn't putting the Amazon FNSKU barcodes on the boxes we could do all that work for you. Joe: And you happen to be in Southern California so if it's coming off a boat it just have to go very far, which is kind of a strategic location, I would imagine. Jesse: Exactly, yeah. Joe: I had a guy named Rocky Cliburn on the podcast in the last, I don't know, maybe it was a year ago and Rocky was just this great buyer in his 60s. He was a general manager of car dealerships, if you can imagine, for his entire life and then he bought a jewelry business; an e-commerce jewelry business from Amanda here on the team. And Rocky and his daughter ran the business and within months improved the margins by like $8,000 to $10,000 a month by working with their fulfillment center in terms of shipping rates and packaging and things of that nature. You and I chatted prerecording here about saving on postage in terms of improving the value of a business and so you understand we always talk about the value of a business and it's really based upon profit, which is actually called seller's discretionary earnings. It's not about topline revenue. It's about what you get to keep. And a lot of folks don't focus on the 3PL potential savings as they prepare an eventual exit of their business. So how do you end up saving thousands of dollars on your shipping and postage like Rocky did if you're working with a 3PL, what kind of recommendations have you implemented for clients of yours? Jesse: That's a great point; a great question. So there's two things there. One is choosing the right shipping methods and another is the packaging that you're choosing. So I'll start with the packaging and for example, a jewelry company they might have one standard box size for all their orders just to they think it's a good solution that's like a catch-all. Every order ships in the same box so it either might be too big or it might be too small. If you optimize that, especially for smaller weight items, every ounce is almost 20 cents with the Postal Service. So if you could figure out a way to ship in a smaller box, maybe a more efficiently sized box, even though you think it might be it's a bigger inconvenience to have to source two different sized boxes or whatever it may be, you're going to knock 5%, 10% off your postage just right there optimizing for box size especially for orders under a pound. Joe: How much do boxes really weigh I mean if we're talking about the size of a shoebox? Jesse: So a shoebox is quite like half; almost half a pound, I would say. Joe: Okay, so if you can save a couple of ounces, you might be saving $400 or $500 a month if you're shipping a thousand orders a month or something like that. Jesse: Easy, yeah. Joe: Back in the day, when I was doing what most folks do that are listening, we had a fulfillment center up in Maine, which is just crazy because I was shipping all over the country but that's where I was from at the time. But they had a subcarrier. It wasn't the US Postal Service. They had somebody else that was sort of a cheaper version of that that would take it to the US Postal Service and then the US Postal Service would deliver it for that last mile or so. I forget what that's called but is that something that a lot of 3PLs can utilize and how do you find out about it if you're working with a 3PL now? Jesse: Yeah, so those are called shipping aggregators or an aggregator service. A lot of the major carriers offer that these days. The FedEx one is called Smart Post, and then there's a DHL product called DHL E-commerce. So those guys would pick up from your 3PL, bring it as close as they can kind of to the customer, then USPS finishes it. So those are good and bad. They're great for saving money. They're bad for making first impressions. Joe: So they take a little longer to ship, right? Jesse: Yeah, exactly because there's more touchpoints. But I think what we spoke about was; like we have a lot of subscription-based companies. Joe: I think we did that. I think that's what we did, is did it on the recurring revenue aspect of it where it didn't need to be there in two days, you could get it in five. Jesse: Exactly. Yeah, so we could set it up. And always look for this in a 3PL to have flexibility with mapping your shipping methods. It's really important that they don't just like put all your orders like this is it, this is what you have to use because we work with all the carriers. We probably have over 100 available methods and we work with our customers to make sure they're using the best ones. So for a subscription-based company, that first-order should go out with like a fairly premium single carrier option like USPS Priority Mail or FedEx Ground or whatever it may be so that is quick and the tracking is seamless. And then once they get into that subscription funnel; the customer, you could set it up programmatically so that instead of the order shipping on the anniversary date, you ship the order like three days in advance and you use one of these slower and cheaper methods. So that way the order is going to arrive within one or two days of the correct window for the subscription renewal, you're going to save easily 30%, 40% on your postage that way, and yeah, everyone is happy. Joe: That could certainly add up, that's for sure. That subcarrier method, is there tracking with it as well or not? Jesse: There is tracking, but it's known to go dark the tracking sometimes. Joe: Okay. Jesse: It's not as reliable as a single carrier because yeah. Joe: Okay, do you have; actually location, does it really matter? As I just said a few minutes ago, my fulfillment center was up in Maine. I was shipping all over the country. Jesse: That might be the worst place, Sanford Fulfillment Center. Joe: Oh really? Okay [INAUDIBLE 00:19:30.4]. Why would that be the worst? Is it just zone wise is the best place inside of the country or is the best place in Southern California where you are? Jesse: Okay, so if you could only choose one fulfillment center or one location, middle of the country is best unless obviously, all your customers are west. Like, if you're a surf brand and all your customers are on the West Coast choose a West Coast 3PL. But if you're just a normal run of the mill brand and you could only have one facility choose something in the middle, that way shipments are never really going to go to the outer edges of the zone map. So if you just Google search a zone map, the country will be split up into kind of columns like a heat map with the further you go, the further the zone and it goes up to nine zones. If you're in the middle of the country, the furthest zone is like six or seven possibly. And so with Maine, the reason why Maine is not so great is New York, historically one of the biggest population centers in terms of e-commerce orders going to that area, that's a zone two or three for Maine. So you're not even getting the benefit of being that close to New York geographically and then everything in L.A. is a zone nine. Joe: Let's talk dollars, though. Jesse: Yeah. Joe: And you've seen this with clients that you've brought in. How much are we talking about? If somebody is; and I know it's hard to quantify, so maybe we're only talking percentages but… Jesse: I could give you an example. Joe: Please. Yeah. Thank you. Jesse: Yeah. So we opened our facility in Ohio last year and we had a customer; one of our better customers, the supplements company, they were shipping everything out of our L.A. warehouse, obviously. Right away they probably spent close to $100,000 a month on postage. Joe: Okay. Jesse: Or they did when we were; they still do it [INAUDIBLE 00:21:34.4]. Right away when we started shipping out of Columbus and Los Angeles; so now you cut it down to furthest the package is going is zone four. Right away they started saving $15,000, $20,000 a month. Joe: Holy cow. Jesse: Not changing anything and the shipping speed… Joe: I hope everybody is listening to this far just because in that situation, $100,000 a month, even if all you spend is $10,000 a month on shipping, you're saving 15% to 20%. Jesse: Yeah. Joe: Go ahead. Jesse: And your customers are getting their orders quicker. Joe: So they're happier too; you're getting no return rates, higher customer satisfaction. Jesse: Yeah. Joe: The value that adds to the company in terms of customer satisfaction is huge but the value in terms of the sellability of the list price of the company for that one spending $100,000 and it drops to $80,000 a month, that's $240,000 of real cash saved on an annual basis. Jesse: Yeah. Joe: The size of that business, I'm going to guess maybe it's at a four-time multiple. They just added nearly a million dollars to the value of their company by saving $240,000 a year. That's that net worth. It's pretty incredible. So as whatever, it's just shipping, I'm going to focus on revenue, just stop focusing on revenue alone and look at some of these other things, because it's just math and logic saves a tremendous amount of money. That's awesome. What other tips and tricks do you have here Jesse? Come on, keep throwing them at us. Jesse: Yeah, so splitting up inventory; that's a big one. So using multiple facilities and find a company that has a few facilities and if you could afford it, there's a lot of fulfillment consultants out there who aren't terribly expensive at all. But it could be a really daunting process for brands going through their whatever they use Excel or the ERP or their inventory systems and be like, how am I going to split up the inventory between two warehouses? I don't know where demand is, all that stuff. There's people out there and software tools out there that could help figure that out for you. Joe: It's not something that a 3PL will do when they've got multiple centers or you'd refer them on to these consultants? Jesse: Yeah, we could do it. For inventory planning, we're building tools for that. It's really complicated to do and to do properly. It's not our core competency. And it's a big responsibility to do that properly. We could totally look at your shipping data and tell you how much you would be saving by using Facility A, Facility B, or them in conjunction. Joe: Okay, so splitting up inventory to the right fulfillment centers you're saving like your client did 15% to 20%. Jesse: Yeah. Joe: Any other sort of immediate thoughts come to mind in terms of somebody that either let's assume that they don't even have a 3PL now what should they; I know obviously you want them to go to ShippingTree.co and work with you but if they're already in a relationship with somebody, how do they improve that relationship and any other tips that you can think of? Jesse: So always think of your 3PL partner not just as another vendor, but really as a partner and part of your business and kind of put yourself in their shoes when it comes to the way you send them inventory, the way you keep them in the loop on sales or promotions you're running. Like really consider them like an outsource or your shipping department that's just outsourced. So if you were doing your fulfillment, you wouldn't run like a flash sale and then call down to the warehouse 20 minutes after the flash sale launch and be like, hey, buddy you have 15,000 orders coming down the pipe. You would tell your people in your own company a few days in advance. So do that with your 3PL, help make their jobs a little easier, send them stuff that's barcoded, clearly divided. We deal with a couple of hundred customers and you could imagine how many different items we have in the warehouse. All our merchants are really passionate, but like, I can't tell the difference between print like bandana print 1 and bandana print 2 you know? Joe: Yeah, we always hear stories of Amazon messing products up. I'm sure it happens in 3PLs as well. It's not you. Jesse: It happens but there's things you could do to mitigate that. Like work with them as a true partner and if you sense any pushback in trying to improve the relationship, I would look elsewhere. Joe: Yeah. Can 3PLs do fulfill by merchant with Amazon Prime? Jesse: Yeah. Joe: And are you in that situation or is it not a 3PL, in general, it's more of the product at the 3PL, how does that work? Jesse: Okay, so fulfilled by merchants we could do no problem. And then there's Seller Fulfilled Prime, which is that is actually on the merchants. They have to get their accounts authorized for Seller Fulfilled Prime. Joe: Even if they're using a 3PL? Jesse: Yeah, so their specific Amazon seller account has to be authorized for Seller Fulfilled Prime. Joe: Is that a daunting task or something? Jesse: Yeah, it's at least 90 days, and yeah. Joe: And what's the benefit to that in your opinion? Jesse: So the benefit there is you get the prime badge on your Amazon listings, you kind of get all the benefits of winning the buy box that you'd get with using FBA but the package could be sent out in your own custom packaging. You control the whole process and generally, it's a little cheaper than Amazon; storage wise and stuff like that. Joe: You still have to abide by the terms of services I would imagine. You still don't own the customer, even though you've got all the customer data minus the phone number, I suppose. Is there any advantage to doing Seller Fulfilled Prime using a 3PL in terms of customer data that you get to keep versus just using FBA? Jesse: I don't think so. It's more like a flexibility piece. Joe: Okay. Jesse: So those sellers that were set up with Seller Fulfilled Prime when COVID hit and FBA stopped allowing shipments in, they didn't skip a beat, they kept their Prime badge, all that stuff. Joe: Yeah, okay. Jesse: It's a little bit more secure. Joe: Having control as opposed to letting Amazon have full control of it, yeah. Okay. This has been great. We're up against the clock here, but this is fantastic stuff. I think that anybody out there listening needs to dig deeper into their expenses on the 3PL side. If all you're doing is fulfilled by Amazon, you might want to look at at least a 3PL like Shipping Tree to do kitting and prepping and getting it shipped off to Amazon so you're not paying exorbitant storage fees at Amazon and then as your offline Amazon sales grow running a Shopify side so on and so forth, I think is great to do. So any last-minute thoughts in terms of other things that people could do to benefit themselves with 3PL negotiations and working with them before we wrap this up? Jesse: No. Just be aware of this. Like I said I think the biggest red flag are those proposals you get back that are like two or three pages long with a ton of line items. That's going to be a headache of a relationship for you to manage. Find someone that keeps it simple for you. It's a complicated process. It's my job to simplify that for our merchant customers and find someone that will do that for you. Joe: I got you. Okay, how do folks reach you and your firm, Jesse? Jesse: If you're going to reach out you could email me directly Jesse@ShippingTree.co or go to our website and fill out the form there. Joe: Awesome. I appreciate your time. We'll look forward to a lot of folks reaching out to you as well. Jesse: Cool. Thanks, man. Take it easy.
Braze's Director of Data Jesse Tao and Tech Alliances Manager at Looker Erin Franz graciously break down BI tools and the value of data for the rest of us civilians. They walk through the marketization of data and the power of Looker blocks. TRANSCRIPT: [0:00:17] PJ: Hi there. This is PJ Bruno. Welcome back to Braze For Impact, your weekly tech industry discussed digest. And I'm thrilled today to have two very good friends of mine, Jesse Tao, our director of data, the man about data. What's the title Jesse? It's just data person? [0:00:33] Jesse: Well my Slack title is just data stuff. [0:00:35] PJ: Right. So we have Jesse Tao, data stuff- [0:00:38] Jesse: Official title is, Head of Data Strategy. [0:00:41] PJ: Head of Data Stuff, Jesse Tao, and also our very good friend joining us from Looker, that's Erin Franz. Hi Erin. [0:00:48] Erin: Hi. [0:00:49] PJ: Good to have you here. [0:00:50] Erin: Yeah, glad to be here. [0:00:51] PJ: How's the day been so far? You guys have been doing workshops right? [0:00:54] Erin: Yeah. Flew in last night. Just starting the day early, east coast time. Feeling great. [0:00:58] PJ: Awesome. Not too jet lagged yet? You're feeling good? [0:01:01] Erin: So far. [0:01:02] PJ: Hitting our stride. That's what I like to hear. So we're here today to talk about data, about insights. I'm sure as you two know, over the past 30 years there's been monumental strides in what that means to companies, and the value that it can add. So let's start really, really general, where we are today. Erin, can you speak to some patterns that we've since in data, since the beginning of it? I guess from relational, to non relational databases, to the kind of stuff that you work with right now? [0:01:34] Erin: Yeah. I mean, I can speak to ... Since I've joined Looker about four years ago, sort of how the landscape's changed and how we've seen sort of the product evolve with the technologies that have become available. So I think when Looker was founded six years ago or so, Redshift was just emerging as this modern analytical data warehouse. And those technologies didn't really exist before. And what this enabled, was the ability to actually expose large volumes of data across an organization in a way that multiple people could be accessing at the same time, and really using it to make data-driven decisions. Luckily, Looker took a bet on SQL being kind of this language of querying that would scale with all these different technologies that have come out. And luckily, that's been the case. With Redshift, we've also seen other databases like Snowflake and Google has BigQuery, that have really enabled organizations to become data-driven and self-serving when it comes to making decisions based on data. [0:02:37] PJ: And making it more accessible to people like me, like pedestrians, plebs, who just don't really understand kind of the technical side of data. It's like- [0:02:47] Erin: Exactly. [0:02:48] PJ: Democratizes it a bit. [0:02:49] Erin: Right. Making it accessible in a way that it's not just accessible to technical folks, to data analysts, to people who understand SQL and know how to code, to people who just want to click and drop and create reports and explore data on their own, products like Looker make that possible. [0:03:06] PJ: And Jesse you work with Looker pretty regularly at this point? [0:03:10] Jesse: Yes, almost every day. [0:03:11] PJ: Almost every day. And I mean we wouldn't call you a pedestrian, you're pretty deep in data, you understand it well enough. [0:03:16] Jesse: Yes I do. [0:03:17] PJ: Why don't you talk to us a little bit about the marketization of data. This is something that's- [0:03:22] Jesse: Yeah, you know, I think today we collect a lot of data. And in my opinion, data has more or less become a commodity now, rather than the hot topic. And what the hot topic of today is, it's insights. Because you're thinking about it, we're collecting a lot of data. We have data coming in from IOT and all these other sources, and most of the data that's collected, isn't being used. So, how useful is something that's sitting in the data of our house, kind of just collecting dust? So, very low value there. The value is from the insights, from actually analyzing the data, getting the data and figuring out what you want to do with it, to drive business decisions. And this is kind of where Braze comes in, and Looker comes in. We're providing the data and also providing the framework and the tools for people who are not using data, to get insights out of it and actually use that data. So I think in terms of the marketing pressure in the industry, we're moving ... We're going to still collect a lot of data, but more of the focus is going to be on how do we actually use that data faster and more efficiently? [0:04:21] PJ: Right. Because if you're not using that data and you're not taking action on it, you're going to be left in the dust, more or less, right? Is that the ... [0:04:27] Jesse: Yes. To put it in Marie Kondo terms, data just sitting there, brings us no joy. [0:04:33] PJ: And you're all about sparking joy. [0:04:36] Jesse: Sure. [0:04:38] PJ: All right then. Okay, well what sorts of data, insights are available to day that wasn't available in the past? Obviously this is kind of a big sweeping generalization, but what can we speak to currently? [0:04:49] Erin: I think some common themes that we've seen emerging are, people are collecting data from tools that they are using in their business, whether that's a Salesforce as a CRM's index as a support system and centralizing all that data in one place. So you're not just accessing one data set, not just your transactional data set, but also the data sets that define your whole business and your whole customer journey. So you're actually able to create kind of that 360 view of the customer that we're all sort of striving for, from as many sources as possible. And that's become possible because of these data warehousing technologies that are now available. [0:05:27] PJ: It's all about that 360 degree view these days, isn't it? [0:05:30] Jesse: Absolutely. [0:05:33] PJ: Because I'm still kind of just getting my feet wet with my understanding of the eco system of products right? You have your attribution, you have your CDPs, Braze is in there somewhere- [0:05:44] Jesse: Mm-hmm (affirmative)- [0:05:46] PJ: Engagement. So Looker is the analysis, it's less the visualization and more the business intelligence right? Because I feel like on our call, we talked a little bit, it's not just graphs right? [0:05:57] Erin: Right. Part of it is graphs for sure- [0:06:00] PJ: Right. [0:06:00] Erin: You need to be able to visualize your data, but much more than that, of course we always say the starting point for Looker is a dashboard, or a visualization. You can really drill into that visualization, see the components that have built that. If you're technical, you can even see the SQL that is being written to the database to supply that result set. And then you can modify that report, you can drill down into the granular level data that's supplying the data for that visualization. So let's say you're looking at event count by day on your application, you can see what those events are just by clicking into one of those data points. [0:06:36] PJ: Gotcha. And that data, that belongs to the company effectively, or that belongs to the user? [0:06:45] Jesse: I have a point of view on that. And before I share my thoughts, I'll preface it by saying I'm not a lawyer, so do not use this as legal advice. [0:06:53] PJ: Okay, thanks for that. [0:06:55] Jesse: I think the data ultimately belongs to the end user, but the company is basically the custodian of that data. Because without the end user, there is no data but without the company, there's not way of collecting or storing that data. So, the company is more or less using, collecting that data on behalf of the user. They're creating some sort of value from it, either from messaging or personalization or just understanding the user a little bit better. Some way of using that data to create insights into the level of value to that user. But ultimately, it is that user's data and the user should own that data. I think that's the point of view that many countries and regulatory bodies are holding as well. If you look at GDPR as well as the upcoming California privacy laws, the focus is really on the end user and their ability to control the data that they collect, the accuracy of it and the right to be forgotten. So, I think there is a common theme where the view point is the end user owns the data whereas, the companies are the ones who are using it to provide value both to the user and to the marketplace. [0:08:03] PJ: That makes sense. And the California protection, that's going to happen at the end of this year right? [0:08:11] Jesse: I don't know the exact timeline. We'll have to refer to our legal team about that. [0:08:14] PJ: Okay. Well we can patch that up later if we need to. So Erin, let's dig more into Looker a little bit. What's the real differentiator for you guys? What do you guys kind of hold up as a torch? This is kind of who we are and what makes us stand out- [0:08:29] Erin: Yeah. [0:08:30] PJ: Amongst the other tools. [0:08:31] Erin: I think luckily, the core Looker product has been fundamentally the same since its inception. With the core differentiators being that it's entirely in database. So as we talked about, the ability to access all of the data, down to the granular ... Most granular level that you're collecting it and exposed that across your organization. And the way that we're able to do that while still providing standards governance, so users are not creating their own one-off definitions of revenue, something that's incredibly important to reporting, is through our modeling layer, which is called LookML. So that's where you define all the business logic that your end data consumers will be using, whether by just exposing them to pre-built dashboards, visualizations or having them build their own content. And the way this works, while still leveraging the database, is it's really just an abstraction of SQL, or the language that you're using to create those database investments. And then finally, it's a web-based modern application. So that makes it really easy to share, collaborate and extend into plenty of other users. We have a fully baked API where you can serve data from Looker elsewhere to bring it into the tools where you need it. [0:09:46] PJ: So LookML, you said it's your own language- [0:09:50] Erin: Yup. [0:09:50] PJ: It's built on another language- [0:09:53] Erin: Mm-hmm (affirmative)- [0:09:54] PJ: And so if you know LookML, it actually is useful outside of Looker as well. [0:09:59] Erin: It's proprietary to the product, but it's very ... What you're doing is modeling the components of SQL, which is a core skillset of any data analyst. It really just makes it easier because instead of writing one-off queries, you're writing the components of those queries so they can be reused, by not only the data analyst, but also by all the data consumers. [0:10:21] PJ: Gotcha. Cool. Well let's talk Looker Blocks. This is what I really want to get into because I first heard about it at LTR 2018, because we announced our first Looker Block right? [0:10:34] Jesse: Two Looker Blocks actually. [0:10:35] PJ: Thank you Jesse. Fact checking on the go. Do you want to talk about that? That was kind of a big release right? [0:10:43] Jesse: Yeah, it was a pretty big release because it was still pretty early on in our relationship with Looker but we saw the immediate value pretty early on, so we decided to move quickly in that direction. And I'll let Erin talk a little bit about what our Looker Blocks, but the two Looker Blocks that we released back in November, are based around our currents data export and it focuses on market engagement and user behavior. So marketing engagement on the Braze data side will be things like email sends, push opens and at message clicks, stuff like that. And user behavior includes things like session starts and app purchases, so the behaviors of the users. We take all of that information together to create really useful insights around how campaigns are performing, user retention, if campaigns are improving your driving purchases, things like that. [0:11:32] PJ: Gotcha. And Looker Blocks for those of us who don't actually know the definition- [0:11:39] Erin: Yeah. [0:11:39] PJ: Are basically ... [0:11:40] Erin: They're basically templates for LookML. So LookML is a text-base modeling language. So we can model expected data sources upfront. So, data sources that are going to have a common schema, so common tables, columns, fields, within that. We've created a bunch of these for sources that are commonly used across our customers like Salesforce, Zendesk, as I mentioned before, Google AdWords, Facebook ads. The sources we're seeing most often, and then also the sources that we want to model proactively with our partners like Braze. [0:12:12] PJ: Cool. And so these two Looker Blocks, these are the first of many. [0:12:16] Jesse: Yes. [0:12:18] PJ: Cool. I mean, do you know what's on deck? Do we know what's coming up or do we want to save that for our next episode? [0:12:24] Jesse: We can save that for the next episode, but I actually want to talk a little bit about why we decided to make these Looker Blocks. And I think it's because we saw in it, the common vision with our product, which is data agility, or what we call, data agility. And that means basically speed to insight for us. As I mentioned before, the value of data is not in the data itself, it's what you can do with it, and how you can actually gets insights out of it. And with Looker Blocks, it acts as a template where we are predefining all the data fields and relationships, and providing those fundamental building blocks for us and out customers to build on top of. So, what would historically take a data engineer or a BI developer weeks, days, potentially even months to model, we do all of that leg work for our customers so they can just drag and drop in those Looker Blocks and be ready to find insights within minutes or hours. [0:13:20] PJ: So that's huge. That's going to save time. [0:13:22] Jesse: Absolutely. [0:13:24] PJ: It's exciting. All right, let's move on down to data tech changing roles. How is data tech ... How is it changing the way people are doing their jobs and what will the change for real expectations be in the future? [0:13:39] Jesse: Sure. Now, I think that people are becoming a lot more data-driven, and thinking about how to both collect and use data in their every day lives. Well not just their every day lives, but every day professional lives. They're using data to not just justify their decisions, but also to understand what the implications are in areas that they may not have seen before. And I think that's going to be a point of differentiation for customers, for our companies, because if you can actually use the data in a very insightful way, you can understand more about your users, your competitors, the marketplace and be able to confidently act in a way that will set you apart. And I think in terms of the data collection, the aspect of privacy is going to be more and more important as well. As I mentioned before, there's GDPR, there's the California privacy laws. I think people are just going to be ... Sorry. I think people are going to have to be more careful about what they collect because in the past, you could collect everything. And now with the privacy breeches you've been seeing at big tech companies, big banks, people have to be careful about both what they're collecting and how they're using it. [0:14:49] PJ: What's the most insightful way you've collected data to make a decision about your life? Putting you on the spot Jesse. I'll start. [0:14:58] Jesse: Okay. [0:15:00] PJ: Mine will have to be using Rotten Tomatoes to decide to not watch movies. That's probably it. That's probably saved me several hours of viewing time. [0:15:11] Jesse: Okay. So I actually have a script that I write, that scrapes lottery websites for the winning numbers, as well as the pay out. And I modeled out something where something like Powerball or Mega Millions, the optimal time to buy is a jackpot of around 3.25 to 3.5 million because at that time, there are not so many buyers where you have to split the pot. So you basically maximize your payout that way. So, we have office lotto pools here and I don't really partake in them up until a certain point where I think there's a higher payout. [0:15:49] PJ: I'm going to keep that in mind Jesse. That's a good one. That was a really good one. [0:15:52] Erin: Yeah, saving time, stress. I'm more on the Rotten Tomatoes path. [0:15:59] PJ: Yeah? [0:15:59] Erin: Yeah. [0:16:00] PJ: Do your homework, do your reviews, leverage the data available. [0:16:03] Erin: I guess restaurants also. [0:16:05] PJ: Yup. Yeah. [0:16:07] Erin: Avid Yelp user. [0:16:08] PJ: I'm a latecomer- [0:16:09] Erin: Not a reviewer but- [0:16:10] PJ: Not a reviewer, right. I'm a voyeur. I hide in the comments and I watch. [0:16:13] Erin: Yes. [0:16:14] PJ: I'm a- [0:16:14] Jesse: Lurker. [0:16:15] PJ: What's that? [0:16:16] Jesse: A lurker. [0:16:16] PJ: I'm a lurker. I'm a lurker, that's right. [0:16:18] Erin: Yeah, I rely on those people who are letting people know their opinions. [0:16:22] PJ: And I'm a latecomer to Reddit actually. I kind of just joined the bandwagon because I needed information on a certain thing. I was like, wow, this isn't just funny comments, there's a lot of really useful information here. Who knew? Anyways, so what were some trends, some hot ideas in the last few years that really didn't deliver on its promise? What are some current trends or hot ideas you think do have promise in the future? Erin, you want to weigh in? [0:16:48] Erin: Well, getting back to technology here, I think that as companies starting becoming more digital, and they were collecting so much more data and they wanted a place to put it, a data lake, and I think you know, I don't know how long ago it was, but Hadoop technology has emerged as kind of this place where you could be putting all your log data, all of your transactional data, all of this data. And it was easy to collect potentially, but not easy to actually self serve. So you were collecting all this data, but you didn't know ... There was no way to expose it to the organization. So I think that these analytical data warehouses have really filled that void and actually made that possible. And we've only seen that within the past five years or so. [0:17:33] PJ: Can you tell me the different between a data warehouse and a data lake? Because I've heard data lake around this office over the past eight or nine months, and is there a big different that I'm missing? [0:17:45] Erin: I can give the high level and then I think Jesse might want to comment on the more details. But you can think of a data lake as more like a file system. So you're putting all these files of data in this place for storage, but that doesn't make it necessarily accessible to the people who need it. [0:18:02] PJ: But the warehouse, you can actually do more with it? [0:18:05] Erin: Right. In a more performant way. [0:18:07] Jesse: Yeah, I mean the way I would kind of think about it is a little bit more literally if you will. A data warehouse you can image as potentially a physical warehouse that you can just put anything in there. In this case, it's going to be data. And a data lake, you can think of as a warehouse that has a giant pool in it. All that data is kind of just swimming around in a, I wouldn't call it a liquid form, but there's ... It's potentially unstructured, it's very fluid, it's just there. [0:18:31] PJ: Makes sense. [0:18:32] Jesse: And then people can go into that data lake with buckets or whatever tool to extract the data that they need. [0:18:40] PJ: That's a good metaphor. And so data lakes versus data ponds, is there ... [0:18:45] Jesse: There have been some ... I've heard the term data ponds before- [0:18:48] PJ: Really? Okay. I thought I was just messing with you, but I guess I wasn't- [0:18:53] Jesse: No I've heard it before. I don't think we're currently using that though. [0:18:56] PJ: All right, Jesse, hot shot, will data proficiency be a core skill for talent in the future? What do you think? [0:19:05] Jesse: Yeah, I think absolutely. I think here at Braze, and just at other companies, just reading the news, you hear more and more about how companies try to be data-driven. If you just look at our job descriptions, by the way we're hiring, and job descriptions of other companies you see, the requirement of understanding the different data warehouses, technologies, how to use data. A move from Excel to more complicated analytics technologies like Looker for example, becoming more and more popular. So it's absolutely going to be more important in the future. And you know I think for data analysts, that's ... Their role has kind of changed over the recent years and will continue to change as well. I think for the data analysts that I see, it's moving more and more towards a full staff knowledge. So before, you would see people focusing on one element of the data pipeline, whereas analysts today tend to have more visibility over how to bring data in, how to clean it, how to do the app analysis and the visualization, everything. And I think there's going to be more focus on the domain knowledge as well because data and insights out of context, is not going to be terribly useful to the organization. So we need to know how to appropriately analyze and interpret enough information in a way that the business or the end users can actually use. Also, I think in terms of the marketplace, you're just going to see more and more technologies. Some better, some worse than others, within the visualization space. Looker is pretty new, they're a ... I would call them a challenger, again something encompassing the place and they're doing very well. But going further upstream, you're seeing a lot of new database, data warehouse technologies, a lot of new ETL technologies. So I think the data analysts of today and tomorrow, are just going to be more familiar with these technologies and how to use these technologies. And then flipping a little bit to the non technical people, so the end consumers of the data. I think you're going to see changes there as well, especially as data becomes more democratized, and easier to use and consume. We're definitely seeing a trend towards self service. So, drag and drop analysis of data rather than actually going into the data warehouse to write the code and analyze it. We're seeing more sophisticated alerting, so we know when data isn't looking the way we think it should be looking. And that's just going to allow people to move a lot more quickly and more confidently as they try out experiments, they do AB tests and iterate quickly. [0:21:37] PJ: Brave new world. [0:21:38] Jesse: Yes. [0:21:40] PJ: Erin, you want to weigh in? What does the future hold for Looker? You don't need to show your full hand. I know you guys have stuff. But anything you want to leave us with? [0:21:49] Erin: Yeah, I think beyond sort of self service, the core BI use case, Looker's really trying to position itself as a data platform. So, not just for internal analytics and reporting, but also serving data elsewhere to other applications, to deliver data where it needs to go, like the action hub integration that we built with Braze. So, basically connecting the dots when it comes to doing analysis and taking action on it. So, building your list of users you want to target a campaign to and not just having to export that and then upload it into a tool, but creating that link directly to that product you're using. [0:22:32] PJ: Awesome. Cool. A lot stuff to look forward to then. [0:22:35] Jesse: Mm-hmm (affirmative)- [0:22:35] PJ: Thank you guys so much for being here with me, and thank you guys for joining us. This has been Jesse Tao, Erin Franz and PJ Bruno. Happy visualizing. [0:22:45]
The Hermit's Lamp Podcast - A place for witches, hermits, mystics, healers, and seekers
Andrew and Jesse connect on this weeks episode to discuss their connect to Santeria and the Orishas. We see how these traditions influence us, our world, and our magick. If you're enjoying the podcast so far why not check out our Patreon. For just a few dollars an episode you'll get special perks and Patreon only episodes! You can do so here. If you want more of this in your life you can subscribe by RSS , iTunes, Stitcher, or email. Check out Jesse's store "Wolf and Goat" here, his podcast here, and his theatre work here. Thanks for joining the conversation. Please share the podcast to help us grow and change the world. Andrew ANDREW: [00:00:00] Welcome to The Hermit's Lamp podcast. I'm here today with Jesse Hathaway, who . . . I have a hard time describing exactly who Jesse is. Jesse does all sorts of traditional magical traditions in [00:00:15] the ATR, as well as, you know, being an author and creator of magical products and a participator in a whole bunch of other traditions as well. So, I'm just going to hand it over to Jesse and say hey, how [00:00:30] how would you introduce yourself here? JESSE: Hi. Wow. I think . . . You know, I'm not, I'm not a big fan of magical CVs as it is, but you know, I, summary-wise, [00:00:45] I guess, I'm an Olocha. I made Obatalá in the Cuban Lukumí Santería tradition. I am a Tata Quimbanda, which . . . I'm a practitioner of Brazilian . . . It's [00:01:00] an Afro-Brazilian sorcerers' tradition that is sometimes paired with Umbanda, or Candomblé. Sometimes people let it stand on its own. It's a Congolese-derived practice, and traditional [00:01:15] witchcraft has always been there for, you know, as long as I can consciously remember, into early teens and things like that. But I study whatever interests me. It doesn't mean I'm initiated in all those things; it doesn't mean I'm practicing [00:01:30] all those things, but I have a passion for magical traditions, folk magic, folklore. I have a huge love of Mexican curanderismo, which is a familial background, although I did not go into that as a [00:01:45] practitioner. And I think also just . . . I'm a babbler, is probably important for my CV as well, that, you know, some of these things, like curanderismo, culturally, you never called yourself [00:02:00] that thing; that was something the community called you. So, I guess in some ways whatever people call me is whatever they call me, and they can come to me for what they come to me. And the main thing is that I'm just trying to do as much training with elders and keep things going as I can. But yeah. [00:02:16] ANDREW: I think that's a really interesting point. You know? And maybe we can start with that. We . . . I mean, we were talking before we got on the line, right? And we were talking about, you know, these sort of questions of authority and [00:02:31] who gets to call oneself authority, you know, who's an expert in these traditions or an elder or even just, you know, an acknowledged practitioner, you know? And I think that this question of where [00:02:46] does the authority come from? And how does that happen sort of inside and outside of traditional practices is a really interesting point, right? JESSE: Yeah. ANDREW: You know, for example, you're talking about, you know, being a curandero, [00:03:01] like, that's not a thing that you call yourself. That's what other people would call you if they're going to call you that, right? JESSE: Yes. ANDREW: I think that that's really fascinating, and I think that we see a lot of change [00:03:16] around that, where traditionally everybody lived in the same place, right? Everybody generally didn't move around that much and people probably saw a person in that practice grow [00:03:31] up, experience their training, they saw that they got the nod from other people who are acknowledged as that, and at some point, they started taking on their own, you know, practice, right? But in the Internet age, right, [00:03:47] that looks more like a good Instagram account, maybe? JESSE: (laughing) ANDREW: You know, maybe a nice website. JESSE: Yeah. ANDREW: You know, what . . . like, I'm curious what you think about those evolutions and those changes that are going [00:04:02] on around that. JESSE: Yeah. I mean, the apprenticeship model, which . . . It's not a certificate model, right? It's something different, where you are under an apprenticeship, you are with the elder and [00:04:17] their clients see you training with their elder. You know, they . . . it's . . .The visibility is a very different thing. It's not just classes. It's not just, you know, herb walks, occasionally. You are the right hand [00:04:32] of that elder for a very long time. And they see you go from incompetence to competence to fluency, and you know, that kind of replacement for that elder if and when they pass is there. And [00:04:47] it's a very different model than what is done now. But even within, I think, the kind of Internet age, of, you know, teachers have dozens and dozens and dozens of students. I look at the Brazilian model of a tahero, where [00:05:02] there is going to be one pai de the santo, who is the head, doing everything. They're doing all the initiating, thousands of people, but each person has a yake baba care [spelling?] that's taking care of their needs that is more individualized in that way. But still, it's . . . [00:05:17] you lose your individuality when you train, and that part is, that sacrifice is very difficult, I think, for a lot of our very Western Internet-friendly minds about promoting individuality. How different you are, how a certain . . . [00:05:32] You know, "I'm studying this tradition," and the tradition is studying you, is part of the thing that we forget too. ANDREW: Well, and I think that it's part of the . . . part of the good training, you know, is learning how [00:05:47] to get out of the way and do the work, right? JESSE: Yeah. ANDREW: You know? Like the . . . you know, I think about the elder Olochas that I trained with and spent time with, or am at ceremonies with, right? And certainly, if there's a [00:06:02] junior person there to put, to open in the coconuts or whatever, they're going to do that, they're going to be like, "Hey, go do that, go mop the floor, go whatever." JESSE: Yeah. ANDREW: But also, if there's not, they're just going to grab the hammer and go, right? And, [00:06:17] you know, there are these funny things that come from that training and that experience. And, you know, opening coconuts is one of them. You know, I watched the people who are new, you know, in my house come and open coconuts, and, you know, I'm like, I always [00:06:32] look over like, "Oh, they're taking forever!" You know, not in a mean way, but just in a like, you know . . . And then, and that feeling of like, I can open a coconut in no time because I've done hundreds and hundreds of them now. JESSE: Yep. ANDREW: And, those subtle things that you would, [00:06:47] you know, you would see being in the space with somebody else . . . JESSE: Yes. ANDREW: Make that big difference, right? JESSE: Mm-hmm. Even the way the way that we mopped the way that . . . we call it watering your elders, you know, just [00:07:02] the, you have to . . . in a good way, not . . . I don't mean that in a . . . But the idea of culturally, like, I'm . . . Those of us that are more on the introverted side, you know, it's a lot to go and say hello to everyone. It's a lot to enter a room and to each person say hello. [00:07:17] It can be exhausting before the ritual even starts. ANDREW: Mm-hmm. JESSE: You know, you learn shorthands. Or you find ways to be able to enter into the social language that is needed to be able to access things. But, going around and asking everyone who's older [00:07:32] than you: "Do you want something to drink? Can I get you a coffee? Can I get you a water? Can I get you something?" Even if they say no, it's a lot, for whatever reason, that service-oriented side of things leaves . . . It works both ways in the sense that it allows people to introduce themselves to each other, in [00:07:47] a way that's not just small talk. But also, people see that you are trying to take care of people in the room, and make sure that everyone is comfortable. And it's an interesting side of things that you know . . . That's [00:08:02] not a critique; it is a critique, but of the Internet culture basis or the book-learned culture of not realizing that the book is still your teacher and it's a one-sided conversation that you don't get to necessarily appeal to the author and ask for clarification, but you didn't [00:08:17] teach yourself. You learned from a book. You didn't teach yourself, because there's a language that you are relying on that is built on clichés and allegories and metaphors and things like that. So, there's, there's . . . This idea of picking yourself up by your [00:08:32] bootstraps into a magical tradition is not quite necessarily the case even when you're doing it by yourself. And, and, if we believe that spirit is intervening, then spirit is also teaching us as well. And [00:08:47] how well we can refine that, our own inner ear, to listen to that, is also something there. In a community, you know, a community setting, people often ask in online groups, like what books can I read? Read the room, first, like [00:09:02] take the temperature of the room and listen, because, I mean, the best conversations happen at 2 a.m. after all the things are done for the day and the cook finally gets to sit down because the kitchen is shut. ANDREW: Sure. Or they're in there and you're talking to them instead of you know, rushing around. [00:09:17] JESSE: Yeah. ANDREW: And then they go, "Hey, come look at this thing that I'm going to do here," right? And even, even in the simplest of things like, you know, cooking the inyales right? Like just cooking the parts of the animals that go to the Orishas. There's all sorts [00:09:32] of stuff to learn about just even a simple thing like that, you know, and if you're engaged with the people and talking to them and have a relationship with them, then they're going to invite you in and be like, "Hey, you know, if you're looking for this, do this this way, or here's a good way to do it," [00:09:47] you know? Otherwise, you're just, you know, you can do it and it will serve the job but you're missing big swaths of the teaching, right? It's always the thing that I'm really aware of in my, you know, in my position as somebody in Toronto, far [00:10:02] away from regular practice, right? My . . . my knowledge is good. You know, my . . . I mean, there's always things to work on, but my fluency and some of those little details, I'm well aware that it's not as strong [00:10:17] as it would be if I was living somewhere where I got to just work more often, you know, because you can never learn those things from a book. Nobody ever thinks to talk about that. You know? Right? Unless you're in the room with the person and then you're watching them, like, "Hey, what was that? Why'd you put that in there? I didn't see [00:10:32] anybody do this before,” you know? JESSE: You know, you can read a book about running a marathon, but it's a very different thing to do it. ANDREW: Right? JESSE: And we talk about that all the time, of like, you know, watching, if someone doesn't know how to mop, and they say they're an active santero. You're like "Hmm, maybe not." But [00:10:48] there's this side of it, of, there's so much, there's different types of knowledge and the modern age promotes one type of knowledge, which is the facts of the, the history of that type of thing that can be transmitted via literature [00:11:03] in that way, in the written word and it's an interesting side of things, but it's very different when the body knows it, when the, when the ways of learning in the body are different from the head. And even . . . [00:11:18] So, it's an interesting side of, you know, really making sure if someone doesn't know how to do certain things, you train them and even, even, for example, my early years [00:11:33] as an Olocha. I come from a house of a lot of old elders. Like physically, they are more aged. And so even though I could be doing other things, they needed someone to lift the big water buckets and up [00:11:48] and down the stairs and do the heavy lifting and open the coconuts. So even though there were other tasks that I could be doing, I was doing the manual labor, because I was younger . . . ANDREW: Mm-hmm. JESSE: And able to do certain things that my amazing elder ladies could not. [00:12:03] And that's an interesting side of things too, because then they sent me out and like, you know, “Go to this house, and start studying with them a little bit here, and then come back and branch out,” so that I could get different experiences. And I think one of the things that's very interesting with . . . In the history of Santería, [00:12:19] is just because the houses started working with each other, things got very homogenized very quickly, through public opinion, both in a good and a bad way. There are variances to the way things are done, but the variances between the houses are actually pretty small. [00:12:34] You know, there's kind of a liturgized homogenized way to do things that is acceptable. And when you vary too much from that, both out of tradition or vary too much from that out of lack of tradition or lack of knowledge, you kind of get [00:12:49] pulled back into what is the acceptable practice . . . ANDREW: Mm-hmm. JESSE: And that's an interesting side of it. So, it's actually preserved a lot through public opinion through the fact that there's seven different lineages represented in a room because you invite [00:13:04] those people to work because in the early days you didn't get a choice on who was coming to work cause you needed people. So, you got anybody, any santero that was in New York City. ANDREW: Yeah. JESSE: "Come, work this thing!" And so, new traditions kind of, or at least parallel traditions start aligning, they start [00:13:19] coming into a common practice and adaptations have to happen for the modern age. You can't do certain things the way that was done in Cuba or in Nigeria. So, it's . . . Those modifications happen, and elders make those decisions. [00:13:34] When one person makes those decisions, it can get a little crazy. But when a community comes together and says, "How do we resolve this problem? How do we take care of this? Then there's more options, I think. ANDREW: Mm-hmm. For sure. Well, and I think that goes [00:13:49] back to . . . It goes back to kind of a couple questions around that. One is for me, I think that where there are differences in lineage, it's important to know what they are. Even if they're small. It's interesting, where there are lineage [00:14:34] differences, that I think it's really important to become aware of those and know what they are, right? You know, I mean, we are initiated into a lineage, and therefore if our lineage does it a certain way, we should do it that way. And you know, [00:14:49] in these different times, where you go, might go to different houses and do things in different ways, I think that it's important to respect, you know, the way other people do it and also know that when you're in your home, you do it a different way, right? Or when it's your event. But [00:15:04] I think it also creates a lot of unnecessary dialogue and drama, and I think that we see this in all the magical communities, right? At least every one that I've ever been in, which is more than a few. It's this thing of "Well, [00:15:19] we don't do it this way. Therefore, it must be wrong," right? JESSE: Yes. ANDREW: You know, "This is . . . this is not . . . I've never seen this; therefore, it must be wrong." And I think that, you know, it's such a such a sticky [00:15:34] topic, right? How do we understand what is tradition? What is traditional variance? How do we understand what is, what comes from experience, and what might be other groups' experience that we could integrate? JESSE: Mm-hmm. ANDREW: And how do [00:15:49] we . . . And how do we judge what is just, you know, manufactured garbage, right? JESSE: (laughs) ANDREW: To make a few bucks, you know? So. I don't know. What do you think? Give us, give us a guide here, give us some solid rules we can live by. JESSE: Because I'm the authority? (laughs) Authority of [00:16:04] that. ANDREW: Yeah, I'm giving you all the authority right here. Community of one gives it to you, Jesse! JESSE: Yeah, yeah. I think, obviously reliable or people that you can [00:16:19] confide in and ask opinions on that . . . The chain of eldership is really important and it's not just for this. You know, I don't, I don't support the complete submission to elder guru style where it allows for physical abuse or emotional abuse and that way . . . That is a [00:16:34] model that does exist and has existed but there is a possibility of an elder and mentor elder and minor model that allows for accessing [00:16:49] opinions that can contextualize things based in the knowledge that they have that is more than your own. ANDREW: Mm-hmm. JESSE: How do you, how do you modify? I think there's the side of it too, that's always interesting, [00:17:04] of when you don't recognize something, if you're secure in what you have, you don't attack the thing you don't know, you just look at it and cook. That's interesting. Let me see where this goes, and you have to wait. Gauge the point of when it seems off and [00:17:19] what is your agenda in making sure that it's correct or incorrect. ANDREW: Mm-hmm. JESSE: And that personal side of it, the, you know, this idea that there's objective . . . one tradition that was passed down from Adam and Eve, it gets a really, it doesn't serve us. And I . . . Certainly [00:17:34] within the ATRs, I mean, the differences between traditions, houses, the differences between Santería and Candomblé and different Orisha practices are huge. And at the same time, the [00:17:49] Orisha are very flexible in what they, what they say and do, and they're not going to sit there and nitpick, but there are ways, specifically, that the tradition has evolved, to make sure that Orisha comes, that Orisha is there, that is unique to each lineage, unique to each house, it has similarities [00:18:04] and commonalities and landmarks, you know, to . . . that are recognizable. But at the same time there's . . . I don't see elders get as upset about something that's off. [00:18:19] Just minorly off. They'll be like, "Oh, we don't do that," and don't worry about it because "come do it, we do it this way." ANDREW: Yeah. JESSE: I see a lot of people who are younger, get really pissed off about keeping tradition intact. ANDREW: And I've talked to elders who talk about that's [00:18:34] how they felt when they were younger. Right? And be like, "Oh, when I was like 18, I was so mad about all these things. But now I'm like, well, I can see both sides, you know." JESSE: Yeah. And it's the question of like, do you spend all the time stamping the thing out that you don't like [00:18:49] or do you spend time investing into the model that you feel is more correct and more profitable for people to follow? ANDREW: Mm-hmm. JESSE: And, you know, fighting for what you want to see as opposed to what you don't want to see. And there's merits on both sides. I think, personally. [00:19:04] You know, when is it that we don't . . . We try not to innovate a lot of times in ATRs, right? Of like, you innovate through necessity only. ANDREW: Mm-hmm. JESSE: And, a temporary thing that you're still asking clarification on from elders or spirits [00:19:19] or things like this, but you try to innovate as little because otherwise it's not necessarily what you're practicing anymore. ANDREW: Mm-hmm. JESSE: It's not recognizable. It's not recognizable. And has its own thing. Certainly. [00:19:35] Opinions change as you get older too, and you . . . More experience, it's not just older. What is the Chinua Achebe quote of "Old age is respected and wisdom is revered"? The same thing is similar in our models here of, like, you know, someone who has worked the room for [00:19:50] five years consistently at the foot of an elder is going to know more than someone who's 20 years old and has never worked the room, as much, or worked it once a year. Someone who births a lot of Orisha constantly or is taking a lot of clients is going to have a different opinion of how things function because they realize, [00:20:05] "I don't do it this way because it gets in the way of blah blah blah blah blah blah blah blah." ANDREW: Mm-hmm. JESSE: Versus, if it's your first time giving, it's like a first-time child. You're going to make a bunch of mistakes. You're going to realize you might put a lot of effort into things that you won't necessarily do on the fourth child down . . . ANDREW: Yeah. JESSE: Because important . . . and that practical [00:20:20] experience, you know, sometimes we just have to suffer through our own inexperience and be humble and keep going to elders and asking opinions and seeing, keeping our eyes open as to what is being done. And if we're in a solitary tradition where it's [00:20:35] less likely that we're going to have an elder who's going to speak to our direct needs, then learn from other things around you that you can, that you admire and can pull in. You know, it's really hard to reinvent the wheel constantly. ANDREW: Well, I think it's . . . You know, I think it's really interesting because [00:20:50] when I . . . The first store that I read out of have a predominantly Afro-Caribbean clientele. JESSE: Mm-hmm. ANDREW: And, you know, so I was . . . Although I was getting involved in [00:21:05] the Orisha traditions and stuff at that point, I didn't have a ton of experience at all and, and I wasn't initiated as a priest, so it was just mostly my own development that I was focused on. But I, you know, I had done a ton of ceremonial work and you know, initiations [00:21:20] along all those lines, and one of the things that was really interesting was, I would end up having these conversations with you know, spiritual Baptist priests and, you know, other people, and they're like, "You really understand," you know, [00:21:35] whatever it was that they were doing, right? They would always say, "You really understand our tradition. You really understand this. You really understand the African mindset," or whatever, and I understand that they felt that that was true. But I think that what I knew was, what I [00:21:50] actually really understand is magic and I understand that there are generally fundamental things that are kind of true across the board if you're really engaged at a deep level and not, not sort of in the "there's only one faith" [00:22:06] or "there's only one source" or any of that kind of like, you know, Victorian colonial nonsense, right? JESSE: Yeah. ANDREW: But in the sense that when you understand that spirits are real, and you have the capacity to genuinely speak with spirits and [00:22:21] you're going to work with materials, nature, candles, whatever, offerings. Then, then though the surface of those things, or the tradition and lineage piece changes those, there is a fundamental mindset [00:22:36] that, that's there, kind of around the world around those kinds of things. And once you get that, then you can relate at that place, right? Which is completely different than sort of going in and sort of saying, you know, as I've certainly seen other people do, "Well, [00:22:51] yeah, exactly, I know this tradition and the spirits gave it to me and therefore I am able to do this and that and whatever, it's like, no no, no. I know how to talk to spirits. And in fact, often even people, spirits of other people's traditions might lean in a bit through a reading and nudge me in [00:23:06] a given direction. But that's not the same as understanding their traditions or whatever, right? JESSE: Yeah. Absolutely. The . . . I think that when we're talking about fluency, and magical fluency, we're talking about a practicality, as far as how [00:23:21] to utilize those things in everyday life, and that, that is something that is, I think, palpable when someone knows and can give practical advice, practical actions to achieve certain things, no matter the, no matter the tradition. And [00:23:36] certainly, when it's still theory in someone's head and less pragmatic, you can tell that too. There can be a struggle to articulate something. What are the next steps? And where do you go from here? And we can [00:23:51] talk about cosmology and philosophy which differ from person to person, let alone town to town, or tradition to tradition. ANDREW: Yeah. JESSE: And those finer points, but the practicality of it, that is, that's something different. You have to be somewhat fluent in order to give [00:24:06] good practical advice on how to move forward, and parroting something is, you know, you first learned by saying what you know, and going off of what you've seen, but the more you can expose yourself to, the more people's styles, you'll start to learn different ways of approaching things. [00:24:21] And certainly, I'm being reminded of a computer search parameter [00:24:36] recently. That was . . . The issue with diagnostic tools from computers or trying to diagnose illness and things like this, is that they're not programmed to look for something that isn't there. ANDREW: Hmm. JESSE: And this is something that humans can still do very well in that . . . not [00:24:51] just looking for the problem out of the common, of the sets of things are there, but to have a revelation of what could still be needed by the person, not necessarily . . . You know, when someone comes for a reading, there, it's not just their conscious problems we're talking about. We're trying to look and [00:25:06] bring those things that are unconscious to the surface too, to see what is actually the root of something that needs to be addressed, and those things come from having a good foundation in the basics, in order to . . . You [00:25:21] know, you have to do primary colors before you start doing secondary colors and understanding what those things are. You can't mix secondary colors trying to get primary colors. You still have to know what that, that order is, and I think it's very similar in magic. You know, there's basic advice on things and [00:25:36] some people will give out the basic like, you know, here's an uncrossing. Here's a, here's a love drawing, here's a bend over type of working, and those are, those are set vocabularies and other people [00:25:51] might tell you to go light a candle at the base of this tree and the spirit is going to take care of it. And that's the model that they were using, and both are pragmatic in this sense, but I . . . I wonder how much materialism [00:26:06] still enters in, the kind of Scientific Revolution atheist materialism that sneaks in because that is the paradigm as Westerners that we are raised in, you know, there's some variance in that and based on familial upbringing and religious upbringing. But the idea that spirit [00:26:23] is not necessarily tangible in the same way and it is actually affecting the materia to do the thing is a less popular model. And it's interesting now, like once you get introduced to the concept [00:26:38] of a charged statue or something like that, people want to put loads and everything in. They don't necessarily know what goes in it. They want to know, "Why, why do I put these things there? Am I putting this there to symbolize this?" Whereas in spirit-based traditions the spirit might possess someone, and it could put [00:26:53] anything it wants in that statue and breathe on it or splash it with whatever and now it's charged. It doesn't necessarily have a logic that we can understand as to why it picked that item to represent that thing because it's not representation. It's [having?] something and that is a battery of power that is being used. [00:27:08] Not, did you have all 732 exact ingredients . . . ANDREW: Exactly. JESSE: To put in. That spirit could go for a walk and pull a clump of herbs and give you one of the most powerful baths you've ever had. Whereas if you try and duplicate it with those same herbs, it's not going to be the same, because you're not . . . ANDREW: Yeah. I was [00:27:23] talking with somebody in the store recently about . . . they were asking me where I get the crystals that I buy, and about the mining practices, you know, and I think that those, those are really important questions, you know, and the short answer is about [00:27:38] half of what I have, what I sell, I know, I know pretty clearly where it comes from, and short of, you know, hopping on a plane and going to the mine, I feel like the people I'm buying from, who are buying directly from the miners, [00:27:53] you know, I believe them, you know. It's the best we can do in this in this day and age, you know. A bunch of the other stuff, I'm far less clear about where that comes from and, and you know, I would like to reduce that [00:28:08] amount, you know, to be clearer that there's no human rights violations and horrible environmental destruction and so on. But it's, but it's complicated and it's difficult and you know in this industry for sure, and in tons of industries. They [00:28:23] were asking me about the magical influence of where, of where something comes from and how it's handled along the way and all of these kinds of things, right? Is the stone that you know where and how it was [00:28:38] mined different than the stone where you don't? And, and when I was talking with them about it, I mean, certainly I have my own political and social view on that stuff, which is, I think that the stuff that is harvested [00:28:53] with respect is always, is always better whenever we can manage it. You know, whenever I harvest things, I always harvest them with a lot of respect. And I think that that's a great thing. But I think that there's kind of a, also another question mixed in that, [00:29:08] which is, where does the actual magic of what you're doing reside, right? And in the context of a stone, right? Is it concretely in the minerals and the energy of that? And [00:29:23] I think that that's, that's part of it, you know, there there's really interesting crystal books that talk about the, you know, how the crystals form and how that magic, how the energy of that relates to their sort of fundamental crystalline structure that varies from different stones [00:29:38] and you know, you've got color and you've got different participations and all that kind of stuff. And what other things activate this, right? JESSE: Mm-hmm. ANDREW: And you know, there's the power of the thing in and of itself, but kind of as you're saying, there's also what the spirit might want, right? JESSE: Yes. ANDREW: Like, you know, if I'm working with, you [00:31:37] know, one of my guides, and my guide says, you know, grab me, grab me a piece of iron pyrite and let's do this with it. JESSE: Mm-hmm. ANDREW: Certainly the . . . certainly the element is important, but the activation of that particular spirit through that element is [00:31:52] way more important in that equation, probably. You know, the actual force through which the spirit makes the change or consecrates that thing, you know, and consecrating a statue is a good example of that, right? You know. That is the force of the spirit making [00:32:07] something and putting it together and anchoring it. And then we get into . . . And then sort of the third thing that I see which is related but not exactly the same which is you know, especially with things like plants and stuff like that, right? There is also the [00:32:22] living entity which is that plant in and of itself right and not necessarily just the specific one that you're working with, but the sort of deeper energy of a given, you know, a given plant in the world, you know, like [00:32:37] ayahuasca or other things. You know, people, you know often talk about that as an entity that wants to return to the world, but I think that that's actually fundamentally true of the bow trees in the front of my shop and, you know my crown [00:32:52] of thorns plant, and all of those things, and it knows I'm definitely, in the way that I'm working with them in the space, connecting with the collective entity of that plant, you know? And so, I think that this [00:33:07] this idea of how are we working and what are we doing is so interesting and I think it's something that people don't really see those distinctions. I don't hear them talked about, you know? JESSE: Mm-hmm. ANDREW: So, I'm curious what you think about them. JESSE: Yeah, [00:33:23] I . . . something that comes to mind. I have, as long as I've been crowned actually, so a dozen years, been working with an experimental Theater Company here in New York City called Dzieci. And it's [00:33:38] using theater as a tool to investigate something else. But that's [lost audio at 33:45?] is unique to each person. But we're talking about investigating the sacred through the tool Le Théâtre. Through the means of theater. [00:33:53] And this intentionality, this question of intentionality is quite interesting to explore. And a question that gets posed a lot by the director, and then as we start something, is when does [00:34:08] the ritual begin? ANDREW: Mm. JESSE: And, is it when you have the audience fully there and in a theater context and the play starts? Well, no, it started long before that with the rehearsal process and then again, when did it start before that? And the question is when you bring [00:34:23] it . . . You know, for me, the answer and it seems to be a common thought on this, is when you bring awareness to it. ANDREW: Mm. JESSE: And so, if I know that I'm doing an important ritual next week and every day I'm waking up going, "I'm doing this next week. What can I do today to manifest that more [00:34:38] smoothly and make sure?" Then making sure all your bills are paid and you know, the bag lunches are done for the day and everything, that becomes part of the ritual. ANDREW: Mm-hmm. JESSE: And there's this interesting question of intentionality, when you know that something is ethically harvested [00:34:53] and you're going to the store but you're in a tizzy and distracted going to the store and you're not present when you're picking up the crystal and you're putting it on the thing and you know, talking on the cell phone and looking at things. What are you doing to destroy the intentionality of that good harvest act? ANDREW: Mm-hmm. JESSE: I mean there's that side of it too, that's always interesting to me, of, [00:35:08] you know, you can have good ingredients prepared by bad chefs. ANDREW: Yep. JESSE: And you can get shitty ingredients prepared by expert chefs that still taste better. You can have ingredients, you can have a horrible angry chef prepare something masterfully because they know how to treat the [00:35:23] food and maybe they're compartmentalizing their emotion. Maybe they're not. They're . . . that missing ingredient of grandmother love that goes into the cookies: Does it make it taste better? Does it not? And you know, it is, I think for all of us, the question of intentionality is an interesting side of it [00:35:38] of what are we bringing to it? And how we contributing to these seeds? You know, I think, I like to look at things as seeds of potential . . . ANDREW: Mm-hmm. JESSE: And certain things allow them to mature and grow. The side of [00:35:53] it of looking at: What is it that that makes something work? What is it that allows something to happen? [00:36:08] I think anybody that can say definitively is selling something . . . ANDREW: (chuckles) And they probably have a great brand name trademark . . . JESSE: Absolutely! ANDREW: Attached to it. Right? JESSE: Yeah, I think the [00:36:23] exploration of that and the curiosity of that is what, for me at least, drives me to constantly keep practicing that you know that you can . . . Like you were saying earlier, that sometimes, you know, if there's someone there that can mop [00:36:38] the floor, open the coconut, there's a way to enter into that, where sometimes the task just has to get done and that person is learning it and they're going to make their mistakes. There's other times. I remember recently . . . We were short staffed at an Ocha ritual and I was the one on my hands and knees mopping, because normally would be someone else [00:36:53] and that's fine, because I'm usually assisting someone. But the . . . I had such pure joy in mopping the floor of just, like it was such an interesting thing of caretaking and, and kind of going into the trance of mopping, which was an interesting thing too, of still remaining present enough to know what [00:37:08] else was going on in the room, so that I'm not mopping something carelessly. ANDREW: Mm-hmm. JESSE: But also, this balance of, I guess it is a little bit of Zen and the art of peeling potatoes. But also for those of us that get lost in our heads, to be present enough and aware [00:37:23] enough of what else is going on, so that if you know the something escapes, you know, whether it's a child, a chicken, or a potato rolling down the hallway, that you're able to notice it and catch it, not that the chick, child is rolling down the hallway, but I [00:37:38] . . . hopefully that metaphor still makes sense. ANDREW: Yeah. Well, it's one of those things too. For me, I think one of the big differences between before making Ocha and after making Ocha. Or maybe [00:37:53] before receiving Orishas and after receiving Orishas is, when I work the tradition, whatever that is, I can feel the joy of the Orishas themselves, you know? JESSE: Mm-hmm. ANDREW: Like [00:38:08] when I tell them, like well I'm going to feed him something, and you know, I mean, that might be a sacrifice, but it might just be like, "I'm just gonna, you know, hey, I'm going to cook you this. I'm going to toast up all this corn for you," and you know, whatever. You can feel that energy, right? [00:38:23] JESSE: Yeah. ANDREW: And I feel like that energy extends to mopping the floor to you know, like all of these kinds of things, right? To, you know, even some of the less pleasant things like plucking, you know, plucking the chickens [00:38:38] after, or, you know wrestling with a ram that got out in the rain, or you know, whatever right? It's just like, it doesn't really matter, from my experience, you know, and maybe this is just me, but I think that it's part of this thing, because that, that service [00:38:53] to the spirits and their pleasure in it, you know, lifts up everything else. Right? JESSE: Well, I think it's an interesting parallel too, of a . . .It would seem to me, at least the way that I understood [00:39:08] Greek myths and Norse myths presented to me as a child, even reading like Edith Hamilton. . . ANDREW: Mm-hmm. JESSE: They were very anthropomorphized, the gods. So, anthropomorphized that there wasn't . . . it was hard to imagine that they were appearing in nature. They just owned [00:39:23] nature. And it seems that, you know, as my understanding of these things matures that perhaps that is a kind of modern revamping of a lot of pagan ideology and pagan theology . . . ANDREW: Mm-hmm. JESSE: But I'm [00:39:38] in one of Matthery's books, I believe, he's interviewing a priestess of Yemayá, in Nigeria, and talks to her and, and she talks about other [00:39:53] people worship their deities. We do our deities. And that when she interacts with water in any conscious level, she is participating in Yemayá. That Yemayá is an act of mopping or washing a body or washing the self or cooking and that water itself has a respect [00:40:08] and a consciousness and that consciousness, for her, was named Yemayá. ANDREW: Mm-hmm. JESSE: So, it was this concept and we talk about this, and the Spanish verb hacer does this very well, hacer tonto, you're doing something, you're making Santo, that when we participate in these [00:40:23] activities, we're actually participating in Orisha. Orisha is not a human. Orishas have incarnated as humans. But Orisha is as much the sound that the drum makes and gives us pleasure. Orisha is the flash of insight of a new idea. [00:40:38] Orisha is the feeling that we should go left and not right at this intersection, you know, there's things that are in the body that is not just in the head. The head leads it, of course, but it is broader and more experiential [00:40:53] and then the body becomes an extension of the head and the head grows because it is experiencing the world and I think there's something different. You know, mopping, you are, you are participating in an Orisha act that is yes, you're finding the joy. But it was also that the deities of [00:41:08] water that are there, that bathing can become a sacred act again. Like when does the ritual begin; when you bring attention to it. And you could make everything about the spirits that you're serving, or you could make very little and only be like a Sunday religionist, as you know, we talk [00:41:23] about. You know, it's a controversial thing to talk about the lack of ability to have separation of church and state but religion is there to justify politic, it always has been, the concept of religion. Karen Armstrong goes into that and I promote her all the time, just [00:41:38] because I find her such a fascinating . . . She's an ex-nun that writes on religion and her book, Fields of Blood, looks at religion and violence. And she talks about that that individual religion and spirituality is a very different thing than organized religion that is sitting there trying [00:41:53] to justify the actions of people in power. ANDREW: Mm-hmm. JESSE: Whereas the concept of religiosity or spirituality and those things . . . But what we do in our day-to-day is up to is each of us, but it's not just about going to church on Sunday, [00:42:08] promoting the separation of that, thinking that going to church on Sunday makes you a good person because you went . . . it's part of it. But how do you treat your family? How do you treat your co-workers? How do you treat the people around you? You know, how do you treat the land you're on? And this is a . . . It's not for everyone, because [00:42:23] it's very difficult to constantly be on in that mode. It takes practice. It's a muscle that you have to build and stretch. ANDREW: Mm-hmm. JESSE: And I do think that mopping, and carrying buckets of groceries up and down stairs, is a way of stretching [00:42:38] that muscle, or at least it can be when you present it in that way. If you're just bossing someone around, and say, "Go do this, go do this, go do this," they may not see that they're stretching a muscle. ANDREW: Yeah. JESSE: That's, you know, that's the thing too, is responsible training. You have to say, why are you doing this? Because if [00:42:53] we all stop to take out the trash, we can't do prepare for the ritual that has to happen. But if you, who cannot be on that side of the curtain or do and be in that room at that time, can take out the trash, then you've helped us do that ritual. ANDREW: Mm-hmm. JESSE: It is [00:43:08] part of it. It's that way of, what was the thing where the man was . . . A president was going to look at the space program and asked the janitor who he was and what he did, and he said, "This is my name and I'm [00:43:23] helping build to send men into space," you know, that it was the responsibility or the contextualized importance of every single task in a temple. ANDREW: Mm-hmm. JESSE: Very real thing. And if that person doesn't know, someone else is going to have to do it and hopefully take away [00:43:38] from that person balancing the books that day, but that's . . . it's an interesting thing. You see it in different religious communities. I'm friends with some nuns in Connecticut, at the Benedictine Abbey there, and it's so interesting to see, because they follow the Liturgy [00:43:53] of the Hours, their work spurts are two hours. They work really really hard for two hours, and they stop, change, and sing for a half hour to an hour depending on which what the liturgy is that day and then go back to work again. Though there's no warm-up [00:44:08] to working. They know they only have two hours, but they also don't rush. Which is like, "You're gonna do it, you're gonna get to work," and that's great. ANDREW: I think that that, also that dedication, right? Like they're gonna, they're gonna stop and sing, you know? It's like before [00:44:24] before I got married, my spirits, you know, my ancestors, in a mass, and a misa, were basically like, "We want you to go to church before you get married. We know you're not getting it in church. That's fine. But we want [00:44:39] you to go to a mass." And we were like, "All right," and so I went, and it was it was me and my partner and one other person in this massive, like, Anglican Church at 5 p.m. on a Friday night. And [00:44:54] I remember being there and it was very obvious that like, all the people in the congregation actually had no idea what to do because the priest was like, "Is anybody actually going to come up and take communion or should we just carry on," right? Like, oh, I didn't know this was the point, right? [00:45:09] Which is amusing, but it was also very obvious to me that if nobody had been there, he would have just done the mass. JESSE: Yeah. ANDREW: You know? And that like, that sort of devotion of, "We're going to stop and sing, [00:45:24] we're going to do this, we're going to do this thing." I think that kind of devotion is just astounding, you know, it's so wonderful. JESSE: you're speaking to me very true to Dzieci. We do a piece every year around this time. We [00:45:39] just had our first performances of it, but, called Fool's Mass, which is based on the kind of feast of fools idea from the, from the early modern and medieval period. But it's a [00:45:54] bunch of fools who are have to do the Christmas Mass, even though the priest just died. The exploration. It's a buffoonery piece and it's, there are extreme elements of humor and tragedy in it. ANDREW: Mm-hmm. JESSE: But the idea that this choir [00:46:09] comes together to sing and normally, you know, we play characters of different ability and, and function and, and responsibilities and some of us are troublemakers and other people are rule followers and what that chaos ensues, but [00:46:24] we know that there's songs that we sing and come together and there's something that's profound there in the in the silence and listening to each other as well. ANDREW: Mm-hmm. JESSE: And the chaos breaks out again, and how do you do this? How do you . . . how do you continue? In what you know, even [00:46:39] if there's no leader, how do you . . . I always find it interesting, like the dynamic of a classroom when the teacher has to leave to take, to go to the bathroom or something like that? Like, does it function as the same? It depends on the . . . how the teacher has run faster a lot of times. But [00:46:55] it's a, it's an interesting side of things. Doing what you know, when you know to do it is still, lots of times we're like, "Oh, the authority figure's not here, I don't have to do it this way. I could do it this other way." ANDREW: Exactly, right? JESSE: And [00:47:10] you go, okay, what did I just lose and what did I gain from that? What was the actual benefit from not doing it the exact way I know how? And so many times I think that, you know, it can come up in our systems [00:47:25] of divination, right? That you have the tools, you know exactly what the problem is, and you're not using them. ANDREW: Yeah. JESSE: You know? There's nothing new here. There's no new problems. You know what, you know, every problem that comes up, you know exactly why it's there and you have the tools to fix it, but you're not doing it. So, what do you what [00:47:40] are you looking for here? You know, that's, that's an interesting thing too. ANDREW: Yeah, I think it's such an interesting question, you know? Again, as somebody who's sort of far away from regular practice, you know, not having not having an extended community here, [00:47:55] you know, I've definitely, I've definitely run into this sort of angsty emotional piece. And I'm like, "Ah, I got nothing to do. I don't know what to work on. I got nothing to practice," or whatever, and this desire to learn more, right? And, and, [00:48:10] what I noticed at one point was, I was like, "Well, that's cool if there's more to learn and there's always more to learn," but also, how solid's your singing of Osain, [00:48:25] right? How solid is this piece? How about you, like, you know, make sure that you can, like, say the prayer for each of the Orishas, you know, the Oríkì, or learn a song for . . . There's often so much [00:48:40] in our immediate vicinity that we can tend to, and if we take that agency back to ourselves, right? JESSE: Yeah. ANDREW: And that way of like, you know, well, what do we, what do we do when there is nobody else watching? Right. JESSE: Mm-hmm. ANDREW: I think it's . . . I think that that [00:48:55] is . . . That's where the real work is, right? JESSE: Yeah. ANDREW: I mean, the rest of it is a bunch of work too and you know, not to dismiss it. But at least for me that real work is: I'm here. I'm doing this thing, whether it's, you [00:49:10] know, Orisha stuff or other stuff with my guides or you know, working on the cards or other projects. It's always that question of like: Okay, what do I need to do? How do I make myself do it? How do I do the stuff that doesn't seem glamorous but moves it all forward, [00:49:25] you know, and how do you find the joy with that, so you can sort of continue with devotion around it, you know, or faith, or those kind of old-fashionedy words, right? JESSE: Yeah, and also the benefit of when you approach things in [00:49:40] that way, it only informs the other things you're doing. ANDREW: Mm-hmm. JESSE: So, meaning, you know, you're going back to basics and finding new interesting things in them. Then it means that the possibility of you finding new ways and new depths to everything you're doing, because again, it's that muscle that you're stretching that is [00:49:55] developing a way of looking at the world, and, and aligning your feet to a new path. Perhaps it's the same path and you're learning it better, you know, it's nice to return to the things we know sometimes and realize that, oh gosh, there's a lot more here to examine. That [00:50:10] side of it. I know that's wonderful to be able to really examine what it [00:50:25] is that we know and develop the questions of ourselves of like okay, you think you know this for sure, and that's great, but what happens when you do it again? Do it one more time! ANDREW: mm-hmm. JESSE: I guess, for me, my background's, undergrad, is in theater [00:50:40] and doing things again is not a problem. ANDREW: Yeah. JESSE: Over and over and over. There is something of benefit when you have something so memorized. It allows for a new freedom in finding things [00:50:55] out. And it's not the same as reading the prayer, you know, there's a difference there. And what is it to do this and how you say it and what it opens your mind up to. It's like Catholic parallel of the rosary, that saying the prayers is just the bare minimum. Saying [00:51:10] the prayers of the rosary is the minimum. The visualization that is supposed to happen, because the prayers are by rote and coming out of your mouth, and your hand knows to feel for what beads it's saying. That you're actually envisioning mysteries as you're going through the rosary, is, that's level 2 and above, but [00:51:25] you know, if all you know is the prayers and that's what you do . . . ANDREW: Yeah. Well, and it's like, you know, watching, you know watching elders conduct ceremony, right? They're singing a song, they're doing a thing. They see somebody doing something they [00:51:40] shouldn't be and they don't even lose a beat and they're like, "Put the bucket of water down, blah blah blah blah," and they go right back to it, you know? And sometimes they even just sing it in the tune of what's going on, right? Which is always amusing. JESSE: Yes. Yeah, it is! (laughing) ANDREW: And, and that kind of fluency is just [00:51:55] you know, it's so profound. And it comes from that showing up and being present and having walked it so many times and all of that kind of stuff. Yeah. It's such a, such a fascinating thing to see in practice. And it comes out of this, [00:52:10] so much experience with it, right? JESSE: Yeah. ANDREW: Like being on theater, you know, on stage, when the person you're across from like, says the wrong line, what do you do, right? JESSE: You don't shoot them the right line. You've got to . . . and successful theater something that is [00:52:25] a wonderful exploration is, making each other look good. ANDREW: Mm-hmm. JESSE: You know, in ritual there's so much correction in the way that things can happen. But how can you correct the person so that they are empowered to embrace this correction you're giving them because you get [00:52:40] flustered. And everybody's gonna respond to that differently. But you know, how can you make the person look good still and explain to them, "Hey, there's this better way, try it like this." ANDREW: Yeah. JESSE: And, and, and really, because then they're open to the critique. They're open to the correction. And they don't feel ashamed. But, there's [00:52:55] also, we have to get over our shame, too. Especially in the oral traditions, because you're going to be corrected in front of other people. ANDREW: All the time! JESSE: And, you know, there's, I remember thinking about the profundity of . . . you know, we talk about our attitudes when were younger and [00:53:10] things, and enter member serving Egun before a ritual once, and everybody's talking and really only the people up at the front right at the shrine are actually paying attention to what's going on, and it was frustrating, and "I can't believe people aren't paying attention!" And realizing like, I am so not present because I'm [00:53:25] so concerned with everybody not paying attention that I'm not paying attention either, and it was just the like, oh my God, it all works if one person is focused up front, the whole thing, the whole ceremony is approved if one person, one conscious act makes [00:53:40] it happen. And then it's like it's great if the whole room is aligned, it's great if everybody will be quiet and focus. Its great of what that is, but it also is humbling to realize how much profound change or acceptance or of a new trajectory can happen with [00:53:55] just one person focusing. ANDREW: Mm-hmm. For sure. JESSE: And being on point and on task and that's really beautiful. ANDREW: Yeah. Well, maybe that's a good place to leave it. Go out there, folks. Be present! Listen, learn, and be kind to yourself and others, [00:54:10] you know, so we can all grow and expand and get wherever it is we're going to go with all of our magical practices. Yeah. Thanks for hanging out with me today, Jesse and being on here. I deeply appreciate it. JESSE: My pleasure. ANDREW: You've got all sorts of great [00:54:25] stuff going on online. People want to check it out. Where should they come and find you? JESSE: The store I run is Wolf and Goat, so wolf-and-goat.com. You can type it without the dashes as well. We're on [00:54:40] Facebook as well. I do a podcast with Dr. Al Cummins, called Radio Free Golgotha intermittently. We're on Facebook as well. But RadioFreeGolgotha.com. If you're interested in Para theater and want to do some strange [00:54:55] explorations of self and the world around you through theater. DzieciTheatre.org DzieciTheatre with an R, E, dot org. ANDREW: Spelled just like it sounds. JESSE: Yeah. (laughs) It means [00:55:10] children in Polish. And, I'm sure there's many other things I'm forgetting. But generally, I'm around a lot online, and even more so, in the back alleys of New York, I suppose, so, it's, [00:55:25] it's a pleasure and thanks for having me on, Andrew. ANDREW: Oh, thank you.
Copywriter and hypnotist Jesse Gernigin joins The Copywriter Club Podcast to talk with Kira and Rob about his freelance business, creating an online summit, and how knowing how to hypnotize people helps him know how to attract customers and sell more products. In this interview, we talk about: • how Jesse went from magician to hypnotist to copywriter • what it takes to bee a hypnotist • the #1 thing he did that made him a successful hypnotist • what he sent potential clients when he was cold contacting • how often he succeeded (and failed) when he was cold emailing and how he increased his chances of success • how Jesse works with clients to get them what they need (not just what they want) • what he did on Upwork to succeed • acting as a strategist in addition to working as a copywriter • what it takes to assemble an online summit and what has surprised him the most from putting on a summit And while talking about his summit, Jesse let us in on the tools he used to get his summit online and we asked him about the two best speakers he included in his summit. Finally Jesse told us what he thinks will happen to copywriting in the future. To get this one... click the play button below, or scroll down for a full transcript. The people and stuff we mentioned on the show: Sponsor: The Copywriter Club In Real Life Geoff Ronning The Ultimate Sales Letter by Dan Kennedy Vander Meide Ramit Sethi Chase Jarvis Paige Poutiainen Danny Marguiles Joanna Wiebe Thrive Architect Rainmaker Wordpress ConvertKit Teachable Vimeo Natalie McGuire Lianna Patch Hillary Weiss Entrepreneur on Fire Live Gold Rich Kira’s website Rob’s website The Copywriter Club Facebook Group Intro: Content (for now) Outro: Gravity Full Transcript: Rob: What if you could hang out with seriously talented copywriters, ask them about their successes and failures, their work processes and their habits, then steal an idea or two to inspire your own work? That’s what Kira and I do every week at The Copywriter Club Podcast. Kira: You’re invited to join the club for episode 71 as we chat with copywriter, marketing consultant, and hypnotist Jesse Gernigin about trading his magic act for high paying copywriting gigs, how he finds and lands freelance clients, what goes on behind the scenes of an online summit, and how hypnotism helps him become a better copywriter! Kira: Welcome, Jesse! Jesse: Thank you guys so much for having me! It’s great to be here. Rob: It’s great to have you. Jesse: Yeah, it’s cool to talk with you guys on this end after having you both on my summit, so this is great! Kira: Yeah! So we’re going to talk about your summit in a bit; you’re a first hypnotist on the show! Jesse: Okay! Yeah. Rob: Yeah, we’re waiting for you to say something like “look into my eyes”—follow the watch... Kira: (laughs) Jesse: (laughs) Kira: I’m actually a little nervous now! I feel like you might hypnotize us and make us say something ridiculous. I don’t know. Jesse: No, no, no. (laughs) Kira: All right, Jesse, a good place to start is just with your story. You know, who are you? How did you get into copywriting? Especially with the magic background? Tell us a little more about your story. Jesse: Oh, this is funny. So we’re going to go back to the days of copywriting books—Dan Kennedy’s, I think 1993 book—The Ultimate Sales Letter. So, I graduated college in 2007, so I came out right at the heart of the recession, and nobody was hiring for anything I had a degree in. And I’d been a magician and a hypnotist, and I’d work, you know, shows and make five or six thousand dollars a year just doing it on the side. And my buddy told me, you should just do this full time until a job opens up! So I went out, found an agent, and I was a really great performer. I don’t like to toot my own horn, because I wasn’t necessarily more talented than anybody else,
Welcome to LPLE, "Let's Practice Listening in English!" Jesse and Andrew reflect on their travels to Japan, and Jesse talks about his favorite thing to do when he's in Japan. Join in the conversation! Follow us on Twitter and Facebook to ask us questions about English conversation and meet other English language learners all over the world. Website: dialog.fm/lple iTunes: bit.ly/LPLEiTunes Facebook: bit.ly/LPLEFacebook Twitter: bit.ly/LPLETwitter Patreon: bit.ly/LPLEPatreon TRANSCRIPT Intro [Jesse]: Hi everyone. My name is Jesse Robbins, and welcome to LPLE from Dialogue FM. We're the podcast that lets you practice listening in English. We speak English slowly and clearly so that you can follow along and understand native English speakers more easily. I'm excited to help you improve your English listening skills, as well as help you learn new vocabulary, grammar, and idioms commonly heard and conversation among native English speakers. If you want to practice listening in English, then we invite you to join our conversation. Jesse: Hey, Andrew. Andrew: Hey, Jesse. Jesse: Before we begin, I'd like to say a special hello to students from two different schools now who are listening to LPLE to improve their English listening skills. Students from EKO English Pronunciation in Ho Chi Minh City in Vietnam, and I also found out that we have some students from our local City University who are also using LPLE for their school assignments and, just in general, to improve their English listening comprehension. So, hello! Andrew: That's great news! Welcome, folks! Jesse: Andrew, one thing you and I have in common is we have both been to Japan. Andrew: Yes. Jesse: When did you go? Andrew: It's been a while; I went back in 2008. Jesse: So, that's about... Andrew: Eight years ago? Jesse: Yeah, that's right. A lot has changed since. Andrew: I'm not surprised. Jesse: The last time I was in Japan was last fall, and before that it had also been about eight years. Let me tell you, I love Japan. I think you feel the same way. Andrew: I really enjoyed my trip. Jesse: There's a few things I really enjoy about Japan. Every time I go, I always have a wonderful time, and it's primarily because, one, I have friends there, and they always take really good care of me. In general, not just because I have friends, but Japanese people, in general, are very welcoming. Did you experience that yourself? Andrew: That's very much what I experienced when I went there. Even just from people on the street, or the people you met in stores or on the train, they were all very kind and very accommodating, and I was going without any Japanese language experience--I was speaking only English--and they were very accommodating of my need to work in my own language and learn my way around the city and find out what I needed to do. Jesse: So, very similar to my experience in Vietnam, because I can speak Japanese, I found that social barriers and any level of potential discomfort in interacting with a foreigner is immediately removed. Andrew: Right. Jesse: It also makes it much easier for me to establish--to create--my own social network in Japan. That's why every time I go back to Japan, I always let my friends know in advance and then I start to schedule time with them to go have dinner, to go have drinks, and, more importantly--I would say probably most importantly--I make sure to schedule time to go to karaoke. Andrew: Hahaha! That's the only reason you're going to Japan, isn't it, Jesse?... Jesse: Let me tell you, if I ever lived in Japan, I sincerely believe that I would be going to karaoke probably every day, if not every other day. I love it that much when I'm in Japan. You know what's funny? I live here in Seattle, and we have plenty of karaoke places. There's actual places that are just for karaoke and then there are bars, and in bars, there are karaoke machines, and you stand up on a stage in front of people, and you sing. When I'm in Seattle, my desire to go to karaoke is pretty small; I don't think about it that much. When I'm in Japan, that's almost all I can think about--is "when am I going to be going to my next karaoke party?" Andrew: Okay, so what is different about going to do karaoke in Japan that makes you so excited about it? Jesse: Well, in Japan the entire system is different when it comes to karaoke. Again, in America, it's very common to stand up on a stage in front of strangers and sing. Andrew: So you have to put on a performance for people you don't know. Jesse: That's right. So, it feels very intimidating, especially for people who are new. However, in Japan, it's very different. All karaoke places have separate rooms that you rent, and the rooms vary on size: Some rooms are very small for only four people--actually maybe even smaller I hear, sometimes for only two people; very small--and they can be as large as enough to fit 10 people or 15 people; an actual big party. So, you're in a room with just your friends, so the level of intimidation and fear to sing in front of people is a lot lower, and even in those rooms, because you feel more comfortable then you can let yourself have a lot of fun, specifically in this particular room I went to last time they had a mini stage with a microphone stand, and you felt like you were giving a performance, but you were giving a performance to all of your friends who are cheering you on. Now, not only do I speak Japanese but I also can sing in Japanese I will not touch you with my singing in Japanese right now you will have to come because I sing in Japanese and his friends and I'm a foreigner who can speak Japanese in Japanese and that much more amusing to watch them no. And it's a positive feedback loop because I'm having fun and my friends are having fun and then cheering me on which makes me have that much more fun when I'm singing for them overall look I love Japan I love going to Japan I love speaking Japanese I love Japanese food I love Japanese culture music everything but if there's one thing I love the most about going to Japan it's going to karaoke. Outro [Jesse]: Thank you for listening to this episode of LPLE, Let's Practice Listening in English, from Dialog.FM. Subscribe to LPLE on iTunes to hear the latest episodes, or listen to past episodes on our website, Dialog.FM. That's d-i-a-l-o-g-dot-f-m. If you have questions or comments about English, or if you would like for us to use a word, grammar, or idiom in our conversation so you can learn how to use it correctly, we would love to hear from you on Twitter at @dialogdotfm or Facebook at facebook.com/dialogFM.
Welcome to LPLE, "Let's Practice Listening in English!" Jesse and Andrew talk about online dating, older methods of meeting people, and the loss of the ability to introduce one's self to a stranger. Join in the conversation! Follow us on Twitter and Facebook to ask us questions about English conversation and meet other English language learners all over the world. Website: dialog.fm/LPLE iTunes: itunes.apple.com/us/podcast/lple/id1098735563 Facebook: facebook.com/LPLEDialogFM Twitter: twitter.com/LPLEDialogFM Patreon: patreon.com/LPLE TRANSCRIPT Intro [Jesse]: Hi everyone. My name is Jesse Robbins, and welcome to LPLE from Dialogue FM. We're the podcast that lets you practice listening in English. We speak English slowly and clearly so that you can follow along and understand native English speakers more easily. I'm excited to help you improve your English listening skills, as well as help you learn new vocabulary, grammar, and idioms commonly heard and conversation among native English speakers. If you want to practice listening in English, then we invite you to join our conversation. Jesse: Hey, Andrew. Andrew: Hey, Jesse. Jesse: Let me just say that dating in our society--modern-day dating--confuses the hell out of me. Andrew: I am right there with you. Jesse: We are both in our early thirties... Andrew: Yes. Jesse: ...and when we look back on our twenties--common dating age--dating was a lot different than what dating looks like now. Andrew: Yes. I think the dating culture that you and I grew up with was a much more traditional form of dating--that's the type that is probably closer to the kind of dating that maybe even our parents did when they were younger--and technology and social networking and the internet have kind of broken all of the rules that you and I used to follow... Jesse: ...and have created their own rules. Andrew: Yes. New standards and also ways of interacting and ways of meeting people that you and I didn't even have when we started going out to meet romantic partners. Jesse: Now, of course, there are pros and cons, there are the benefits and drawbacks, to this new form of dating, and you can say there's benefits and drawbacks, pros and cons, to the old way of dating back in our twenties. Andrew: Right! Jesse: I feel confident enough to say that if I were a single man now, I just could not handle the modern-day dating scene, which is-... I'm mean, what are the common apps that people use nowadays? Andrew: I think that nowadays rather than going to the bar and trying to meet someone you've never seen before, or being introduced through a friend, or asking someone out who you knew in a different context, either at work or to school, is not the most common way anymore because we have matchmaking websites like OkCupid or Match.com in the United States, or I think the most popular in our city is Tinder, which is the mobile app on our phone that lets us meet people without a whole lot of introduction and without a whole lot of detail about who the person is. It is like Facebook reduced to a picture and two sentences, and people make decisions on the spot about whether or not they are interested in meeting the other person. And, my understanding--I've never used this; I feel like a very old man just because I'm no longer in the market for this service--but, my understanding is you look at a bunch of pictures, see the person's first name, and read two sentences about them, and you say "yes, I'd like to meet this person," or "no, I wouldn't," and that is enough to carry on the conversation to the next level, because once both people say "yes" then they're allowed to contact each other and make plans for a date, which is about like walking into a coffee shop, and making an order, finding out the person behind you made the same order, and then going on a date next, which seems kind of crazy fast to me and without a whole lot of planning or preparation. Jesse: The screen on your phone, even with a [iPhone] 6 Plus, is not that large to contain a lot of information about someone, and then make a determination of whether or not you want to spend the next hour, or two hours, or five hours of your life with them. Andrew: Hahaha... Jesse: That's really what you're doing. You're trying to create-... Have the best looking photo of yourself and combine the best mix of words such that you will attract somebody. Andrew: Right. I think we are coming off sounding a bit old and out of fashion with this... Jesse: Yeah, out of touch. Andrew: ...out of touch, really, because we are used to meeting people earlier in the process and kind of evaluating from afar or being introduced, and things move a bit more slowly, and you are considering a person rather than an idea, and I think that's just us not being ready for this technology more than anything else, because I think you and I, because we have partners, don't need to be out finding new people all the time. And, what this technology lets you do is skip the part where you have to bravely introduce yourself to someone you don't know in the bar, or beg all your friends for introductions, or hope you are lucky enough to meet someone at a sports team, or a job, or a school that meets your interests. There's a lot of research that we had to do--I guess, in general--that people can skip and go to meeting people right away, and that seems to be what everyone wants in the first place. So, I think it's actually probably a good thing. Jesse: In the end, it's definitely made the process a lot easier for people. The barrier to talking to new people is dramatically reduced. Andrew: Right. Jesse: But, therein also lies a problem: We've lost the ability to talk to other people, right? If you think about, we've somewhat lost the ability to effectively introduce ourselves to a complete stranger and strike up a normal conversation. Andrew: Do you think that people had to practice being social and charming in person because they were forced to because of all of the dating that needed to happen before you got to know someone in the past, and that now people don't need to practice, and so they aren't as good at it? Jesse: I think there's an element of training that goes on--trial by fire, learning through experience. When I look back, before I met my current wife, when I was in the dating scene, when I was engaged in dating and introducing myself to women, yeah, there was an element of learning how to introduce myself appropriately, there were some successes and there were plenty of failures. Overall, I felt like it was a good skill to learn, primarily because you eventually learn how to get over that fear of introducing yourself to a stranger, and I think that skill carries over into a lot of other things. You can carry that skill over into your professional life, which is just learning how to go into a new room and introduce yourself to new people. I think that there's-... As nice as it is to streamline, to make it easier to meet people for the purpose of a potentially romantic relationship, I do believe that there's an element of social interaction that is lost, a particular skill that is lost. Andrew: And, I think I agree with you in terms of the software not being able to add that social interaction. What I think is happening is that people get to know more about the person before they have to expend the energy, so people who are naturally social, naturally friendly can go out there and meet people the old fashion way. But, they also have this tool to meet more people sooner, which lets them practice sooner with more people, if that is their stumbling block. So, I think, on balance, it is probably better, it is-... We have the old ways and we have the new ways, and people have more options, in general, when they're going out to find their next date. Jesse: Much like I would assume everybody in the dating scene agrees: The more options, the better. Andrew: Agreed. Outro [Jesse]: Thank you for listening to this episode of LPLE, Let's Practice Listening in English, from Dialog.FM. Subscribe to LPLE on iTunes to hear the latest episodes, or listen to past episodes on our website, Dialog.FM. That's d-i-a-l-o-g-dot-f-m. If you have questions or comments about English, or if you would like for us to use a word, grammar, or idiom in our conversation so you can learn how to use it correctly, we would love to hear from you on Twitter at @dialogdotfm or Facebook at facebook.com/dialogFM.
Welcome to LPLE, "Let's Practice Listening in English!" Jesse talks about moving into a new house. Andrew explains states, cities, and neighborhoods in America. Join in the conversation! Follow us on Twitter and Facebook to ask us questions about English conversation and meet other English language learners all over the world. Twitter: @LPLEDialogFM Facebook: facebook.com/LPLEDialogFM TRANSCRIPT Intro [Jesse]: Hi everyone. My name is Jesse Robbins, and welcome to LPLE from Dialogue FM. We're the podcast that lets you practice listening in English. We speak English slowly and clearly so that you can follow along and understand native English speakers more easily. I'm excited to help you improve your English listening skills, as well as help you learn new vocabulary, grammar, and idioms commonly heard and conversation among native English speakers. If you want to practice listening in English, then we invite you to join our conversation. Jesse: Hi, Andrew! Andrew: Hey, Jesse. Jesse: Cool story. A couple of weeks ago, my wife and I moved into a new house. Andrew: I know! We're actually sitting in it right now. Jesse: We're actually recording this podcast on our new dining table in our new living room. It's quite nice! Andrew: It's a very nice, brand new place. Jesse: Now, we live in the Rainier Valley neighborhood. Now, for those who are unfamiliar with how geography and...what, what's a good word? Municipalities? Andrew: I would just say how cities are laid out... Jesse: How cities are laid out. Andrew: Or, how Seattle is laid out. Jesse: Right, because some cities do it differently. Andrew: Right. Jesse: One big example is New York, where they have something that I don't think any other city has in the nation, which is burrows. Andrew: Well, yes. And, I would call those neighborhoods, but the burrows are mainly-... The burrows are defined by geography, right? By the islands that make up part of New York City and also where you are in relation to the freeway and downtown, is that right? Jesse: I have no idea how burrows work, honestly... Andrew: [hahaha] Jesse: Well, skipping that for just a moment here. How Seattle works is you have the Washington State, you have counties within the state, you have cities within the counties, and then you have neighborhoods, within the cities. Andrew: That's right. Jesse: So, we live in the Rainier Valley neighborhood. The old neighborhood we lived in before was called Judkins Park. We moved from Judkins Park to the Rainier Valley. Andrew: That's interesting, actually, because when you spoke about neighborhoods I was actually thinking about, I guess, a larger version of the "neighborhood" definition. So, Seattle is broken down by different areas, which I would consider to be places like Capitol Hill, First Hill, the Central District, North Beacon Hill, and so on. What you're describing are actually smaller parts of those areas, which are the actual, I guess, communities inside those neighborhoods like Judkins or Rainier Valley, and they refer more closely to the roads and the intersections that are around the area where you live, is that right? Jesse: Yeah, that's correct. Now, originally where we lived before in Judkins Park, we were about seven minutes to 10 minutes away from Chinatown and downtown. Andrew: That's right. Jesse: Now we live five to seven minutes away from Chinatown and downtown. So, we're moving ever closer to Chinatown and downtown, without actually living inside either one of those two areas. Andrew: Yes, which is interesting because you are actually moving south, away from most of Seattle, a little ways away. Jesse: Now, we live in a house-... a style of house that's called a "townhouse." How do we describe a townhouse for people who are unfamiliar with this kind of architecture? Andrew: That's a good question. I think when people think of normal family homes in the United States, in general, they are usually a traditional structure with a sloping roof, they are usually one or two stories tall, and usually take up a lot of space on one floor with a large yard around side it--around it outside. I think I would describe a townhome as taking up much less space with much less yard, and having more floors instead so that they are about the same size inside the home, but on maybe three or four floors instead of one or two. Jesse: That's right, that's right. On our ground floor, immediately when you enter the front door there are stairs going up to the, kind of the main area the living room, the kitchen. But, also on the ground floor when you enter you have the option of going to the side of the stairs to two different bedrooms and a bathroom. Andrew: Right. Jesse: So, they're basically compressing, they're making--for maybe lack of a better word--shrinking the size of a normal house; instead of building wider they're building taller. Andrew: That's correct, yes. And, I would say that it is not--... again it is not smaller, it is just stacked differently. So, like you say, there are only two bedrooms on the ground floor, which means that the floor is smaller, but then the next floor up you have a living room and a kitchen, which in a more traditional American home might all be on the same floor. Jesse: Right, right. Are there townhouses in other states? I think that maybe townhouses are more commonly found in denser cities where land is sma-... where land is fewer. Andrew: I think land is more expensive near big cities, and that is why people choose to build taller rather than wider. Jesse: Yes. Andrew: I think traditional American cities had more space, and many of them are still like that. So, for example, in the middle of the country, in the midwest cities like St. Louis or Chicago, tend to have more space and so they have more single family homes with yards. In cities that are denser like New York or like Seattle or San Francisco, there's not as much space to have a yard and to build out, and so they build up instead, and that's why town homes have become more popular. But, they're also very nice because they are built with the newest technology. Jesse: Yes. Andrew: So, they have bigger windows, they have better insulation so they don't get as cold or as hot in the weather, and they're cheaper to run, so it costs less money to keep them warm in the winter and cool in the summer. And, they stand up to weather well, as well. Jesse: That's right. You talk about yards. Now, how do you feel about yards? Andrew: I personally don't care much for them. I don't-... Let me say that differently, I don't value them very highly because I don't spend my time out in them. I am usually out in the city, and when I want to go out into nature, I drive to the mountains and the forests nearby. So, to me the yard is pretty to look at, but it also means a lot of work. I need to mow the grass, I need to pull weeds, I need to plant flowers or a garden, and these are things that I would not want to do normally for myself. So, they are kind of a responsibility that I don't want. I like living in the city because I am close to everything that I like to do, so bars, restaurants, theater, bands, and other performances, and also to be close to my friends. And, so, I don't feel like it is as important to have an estate, a big piece of land to live on, as well. Outro [Jesse]: Thank you for listening to this episode of LPLE, Let's Practice Listening in English, from Dialog.FM. Subscribe to LPLE on iTunes to hear the latest episodes, or listen to past episodes on our website, Dialog.FM. That's d-i-a-l-o-g-dot-f-m. If you have questions or comments about English, or if you would like for us to use a word, grammar, or idiom in our conversation so you can learn how to use it correctly, we would love to hear from you on Twitter at @dialogdotfm or Facebook at facebook.com/dialogFM.
Welcome to LPLE, "Let's Practice Listening in English!" Jesse shares another story about his experience visiting Vietnam and is impressed by how local Vietnamese try to practice English with foreigners. Andrew wonders how foreigners might feel about random locals coming up to foreign travelers to practice speaking English. Join in the conversation! Follow us on Twitter and Facebook to ask us questions about English conversation and meet other English language learners all over the world. Twitter: @LPLEDialogFM Facebook: facebook.com/LPLEDialogFM TRANSCRIPT Intro [Jesse]: Hi everyone. My name is Jesse Robbins, and welcome to LPLE from Dialogue FM. We're the podcast that lets you practice listening in English. We speak English slowly and clearly so that you can follow along and understand native English speakers more easily. I'm excited to help you improve your English listening skills, as well as help you learn new vocabulary, grammar, and idioms commonly heard and conversation among native English speakers. If you want to practice listening in English, then we invite you to join our conversation. Jesse: Hi, Andrew. Andrew: Hey, Jesse. Jesse: Another interesting story about Vietnam. Remember, I was there for two-and-a-half weeks, and during this trip to Vietnam I actually had the chance to visit another province. Now, when most people think about going to Vietnam they think about going to Ho Chi Minh City, Hanoi, maybe even Ha Long Bay. You visited those places didn't you? Andrew: I did. You took me around and played tour guide. Thank you! Jesse: During this particular trip I spent one week in a province called Dong Thap. It's about a three-and-a-half hour to four-hour bus ride southwest of Ho Chi Minh City. Andrew: Okay, so in the south of the country. Jesse: Yeah. I was there for one week and it was a very fun experience. The city itself where I was in, which is called Cao Lanh, is a pretty small but rapidly developing city. It's small compared to, of course, Ho Chi Minh City. But, you start to see a lot of commercial businesses starting to grow. Andrew: That's good. Jesse: Yeah. Exactly, there's big hotels, there's stores that sell lots of computer peripherals and whatnot, there's...it's a rapidly growing city, which is really exciting. And, the people were so welcoming. That's not to say that other parts of Vietnam aren't. I'm sure they are. In this city we had the chance to meet college students at the local university; and, these students are practicing English. And, because there's not a lot of foreigners that come into the city or province in general, they are so excited to meet us. Andrew: That's great that they had a chance to speak with people who were native speakers. Jesse: There was one afternoon I walked around the lake--there's a popular lake there. It's not as big as our Green Lake, right, in terms of size, but it is still made for a pleasant walk around the lake. And, multiple times as I walked around the lake I was stopped by local Vietnamese just because they wanted to say "hi" and ask where I'm from. Andrew: Do you think that was because they recognized that you were foreign to Vietnam and that you probably spoke English? Do you think it was an opportunity for them to practice their language skills? Jesse: It's a combination of both. I think it's a combination of, one, I'm a foreigner, more specifically, I am an atypical-looking foreigner... Andrew: ...Meaning you don't look like a white American. Jesse: Correct. Now, in Vietnam, it was very hot, so I tanned very quickly. I got darker skin very quickly, so any chance of me looking even remotely American or European was gone. So, there was an element of 'I'm a foreigner' but there's also a sense of 'I'm a strange-looking foreigner.' Andrew: You felt like you looked unique? Jesse: Very much so. And, that's not a bad thing; it's fine. I kind of expected it at this point. And then there's also the element of them wanting to practice their English, which is also fine. So, that leads me to another story I wanted to talk to you about. It's not just about how friendly the local residents of the city were. It's not how welcoming the university students were for us. There's one common theme I've noticed that makes me admire people studying the English language in general. The Vietnamese I met work so hard to find a way to practice English. They find every opportunity they can, and they are not shy about it. Andrew: Does this make them rude or did they interrupt your other events or conversations? Jesse: Not at all. So, they were really respectful. Now, you know, maybe one could say that it might be rude of them to yell "hello" when I'm just trying to have a peaceful walk around the lake, but they don't know I'm trying to have a peaceful walk around the lake. Andrew: They reached out and introduce themselves and engaged in a conversation from scratch without any introduction Jesse: Exactly, and I admire that. I admire that tenacity. I admire that enthusiasm. I admire that dedication. And, I admire that energy from them. When learning a foreign language, one of the biggest challenges I think we as Americans have is we are so afraid of making a mistake we don't want to try to practice our Spanish that we learned for one year because we're somehow embarrassed by it. Whereas these students who have been practicing English for, of course, over one year but who have never left Vietnam in their life let alone seen many foreigners in their city... Andrew: ...Were completely ready to walk up to a stranger and start speaking in their new language. Jesse: Exactly, and I truly admire that. So, for many foreigners who are unfamiliar with traveling in a country like Vietnam where people are working so hard to practice English because they know that English is going to provide them with an economic opportunity. Andrew: Right. It gives them a better jobs. It gives them access to opportunities they wouldn't have if they don't speak the language of business, which is usually English. Jesse: Right. If you're a foreigner who goes to this kind of country and you're not familiar with that kind of mentality, of course it could seem pretty rude or disruptive to your schedule because maybe you're just trying to enjoy the scenery or take some photos, you just want time to yourself. I want to encourage people listening to this, you know, as you, you in the audience, as you practice English by listening to this podcast and as you introduce yourself to foreigners and say "hello" just know that there are many people who admire what you're doing because what you're doing is not easy at all Andrew: Agreed! Outro [Jesse]: Thank you for listening to this episode of LPLE, Let's Practice Listening in English, from Dialog.FM. Subscribe to LPLE on iTunes to hear the latest episodes, or listen to past episodes on our website, Dialog.FM. That's d-i-a-l-o-g-dot-f-m. If you have questions or comments about English, or if you would like for us to use a word, grammar, or idiom in our conversation so you can learn how to use it correctly, we would love to hear from you on Twitter at @dialogdotfm or Facebook at facebook.com/dialogFM
Welcome to LPLE, "Let's Practice Listening in English!" Jesse and Andrew talk about how they celebrated Christmas and the turn of the New Year. Jesse introduces the concept of a "bachelor party." Join in the conversation! Follow us on Twitter and Facebook to ask us questions about English conversation and meet other English language learners all over the world. Twitter: @LPLEDialogFM Facebook: facebook.com/LPLEDialogFM TRANSCIPT Intro [Jesse]: Hi everyone. My name is Jesse Robbins, and welcome to LPLE from Dialogue FM. We're the podcast that lets you practice listening in English. We speak English slowly and clearly so that you can follow along and understand native English speakers more easily. I'm excited to help you improve your English listening skills, as well as help you learn new vocabulary, grammar, and idioms commonly heard and conversation among native English speakers. If you want to practice listening in English, then we invite you to join our conversation. Jesse: Andrew. Andrew: Jesse. Jesse: Happy New Year! Andrew: Same to you. Jesse: We are in January; it is 2016. I'm curious, how was your holiday vacation back in December? Andrew: Very good. It has been a busy couple of weeks, but I had a lot of time to visit family and friends, and, so, in addition to the Christmas holiday at the end of December, I also went to a wedding that happened on New Year's Eve, and then celebrated the New Year before coming back to work in January. Jesse: That's fantastic! Whose wedding? Andrew: This was a wedding for two of my good friends who live across the border in Canada. We are here in Seattle, and the wedding and the New Year's Eve celebration we had was up in Vancouver, British Columbia in Canada, about 2 to 3 hours drive away. Jesse: How long did you stay in Canada? Andrew: Let me see... We went up the day before New Year's Eve, so December 30th, the wedding was on December 31st, and, so, we had the wedding ceremony during the day, and after the wedding was finished we went to downtown Vancouver to watch the fireworks at midnight to celebrate the New Year, and then went home with lots of traffic in the early morning of January 1st. Jesse: For New Year's celebration, every year my wife and I go to our friends' house. Our friend lives in a neighborhood called Capitol Hill. They live in an apartment and their apartment has a rooftop that people can go on to to have a beautiful view of downtown Seattle. Andrew: So, this is a tall building on a tall hill in the middle of the city. Jesse: Correct. Andrew: And there's a good view of the downtown city area where the fireworks happen. Jesse: Yes. So, we hung out at a friends' place for a few hours. And then, 15 minutes before midnight we took a bottle of champagne, we went up to the rooftop, and we waited for the fireworks. The fireworks event was happening at a place in downtown called Seattle Center. On the rooftop of our friend's apartment there were many people, so it was a little bit crowded. However, we had enough space to have a clear view of the Space Needle. Andrew: That's great! Jesse: So, we waited, and then when it was New Year's we all drink champagne and we gave each other a hug, we wished each other "Happy New Year," and then after that I quickly went back inside the house because it is very cold outside... Andrew: ...in December in Seattle, yeah. Jesse: How was your Christmas? Andrew: My Christmas was also very good. It was a little bit busy because this was the first year that my girlfriend and I were trying to see both my parents and her parents all on the holiday. So, on Christmas Eve we went down to my parents' house, which is about 30 minutes away, and spend the night there to get up in the morning, and exchange gifts, and have a Christmas dinner meal in the middle of the day. And, my dad's parents--my grandparents--were also there to celebrate with us. And then, we had to leave that evening to go to her parents' house up north of the city, about an hour away. So, there was a lot of driving involved, but we got to see everyone and have dinner with both sides and get back in time to jump into the other activity that I had to fit into this busy season, which was the bachelor party for my friend who was getting married. So, December 25th and 26th it was Christmas celebrations, and December 27th and 28th, which was the weekend, my friend from Vancouver came to Seattle with some friends, and we went around the city to drink and celebrate his wedding that was coming up very soon. Jesse: Let's explain a bachelor party really quickly. A bachelor party in American society is when the groom--the guy who is, the man who is going to get married--has a party with his guy friends before the wedding. Sometimes it's the night before the wedding... Andrew: And that's a bad idea [haha]. Jesse: Sometimes it's the week before the wedding. Sometimes it can even be a couple of months before the wedding. Typically a groom has a bachelor party, and usually a bachelor party includes drinking alcohol and beer, going out into the city, or...or even going to a destination, a far away place like a cabin or... Andrew: Another city to take a vacation. Jesse: Exactly. Andrew: And, usually there are other activities to celebrate the occasion of his getting married like going out to dance, or going to a show, or celebrating at someone's house. We did a lot of those things and had a very good time. Although, some of the guests had a little bit too much to drink and had headaches the next day, they had a hangover. I believe the bride also had her version of this party, so she had a bachelorette party that was up in Vancouver. So, they were separated to celebrate on their own independently, and then came back together for the wedding the next week. Jesse: Typically, for a bachelor party there are no women in the group Andrew: Correct. Jesse: Likewise, typically, for a bachelorette party for the bride there are no men in the group, it's only women. Andrew: Correct. Jesse: For my Christmas holiday, it was very relaxing. We had Christmas dinner at my wife's family's house. They live very nearby. And then, throughout the day I relaxed at home and watched four movies. Andrew: [haha] That's a long...a long day! Jesse: Four movies, back-to-back-to-back-to-back. Andrew: Eight hours or so of film? Jesse: Yes. Whenever I have a holiday vacation like Christmas, or like Thanksgiving, or like New Year's, I like to spend time watching movies at home. It's a chance for me to catch up on many movies that I missed over the past four or five years. Andrew: Four or five years? Jesse: Yeah! Not just this past year, but over the past few years. So, I stayed at home watching movies and just relaxing. That was it. Andrew: Sounds like a good time and a good way to relax before coming back to work and your normal life after the holiday. Jesse: Exactly. Outro [Jesse]: Thank you for listening to this episode of LPLE, Let's Practice Listening in English, from Dialog.FM. Subscribe to LPLE on iTunes to hear the latest episodes, or listen to past episodes on our website, Dialog.FM. That's d-i-a-l-o-g-dot-f-m. If you have questions or comments about English, or if you would like for us to use a word, grammar, or idiom in our conversation so you can learn how to use it correctly, we would love to hear from you on Twitter at @dialogdotfm or Facebook at facebook.com/dialogFM.
Episode 151: Reflecting Back on Theatre School Jesse Wilson was born in LA, started acting at an early age and was accepted into Juilliard. He was on track to becoming a traditional professional actor, but twenty years later has found purpose in the education aspect of the arts. In this podcast Jesse reflects back on his time at Juilliard, his expectations of the experience and what really happened, and what do today's students need to do to prepare for life after school. Show Notes Face the City promotional package Face the City Video Theatre Beyond the Stage podcast Theatrefolk Blog Episode Transcript Welcome to TFP – The Theatrefolk Podcast – the place to be for Drama teachers, Drama students, and theatre educators everywhere. I'm Lindsay Price, resident playwright for Theatrefolk. Hello! I hope you're well. Thanks for listening! This is Episode 151. You can find any links to this episode in the show notes which are at theatrefolk.com/episode151. All right. Today, we're looking back. We're reflecting. Reflection – the great buzzword of education in the 21st Century. “Everybody walk across the room. Okay, everybody, reflect on the experience.” (You know, I'm kidding. I'm kidding.) Actually, I use reflection all the time in our educational materials. I think it's important. It's really key to comprehension and connection – good C words. So, we're going to reflect and we're going to reflect on theatre school. We're going to look back on going to theatre school twenty years after the fact with Jesse Wilson. He was accepted into Juilliard and one might think that that alone would be a one-way ticket to a professional theatre life because going to Juilliard is a dream for many theatre students. When I think of theatre school, you know, that's the name at the top of my list. But, as we all know, dreams are different than reality – in good and bad ways, right? There's a little teaser for you. You have to listen for good and bad. Okay, let's get to it. LINSDAY: Hello, everybody! All right, I am speaking with Jesse Wilson. Hello, Jesse! JESSE: What's up, Lindsay? LINSDAY: Oh, everything is up! Everything is up, nothing is down. Actually, everything is up and everything is down which means we're all in a good place. JESSE: That's great. LINSDAY: Or something like that. JESSE: Isn't that like a She Silverstein or something? “Everything is up, everything is down”? Something like that? I don't know. LINSDAY: Oh, I like that. I think that's wonderful. JESSE: It sounds like it. If not, it's something that should be written. I love it. LINSDAY: It's a good point of view and we'll give the credit to Shel Silverstein. JESSE: Yeah. LINSDAY: So, Jesse, you and I talked last year about a wonderful project that you sort of got involved in with theatre in prisons. JESSE: Yes. LINSDAY: We also got a little bit into your beginnings and how where you are is probably not where you were expected when you first started out in your theatre career. JESSE: Not at all. LINSDAY: That's kind of what we're talking about this month on the podcast. We're talking about, you know, what happens about your expectations of going into a theatre program and what you think your theatre life is going to be and maybe how you can maybe temper some of those expectations or have a strategy, right? Have a sort of strategy for hitting the real world. I think that's it in a nutshell. So, let's go back – way back – and just talk about your beginnings. You grew up in LA. JESSE: I did. LINSDAY: What was that like? JESSE: Born and raised. LINSDAY: Ah, you're one of the twenty, right? JESSE: I think so, yeah! I was born and raised in Hollywood, California – about five minutes away from the Hollywood sign. You could look outside my bedroom window and I saw a very clear view of the Dodgers' stadium way in the distance and then, to my left,
Nasir and Matt welcome Jesse Lindsleyto talk about people are ripping off successful mobile games and answer the question, "My partners and I have been developing an online software and a mobile app, but we are wondering if we should split the mobile aspect of our business into a separate LLC since not everyone is developing that and the app could stand on its own. Is that advisable?" Full Podcast Transcript NASIR: Welcome to Legally Sound Smart Business! This is Nasir Pasha. MATT: And this is Matt Staub. NASIR: Welcome to our podcast where we cover business in the news and add our legal twist and also answer some of your business legal questions that you, the listener, can send in at ask@legallysoundsmartbusiness.com and we’re ready to go. MATT: Yeah, I hope we make this a good one. The Wednesday episode seemed to be the most popular one. NASIR: Yeah, it’s the hump of the week and the top of the week, I think, for us. MATT: Don’t screw up, Nasir. NASIR: I know. It’s a lot of pressure now. Great. MATT: Let’s get into the story we have for today. It has to deal with this game, 2048, which I’ve heard of but I never play. I actually downloaded it yesterday. NASIR: Really? Okay. MATT: Just to see what it was and played it a couple of times. I get the gist of it and I can get how it would be addicting. I try not to do any of these games just because I don’t want to get sucked in. NASIR: I agree. I’ve played it and I admit that I actually have played it quite a bit but I’m not much of a mobile app gamer, I would consider myself, but for whatever reason, I heard other people playing it so I took a look and I like puzzle games. I got sucked in, I suppose. MATT: I’m surprised you’ve played because anyone that listens to this podcast knows that your math skills aren’t always 100 percent. NASIR: Oh, wow, that’s a low-blow. That is a low-blow. I was a computer science major. Math was important at that time. But, anyway, my AP Calc teacher is going to be upset at that comment from high school. I was just joking. MATT: So, what we’re dealing with here, the underlying story is the intellectual property behind these games and, more importantly, I guess the infringement thereof. But when we’re dealing with mobile games and I think they talked about board games as well, not everything you think would be able to be protected is protected. They mentioned how the protection is of the expression of an idea but you can’t protect the idea itself. NASIR: Yeah, and if you notice these 2048 games are everywhere and apparently it’s some kind of version of some game called Threes and I think everyone knows that, well, I never played but that game Flappy Bird which became really popular but then went off the market and then everyone put clones up, the reason is basically there was no patent on that idea and there’s maybe copyright as far as the graphics go and so forth but there’s limitations on that. But I wanted to get Jesse from Thrust. They have a pretty cool mobile app development company and kind of want to get their perspective. I’m sure this issue has been brought up in their development as well. Jesse, how are you doing? JESSE: I’m doing great. Thanks for having me. NASIR: Very good. So, you kind of heard about what’s going on with 2048. You’ve seen all these clones and it’s weird. I use an Android. If you go to the top ten mobile apps that are free downloads or whatever, I think three of them are 2048 variations. I don’t know if you’ve ever had any experience with these kinds of issues dealing with your development. JESSE: Yeah, we come across this kind of stuff all the time. As developers, we get frustrated when people copy our ideas. I’ve been involved in these kinds of discussions from the gaming gambling days, early 2000 to the Facebook launching of games and a lot of the games were launched by Zynga and others were pure copycats of other games. So, it’s pretty standard and there’s court cases where sometimes the little gu...