Podcast appearances and mentions of mike julian

Today's guest is Peter Moskos, a professor at John Jay College of Criminal Justice. He spent two years as a police officer in Baltimore. I asked him to come on and talk about his new book, Back from the Brink, Inside the NYPD and New York City's Extraordinary 1990s Crime Drop. It's one of my favorite books I've read this year (and it was one of my three book recommendations on Ezra Klein's show last week).Peter spoke with hundreds of police officers and NYC officials to understand and describe exactly how the city's leaders in the early 1990s managed to drive down crime so successfully.We discussed:* How bad did things get in the 1970s?* Why did processing an arrest take so long?* What did Bill Bratton and other key leaders do differently?* How did police get rid of the squeegee men?I've included my reading list at the bottom of this piece. Thanks to Harry Fletcher-Wood for his judicious transcript edits.Subscribe for one new interview a week.Peter, how would you describe yourself?I would say I'm a criminologist: my background is sociology, but I am not in the sociology department. I'm not so big on theory, and sociology has a lot of theory. I was a grad student at Harvard in sociology and worked as a police officer [in Baltimore] and that became my dissertation and first book, Cop in the Hood. I've somewhat banked my career on those 20 months in the police department.Not a lot of sociologists spend a couple of years working a police beat.It's generally frowned upon, both for methodological reasons and issues of bias. But there is also an ideological opposition in a lot of academia to policing. It's seen as going to the dark side and something to be condemned, not understood.Sociologists said crime can't go down unless we fix society first. It's caused by poverty, racism, unemployment, and social and economic factors — they're called the root causes. But they don't seem to have a great impact on crime, as important as they are. When I'm in grad school, murders dropped 30-40% in New York City. At the same time, Mayor Giuliani is slashing social spending, and poverty is increasing. The whole academic field is just wrong. I thought it an interesting field to get into.We're going to talk about your new book, which is called Back from the Brink, Inside the NYPD and New York City's Extraordinary 1990s Crime Drop. I had a blast reading it. Tell me about the process of writing it.A lot of this is oral history, basically. But supposedly people don't like buying books that are called oral histories. It is told entirely from the perspective of police officers who were on the job at the time. I would not pretend I talked to everyone, because there were 30,000+ cops around, but I spoke to many cops and to all the major players involved in the 1990s crime drop in New York City.I was born in the ‘90s, and I had no idea about a crazy statistic you cite: 25% of the entire national crime decline was attributable to New York City's crime decline.In one year, yeah. One of the things people say to diminish the role of policing is that the crime drop happened everywhere — and it did end up happening almost everywhere. But I think that is partly because what happened in New York City was a lot of hard work, but it wasn't that complicated. It was very easy to propagate, and people came to New York to find out what was going on. You could see results, literally in a matter of months.It happened first in New York City. Really, it happened first in the subways and that's interesting, because if crime goes down in the subways [which, at the time, fell under the separate New York City Transit Police] and not in the rest of the city, you say, “What is going on in the subways that is unique?” It was the exact same strategies and leadership that later transformed the NYPD [New York Police Department].Set the scene: What was the state of crime and disorder in New York in the ‘70s and into the ‘80s?Long story short, it was bad. Crime in New York was a big problem from the late ‘60s up to the mid ‘90s, and the ‘70s is when the people who became the leaders started their careers. So these were defining moments. The city was almost bankrupt in 1975 and laid off 5,000 cops; 3,000 for a long period of time. That was arguably the nadir. It scarred the police department and the city.Eventually, the city got its finances in order and came to the realization that “we've got a big crime problem too.” That crime problem really came to a head with crack cocaine. Robberies peaked in New York City in 1980. There were above 100,000 robberies in 1981, and those are just reported robberies. A lot of people get robbed and just say, “It's not worth it to report,” or, “I'm going to work,” or, “Cops aren't going to do anything.” The number of robberies and car thefts was amazingly high. The trauma, the impact on the city and on urban space, and people's perception of fear, all comes from that. If you're afraid of crime, it's high up on the hierarchy of needs.To some extent, those lessons have been lost or forgotten. Last year there were 16,600 [robberies], which is a huge increase from a few years ago, but we're still talking an 85% reduction compared to the worst years. It supposedly wasn't possible. What I wanted to get into in Back from the Brink was the actual mechanisms of the crime drop. I did about fifty formal interviews and hundreds of informal interviews building the story. By and large, people were telling the same story.In 1975, the city almost goes bankrupt. It's cutting costs everywhere, and it lays off more than 5,000 cops, about 20% of the force, in one day. There's not a new police academy class until 1979, four years later. Talk to me about where the NYPD was at that time.They were retrenched, and the cops were demoralized because “This is how the city treats us?” The actual process of laying off the cops itself was just brutal: they went to work, and were told once they got to work that they were no longer cops. “Give me your badge, give me your gun."The city also was dealing with crime, disorder, and racial unrest. The police department was worried about corruption, which was a legacy of the Knapp Commission [which investigated NYPD corruption] and [Frank] Serpico [a whistleblowing officer]. It's an old police adage, that if you don't work, you can't get in trouble. That became very much the standard way of doing things. Keep your head low, stay out of trouble, and you'll collect your paycheck and go home.You talk about the blackout in 1977, when much of the city lost power and you have widespread looting and arson. 13,000 off-duty cops get called in during the emergency, and only about 5,000 show up, which is a remarkable sign of the state of morale.The person in my book who's talking about that is Louis Anemone. He showed up because his neighbor and friend and partner was there, and he's got to help him. It was very much an in-the-foxholes experience. I contrast that with the more recent blackout, in which the city went and had a big block party instead. That is reflective of the change that happened in the city.In the mid-80s you get the crack cocaine epidemic. Talk to me about how police respond.From a political perspective, that era coincided with David Dinkins as [New York City's first black] mayor. He was universally disliked, to put it mildly, by white and black police officers alike. He was seen as hands off. He was elected in part to improve racial relations in New York City, to mitigate racial strife, but in Crown Heights and Washington Heights, there were riots, and racial relations got worse. He failed at the level he was supposed to be good at. Crime and quality of life were the major issues in that election.Dinkins's approach to the violence is centered around what they called “community policing.” Will you describe how Dinkins and political leaders in the late ‘80s and early ‘90s thought about policing?This is under Ben Ward, the [NYPD] Commissioner at the time. The mayor appoints the police commissioner — and the buck does stop with the mayor — but the mayor is not actively involved in day-to-day operations. That part does go down to the police department.Community policing was seen as an attempt to improve relations between the police and the community. The real goal was to lessen racial strife and unrest between black (and to a lesser extent Hispanic) communities and the NYPD. Going back to the ‘60s, New York had been rocked by continued unrest in neighborhoods like Central Harlem, East New York, and Bushwick. Community policing was seen as saying that police are partly to blame, and we want to improve relations. Some of it was an attempt to get the community more involved in crime fighting.It's tough. It involves a certain rosy view of the community, but that part of the community isn't causing the problems. It avoids the fact there are people who are actively criming and are willing to hurt people who get in their way. Community policing doesn't really address the active criminal element, that is a small part of any community, including high-crime communities.Arrests increased drastically during this era, more than in the ‘90s with broken windows policing. If the idea is to have fewer arrests, it didn't happen in the ‘80s. Some good came out of it, because it did encourage cops to be a bit more active and cops are incentivized by overtime. Arrests were so incredibly time-consuming, which kind of defeated the purpose of community policing. If you made an arrest in that era, there was a good chance you might spend literally 24 hours processing the arrest.Will you describe what goes into that 24 hours?From my experience policing in Baltimore, I knew arrests were time-consuming and paperwork redundant, but I could process a simple arrest in an hour or two. Even a complicated one that involved juveniles and guns and drugs, we're talking six to eight hours.In the ‘80s, Bob Davin, [in the] Transit Police, would say they'd make an arrest, process at the local precinct, search him in front of a desk officer, print him, and then they would have to get a radio car off patrol to drive you down to central booking at 100 Centre Street [New York City Criminal Court]. Then they would fingerprint him. They didn't have the live scan fingerprints machine, it was all ink. It had to be faxed up to Albany and the FBI to see if it hit on any warrant federally and for positive identification of the person. Sometimes it took 12 hours to have the prints come back and the perp would be remanded until that time. Then you'd have to wait for the prosecutor to get their act together and to review all the paperwork. You couldn't consider bail unless the prints came back either positive or negative and then you would have that initial arraignment and the cop could then go home. There are a lot of moving parts, and they moved at a glacial pace.The system often doesn't work 24/7. A lot of this has changed, but some of it was having to wait until 9 am for people to show up to go to work, because it's not a single system. The courts, the jails, and policing all march to their own drummer, and that created a level of inefficiency.So much of the nitty-gritty of what cops actually do is boring, behind-the-scenes stuff: How do we speed up the paperwork? Can we group prisoners together? Can we do some of this at the police station instead of taking it downtown? Is all of this necessary? Can we cooperate with the various prosecutors? There are five different prosecutors in New York City, one for each borough.There's not a great incentive to streamline this. Cops enjoyed the overtime. That's one of the reasons they would make arrests. So during this time, if a cop makes an arrest for drug dealing, that cop is gone and no cop was there to replace him. If it's a minor arrest, there's a good chance in the long run charges will be dropped anyway. And you're taking cops off the street. In that sense, it's lose-lose. But, you have to think, “What's the alternative?”Bob Davin is a fascinating guy. There's a famous picture from 1981 by Martha Cooper of two cops on a subway train. It's graffitied up and they're in their leather jackets and look like cops from the ‘70s. Martha Cooper graciously gave me permission to use the picture, but she said, "You have to indemnify me because I don't have a release form. I don't know who the cops are." I said, "Martha, I do know who the cop is, because he's in my book and he loves the picture.” Bob Davin is the cop on the right.Davin says that things started to get more efficient. They had hub sites in the late ‘80s or ‘90s, so precincts in the north of Manhattan could bring their prisoners there, and you wouldn't have to take a car out of service to go back to Central Booking and deal with traffic. They started collecting prisoners and bringing them en masse on a small school bus, and that would cut into overtime. Then moving to electronic scan fingerprints drastically saves time waiting for those to come back.These improvements were made, but some of them involve collective bargaining with unions, to limit overtime and arrests that are made for the pure purpose of overtime. You want cops making arrests for the right reason and not simply to make money. But boy, there was a lot of money made in arrests.In 1991, you have the infamous Crown Heights riot in Brooklyn. Racial tensions kick off. It's a nightmare for the mayor, there's this sense that he has lost control. The following year, you have this infamous police protest at City Hall where it becomes clear the relationship between the cops and the mayor has totally evaporated. How does all that play into the mayoral race between Dinkins and Giuliani?It was unintentional, but a lot of the blame for Crown Heights falls on the police department. The part of the story that is better known is that there was a procession for a Hasidic rabbi that was led by a police car. He would go to his wife's grave, and he got a little three-car motorcade. At some point, the police look at this and go "Why are we doing this? We're going to change it." The man who made the deal said ‘I"m retiring in a couple weeks, can we just leave it till then? Because I gave him my word." They're like, "Alright, whatever."This motor car procession is then involved in a car crash, and a young child named Gavin Cato is killed, and another girl is severely injured. The volunteer, Jewish-run ambulance shows up and decides they don't have the equipment: they call for a professional city ambulance. Once that ambulance is on the way, they take the mildly-injured Jewish people to the hospital. The rumor starts that the Jewish ambulance abandoned the black children to die.This isn't the first incident. There's long been strife over property and who the landlord is. But this was the spark that set off riots. A young Jewish man was randomly attacked on the street and was killed.As an aside, he also shouldn't have died, but at the hospital they missed internal bleeding.Meanwhile, the police department has no real leadership at the time. One chief is going to retire, another is on vacation, a third doesn't know what he's doing, and basically everyone is afraid to do anything. So police do nothing. They pull back, and you have three days of very anti-Semitic riots. Crowds chanting "Kill the Jews" and marching on the Lubavitch Hasidic Headquarters. Al Sharpton shows up. The riots are blamed on Dinkins, which is partly fair, but a lot of that's on the NYPD. Finally, the mayor and the police commissioner go to see what's going on and they get attacked. It's the only time in New York City history that there's ever been an emergency call from the police commissioner's car. People are throwing rocks at it.It took three days to realise this, but that's when they say “We have to do something here,” and they gather a group of officers who later become many of Bratton's main chiefs at the time [Bill Bratton was Commissioner of the NYPD from 1994-1996, under Giuliani]: Mike Julian, Louis Anemone, Ray Kelly, and [John] Timoney. They end the unrest in a day. They allow people to march, they get the police department to set rules. It still goes on for a bit, but no one gets hurt after that, and that's it.It was a huge, national story at the time, but a lot of the details were not covered. Reporters were taken from their car and beaten and stripped. The significance was downplayed at the time, especially by the New York Times, I would say.That's followed by the Washington Heights riots, which is a different story. A drug dealer was shot and killed by cops. There were rumors, which were proven to be false, that he was executed and unarmed. Then there were three days of rioting there. It wasn't quite as severe, but 53 cops were hurt, 120 stores were set on fire, and Mayor Dinkins paid for the victim's family to go to the Dominican Republic for the funeral. The police perspective again was, “You're picking the wrong side here.”Then there's the so-called Police Riot at City Hall. Nominally, it was about the CCRB, the Civilian Complaint Review Board, and setting up an accountability mechanism to control cops. But really it was just an anti-Dinkins protest. It was drunken and unruly. The cops stormed the steps of City Hall. I have the account of one of the cops who was on the top of those steps looking at this mob of cops storming to him, and he's getting worried he's going to be killed in a crush. There were racist chants from off-duty cops in the crowd. It did not reflect well on police officers. But it showed this hatred of David Dinkins, who was seen as siding with criminals and being anti-police. The irony is that Dinkins is the one who ends up hiring all the cops that Giuliani gets credit for.In the “Safe Streets, Safe City” program?Yes. That was because a white tourist, Brian Watkins, was killed in a subway station protecting his parents who were getting robbed. That led to the famous headline [in the New York Post] of “Dave, do something! Crime-ravaged city cries out for help.” He, with City Council President Peter Vallone, Sr., drafted and pushed through this massive hiring of police officers, “Safe Streets, Safe City.”The hiring wasn't fast-tracked. It might be because Dinkins's people didn't really want more cops. But it was a Dinkins push that got a massive hiring of cops. When the first huge class of police officers graduated, Bill Bratton was there and not David Dinkins.Some interviewees in your book talk about how there's physically not enough room in the police academies at this time, so they have to run classes 24/7. You cycle cohorts in and out of the same classroom, because there are too many new cops for the facilities.You have thousands of cops going through it at once. Everyone describes it as quite a chaotic scene. But it would have been hard to do what the NYPD did without those cops. Ray Kelly, who was police commissioner under Dinkins at the end [from 1992 to 1994] before he became police commissioner for 12 years under Bloomberg [from 2002 to 2013] probably could have done something with those cops too, but he never had the chance, because the mayoral leadership at the time was much more limiting in what they wanted cops to do.Crime starts declining slowly in the first few years of the ‘90s under Dinkins, and then in ‘93 Giuliani wins a squeaker of a mayoral election against Dinkins.One of the major issues was the then-notorious “squeegee men” of New York City. These were guys who would go to cars stopped at bridges and tunnel entrances and would rub a squeegee over the windshield asking for money. It was unpleasant, intimidating, and unwanted, and it was seen as one of those things that were just inevitable. Like graffiti on the subway in the ‘80s. Nothing we can do about it because these poor people don't have jobs or housing or whatever.The irony is that Bratton and Giuliani were happy to take credit for that, and it was an issue in the mayoral campaign, but it was solved under David Dinkins and Ray Kelly and Mike Julian with the help of George Kelling [who, with James Wilson, came up with broken windows theory]. But they never got credit for it. One wonders if, had they done that just a few months earlier, it would have shifted the entire campaign and we'd have a different course of history in New York City.It's a great example of a couple of things that several people in your book talk about. One is that disorder is often caused by a very small set of individuals. There's only like 70 squeegee men, yet everybody sees them, because they're posted up at the main tunnel and bridge entrances to Manhattan. And getting them off the streets solves the problem entirely.Another emphasis in the book is how perceptions of crime are central. You quote Jack Maple, the father of Compstat, as saying, “A murder on the subway counts as a multiple murder up on the street, because everybody feels like that's their subway.” The particular locations of crimes really affect public perception.Absolutely. Perception is reality for a lot of these things, because most people aren't victimized by crime. But when people perceive that no one is in control they feel less safe. It's not that this perception is false, it just might not be directly related to an actual criminal act.The other thing I try to show is that it's not just saying, “We've got to get rid of squeegee men. How do you do it?” They had tried before, but this is why you need smart cops and good leadership, because it's a problem-solving technique, and the way to get rid of graffiti is different to the way you get rid of squeegee men.This book is in opposition to those who just say, “We can't police our way out of this problem.” No, we can. We can't police our way out of every problem. But if you define the problem as, we don't want people at intersections with squeegees, of course we can police our way out of the problem, using legal constitutional tools. You need the political will. And then the hard work starts, because you have to figure out how to actually do it.Will you describe how they tackle the squeegee men problem?Mike Julian was behind it. They hired George Kelling, who's known for broken windows. They said, “These people are here to make money. So to just go there and make a few arrests isn't going to solve the problem.” First of all, he had to figure out what legal authority [to use], and he used Traffic Reg 44 [which prohibits pedestrians from soliciting vehicle occupants]. He talked to Norm Siegel of the NYCLU [New York Civil Liberties Union] about this, who did not want this crackdown to happen. But Norman said, “Okay, this is the law, I can't fight that one. You're doing it legally. It's all in the books.” And So that took away that opposition.But the relentless part of it is key. First they filmed people. Then, when it came to enforcement, they warned people. Then they cited people, and anybody that was left they arrested. They did not have to arrest many people, because the key is they did this every four hours. It was that that changed behavior, because even a simple arrest isn't going to necessarily deter someone if it's a productive way to make money. But being out there every four hours for a couple of weeks or months was enough to get people to do something else. What that something else is, we still don't know, but we solved the squeegee problem.So in 93, Giuliani is elected by something like 50,000 votes overall. Just as an aside, in Prince of the City, Fred Siegel describes something I had no idea about. There's a Puerto Rican Democratic Councilman who flips and supports Giuliani. Mayor Eric Adams, who at the time was the head of a nonprofit for black men in law enforcement, calls him a race traitor for doing that and for being married to a white woman. There was a remarkable level of racial vitriol in that race that I totally missed.10 years ago when I started this, I asked if I could interview then-Brooklyn borough president Eric Adams, and he said yes, and the interview kept getting rescheduled, and I said, “Eh, I don't need him.” It's a regret of mine. I should have pursued that, but coulda, woulda, shoulda.Giuliani is elected, and he campaigns very explicitly on a reducing crime and disorder platform. And he hires Bill Bratton. Tell me about Bratton coming on board as NYPD commissioner.Bratton grew up in Boston, was a police officer there, became head of the New York City Transit Police when that was a separate police department. Right before he becomes NYPD Commissioner, he's back in Boston, as the Chief of Police there, and there is a movement among certain people to get Bratton the NYC job. They succeed in that, and Bratton is a very confident man. He very much took a broken windows approach and said, “We are going to focus on crime.” He has a right-hand man by the name of Jack Maple who he knows from the Transit Police. Maple is just a lieutenant in transit, and Bratton makes him the de facto number two man in the police department.Jack Maple passed away in 2001 and I didn't know what I was going to do, because it's hard to interview a man who's no longer alive. Chris Mitchell co-wrote Jack Maple's autobiography called Crime Fighter and he graciously gave me all the micro-cassettes of the original interviews he conducted with Maple around 1998. Everyone has a Jack Maple story. He's probably the most important character in Back from the Brink.Jack Maple comes in, no one really knows who he is, no one respects him because he was just a lieutenant in Transit. He goes around and asks a basic question — this is 1994 — he says, “How many people were shot in New York City in 1993?” And nobody knows. That is the state of crime-fighting in New York City before this era. There might have been 7,000 people shot in New York City in 1990 and we just don't know, even to this day.One citation from your book: in 1993, an average of 16 people were shot every day. Which is just remarkable.And remember, shootings have been declining for two or three years before that! But nobody knew, because they weren't keeping track of shootings, because it's not one of the FBI Uniform Crime Report [which tracks crime data nationally] index crimes. But wouldn't you be curious? It took Jack Maple to be curious, so he made people count, and it was findable, but you had to go through every aggravated assault and see if a gun was involved. You had to go through every murder from the previous year and see if it was a shooting. He did this. So we only have shooting data in New York City going back to 1993. It's just a simple process of caring.The super-short version of Back from the Brink is it was a change in mission statement: “We're going to care about crime.” Because they hadn't before. They cared about corruption, racial unrest, brutality, and scandal. They cared about the clearance rate for robbery a bit. You were supposed to make three arrests for every ten robberies. It didn't matter so much that you were stopping a pattern or arresting the right person, as long as you had three arrests for every ten reported crimes, that was fine.This is a story about people who cared. They're from this city — Bratton wasn't, but most of the rest are. They understood the trauma of violence and the fact that people with families were afraid to go outside, and nobody in the power structure seemed to care. So they made the NYPD care about this. Suddenly, the mid-level police executives, the precinct commanders, had to care. and the meetings weren't about keeping overtime down, instead they were about ”What are you doing to stop this shooting?”Tell listeners a little bit more about Jack Maple, because he's a remarkable character, and folks may not know what a kook he was.I think he was a little less kooky than he liked to present. His public persona was wearing a snazzy cat and spats and dressing like a fictional cartoon detective from his own mind, but he's a working-class guy from Queens who becomes a transit cop.When Bratton takes over, he writes a letter up the chain of command saying this is what we should do. Bratton read it and said, “This guy is smart.” Listening to 80 hours of Jack Maple, everyone correctly says he was a smart guy, but he had a very working-class demeanor and took to the elite lifestyle. He loved hanging out and getting fancy drinks at the Plaza Hotel. He was the idea man of the NYPD. Everyone has a Jack Maple imitation. “You're talking to the Jackster,” he'd say. He had smart people working under him who were supportive of this. But it was very much trying to figure out as they went along, because the city doesn't stop nor does it sleep.He was a bulls***er, but he's the one who came up with the basic outline of the strategy of crime reduction in New York City. He famously wrote it on a napkin at Elaine's, and it said, “First, we need to gather accurate and timely intelligence.” And that was, in essence, CompStat. “Then, we need to deploy our cops to where they need to be.” That was a big thing. He found out that cops weren't working: specialized units weren't working weekends and nights when the actual crime was happening. They had their excuses, but basically they wanted a cushy schedule. He changed that. Then, of course, you have to figure out what you're doing, what the effective tactics are. Then, constant follow up and assessment.You can't give up. You can't say “Problem solved.” A lot of people say it wasn't so much if your plan didn't work, you just needed a Plan B. It was the idea that throwing your hands in the air and saying, “What are you going to do?” that became notoriously unacceptable under Chief Anemone's stern demeanor at CompStat. These were not pleasant meetings. Those are the meetings that both propagated policies that work and held officers accountable. There was some humiliation going on, so CompStat was feared.Lots of folks hear CompStat and think about better tracking of crime locations and incidents. But as you flesh out, the meat on the bones of CompStat was this relentless follow-up. You'd have these weekly meetings early in the morning with all the precinct heads. There were relentless asks from the bosses, “What's going on in your district or in your precinct? Can you explain why this is happening? What are you doing to get these numbers down?” And follow-ups the following week or month. It was constant.CompStat is often thought of as high-tech computer stuff. It wasn't. There was nothing that couldn't have been done with old overhead projectors. It's just that no one had done it before. Billy Gorta says it's a glorified accountability system at a time when nobody knew anything about computers. Everyone now has access to crime maps on a computer. It was about actually gathering accurate, timely data.Bratton was very concerned that these numbers had to be right. It was getting everyone in the same room and saying, “This is what our focus is going to be now.” And getting people to care about crime victims, especially when those crime victims might be unsympathetic because of their demeanor, criminal activity, or a long arrest record. “We're going to care about every shooting, we're going to care about every murder.”Part of it was cracking down on illegal guns. There were hundreds of tactics. The federal prosecutors also played a key role. It was getting this cooperation. Once it started working and Giuliani made it a major part of claiming success as mayor, suddenly everyone wanted to be part of this, and you had other city agencies trying to figure it out. So it was a very positive feedback loop, once it was seen as a success.When Bratton came on the job, he said, “I'm going to bring down crime 15%.” No police commissioner had ever said that before. In the history of policing before 1994, no police commissioner ever promised a double-digit reduction in crime or even talked about it. People said “That's crazy.” It was done, and then year after year. That's the type of confidence that they had. They were surprised it worked as well as it did, but they all had the sense that there's a new captain on this ship, and we're trying new things. It was an age of ideas and experiment.And it was a very short time.That's the other thing that surprised me. Giuliani fired Bratton in the middle of ‘96.It's remarkable. Bratton comes in ‘94, and August 1994 is where you see crime drop off a cliff. You have this massive beginning of the reduction that continues.That inflection point is important for historical knowledge. I don't address alternatives that other people have proposed [to explain the fall in crime] — For example, the reduction in lead [in gasoline, paint, and water pipes] or legalized abortion with Roe v. Wade [proposed by Stephen Dubner].Reasonable people can differ. Back from the Brink focuses on the police part of the equation. Today, almost nobody, except for a few academics, says that police had nothing to do with the crime drop. That August inflection is key, because there is nothing in a lagged time analysis going back 20 years that is going to say that is the magic month where things happened. Yet if you look at what happened in CompStat, that's the month they started getting individual officer data, and noticing that most cops made zero arrests, and said, “Let's get them in the game as well.” And that seemed to be the key; that's when crime fell off the table. The meetings started in April, I believe, but August is really when the massive crime drop began.To your point about the confidence that crime could be driven down double digits year over year, there's a great quote you have from Jack Maple, where he says to a fellow cop, “This is going to be like shooting fish in a barrel. As long as we have absolute control, we can absolutely drive this number into the floor.”One detail I enjoyed was that Jack Maple, when he was a transit cop, would camp out under a big refrigerator box with little holes cut out for eyes and sit on the subway platform waiting for crooks.For people who are interested in Jack Maple, it is worth reading his autobiography, Crime Fighter. Mike Daly wrote New York's Finest, which uses the same tapes that I had access to, and he is much more focused on that. He's actually the godfather of Jack Maple's son, who is currently a New York City police officer. But Maple and co were confident, and it turned out they were right.As well as having changes in tactics and approach and accountability across the NYPD, you also have a series of specific location cleanups. You have a specific initiative focused on the Port Authority, which is a cesspool at the time, an initiative in Times Square, the Bryant Park cleanup, and then Giuliani also focuses on organized crime on the Fulton Fish Market, and this open-air market in Harlem.I was struck that there was both this general accountability push in the NYPD through CompStat, and a relentless focus on cleaning up individual places that were hubs of disorder.I'm not certain the crime drop would have happened without reclamation of public spaces and business improvement districts. Bryant Park's a fascinating story because Dan Biederman, who heads the Corporation, said, “People just thought it was like a lost cause, this park can't be saved. The city is in a spiral of decline.” He uses Jane Jacobs' “eyes on the street” theory and then George Kelling and James Q. Wilson's broken windows theory. The park has money — not city money, but from local property owners — and it reopens in 1991 to great acclaim and is still a fabulous place to be. It showed for the first time that public space was worth saving and could be saved. New York City at the time needed that lesson. It's interesting that today, Bryant Park has no permanent police presence and less crime. Back in the ‘80s, Bryant Park had an active police presence and a lot more crime.The first class I ever taught when I started at John Jay College in 2004, I was talking about broken windows. A student in the class named Jeff Marshall, who is in my book, told me about Operation Alternatives at the Port Authority. He had been a Port Authority police officer at the time, and I had not heard of this. People are just unaware of this part of history. It very much has lessons for today, because in policing often there's nothing new under the sun. It's just repackaged, dusted off, and done again. The issue was, how do we make the Port Authority safe for passengers? How do we both help and get rid of people living in the bus terminal? It's a semi-public space, so it makes it difficult. There was a social services element about it, that was Operational Alternatives. A lot of people took advantage of that and got help. But the flip side was, you don't have to take services, but you can't stay here.I interviewed the manager of the bus terminal. He was so proud of what he did. He's a bureaucrat, a high-ranking one, but a port authority manager. He came from the George Washington Bridge, which he loved. And he wonders, what the hell am I going to do with this bus terminal? But the Port Authority cared, because they're a huge organization and that's the only thing with their name on it — They also control JFK Airport and bridges and tunnels and all the airports, but people call the bus terminal Port Authority.They gave him almost unlimited money and power and said, “Fix it please, do what you've got to do,” and he did. It was environmental design, giving police overtime so they'd be part of this, a big part of it was having a social service element so it wasn't just kicking people out with nowhere to go.Some of it was also setting up rules. This also helped Bratton in the subway, because this happened at the same time. The court ruled that you can enforce certain rules in the semi-public spaces. It was not clear until this moment whether it was constitutional or not. To be specific, you have a constitutional right to beg on the street, but you do not have a constitutional right to beg on the subway. That came down to a court decision. Had that not happened, I don't know if in the long run the crime drop would have happened.That court decision comes down to the specific point that it's not a free-speech right on the subway to panhandle, because people can't leave, because you've got them trapped in that space.You can't cross the street to get away from it. But it also recognized that it wasn't pure begging, that there was a gray area between aggressive begging and extortion and robbery.You note that in the early 1990s, one-third of subway commuters said they consciously avoided certain stations because of safety, and two thirds felt coerced to give money by aggressive panhandling.The folks in your book talk a lot about the 80/20 rule applying all over the place. That something like 20% of the people you catch are committing 80% of the crimes.There's a similar dynamic that you talk about on the subways, both in the book and in your commentary over the past couple years about disorder in New York. You say approximately 2,000 people with serious mental illness are at risk for street homelessness, and these people cycle through the cities, streets, subways, jails, and hospitals.What lessons from the ‘90s can be applied today for both helping those people and stopping them being a threat to others?Before the ‘80s and Reagan budget cuts there had been a psychiatric system that could help people. That largely got defunded. [Deinstitutionalization began in New York State earlier, in the 1960s.] We did not solve the problem of mental health or homelessness in the ‘90s, but we solved the problem of behavior. George Kelling [of broken windows theory] emphasized this repeatedly, and people would ignore it. We are not criminalizing homelessness or poverty. We're focusing on behavior that we are trying to change. People who willfully ignore that distinction almost assume that poor people are naturally disorderly or criminal, or that all homeless people are twitching and threatening other people. Even people with mental illness can behave in a public space.Times have changed a bit. I think there are different drugs now that make things arguably a bit worse. I am not a mental health expert, but we do need more involuntary commitment, not just for our sake, but for theirs, people who need help. I pass people daily, often the same person, basically decomposing on a subway stop in the cold. They are offered help by social services, and they say no. They should not be allowed to make that choice because they're literally dying on the street in front of us. Basic humanity demands that we be a little more aggressive in forcing people who are not making rational decisions, because now you have to be an imminent threat to yourself or others. That standard does need to change. But there also need to be mental health beds available for people in this condition.I don't know what the solution is to homelessness or mental health. But I do know the solution to public disorder on the subway and that's, regardless of your mental state or housing status, enforcing legal, constitutional rules, policing behavior. It does not involve locking everybody up. It involves drawing the line between acceptable and unacceptable behavior. It's amazing how much people will comply with those rules.That presents the idea that someone's in charge, it's not a free-for-all. You get that virtuous loop, which New York had achieved in 2014–2016, when crime was at an all-time low in the city. Then the politicians decided public order wasn't worth preserving anymore. These are political choices.I had a similar version of this conversation with a friend who was shocked that there were zero murders on the subway in 2017 and that that number was stable: you had one or two a year for several years in the mid-2010s.It was five or fewer a year from 1997 to 2019, and often one or two. Then you have zero in 2017. There were [ten in 2022]. It coincides perfectly with an order from [Mayor] de Blasio's office and the homeless czar [Director of Homeless Services Steven] Banks [which] told police to stop enforcing subway rules against loitering. The subways became — once again — a de facto homeless shelter. Getting rule-violating homeless people out of the subway in the late ‘80s was such a difficult and major accomplishment at the time, and to be fair it's not as bad as it was.The alternative was that homeless outreach was supposed to offer people services. When they decline, which 95% of people do, you're to leave them be. I would argue again, I don't think that's a more humane stance to take. But it's not just about them, it's about subway riders.There's one story that I think was relevant for you to tell. You were attacked this fall on a subway platform by a guy threatening to kill you. It turns out he's had a number of run-ins with the criminal justice system. Can you tell us where that guy is now?I believe he's in prison now. The only reason I know who it is is because I said, one day I'm going to see his picture in the New York Post because he's going to hurt somebody. Am I 100 percent certain it's Michael Blount who attacked me? No, but I'm willing to call him out by name because I believe it is. He was out of prison for raping a child, and he slashed his ex-girlfriend and pushed her on the subway tracks. And then was on the lam for a while. I look at him and the shape of his face, his height, age, build, complexion, and I go, that's got to be him.I wasn't hurt, but he gave me a sucker punch trying to knock me out and then chased me a bit threatening to kill me, and I believe he wanted to. It's the only time I ever was confronted by a person who I really believe wanted to kill me, and this includes policing in the Eastern District in Baltimore. It was an attempted misdemeanor assault in the long run. But I knew it wasn't about me. It was him. I assume he's going to stay in prison longer for what he did to his ex-girlfriend. But I never thought it would happen to me. I was lucky the punch didn't connect.Peter Moskos's new book is Back from the Brink, Inside the NYPD and New York City's Extraordinary 1990s Crime Drop.My reading listEssays:Johnny Hirschauer's reporting, including “A Failed 'Solution' to 'America's Mental Health Crisis',“ “Return to the Roots,” and “The Last Institutions.” “Broken Windows: The Police and Neighborhood Safety,” by George L. Kelling and James Q. Wilson. ​“It's Time to Talk About America's Disorder Problem,” Charles Lehman.Books:Ghettoside: A True Story of Murder in America, Jill Leovy.​Prince of the City: Giuliani, New York, and the Genius of American Life, Fred Siegel.​ Cop in the Hood: My Year Policing Baltimore's Eastern District, Peter Moskos.​Dreamland: The True Tale of America's Opiate Epidemic, Sam Quinones.​Bonfire of the Vanities, Tom Wolfe. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.statecraft.pub

AWS Morning Brief for the week of September 3rd, with Mike Julian. Links:Amazon Braket adds support for Rigetti's 84-Qubit Ankaa™-2 system, our largest gate-based superconducting deviceAnnouncing general availability of Amazon EC2 G6e instancesAmazon EC2 status checks now support reachability health of attached EBS volumesAmazon EMR support prioritized and capacity-optimized-prioritized allocation strategies for EC2 instancesAmazon OpenSearch Service now supports Graviton3 (C7g, M7g, R7g, R7gd) instances AWS announces Amazon-provided contiguous IPv4 blocksAmazon S3 now supports conditional writesAmazon S3 adds additional context to HTTP 403 Access Denied error messagesAmazon S3 no longer charges for several HTTP error codesAWS CodeBuild now supports Mac buildsAWS Identity and Access Management now supports AWS PrivateLink in all commercial RegionsAWS Network Firewall introduces GeoIP Filtering to inspect traffic based on geographic locationAnnouncing AWS Parallel Computing ServiceNow open — AWS Asia Pacific (Malaysia) RegionAWS Lambda introduces recursive loop detection APIsAnnouncing AWS KMS Elliptic Curve Diffie-Hellman (ECDH) support

Cardinal's Chief Strategy Officer, Rich Briddock is joined by VP of Sales at Definitive Healthcare, Mike Julian to explore how advanced data solutions can transform your healthcare marketing strategies into powerful revenue-generating engines. You'll learn how to target the right patient populations, optimize your marketing spend, and confidently justify your strategies to stakeholders. This episode is packed with insights to help you navigate market dynamics and enhance your campaign effectiveness for better returns and improved patient care. RELATED RESOURCES Reporting Digital Marketing Performance to the C-Suite - https://www.cardinaldigitalmarketing.com/healthcare-resources/blog/marketing-reporting-executive-board/  Privacy First: Marketing Technologies That Prioritize HIPAA Compliance - https://www.cardinaldigitalmarketing.com/healthcare-resources/blog/hipaa-compliant-martech/  Patient-Centric Marketing: How to Define and Reach Your Ideal Audience -  https://www.cardinaldigitalmarketing.com/healthcare-resources/podcast/patient-centric-marketing-define-reach-audience/ 

AWS Morning Brief for the week of Monday, August 19th with Mike Julian. Links:How AWS powered Prime Day 2024 for record-breaking salesNew capabilities for Amazon EC2 On-Demand Capacity Reservations: Split, Move, and Modify additional attributesAmazon QuickSight now includes nested filtersAnnouncing Amazon S3 Express One Zone storage class support on Amazon EMRAmazon Verified Permissions improves support for OIDC identity providersAWS announces support for Cost Allocation Tags on AWS Transit GatewayAnnouncing Karpenter 1.0Visualize enterprise IP address management and planning with CIDR map

AWS Morning Brief for the week of Monday, August 12th with Mike Julian. Links:Introducing AWS End User MessagingAmazon EFS now supports up to 30 GiB/s (a 50% increase) of read throughputAmazon RDS for Db2 supports loading data from Amazon S3AWS announces private IPv6 addressing for VPCs and subnetsAnnouncing delegated administrator for Cost Optimization HubOpenSearch optimized instance (OR1) is game changing for indexing performance and cost

AWS Morning Brief for the week of Monday, August 5th with Mike Julian. Links:Introducing AWS End User MessagingAWS Graviton-based EC2 instances now support hibernationNew Amazon CloudWatch dimensions for Amazon EC2 On Demand Capacity ReservationsAWS and Multicloud: Existing capabilities & continued enhancementsDeliver Amazon CloudWatch logs to Amazon OpenSearch ServerlessCost Optimizer for Amazon WorkSpaces 2.7 releasedJeff Barr, Chief Evangelist at AWS, confirms service deprecations via Twitter

AWS Morning Brief for the week of Monday, July 29th, with Mike Julian.Links:Amazon VPC IPAM now supports BYOIP for IPs registered with any Internet RegistryAWS Cost Categories now supports “Billing Entity” dimension Enhance database performance with Amazon RDS dedicated log volumesQLDB deprecation--more on the way?

NOVA of Virginia Aquatics coaches Mike Wenert & Mike Julian join the SSDL Podcast this week! Tune in to hear more about Mike & Mike's backgrounds in coaching, their approaches to coaching, philosophy, training approach, and TONS of more advice for being the best coach you can be for your team!

Highlight from the "Faces in FinOps" Podcast. Full EP Here: Spotify HERE: In this episode of the Faces In FinOps Podcast, we dive deep into the world of FinOps with a special guest, Mike Julian, the CEO and Co-founder of The Duckbill Group. Mike shares his journey and expertise in the realm of cloud cost management, shedding light on their unique approach to optimizing AWS costs. As the leader of The Duckbill Group, Mike emphasizes the importance of architectural choices over traditional cost-saving methods like Reserved Instances and Savings Plans. Their focus on understanding and influencing costs at the architectural level has led to impressive results, with over $4 billion in contracts negotiated for their clients. Mike and Jon discuss the evolution of FinOps maturity within organizations, emphasizing that it's not always a linear path. Instead, it depends on the organization's growth and changing priorities. They also touch on the common mistakes mature FinOps teams make, such as adopting a gatekeeping posture and the challenge of educating engineering teams about cost-saving options. In the spirit of recognizing the unsung heroes of FinOps, Mike gives a shout-out to the countless practitioners working behind the scenes, making incredible strides in optimizing cloud costs within their organizations, even if they can't always be in the public eye. Join us for this insightful conversation with Mike Julian, where we explore the dynamic landscape of FinOps and cloud cost management.

Highlight from the "Faces in FinOps" Podcast. Full EP Here: Spotify HERE: In this episode of the Faces In FinOps Podcast, we dive deep into the world of FinOps with a special guest, Mike Julian, the CEO and Co-founder of The Duckbill Group. Mike shares his journey and expertise in the realm of cloud cost management, shedding light on their unique approach to optimizing AWS costs. As the leader of The Duckbill Group, Mike emphasizes the importance of architectural choices over traditional cost-saving methods like Reserved Instances and Savings Plans. Their focus on understanding and influencing costs at the architectural level has led to impressive results, with over $4 billion in contracts negotiated for their clients. Mike and Jon discuss the evolution of FinOps maturity within organizations, emphasizing that it's not always a linear path. Instead, it depends on the organization's growth and changing priorities. They also touch on the common mistakes mature FinOps teams make, such as adopting a gatekeeping posture and the challenge of educating engineering teams about cost-saving options. In the spirit of recognizing the unsung heroes of FinOps, Mike gives a shout-out to the countless practitioners working behind the scenes, making incredible strides in optimizing cloud costs within their organizations, even if they can't always be in the public eye. Join us for this insightful conversation with Mike Julian, where we explore the dynamic landscape of FinOps and cloud cost management.

In this special live-recorded episode of Screaming in the Cloud, Corey interviews himself— well, kind of. Corey hosts an AMA session, answering both live and previously submitted questions from his listeners. Throughout this episode, Corey discusses misconceptions about his public persona, the nature of consulting on AWS bills, why he focuses so heavily on AWS offerings, his favorite breakfast foods, and much, much more. Corey shares insights into how he monetizes his public persona without selling out his genuine opinions on the products he advertises, his favorite and least favorite AWS services, and some tips and tricks to get the most out of re:Invent.About CoreyCorey is the Chief Cloud Economist at The Duckbill Group. Corey's unique brand of snark combines with a deep understanding of AWS's offerings, unlocking a level of insight that's both penetrating and hilarious. He lives in San Francisco with his spouse and daughters.Links Referenced: lastweekinaws.com/disclosures: https://lastweekinaws.com/disclosures duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: As businesses consider automation to help build and manage their hybrid cloud infrastructures, deployment speed is important, but so is cost. Red Hat Ansible Automation Platform is available in the AWS Marketplace to help you meet your cloud spend commitments while delivering best-of-both-worlds support.Corey: Well, all right. Thank you all for coming. Let's begin and see how this whole thing shakes out, which is fun and exciting, and for some godforsaken reason the lights like to turn off, so we're going to see if that continues. I've been doing Screaming in the Cloud for about, give or take, 500 episodes now, which is more than a little bit ridiculous. And I figured it would be a nice change of pace if I could, instead of reaching out and talking to folks who are innovative leaders in the space and whatnot, if I could instead interview my own favorite guest: myself.Because the entire point is, I'm usually the one sitting here asking questions, so I'm instead going to now gather questions from you folks—and feel free to drop some of them into the comments—but I've solicited a bunch of them, I'm going to work through them and see what you folks want to know about me. I generally try to be fairly transparent, but let's have fun with it. To be clear, if this is your first exposure to my Screaming in the Cloud podcast show, it's generally an interview show talking with people involved with the business of cloud. It's not intended to be snarky because not everyone enjoys thinking on their feet quite like that, but rather a conversation of people about what they're passionate about. I'm passionate about the sound of my own voice. That's the theme of this entire episode.So, there are a few that have come through that are in no particular order. I'm going to wind up powering through them, and again, throw some into the comments if you want to have other ones added. If you're listening to this in the usual Screaming in the Cloud place, well, send me questions and I am thrilled to wind up passing out more of them. The first one—a great one to start—comes with someone asked me a question about the video feed. “What's with the Minecraft pickaxe on the wall?” It's made out of foam.One of my favorite stories, and despite having a bunch of stuff on my wall that is interesting and is stuff that I've created, years ago, I wrote a blog post talking about how machine learning is effectively selling digital pickaxes into a gold rush. Because the cloud companies pushing it are all selling things such as, you know, they're taking expensive compute, large amounts of storage, and charging by the hour for it. And in response, Amanda, who runs machine learning analyst relations at AWS, sent me that by way of retaliation. And it remains one of my absolute favorite gifts. It's, where's all this creativity in the machine-learning marketing? No, instead it's, “We built a robot that can think. But what are we going to do with it now? Microsoft Excel.” Come up with some of that creativity, that energy, and put it into the marketing side of the world.Okay, someone else asks—Brooke asks, “What do I think is people's biggest misconception about me?” That's a good one. I think part of it has been my misconception for a long time about what the audience is. When I started doing this, the only people who ever wound up asking me anything or talking to me about anything on social media already knew who I was, so I didn't feel the need to explain who I am and what I do. So, people sometimes only see the witty banter on Twitter and whatnot and think that I'm just here to make fun of things.They don't notice, for example, that my jokes are never calling out individual people, unless they're basically a US senator, and they're not there to make individual humans feel bad about collectively poor corporate decision-making. I would say across the board, people think that I'm trying to be meaner than I am. I'm going to be honest and say it's a little bit insulting, just from the perspective of, if I really had an axe to grind against people who work at Amazon, for example, is this the best I'd be able to do? I'd like to think that I could at least smack a little bit harder. Speaking of, we do have a question that people sent in in advance.“When was the last time that Mike Julian gave me that look?” Easy. It would have been two days ago because we were both in the same room up in Seattle. I made a ridiculous pun, and he just stared at me. I don't remember what the pun is, but I am an incorrigible punster and as a result, Mike has learned that whatever he does when I make a pun, he cannot incorrige me. Buh-dum-tss. That's right. They're no longer puns, they're dad jokes. A pun becomes a dad joke once the punch line becomes a parent. Yes.Okay, the next one is what is my favorite AWS joke? The easy answer is something cynical and ridiculous, but that's just punching down at various service teams; it's not my goal. My personal favorite is the genie joke where a guy rubs a lamp, Genie comes out and says, “You can have a billion dollars if you can spend $100 million in a month, and you're not allowed to waste it or give it away.” And the person says, “Okay”—like, “Those are the rules.” Like, “Okay. Can I use AWS?” And the genie says, “Well, okay, there's one more rule.” I think that's kind of fun.Let's see, another one. A hardball question: given the emphasis on right-sizing for meager cost savings and the amount of engineering work required to make real architectural changes to get costs down, how do you approach cost controls in companies largely running other people's software? There are not as many companies as you might think where dialing in the specifics of a given application across the board is going to result in meaningful savings. Yes, yes, you're running something in hyperscale, it makes an awful lot of sense, but most workloads don't do that. The mistakes you most often see are misconfigurations for not knowing this arcane bit of AWS trivia, as a good example. There are often things you can do with relatively small amounts of effort. Beyond a certain point, things are going to cost what they're going to cost without a massive rearchitecture and I don't advise people do that because no one is going to be happy rearchitecting just for cost reasons. Doesn't go well.Someone asks, “I'm quite critical of AWS, which does build trust with the audience. Has AWS tried to get you to market some of their services, and would I be open to do that?” That's a great question. Yes, sometimes they do. You can tell this because they wind up buying ads in the newsletter or the podcast and they're all disclaimed as a sponsored piece of content.I do have an analyst arrangement with a couple of different cloud companies, as mentioned lastweekinaws.com/disclosures, and the reason behind that is because you can buy my attention to look at your product and talk to you in-depth about it, but you cannot buy my opinion on it. And those engagements are always tied to, let's talk about what the public is seeing about this. Now, sometimes I write about the things that I'm talking about because that's where my mind goes, but it's not about okay, now go and talk about this because we're paying you to, and don't disclose that you have a financial relationship.No, that is called fraud. I figure I can sell you as an audience out exactly once, so I better be able to charge enough money to never have to work again. Like, when you see me suddenly talk about multi-cloud being great and I became a VP at IBM, about three to six months after that, no one will ever hear from me again because I love nesting doll yacht money. It'll be great.Let's see. The next one I have on my prepared list here is, “Tell me about a time I got AWS to create a pie chart.” I wish I'd see less of it. Every once in a while I'll talk to a team and they're like, “Well, we've prepared a PowerPoint deck to show you what we're talking about.” No, Amazon is famously not a PowerPoint company and I don't know why people feel the need to repeatedly prove that point to me because slides are not always the best way to convey complex information.I prefer to read documents and then have a conversation about them as Amazon tends to do. The visual approach and the bullet lists and all the rest are just frustrating. If I'm going to do a pie chart, it's going to be in service of a joke. It's not going to be anything that is the best way to convey information in almost any sense.“How many internal documents do I think reference me by name at AWS,” is another one. And I don't know the answer to documents, but someone sent me a screenshot once of searching for my name in their Slack internal nonsense thing, and it was about 10,000 messages referenced me that it found. I don't know what they were saying. I have to assume, on some level, just something that does a belt feed from my Twitter account where it lists my name or something. But I choose to believe that no, they actually are talking about me to that level of… of extreme.Let's see, let's turn back to the chat for a sec because otherwise it just sounds like I'm doing all prepared stuff. And I'm thrilled to do that, but I'm also thrilled to wind up fielding questions from folks who are playing along on these things. “I love your talk, ‘Heresy in the Church of Docker.' Do I have any more speaking gigs planned?” Well, today's Wednesday, and this Friday, I have a talk that's going out at the CDK Community Day.I also have a couple of things coming up that are internal corporate presentations at various places. But at the moment, no. I suspect I'll be giving a talk if they accept it at SCALE in Pasadena in March of next year, but at the moment, I'm mostly focused on re:Invent, just because that is eight short weeks away and I more or less destroy the second half of my year because… well, holidays are for other people. We're going to talk about clouds, as Amazon and the rest of us dance to the tune that they play.“Look in my crystal ball; what will the industry look like in 5, 10, or 20 years?” Which is a fun one. You shouldn't listen to me on this. At all. I was the person telling you that virtualization was a flash in the pan, that cloud was never going to catch on, that Kubernetes and containers had a bunch of problems that were unlikely to be solved, and I'm actually kind of enthused about serverless which probably means it's going to flop.I am bad at predicting overall trends, but I have no problem admitting that wow, I was completely wrong on that point, which apparently is a rarer skill than it should be. I don't know what the future the industry holds. I know that we're seeing some AI value shaping up. I think that there's going to be a bit of a downturn in that sector once people realize that just calling something AI doesn't mean you make wild VC piles of money anymore. But there will be use cases that filter out of it. I don't know what they're going to look like yet, but I'm excited to see it.Okay, “Have any of the AWS services increased costs in the last year? I was having a hard time finding historical pricing charts for services.” There have been repricing stories. There have been SMS charges in India that have—and pinpointed a few other things—that wound up increasing because of a government tariff on them and that cost was passed on. Next February, they're going to be charging for public IPV4 addresses.But those tend to be the exceptions. The way that most costs tend increase have been either, it becomes far cheaper for AWS to provide a service and they don't cut the cost—data transfer being a good example—they'll also often have stories in that they're going to start launching a bunch of new things, and you'll notice that AWS bills tend to grow in time. Part of that growth, part of that is just cruft because people don't go back and clean things up. But by and large, I have not seen, “This thing that used to cost you $1 is now going to cost you $2.” That's not how AWS does pricing. Thankfully. Everyone's always been scared of something like that happening. I think that when we start seeing actual increases like that, that's when it's time to start taking a long, hard look at the way that the industry is shaping up. I don't think we're there yet.Okay. “Any plans for a Last Week in Azure or a Last Week in GCP?” Good question. If so, I won't be the person writing it. I don't think that it's reasonable to expect someone to keep up with multiple large companies and their releases. I'd also say that Azure and GCP don't release updates to services with the relentless cadence that AWS does.The reason I built the thing to start with is simply because it was difficult to gather all the information in one place, at least the stuff that I cared about with an economic impact, and by the time I'd done that, it was, well, this is 80% of the way toward republishing it for other people. I expected someone was going to point me at a thing so I didn't have to do it, and instead, everyone signed up. I don't see the need for it. I hope that in those spaces, they're better at telling their own story to the point where the only reason someone would care about a newsletter would be just my sarcasm tied into whatever was released. But that's not something that I'm paying as much attention to, just because my customers are on AWS, my stuff is largely built on AWS, it's what I have to care about.Let's see here. “What do I look forward to at re:Invent?” Not being at re:Invent anymore. I'm there for eight nights a year. That is shitty cloud Chanukah come to life for me. I'm there to set things up in advance, I'm there to tear things down at the end, and I'm trying to have way too many meetings in the middle of all of that. I am useless for the rest of the year after re:Invent, so I just basically go home and breathe into a bag forever.I had a revelation last year about re:Play, which is that I don't have to go to it if I don't want to go. And I don't like the cold, the repetitive music, the giant crowds. I want to go read a book in a bathtub and call it a night, and that's what I hope to do. In practice, I'll probably go grab dinner with other people who feel the same way. I also love the Drink Up I do there every year over at Atomic Liquors. I believe this year, we're partnering with the folks over at RedMonk because a lot of the people we want to talk to are in the same groups.It's just a fun event: show up, let us buy you drinks. There's no badge scan or any nonsense like that. We just want to talk to people who care to come out and visit. I love doing that. It's probably my favorite part of re:Invent other than not being at re:Invent. It's going to be on November 29th this year. If you're listening to this, please come on by if you're unfortunate enough to be in Las Vegas.Someone else had a good question I want to talk about here. “I'm a TAM for AWS. Cost optimization is one of our functions. What do you wish we would do better after all the easy button things such as picking the right instance and family, savings plans RIs, turning off or delete orphan resources, watching out for inefficient data transfer patterns, et cetera?” I'm going to back up and say that you're begging the question here, in that you aren't doing the easy things, at least not at scale, not globally.I used to think that all of my customer engagements would be, okay after the easy stuff, what's next? I love those projects, but in so many cases, I show up and those easy things have not been done. “Well, that just means that your customers haven't been asking their TAM.” Every customer I've had has asked their TAM first. “Should we ask the free expert or the one that charges us a large but reasonable fixed fee? Let's try the free thing first.”The quality of that advice is uneven. I wish that there were at least a solid baseline. I would love to get to a point where I can assume that I can go ahead and be able to just say, “Okay, you've clearly got your RI stuff, you're right-sizing, you're deleting stuff you're not using, taken care of. Now, let's look at the serious architecture stuff.” It's just rare that I get to see it.“What tool, feature, or widget do I wish AWS would build into the budget console?” I want to be able to set a dollar figure, maybe it's zero, maybe it's $20, maybe it is irrelevant, but above whatever I set, the account will not charge me above that figure, period. If that means they have to turn things off if that means they had to delete portions of data, great. But I want that assurance because even now when I kick the tires in a new service, I get worried that I'm going to wind up with a surprise bill because I didn't understand some very subtle interplay of the dynamics. And if I'm worried about that, everyone else is going to wind up getting caught by that stuff, too.I want the freedom to experiment and if it smacks into a wall, okay, cool. That's $20. That was worth learning that. Whatever. I want the ability to not be charged unreasonable overages. And I'm not worried about it turning from 20 into 40. I'm worried about it turning from 20 into 300,000. Like, there's the, “Oh, that's going to have a dent on the quarterlies,” style of [numb 00:16:01]—All right. Someone also asked, “What is the one thing that AWS could do that I believe would reduce costs for both AWS and their customers. And no, canceling re:Invent doesn't count.” I don't think about it in that way because believe it or not, most of my customers don't come to me asking to reduce their bill. They think they do at the start, but what they're trying to do is understand it. They're trying to predict it.Yes, they want to turn off the waste in the rest, but by and large, there are very few AWS offerings that you take a look at and realize what you're getting for it and say, “Nah, that's too expensive.” It can be expensive for certain use cases, but the dangerous part is when the costs are unpredictable. Like, “What's it going to cost me to run this big application in my data center?” The answer is usually, “Well, run it for a month, and then we'll know.” But that's an expensive and dangerous way to go about finding things out.I think that customers don't care about reducing costs as much as they think; they care about controlling them, predicting them, and understanding them. So, how would they make things less expensive? I don't know. I suspect that data transfer if they were to reduce that at least cross-AZ or eliminate it ideally, you'd start seeing a lot more compute usage in multiple AZs. I've had multiple clients who are not spinning things up in multi-AZ, specifically because they'll take the reliability trade-off over the extreme cost of all the replication flowing back and forth. Aside from that, they mostly get a lot of the value right in how they price things, which I don't think people have heard me say before, but it is true.Someone asked a question here of, “Any major trends that I'm seeing in EDP/PPA negotiations?” Yeah, lately, in particular. Used to be that you would have a Marketplace as the fallback, where it used to be that 50 cents of every dollar you spent on Marketplace would count. Now, it's a hundred percent up to a quarter of your commit. Great.But when you have a long-term commitment deal with Amazon, now they're starting to push for all—put all your other vendors onto the AWS Marketplace so you can have a bigger commit and thus a bigger discount, which incidentally, the discount does not apply to Marketplace spend. A lot of folks are uncomfortable with having Amazon as the middleman between all of their vendor relationships. And a lot of the vendors aren't super thrilled with having to pay percentages of existing customer relationships to Amazon for what they perceive to be remarkably little value. That's the current one.I'm not seeing generative AI play a significant stake in this yet. People are still experimenting with it. I'm not seeing, “Well, we're spending $100 million a year, but make that 150 because of generative AI.” It's expensive to play with gen-AI stuff, but it's not driving the business spend yet. But that's the big trend that I'm seeing over the past, eh, I would say, few months.“Do I use AWS for personal projects?” The first problem there is, well, what's a personal project versus a work thing? My life is starting to flow in a bunch of weird different ways. The answer is yes. Most of the stuff that I build for funsies is on top of AWS, though there are exceptions. “Should I?” Is the follow-up question and the answer to that is, “It depends.”The person is worrying about cost overruns. So, am I. I tend to not be a big fan of uncontrolled downside risk when something winds up getting exposed. I think that there are going to be a lot of caveats there. I know what I'm doing and I also have the backstop, in my case, of, I figure I can have a big billing screw-up or I have to bend the knee and apologize and beg for a concession from AWS, once.It'll probably be on a billboard or something one of these days. Lord knows I have it coming to me. That's something I can use as a get-out-of-jail-free card. Most people can't make that guarantee, and so I would take—if—depending on the environment that you know and what you want to build, there are a lot of other options: buying a fixed-fee VPS somewhere if that's how you tend to think about things might very well be a cost-effective for you, depending on what you're building. There's no straight answer to this.“Do I think Azure will lose any market share with recent cybersecurity kerfuffles specific to Office 365 and nation-state actors?” No, I don't. And the reason behind that is that a lot of Azure spend is not necessarily Azure usage; it's being rolled into enterprise agreements customers negotiate as part of their on-premises stuff, their operating system licenses, their Office licensing, and the rest. The business world is not going to stop using Excel and Word and PowerPoint and Outlook. They're not going to stop putting Windows on desktop stuff. And largely, customers don't care about security.They say they do, they often believe that they do, but I see where the bills are. I see what people spend on feature development, I see what they spend on core infrastructure, and I see what they spend on security services. And I have conversations about budgeting with what are you doing with a lot of these things? The companies generally don't care about this until right after they really should have cared. And maybe that's a rational effect.I mean, take a look at most breaches. And a year later, their stock price is larger than it was when they dispose the breach. Sure, maybe they're burning through their ablated CISO, but the business itself tends to succeed. I wish that there were bigger consequences for this. I have talked to folks who will not put specific workloads on Azure as a result of this. “Will you talk about that publicly?” “No, because who can afford to upset Microsoft?”I used to have guests from Microsoft on my show regularly. They don't talk to me and haven't for a couple of years. Scott Guthrie, the head of Azure, has been on this show. The problem I have is that once you start criticizing their security posture, they go quiet. They clearly don't like me.But their options are basically to either ice me out or play around with my seven seats for Office licensing, which, okay, whatever. They don't have a stick to hit me with, in the way that they do most companies. And whether that's true or not that they're going to lash out like that, companies don't want to take the risk of calling Microsoft out in public. Too big to be criticized as sort of how that works.Let's see, someone else asks, “How can a startup get the most out of its startup status with AWS?” You're not going to get what you think you want from AWS in this context. “Oh, we're going to be a featured partner so they market us.” I've yet to hear a story about how being featured by AWS for something has dramatically changed the fortunes of a startup. Usually, they'll do that when there's either a big social mission and you never hear about the company again, or they're a darling of the industry that's taking the world by fire and they're already [at 00:22:24] upward swing and AWS wants to hang out with those successful people in public and be seen to do so.The actual way that startup stuff is going to manifest itself well for you from AWS is largely in the form of credits as you go through Activate or one of their other programs. But be careful. Treat them like actual money, not this free thing you don't have to worry about. One day they expire or run out and suddenly you're going from having no dollars going to AWS to ten grand a month and people aren't prepared for that. It's, “Wait. So you mean this costs money? Oh, my God.”You have to approach it with a sense of discipline. But yeah, once you—if you can do that, yeah, free money and a free cloud bill for a few years? That's not nothing. I also would question the idea of being able to ask a giant company that's worth a trillion-and-a-half dollars and advice for how to be a startup. I find that one's always a little on the humorous side myself.“What do I think is the most underrated service or feature release from 2023? Full disclosures, this means I'll make some content about it,” says Brooke over at AWS. Oh, that's a good question. I'm trying to remember when various things have come out and it all tends to run together. I think that people are criticizing AWS for charging for IPV4 an awful lot, and I think that that is a terrific change, just because I've seen how wasteful companies are with public IP addresses, which are basically an exhausted or rapidly exhausting resource.And they just—you spend tens or hundreds of thousands of these things and don't use reason to think about that. It'll be one of the best things that we've seen for IPV6 adoption once AWS figures out how to make that work. And I would say that there's a lot to be said for since, you know, IPV4 is exhausted already, now we're talking about can we get them on the secondary markets, you need a reasonable IP plan to get some of those. And… “Well, we just give them the customers and they throw them away.” I want AWS to continue to be able to get those for the stuff that the rest of us are working on, not because one big company uses a million of them, just because, “Oh, what do you mean private IP addresses? What might those be?” That's part of it.I would say that there's also been… thinking back on this, it's unsung, the compute optimizer is doing a lot better at recommending things than it used to be. It was originally just giving crap advice, and over time, it started giving advice that's actually solid and backs up what I've seen. It's not perfect, and I keep forgetting it's there because, for some godforsaken reason, it's its own standalone service, rather than living in the billing console where it belongs. But no one's excited about a service like that to the point where they talk about or create content about it, but it's good, and it's getting better all the time. That's probably a good one. They recently announced the ability for it to do GPU instances which, okay great, for people who care about that, awesome, but it's not exciting. Even I don't think I paid much attention to it in the newsletter.Okay, “Does it make economic sense to bring your own IP addresses to AWS instead of paying their fees?” Bring your own IP, if you bring your own allocation to AWS, costs you nothing in terms of AWS costs. You take a look at the market rate per IP address versus what AWS costs, you'll hit break even within your first year if you do it. So yeah, it makes perfect economic sense to do it if you have the allocation and if you have the resourcing, as well as the ability to throw people at the problem to do the migration. It can be a little hairy if you're not careful. But the economics, the benefit is clear on that once you account for those variables.Let's see here. We've also got tagging. “Everyone nods their heads that they know it's the key to controlling things, but how effective are people at actually tagging, especially when new to cloud?” They're terrible at it. They're never going to tag things appropriately. Automation is the way to do it because otherwise, you're going to spend the rest of your life chasing developers and asking them to tag things appropriately, and then they won't, and then they'll feel bad about it. No one enjoys that conversation.So, having derived tags and the rest, or failing that, having some deployment gate as early in the process as possible of, “Oh, what's the tag for this?” Is the only way you're going to start to see coverage on this. And ideally, someday you'll go back and tag a bunch of pre-existing stuff. But it's honestly the thing that everyone hates the most on this. I have never seen a company that says, “We are thrilled with our with our tag coverage. We're nailing it.” The only time you see that is pure greenfield, everything done without ClickOps, and those environments are vanishingly rare.“Outside a telecom are customers using local zones more, or at all?” Very, very limited as far as what their usage looks like on that. Because that's… it doesn't buy you as much as you'd think for most workloads. The real benefit is a little more expensive, but it's also in specific cities where there are not AWS regions, and at least in the United States where the majority of my clients are, there is not meaningful latency differences, for example, from in Los Angeles versus up to Oregon, since no one should be using the Northern California region because it's really expensive. It's a 20-millisecond round trip, which in most cases, for most workloads, is fine.Gaming companies are big exception to this. Getting anything they can as close to the customer as possible is their entire goal, which very often means they don't even go with some of the cloud providers in some places. That's one of those actual multi-cloud workloads that you want to be able to run anywhere that you can get a baseline computer up to run a container or a golden image or something. That is the usual case. The rest are, for local zones, is largely going to be driven by specific one-off weird things. Good question.Let's see, “Is S3 intelligent tiering good enough or is it worth trying to do it yourself?” Your default choice for almost everything should be intelligent tiering in 2023. It winds up costing you more only in very specific circumstances that are unlikely to be anything other than a corner case for what you're doing. And the exceptions to this are, large workloads that are running a lot of S3 stuff where the lifecycle is very well understood, environments where you're not going to be storing your data for more than 30 days in any case and you can do a lifecycle policy around it. Other than those use cases, yeah, the monitoring fee is not significant in any environment I've ever seen.And people view—touch their data a lot less than they believe. So okay, there's a monitoring fee for object, yes, but it also cuts your raw storage cost in half for things that aren't frequently touched. So, you know, think about it. Run your own numbers and also be aware that first month as it transitions in, you're going to see massive transition charges per object, but wants it's an intelligent tiering, there's no further transition charges, which is nice.Let's see here. “We're all-in on serverless”—oh good, someone drank the Kool-Aid, too—“And for our use cases, it works great. Do I find other customers moving to it and succeeding?” Yeah, I do when they're moving to it because for certain workloads, it makes an awful lot of sense. For others, it requires a complete reimagining of whatever it is that you're doing.The early successes were just doing these periodic jobs. Now, we're seeing full applications built on top of event-driven architectures, which is really neat to see. But trying to retrofit something that was never built with that in mind can be more trouble than it's worth. And there are corner cases where building something on serverless would cost significantly more than building it in a server-ful way. But its time has come for an awful lot of stuff. Now, what I don't subscribe to is this belief that oh, if you're not building something serverless you're doing it totally wrong. No, that is not true. That has never been true.Let's see what else have we got here? Oh, “Following up on local zones, how about Outposts? Do I see much adoption? What's the primary use case or cases?” My customers inherently are coming to me because of a large AWS bill. If they're running Outposts, it is extremely unlikely that they are putting significant portions of their spend through the Outpost. It tends to be something of a rounding error, which means I don't spend a lot of time focusing on it.They obviously have some existing data center workloads and data center facilities where they're going to take an AWS-provided rack and slap it in there, but it's not going to be in the top 10 or even top 20 list of service spend in almost every case as a result, so it doesn't come up. One of the big secrets of how we approach things is we start with a big number first and then work our way down instead of going alphabetically. So yes, I've seen customers using them and the customers I've talked to at re:Invent who are using them are very happy with them for the use cases, but it's not a common approach. I'm not a huge fan of the rest.“Someone said the Basecamp saved a million-and-a-half a year by leaving AWS. I know you say repatriation isn't a thing people are doing, but has my view changed at all since you've published that blog post?” No, because everyone's asking me about Basecamp and it's repatriation, and that's the only use case that they've got for this. Let's further point out that a million-and-a-half a year is not as many engineers as you might think it is when you wind up tying that all together. And now those engineers are spending time running that environment.Does it make sense for them? Probably. I don't know their specific context. I know that a million-and-a-half dollars a year to—even if they had to spend that for the marketing coverage that they're getting as a result of this, makes perfect sense. But cloud has never been about raw cost savings. It's about feature velocity.If you have a data center and you move it to the cloud, you're not going to recoup that investment for at least five years. Migrations are inherently expensive. It does not create the benefits that people often believe that they do. That becomes a painful problem for folks. I would say that there's a lot more noise than there are real-world stories [hanging 00:31:57] out about these things.Now, I do occasionally see a specific workload that is moved back to a data center for a variety of reasons—occasionally cost but not always—and I see proof-of-concept projects that they don't pursue and then turn off. Some people like to call that a repatriation. No, I call it as, “We tried and it didn't do what we wanted it to do so we didn't proceed.” Like, if you try that with any other project, no one says, “Oh, you're migrating off of it.” No, you're not. You tested it, it didn't do what it needed to do. I do see net-new workloads going into data centers, but that's not the same thing.Let's see. “Are the talks at re:Invent worth it anymore? I went to a lot of the early re:Invents and haven't and about five years. I found back then that even the level 400 talks left a lot to be desired.” Okay. I'm not a fan of attending conference talks most of the time, just because there's so many things I need to do at all of these events that I would rather spend the time building relationships and having conversations.The talks are going to be on YouTube a week later, so I would rather get to know the people building the service so I can ask them how to inappropriately use it as a database six months later than asking questions about the talk. Conference-ware is often the thing. Re:Invent always tends to have an AWS employee on stage as well. And I'm not saying that makes these talks less authentic, but they're also not going to get through slide review of, “Well, we tried to build this onto this AWS service and it was a terrible experience. Let's tell you about that as a war story.” Yeah, they're going to shoot that down instantly even though failure stories are so compelling, about here's what didn't work for us and how we got there. It's the lessons learned type of thing.Whenever you have as much control as re:Invent exhibits over its speakers, you know that a lot of those anecdotes are going to be significantly watered down. This is not to impugn any of the speakers themselves; this is the corporate mind continuing to grow to a point where risk mitigation and downside protection becomes the primary driving goal.Let's pull up another one from the prepared list here. “My most annoying, overpriced, or unnecessary charge service in AWS.” AWS Config. It's a tax on using the cloud as the cloud. When you have a high config bill, it's because it charges you every time you change the configuration of something you have out there. It means you're spinning up and spinning down EC2 instances, whereas you're going to have a super low config bill if you, you know, treat it like a big dumb data center.It's a tax on accepting the promises under which cloud has been sold. And it's necessary for a number of other things like Security Hub. Control Towers magic-deploys it everywhere and makes it annoying to turn off. And I think that that is a pure rent-seeking charge because people aren't incurring config charges if they're not already using a lot of AWS things. Not every service needs to make money in a vacuum. It's, “Well, we don't charge anything for this because our users are going to spend an awful lot of money on storing things in S3 to use our service.” Great. That's a good thing. You don't have to pile charge upon charge upon charge upon charge. It drives me a little bit nuts.Let's see what else we have here as far as questions go. “Which AWS service delights me the most?” Eesh, depends on the week. S3 has always been a great service just because it winds up turning big storage that usually—used to require a lot of maintenance and care into something I don't think about very much. It's getting smarter and smarter all the time. The biggest lie is the ‘Simple' in its name: ‘Simple Storage Service.' At this point, if that's simple, I really don't want to know what you think complex would look like.“By following me on Twitter, someone gets a lot of value from things I mention offhandedly as things everybody just knows. For example, which services are quasi-deprecated or outdated, or what common practices are anti-patterns? Is there a way to learn this kind of thing all in one go, as in a website or a book that reduces AWS to these are the handful of services everybody actually uses, and these are the most commonly sensible ways to do it?” I wish. The problem is that a lot of the stuff that everyone knows, no, it's stuff that at most, maybe half of the people who are engaging with it knew.They find out by hearing from other people the way that you do or by trying something and failing and realizing, ohh, this doesn't work the way that I want it to. It's one of the more insidious forms of cloud lock-in. You know how a service works, how a service breaks, what the constraints are around when it starts and it stops. And that becomes something that's a hell of a lot scarier when you have to realize, I'm going to pick a new provider instead and relearn all of those things. The reason I build things on AWS these days is honestly because I know the ways it sucks. I know the painful sharp edges. I don't have to guess where they might be hiding. I'm not saying that these sharp edges aren't painful, but when you know they're there in advance, you can do an awful lot to guard against that.“Do I believe the big two—AWS and Azure—cloud providers have agreed between themselves not to launch any price wars as they already have an effective monopoly between them and [no one 00:36:46] win in a price war?” I don't know if there's ever necessarily an explicit agreement on that, but business people aren't foolish. Okay, if we're going to cut our cost of service, instantly, to undercut a competitor, every serious competitor is going to do the same thing. The only reason to do that is if you believe your margins are so wildly superior to your competitors that you can drive them under by doing that or if you have the ability to subsidize your losses longer than they can remain a going concern. Microsoft and Amazon are—and Google—are not in a position where, all right, we're going to drive them under.They can both subsidize losses basically forever on a lot of these things and they realize it's a game you don't win in, I suspect. The real pricing pressure on that stuff seems to come from customers, when all right, I know it's big and expensive upfront to buy a SAN, but when that starts costing me less than S3 on a per-petabyte basis, that's when you start to see a lot of pricing changing in the market. The one thing I haven't seen that take effect on is data transfer. You could be forgiven for believing that data transfer still cost as much as it did in the 1990s. It does not.“Is AWS as far behind in AI as they appear?” I think a lot of folks are in the big company space. And they're all stammering going, “We've been doing this for 20 years.” Great, then why are all of your generative AI services, A, bad? B, why is Alexa so terrible? C, why is it so clear that everything you have pre-announced and not brought to market was very clearly not envisioned as a product to be going to market this year until 300 days ago, when Chat-Gippity burst onto the scene and OpenAI [stole a march 00:38:25] on everyone?Companies are sprinting to position themselves as leaders in the AI space, despite the fact that they've gotten lapped by basically a small startup that's seven years old. Everyone is trying to work the word AI into things, but it always feels contrived to me. Frankly, it tells me that I need to just start tuning the space out for a year until things settle down and people stop describing metric math or anomaly detection is AI. Stop it. So yeah, I'd say if anything, they're worse than they appear as far as from behind goes.“I mostly focus on AWS. Will I ever cover Azure?” There are certain things that would cause me to do that, but that's because I don't want to be the last Perl consultancy is the entire world has moved off to Python. And effectively, my focus on AWS is because that's where the painful problems I know how to fix live. But that's not a suicide pact. I'm not going to ride that down in flames.But I can retool for a different cloud provider—if that's what the industry starts doing—far faster than AWS can go from its current market-leading status to irrelevance. There are certain triggers that would cause me to do that, but at the time, I don't see them in the near term and I don't have any plans to begin covering other things. As mentioned, people want me to talk about the things I'm good at not the thing that makes me completely nonsensical.“Which AWS services look like a good idea, but pricing-wise, they're going to kill you once you have any scale, especially the ones that look okay pricing-wise but aren't really and it's hard to know going in?” CloudTrail data events, S3 Bucket Access logging any of the logging services really, Managed NAT Gateways in a bunch of cases. There's a lot that starts to get really expensive once you hit certain points of scale with a corollary that everyone thinks that everything they're building is going to scale globally and that's not true. I don't build things as a general rule with the idea that I'm going to get ten million users on it tomorrow because by the time I get from nothing to substantial workloads, I'm going to have multiple refactors of what I've done. I want to get things out the door as fast as possible and if that means that later in time, oh, I accidentally built Pinterest. What am I going to do? Well, okay, yeah, I'm going to need to rebuild a whole bunch of stuff, but I'll have the user traffic and mindshare and market share to finance that growth.Early optimization on stuff like this causes a lot more problems than it solves. “Best practices and anti-patterns in managing AWS costs. For context, you once told me about a role that I had taken that you'd seen lots of companies tried to create that role and then said that the person rarely lasts more than a few months because it just isn't effective. You were right, by the way.” Imagine that I sometimes know what I'm talking about.When it comes to managing costs, understand what your goal is here, what you're actually trying to achieve. Understand it's going to be a cross-functional work between people in finance and people that engineering. It is first and foremost, an engineering problem—you learn that at your peril—and making someone be the human gateway to spin things up means that they're going to quit, basically, instantly. Stop trying to shame different teams without understanding their constraints.Savings Plans are a great example. They apply biggest discount first, which is what you want. Less money going out the door to Amazon, but that makes it look like anything with a low discount percentage, like any workload running on top of Microsoft Windows, is not being responsible because they're always on demand. And you're inappropriately shaming a team for something completely out of their control. There's a point where optimization no longer makes sense. Don't apply it to greenfield projects or skunkworks. Things you want to see if the thing is going to work first. You can optimize it later. Starting out with a, ‘step one: spend as little as possible' is generally not a recipe for success.What else have we got here? I've seen some things fly by in the chat that are probably worth mentioning here. Some of it is just random nonsense, but other things are, I'm sure, tied to various questions here. “With geopolitics shaping up to govern tech data differently in each country, does it make sense to even build a globally distributed B2B SaaS?” Okay, I'm going to tackle this one in a way that people will probably view as a bit of an attack, but it's something I see asked a lot by folks trying to come up with business ideas.At the outset, I'm a big believer in, if you're building something, solve it for a problem and a use case that you intrinsically understand. That is going to mean the customers with whom you speak. Very often, the way business is done in different countries and different cultures means that in some cases, this thing that's a terrific idea in one country is not going to see market adoption somewhere else. There's a better approach to build for the market you have and the one you're addressing rather than aspirational builds. I would also say that it potentially makes sense if there are certain things you know are going to happen, like okay, we validated our marketing and yeah, it turns out that we're building an image resizing site. Great. People in Germany and in the US all both need to resize images.But you know, going in that there's going to be a data residency requirement, so architecting, from day one with an idea that you can have a partition that winds up storing its data separately is always going to be to your benefit. I find aligning whatever you're building with the idea of not being creepy is often a great plan. And there's always the bring your own storage approach to, great, as a customer, you can decide where your data gets stored in your account—charge more for that, sure—but then that na—it becomes their problem. Anything that gets you out of the regulatory critical path is usually a good idea. But with all the problems I would have building a business, that is so far down the list for almost any use case I could ever see pursuing that it's just one of those, you have a half-hour conversation with someone who's been down the path before if you think it might apply to what you're doing, but then get back to the hard stuff. Like, worry on the first two or three steps rather than step 90 just because you'll get there eventually. You don't want to make your future life harder, but you also don't want to spend all your time optimizing early, before you've validated you're actually building something useful.“What unique feature of AWS do I most want to see on other cloud providers and vice versa?” The vice versa is easy. I love that Google Cloud by default has the everything in this project—which is their account equivalent—can talk to everything else, which means that humans aren't just allowing permissions to the universe because it's hard. And I also like that billing is tied to an individual project. ‘Terminate all billable resources in this project' is a button-click away and that's great.Now, what do I wish other cloud providers would take from AWS? Quite honestly, the customer obsession. It's still real. I know it sounds like it's a funny talking point or the people who talk about this the most under the cultists, but they care about customer problems. Back when no one had ever heard of me before and my AWS Bill was seven bucks, whenever I had a problem with a service and I talked about this in passing to folks, Amazonians showed up out of nowhere to help make sure that my problem got answered, that I was taken care of, that I understood what I was misunderstanding, or in some cases, the feedback went to the product team.I see too many companies across the board convinced that they themselves know best about what customers need. That occasionally can be true, but not consistently. When customers are screaming for something, give them what they need, or frankly, get out of the way so someone else can. I mean, I know someone's expecting me to name a service or something, but we've gotten past the point, to my mind, of trying to do an apples-to-oranges comparison in terms of different service offerings. If you want to build a website using any reasonable technology, there's a whole bunch of companies now that have the entire stack for you. Pick one. Have fun.We've got time for a few more here. Also, feel free to drop more questions in. I'm thrilled to wind up answering any of these things. Have I seen any—here's one that about Babelfish, for example, from Justin [Broadly 00:46:07]. “Have I seen anyone using Babelfish in the wild? It seems like it was a great idea that didn't really work or had major trade-offs.”It's a free open-source project that translates from one kind of database SQL to a different kind of database SQL. There have been a whole bunch of attempts at this over the years, and in practice, none of them have really panned out. I have seen no indications that Babelfish is different. If someone at AWS works on this or is a customer using Babelfish and say, “Wait, that's not true,” please tell me because all I'm saying is I have not seen it and I don't expect that I will. But I'm always willing to be wrong. Please, if I say something at some point that someone disagrees with, please reach out to me. I don't intend to perpetuate misinformation.“Purely hypothetically”—yeah, it's always great to ask things hypothetically—“In the companies I work with, which group typically manages purchasing savings plans, the ops team, finance, some mix of both?” It depends. The sad answer is, “What's a savings plan,” asks the company, and then we have an educational path to go down. Often it is individual teams buying them ad hoc, which can work, cannot as long as everyone's on the same page. Central planning, in a bunch of—a company that's past a certain point in sophistication is where everything winds up leading to.And that is usually going to be a series of discussions, ideally run by that group in a cross-functional way. They can be cost engineering, they can be optimization engineering, I've heard it described in a bunch of different ways. But that is—increasingly as the sophistication of your business and the magnitude of your spend increases, the sophistication of how you approach this should change as well. Early on, it's the offense of some VP of engineering at a startup. Like, “Oh, that's a lot of money,” running the analyzer and clicking the button to buy what it says. That's not a bad first-pass attempt. And then I think getting smaller and smaller buys as you continue to proceed means you can start to—it no longer becomes the big giant annual decision and instead becomes part of a frequently used process. That works pretty well, too.Is there anything else that I want to make sure I get to before we wind up running this down? To the folks in the comments, this is your last chance to throw random, awkward questions my way. I'm thrilled to wind up taking any slings, arrows, et cetera, that you care to throw my way a going once, going twice style. Okay, “What is the most esoteric or shocking item on the AWS bill that you ever found with one of your customers?” All right, it's been long enough, and I can say it without naming the customer, so that'll be fun.My personal favorite was a high five-figure bill for Route 53. I joke about using Route 53 as a database. It can be, but there are better options. I would say that there are a whole bunch of use cases for Route 53 and it's a great service, but when it's that much money, it occasions comment. It turned out that—we discovered, in fact, a data exfiltration in progress which made it now a rather clever security incident.And, “This call will now be ending for the day and we're going to go fix that. Thanks.” It's like I want a customer testimonial on that one, but for obvious reasons, we didn't get one. But that was probably the most shocking thing. The depressing thing that I see the most—and this is the core of the cost problem—is not when the numbers are high. It's when I ask about a line item that drives significant spend, and the customer is surprised.I don't like it when customers don't know what they're spending money on. If your service surprises customers when they realize what it costs, you have failed. Because a lot of things are expensive and customers know that and they're willing to take the value in return for the cost. That's fine. But tricking customers does not serve anyone well, even your own long-term interests. I promise.“Have I ever had to reject a potential client because they had a tangled mess that was impossible to tackle, or is there always a way?” It's never the technology that will cause us not to pursue working with a given company. What will is, like, if you go to our website at duckbillgroup.com, you're not going to see a ‘Buy Here' button where you ‘add one consulting, please' to your shopping cart and call it a day.It's a series of conversations. And what we will try to make sure is, what is your goal? Who's aligned with it? What are the problems you're having in getting there? And what does success look like? Who else is involved in this? And it often becomes clear that people don't like the current situation, but there's no outcome with which they would be satisfied.Or they want something that we do not do. For example, “We want you to come in and implement all of your findings.” We are advisory. We do not know the specifics of your environment and—or your deployment processes or the rest. We're not an engineering shop. We charge a fixed fee and part of the way we can do that is by controlling the scope of what we do. “Well, you know, we have some AWS bills, but we really want to—we really care about is our GCP bill or our Datadog bill.” Great. We don't focus on either of those things. I mean, I can just come in and sound competent, but that's not what adding value as a consultant is about. It's about being authoritatively correct. Great question, though.“How often do I receive GovCloud cost optimization requests? Does the compliance and regulation that these customers typically have keep them from making the needed changes?” It doesn't happen often and part of the big reason behind that is that when we're—and if you're in GovCloud, it's probably because you are a significant governmental entity. There's not a lot of private sector in GovCloud for almost every workload there. Yes, there are exceptions; we don't tend to do a whole lot with them.And the government procurement process is a beast. We can sell and service three to five commercial engagements in the time it takes to negotiate a single GovCloud agreement with a customer, so it just isn't something that we focused. We don't have the scale to wind up tackling that down. Let's also be clear that, in many cases, governments don't view money the same way as enterprise, which in part is a good thing, but it also means that, “This cloud thing is too expensive,” is never the stated problem. Good question.“Waffles or pancakes?” Is another one. I… tend to go with eggs, personally. It just feels like empty filler in the morning. I mean, you could put syrup on anything if you're bold enough, so if it's just a syrup delivery vehicle, there are other paths to go.And I believe we might have exhausted the question pool. So, I want to thank you all for taking the time to talk with me. Once again, I am Cloud Economist Corey Quinn. And this is a very special live episode of Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review wherever you can—or a thumbs up, or whatever it is, like and subscribe obviously—whereas if you've hated this podcast, same thing: five-star review, but also go ahead and leave an insulting comment, usually around something I've said about a service that you deeply care about because it's tied to your paycheck.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

About MikeBesides his duties as The Duckbill Group's CEO, Mike is the author of O'Reilly's Practical Monitoring, and previously wrote the Monitoring Weekly newsletter and hosted the Real World DevOps podcast. He was previously a DevOps Engineer for companies such as Taos Consulting, Peak Hosting, Oak Ridge National Laboratory, and many more. Mike is originally from Knoxville, TN (Go Vols!) and currently resides in Portland, OR.Links Referenced: Twitter: https://twitter.com/Mike_Julian mikejulian.com: https://mikejulian.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is brought to us in part by our friends at Datadog. Datadog is a SaaS monitoring and security platform that enables full-stack observability for modern infrastructure and applications at every scale. Datadog enables teams to see everything: dashboarding, alerting, application performance monitoring, infrastructure monitoring, UX monitoring, security monitoring, dog logos, and log management, in one tightly integrated platform. With 600-plus out-of-the-box integrations with technologies including all major cloud providers, databases, and web servers, Datadog allows you to aggregate all your data into one platform for seamless correlation, allowing teams to troubleshoot and collaborate together in one place, preventing downtime and enhancing performance and reliability. Get started with a free 14-day trial by visiting datadoghq.com/screaminginthecloud, and get a free t-shirt after installing the agent.Corey: Forget everything you know about SSH and try Tailscale. Imagine if you didn't need to manage PKI or rotate SSH keys every time someone leaves. That'd be pretty sweet, wouldn't it? With Tailscale SSH, you can do exactly that. Tailscale gives each server and user device a node key to connect to its VPN, and it uses the same node key to authorize and authenticate SSH.Basically you're SSHing the same way you manage access to your app. What's the benefit here? Built in key rotation permissions is code connectivity between any two devices, reduce latency and there's a lot more, but there's a time limit here. You can also ask users to reauthenticate for that extra bit of security. Sounds expensive?Nope, I wish it were. tail scales. Completely free for personal use on up to 20 devices. To learn more, visit snark.cloud/tailscale. Again, that's snark.cloud/tailscaleCorey: Welcome to Screaming in the Cloud, I'm Corey Quinn and I'm having something of a crisis of faith based upon a recent conversation I've had with my returning yet again guest, Mike Julian, my business partner and CEO of The Duckbill Group. Welcome back, Mike.Mike: Hi, everyone.Corey: So, the revelation that had surfaced unexpectedly was, based upon a repeated talking point where I am a terrible employee slash expensive to manage, et cetera, et cetera, and you pointed out that you've been managing me for four years or so now, at which point I did a spit take, made all the more impressive by the fact that I wasn't drinking anything at the time, and realized, “Oh, my God, you're right, but I haven't had any of the usual problems slash friction with you that I have with basically every boss I've ever had in my entire career.” So, I'm spiraling. Let's talk about that.Mike: My recollection of that conversation is slightly different than yours. Mine is that you called me and said, “Mike, I just realized that you're my boss.” And I'm like, “How do you feel about that?” He's like, “I'm not really sure.”Corey: And I'm still not entirely sure how I feel if I'm being fully honest with you. Just because it's such a weird thing to have to deal with. Because historically, I always view a managerial relationship as starting from a place of a power imbalance. And that is the one element that is missing from our relationship. We each own half the company, we can fire each other, but it takes the form of tearing the company apart, and that isn't something that we're really set up to entertain.Mike: And you know, I actually think it's deeper than that because you owning the other half of the company is not really… it's not really power in itself. Like, yeah, it is, but you could easily own half the company and have no power. Because, like, really when we talk about power, we're talking about political power, influence, and I think the reason that there is no power imbalance is because each of us does something in the company that is just as important as the other. And they're both equally valuable to the company and we both recognize the other's contributions, as that, as being equally valuable to the company. It's less to do about how much we own and more about the work that we do.Corey: Oh, of course. The ownership starts and stops entirely with the fact that neither one of us can force the other out. So it's, as opposed to well, I own 51% of the company, so when I'm tired of your bullshit, you're leaving. And that is a dynamic that's never entered into it. I'm also going to add one more thing onto what you just said, which is, both of us would sooner tear off our own skin than do the other's job.Mike: Yeah. God, I would hate to do your job, but I know you'd hate to do mine.Corey: You look at my calendar on a busy meeting day and you have a minor panic attack just looking at it where, “Oh, my God, talking to that many people.” And you are going away for a while and you come back with a whole analytical model where your first love language feels like it's spreadsheets on some days, and I look at this and it's like, “Yeah, I know what some of those numbers mean.” And it just drives me up a wall, the idea of building out a plan and an execution thing and then delegating a lot of it to other people, it does not work for my worldview in so many different ways. It's the reason I think that you and I get along. That and our shared values.Mike: I remember the first time that you and I did a consulting engagement together. We went on a multi-day trip. And at the end of, like, three days of nonstop conversations, you made a comment, it was like, “Cool. So, what are we going to do that again?” Like, you were excited by it. I can tell you're energized. And I was just thinking, “Please for love of God, I want to die right now.”Corey: One of the weirdest parts about all of it, though, is neither one of us is in a scenario where what we do for a living and how we go about it works without the other.Mike: Right. Yeah, like, this is one of the interesting things about the company we have built is that it would not work with just you or just me; it's us being co-founders is what makes it successful.Corey: The thing that I do not understand and I don't think I ever will is the idea of co-founder speed dating, where you basically go to some big networking mixer event, pick some rando off the street, and congratulations, that's your business partner. Have fun. It is not that much of an exaggeration to say that co-founding a company with someone else is like a marriage. You are creating a legal entity that without very specific controls and guidelines, you are opening yourself up to massive liability issues if the other person decides to screw you over. That is part of the reason that the values match was so important for us.Mike: Yeah, it is surprising to me how similar being co-founders and business partners is to being married. I did not expect how close those two things were. You and I spend an incredible amount of time just on the relationship for each of us, which I never expected, but makes sense in hindsight.Corey: That's I think part of it makes the whole you managing me type of relationship work is because not only can you not, “Fire me,” quote-unquote, but I can't quit without—Mike: [laugh].Corey: Leaving behind a giant pile of effort with nothing to show for it over the last four years. So, it's one of those conversation styles where we go into the conversation knowing, regardless of how heated it gets or how annoyed we are with each other, that we are not going to blow the company up because one of us is salty that week.Mike: Right. Yeah, I remember from the legal perspective, when we put together a partnership agreement, our attorneys were telling us that we really needed to have someone at the 51% owner, and we were both adamant that no, that doesn't work for us. And finally, the way that we handled it is if you and I could not handle a dispute, then the only remedy left was to shut the entire thing down. And that would be an automatic trigger. We've never ever, ever even got close to that point.But like, I like that's the structure because it really means that if you and I can't agree on something and it's a substantial thing, then there's no business, which really kind of sets the stage for how important the conversations that we have are. And of course, you and I, we're close, we have a great relationship, so that's never come up. But I do like that it's still there.Corey: I like the fact that there's always going to be an option to get out. It's not a suicide pact, for lack of a better term. But it's also something that neither one of us would ever entertain lightly. And credit where due, there have been countless conversations where you and I were diametrically opposed; we each talk through it, and one or the other of us will just do a complete one-eighty our position where, “Okay, you convinced me,” and that's it. What's so odd about that is because we don't have too many examples of that in public society, it just seems like there's now this entire focus on, “Oh, if you make an observation or a point, that's wrong, you've got to double down on it.” Why would you do that? That makes zero sense. When you've considered something of a different angle and change your mind, why waste more time on it?Mike: I think there's other interesting ones, too, where you and I have come at something from a different angle and one of us will realize that we just actually don't care as much as we thought we did. And we'll just back down because it's not the hill we want to die on.Corey: Which brings us to a good point. What hill do we want to die on?Mike: Hmm. I think we've only got a handful. I mean, as it should; like, there should not be there should not be many of them.Corey: No, no because most things can change, in the fullness of time. Just because it's not something we believe is right for the business right now does not mean it never will be.Mike: Yeah. I think all of them really come down to questions of values, which is why you and I worked so well together, in that we don't have a lot of common interests, we're at completely different stages in our lives, but we have very tightly aligned values. Which means that when we go into a discussion about something, we know where the other stands right away, like, we could generally make a pretty good guess about it. And there's often very little question about how some values discussion is going to go. Like, do we take on a certain client that is, I don't know, they build landmines? Is that a thing that we're going to do? Of course not. Like—Corey: I should clarify, we're talking here about physical landmines; not whatever disastrous failure mode your SaaS application has.Mike: [laugh]. Yeah.Corey: We know what those are.Mike: Yeah, and like, that sort of thing, you and I would never even pose the question to each other. We would just make the decision. And maybe we tell each other later because and, like, “Hey, haha, look what happened,” but there will never be a discussion around it because it just—our values are so tightly aligned that it wouldn't be necessary.Corey: Whenever we're talking to someone that's in a new sector or a company that has a different expression, we always like to throw it past each other just to double-check, you don't have a problem with—insert any random thing here; the breadth of our customer base just astounds me—and very rarely as either one of us thrown a flag on something just because we do have this affinity for saying[ yes and making money.Mike: Yeah. But you actually wanted to talk about the terribleness of managing you.Corey: Yeah. I am very curious as to what your experience has been.Mike: [laugh].Corey: And before we dive into it, I want to call out a couple of things that make me a little atypical for your typical problem employee. I am ADHD personified. My particular expression of that means that my energy level is very different at different times of day, there are times where I will get nothing done for a day or two, and then in four hours, get three weeks of work done. It is hard to predict and it's hard to schedule around and it's never clear exactly what that energy level is going to be at any given point in time. That's the starting point of this nonsense. Now, take it away.Mike: Yeah. What most people know about Corey is what everyone sees on Twitter, which is what I would call the high highs. Everyone sees you as your most energetic, or at least perceived as the most energetic. If they see you in person at a conference, it's the same sort of thing. What people don't see are your lows, which are really, really low lows.And it's not a matter of, like, you don't get anything done. Like, you know, we can handle that; it's that you disappear. And it may be for a couple hours, it may be for a couple of days, and we just don't really know what's going on. That's really hard. But then, in your high highs, they're really high, but they're also really unpredictable.So, what that means is that because you have ADHD, like, the way that your brain thinks, the way your brain works, is that you don't control what you're going to focus on, and you never know what you're going to focus on. It may be exactly what you should be focusing on, which is a huge win for everyone involved, but sometimes you focus on stuff that doesn't matter to anyone except you. Sometimes really interesting stuff comes out of that, but oftentimes it doesn't. So, helping build a structure to work around those sorts of things and to also support those sorts of things, has been one of the biggest challenges that I've had. And most of my job is really about building a support structure for you and enabling you to do your best work.So, that's been really interesting and really challenging because I do not think that way. Like, if I need to focus on something, I just say, “Great. I'm just going to focus on this thing,” and I'll focus on it until I'm done. But you don't work that way, and you couldn't conceivably work that way, ever. So, it's always been hard because I say things like, “Hey, Corey, I need you to go write this series of emails.” And you'll write them when your brain decides that wants to write them, which might be never.Corey: That's part of the problem. I've also found that if I have an idea floating around too long, it'll linger for years and I'll never write anything about it, whereas there are times when I have—the inspiration strikes, I write a one- to 2000-word blog post every week that goes out, and there are times it takes me hours and there are times I bust out the entire thing in first draft form in 20 minutes or less. Like, if it's Domino's, like, there's not going to be a refund on it. So, it's kind of wild and I wish I could harness that somehow I don't know how, but… that's one of the biggest challenges.Mike: I wish I could too, but it's one of the things that you learn to get used to. And with that, because we've worked together for so long, I've gotten to be able to tell in what state of mind you are. Like, are you in a state where if I put something in front of you, you're going to go after it hard, and like, great things are going to happen, or are you more likely to ignore that I said anything? And I can generally tell within the first sentence or so of bringing something up. But that also means that I have other—I have to be careful with how I structure requests that I have for you.In some cases, I come with a punch list of, like, here's six things I need to get through and I'm going to sit on this call while we go through them. In other cases, I have to drip them out one at a time over the span of a week just because that's how your mind is those days. That makes it really difficult because that's not how most people are managed and it's not how most people expect to manage. So, coming up with different ways to do that has been one of the trickiest things I've done.Corey: Let's move on a little bit other than managing my energy levels because that does not sound like a particularly difficult employee to manage. “Okay, great. We've got to build some buffer room into the schedule in case he winds up not delivering for a few days. Okay, we can live with that.” But oh, working with me gets so much worse.Mike: [laugh]. It absolutely does.Corey: This is my performance review. Please hit me with it.Mike: Yeah. The other major concern that has been challenging to work through that makes you really frustrating to work with, is you hate conflict. Actually, I don't actually—let me clarify that further. You avoid conflict, except your definition of conflict is more broad than most. Because when most people think of conflicts, like, “Oh, I have to go have this really hard conversation, it's going to be uncomfortable, and, like—”Corey: “Time to go fire Steven.”Mike: Right, or things like, “I have to have our performance conversation with someone.” Like, everyone hates those, but, like, there's good ways and bad ways to them, like, it's uncomfortable even at the best of times. But with you, it's more than that, it's much more broad. You avoid giving direction because you perceive giving direction as potential for conflict, and because you're so conflict-avoidant, you don't give direction to people.Which means that if someone does something you don't like, you don't say anything and then it leaves everyone on the team to say, like, “I really wish Corey would be more explicit about what he wants. I wish he was more vocal about the direction he wanted to go.” Like, “Please tell us something more.” But you're so conflict-avoidant that you don't, and no amount of begging or we're asking for it has really changed that, so we end up with these two things where you're doing most of the work yourself because you don't want to direct other people to do it.Corey: I will push back slightly on one element of that, which is when I have a strong opinion about something, I am not at all hesitant about articulating that. I mean, this is not—like, my Twitter is not performance art; it's very much what I believe. The challenge is that for so much of what we talk about internally on a day-to-day basis, I don't really have a strong opinion. And what I've always shied away from is the idea of telling people how to do their jobs. So, I want to be very clear that I'm not doing that, except when it's important.Because we've all been in environments in the corporate world where the president of the company wanders past or your grand-boss walks into the room and asks an idle question, or, “Maybe we should do this,” and it never feels like it's really just idle pondering. It's, “Welp, new strategic priority just dropped from on high.”Mike: Right.Corey: And every senior manager has a story about screwing that one up. And I have led us down that path once or twice previously. So—Mike: That's true.Corey: When I don't have a strong opinion, I think what I need to get better at is saying, “I don't give a shit,” but when I frame it like that it causes different problems.Mike: Yeah. Yeah, that's very true. I still don't completely agree with your disagreement there, but I understand your perspective. [laugh].Corey: Oh, he's not like you can fire me, so it doesn't really matter. I kid. I kid.Mike: Right. Yeah. So, I think those are the two major areas that make you a real challenge to manage and a challenge to direct. But one of the reasons why I think we've been successful at it, or at least I'll say I've been successful at managing you, is I do so with such a gentle touch that you don't realize that I'm doing anything, and I have all these different—Corey: Well, it did take me four years to realize what was going on.Mike: Yeah, like, I have all these different ways of getting you to do things, and you don't realize I'm doing them. And, like, I've shared many of them here for you for the first time. And that's really is what has worked out well. Like, a lot of the ways that I manage you, you don't realize are management.Corey: Managing shards. Maintenance windows. Overprovisioning. ElastiCache bills. I know, I know. It's a spooky season and you're already shaking. It's time for caching to be simpler. Momento Serverless Cache lets you forget the backend to focus on good code and great user experiences. With true autoscaling and a pay-per-use pricing model, it makes caching easy. No matter your cloud provider, get going for free at gomomento.co/screaming That's GO M-O-M-E-N-T-O dot co slash screamingCorey: What advice would you have for someone for whom a lot of these stories are resonating? Because, “Hey, I have a direct report is driving me to distraction and a lot sounds like what you're describing.” What do you wish you'd known sooner about how to coax performance out of me, for lack of a better phrasing?Mike: When we first started really working together, I knew what ADHD was, but I knew it from a high school paper that I did on ADHD, and it's um—oh, what was it—“The Overdiagnosis of ADHD,” which was a thing when you and I were at high school. That's all I knew is just that ADHD was suspected to be grossly overdiagnosed and that most people didn't have it. What I have learned is that yeah, that might have been true—maybe; I don't know—but for people that do have any ADHD, it's a real thing. Like, it does have some pretty substantial impact.And I wish I had known more about how that manifests, and particularly how it manifests in different people. And I wish I'd known more earlier on about the coping mechanisms that different people create for themselves and how they manage and how they—[sigh], I'm struggling to come up with the right word here, but many people who are neurodivergent in some way create coping mechanisms and ways to shift themselves to appear more neurotypical. And I wish I had understood that better. Particularly, I wish I had understood that better for you when we first started because I've kind of learned about it over time. And I spent so much time trying to get you to work the way that I work rather than understand that you work different. Had I spent more time to understand how you work and what your coping mechanisms were, the earlier years of Duckbill would have been so much smoother.Corey: And again, having this conversation has been extraordinarily helpful. On my side of it, one of the things that was absolutely transformative and caused a massive reduction in our interpersonal conflict was the very simple tool of, it's not necessarily a problem when I drop something on the floor and don't get to it, as long as I throw a hand up and say, “I'm dropping this thing,” and so someone else can catch it as we go. I don't know how much of this is ADHD speaking versus how much of it is just my own brokenness in some ways, but I feel like everyone has this neverending list of backlog tasks that they'll get to someday that generally doesn't ever seem to happen. More often than not, I wind up every few months, just looking at my ever-growing list, reset to zero and we'll start over. And every once in a while, I'll be really industrious and knock a thing or two off the list. But so many that don't necessarily matter or need to be me doing them, but it drives people to distraction when something hits my email inbox, it just dies there, for example.Mike: Yeah. One of the systems that we set up here is that if there's something that Corey does not immediately want to do, I have you send it to someone else. And generally it's to me and then I become a router for you. But making that more explicit and making that easier for you—I'm just like, “If this is not something that you're going to immediately take care of yourself, forward it to me.” And that was huge. But then other things, like when you take time off, no one knows you're taking time off. And it's an—the easiest thing is no one cares that you're taking time off; just, you know, tell us you're doing it.Corey: Yeah, there's a difference between, “I'm taking three days off,” and your case, the answer is generally, “Oh, thank God. He's finally using some of that vacation.”Mike: [laugh].Corey: The problem is there's a world of difference between, “Oh, I'm going to take these three days off,” and just not showing up that day. That tends to cause problems for people.Mike: Yeah. They're just waving a hand in the air and saying, “Hey, this is happening,” that's great. But not waving it, not saying anything at all, that's where the pain really comes from.Corey: When you take a look across your experience managing people, which to my understanding your first outing with it was at this company—Mike: Yeah.Corey: What about managing me is the least surprising and the most surprising that you've picked up during that pattern? Because again, the story has always been, “Oh, yeah, you're a terrible manager because you've never done it before,” but I look back and you're clearly the best manager I've ever had, if for no other reason than neither one of us can rage-quit. But there's a lot of artistry to how you've handled a lot of challenges that I present to you.Mike: I'm the best manager you've had because I haven't fired you. [laugh].Corey: And also, some of the best ones I have had fired me. That doesn't necessarily disqualify someone.Mike: Yeah. I want to say, I am by no means experienced as a manager. As you mentioned, this is my first outing into doing management. As my coach tells me, I'm getting better every day. I am not terrible [laugh].The—let's see—most surprising; least surprising. I don't think I have anything for least surprising. I think most surprising is how easy it is for you to accept feedback and how quickly you do something about it, how quickly you take action on that feedback. I did not expect that, given all your other proclivities for not liking managers, not liking to be managed, for someone to give feedback to you and you say, “Yep, that sounds good,” and then do it, like, that was incredibly surprising.Corey: It's one of those areas where if you're not embracing or at least paying significant attention to how you are being perceived, maybe that's a problem, maybe it's not, let's be very clear. However, there's also a lot of propensity there to just assume, “Oh, I'm right and screw everyone else.” You can do an awful lot of harm that way. And that is something I've had to become incredibly aware of, especially during the pandemic, as the size of my audience at this point more than quadrupled from the start of the pandemic. These are a bunch of people now who have never met me in person, they have no context on what I do.And I tend to view the world the way you might expect a dog to behave, who caught a car that he has absolutely no idea how to drive, and he's sort of winging it as he goes. Like, step one, let's not kill people. Step two, eh, we'll figure that out later. Like, step one is the most important.Mike: Mm-hm. Yeah.Corey: And feedback is hard to get, past a certain point. I often lament from time to time that it's become more challenging for me to weed out who the jerks are because when you're perceived to have a large platform and more or less have no problem calling large companies and powerful folk to account, everyone's nice to you. And well, “Really? He's terrible and shitty to women. That's odd. He's always been super nice to me.” Is not the glowing defense that so many people seem to think that it is. It's I have learned to listen a lot more clearly the more I speak.Mike: That's a challenge for me as well because, as we've mentioned, my first foray into management. As we've had more people in the company, that has gotten more of a challenge of I have to watch what I say because my word carries weight on its own, by virtue of my position. And you have the same problem, except yours is much more about your weight in public, rather than your weight internally.Corey: I see it as different sides of the same coin. I take it as a personal bit of a badge of honor that almost every person I meet, including the people who've worked here, have come away, very surprised by just how true to life my personality on Twitter is to how actually am when I interact with humans. You're right, they don't see the low sides, but I also try not to take that out on the staff either.Mike: [laugh]. Right.Corey: We do the best of what we have, I think, and it's gratifying to know that I can still learn new tricks.Mike: Yeah. And I'm not firing anytime soon.Corey: That's right. Thank you again for giving me the shotgun performance review. It's always appreciated. If people want to learn more, where can they find you, to get their own performance preview, perhaps?Mike: Yeah, you can find me on Twitter at @Mike_Julian. Or you can sign up for our newsletter, where I'm talking about my upcoming book on consulting at mikejulian.com.Corey: And we will put links to that into the show notes. Thanks again, sir.Mike: Thank you.Corey: Mike Julian, CEO of The Duckbill Group, my business partner, and apparently my boss. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that demonstrates the absolute worst way to respond to a negative performance evaluation.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About MikeBeside his duties as The Duckbill Group's CEO, Mike is the author of O'Reilly's Practical Monitoring, and previously wrote the Monitoring Weekly newsletter and hosted the Real World DevOps podcast. He was previously a DevOps Engineer for companies such as Taos Consulting, Peak Hosting, Oak Ridge National Laboratory, and many more. Mike is originally from Knoxville, TN (Go Vols!) and currently resides in Portland, OR.Links Referenced: @Mike_Julian: https://twitter.com/Mike_Julian mikejulian.com: https://mikejulian.com duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at AWS AppConfig. Engineers love to solve, and occasionally create, problems. But not when it's an on-call fire-drill at 4 in the morning. Software problems should drive innovation and collaboration, NOT stress, and sleeplessness, and threats of violence. That's why so many developers are realizing the value of AWS AppConfig Feature Flags. Feature Flags let developers push code to production, but hide that that feature from customers so that the developers can release their feature when it's ready. This practice allows for safe, fast, and convenient software development. You can seamlessly incorporate AppConfig Feature Flags into your AWS or cloud environment and ship your Features with excitement, not trepidation and fear. To get started, go to snark.cloud/appconfig. That's snark.cloud/appconfig.Corey: Forget everything you know about SSH and try Tailscale. Imagine if you didn't need to manage PKI or rotate SSH keys every time someone leaves. That'd be pretty sweet, wouldn't it? With Tailscale SSH, you can do exactly that. Tailscale gives each server and user device a node key to connect to its VPN, and it uses the same node key to authorize and authenticate SSH.Basically you're SSHing the same way you manage access to your app. What's the benefit here? Built in key rotation, permissions is code, connectivity between any two devices, reduce latency and there's a lot more, but there's a time limit here. You can also ask users to reauthenticate for that extra bit of security. Sounds expensive?Nope, I wish it were. Tailscale is completely free for personal use on up to 20 devices. To learn more, visit snark.cloud/tailscale. Again, that's snark.cloud/tailscaleCorey: Welcome to Screaming in the Cloud. I'm Cloud Economist Corey Quinn, and my guest is a returning guest on this show, my business partner and CEO of The Duckbill Group, Mike Julian. Mike, thanks for making the time.Mike: Lucky number three, I believe?Corey: Something like that, but numbers are hard. I have databases for that of varying quality and appropriateness for the task, but it works out. Anything's a database. If you're brave enough.Mike: With you inviting me this many times, I'm starting to think you'd like me or something.Corey: I know, I know. So, let's talk about something that is going to put that rumor to rest.Mike: [laugh].Corey: Clearly, you have made some poor choices in the course of your career, like being my business partner being the obvious one. But what's really in a dead heat for which is the worst decision is you've written a book previously. And now you are starting the process of writing another book because, I don't know, we don't keep you busy enough or something. What are you doing?Mike: Making very bad decisions. When I finished writing Practical Monitoring—O'Reilly, and by the way, you should go buy a copy if interested in monitoring—I finished the book and said, “Wow, that was awful. I'm never doing it again.” And about a month later, I started thinking of new books to write. So, that was 2017, and Corey and I started Duckbill and kind of stopped thinking about writing books because small companies are basically small children. But now I'm going to write a book about consulting.Corey: Oh, thank God. I thought you're going to go down the observability path a second time.Mike: You know, I'm actually dreading the day that O'Reilly asks me to do a second edition because I don't really want to.Corey: Yeah. Effectively turn it into an entire story where the only monitoring tool you really need is the AWS bill. That'll go well.Mike: [laugh]. Yeah. So yeah, like, basically, I've been doing consulting for such a long time, and most of my career is consulting in some form or fashion, and I head up all the consulting at Duckbill. I've learned a lot about consulting. And I've found that people have a lot of questions about consulting, particularly at the higher-end levels. Once you start getting into advisory sort of stuff, there's not a lot of great information out there aimed at engineering.Corey: There's a bunch of different views on what consulting is. You have independent contractors billing by the hour as staff replacement who call what they do consulting; you have the big consultancies, like Bain or BCG; you've got what we do in an advisory sense, and of course, you have a bunch of MBA new grads going to a lot of the big consultancies who are going to see a book on consulting and think that it's potentially for them. I don't know that you necessarily have a lot of advice for the new grad type, so who is this for? What is your target customer for this book?Mike: If you're interested in joining McKinsey out of college, I don't have a lot to add; I don't have a lot to tell you. The reason for that is kind of twofold. One is that shops like McKinsey and Deloitte and Accenture and BCG and Bain, all those, are playing very different games than what most of us think about when we think consulting. Their entire model revolves around running a process. And it's the same process for every client they work with. But, like, you're buying them because of their process.And that process is nothing new or novel. You don't go to those firms because you want the best advice possible. You go to those firms because it's the most defensible advice. It's sort of those things like, “No one gets fired for buying Cisco,” no one got fired for buying IBM, like, that sort of thing, it's a very defensible choice. But you're not going to get great results from it.But because of that, their entire model revolves around throwing dozens, in some cases, hundreds of new grads at a problem and saying, “Run this process. Have fun. Let us know if you need help.” That's not consulting I have any experience with. It's honestly not consulting that most of us want to do.Most of that is staffed by MBAs and accountants. When I think consulting, I think about specialized advice and providing that specialized advice to people. And I wager that most of us think about that in the same way, too. In some cases, it might just be, “I'm going to write code for you as a freelancer,” or I'm just going to tell you like, “Hey, put the nail in here instead of over here because it's going to be better for you.” Like, paying for advice is good.But with that, I also have a… one of the first things I say in the beginning of the book, which [laugh] I've already started writing because I'm a glutton for punishment, is I don't think junior people should be consultants. I actually think it's really bad idea because to be a consultant, you have to have expertise in some area, and junior staff don't. They haven't been in their careers long enough to develop that yet. So, they're just going to flounder. So, my advice is generally aimed at people that have been in their careers for quite some time, generally, people that are 10, 15, 20 years into their career, looking to do something.Corey: One of the problems that we see when whenever we talk about these things on Twitter is that we get an awful lot of people telling us that we're wrong, that it can't be made to work, et cetera, et cetera. But following this model, I've been independent for—well, I was independent and then we became The Duckbill Group; add them together because figuring out exactly where that divide happened is always a mental leap for me, but it's been six years at this point. We've definitely proven our ability to not go out of business every month. It's kind of amazing. Without even an exception case of, “That one time.”Mike: [laugh]. Yeah, we are living proof that it does work, but you don't really have to take just our word for it because there are a lot of other firms that exist entirely on an advisory-only, high-expertise model. And it works out really well. We've worked with several of them, so it does work; it just isn't very common inside of tech and particularly inside of engineering.Corey: So, one of the things that I find is what differentiates an expert from an enthusiastic amateur is, among other things, the number of mistakes that they've made. So, I guess a different way of asking this is what qualifies you to write this book, but instead, I'm going to frame it in a very negative way. What have you screwed up on that puts you in a position of, “Ah, I'm going to write a book so that someone else can make better choices.”Mike: One of my favorite stories to tell—and Corey, I actually think you might not have heard this story before—Corey: That seems unlikely, but give it a shot.Mike: Yeah. So, early in my career, I was working for a consulting firm that did ERP implementations. We worked with mainly large, old-school manufacturing firms. So, my job there was to do the engineering side of the implementation. So, a lot of rack-and-stack, a lot of Windows Server configuration, a lot of pulling cables, that sort of thing. So, I thought I was pretty good at this. I quickly learned that I was actually not nearly as good as I thought I was.Corey: A common affliction among many different people.Mike: A common affliction. But I did not realize that until this one particular incident. So, me and my boss are both on site at this large manufacturing facility, and the CFO pulls my boss aside and I can hear them talking and, like, she's pretty upset. She points at me and says, “I never want this asshole in my office ever again.” So, he and I have a long drive back to our office, like an hour and a half.And we had a long chat about what that meant for me. I was not there for very long after that, as you might imagine, but the thing is, I still have no idea to this day what I did to upset her. I know that she was pissed and he knows that she was pissed. And he never told me exactly what it was, only that's you take care of your client. And the client believes that I screwed up so massively that she wanted me fired.Him not wanting to argue—he didn't; he just kind of went with it—and put me on other clients. But as a result of that, it really got me thinking that I screwed something up so badly to make this person hate me so much and I still have no idea what it was that I did. Which tells me that even at the time, I did not understand what was going on around me. I did not understand how to manage clients well, and to really take care of them. That was probably the first really massive mistake that I've made my career—or, like, the first time I came to the realization that there's a whole lot I don't know and it's really costing me.Corey: From where I sit, there have been a number of things that we have done as we've built our consultancy, and I'm curious—you know, let's get this even more personal—in the past, well, we'll call it four years that we have been The Duckbill Group—which I think is right—what have we gotten right and what have we gotten wrong? You are the expert; you're writing a book on this for God's sake.Mike: So, what I think we've gotten right is one of my core beliefs is never bill hourly. Shout out to Jonathan Stark. He wrote I really good book that is a much better explanation of that than I've ever been able to come up with. But I've always had the belief that billing hourly is just a bad idea, so we've never done that and that's worked out really well for us. We've turned down work because that's the model they wanted and it's like, “Sorry, that's not what we do. You're going to have to go work for someone else—or hire someone else.”Other things that I think we've gotten right is a focus on staying on the advisory side and not doing any implementation. That's allowed us to get really good at what we do very quickly because we don't get mired in long-term implementation detail-level projects. So, that's been great. Where we went a little wrong, I think—or what we have gotten wrong, lessons that we've learned. I had this idea that we could build out a junior and mid-level staff and have them overseen by very senior people.And, as it turns out, that didn't work for us, entirely because it didn't work for me. That was really my failure. I went from being an IC to being the leader of a company in one single step. I've never been a manager before Duckbill. So, that particular mistake was really about my lack of abilities in being a good manager and being a good leader.So, building that out, that did not work for us because it didn't work for me and I didn't know how to do it. So, I made way too many mistakes that were kind of amateur-level stuff in terms of management. So, that didn't work. And the other major mistake that I think we've made is not putting enough effort into marketing. So, we get most of our leads by inbound or referral, as is common with boutique consulting firms, but a lot of the income that we get comes through Last Week in AWS, which is really awesome.But we don't put a whole lot of effort into content or any marketing stuff related to the thing that we do, like cost management. I think a lot of that is just that we don't really know how, aside from just creating content and publishing it. We don't really understand how to market ourselves very well on that side of things. I think that's a mistake we've made.Corey: It's an effective strategy against what's a very complicated problem because unlike most things, if—let's go back to your old life—if we have an observability problem, we will talk about that very publicly on Twitter and people will come over and get—“Hey, hey, have you tried to buy my company's product?” Or they'll offer consulting services, or they'll point us in the right direction, all of which is sometimes appreciated. Whereas when you have a big AWS bill, you generally don't talk about it in public, especially if you're a serious company because that's going to, uh, I think the phrase is, “Shake investor confidence,” when you're actually live tweeting slash shitposting about your own AWS bill. And our initial thesis was therefore, since we can't wind up reaching out to these people when they're having the pain because there's no external indication of it, instead what we have to do is be loud enough and notable in this space, where they find us where it shouldn't take more than them asking one or two of their friends before they get pointed to us. What's always fun as the stories we hear is, “Okay, so I asked some other people because I wanted a second opinion, and they told us to go to you, too.” Word of mouth is where our customers come from. But how do you bootstrap that? I don't know. I'm lucky that I got it right the first time.Mike: Yeah, and as I mentioned a minute ago, that a lot of that really comes through your content, which is not really cost management-related. It's much more AWS broad. We don't put out a lot of cost management specific content. And honestly, I think that's to our detriment. We should and we absolutely can. We just haven't. I think that's one of the really big things that we've missed on doing.Corey: There's an argument that the people who come to us do not spend their entire day thinking about AWS bills. I mean, I can't imagine what that would be like, but they don't for whatever reason; they're trying to do something ridiculous, like you know, run a profitable company. So, getting in front of them when they're not thinking about the bills means, on some level, that they're going to reach out to us when the bill strikes. At least that's been my operating theory.Mike: Yeah, I mean, this really just comes down to content strategy and broader marketing strategy. Because one of the things you have to think about with marketing is how do you meet a customer at the time that they have the problem that you solve? And what most marketing people talk about here is what's called the triggering event. Something causes someone to take an action. What is that something? Who is that someone, and what is that action?And for us, one of the things that we thought early on is that well, the bill comes out the first week of the month, every month, so people are going to opened the bill freak out, and a big influx of leads are going to come our way and that's going to happen every single month. The reality is that never happened. That turns out was not a triggering event for anyone.Corey: And early on, when we didn't have that many leads coming in, it was a statistical aberration that I thought I saw, like, “Oh, out of the three leads this month, two of them showed up in the same day. Clearly, it's an AWS billing day thing.” No. It turns out that every company's internal cadence is radically different.Mike: Right. And I wish I could say that we have found what our triggering events are, but I actually don't think we have. We know who the people are and we know what they reach out for, but we haven't really uncovered that triggering event. And it could also be there, there isn't a one. Or at least, if there is one, it's not one that we could see externally, which is kind of fine.Corey: Well, for the half of our consulting that does contract negotiation for large-scale commitments with AWS, it comes up for renewal or the initial discount contract gets offered, those are very clear triggering events but the challenge is that we don't—Mike: You can't see them externally.Corey: —really see that from the outside. Yeah.Mike: Right. And this is one of those things where there are triggering events for basically everything and it's probably going to be pretty consistent once you get down to specific services. Like we provide cost optimization services and contract negotiation services. I'm willing to bet that I can predict exactly what the trigger events for both of those will be pretty well. The problem is, you can never see those externally, which is kind of fine.Ideally, you would be able to see it externally, but you can't, so we roll with it, which means our entire strategy has revolved around always being top-of-mind because at the time where it happens, we're already there. And that's a much more difficult strategy to employ, but it does work.Corey: All it takes is time and being really lucky and being really prolific, and, and, and. It's one of those things where if I were to set out to replicate it, I don't even know how I'd go about doing it.Mike: People have been asking me. They say, “I want to create The Duckbill Group for X. What do I do?” And I say, “First step, get yourself a Corey Quinn.” And they're like, “Well, I can't do that. There's only one.” I'm like, “Yep. Sucks to be you.” [laugh].Corey: Yeah, we called the Jerk Store. They're running out of him. Yeah, it's a problem. And I don't think the world needs a whole lot more of my type of humor, to be honest, because the failure mode that I have experienced brutally and firsthand is not that people don't find me funny; it's that it really hurts people's feelings. I have put significant effort into correcting those mistakes and not repeating them, but it sucks every time I get it wrong.Mike: Yeah.Corey: Another question I have for you around the book targeting, are you aiming this at individual independent consultants or are you looking to advise people who are building agencies?Mike: Explicitly not the latter. My framing around this is that there are a number of people who are doing consulting right now and they've kind of fell into it. Often, they'll leave one job and do a little consulting while they're waiting on their next thing. And in some cases, that might be a month or two. In some cases, it might go on years, but that whole time, they're just like, “Oh, yeah, I'm doing consulting in between things.”But at some point, some of those think, “You know what? I want this to be my thing. I don't want there to be a next thing. This is my thing. So therefore, how do I get serious about doing consulting? How do I get serious about being a consultant?”And that's where I think I can add a lot of value because casually consulting of, like, taking whatever work just kind of falls your way is interesting for a while, but once you get serious about it, and you have to start thinking, well, how do I actually deliver engagements? How do I do that consistently? How do I do it repeatedly? How to do it profitably? How do I price my stuff? How do I package it? How do I attract the leads that I want? How do I work with the customers I want?And turning that whole thing from a casual, “Yeah, whatever,” into, “This is my business,” is a very different way of thinking. And most people don't think that way because they didn't really set out to build a business. They set out to just pass time and earn a little bit of money before they went off to the next job. So, the framing that I have here is that I'm aiming to help people that are wanting to get serious about doing consulting. But they generally have experience doing it already.Corey: Managing shards. Maintenance windows. Overprovisioning. ElastiCache bills. I know, I know. It's a spooky season and you're already shaking. It's time for caching to be simpler. Momento Serverless Cache lets you forget the backend to focus on good code and great user experiences. With true autoscaling and a pay-per-use pricing model, it makes caching easy. No matter your cloud provider, get going for free at gomemento.co/screaming That's GO M-O-M-E-N-T-O dot co slash screamingCorey: We went from effectively being the two of us on the consulting delivery side, two scaling up to, I believe, at one point we were six of us, and now we have scaled back down to largely the two of us, aided by very specific external folk, when it makes sense.Mike: And don't forget April.Corey: And of course. I'm talking delivery.Mike: [laugh].Corey: There's a reason I—Mike: Delivery. Yes.Corey: —prefaced it that way. There's a lot of support structure here, let's not get ourselves, and they make this entire place work. But why did we scale up? And then why did we scale down? Because I don't believe we've ever really talked about that publicly.Mike: No, not publicly. In fact, most people probably don't even notice that it happened. We got pretty big for—I mean, not big. So, we hit, I think, six full-time people at one point. And that was quite a bit.Corey: On the delivery side. Let's be clear.Mike: Yeah. No, I think actually with support structure, too. Like, if you add in everyone that we had with the sales and marketing as well, we were like 11 people. And that was a pretty sizable company. But then in July this year, it kind of hit a point where I found that I just wasn't enjoying my job anymore.And I looked around and noticed that a lot of other people was kind of feeling the same way, is just things had gotten harder. And the business wasn't suffering at all, it was just everything felt more difficult. And I finally realized that, for me personally at least, I started Duckbill because I love working with clients, I love doing consulting. And what I have found is that as the company grew larger and larger, I spent most of my time keeping the trains running and taking care of the staff. Which is exactly what I should be doing when we're that size, like, that is my job at that size, but I didn't actually enjoy it.I went into management as, like, this job going from having never done it before. So, I didn't have anything to compare it to. I didn't know if I would like it or not. And once I got here, I realized I actually don't. And I spent a lot of efforts to get better at it and I think I did. I've been working with a leadership coach for years now.But it finally came to a point where I just realized that I wasn't actually enjoying it anymore. I wasn't enjoying the job that I had created. And I think that really panned out to you as well. So, we decided, we had kind of an opportune time where one of our team decided that they were also wanting to go back to do independent consulting. I'm like, “Well, this is actually pretty good time. Why don't we just start scaling things back?” And like, maybe we'll scale it up again in the future; maybe we won't. But like, let's just buy ourselves some breathing room.Corey: One of the things that I think we didn't spend quite enough time really asking ourselves was what kind of place do we want to work at. Because we've explicitly stated that you and I both view this as the last job either of us is ever going to have, which means that we're not trying to do the get big quickly to get acquired, or we want to raise a whole bunch of other people's money to scale massively. Those aren't things either of us enjoy. And it turns out that handling the challenges of a business with as many people working here as we had wasn't what either one of us really wanted to do.Mike: Yeah. You know what—[laugh] it's funny because a lot of our advisors kept asking the same thing. Like, “So, what kind of company do you want?” And like, we had some pretty good answers for that, in that we didn't want to build a VC-backed company, we didn't ever want to be hyperscale. But there's a wide gulf of things between two-person company and hyperscale and we didn't really think too much about that.In fact, being a ten-person company is very different than being a three-person company, and we didn't really think about that either. We should have really put a lot more thought into that of what does it mean to be a ten-person company, and is that what we want? Or is three, four, or five-person more our style? But then again, I don't know that we could have predicted that as a concern had we not tried it first.Corey: Yeah, that was very much something that, for better or worse, we pay advisors for their advice—that's kind of definitionally how it works—and then we ignored it, on some level, though we thought we were doing something different at the time because there's some lessons you've just got to learn by making the mistake yourself.Mike: Yeah, we definitely made a few of those. [laugh].Corey: And it's been an interesting ride and I've got zero problem with how things have shaken out. I like what we do quite a bit. And honestly, the biggest fear I've got going forward is that my jackass business partner is about to distract the hell out of himself by writing a book, which is never as easy as even the most pessimistic estimates would be. So, that's going to be awesome and fun.Mike: Yeah, just wait until you see the dedication page.Corey: Yeah, I wasn't mentioned at all in the last book that you wrote, which I found personally offensive. So, if I'm not mentioned this time, you're fired.Mike: Oh, no, you are. It's just I'm also adding an anti-dedication page, which just has a photo of you.Corey: Oh, wonderful, wonderful. This is going to be one of those stories of the good consultant and the bad consultant, and I'm going to be the Goofus to your Gallant, aren't I?Mike: [laugh]. Yes, yes. You are.Corey: “Goofus wants to bill by the hour.”Mike: It's going to have a page of, like, “Here's this [unintelligible 00:25:05] book is dedicated to. Here's my acknowledgments. And [BLEEP] this guy.”Corey: I love it. I absolutely love it. I think that there is definitely a bright future for telling other people how to consult properly. May just suggest as a subtitle for the book is Consulting—subtitle—You Have Problems and Money. We'll Take Both.Mike: [laugh]. Yeah. My working title for this is Practical Consulting, but only because my previous book was Practical Monitoring. Pretty sure O'Reilly would have a fit if I did that. I actually have no idea what I'm going to call the book, still.Corey: Naming things is super hard. I would suggest asking people at AWS who name services and then doing the exact opposite of whatever they suggest. Like, take their list of recommendations and sort by reverse order and that'll get you started.Mike: Yeah. [laugh].Corey: I want to thank you for giving us an update on what you're working on and why you have less hair every time I see you because you're mostly ripping it out due to self-inflicted pain. If people want to follow your adventures, where's the best place to keep updated on this ridiculous, ridiculous nonsense that I cannot talk you out of?Mike: Two places. You can follow me on Twitter, @Mike_Julian, or you can sign up for the newsletter on my site at mikejulian.com where I'll be posting all the updates.Corey: Excellent. And I look forward to skewering the living hell out of them.Mike: I look forward to ignoring them.Corey: Thank you, Mike. It is always a pleasure.Mike: Thank you, Corey.Corey: Mike Julian, CEO at The Duckbill Group, and my unwilling best friend. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, annoying comment in which you tell us exactly what our problem is, and then charge us a fixed fee to fix that problem.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Episode 149:     Welcome to PI-Perspectives. On July 4th, our country once again faced the horrors of an active shooter situation in Chicago, Illinois. At least 6 people were killed and over 30 were injured. Matt reached out to Mike Julian, the founder of the ALIVE program that teaches people how to survive these situations. Mike actually is doing instructor training on July 21st and 22nd. The course is live in MAryland or via Zoom. If you have the desire to become an instructor and teach people as a paid consultant, you should consider registering. The information is in the show notes. Please welcome, Mike Julian and your host, private Investigator, Matt Spaier  Links:      Matt's email: MatthewS@Satellitepi.com   Linkedin: Matthew Spaier       www.investigators-toolbox.com   PI-Perspectives Youtube link:  https://www.youtube.com/channel/UCYB3MaUg8k5w3k7UuvT6s0g Mike Email: mjulian@investigations-nbi.com A.L.I.V.E. website https://activeshootersurvivaltraining.com/ Instructor course https://activeshootersurvivaltraining.com/become-a-certified-alive-instructor/ Online training https://activeshootersurvivaltraining.com/online-active-shooter-training-courses/ Book https://activeshootersurvivaltraining.com/product/10-minutes-live-surviving-an-active-shooter-using-alive-book/ Sponsors:     https://piinstitute.com/ https://apps.crosstrax.co/signup/index/refcd/LY3R7VUW69 https://www.conflictinternational.com/ https://www.nciss.org/ https://www.skopenow.com/

About AlexAlex holds a Ph.D. in Computer Science and Engineering from UC San Diego, and has spent over a decade building high-performance, robust data management and processing systems. As an early member of a couple fast-growing startups, he's had the opportunity to wear a lot of different hats, serving at various times as an individual contributor, tech lead, manager, and executive. Prior to joining the Duckbill Group, Alex spent a few years as a freelance data engineering consultant, helping his clients build, manage and maintain their data infrastructure. He lives in Los Angeles, CA.Links: Twitter: https://twitter.com/alexras/ Personal page: https://alexras.info Old Consulting website with blog: https://bitsondisk.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: The company 0x4447 builds products to increase standardization and security in AWS organizations. They do this with automated pipelines that use well-structured projects to create secure, easy-to-maintain and fail-tolerant solutions, one of which is their VPN product built on top of the popular OpenVPN project which has no license restrictions; you are only limited by the network card in the instance. To learn more visit: snark.cloud/deployandgoCorey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm the chief cloud economist at The Duckbill Group, which people are generally aware of. Today, I'm joined by our most recent principal cloud economist, Alex Rasmussen. Alex, thank you for joining me today, it is a pleasure to talk to you, as if we aren't talking to each other constantly, now that you work here.Alex: Thanks, Corey. It's great being here.Corey: So, I followed a more, I'd say traditional path for a cloud economist, but given that I basically had to invent the job myself, the more common path because imagine that you start building a role from scratch and the people you wind up looking for initially look a lot like you. And that is grumpy sysadmin, historically, turned into something, kind of begrudgingly, that looks like an SRE, which I still maintain are the same thing, but it is imperative people not email me about that. Yes, I know, you work at Google. But instead, what I found during my tenure as a sysadmin, is that I was working with certain things an awful lot, like web servers, and other things almost never, like databases and data warehouses. Because if you screw up a web server, we all have a good laugh, the site's down for a couple of minutes, life goes on, you have a shame trophy on your desk if that's your corporate culture, things continue.Mess up the data severely enough, and you don't have a company anymore. So, I was always told to keep my aura away from the expensive spendy things that power a company. You are sort of the first of a cloud economist subtype that doesn't resemble that. Before you worked here, you were effectively an independent consultant working on data engineering. Before that, you had a couple of jobs, but you had gotten a PhD in computer science, which means, first, you are probably one of the people in this world most qualified to pass some crappy job interview of solving a sorting algorithm on a whiteboard, but how did you get here from where you were?Alex: Great question. So, I like to joke that I kind of went to school until somebody told me that I had to stop. And I took that and went and started—or didn't start, but I was an early engineer at a startup and then was an executive at another early-stage one, and did a little bit of everything. And went freelance, did that for a couple of years, and worked with all kinds of different companies—vast majority of those being startups—helping them with data infrastructure problems. I've done a little bit of everything throughout my career.I've been, you know, IC, manager, manager, manager, IT guy, everything in between. I think on the data side of things, it just sort of happened, to be honest with you, it kind of started with the stuff that I did for my dissertation and parlayed that into a job back when the big data wave was starting to kind of truly crest. And I've been working on data infrastructure, basically my entire career. So, it wasn't necessarily something that was intentional. I've just been kind of taking the opportunity that makes the most sense for me it kind of every juncture. And my career path has been a little bit strange, both by academic and industrial standards. But I like where I'm at and I gained something really valuable from each of those experiences. So.Corey: It's been an interesting area of I won't say weakness here, but it's definitely been a bit of a challenge when we look at an AWS environment and even talking about a typical AWS customer without thinking of any of them in particular, I can already tell you a few things are likely to be true. For example, the number one most expensive line item in their bill is going to be EC2, and compute is the thing that powers it. Now, maybe that is they're running a bunch of instances the old-fashioned way. Maybe they're running Kubernetes but that's how it shows up. There's a lot of things that could be, and we look at what rounds that out.Now, the next item down should almost certainly not be data transfer and if so we should have a conversation, but data in one form or another is very often going to be number two. And that can mean a bunch of different things, historically. It could mean, “Oh, you have a whole bunch of stuff in S3. Let's talk about access patterns. Let's talk about lifecycle policies. Let's talk about making sure the really important stuff is backed up somewhere. Maybe you want to spend more on that particular aspect of it.”If it's on EBS volumes, that's interesting and definitely worth looking into and trying to understand the context of what's going on. Periodically we'll see a whole bunch of additional charges that speak to some of that EC2 charge in the form of EMR, AWS's Elastic MapReduce, which charges a per-hour instance charge, but also charges you for the instances that are running under the hood and under the EC2 line item. So, there's a lot of data lifecycle stuff, there's a lot of data ecosystem stories, that historically we've consulted out with experts in that particular space. And that's great, but we were starting to have to drag those people in on more and more engagements as we saw them. And we realized that was really something we had to build out as a core competency for ourselves.And we started out not intending to hire for someone with that specialty, but the more we talked to you, the more it became clear that this was a very real and very growing need that we and our customers have. How closely it is what you're doing now as far as AWS bill analysis and data pattern deep-dive align with what you were doing as a freelance consultant in the space?Alex: A lot more than you might expect. You know, I think that increasingly, what you're seeing now is that a company's core differentiator is its data, right, how much of it they have, what they do with it. And so, you know, to your point, I think when you look at any company's cloud spend, it's going to be pretty heavy on the data side in terms of, like, where have you put it? What are you doing to process it? Where is it going once it's been processed? And then how is that—Corey: And data transfer is a very important first word in that two-word sequence.Alex: Oh, sure is. And so I think that, like, in a lot of ways, the way that a customer's cloud architecture looks and the way that their bill looks kind of as a consequence of that is kind of a reification in a way of the way that the data flows from one place to another and what's done with it at each step along the way. I think what complicates this is that companies that have been around for a little while have lived through this kind of very amorphous, kind of, polyglot way that we're approaching data. You know, back when I was first getting started in the big data days, it was MapReduce, MapReduce, MapReduce, right? And we quickly [crosstalk 00:07:29]—Corey: Oh, yes. The MapReduce white paper out of Google, a beautiful April Fool's Day prank that the folks at Yahoo fell for hook, line, and sinker. They wrote Hadoop, and now we're all stuck with that pattern. Great gag, they really should have clarified they were kidding. Here we are.Alex: Exactly. So—Corey: I mostly kid.Alex: No, for sure. But I think especially when it comes to data, we tend to over-index on what the large companies do and then quickly realize that we've made a mistake and correct backwards, right? So, there was this big push toward MapReduce for everything until people realize that it was just a pain in the neck to operate and to build. And so then we moved into Spark, so kind of up-leveled a little bit. And then there was this kind of explosion of NoSQL and NewSQL databases that hit the market.And MongoDB inexplicably won that war and now we're kind of in this world where everything is cloud data warehouse, right? And now we're trying to wrestle with, like, is it actually a good idea to put everything in one warehouse and have SQL be the lingua franca on top of it? But it's all changing so rapidly. And when you come into a customer that's been around for 10 or 15 years, and has, you know, been in the cloud for a substantial—Corey: Yeah, one of those ancient customers. That is—Alex: I know, right?Corey: —basically old enough to almost get a driver's license? Oh, yeah.Alex: Right. It's one of those things where it's like, “Ah, yes, in startup years, you're, like, a hundred years old,” right? But still, you know, I think you see this, kind of—I wouldn't call it a graveyard of failed experiments, right, but it's a collection of, like, “Well, we tried this, and it kind of worked and we're keeping it around because the cost of moving this stuff around—the kind of data gravity, so to speak—is high enough that we're not going to bother transitioning it over.” But then you get into this situation where you have to bend over backwards to integrate anything with anything else. And we're still kind of in the early days of fixing that.Corey: And the AWS bill pattern that we see all the time across the board of those experiments were not successful and do not need to exist, but there's no context into that. The person that set them up left five years ago, the jobs are still running on time. What's happening with them? Well, we could stop them and see who screams, but very often, that's not the right answer either.Alex: And I think there's also something to note there, too, which is like, getting rid of data is very scary, right? I mean, if you resize a Kubernetes cluster from 15 nodes to 10, nobody's going to look at you sideways. But if you go, “Hey, we're just going to drop these tables.” The immediate reaction that you get, particularly from your data science team more often than not is, “Oh, God, what if we need that?” And so the conversation never really happens, and that causes this kind of snowball of data debt that persists in some cases for many, many years.Corey: Yeah, in some cases, what I found has been successful on those big unknown questions is don't delete the data, but restrict access to it for a few weeks and see what happens. Look into it a bit and make sure that it's not like, “Oh, cool. We just did for a month, and now we don't need that data. Let's get rid of it.” And then another month goes by it's like, “So, time to report quarterly earnings. Where's the data?”Oh, dear, that's not going to go well, for anyone. And understanding what's happening, the idea of cloning a petabyte of data so you can run an experiment on it. And okay, turns out the experiment wasn't needed. Do we still need to keep all of that?Alex: Yeah.Corey: The underlying platform advancements have been helpful toward this as well, a petabyte of data now in Glacier Deep Archive cost the princely sum of a thousand bucks a month, which is pretty close to the idea of why would I ever delete data ever again? I can get it back within a day if I need it, so let's just put it there instead.Alex: Right. You know, funny story. When I was in graduate school, we were dealing with, you know, 100 terabyte datasets on the regular that we had to generate every time because we only had 200 terabytes of raw storage. [laugh]. And this was before cloud was yet mature enough that we could get the kind of performance numbers that we wanted off of it.And we would end up having to delete the input data to make room for the output data. [laugh]. And thankfully, we don't need to do that anymore. But there are a lot of, kind of, anti-patterns that arise from that too, right? If data is easy to keep around forever, it stays around forever.And if it's easy to, let's say, run a SQL command against your Snowflake instance that scans 20 terabytes of data, you're just going to do it, and the exposure of that to you is so minimal that you can end up causing a whole bunch of problems for yourself by the fact that you don't have to deal with stuff at that low-level of abstraction anymore.Corey: It's always fun watching how this stuff manifests—because I'm dipping a toe into it from time to time—the easy, naive answer that we could give every customer but we don't is, “Huh. So, you have a whole bunch of EMR stuff? Well, you know, if you migrate that into something else, you'll save a whole bunch of money on that.” With no regard for the 500 jobs that run against that EMR cluster on a consistent basis that form is a key part of business process. “Yeah, if you could just do the entire flow of how data is operated with throughout your entire business that would be swell because you can save tens of thousands of dollars a month on that.” Yeah, how about we don't suggest things that are just absolute buffoonery.Alex: Well, and it's like, you know, you hit on a good point. Like, one of my least favorite words in the English language is the word ‘just.' And you know, I spent a few years as a freelance data consultant, and you know, a lot of what I would hear sometimes from customers is, “Well, why don't we ‘just' deprecate X?”Corey: “Why don't we just—” “I'm going to stop you there because there is no ‘just.'”Alex: Exactly.Corey: There's always context that we cannot have as outsiders.Alex: Precisely. Precisely. And digging into that really is—it's the fun part of the job, but it's also the hard part of the job.Corey: Before we created The Duckbill Group, which was really when I took Mike Julian on as business partner and CEO and formed the entity, I had something in common with you; I was freelancing for a couple of years beforehand. Now, I know why I wound up deciding, all right, we're going to turn this into a company, but what was it that I guess made you decide to, you know, freelancing is all well and good, but it's time to get something that looks a lot more like a quote-unquote, “Traditional job.”Alex: So, I think, on one level, I went freelance because I wasn't exactly sure what I wanted to do next. And I knew what I was good at. I knew what I had a lot of experience at, and I thought, “Well, I can just go out and kind of find a bunch of people that are willing to hire me to do what I'm good at doing, and then maybe eventually I'll find one of them that I like enough that I'll go and work for them. Or maybe I'll come up with some kind of a business model that I can repeat enough times that I don't have to worry that I wake up tomorrow and all of my clients are gone and then I have to go live in a van down by the river.”And I think when I heard about the opening at The Duckbill Group, I had been thinking for a little while about well, this has been going fine for a long time, but effectively what I've been doing is I've been you know, a staff-level data engineer for hire. And do I want to do something more than that, you know? Do I want to do something more comp—perhaps more sophisticated or more complex than that? And I rapidly came to the conclusion that in order to do that, I would have to have sales and marketing, and I would have to, you know, spend a lot of my time bringing in business. And that's just not something that I have really any experience in or I'm any good at.And, you know, I also recognize that, you know, I'm a relatively small fish in a relatively large pond, and if I wanted to get the kind of like, large scale people, the like the big, you know, Fortune 1000 company kind of customers, they may not pay attention to somebody like me. And so I think that ultimately, what I saw with The Duckbill Group was, number one, a group of people that were strongly aligned to the way that I wanted to keep doing this sort of work, right? Cultural alignment was really strong, good people, but also, you know, you folks have a thing that you figured out, and that puts you 10 to 15 steps ahead of where I was. And I was kind of staring down the barrel that, I'm like, am I going to have to take six months not doing client work so that I can figure out how to make this business sustain? And, you know, I think that ultimately, like, I just looked at it, and I said, this just makes sense to me, like, as a next step. And so here we all are.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: It's always fun seeing how people perceive what we've done from the outside. Like, “Oh, yeah, you just stumbled right onto the thing that works, and you've just been going, like, gangbusters ever since.” Then you come aboard, it's like, “Here, look at this pile of things that didn't pan out over here.” And it's, you get to see how the sausage is made in a way that we talk about from time to time externally, but surprisingly, most of our marketing efforts aren't really focused on, “And here's this other time we screwed up as well.” And we're honest about it, but it's not sort of the thing that we promote as the core message of what we do and who we are.A question I like to ask people during job interviews, and I definitely asked you this, and I'll ask you now, which is going to probably throw some folks for a loop because who talks to their current employees like this? But what's next for you? When it comes time for you to leave the Duckbill Group, what do you want to do after this job?Alex: That's a great question. So, I mean, as we've mentioned before, you know, my career trajectory has been very weird and circuitous. And, you know, I would be lying to you if I said that I had absolute certainty about what the rest of that looks like. I've learned a few things about myself in the course of my career, such as it is. In my kind of warm, gooey center, I build stuff. Like, that is what gives me joy, it is what makes me excited to wake up in the morning.I love looking at big, complicated things, breaking them down into pieces, and figuring out how to make the pieces work in a way that makes sense. And, you know, I've spent a long time in the data ecosystem. I don't know, necessarily, if that's something that I'm going to do forever. I'm not necessarily pigeonholing myself into that part of the space just yet, but as long as I get to kind of wake up in the morning, and say, “I'm going to go and build things and it's not going to actively make the world any worse,” I'm happy with that. And so that's really—you know, might go back to freelancing, might go and join another group, another company, big small, who knows. I'm kind of leaving that up to the winds of destiny, so to speak.Corey: One thing that I have found incredi—sorry. Let me just address that first. Like that—Alex: Sure.Corey: —is the right way to think about it. My belief has always been that you don't necessarily have, like, the ten-year plan, or the five-year plan or whatever it is because that's where you're going to go so much as it gives you direction and forces you to keep moving so you don't wind up sitting in the same place for five years with one year of experience repeated five times. It helps you remember the bigger picture. Because I've always despised this fiction that we see in job interviews where average tenure in our industry is 18 to 36 months, give or take, but somehow during the interviews, we all talk like this is now your forever job, and after 25 years, you'll retire. And yeah, let's be a little more realistic than that.My question is always what is next and how can we align in a way that helps you get to what's coming? That's the purpose behind the question, and that's—the only way to make that not just a drippingly insincere question is to mean it and to continue to focus on it from time to time of, great. What are you learning what's next? Now, at the time of this recording, you've been here, I believe three weeks if I'm not mistaken?Alex: I've—this is week two for me at time of recording.Corey: Excellent. Yes, my grasp of time is sort of hazy at the best of times. I have a—I do a lot of things.Alex: For sure.Corey: But yeah, it has been an eye-opening experience for me, not because, “Oh, wow, we have an employee.” Yeah, we've done that a few times before. But rather because of your background, you are asking different questions than we typically get during onboarding. I had a blog post go out recently—or will be by the time this airs—about a question that you asked about, “Wow, onboarding into our internal account structure for AWS is way more polished than I've ever seen it before. Is that something you built in-house? What is that?”And great. Oh, terrific, I'd forgotten that this is kind of a novel thing. No. What we're using is AWS's SSO offering, which is such a well-built, polished product that I can only assume that it's under NDA because Amazonians don't talk about it ever. But it's great.It has a couple of annoyances, but beyond that, it's something that I'm a big fan of, but I'd forgotten how transformative that is, compared to the usual approach of all right, here's your username, here's a password you're going to have to change, here are your IAM credentials to store on disk forever. It's the ability to look at what we're doing through the eyes of someone who is clearly deep into the technical weeds, but not as exposed to all of the minutiae of the 300-some-odd AWS services is really a refreshing thing for all of us, just because it helps us realize what it's like to see some of this stuff for the first time, as well as gives me content ideas because if it's new to you, I promise you are not the only person who's seeing it that way. And if you don't really understand something well enough to explain it, I would argue you don't really understand the thing, so it forces me to get more awareness around exactly how different facets work. It's been an absolutely fantastic experience so far, from my perspective.Alex: Thank you. Right back at you. I mean, spending so many years working with startups, my kind of level of expected sophistication is, “I'm going to write your password on the back of a napkin. I have fifteen other things to do. Go figure it out.” And so you know, it's always nice to see—particularly players like AWS that are such 800-pound gorillas—going in and trying to uplevel that experience in a way that feels like—because I mean, like, look, AWS could keep us with the, “Here's a CSV with your username and password. Good luck, have fun.” And you know, they would still make—Corey: And they're going to have to because so much automation is built around that—Alex: Oh yeah—Corey: In so many places.Alex: —so much.Corey: It's always net-additive, they never turn anything off, which is increasingly an operational burden.Alex: Yeah, absolutely. Absolutely. But yeah, it's nice to see them up-level this in a way that feels like they're paying attention to their customers' pain. And that's always nice to see.Corey: So, we met a few years ago—in the before times—at a mixer that we wound up throwing—slash meetup. It was in Southern California for some AWS event or another. You've been aware of who we are and what we do for a while now, so I'm very curious to know—and the joy of having these conversations is that I don't actually know what the answer is going to be, so this may never see the light of day if it goes to weird—Alex: [laugh].Corey: —in the wrong direction, but—no I'm kidding. What has been, I guess, the biggest points of dissonance or surprises based upon your perception of who we are and what we do externally, versus joining and seeing how the sausage is made?Alex: You know, I think the first thing is—um, well, how to put this. I think that a lot of what I was expecting, given how much work you all do and how big—well, ‘you all;' we do—and how big the list of clients is and how it gets bigger every day, I was expecting this to be, like, this very hyper put together, like, every little detail has been figured out kind of engagement where I would have to figure out how you all do this. And coming in and realizing that a lot of it is just having a lot of in-depth knowledge born from experience of a bunch of stuff inside of this ecosystem, and then the rest of it is kind of free jazz, is kind of encouraging. Because as someone that was you know, as a freelancer, right, who do you see, right? You see people who have big public presences or people who are giant firms, right?On the GCP side, SADA Systems is a great example. They're another local company for me here in Los Angeles, and—Corey: Oh, yes. [unintelligible 00:24:48] Miles has been a recurring guest on the show.Alex: Yeah. And he's great. And, like, they have this enormous company that's got, like, all these different specializations and they're basically kind of like the middleman for GCP on a lot of things. And, like, you see that, and then you kind of see the individual people that are like, “Yeah, you know, I'm not really going to tell you that I only have two clients and that if both of them go away, I'm screwed, but, like, I only have two clients, and if both of them go away, I'm screwed.” And so, you know, I think honestly seeing that, like, what you've built so far and what I hope to help you continue to build is, you know, you've got just enough structure around the thing so that it makes sense, and the rest of it, you're kind of admitting that no plan ever survives contact with the client, right, and that everybody's going to be different than that everybody's problems are going to be different.And that you can't just go in and say, “Here's a dashboard, here's a calculator, have fun, give me my money,” right? Because that feels like—in optimization spaces of any kind, be that cloud, or data or whatever, there's this, kind of, push toward, how do I automate myself out of a job, and the realization that you can't for something like this, and that ultimately, like, you're just going to have to go with what you know, is something that I kind of had a suspicion was the case, but this really made it clear to me that, like, oh, this is actually a reasonable way of going about this.Corey: We thought otherwise at one point. We thought that this was something could be easily addressed their software. We launched our DuckTools SaaS platform in beta and two months later, did the—our incredible journey has come to an end, and took it off of a public offering. Because it doesn't lend itself to solving these problems in software in any reasonable way. I am ever more convinced over time that the idea of being able to solve cloud cost optimization with software at VC-scale is a red herring.And yeah, it just isn't going to work because it's one size fits some. Our customers are, by definition, exceptional in many respects, and understanding the context behind why things are the way that they are mean that we can only go so far with process because then it becomes a let's have a conversation and let's be human. Otherwise, we try to overly codify the process, and congratulations, we just now look like really crappy software, but expensive because it's all people doing it. It doesn't work that way. We have tools internally that help smooth over a lot of those edges, but by and large, people who are capable of performing at especially at the principal level for a cloud economics role, inherently are going to find themselves stifled by too much process because they need to have the freedom to dig into the areas that are relevant to the customer.It's why we can't recraft all of our statements of work in ways that tend to shy away from explicitly defined deliverables. Because we deliver an outcome, but it's going to depend entirely, in most cases, up on what we discover along the way. Maybe a full-on report isn't the best way of presenting the data in the way that we see it. Maybe it's a small proof of concept script or something like that. Maybe it's, I don't know, an interpretive dance in front of the company's board.Alex: [laugh]. Right.Corey: I'm open to exploring opportunities. But it comes down to what is right for the customer. There's a reason we only ever charge a fixed fee for these things, and it's because at that point, great, we're giving you the advice that we'd implement ourselves. We have no partnerships with any vendor in the space just to avoid bias or the perception of same. It's important that we are the authoritative source around these things.Honestly, the thing that surprised me the most about all this is how true to that vision we've stayed as we've as we flushed out what works, what doesn't. And we can distantly fail to go out of business every month. I am ecstatic about that. I expected this to wind up cratering into a mountain four months after I went freelance. Not yet.Alex: Well, I mean, I think there's another aspect of this too, right? Because I've spent a lot of my career working inside of venture capital-backed companies. And there's a lot of positive things to be said about having ready access to that kind of cash, but it does something to your business the second you take it. And I've been in a couple of situations where, like, once you actually have that big bucket of money, the incentive is grow, right? Hire more people get more customers, go, go, go, go, go.And sometimes what you'll find is that you'll spend the time and the money on an initiative and it's clearly not working. And you just kind of have to keep doubling down because now you've got customers that are using this thing and now you have to maintain it, and before you know it, you've got this albatross hanging around your neck. And like one of the things that I really respect about the way that Duckbill Group is is handling this by not taking outside cash is, like, it frees you up to make these kinds of bets, and then two months later say, “Well, that didn't work,” and try something else. And you know, that's very difficult to do once you have to go and convince someone with, you know, money flowing out of their ears, that that's the right thing to do.Corey: We have to be intentional about what we're doing. One of the benefits of bringing you aboard is that one, it does improve our capacity for handling more engagements at the same time, but it also improves the quality of the engagements that we are delivering. Instead of basically doing a round-robin assignment policy we can—Alex: Right.Corey: —we consult with each other; we talk about specific areas in which we have specific expertise. You get dragged into a lot of data portions of existing engagements, and the rest of us get pulled into other areas in which you might not be as strong. For example, “What are all of these ridiculous services? I can't make heads or tails have the ridiculous naming side of it.” Surprise, that's not a you problem.It comes down to being able to work collaboratively and let each other shine in a way that doesn't mean we load people up with work. We're very strict about having a 40-hour or less work week, just because we're not rushing for an exit. We want to enjoy our time working, we want to enjoy what we're doing, and then we want to go home and don't think about work until it's time to come back and think about these things. Like, it's a lifestyle company, but that lifestyle doesn't need to be run, run, run, run, run all the time, and it doesn't need to be something that people barely tolerate.Alex: Yeah. And I think that, you know, especially coming from being an army of one in a lot of engagements, it is really refreshing to be able to—see because, you know, I'm fortunate enough, I have friends in the industry that I can go and say like, “I have no idea how to make heads or tails of X.” And you know, I can get help that way, but ultimately, like, the only other outlet that I have here is the customer and they're not bringing me in if they have those answers readily to hand. And so being able to bounce stuff off of other people inside of an organization like this has been really refreshing.Corey: One of the things I've appreciated about your tenure here so far is the questions that you ask are pitched at the perfect level, by which I mean, it is never something you could answer with a three-second visit to Google, but it's also not something that you've spent three days spinning your wheels on trying to understand. You do a bit of digging; it's a little unclear, especially since there are multiple paths to go down, and then you flag it for clarification. And there's really so much to be said for that. Really, when we're looking for markers of seniority in the interview process, it's admitting you don't know something, but then also talking about how you would go about getting the answer. And it's—because no one has all this stuff in their head. I spend a disturbing amount of time looking at search engines and trying to reformulate queries and to get answers that make sense.I don't have the entirety of AWS shoved into my head. Yet. I'm sure there's something at re:Invent that's going to be scary and horrifying that will claim to do it and basically have a poor user interface, but all right. When that comes, we'll reevaluate then because this industry is always changing.Alex: For sure. For sure. And I think it's, it's worth pointing out that, like, one of the things that having done this for a long time gives you is this kind of scaffolding in your head that you can hang things over. We're like, you don't need to have every single AWS service memorized, but if you've got that scaffold in your head going, “Oh, like, this thing sounds like it hangs over this part of the mental scaffold, and I've seen other things that do that, so I wonder if it does this and this and this,” right? And that's a lot of it, honestly.Because especially, like, when I was solely in the data space, there's a new data wareho—or a new, like, data catalog system coming out every other week. You know, there are a thousand different things that claim to do MLOps, right? And whenever, like, someone comes to me and says, “Do you have experience with such and such?” And the answer was usually, “Well if you hum a few bars, I can fake it.” And, you know, that tends to help a great deal.Corey: Yeah. “No, but I'll find out and get back to you,” the right answer. Making it up and being wrong is the best way to get rejected from an environment. That's not just consulting; that's employment, too. If 95% of the time, you give the right answer, but that one time and 20 you're going to just make it up, well, I have to validate the other 19 because I never know when someone's faking it or not. There's that level of earned trust that's important.Alex: Well, yeah. And you're being brought in to be the expert in the room. That doesn't necessarily mean that you are the all-seeing, all-knowing oracle of knowledge but, like, if you say a thing, people are just going to believe you. And so, you know, it's beholden on you—Corey: If not, we have a different problem.Alex: Well, yeah, exactly. Hopefully, right? But yeah, I mean, it's beholden on you to be honest with your customer at a certain point, I think.Corey: I really want to thank you for taking the time out of your day to got with me about this. And I would love to have you back on in a couple of months once you're fully up to speed and spinning at the proper RPMs and see what's happened then. I—Alex: Thank you. I'd—Corey: —really appreciate—Alex: —love to.Corey: —your time where's the best place for people to learn more about you if they haven't heard your name before?Alex: Well, let's see. I am @alexras on Twitter, A-L-E-X-R-A-S. My personal website is alexras.info.I've done some writing on data stuff, including a pretty big collection of blog posts on the data side of the AWS ecosystem that are still on my consulting page, bitsondisk.com. Other than that—I mean, yeah, Twitter is probably the best place to find me, so if you want to talk more about any weird, nerd data stuff, then please feel free to reach out there.Corey: And links to that will, of course, be in the [show notes 00:35:57]. Thanks again for your time. I really appreciate it.Alex: Thank you. It's been a pleasure.Corey: Alex Rasmussen, principal cloud economist here at The Duckbill Group. I am Corey Quinn, cloud economist to the stars, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that you then submit to three other podcast platforms just to make sure you have a backup copy of that particular piece of data.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Episode 130:     Welcome to Pi- Perspectives. Today, we have a great show. We are talking about the ALIVE active Shooter training program with Michael Julian. Mike is a PI in California and will be one of the speakers at the Intellenet conference in Vegas next month. Mike has been working in this industry since he was 16 years old and has a great perspective on the industry. Please welcome Michael Julian and your host, Private Investigator, Matt Spaier Links:      Matt's email: MatthewS@Satellitepi.com   Linkedin: Matthew Spaier       www.investigators-toolbox.com   PI-Perspectives Youtube link:  https://www.youtube.com/channel/UCYB3MaUg8k5w3k7UuvT6s0g Michael Julian Info: Linkedin: Michael julian Email: mjulian@investigations-nbi.com https://sociatap.com/ALIVEActiveShooter/ https://www.cali-pi.org/events Sponsors:     https://apps.crosstrax.co/signup/index/refcd/LY3R7VUW69 https://investigationeducation.com/ https://www.skopenow.com/

Julian Schwarzenbach is a Consultant, Author, Trainer & Director at Data and Process Advantage where he is passionate about helping organizations improve their profitability, efficiency and quality through better exploitation of their data, particularly for asset intensive organizations. Mike Petrusky asks Julian about his role as a 'Data Evangelist' why he believes that "Data does not have to be difficult!" They discuss how to spot and avoid the most common "data fallacies", strike the right balance between data and software, and set up long-term data management systems. Check out these highlights from their recent live Asset Champion broadcast and then download the full video recording! Connect with Julian on LinkedIn: https://www.linkedin.com/in/julianschwarzenbach/ Learn more about DPA: https://www.dpadvantage.co.uk/ Watch the full video recording with Mike & Julian: https://managerplus.iofficecorp.com/webinar-download-why-good-data-is-never-enough Connect with Mike on LinkedIn: https://www.linkedin.com/in/mikepetrusky/ Learn more about the iOFFICE + SpaceIQ Asset Division and explore more interviews at: https://www.assetchampion.com/ Share your thoughts with Mike via email: podcast@iOFFICECORP.com  

About DanielleDanielle Baskin is a serial entrepreneur and multimedia artist whose work has been featured in The New York Times, The Guardian, NPR, The New Yorker, WSJ, and more. She's also the CEO of Dialup, a globally acclaimed voice-chat app.Links: Dialup: https://dialup.com Twitter: https://twitter.com/djbaskin Cofounder Quest: https://cofounder.quest Personal Website: https://daniellebaskin.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It seems like there is a new security breach every day. Are you confident that an old SSH key, or a shared admin account, isn't going to come back and bite you? If not, check out Teleport. Teleport is the easiest, most secure way to access all of your infrastructure. The open source Teleport Access Plane consolidates everything you need for secure access to your Linux and Windows servers—and I assure you there is no third option there. Kubernetes clusters, databases, and internal applications like AWS Management Console, Yankins, GitLab, Grafana, Jupyter Notebooks, and more. Teleport's unique approach is not only more secure, it also improves developer productivity. To learn more visit: goteleport.com. And not, that is not me telling you to go away, it is: goteleport.com. Corey: You know how Git works right?Announcer: Sorta, kinda, not really. Please ask someone else.Corey: That's all of us. Git is how we build things, and Netlify is one of the best ways I've found to build those things quickly for the web. Netlify's Git-based workflows mean you don't have to play slap-and-tickle with integrating arcane nonsense and web hooks, which are themselves about as well understood as Git. Give them a try and see what folks ranging from my fake Twitter for Pets startup, to global Fortune 2000 companies are raving about. If you end up talking to them—because you don't have to; they get why self-service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y dot com.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. It's always fun when I get the opportunity to talk to people whose work inspires me, and makes me reflect more deeply upon how I go about doing things in various ways. Now, for folks who have been following my journey for a while, it's pretty clear that humor plays a big part in this, but that is not something that I usually talk about with respect to whose humor inspires me.Today that's going to change a little bit. My guest is Danielle Baskin, who among so many other things is the CEO of a company called Dialup, but more notably is renowned for pulling a bunch of—I don't know if we'd call them pranks. I don't know if we would call them performance art. I don't know if we would call them shitposting in real life, but they are all amazing. Danielle, thank you so much for joining. How do you describe what it is that you do?Danielle: Thanks for having me. Yeah, I've used a few different terms. I've called it situation design. I've called it serious jokes. I have called what I do business art, but all the things you said, shitposting IRL, that's part of it too.Corey: It's been an absolute pleasure to just watch what you've done since I first became aware of you, which our mutual friend, Chloe Condon first pointed me in your general direction with, “Hey, Corey, you think you're funny? You should watch what Danielle is doing.” That's not how she framed it, but that's what I took from it because I'm incredibly egotistical, which is now basically a brand slash core personality trait. There you have it.And I encountered you for the first time in person—I believe only time to date—at I believe it was Oracle OpenWorld on the expo floor. She had been talking about you a couple of days before, and I saw someone who could only be you because you were dressed as a seer to be at Oracle OpenWorld. The joke should be clear to folks but we'll explain it later for the folks who are—might need to replay that a bit. I staggered up to you with, “Hey, are you Chloe's friend?”Let me give listeners here some advice through counterexample. Don't do that. It makes you look like a sketchy person who has no clue how social graces work. No one has any context and as soon as you said, “No,” I realized, “Oh, I came across as a loon.” I am going to say, “Never mind. My mistake,” and walk away like a sensible person will after bungling an introduction like that. I'm not usually that inartful about these things. I don't know what the hell happened, but it happens often when we meet people that we consider celebrities, and sorry, for some of us that's you.Danielle: [laugh] yeah, also in fairness to you I was probably fully immersed in character being my wizard self, and so I was not there to, you know, be pulled back to reality. For some context, I was at Oracle OpenWorld because I made a thing called same exact name, oracleopenworld.org, but it's a divination conference for oracles, for fortune-tellers, for wizards, for seers, and it happened at the exact same place in time, so there was a whole crew of people dressed up with capes, and robes, and tall pointy hats doing tarot readings and practicing our divination skills.Corey: Now, I could wind up applying about two dozen different adjectives to Oracle, but playful is absolutely not one of them. I would not ever accuse Oracle, or frankly any large company of that scale of having anything even remotely resembling a sense of humor. As someone who does have to factor in the not that remote possibility of getting kicked out of events that I attend, how do you handle that and not find yourself arrested?Danielle: Oh, we were kicked out every single time.Corey: Oh, good good good.Danielle: I've done this for four years. The first year we were kicked out just because we didn't have badges. I made up our own conference lanyard; of course, there's security issues with that. We were pushed out onto the sidewalk, but I wanted to be inside the conference and closer to the building.The next year I did a two-layer conference badge, so I put the real one underneath the fake one so that if security went up to us we had the right to be there. What sort of happened—so, like, the first year we got kicked out was because we were all distributed; maybe there was like 20 of us. Sometimes we were together. Sometimes we were having our own adventures. My friend Brian decided do a séance for the Deloitte team.Corey: Well, that's Deloitte-ful. Tell me more.Danielle: [laugh]. Brian has never done a séance before, but he is a good improv actor and also a spiritual person, so this is, like, perfect for him. As the Deloitte team if they wanted to do a séance they were, like, sure because I think they didn't have anything going—I mean, people are bored at this conference.Corey: Oh, of course, they are.Danielle: Especially if your boss flew you there to stand at your booth and you've been saying the same thing over and over again; you're looking for something interesting. So, he grabs the pillows from a lounge area and little tea light candles and makes a whole circle so that the team can sit down.He's wearing a bright rainbow cape and he stands in the middle and he could have a booming voice if he wants to. So, he just starts riffing and going—he just goes into séance mode, and this was enough to trigger security noticing that something really weird was happening. And when they went—Corey: They come over and say, “What the hell is this?” The answer was “Kubernetes.”Danielle: I had said everyone can blame—if you get in trouble just blame me just say, “I'm doing this with my friend, Danielle,” and have them talk to me. I wanted more people to come and be wizards. I don't want them to worry about it, so I will take all of the issues on me. He said that he should talk to his manager, Danielle, or I don't know.He said something that made it seem we were all part of a company. Which then makes it seem like our whole project was secret guerilla marketing for something. And we didn't pay for booth. We were not selling anything. We were just trolling. Or not troll—I mean, we were having our own divination summit. We were genuinely—Corey: You were virally marketing is the right answer and from my perspective—Danielle: Yeah, no, I wasn't doing viral marketing. They think anything that's unusual and getting people's attention has the ultimate goal of selling something, which it's not a philosophy I live by.Corey: No, it feels like the weird counter-intuitive thing here is the way to get the blessing of everyone from this would've—the only step you missed was charging Deloitte for doing it at their booth because it attracts attention.Danielle: Oh, sure. Oracle should have been paying us a lot of money for entertaining people. Actually, genuinely I had some real heart-to-heart conversations with people who wanted to have a tarot reading about how should they talk to their boss about not listening to them. This is something magical that happens when you are dressed up in costume and you are acting really weird people feel they can say anything because you're acting way more unusual than them, so it sort of takes away people's barriers. So, people are very honest with me about their situation.People had questions about their family. Anyway, I was in the middle of a heart-to-heart tarot reading, and security at Oracle was alerted to find anyone with a cape. Find the wizards and kick them out because they didn't pay to be here. There's some weird marketing thing happen.Corey: “Find and eject the wizards,” is probably the most surreal thing that they have been told that year.Danielle: Oh, yeah. And they didn't know why. The message why I did not transmit to all the security, but they were just told to find us. Two guards with their walkie-talkies in their uniforms went up to me and they had to escort me off the premises. Which means we had to walk through the conference together and I asked them, “Why?” They're like, “We don't know. We were just told to find you.”Corey: Imagine them trying to find you stopping and asking people, “Excuse me, have you seen the wizard?”Danielle: Exactly.Corey: It is hard to be taken seriously when asking questions like that.Danielle: Totally, totally. So yeah, unfortunately, we had to leave and that has consistently happened because I've done it four times. The final year I went, there was a message before the event even started that you're not allowed to wear a cape.Corey: The fact that you can have actual changes made to company policy for large-scale, incredibly expensive events like that is a sign that you've made it.Danielle: It doesn't even point to any particular incident. Yeah, it's cool to have this sort of lore. When I asked in the last year I went, “I asked why can't we wear a cape?” And one of the event organizer security, I don't know what her role was. She said, “There was an incident the previous year.” Which she was talking about me and my friends.Corey: Of course, but that is the best part of it.Danielle: It's just lore than something once happened with these, like, dark spirits that tried to mess up the Oracle conference with their magic.Corey: Times change and events evolve. Years ago I attended an AWS Summit with a large protest sign that said on it AMI has three syllables, and it got a bit of an eyebrow raise from people at the door, but okay, great. Then people started protesting those events for one of the very many reasons people have to protest Amazon, and they keep piling more on that pile all the time which is neither here nor there.I realized, okay, I can't do that anymore because regardless of what the sign says I will get tackled at the door for trying to bring something like that in, and I don't try and actively disrupt keynotes. So okay, it's time to move on and not get myself viewed through certain lenses that are unhelpful, but it's always a question of moving on and try to top what I did previous years. Weren't you also at Dreamforce wearing pajamas?Danielle: I did a few things at Dreamforce. One year I literally set up a tent. They spend millions of dollars on beautiful fake trees and rocks, and also Dreamforce gets taken over every time the event occurs. I did a few things. I thought I should make it seem like this is real nature so I brought camping gear and a tent and just brought a hiking backpack in.Set it up in the middle of the conference floor laying by the waterfall, but there were people in suits networking around me that did not ask me any questions. I just stayed in the tent, but then I decided to list it on Airbnb. So, inside my tent, I was making an Airbnb listing telling people that they could stay at Dreamforce and explore the beautiful nature there, but it took an hour-and-a-half to get kicked out.Corey: The emails that you must have back and forth with places like Airbnb's customer support line and the rest have got to be legendary at this point.Danielle: [laugh] I get interesting cease-and-desists. I wish there was more dialogue. With Airbnb I just got my listing taken down and I couldn't talk to a human, and even when I got kicked out of Dreamforce they wanted me to leave immediately. I totally snuck in; I didn't have a badge or anything. So, I guess they're in the right for that. The second year at Dreamforce I wore a ghillie suit so I hid. So, I stayed a little bit after the conference ended by hiding as a bush.Corey: That is both amazing and probably terrifying for the worker that encountered you while trying to clean up.Danielle: Oh, I mean often employees—like it depends. Some people find my pranks really delightful because it shakes up their day. Security guards also find this amusing. There's some type of organizer that absolutely hates my pranks.Corey: There's something to be said for self-selecting your own audience. One question that I—sure you get; if I get it I know you get it—where it's difficult for people to sometimes draw the line between the fun whimsical things that you do as pranks and the actual things that you do. A great example of this is something you've been doing for, I think, four years now, the decruiter.Danielle: Yeah. The decruiter a service that's the opposite of a recruiter so it is—Corey: At the first re:Invent AWS had a slide that was apparently he made the night before or something and they misspelled security as decurity. From that perspective, what's a decruiter?Danielle: Yes, I love decurity as a way to talk about infiltrating a space, like, “No I'm a decurity officer.” Yeah, decruiter is basically a service where you talk to us to find out if you should quit your job. Instead of finding out if you should work at a place or figuring out what opportunities there are, we discuss the unemployed life—or the inbet—like, being self-employed, between jobs, switching careers, it's a whole spectrum but there's a few recruiters and we're all like very experienced not having an employer or working for a company. And so, we ask people about how would you spend your free time. What's your financial situation? Are you able to afford leaving? It gets pretty personal, but it's highly specific therapy, but we also don't have a high acceptance rate. I've only decruited like 15% people that I've talked to.Corey: Most of them realize that, oh, there's a lot of things I would have to do if I didn't have a job and I'm just going to stay where I am?Danielle: Yeah. Well, I think a lot of people think that as soon as they leave their job a lot of other things in their life will magically transform, or they'll finally be able to do their creative project they've always wanted to do. This is true some percentage of the time, but I always encourage people to do things outside of work and not seek in their whole fulfillment through their job.There's plenty of time where you can explore other ideas and even overlap them to make sure that like when you quit you have things lined up. A lot of people don't know how to answer, “If you suddenly left tomorrow and could just float for three months, what would you do?” If people give me a good answer—and this is similar to an actual job interview I was like, “Why are you excited about working this company?”If people give me a good answer, that's a conversation. A lot of people have no idea, but they're just stuck in a situation where there's things they could do in their outside of work life that would make them feel happier. That's why it's sort of like therapy, but there's a lot of internal company issues that I talk about. A common reason that people want to leave is that they love their role, they love the company's mission, but they do not like their manager, but their manager is really good friends with the CEO and they absolutely can't say anything. This is so common.Corey: They always say people they'll quit jobs they quit managers and there is something to be said for that.Danielle: Yes, it's scary for people to speak up or who do you write a letter to? How do you secretly talk with your team about it? Are you the only one feeling that way? Typically the people that are the most nervous about saying anything are kind of young either in their early 20s and they feel like they can't say anything.I encourage them to come up with a strategy for making change within their corporation but sometimes it's not worth it. If there's tons of other opportunities for them it's not worth them fixing their company.Corey: It's also I think not incumbent upon people to fix their entire corporate culture unless they're at a somewhat higher executive level. That's a fun thing. The derecruiter.com we'll definitely throw a link to that in the [show notes 00:15:49] and I'll start driving people to it when they ask me for advice on these things. Then you decided, okay, that's fun.You're one of those people I feel has a bit of the same alignment that I do which is, why do one thing when I could do a bunch of things? And you decided, ah, you're going to do a startup. What is the best thing that you can do that really can capitalize on emerging cultural trends? That's right. Getting millennial to make phone calls to each other. Tell me about that story.Danielle: Yeah, and it's not just millennials, though I'm millennial. So, a lot of millennials use Dialup. I mean, Dialup started as a project where basically me and a friend set up a robocall between ourselves. So, like a bot would call our phones and if we would pick up we'd both be connected, but neither of us was actually calling each other. So, it was a way to just always be catching up with each other.So, many friends asked me if they could join the robocalls. That was sort of the seat of Dialup is getting serendipitous phone calls throughout the day that connect you to a person that you might know or might want to meet. Because there's overlap of interest or overlap of someone you know. It grew from me and 20 friends to now 31,000 people who are actively using it all over the world and these conversations can be really incredible.Sometimes people stay on the phone for four hours. People have flown out to meet each other. I get notes every day of how a call has impacted someone one. So, that's what I'm up to now, but I'm trying to do more interesting things with voice technology. I just like realized, oh, the voice as a medium it just transports you to other worlds. You have space to imagine.I mean, people listening to this podcast right now they're not seeing us, but they probably are imagining us, what our rooms look like, what we look like. They're imagining the stories that we're telling them without the distraction of video. I want to do more interesting things with intimate audio—not broadcast stuff. Not Clubhouse or Spaces or anything like that, but just more interesting ways to connect people in one-on-ones.Corey: Something I've noticed is that the voice has a power that text does not. It makes it easier to remember that there's a human on the other side of things. It is far easier for me to send off an incendiary tweet at someone than it is for me to call them up and then berate them, not really my style.The more three-dimensional someone becomes in various capacities and the higher bandwidth the communication takes on, I think the easier it is to remember that most people who don't work at Facebook wake up in the morning hoping to do a good job today. Extending empathy to the rest of the world, that's an important thing.Danielle: Yeah, for sure. It's incredible that humans can detect emotional qualities in a voice call. It's hard to describe why, but people can detect pauses and little mutters. You can sort of know when someone's laughing or when someone's listening even though you're missing all of the visual cues.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: Taking a glance at dialup.com, it appears to be a completely free service. You mentioned that it has 30,000 folks involved. Are you taking the VC model of we're going to get a whole bunch of users first and then figure out how to make money later? Sometimes it works super well. Other times it basically becomes Docker retold.Danielle: I've been thinking about this a lot and I swing back and forth. Right now Dialup is its own thing, connecting strangers. It's free though I do have some paying clients because I do serendipitous one-on-ones within organizations. I've got a secret B2B page, and so that is a little bit of revenue. Right now I'm trying to sort of expand beyond Dialup and make a new thing, in which case I am leaning more towards building a sustainable and profitable company rather than do the raise-VC-money-until-you-die model.Corey: I think it's long past time to disrupt the trope of starving artist. What about well-paid artist? It seems like that would inspire and empower people to create a lot more art when they're not worrying about freezing to death. To that end or presumably to that end you are in the process of looking for a co-founder in what is arguably the most Danielle Baskin possible way. How are you doing it?Danielle: Oh, yeah. I could have done a regular LinkedIn post linking to a Google Doc, but that is not my style, and as a self-employed person I can't reach out to old coworkers and be like, “Oh, you're on my team a few years ago. What are you up to now?” So, I'm sort of under-networked and I thought I should make a game that sort of explains what I'm doing, but have people discover the game in an interesting way. So, I bought a bunch of floppy discs—I have a floppy disc dealer outside of LA.Corey: For those who are not millennials and are in fact younger than that—and of course let's not forget Gen X, the Baby Boom Generation, the Silent Generation which I can only assume is comprised entirely of people who represent big companies from a PR point of view because they never comment on anything. What is a floppy disc for someone who was born in, I don't know, 2005?Danielle: Oh, a floppy disk is how you would run software on your computer.Corey: Yeah, a USB stick with no capacity you can wreck with a magnet.Danielle: Yes, it's like a flat wide USB stick, but it only contains—Corey: 1.44 megabytes on the three-and-a-half-inch version.Danielle: I think some of them then went up to 2.88.Corey: Ohh.Danielle: You can't even fit a picture—a modern picture. You could do a super low-resolution pixel art.Corey: This picture of grandma has a whopping eight pixels in it. Oh, okay, great. I guess.Danielle: Yeah. More complex software would be eight floppy disks that you have to insert disk A, insert disk B.Corey: Anti-piracy warnings in that day of ‘don't copy that floppy.' It was a seminal thing for a long time.Danielle: I have it in my game; it says ‘don't make illegal copies of this game.' My game is not literally on the floppy disc. All floppy discs come with pretty interesting artwork on the label. There's a little space for a sticker, and because I have hundreds of floppy disks, I sort of looked at—I had a ton of design inspiration.So, I made floppy discs in the aesthetic of the other ones that say Cofounder Quest—like it's this game—and it leads you to a website. I scattered these in strategic places around the bay area, and I also mailed some to people outside of the bay area. If you stumble across this in person or on the internet, it leads you to this adventure game that's around seven minutes to play.It really explains what I want to do with Dialup, and explains me, and explains my aesthetic, and the sort of playful experiences that I'm into without telling you. So, you get to really experience it. At the end, it basically leads you to a job description and tells you to reach out to me if you're interested.Corey: I was independent for years and I finally decided to take on a business partner. As it turns out, Mike Julian, who's the CEO of The Duckbill Group and I go back ten years, he's my best friend. I kept correcting him. He introduced me as his friend. I said, “No, Mike, your best friend.” Then I got him on audio at one point saying, “Oh, Corey Quinn? He's my best friend.” I have that on my soundboard and I play it every time he gets uppity. That's the sort of nonsense it's important in a co-founder relationship. It is a marriage in some respects.Danielle: Oh, for sure.Corey: It's a business entity. Each one of you can destroy the other financially in different ways. You have to have shared values. The idea of speed-dating your way through finding some random co-founder as a job application, on some level, has always struck me as a little dissonant. I like the approach you're taking of this is who I am and how I go about things. If this aligns then we should talk, and if you don't like this you're not going to like any of the rest of this.Danielle: For sure. I'm definitely self-selecting with who would actually reach out after playing. I also understand. I'm not going to find a co-founder in a few weeks. I'm just starting conversations with people and then seeing who I should continue talking to or seeing if we could do a mini-project together.Yeah, it's weird. It's a very intense relationship. That's why people do end up becoming co-founders with someone that they already know who's a friend. It's possible I already know my co-founder and they've been in front of me this whole time. I think these sorts of moments happen, but I also think that it's cool to totally expand your network and meet someone who maybe has an overlap in spirit, but is someone that you would've never otherwise met. That there could be this great overlap or convergence there. I wanted to cast a very wide net with who this would reach, but it's still going to be a multi-month-long process or longer.Corey: It's not these one-off projects that are the most interesting part to me. It is the sheer variety and consistency of this. During the pandemic I believe you wound up having the verified checkmark badges for houses and fill out this form if you want one and for folks in San Francisco. Absolutely, of course, I filled that out. I read a fairly bad take news article on it of a bunch of people fell for this prank.No, absolutely not. If people are familiar with your work then they know exactly what they're getting into with something like this and you support the kinds of things you want to see more of in the world. I didn't fall for anything. I wanted to see where it led and that's how I feel on everything you do.Danielle: Yeah, you appreciated the joke.Corey: Yeah.Danielle: Yeah, I think people who are familiar with my work understand that I take jokes very seriously. So, it's not simply—like, usually it's not just a website that's like, huh, this was a trick. It's more of an ongoing theater piece. So, I actually did go through all of the applicants for the Blue Check Homes. Oh, for some context, I made a website where you could apply to have a blue verified badge and a plaster crest put on your house if you are a dignified authentic person that lives in the house.So, I'm interviewing—I narrowed it down to 50 people from all the applicants and I'm going through and interviewing people with a committee. I'm recording all of the interviews because I think this will make an interesting mini-documentary. I'm actually making one in installing one, but I'm documenting all of it.When I started it—for a lot of projects I don't have the ending planned yet. I like the sort of joke to unfold on the internet in real-time, and then figure out what the next thing I should do from there is and continue the project in a sort of curious exploratory mindset as opposed to just saying, “All right, the joke is done.”Corey: What is your process for coming up with this stuff? Because for me the most intimidating thing I ever see in the course of a week is not the inevitable cease and desist I get from every large cloud company for everything I do. Rather an empty page where it's all right time for me to write a humorous blog post, or start drafting the bones of a Twitter thread, or start writing my resignation and if I don't come with an idea by the end of it, I'll submit it. Where does the creative process start from with you?Danielle: Yeah. I rarely have creative brainstorming sessions. I'm a person who thinks of a million bad ideas and then there's one good one. My mind leaps to a ton of ideas. I rarely write down ideas. I don't do any sort of—you might imagine I'm in a room of whiteboards and post-it notes, workshopping things and doing creative brainstorm sessions, but I don't.I think I act upon the things that I feel just extremely excited about and feel like I must do this immediately. It's hard to explain, but with a lot of my ideas, I just feel this surge of energy. I have to do this because no one else will do it and it's funny at this moment. If I don't feel that way I kind of don't do anything and see if the idea keeps reemerging. With a lot of ideas I may be thought of it a year ago and it just kept resurfacing, but I don't really force myself to churn out creative projects if that makes sense. People have told me that my work reminds them of Mischief. It's like as a company that puts out a prank on a Tuesday every two weeks.Corey: Not familiar with them, but there have been a whole bunch of flash mob groups, and other folks who affected just wind up being professional pranksters, which I love the concept.Danielle: Yeah, yeah, for sure. I do churn out a lot of pranks and I even have my own prank calendar. I'm not strict with my own deadlines and I also think timing is important. So, you might think of a good idea, but then it's just the spirit of the zeitgeist doesn't want you to do it that week. I improvise the things that I want to launch. I mostly do things that I just feel are rich in something I could explore.Like, with Cofounder Quest I was always on the fence about it because it feels to me annoying to tell people you're trying to hire someone or to put yourself out there and be pitching your startup. So, I was kind of nervous about that, but I also thought if I leave a floppy disk in the park, and then put a picture on the internet it'll lead to something—there's something that it will lead to.It might lead to finding a co-founder. It might lead to meeting interesting people, but also I've never built an interactive game with audio and so I was interested in learning that, but yeah, I tend to land on ideas that I think are rich in terms of things I could learn. Things that I could turn into more immersive theater and things that keep resurfacing as opposed to keeping myself on a strict schedule of creative ideas if that makes sense.Corey: It makes a lot of sense. It's one of those things that it is not commonly understood for those of us who came up in the nose of the grindstone 40 hours a week, have a work ethic. Even if you're not busy look busy. Sometimes work looks a lot more like getting up and going to a coffee shop and meeting some stranger from the internet than it does sitting down churning out code.Danielle: For sure. I think that it is important to continue being in conversations with people. I think good ideas emerge while you're in the middle of talking, and you realize your own limitations and ideas when you have to explain things to other people. While something you're very clear in your head as soon as there's a person you don't know and they ask you, “What are you working on?” You realize, oh, there's so many gaps. It made perfect sense to me, but there's a lot of gaps. So yeah, I think it's important to stay in dialogue and also have to explain yourself to new people instead of just sort of making ideas in a vacuum.Corey: I want to thank you for being so generous with your time and talking to me about all the various things you have going on. If people want to follow along and learn more about what you're up to, where can they find you?Danielle: I post a lot of my projects on Twitter. So, I'm @djbaskin. If you want to play Cofounder Quest, it's cofounder.quest. That is an actual domain. I also have a website daniellebaskin.com, which has a lot of my projects, many of which we didn't discuss. I also do, similar to Oracle OpenWorld, I like to host popup events that involve lots of people trolling. So, if you want to get involved in anything you see I'm always happy to bring more wizards on board.Corey: We will, of course, put links to that in the [show notes 00:31:10]. Danielle, thank you so much for taking the time to speak with me today.Danielle: Oh yeah, thanks for having me. It was great talking with you.Corey: Danielle Baskin, CEO of Dialup, and oh so very much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast please leave a five-star review on your podcast platform of choice, along with a long rambling comment applying to be the co-host of this podcast, viewing it of course as a podcasting call.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About ThomasThomas Hazel is Founder, CTO, and Chief Scientist of ChaosSearch. He is a serial entrepreneur at the forefront of communication, virtualization, and database technology and the inventor of ChaosSearch's patented IP. Thomas has also patented several other technologies in the areas of distributed algorithms, virtualization and database science. He holds a Bachelor of Science in Computer Science from University of New Hampshire, Hall of Fame Alumni Inductee, and founded both student & professional chapters of the Association for Computing Machinery (ACM).Links:ChaosSearch: https://www.chaossearch.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by my friends at ThinkstCanary. Most companies find out way too late that they've been breached. ThinksCanary changes this and I love how they do it. Deploy canaries and canary tokens in minutes and then forget about them. What's great is the attackers tip their hand by touching them, giving you one alert, when it matters. I use it myself and I only remember this when I get the weekly update with a “we're still here, so you're aware” from them. It's glorious! There is zero admin overhead  to this, there are effectively no false positives unless I do something foolish. Canaries are deployed and loved on all seven continents. You can check out what people are saying at canary.love. And, their Kub config canary token is new and completely free as well. You can do an awful lot without paying them a dime, which is one of the things I love about them. It is useful stuff and not an, “ohh, I wish I had money.” It is speculator! Take a look; that's canary.love because it's genuinely rare to find a security product that people talk about in terms of love. It really is a unique thing to see. Canary.love. Thank you to ThinkstCanary for their support of my ridiculous, ridiculous non-sense.   Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted episode is brought to us by our friends at ChaosSearch.We've been working with them for a long time; they've sponsored a bunch of our nonsense, and it turns out that we've been talking about them to our clients since long before they were a sponsor because it actually does what it says on the tin. Here to talk to us about that in a few minutes is Thomas Hazel, ChaosSearch's CTO and founder. First, Thomas, nice to talk to you again, and as always, thanks for humoring me.Thomas: [laugh]. Hi, Corey. Always great to talk to you. And I enjoy these conversations that sometimes go up and down, left and right, but I look forward to all the fun we're going to have.Corey: So, my understanding of ChaosSearch is probably a few years old because it turns out, I don't spend a whole lot of time meticulously studying your company's roadmap in the same way that you presumably do. When last we checked in with what the service did-slash-does, you are effectively solving the problem of data movement and querying that data. The idea behind data warehouses is generally something that's shoved onto us by cloud providers where, “Hey, this data is going to be valuable to you someday.” Data science teams are big proponents of this because when you're storing that much data, their salaries look relatively reasonable by comparison. And the ChaosSearch vision was, instead of copying all this data out of an object store and storing it on expensive disks, and replicating it, et cetera, what if we queried it in place in a somewhat intelligent manner?So, you take the data and you store it, in this case, in S3 or equivalent, and then just query it there, rather than having to move it around all over the place, which of course, then incurs data transfer fees, you're storing it multiple times, and it's never in quite the format that you want it. That was the breakthrough revelation, you were Elasticsearch—now OpenSearch—API compatible, which was great. And that was, sort of, a state of the art a year or two ago. Is that generally correct?Thomas: No, you nailed our mission statement. No, you're exactly right. You know, the value of cloud object stores, S3, the elasticity, the durability, all these wonderful things, the problem was you couldn't get any value out of it, and you had to move it out to these siloed solutions, as you indicated. So, you know, our mission was exactly that, transformed customers' cloud storage into an analytical database, a multi-model analytical database, where our first use case was search and log analytics, replacing the ELK stack and also replacing the data pipeline, the schema management, et cetera. We automate the entire step, raw data to insights.Corey: It's funny we're having this conversation today. Earlier, today, I was trying to get rid of a relatively paltry 200 gigs or so of small files on an EFS volume—you know, Amazon's version of NFS; it's like an NFS volume except you're paying Amazon for the privilege—great. And it turns out that it's a whole bunch of operations across a network on a whole bunch of tiny files, so I had to spin up other instances that were not getting backed by spot terminations, and just firing up a whole bunch of threads. So, now the load average on that box is approaching 300, but it's plowing through, getting rid of that data finally.And I'm looking at this saying this is a quarter of a terabyte. Data warehouses are in the petabyte range. Oh, I begin to see aspects of the problem. Even searching that kind of data using traditional tooling starts to break down, which is sort of the revelation that Google had 20-some-odd years ago, and other folks have since solved for, but this is the first time I've had significant data that wasn't just easily searched with a grep. For those of you in the Unix world who understand what that means, condolences. We're having a support group meeting at the bar.Thomas: Yeah. And you know, I always thought, what if you could make cloud object storage like S3 high performance and really transform it into a database? And so that warehouse capability, that's great. We like that. However to manage it, to scale it, to configure it, to get the data into that, was the problem.That was the promise of a data lake, right? This simple in, and then this arbitrary schema on read generic out. The problem next came, it became swampy, it was really hard, and that promise was not delivered. And so what we're trying to do is get all the benefits of the data lake: simple in, so many services naturally stream to cloud storage. Shoot, I would say every one of our customers are putting their data in cloud storage because their data pipeline to their warehousing solution or Elasticsearch may go down and they're worried they'll lose the data.So, what we say is what if you just said activate that data lake and get that ELK use case, get that BI use case without that data movement, as you indicated, without that ETL-ing, without that data pipeline that you're worried is going to fall over. So, that vision has been Chaos. Now, we haven't talked in, you know, a few years, but this idea that we're growing beyond what we are just going after logs, we're going into new use cases, new opportunities, and I'm looking forward to discussing with you.Corey: It's a great answer that—though I have to call out that I am right there with you as far as inappropriately using things as databases. I know that someone is going to come back and say, “Oh, S3 is a database. You're dancing around it. Isn't that what Athena is?” Which is named, of course, after the Greek Goddess of spending money on AWS? And that is a fair question, but to my understanding, there's a schema story behind that does not apply to what you're doing.Thomas: Yeah, and that is so crucial is that we like the relational access. The time-cost complexity to get it into that, as you mentioned, scaled access, I mean, it could take weeks, months to test it, to configure it, to provision it, and imagine if you got it wrong; you got to redo it again. And so our unique service removes all that data pipeline schema management. And because of our innovation because of our service, you do all schema definition, on the fly, virtually, what we call views on your index data, that you can publish an elastic index pattern for that consumption, or a relational table for that consumption. And that's kind of leading the witness into things that we're coming out with this quarter into 2022.Corey: I have to deal with a little bit of, I guess, a shame here because yeah, I'm doing exactly what you just described. I'm using Athena to wind up querying our customers' Cost and Usage Reports, and we spend a couple hundred bucks a month on AWS Glue to wind up massaging those into the way that they expect it to be. And it's great. Ish. We hook it up to Tableau and can make those queries from it, and all right, it's great.It just, burrr goes the money printer, and we somehow get access and insight to a lot of valuable data. But even that is knowing exactly what the format is going to look like. Ish. I mean, Cost and Usage Reports from Amazon are sort of aspirational when it comes to schema sometimes, but here we are. And that's been all well and good.But now the idea of log files, even looking at the base case of sending logs from an application, great. Nginx, or Apache, or [unintelligible 00:07:24], or any of the various web servers out there all tend to use different logging formats just to describe the same exact things, start spreading that across custom in-house applications and getting signal from that is almost impossible. “Oh,” people say, “So, we'll use a structured data format.” Now, you're putting log and structuring requirements on application developers who don't care in the first place, and now you have a mess on your hands.Thomas: And it really is a mess. And that challenge is, it's so problematic. And schemas changing. You know, we have customers and one reasons why they go with us is their log data is changing; they didn't expect it. Well, in your data pipeline, and your Athena database, that breaks. That brings the system down.And so our system uniquely detects that and manages that for you and then you can pick and choose how you want to export in these views dynamically. So, you know, it's really not rocket science, but the problem is, a lot of the technology that we're using is designed for static, fixed thinking. And then to scale it is problematic and time-consuming. So, you know, Glue is a great idea, but it has a lot of sharp [pebbles 00:08:26]. Athena is a great idea but also has a lot of problems.And so that data pipeline, you know, it's not for digitally native, active, new use cases, new workloads coming up hourly, daily. You think about this long-term; so a lot of that data prep pipelining is something we address so uniquely, but really where the customer cares is the value of that data, right? And so if you're spending toils trying to get the data into a database, you're not answering the questions, whether it's for security, for performance, for your business needs. That's the problem. And you know, that agility, that time-to-value is where we're very uniquely coming in because we start where your data is raw and we automate the process all the way through.Corey: So, when I look at the things that I have stuffed into S3, they generally fall into a couple of categories. There are a bunch of logs for things I never asked for nor particularly wanted, but AWS is aggressive about that, first routing through CloudTrail so you can get charged 50-cent per gigabyte ingested. Awesome. And of course, large static assets, images I have done something to enter colloquially now known as shitposts, which is great. Other than logs, what could you possibly be storing in S3 that lends itself to, effectively, the type of analysis that you built around this?Thomas: Well, our first use case was the classic log use cases, app logs, web service logs. I mean, CloudTrail, it's famous; we had customers that gave up on elastic, and definitely gave up on relational where you can do a couple changes and your permutation of attributes for CloudTrail is going to put you to your knees. And people just say, “I give up.” Same thing with Kubernetes logs. And so it's the classic—whether it's CSV, where it's JSON, where it's log types, we auto-discover all that.We also allow you, if you want to override that and change the parsing capabilities through a UI wizard, we do discover what's in your buckets. That term data swamp, and not knowing what's in your bucket, we do a facility that will index that data, actually create a report for you for knowing what's in. Now, if you have text data, if you have log data, if you have BI data, we can bring it all together, but the real pain is at the scale. So classically, app logs, system logs, many devices sending IoT-type streams is where we really come in—Kubernetes—where they're dealing with terabytes of data per day, and managing an ELK cluster at that scale. Particularly on a Black Friday.Shoot, some of our customers like—Klarna is one of them; credit card payment—they're ramping up for Black Friday, and one of the reasons why they chose us is our ability to scale when maybe you're doing a terabyte or two a day and then it goes up to twenty, twenty-five. How do you test that scale? How do you manage that scale? And so for us, the data streams are, traditionally with our customers, the well-known log types, at least in the log use cases. And the challenge is scaling it, is getting access to it, and that's where we come in.Corey: I will say the last time you were on the show a couple of years ago, you were talking about the initial logging use case and you were speaking, in many cases aspirationally, about where things were going. What a difference a couple years is made. Instead of talking about what hypothetical customers might want, or what—might be able to do, you're just able to name-drop them off the top of your head, you have scaled to approximately ten times the number of employees you had back then. You've—Thomas: Yep. Yep.Corey: —raised, I think, a total of—what, 50 million?—since then.Thomas: Uh, 60 now. Yeah.Corey: Oh, 60? Fantastic.Thomas: Yeah, yeah.Corey: Congrats. And of course, how do you do it? By sponsoring Last Week in AWS, as everyone should. I'm taking clear credit for that every time someone announces around, that's the game. But no, there is validity to it because telling fun stories and sponsoring exciting things like this only carry you so far. At some point, customers have to say, yeah, this is solving a pain that I have; I'm willing to pay you money to solve it.And you've clearly gotten to a point where you are addressing the needs of those customers at a pretty fascinating clip. It's bittersweet from my perspective because it seems like the majority of your customers have not come from my nonsense anymore. They're finding you through word of mouth, they're finding through more traditional—read as boring—ad campaigns, et cetera, et cetera. But you've built a brand that extends beyond just me. I'm no longer viewed as the de facto ombudsperson for any issue someone might have with ChaosSearch on Twitters. It's kind of, “Aww, the company grew up. What happened there?”Thomas: No, [laugh] listen, this you were great. We reached out to you to tell our story, and I got to be honest. A lot of people came by, said, “I heard something on Corey Quinn's podcasts,” or et cetera. And it came a long way now. Now, we have, you know, companies like Equifax, multi-cloud—Amazon and Google.They love the data lake philosophy, the centralized, where use cases are now available within days, not weeks and months. Whether it's logs and BI. Correlating across all those data streams, it's huge. We mentioned Klarna, [APM Performance 00:13:19], and, you know, we have Armor for SIEM, and Blackboard for [Observers 00:13:24].So, it's funny—yeah, it's funny, when I first was talking to you, I was like, “What if? What if we had this customer, that customer?” And we were building the capabilities, but now that we have it, now that we have customers, yeah, I guess, maybe we've grown up a little bit. But hey, listen to you're always near and dear to our heart because we remember, you know, when you stop[ed by our booth at re:Invent several times. And we're coming to re:Invent this year, and I believe you are as well.Corey: Oh, yeah. But people listening to this, it's if they're listening the day it's released, this will be during re:Invent. So, by all means, come by the ChaosSearch booth, and see what they have to say. For once they have people who aren't me who are going to be telling stories about these things. And it's fun. Like, I joke, it's nothing but positive here.It's interesting from where I sit seeing the parallels here. For example, we have both had—how we say—adult supervision come in. You have a CEO, Ed, who came over from IBM Storage. I have Mike Julian, whose first love language is of course spreadsheets. And it's great, on some level, realizing that, wow, this company has eclipsed my ability to manage these things myself and put my hands-on everything. And eventually, you have to start letting go. It's a weird growth stage, and it's a heck of a transition. But—Thomas: No, I love it. You know, I mean, I think when we were talking, we were maybe 15 employees. Now, we're pushing 100. We brought on Ed Walsh, who's an amazing CEO. It's funny, I told him about this idea, I invented this technology roughly eight years ago, and he's like, “I love it. Let's do it.” And I wasn't ready to do it.So, you know, five, six years ago, I started the company always knowing that, you know, I'd give him a call once we got the plane up in the air. And it's been great to have him here because the next level up, right, of execution and growth and business development and sales and marketing. So, you're exactly right. I mean, we were a young pup several years ago, when we were talking to you and, you know, we're a little bit older, a little bit wiser. But no, it's great to have Ed here. And just the leadership in general; we've grown immensely.Corey: Now, we are recording this in advance of re:Invent, so there's always the question of, “Wow, are we going to look really silly based upon what is being announced when this airs?” Because it's very hard to predict some things that AWS does. And let's be clear, I always stay away from predictions, just because first, I have a bit of a knack for being right. But also, when I'm right, people will think, “Oh, Corey must have known about that and is leaking,” whereas if I get it wrong, I just look like a fool. There's no win for me if I start doing the predictive dance on stuff like that.But I have to level with you, I have been somewhat surprised that, at least as of this recording, AWS has not moved more in your direction because storing data in S3 is kind of their whole thing, and querying that data through something that isn't Athena has been a bit of a reach for them that they're slowly starting to wrap their heads around. But their UltraWarm nonsense—which is just, okay, great naming there—what is the point of continually having a model where oh, yeah, we're going to just age it out, the stuff that isn't actively being used into S3, rather than coming up with a way to query it there. Because you've done exactly that, and please don't take this as anything other than a statement of fact, they have better access to what S3 is doing than you do. You're forced to deal with this thing entirely from a public API standpoint, which is fine. They can theoretically change the behavior of aspects of S3 to unlock these use cases if they chose to do so. And they haven't. Why is it that you're the only folks that are doing this?Thomas: No, it's a great question, and I'll give them props for continuing to push the data lake [unintelligible 00:17:09] to the cloud providers' S3 because it was really where I saw the world. Lakes, I believe in. I love them. They love them. However, they promote the move the data out to get access, and it seems so counterintuitive on why wouldn't you leave it in and put these services, make them more intelligent? So, it's funny, I've trademark ‘Smart Object Storage,' I actually trademarked—I think you [laugh] were a part of this—‘UltraHot,' right? Because why would you want UltraWarm when you can have UltraHot?And the reason, I feel, is that if you're using Parquet for Athena [unintelligible 00:17:40] store, or Lucene for Elasticsearch, these two index technologies were not designed for cloud storage, for real-time streaming off of cloud storage. So, the trick is, you have to build UltraWarm, get it off of what they consider cold S3 into a more warmer memory or SSD type access. What we did, what the invention I created was, that first read is hot. That first read is fast.Snowflake is a good example. They give you a ten terabyte demo example, and if you have a big instance and you do that first query, maybe several orders or groups, it could take an hour to warm up. The second query is fast. Well, what if the first query is in seconds as well? And that's where we really spent the last five, six years building out the tech and the vision behind this because I like to say you go to a doctor and say, “Hey, Doc, every single time I move my arm, it hurts.” And the doctor says, “Well, don't move your arm.”It's things like that, to your point, it's like, why wouldn't they? I would argue, one, you have to believe it's possible—we're proving that it is—and two, you have to have the technology to do it. Not just the index, but the architecture. So, I believe they will go this direction. You know, little birdies always say that all these companies understand this need.Shoot, Snowflake is trying to be lake-y; Databricks is trying to really bring this warehouse lake concept. But you still do all the pipelining; you still have to do all the data management the way that you don't want to do. It's not a lake. And so my argument is that it's innovation on why. Now, they have money; they have time, but, you know, we have a big head start.Corey: I remembered last year at re:Invent they released a, shall we say, significant change to S3 that it enabled read after write consistency, which is awesome, for again, those of us in the business of misusing things as databases. But for some folks, the majority of folks I would say, it was a, “I don't know what that means and therefore I don't care.” And that's fine. I have no issue with that. There are other folks, some of my customers for example, who are suddenly, “Wait a minute. This means I can sunset this entire janky sidecar metadata system that is designed to make sure that we are consistent in our use of S3 because it now does it automatically under the hood?” And that's awesome. Does that change mean anything for ChaosSearch?Thomas: It doesn't because of our architecture. We're append-only, write-once scenario, so a lot of update-in-place viewpoints. My viewpoint is that if you're seeing S3 as the database and you need that type of consistency, it make sense of why you'd want it, but because of our distributive fabric, our stateless architecture, our append-only nature, it really doesn't affect us.Now, I talked to the S3 team, I said, “Please if you're coming up with this feature, it better not be slower.” I want S3 to be fast, right? And they said, “No, no. It won't affect performance.” I'm like, “Okay. Let's keep that up.”And so to us, any type of S3 capability, we'll take advantage of it if benefits us, whether it's consistency as you indicated, performance, functionality. But we really keep the constructs of S3 access to really limited features: list, put, get. [roll-on 00:20:49] policies to give us read-only access to your data, and a location to write our indices into your account, and then are distributed fabric, our service, acts as those indices and query them or searches them to resolve whatever analytics you need. So, we made it pretty simple, and that is allowed us to make it high performance.Corey: I'll take it a step further because you want to talk about changes since the last time we spoke, it used to be that this was on top of S3, you can store your data anywhere you want, as long as it's S3 in the customer's account. Now, you're also supporting one-click integration with Google Cloud's object storage, which, great. That does mean though, that you're not dependent upon provider-specific implementations of things like a consistency model for how you've built things. It really does use the lowest common denominator—to my understanding—of object stores. Is that something that you're seeing broad adoption of, or is this one of those areas where, well, you have one customer on a different provider, but almost everything lives on the primary? I'm curious what you're seeing for adoption models across multiple providers?Thomas: It's a great question. We built an architecture purposely to be cloud-agnostic. I mean, we use compute in a containerized way, we use object storage in a very simple construct—put, get, list—and we went over to Google because that made sense, right? We have customers on both sides. I would say Amazon is the gorilla, but Google's trying to get there and growing.We had a big customer, Equifax, that's on both Amazon and Google, but we offer the same service. To be frank, it looks like the exact same product. And it should, right? Whether it's Amazon Cloud, or Google Cloud, multi-select and I want to choose either one and get the other one. I would say that different business types are using each one, but our bulk of the business isn't Amazon, but we just this summer released our SaaS offerings, so it's growing.And you know, it's funny, you never know where it comes from. So, we have one customer—actually DigitalRiver—as one of our customers on Amazon for logs, but we're growing in working together to do a BI on GCP or on Google. And so it's kind of funny; they have two departments on two different clouds with two different use cases. And so do they want unification? I'm not sure, but they definitely have their BI on Google and their operations in Amazon. It's interesting.Corey: You know its important to me that people learn how to use the cloud effectively. Thats why I'm so glad that Cloud Academy is sponsoring my ridiculous non-sense. They're a great way to build in demand tech skills the way that, well personally, I learn best which I learn by doing not by reading. They have live cloud labs that you can run in real environments that aren't going to blow up your own bill—I can't stress how important that is. Visit cloudacademy.com/corey. Thats C-O-R-E-Y, don't drop the “E.” Use Corey as a promo-code as well. You're going to get a bunch of discounts on it with a lifetime deal—the price will not go up. It is limited time, they assured me this is not one of those things that is going to wind up being a rug pull scenario, oh no no. Talk to them, tell me what you think. Visit: cloudacademy.com/corey,  C-O-R-E-Y and tell them that I sent you!Corey: I know that I'm going to get letters for this. So, let me just call it out right now. Because I've been a big advocate of pick a provider—I care not which one—and go all-in on it. And I'm sitting here congratulating you on extending to another provider, and people are going to say, “Ah, you're being inconsistent.”No. I'm suggesting that you as a provider have to meet your customers where they are because if someone is sitting in GCP and your entire approach is, “Step one, migrate those four petabytes of data right on over here to AWS,” they're going to call you that jackhole that you would be by making that suggestion and go immediately for option B, which is literally anything that is not ChaosSearch, just based upon that core misunderstanding of their business constraints. That is the way to think about these things. For a vendor position that you are in as an ISV—Independent Software Vendor for those not up on the lingo of this ridiculous industry—you have to meet customers where they are. And it's the right move.Thomas: Well, you just said it. Imagine moving terabytes and petabytes of data.Corey: It sounds terrific if I'm a salesperson for one of these companies working on commission, but for the rest of us, it sounds awful.Thomas: We really are a data fabric across clouds, within clouds. We're going to go where the data is and we're going to provide access to where that data lives. Our whole philosophy is the no-movement movement, right? Don't move your data. Leave it where it is and provide access at scale.And so you may have services in Google that naturally stream to GCS; let's do it there. Imagine moving that amount of data over to Amazon to analyze it, and vice versa. 2020, we're going to be in Azure. They're a totally different type of business, users, and personas, but you're getting asked, “Can you support Azure?” And the answer is, “Yes,” and, “We will in 2022.”So, to us, if you have cloud storage, if you have compute, and it's a big enough business opportunity in the market, we're there. We're going there. When we first started, we were talking to MinIO—remember that open-source, object storage platform?—We've run on our laptops, we run—this [unintelligible 00:25:04] Dr. Seuss thing—“We run over here; we run over there; we run everywhere.”But the honest truth is, you're going to go with the big cloud providers where the business opportunity is, and offer the same solution because the same solution is valued everywhere: simple in; value out; cost-effective; long retention; flexibility. That sounds so basic, but you mentioned this all the time with our Rube Goldberg, Amazon diagrams we see time and time again. It's like, if you looked at that and you were from an alien planet, you'd be like, “These people don't know what they're doing. Why is it so complicated?” And the simple answer is, I don't know why people think it's complicated.To your point about Amazon, why won't they do it? I don't know, but if they did, things would be different. And being honest, I think people are catching on. We do talk to Amazon and others. They see the need, but they also have to build it; they have to invent technology to address it. And using Parquet and Lucene are not the answer.Corey: Yeah, it's too much of a demand on the producers of that data rather than the consumer. And yeah, I would love to be able to go upstream to application developers and demand they do things in certain ways. It turns out as a consultant, you have zero authority to do that. As a DevOps team member, you have limited ability to influence it, but it turns out that being the ‘department of no' quickly turns into being the ‘department of unemployment insurance' because no one wants to work with you. And collaboration—contrary to what people wish to believe—is a key part of working in a modern workplace.Thomas: Absolutely. And it's funny, the demands of IT are getting harder; the actual getting the employees to build out the solutions are getting harder. And so a lot of that time is in the pipeline, is the prep, is the schema, the sharding, and et cetera, et cetera, et cetera. My viewpoint is that should be automated away. More and more databases are being autotune, right?This whole knobs and this and that, to me, Glue is a means to an end. I mean, let's get rid of it. Why can't Athena know what to do? Why can't object storage be Athena and vice versa? I mean, to me, it seems like all this moving through all these services, the classic Amazon viewpoint, even their diagrams of having this centralized repository of S3, move it all out to your services, get results, put it back in, then take it back out again, move it around, it just doesn't make much sense. And so to us, I love S3, love the service. I think it's brilliant—Amazon's first service, right?—but from there get a little smarter. That's where ChaosSearch comes in.Corey: I would argue that S3 is in fact, a modern miracle. And one of those companies saying, “Oh, we have an object store; it's S3 compatible.” It's like, “Yeah. We have S3 at home.” Look at S3 at home, and it's just basically a series of failing Raspberry Pis.But you have this whole ecosystem of things that have built up and sprung up around S3. It is wildly understated just how scalable and massive it is. There was an academic paper recently that won an award on how they use automated reasoning to validate what is going on in the S3 environment, and they talked about hundreds of petabytes in some cases. And folks are saying, ah, S3 is hundreds of petabytes. Yeah, I have clients storing hundreds of petabytes.There are larger companies out there. Steve Schmidt, Amazon's CISO, was recently at a Splunk keynote where he mentioned that in security info alone, AWS itself generates 500 petabytes a day that then gets reduced down to a bunch of stuff, and some of it gets loaded into Splunk. I think. I couldn't really hear the second half of that sentence because of the sound of all of the Splunk salespeople in that room becoming excited so quickly you could hear it.Thomas: [laugh]. I love it. If I could be so bold, those S3 team, they're gods. They are amazing. They created such an amazing service, and when I started playing with S3 now, I guess, 2006 or 7, I mean, we were using for a repository, URL access to get images, I was doing a virtualization [unintelligible 00:29:05] at the time—Corey: Oh, the first time I played with it, “This seems ridiculous and kind of dumb. Why would anyone use this?” Yeah, yeah. It turns out I'm really bad at predicting the future. Another reason I don't do the prediction thing.Thomas: Yeah. And when I started this company officially, five, six years ago, I was thinking about S3 and I was thinking about HDFS not being a good answer. And I said, “I think S3 will actually achieve the goals and performance we need.” It's a distributed file system. You can run parallel puts and parallel gets. And the performance that I was seeing when the data was a certain way, certain size, “Wait, you can get high performance.”And you know, when I first turned on the engine, now four or five years ago, I was like, “Wow. This is going to work. We're off to the races.” And now obviously, we're more than just an idea when we first talked to you. We're a service.We deliver benefits to our customers both in logs. And shoot, this quarter alone we're coming out with new features not just in the logs, which I'll talk about second, but in a direct SQL access. But you know, one thing that you hear time and time again, we talked about it—JSON, CloudTrail, and Kubernetes; this is a real nightmare, and so one thing that we've come out with this quarter is the ability to virtually flatten. Now, you heard time and time again, where, “Okay. I'm going to pick and choose my data because my database can't handle whether it's elastic, or say, relational.” And all of a sudden, “Shoot, I don't have that. I got to reindex that.”And so what we've done is we've created a index technology that we're always planning to come out with that indexes the JSON raw blob, but in the data refinery have, post-index you can select how to unflatten it. Why is that important? Because all that tooling, whether it's elastic or SQL, is now available. You don't have to change anything. Why is Snowflake and BigQuery has these proprietary JSON APIs that none of these tools know how to use to get access to the data?Or you pick and choose. And so when you have a CloudTrail, and you need to know what's going on, if you picked wrong, you're in trouble. So, this new feature we're calling ‘Virtual Flattening'—or I don't know what we're—we have to work with the marketing team on it. And we're also bringing—this is where I get kind of excited where the elastic world, the ELK world, we're bringing correlations into Elasticsearch. And like, how do you do that? They don't have the APIs?Well, our data refinery, again, has the ability to correlate index patterns into one view. A view is an index pattern, so all those same constructs that you had in Kibana, or Grafana, or Elastic API still work. And so, no more denormalizing, no more trying to hodgepodge query over here, query over there. You're actually going to have correlations in Elastic, natively. And we're excited about that.And one more push on the future, Q4 into 2022; we have been given early access to S3 SQL access. And, you know, as I mentioned, correlations in Elastic, but we're going full in on publishing our [TPCH 00:31:56] report, we're excited about publishing those numbers, as well as not just giving early access, but going GA in the first of the year, next year.Corey: I look forward to it. This is also, I guess, it's impossible to have a conversation with you, even now, where you're not still forward-looking about what comes next. Which is natural; that is how we get excited about the things that we're building. But so much less of what you're doing now in our conversations have focused around what's coming, as opposed to the neat stuff you're already doing. I had to double-check when we were talking just now about oh, yeah, is that Google cloud object store support still something that is roadmapped, or is that out in the real world?No, it's very much here in the real world, available today. You can use it. Go click the button, have fun. It's neat to see at least some evidence that not all roadmaps are wishes and pixie dust. The things that you were talking to me about years ago are established parts of ChaosSearch now. It hasn't been just, sort of, frozen in amber for years, or months, or these giant periods of time. Because, again, there's—yeah, don't sell me vaporware; I know how this works. The things you have promised have come to fruition. It's nice to see that.Thomas: No, I appreciate it. We talked a little while ago, now a few years ago, and it was a bit of aspirational, right? We had a lot to do, we had more to do. But now when we have big customers using our product, solving their problems, whether it's security, performance, operation, again—at scale, right? The real pain is, sure you have a small ELK cluster or small Athena use case, but when you're dealing with terabytes to petabytes, trillions of rows, right—billions—when you were dealing trillions, billions are now small. Millions don't even exist, right?And you're graduating from computer science in college and you say the word, “Trillion,” they're like, “Nah. No one does that.” And like you were saying, people do petabytes and exabytes. That's the world we're living in, and that's something that we really went hard at because these are challenging data problems and this is where we feel we uniquely sit. And again, we don't have to break the bank while doing it.Corey: Oh, yeah. Or at least as of this recording, there's a meme going around, again, from an old internal Google Video, of, “I just want to serve five terabytes of traffic,” and it's an internal Google discussion of, “I don't know how to count that low.” And, yeah.Thomas: [laugh].Corey: But there's also value in being able to address things at much larger volume. I would love to see better responsiveness options around things like Deep Archive because the idea of being able to query that—even if you can wait a day or two—becomes really interesting just from the perspective of, at that point, current cost for one petabyte of data in Glacier Deep Archive is 1000 bucks a month. That is ‘why would I ever delete data again?' Pricing.Thomas: Yeah. You said it. And what's interesting about our technology is unlike, let's say Lucene, when you index it, it could be 3, 4, or 5x the raw size, our representation is smaller than gzip. So, it is a full representation, so why don't you store it efficiently long-term in S3? Oh, by the way, with the Glacier; we support Glacier too.And so, I mean, it's amazing the cost of data with cloud storage is dramatic, and if you can make it hot and activated, that's the real promise of a data lake. And, you know, it's funny, we use our own service to run our SaaS—we log our own data, we monitor, we alert, have dashboards—and I can't tell you how cheap our service is to ourselves, right? Because it's so cost-effective for long-tail, not just, oh, a few weeks; we store a whole year's worth of our operational data so we can go back in time to debug something or figure something out. And a lot of that's savings. Actually, huge savings is cloud storage with a distributed elastic compute fabric that is serverless. These are things that seem so obvious now, but if you have SSDs, and you're moving things around, you know, a team of IT professionals trying to manage it, it's not cheap.Corey: Oh, yeah, that's the story. It's like, “Step one, start paying for using things in cloud.” “Okay, great. When do I stop paying?” “That's the neat part. You don't.” And it continues to grow and build.And again, this is the thing I learned running a business that focuses on this, the people working on this, in almost every case, are more expensive than the infrastructure they're working on. And that's fine. I'd rather pay people than technologies. And it does help reaffirm, on some level, that—people don't like this reminder—but you have to generate more value than you cost. So, when you're sitting there spending all your time trying to avoid saving money on, “Oh, I've listened to ChaosSearch talk about what they do a few times. I can probably build my own and roll it at home.”It's, I've seen the kind of work that you folks have put into this—again, you have something like 100 employees now; it is not just you building this—my belief has always been that if you can buy something that gets you 90, 95% of where you are, great. Buy it, and then yell at whoever selling it to you for the rest of it, and that'll get you a lot further than, “We're going to do this ourselves from first principles.” Which is great for a weekend project for just something that you have a passion for, but in production mistakes show. I've always been a big proponent of buying wherever you can. It's cheaper, which sounds weird, but it's true.Thomas: And we do the same thing. We have single-sign-on support; we didn't build that ourselves, we use a service now. Auth0 is one of our providers now that owns that [crosstalk 00:37:12]—Corey: Oh, you didn't roll your own authentication layer? Why ever not? Next, you're going to tell me that you didn't roll your own payment gateway when you wound up charging people on your website to sign up?Thomas: You got it. And so, I mean, do what you do well. Focus on what you do well. If you're repeating what everyone seems to do over and over again, time, costs, complexity, and… service, it makes sense. You know, I'm not trying to build storage; I'm using storage. I'm using a great, wonderful service, cloud object storage.Use whats works, whats works well, and do what you do well. And what we do well is make cloud object storage analytical and fast. So, call us up and we'll take away that 2 a.m. call you have when your cluster falls down, or you have a new workload that you are going to go to the—I don't know, the beach house, and now the weekend shot, right? Spin it up, stream it in. We'll take over.Corey: Yeah. So, if you're listening to this and you happen to be at re:Invent, which is sort of an open question: why would you be at re:Invent while listening to a podcast? And then I remember how long the shuttle lines are likely to be, and yeah. So, if you're at re:Invent, make it on down to the show floor, visit the ChaosSearch booth, tell them I sent you, watch for the wince, that's always worth doing. Thomas, if people have better decision-making capability than the two of us do, where can they find you if they're not in Las Vegas this week?Thomas: So, you find us online chaossearch.io. We have so much material, videos, use cases, testimonials. You can reach out to us, get a free trial. We have a self-service experience where connect to your S3 bucket and you're up and running within five minutes.So, definitely chaossearch.io. Reach out if you want a hand-held, white-glove experience POV. If you have those type of needs, we can do that with you as well. But we booth on re:Invent and I don't know the booth number, but I'm sure either we've assigned it or we'll find it out.Corey: Don't worry. This year, it is a low enough attendance rate that I'm projecting that you will not be as hard to find in recent years. For example, there's only one expo hall this year. What a concept. If only it hadn't taken a deadly pandemic to get us here.Thomas: Yeah. But you know, we'll have the ability to demonstrate Chaos at the booth, and really, within a few minutes, you'll say, “Wow. How come I never heard of doing it this way?” Because it just makes so much sense on why you do it this way versus the merry-go-round of data movement, and transformation, and schema management, let alone all the sharding that I know is a nightmare, more often than not.Corey: And we'll, of course, put links to that in the [show notes 00:39:40]. Thomas, thank you so much for taking the time to speak with me today. As always, it's appreciated.Thomas: Corey, thank you. Let's do this again.Corey: We absolutely will. Thomas Hazel, CTO and Founder of ChaosSearch. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast episode, please leave a five-star review on your podcast platform of choice, whereas if you've hated this episode, please leave a five-star review on your podcast platform of choice along with an angry comment because I have dared to besmirch the honor of your homebrewed object store, running on top of some trusty and reliable Raspberries Pie.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Some time after we recorded this interview, Mike merged his business endeavors with those of his colleage Cory Quinn to form Duckbill Group, but any chance to share Mike's insight -- whether it reflects his current focus or not -- is worth doing, so I am publishing this interview despite it not reflecting Mike's latest trajectory.

About TimTim's tech career spans over 20 years through various sectors. Tim's initial journey into tech started as a US Marine. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for largeUnix-based datastores.Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with clients in his current role. Tim is also a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and 3-time Pan American Brazilian Jiu-Jitsu champion in his division.Links: Twitter: https://twitter.com/elchefe The Duckbill Group: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, I have a whole bunch of guests come on up, second time. Now, it's easy to take the naive approach of assuming that it's because it's easier for me to find a guest if I know them and don't have to reach out to brand new people all the time. This is absolutely correct; I'm exceedingly lazy. But I don't have too many folks on a third time, but that changes today.My guest is Tim Banks. I've had him on the show twice before, both times it led to really interesting conversations around a wide variety of things. Since those episodes, Tim has taken the job as a principal cloud economist here at The Duckbill Group. Yes, that is probably the strangest interview process you can imagine, but here we are. Tim, thank you so much for joining me both on the show and in the business.Tim: My pleasure, Corey. It was definitely an interesting interview process, you know, but I was glad to be here. So, I'm happy to be here a third time. I don't know if you get a jacket like you do in Saturday Night Live, if you host, like, a fifth time, but we'll see. Maybe it's a vest. A cool vest would be nice.Corey: We can come up with something.[ effectively, it can be like reverse hangman where you wind up getting a vest and every time you come on after that you get a sleeve, then you get a second sleeve, and then you get a collar, and we can do all kinds of neat stuff.Tim: I actually like that idea a lot.Corey: So, I'm super excited to be able to have this conversation with you because I don't normally talk a lot on this show about what cloud economics is because my guest usually is not as deep into the space as I am, and that's fine; people should never be as deep into this space as I am, in the general sense, unless they work here. Awesome. But I do guest on other shows, and people ask me all kinds of questions about AWS billing and cloud economics, and that's fine, it's great, but they don't ask the questions about the space in the same way that I would and the way that I think about it. So, it's hard for me to interview myself. Now, I'm not saying I won't try it someday, but it's challenging. But today, I get to take the easy path out and talk to you about it. So Tim, what the hell is a principal cloud economist?Tim: So, a principal cloud economist, is a cloud computing expert, both in architecture and practice, who looks at cloud cost in the same way that a lot of folks look at cloud security, or cloud resilience, or cloud performance. So, the same engineering concerns you have about making sure that your API stays up all the time, or to make sure that you don't have people that are able to escape containers or to make sure that you can have super, super low response times, is the same engineering fundamentals that I look at when I'm trying to find a way to reduce your AWS bill.Corey: Okay. When we say cloud cost and cloud economics, the natural picture that leads to mind is, “Oh, I get it. You're an Excel jockey.” And sometimes, yeah, we all kind of play those roles, but what you're talking about is something else entirely. You're talking about engineering expertise.And sure enough, if you look at the job postings we have for roles on the team from time to time, we have not yet hired anyone who does not have an engineering and architecture background. That seems odd to folks who do not spend a lot of time thinking about the AWS bill. I'm told those people are what is known as ‘happy.' But here we are. Why do we care about the engineering aspect of any of this?Tim: Well, I think first and foremost because what we're doing in essence, is still engineering. People aren't putting construction paper up on [laugh] AWS; sometimes they do put recipes up on there, but it still involves working on a computer, and writing code, and deploying it somewhere. So, to have that basic understanding of what it is that folks are doing on the platform, you have to have some engineering experience, first and foremost. Secondly, the fact of the matter is that most cost optimization, in my opinion, can be done on the whiteboard, before anything else, and really I think should be done on the whiteboard before anything else. And so the Excel aspect of it is always reactive. “We have now spent this much. How much was it? Where did it go?” And now we have to figure out where it went.I like to figure out and get a ballpark on how much something is going to cost before I write the first line of code. I want to know, hey, we have a tier here, we're using this kind of storage, it's going to take this kind of instance types. Okay, well, I've got an idea of how much it's going to cost. And I was like, “You know, that's going to be expensive. Before we do anything, is there a way that we can reduce costs there?”And so I'm reverse engineering that on already deployed workloads. Or when customers want to say, “Hey, we were thinking about doing this, and this is our proposed architecture,” I'm going to look at it and say, “Well, if you do this and this and this and this, you can save money.”Corey: So, it sounds like you and I have a bit of a philosophical disagreement in some ways. One of my recurring talking points has always been that, “Oh, by and large, application developers don't need to think overly much about cloud cost. What they need to know generally fits on an index card.” It's, okay, big things cost more than small things; if you turn something on, it will never get turned off and will bill you in perpetuity; data transfer has some weird stuff; and if you store data, you pay for data, like, that level of baseline understanding. When I'm trying to build something out my immediate thought is, great, is this thing possible?Because A, I don't always know that it is, and B, I'm super bad at computers so for me, it may absolutely not be, whereas you're talking about baking cost assessments into the architecture as a day one type of approach, even when sketching ideas out on the whiteboard. I'm curious as to how we diverge there. Can you talk more about your philosophy?Tim: Sure. And the reason I do that is because, as most folks that have an engineering background in cloud infrastructure will tell you, you want to build resilience in, on the whiteboard. You certainly want to build performance in, on the whiteboard, right? And security folks will tell you you want to do security on the whiteboard. Because those things are hard to fix after they're deployed.As soon as they're deployed, without that, you now have technical debt. If you don't consider cost optimization and cost efficiency on the whiteboard, and then you try and do it after it's deployed, you not only have technical debt, you may have actual real debt.Corey: One of the comments I tend to give a lot is that architecture and cost are the same thing in the world of cloud. And I think that we might be in violent agreement, as Liz Fong-Jones is fond of framing it, where I am acutely aware of aspects of cost and that does factor into how I build things on the whiteboard—let's also be very clear, most of the things that I build are very small scale; the largest cost by a landslide is the time I spend building it—in practice, that's an awful lot of environments; people are always more expensive than the AWS environment they're working on. But instead, it's about baking in the assumptions and making sure you're not coming up with something that is going to just be wasteful and horrible out of the gate, and I guess part of that also is the fact that I am at a level of billing understanding that I sort of absorbed these concepts intrinsically. Because to me, there is no difference between cost and architecture in an environment like this. You're right, there's always an inherent trade-off between cost and durability. On the one hand, I don't like that. On the other, it feels like it's been true forever and I don't see a way out of it.Tim: It is inescapable. And it's interesting because you talk about the level of an application developer or something like that, like what is your level of concern, but retroactively, we'll go in for cost optimization houses—and I've done this as far back as when I was working at AWS has a TAM—and I'll ask the question to an application developer or database administrator, and I'm like, “Why do you do this? What do you have a string value for something that could be a Boolean?” And you'll ask, “Well, what difference does that make?” Well, it makes a big difference when you're talking about cycles for CPU.You can reduce your CPU consumption on a database instance by changing a string to a Boolean, you need fewer instances, or you need a less powerful instance, or you need less memory. And now you can run a less expensive instance for your database architecture. Well, maybe for one node it's not that biggest difference, but if you're talking about something that's multi-AZ and multi-node, I mean, that can be a significant amount of savings just by making one simple change.Corey: And that might be the difference right there. I didn't realize that, offhand. It makes sense if you think about it, but just realizing that I've made that mistake on one of my DynamoDB tables. It costs something like seven cents a month right now, so it's not something I'm rushing to optimize, but you're right, expand that out by a factor of a million or so, and we're talking serious money, and then that sort of optimization makes an awful lot of sense. I think that my position on it is that when you're building out something small scale as a demo or a proof of concept, spending time on optimizations like this is not the best use of anyone's time or brain sweat, for lack of a better term. How do you wind up deciding when it's time to focus on stuff like that?Tim: Well, first, I will say that—I daresay that somewhere in the 80% of production workloads are just—were the POC, [laugh] right? Because, like, “It worked for this to get funding, let's run it,” right?Corey: Let they who does not have a DynamoDB table in production with the word ‘test' or ‘dev' in it cast the first stone.Tim: It's certainly not me. So, I understand how some of those decisions get made. And that's why I think it's better to think about it early. Because as I mentioned before, when you start something and say, “Hey, this works for now,” and you don't give consideration to that in the future, or consideration for what it's going to be like in the future, and when you start doing it, you'll paint yourself into corners. That's how you get something like static values put in somewhere, or that's how you get something like, well, “We have to run this instance type because we didn't build in the ability to be more microservice-based or stateless or anything like that.”You've seen people that say, “Hey, we could save you a lot of money if you can move this thing off to a different tier.” And it's like, “Well, that would be an extensive rewrite of code; that'd be very expensive.” I daresay that's the main reason why most AS/400s are still being used right now is because it's too expensive to rewrite the code.Corey: Yeah, and there's no AWS/400 that they can migrate to. Yet. Re:Invent is nigh.Tim: So, I think that's why, even at the very beginning, even if you were saying, “Well, this is something we will do later.” Don't make it impossible for you to do later in your code. Don't make it impossible for you to do later in your architecture. Make things as modular as possible, so that way you can say, “Hey”—later on down the road—“Oh, we can switch this instance type.” Or, “Here's a new managed service that we can maybe save money on doing this.”And you allow yourself to switch things out, or turn different knobs, or change the way you do things, and give yourself more options in the future, whether those options are for resilience, or those options or for security, or those options are for performance, or they're for cost optimizations. If you make binding decisions earlier on, you're going to have debt that's going to build up at some point in the future, and then you're going to have to pay the piper. Sometimes that piper is going to be AWS.Corey: One thing that I think gets lost in a lot of conversations about cloud economics—because I know that it happened to me when I first started this place—where I am planning to basically go out and be the world's leading expert in AWS cost analysis and understanding and optimization. Great. Then I went out into the world and started doing some of my first engagements, and they looked a lot less like far-future cost attribution projections and a lot more like, “What's a reserved instance?” And, “We haven't bought any of those in 18 months.” And, “Oh, yeah, we shut down an entire project six months ago. We should probably delete all the resources, huh?”The stuff that I was preparing for at the high end of the maturity curve are great and useful and terrific to have conversations about in some very nuanced depth, but very often there's a walk before you can run style of conversation where, okay, let's do the easy stuff first before we start writing a whole bunch of bespoke internal stuff that maps your business needs to the AWS bill. How do you, I guess, reconcile those things where you're on the one hand, you see the easy stuff and on the other, you see some of the just the absolutely challenging, very hard, five-years-of-engineering-effort-style problems on the other?Tim: Well, it's interesting because I've seen one customer very recently who has brilliant analyses as to their cost; just well-charted, well-tagged, well-documented, well—you know, everything is diagrammed quite nicely and everything like that, and they're very, very aware of their costs, but they leave test instances running all weekend, you know, and their associated volumes and things like that. And that's a very easy thing to fix. That is a very, very low-hanging fruit. And so sometimes, you just have to look at where they're spending their efforts where sometimes they do spend so much time chasing those hard to do things because they are hard to do and they're exciting in an engineering aspect, and then something as simple as, “Hey, how about we delete these old volumes?” It just isn't there.Or, “How about we switch to your S3 bucket storage type?” Those are easy, low-hanging fruits, and you would be surprised how sometimes they just don't get that. But at the same time, sometimes customers have, like, “Hey, we could knock this thing out, we knock this thing out,” because it's Trusted Advisor. Every AI cost optimization recommendation you can get will tell you these five things to do, no matter who you are or where you are, but they don't do the conceptual things like understanding some of the principles behind cost optimization and cost optimization architecture, and proactive cost optimization versus react with cost optimizations. So, you're doing very conceptual education and conversations with folks rather than the, “Do these five things.” And I've not often found a customer that you have to do both on; it's usually one or the other.Corey: It's funny that you made that specific reference to that example. One of my very first projects—not naming names. Generally, when it comes to things like this, you can tell stories or you can name names; I bias for stories—I was talking to a company who was convinced that their developer environments were incredibly overwrought, expensive, et cetera, and burning money. Okay, great. So, I talked about the idea of turning those things off at night or between test runs, deleting volumes to snapshot, and restore them on a schedule when people come in in the morning because all your developers sit in the same building in the same time zones. Great. They were super on board with the idea, and it was going to be a little bit of work, but all right, this was in the days before the EC2 Instance Scheduler, for example.But first, let's go ahead and do some analysis. This is one of those early engagements that really reinforced my idea of, yeah, before we start going too far down the rabbit hole, let's double-check what's going on in the account. Because periodically you encounter things that surprise people. Like, “What's up with those Australia instances?” “Oh, we don't have anything in that region.” “I believe you're being sincere when you say this, however, the API generally doesn't tell lies.”So, that becomes a, oh, security incident time. But looking at this, they were right; they had some fairly sizable developer instances that were running all the time, but doing some analysis, their developer environment was 3% of their bill at the time and they hadn't bought RIs in a year-and-a-half. And looking at what they were doing, there was so much easier stuff that they could do to generate significant savings without running the potential of turning a developer environment off at night in the middle of an incident or something like that. The risk factor and effort were easier just do the easy stuff, then do another pass and look at the deep stuff. And to be clear, they weren't lying to me; they weren't wrong.Back when they started building this stuff out, their developer environments were significantly large and were a significant portion of their spend. And then they hit product-market fit, and suddenly their production environment had to scale significantly in a short period of time. Which, yay, cloud. It's good at that. Then it just became such a small portion that developer environments weren't really a thing. But the narrative internally doesn't get updated very often because once people learn something, they don't go back to relearn whether or not it's still true. It's a constant mistake; I make it myself frequently.Tim: I think it's interesting, there are things that we really need to put into buckets as far as what's an engineering effort and what's an administrative effort. And when I say ‘administrative effort,' I mean if I can save money with a stroke of a pen, well, that's going to be pretty easy, and that's usually going to be RIs; that's going to be EDPs, or PPAs or something like that, that don't require engineering effort. It just requires administrative effort, I think RIs being the simplest ones. Like, “Oh, all I have to do is go in here and click these things four times and I'm going to save money?” “Well, let's do that.”And it's surprising how often people don't do that. But you still have to understand that, and whether it's RIs or whether it's a savings plan, it's still a commitment of some kind, but if you are willing to make that commitment, you can save money with no engineering effort whatsoever. That's almost free money.Corey: So, much of what we do here comes down to psychology, in many ways, more than it does math. And a lot of times you're right, everything you say is right, but in a large-scale environment, go ahead and click that button to buy the savings plan or the reserved instance, and that's a $20 million purchase. And companies will stall for months trying to run a different series of analyses on this and what if this happens, what if that happens, and I get it because, “Yeah, I'm going to click this button that's going to cost more money than I'll make in my lifetime,” that's a scary thing to do; I get it. But you're going to spend the money, one way or the other, with the provider, and if you believe that number is too high, I get it; I am right there with you. Buy half of them right now and then you can talk about the rest until you get to a point of being comfortable with it.Do it incrementally; it's not all or nothing, you have one shot to make the buy. Take pieces out of it that makes sense. You know you're probably not going to turn off your database cluster that handles all of production in the next year, so go ahead and go for it; it saves some money. Do the thing that makes sense. And that doesn't require deep-dive analytics that requires, on some level, someone who's seen a lot of these before who gets what customers are going through. And honestly, it's empathy in many respects, becomes one of those powerful things that we can apply to our customer accounts.Tim: Absolutely. I mean, people don't understand that decision paralysis, about making those commitments costs you money. You can spend months doing analysis, but those months doing analysis, you're going to spend 30, 40, 50, 60, 70% more on your EC2 instances or other compute than you would otherwise, and that can be quite significant. But it's one of those cases where we talk about psychology around perfect being the enemy of good. You don't have to make the perfect purchase of RIs or savings plans and have that so tuned perfectly that you're going to get one hundred percent utilization and zero—like, you don't have to do that.Just do something. Do a little bit. Like you said, buy half; buy anything; just something, and you're going to save money. And then you can run analysis later on, while you're saving money [laugh] and get a little better and tune it up a little more and get more analysis on and maybe fine-tune it, but you don't actually ever need to have it down to the penny. Like, it never has to be that good.Corey: At some point, one of the value propositions we have for our customers has always been that we tell you when to stop focusing on saving money because there's a theoretical cap of a hundred percent of the cloud bill that you can save, but you can make so much more than that by launching the right feature to the right market a little sooner; focus on that. Be responsible stewards of the money that's invested with you, but by and large, as a general piece of guidance, at some point, stop cutting and go back to doing the thing that makes your company work. It's not all about saving money at all costs for almost all of us. It is for us, but we're sort of a special case.Tim: Well, it's a conversation I often have. It's like, all right, are you trying to save money on AWS or are you trying to save money overall? So, if you're going to spend $400,000 worth of engineering effort to save $10,000 on your AWS bill, that doesn't make no sense. So—[laugh]—Corey: Right. There has to be a strategic reason to do things like that—Tim: Exactly.Corey: —and make sure you understand the value of what you're getting for this. One reason that we wind up charging the way that we do—and we've gotten questions on this for a while—has been that we charge a fixed fee for what we do on engagements. And similarly—people have asked this, but haven't tied the two things together—you talk about cost optimization, but never cost-cutting. Why is that? Is that just a negative term?And the answer has been no, they're aligned. What we do focuses on what is best for the customer. Once that fixed fee is decided upon, every single thing that we say is what we would do if we were in the customer's position. There are times we'll look at what they have going on and say, “Ah, you really should spend more money here for resiliency, or durability,” or, “Okay, that is critical data that's not being backed up. You should consider doing that.”It's why we don't take percentages of things because, at that point, we're not just going with the useful stuff, it's, well we're going to basically throw the entire kitchen sink at you. We had an early customer and I was talking to their AWS account manager about what we were going to be doing and their comment was, “Oh, saving money on AWS bills is great, make sure you check the EBS snapshots.” Yeah, I did that. They were spending 150 bucks a month on EBS snapshots, which is basically nothing. It's one of those stories where if, in the course of an hour-long meeting, I can pay for that entire service, by putting a quarter on the table, I'm probably not going to talk about it barring [laugh] some extenuating circumstances.Focus on the big things, not the things that worked in a different environment with a different account and different constraints. It's hard to context switch like that, but it gets a lot easier when it is basically the entirety of what we do all day.Tim: The difference I draw between cost optimization and cost-cutting is that cost optimization is ensuring that you're not spending money unnecessarily, or that you're maximizing your dollar. And so sometimes we get called in there, and we're just validation for the measures they've already done. Like, “Your team is doing this exactly right. You're doing the things you should be doing. We can nitpick if you want to; we're going to save you $7 a year, but who cares about that? But y'all are doing what you should be doing. This is great. Going forward, you want to look for these things and look for these things and look for these things. We're going to give you some more concepts so that you are cost-optimized in the future.” But it doesn't necessarily mean that we have to cut your bill. Because if you're already spending efficiently, you don't need your bill cut; you're already cost-optimized.Corey: Oh, we're not going to nitpick on that, you're mostly optimized there. It's like, “Yeah, that workload's $140 million a year and rising; please, pick nits.” At which point? “Okay, great.” That's the strategic reason to focus on something. But by and large, it comes down to understanding what the goals of clients are. I think that is widely misunderstood about what we do and how we do it.The first question I always ask when someone does outreach of, “Hey, we'd like to talk about coming in here and doing a consulting engagement with us.” “Great.” I always like to ask the quote-unquote, “Foolish question” of, “Why do you care about the AWS bill?” And occasionally I'll get people who look at me like I have two heads of, “Why wouldn't I care about the AWS bill?” Because there are more important things to care about for the business, almost certainly.Tim: One of the things I try and do, especially when we're talking about cost optimization, especially trying to do something for the right now so they can do things going forward, it's like, you know, all right, so if we cut this much from your bill—if you just do nothing else, but do reserved instances or buy a savings plan, right, you're going to save enough money to hire four engineers. Think about what four engineers would do for your overall business? And that's how I want you to frame it; I want you to look at what cost optimization is going to allow you to do in the future without costing you any more money. Or maybe you save a little more money and you can shift it; instead of paying for your AWS bill, maybe you can train your developers, maybe you can get more developers, maybe you can get some ProServ, maybe you can do whatever, buy newer computers for your people so they can do—whatever it is, right? We're not saying that you no longer have to spend this money, but saying, “You can use this money to do something other than give it to Jeff Bezos.”Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: There was an article recently, as of the time of this recording, where Pinterest discussed what they had disclosed in one of their regulatory filings which was, over the next eight years, they have committed to pay AWS $3.2 billion. And in this article, they have the head of engineering talking to the reporter about how they're thinking about these things, how they're looking at things that are relevant to their business, and they're talking about having a dedicated team that winds up doing a whole bunch of data analysis and running some analytics on all of these things, from piece to piece to piece. And that's great. And I worry, on some level, that other companies are saying, “Oh, Pinterest is doing that. We should, too.” Yeah, for the course of this commitment, a 1% improvement is $32 million, so yeah, at that scale I'm going to hire a team of data scientists, too, look at these things. Your bill is $50,000 a month. Perhaps that's not worth the effort you're going to put into it, barring other things that contribute to it.Tim: It's interesting because we will get folks that will approach us that have small accounts—very small, small spend—and like, “Hey, can you come in and talk to us about this whatever.” And we can say very honestly, “Look, we could, but the amount of money we're going to charge you is going to—it's not going to be worth your while right now. You could probably get by on the automated recommendations, on the things that already out there on the internet that everybody can do to optimize their bill, and then when you grow to a point where now saving 10% is somebody's salary, that's when it, kind of, becomes more critical.” And it's hard to say what point that is in anyone's business, but I can say sometimes, “Hey, you know what? That's not really what you need to focus on.” If you need to save $100 a month on your AWS bill, and that's critical, you've got other concerns that are not your AWS bill.Corey: So, back when you were interviewing to work here, one of the areas of focus that you kept bringing up was the concept of observability, and my response to this was, “Ah, hell. Another one.” Because let's be clear, Mike Julian—my business partner and our CEO—has written a book called Practical Monitoring, and apparently what we learned from this is as soon as you finish writing a book on the topic, you never want to talk about that topic ever again, which yeah, in hindsight makes sense. Why do you care about observability when you're here to look at cloud costs?Tim: Because cloud costs is another metric, just like you would use for performance, or resilience, or security. You do real-time monitoring to see if somebody has compromised the system, you do real-time monitoring to see if you have bad performance, if response times are too slow. You do real-time monitoring to know if something has gone down and then you need to make adjustments, or that the automated responses you have in response to that downtime are working. But cloud costs, you send somebody a report at the end of the month. Can you imagine, if you will—just for a second—if you got a downtime report at the end of month, and then you can react to something that has gone down?Or if you get a security report at the end of the month, and then you can react to the fact that somebody has your root keys? Or if you get [laugh] a report at the end of month, this said, “Hey, the CPU on this one was pegged. You should probably scale up.” That's outrageous to anybody in this industry right now. But why do we accept that for cloud cost?Corey: It's worse than that. There are a number of startups that talk about, “Oh, real-time cloud cost monitoring. Okay, the only way you're going to achieve such a thing is if you build an API shim that interprets everything that you're telling your cloud control plane to do, taking cost metrics out of it, and then passing it on to the actual cloud control plane.” Otherwise, you're talking about it showing up in the billing record in—ideally, eight hours; in practice, several days, or you're talking about the CloudTrail events, which is not holistic but gives you some rough idea, but it's also in some cases, 5 to 20 minutes delayed. There's no real-time way to do this without significant disruption to what's going on in your environment.So, when I hear about, “Oh, we do real-time bill analysis.” Yeah, it feels—to be very direct—you don't know enough about the problem space you're working within to speak intelligently about it because anyone who's played in this space for a while knows exactly how hard it is to get there. Now, I've talked to companies that have built real-time-ish systems that take that shim approach and acts sort of as a metadata sidecar ersatz billing system that tracks all of this so they can wind up intercepting potentially very expensive configuration mistakes. And that's great. That's also a bit beyond for a lot of folks today, but it's where the industry is going. But there is no way to get there today, short of effectively intercepting all of those calls, in a way that is cohesive and makes sense. How do you square that circle given the complete lack of effective tooling?Tim: Honestly, I'm going to point that right back at the cloud provider because they know how much you're spending, real-time. They know exactly how much you spend in real-time. They've figured it out. They have the buckets, they have APIs for it internally. I'm sure they do; it would make no sense for them not to. Without giving anything anyway, I know that when I was at AWS, I knew how much they were spending, almost real-time.Corey: That's impressive. I wish that existed. My never having worked at AWS perspective on it is that they, of course, have the raw data effective immediately, or damn close to it, but the challenge for the billing system is distilling and summarizing and attributing all of that in a reasonable timeframe; it is an exabyte-scale problem. I've talked to folks there who have indicated it is comfortably north of a petabyte in raw data per day. And that was a couple of years ago, so one can only imagine as the footprint has increased, so has all of this.I mean, the billing system is fundamentally magic from the outside. I'm not saying it's good magic, but it is magic, and it's something that is unappreciated, that every customer uses, and is one of those areas that doesn't get the attention it deserves. Because, let's be clear, here, we talk about observability; the bill is still the only thing that AWS offers that gives you a holistic overview of everything running in your account, in one place.Tim: What I think is interesting is that you talk about this, the scale of the problem and that it makes it difficult to solve. At the same time, I can have a conversation with my partner about kitty litter, and then all of a sudden, I'm going to start getting ads about kitty litter within minutes. So, I feel like it's possible to emit cost as a metric like you would CPU or disk. And if I'm going to look at who's going to do that, I'm going to look right back at AWS. The fun part about that, though, is I know from AWS's business model, that if that's something they were to emit, it would also cost you, like, 25 cents per call, and then you would actually, like, triple your cloud costs just trying to figure out how much it costs you.Corey: Only with 16 other billing dimensions because of course it would. And again, I'm talking about stuff, because of how I operate and how I think about this stuff, that is inherently corner case, or [vertex 00:31:39] case in many cases. But for the vast majority of folks, it's not the, “Oh, you have this really weird data transfer paradigm between these two resources,” which yeah, that's a problem that needs to be addressed in an awful lot of cases because data transfer pricing is bonkers, but instead it's the, “Huh. You just spun up a big cluster that's going to cost $20,000 a month.” You probably don't need to wait a full day to flag that.And you also can't put this on the customer in the sense of, “Oh, just set some budget alarms, that's great. That's the first thing you should do in a new AWS account.” “Well, jackhole, I've done an awful lot of first things I'm supposed to do in an AWS account, in my dedicated test account for these sorts of things. It's been four months, I'm not done yet with all of those first things I'm supposed to do.” It's incredibly secure, increasingly expensive, and so far all it runs is a single EC2 instance that is mostly there just so that everything else doesn't error out trying to divide by zero.Tim: There are some things that are built-in. If I stand up an EC2 instance and it goes down, I'm going to get an alert that this instance terminated for some reason. It's just going to show up informationally.Corey: In the console. You're not going to get called about it or paged about it, unless—Tim: Right.Corey: —you have something else in the business that will, like a boss that screams at you two o'clock in the morning. This is why we have very little that's production-facing here.Tim: But if I know that alert exists somewhere in the console, that's easy for me to write a trap for. That's easy for me to write, say hey, I'm going to respond to that because this call is going to come out somewhere; it's going to get emitted somewhere. I can now, as an engineer, write a very easy trap that says, “Hey, pop this in the Slack. Send an alert. Send a page.”So, if I could emit a cost metric, and I could say, “Wow. Somebody has spun up this thing that's going to cost X amount of money. Someone should get paged about this.” Because if they don't page about this and we wait eight hours, that's my month's salary. And you would do that if your database server went down; you would do that if someone rooted that database server; you would do that if the database server was [bogging 00:33:48] you to scale up another one. So, why can't you do that if that database server was all of sudden costing you way more than you had calculated?Corey: And there's a lot of nuance here because what you're talking about makes perfect sense for smaller-scale accounts, but even some of the very large accounts where we're talking hundreds of millions a year in spend, you can set compromised keys up on GitHub, put them in Payspin, whatever, and then people start spinning up Bitcoin miners everywhere. Great. It takes a long time to materially move the needle on that level of spend; it gets lost in the background noise. I lose my mind when I wind up leaving a managed NAT gateway running and it cost me 70 bucks a month in my $5 a month test account. Yeah, but you realize you could basically buy an island and it gets lost in the AWS bill at some of the high watermarks for some of these larger accounts.“Oh, someone spun up a cluster that's going to cost $400,000 a year?” Yeah, do I need to re-explain to you what a data science team does? They light money on fire in return for questionable returns, as a general rule. You knew that when you hired them; leave them alone. Whereas someone in their developer account does this, yeah, you kind of want to flag that immediately.It always comes down to rules and context. But I'd love to have some templates ready to go of, “I'm a starving student, please alert me anytime it looks like I might possibly exceed the free tier,” or better yet, “Don't let me, and if I do, it's on you and you eat the cost.” Conversely, it's, “Yeah, this is a Netflix sub-account or whatnot. Maybe don't bother me for anything whatsoever because freedom and responsibility is how we roll.” I imagine that's what they do internally on a lot of their cloud costing stuff because freedom and responsibility is ingrained in their culture. It's great. It's the freedom from having to think about cloud bills and the responsibility for paying it, of the cloud bill.Tim: Yeah, we will get internally alerted if things are [laugh] up too long, and then we will actually get paged, and then our manager would get paged, [laugh] and it would go up the line. If you leave something that's running too expensive, too long. So, there is a system there for it.Corey: Oh, yeah. The internal AWS systems for employees are probably my least favorite AWS service, full stop. And I've seen things posted about it; I believe it's called Isengard, for spinning up internal accounts and the rest—there's a separate one, I think, called Conduit, but I digress—that you spin something up, and apparently if it doesn't wind up—I don't need you to comment on this because you worked there and confidentiality is super important, but to my understanding it's, great, it has a whole bunch of formalized stuff like that and it solves for a whole lot of nifty features that bias for the way that AWS focuses on accounts and how they've view security and the rest. And, “Oh, well, we couldn't possibly ship this to customers because it's not how they operate.” And that's great.My problem with this internal provisioning system is it isolates and insulates AWS employees from the real pain of working with multiple accounts as a customer. You don't have to deal with the provisioning process of Control Tower or whatnot; you have your own internal thing. Eat your own dog food, gargle your own champagne, whatever it takes to wind up getting exposure to the pain that hits customers and suddenly you'll see those things improve. I find that the best way to improve a product is to make the people building it live with the painful parts.Tim: I think it's interesting that the stance is, “Well, it's not how the customers operate, and we wouldn't want the customers to have to deal with this.” But at the same time, you have to open up, like, 100 accounts if you need more than a certain number of S3 buckets. So, they are very comfortable with burdening the customer with a lot of constraints, and they say, “Well, constraints drive innovation.” Certainly, this is a constraint that you could at least offer and let the customers innovate around that.Corey: And at least define who the customer is. Because yeah, “I'm a Netflix sub-account is one story,” “I'm a regulated bank,” is another story, and, “I'm a student in my dorm room, trying to learn how this whole cloud thing works,” is another story. From risk tolerance, from a data protection story, from a billing surprise story, from a, “I'm trying to learn what the hell this is, and all these other service offerings you keep talking to me about confuse the hell out of me; please streamline the experience.” There's a whole universe of options and opportunity that isn't being addressed here.Tim: Well, I will say it very simply like this: we're talking about a multi-trillion dollar company versus someone who, if their AWS bill is too high, they don't pay rent; maybe they don't eat; maybe they have other issues, they don't—medical bill doesn't get paid; child care doesn't get paid. And if you're going to tell me that this multi-trillion dollar company can't solve for that so that doesn't happen to that person and tells them, “Well, if you come in afterwards, after your bill gets there, maybe we can do something about it, but in the meantime, suffer through this.” That's not ethical. Full stop.Corey: There are a lot of things that AWS gets right, and I want to be clear that I'm not sitting here trying to cast blame and say that everything they're doing is terrible. I feel like every time I talk about billing in any depth, I have to throw this disclaimer in. Ninety to ninety-five percent of what they do is awesome. It's just the missing piece that is incredibly painful for customers, and that's what I spend most of my time focusing on. It should not be interpreted to think that I hate the company.I just want them to do better than they are, and what they're doing now is pretty decent in most respects. I just want to fix the painful parts. Tim, thank you for joining me for a third time here. I'm certain I'll have you back in the somewhat near future to talk about more aspects of this, but until then, where can people find you slash retain your services?Tim: Well, you can find me on Twitter at @elchefe. If you want to retain my services for which you would be very, very happy to have, you can go to duckbillgroup.com and fill out a little questionnaire, and I will magically appear after an exchange of goods and services.Corey: Make sure to reference Tim by name just so that we can make our sales team facepalm because they know what's coming next. Tim, thank you so much for your time; it's appreciated.Tim: Thank you so much, Corey. I loved it.Corey: Principal cloud economist here at The Duckbill Group, Tim Banks. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, wait at least eight hours—possibly as many as 48 to 72—and then leave a comment explaining what you didn't like.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Mike Colleluori is a Non-Profit Executive and Entrepreneur. In 2006, after the tragic passing of his brother, Nick, following a battle with Lymphoma––Mike and his family established the HEADStrong Foundation (which was a concept Nick had created shortly before his death). Over the past fifteen years, the Colleluoris have continued Nick's legacy through HEADStrong's mission to offer financial, residential, and emotional support to families affected by cancer. The Non-Profit raises millions of dollars annually and impacts the lives of so many brave individuals fighting back against the disease. In addition to his work with HEADStrong, Mike (along with former NFL All-Pro Defensive End, Trent Cole) is also the Co-Founder of ProVantage Solutions––a  marketing, sponsorship, and business management agency for brands and athletes. For more information about The HEADStrong Foundation & how you can help, visit their website: https://headstrong.org/ ***TIMESTAMPS*** 6:50 - Mike tells a story about his former flow; PLL NLL, and the pro lacrosse leagues; Kids picking a main sport too early; How Mike got into lacrosse and got a scholarship to Hofstra 25:41 - The story behind Mike's brother Nick's battle against and untimely death from cancer; The medical costs of fighting cancer; Why Mike wonders if they're really trying to cure cancer; Chemo, Radiation, & CAR-T Therapy 44:24 - The HEADStrong Foundation and what they do for cancer patients; Mike talks about Nick's dying wish that the Colleluoris start HEADStrong; The stories behind why HEADStrong put a major focus on residential housing for cancer patients and their families 58:02 - That time Lance Armstrong & Livestrong tried to sue HEADStrong (lol); Lance Armstrong's fall from grace and why you have to use your platform for good; Mike tells the story of meeting the man who would later become the Chairman of The HEADStrong Board; The heartbreak of having to bury your child 1:12:39 - The struggle of working on behalf of the disease that killed someone you loved; Mike shares what Nick's last words to him were; “There isn't a manual that tells you how to deal with death”; Perspective & Why we need to “Be Nice!” 1:27:56 - The problems with many Non-Profits and misappropriated funds; Why Mike thinks “People should run a Non-Profit like a For-Profit and a For-Profit like a Non-Profit” 1:37:16 - Revisiting cancer research, whether or not the objective truly is “a cure,” and the capitalism incentives Big Pharma companies have to create to attract medical talent; Mike talks about how he tells the story of HEADStrong's patients; The importance of giving back 1:48:22 - Being present; Mike & Julian talk about their respective approaches to cell phone use; Kids and iPhones 2:01:14 - Mike talks about balancing HEADStrong with his company, ProVantage Solutions; Mike tells the story of how he and former NFL All-Pro, Trent Cole, became business partners; The deep conversations Mike and Trent have 2:10:43 - How bad people often take advantage of Professional Athletes; Athletes finding purpose in their second career after sports; A story about Allen Iverson and luggage; The value of having a camera on you while you're an active player; The massive competition for attention among celebrities compared to a decade ago; Pro Athletes and their various approaches to charity 2:45:52 - Pro Athlete Activism; Mike talks about a business associate whose software automatically reads for all risk-related social media posts from a person's entire life ~ YouTube EPISODES & CLIPS: https://www.youtube.com/channel/UC0A-v_DL-h76F75xik8h03Q  ~ Get $100 Off The Eight Sleep Pod Pro Mattress / Mattress Cover: https://eight-sleep.ioym.net/trendifier  Julian's Instagram: https://www.instagram.com/julianddorey  ~ Beat provided by: https://freebeats.io  Music Produced by White Hot

Mike Julian has been a great friend and taught me a LOT over the past six years or so! Shortly after I kicked it off the Optimising Nutrition Facebook Group, Mike was making so many insightful contributions I asked him to become an admin. Back in the day, we had a very unique group of people who would share their research and insights - Tommy Wood, Robb Wolf, Raymund Edwards, Luis Villasenor and a cast of people who were eager to learn more about nutrition and metabolism. It made for a unique learning environment. It was a great hive mind of sharing of ideas. We all learned a LOT and were able to see things in a fairly impartial manner. You may or may not have heard of Mike Julian. But I think of him as like the Illuminati pulling the strings behind many of the biggest names in nutrition including Robb Wolf, Ted Naiman, Ketogains & Craig Emmerich's recent book Keto. I have learned so much about how human metabolism really works, and how to improve it, from Mike who has been a pioneer in digesting the research, joining the dots and sharing it freely. In this chat, we discussed many of the key concepts in Big Fat Keto Lies, including: - What are the biggest misunderstandings and misconceptions in nutrition and metabolism at the moment? - Why is the Personal Fat Threshold central to our understanding of diabetes and metabolic health? - What is ketosis? How does it work? What do you think is the biggest misunderstanding around ketosis? - Do fat calories matter? Why is there so much confusion? - How are blood glucose and blood lipids interrelated? - What do people really need to know about lipidology? - What are the limitations of high protein % or MORE protein? How do you know when you've had too much? - What are your hopes for the future in the realm of nutrition and metabolism? Links Optimising Nutrition Data-Driven Fasting Nutrient Optimiser Optimising Nutrition Podcast Follow Facebook YouTube Twitter Instagram Pinterest Products Big Fat Keto Lies (Book) NutriBooster Recipe Books Data-Driven Fasting 30 Day Challenge Nutritional Optimisation Masterclass Groups NutriBooster Recipes Optimising Nutrition Data-Driven Fasting Nutrient Optimiser

About MikeBeside his duties as The Duckbill Group's CEO, Mike is the author of O'Reilly's Practical Monitoring, and previously wrote the Monitoring Weekly newsletter and hosted the Real World DevOps podcast. He was previously a DevOps Engineer for companies such as Taos Consulting, Peak Hosting, Oak Ridge National Laboratory, and many more. Mike is originally from Knoxville, TN (Go Vols!) and currently resides in Portland, OR.Links: Software Engineering Daily podcast: https://softwareengineeringdaily.com/category/all-episodes/exclusive-content/Podcast/ Duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself; make one of them go away. To learn more, visit lumigo.io.Corey: This episode is sponsored in part by ChaosSearch. As basically everyone knows, trying to do log analytics at scale with an ELK stack is expensive, unstable, time-sucking, demeaning, and just basically all-around horrible. So why are you still doing it—or even thinking about it—when there's ChaosSearch? ChaosSearch is a fully managed scalable log analysis service that lets you add new workloads in minutes, and easily retain weeks, months, or years of data. With ChaosSearch you store, connect, and analyze and you're done. The data lives and stays within your S3 buckets, which means no managing servers, no data movement, and you can save up to 80 percent versus running an ELK stack the old-fashioned way. It's why companies like Equifax, HubSpot, Klarna, Alert Logic, and many more have all turned to ChaosSearch. So if you're tired of your ELK stacks falling over before it suffers, or of having your log analytics data retention squeezed by the cost, then try ChaosSearch today and tell them I sent you. To learn more, visit chaossearch.io.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I spent the past week guest hosting the Software Engineering Daily podcast, taking listeners over there on a tour of the clouds. Each day, I picked a different cloud and had a guest talk to me about their experiences with that cloud.Now, there was one that we didn't talk about, and we're finishing up that tour here today on Screaming in the Cloud. That cloud is the obvious one, and that is your own crappy data center. And my guest is Duckbill Group's CEO and my business partner, Mike Julian. Mike, thanks for joining me.Mike: Hi, Corey. Thanks for having me back.Corey: So, I frequently say that I started my career as a grumpy Unix sysadmin. Because it isn't like there's a second kind of Unix sysadmin you're going to see. And you were in that same boat. You and I both have extensive experience working in data centers. And it's easy sitting here on the tech coast of the United States—we're each in tech hubs cities—and we look around and yeah, the customers we talked to have massive cloud presences; everything we do is in cloud, it's easy to fall into the trap of believing that data centers are a thing of yesteryear. Are they?Mike: [laugh]. Absolutely not. I mean, our own customers have tons of stuff in data centers. There are still companies out there like Equinix, and CoreSite, and DRC—is that them? I forget the name of them.Corey: DRT. Digital Realty [unintelligible 00:01:54].Mike: Digital Realty. Yeah. These are companies still making money hand over fist. People are still putting new workloads into data centers, so yeah, we're kind of stuck with him for a while.Corey: What's fun is when I talked to my friends over in the data center sales part of the world, I have to admit, I went into those conversations early on with more than my own fair share of arrogance. And it was, “[laugh]. So, who are you selling to these days?” And the answer was, “Everyone, fool.” Because they are.People at large companies with existing data center footprints are not generally doing fire sales of their data centers, and one thing that we learned about cloud bills here at The Duckbill Group is that they only ever tend to go up with time. That's going to be the case when we start talking about data centers as well. The difference there is that it's not just an API call away to lease more space, put in some racks, buy some servers, get them racked. So, my question for you is, if we sit here and do the Hacker News—also known as the worst website on the internet—and take their first principles approach to everything, does that mean the people who are building out data centers are somehow doing it wrong? Did they miss a transformation somewhere?Mike: No, I don't think they're doing it wrong. I think there's still a lot of value in having data centers and having that sort of skill set. I do think the future is in cloud infrastructure, though. And whether that's a public cloud, or private cloud, or something like that, I think we're getting increasingly away from building on top of bare metal, just because it's so inefficient to do. So yeah, I think at some point—and I feel like we've been saying this for years that, “Oh, no, everyone's missed the boat,” and here we are saying it yet again, like, “Oh, no. Everyone's missing the boat.” You know, at some point, the boat's going to frickin' leave.Corey: From my perspective, there are advantages to data centers. And we can go through those to some degree, but let's start at the beginning. Origin stories are always useful. What's your experience working in data centers?Mike: [laugh]. Oh, boy. Most of my career has been in data centers. And in fact, one interesting tidbit is that, despite running a company that is built on AWS consulting, I didn't start using AWS myself until 2015. So, as of this recording, it's 2021 now, so that means six years ago is when I first started AWS.And before that, it was all in data centers. So, some of my most interesting stuff in the data center world was from Oak Ridge National Lab where we had hundreds of thousands of square feet of data center floor space across, like, three floors. And it was insane, just the amount of data center stuff going on there. A whole bunch of HPC, a whole bunch of just random racks of bullshit. So, it's pretty interesting stuff.I think probably the most really interesting bit I've worked on was when I was at a now-defunct company, Peak Hosting, where we had to figure out how to spin up a data center without having anyone at the data center, as in, there was no one there to do the spin up. And that led into interesting problems, like you have multiple racks of equipment, like, thousands of servers just showed up on the loading dock. Someone's got to rack them, but from that point, it all has to be automatic. So, how do you bootstrap entire racks of systems from nothing with no one physically there to start a bootstrap process? And that led us to build some just truly horrific stuff. And thank God that's someone else's problem, now. [laugh].Corey: It makes you wonder if under the hood at all these cloud providers if they have something that's a lot cleaner, and more efficient, and perfect, or if it's a whole bunch of Perl tied together with bash and hope, like we always built.Mike: You know what? I have to imagine that even at AWS at a—I know if this is true at Facebook, where they have a massive data center footprint as well—there is a lot of work that goes into the bootstrap process, and a lot of these companies are building their own hardware to facilitate making that bootstrap process easier. When you're trying to bootstrap, say, like, Dell or HP servers, the management cards only take you so far. And a lot of the stuff that we had to do was working around bugs in the HP management cards, or the Dell DRACs.Corey: Or you can wind up going with some budget whitebox service. I mean, Supermicro is popular, not that they're ultra-low budget. But yeah, you can effectively build your own. And that leads down interesting paths, too. I feel like there's a sweet spot where working on a data center and doing a build-out makes sense for certain companies.If you're trying to build out some proof of concept, yeah, do it in the cloud; you don't have to wait eight weeks and spend thousands of dollars; you can prove it out right now and spend a total of something like 17 cents to figure out if it's going to work or not. And if it does, then proceed from there, if not shut it down, and here's a quarter; keep the change. With data centers, a lot more planning winds up being involved. And is there a cutover at which point it makes sense to evacuate from a public cloud into a physical data center?Mike: You know, I don't really think so. This came up on a recent Twitter Spaces that you and I did around, at what point does it really make sense to be hybrid, or to be all-in on data center? I made the argument that a large-scale HPC does not fit cloud workloads, and someone made a comment that, like, “What is large-scale?” And to me, large-scale was always, like—so Oak Ridge was—or is famous—for having supercomputing, and they have largely been in the top five supercomputers in the world for quite some time. A supercomputer of that size is tens of thousands of cores. And they're running pretty much constant because of how expensive that stuff is to get time on. And that sort of thing would be just astronomically expensive in a cloud. But how many of those are there really?Corey: Yeah, if you're an AWS account manager listening to this and reaching out with, “No, that's not true. After committed spend, we'll wind up giving you significant discounts, and a whole bunch of credits, and jump through all these hoops.” And, yeah, I know, you'll give me a bunch of short-term contractual stuff that's bounded for a number of years, but there's no guarantee that stuff gets renewed at that rate. And let's face it. If you're running those kinds of workloads today, and already have the staff and tooling and processes that embrace that, maybe ripping all that out in a cloud migration where there's no clear business value derived isn't the best plan.Mike: Right. So, while there is a lot of large-scale HPC infrastructure that I don't think particularly fits well on the cloud, there's not a lot of that. There's just not that many massive HPC deployments out there. Which means that pretty much everything below that threshold could be a candidate for cloud workloads, and probably would be much better. One of the things that I noticed at Oak Ridge was that we had a whole bunch of SGI HPC systems laying around, and 90% of the time they were idle.And those things were not cheap when they were bought, and at the time, they're basically worth nothing. But they were idle most of the time, but when they were needed, they're there, and they do a great job of it. With AWS and GCP and Azure HPC offerings, that's a pretty good fit. Just migrate that whole thing over because it'll cost you less than buying a new one. But if I'm going to migrate Titan or Gaia from Oak Ridge over to there, yeah, some AWS rep is about to have a very nice field day. That'd just be too much money.Corey: Well, I'd be remiss as a cloud economist if I didn't point out that you can do this stuff super efficiently in someone else's AWS account.Mike: [laugh]. Yes.Corey: There's also the staffing question where if you're a large blue-chip company, you've been around for enough decades that you tend to have some revenue to risk, where you have existing processes and everything is existing in an on-prem environment, as much as we love to tell stories about the cloud being awesome, and the capability increase and the rest, yadda, yadda, yadda, there has to be a business case behind moving to the cloud, and it will knock some nebulous percentage off of your TCO—because lies, damned lies, and TCO analyses are sort of the way of the world—great. That's not exciting to most strategic-level execs. At least as I see the world. Given you are one of those strategic level execs, do you agree? Am I lacking nuance here?Mike: No, I pretty much agree. Doing a data center migration, you got to have a reason to do it. We have a lot of clients that are still running in data centers as well, and they don't move because the math doesn't make sense. And even when you start factoring in all the gains from productivity that they might get—and I stress the word might here—even when you factor those in, even when you factor in all the support and credits that Amazon might give them, it still doesn't make enough sense. So, they're still in data centers because that's where they should be for the time because that's what the finances say. And I'm kind of hard-pressed to disagree with them.Corey: While we're here playing ‘ask an exec,' I'm going to go for another one here. It's my belief that any cloud provider that charges a penny for professional services, or managed services, or any form of migration tooling or offering at all to their customers is missing the plot. Clearly, since they all tend to do this, I'm wrong somewhere. But I don't see how am I wrong or are they?Mike: Yeah, I don't know. I'd have to think about that one some more.Corey: It's an interesting point because it's—Mike: It is.Corey: —it's easy to think of this as, “Oh, yeah. You should absolutely pay people to migrate in because the whole point of cloud is that it's kind of sticky.” The biggest indicator of a big cloud bill this month is a slightly smaller one last month. And once people wind up migrating into a cloud, they tend not to leave despite all of their protestations to the contrary about multi-cloud, hybrid, et cetera, et cetera. And that becomes an interesting problem.It becomes an area—there's a whole bunch of vendors that are very deeply niched into that. It's clear that the industry as a whole thinks that migrating from data centers to cloud is going to be a boom industry for the next three decades. I don't think they're wrong.Mike: Yeah, I don't think they're wrong either. I think there's a very long tail of companies with massive footprint staying in a data center that at some point is going to get out of a data center.Corey: For those listeners who are fortunate enough not to have to come up the way that we did. Can you describe what a data center is like inside?Mike: Oh, God.Corey: What is a data center? People have these mythic ideas from television and movies, and I don't know, maybe some Backstreet Boys music video; I don't know where it all comes from. What is a data center like? What does it do?Mike: I've been in many of these over my life, and I think they really fall into two groups. One is the one managed by a professional data center manager. And those tend to be sterile environments. Like, that's the best way to describe it. They are white, filled with black racks. Everything is absolutely immaculate. There is no trash or other debris on the floor. Everything is just perfect. And it is freezingly cold.Corey: Oh, yeah. So, you're in a data center for any length of time, bring a jacket. And the soulless part of it, too, is that it's well-lit with fluorescent lights everywhere—Mike: Oh yeah.Corey: —and it's never blinking, never changing. There are no windows. Time loses all meaning. And it's strange to think about this because you don't walk in and think, “What is that racket?” But there's 10,000, 100,000 however many fans spinning all the time. It is super loud. It can clear 120 decibels in there, but it's a white noise so you don't necessarily hear it. Hearing protection is important there.Mike: When I was at Oak Ridge, we had—all of our data centers, we had a professional data center manager, so everything was absolutely pristine. And to get into any of the data centers, you had to go through a training; it was very simple training, but just, like, “These are things you do and don't do in the data center.” And when you walked in, you had to put in earplugs immediately before you walked in the door. And it's so loud just because of that, and you don't really notice it because you can walk in without earplugs and, like, “Oh, it's loud, but it's fine.” And then you leave a couple hours later and your ears are ringing. So, it's a weird experience.Corey: It's awful. I started wearing earplugs every time I went in, just because it's not just the pain because hearing loss doesn't always manifest that way. It's, I would get tired much more quickly.Mike: Oh, yeah.Corey: I would not be as sharp. It was, “What is this? Why am I so fatigued?” It's noise.Mike: Yeah. And having to remember to grab your jacket when you head down to the data center, even though it's 95 degrees outside.Corey: At some point, if you're there enough—which you probably shouldn't be—you start looking at ways to wind up storing one locally. I feel like there could be some company that makes an absolute killing by renting out parkas at data centers.Mike: Yeah, totally. The other group of data center stuff that I generally run into is the exact opposite of that. And it's basically someone has shoved a couple racks in somewhere and they just kind of hope for the best.Corey: The basement. The closet. The hold of a boat, with one particular client we work with.Mike: Yeah. That was an interesting one. So, we had a—Corey and I had a client where they had all their infrastructure in the basement of a boat. And we're [laugh] not even kidding. It's literally in the basement of a boat.Corey: Below the waterline.Mike: Yeah below the waterline. So, there was a lot of planning around, like, what if the hold gets breached? And like, who has to plan for that sort of thing? [laugh]. It was a weird experience.Corey: It turns out that was—was hilarious about that was while they were doing their cloud migration into AWS, their account manager wasn't the most senior account manager because, at that point, it was a small account, but they still stuck to their standard talking points about TCO, and better durability, and the rest, and it didn't really occur to them to come back with a, what if the boat sinks? Which is the obvious reason to move out of that quote-unquote, “data center?”Mike: Yeah. It was a wild experience. So, that latter group of just everything's an absolute wreck, like, everything—it's just so much of a pain to work with, and you find yourself wanting to clean it up. Like, install new racks, do new cabling, put in a totally new floor so you're not standing on concrete. You want to do all this work to it, and then you realize that you're just putting lipstick on a pig; it's still going to be a dirty old data center at the end of the day, no matter how much work you do to it. And you're still running on the same crappy hardware you had, you're still running on the same frustrating deployment process you've been working on, and everything still sucks, despite it looking good.Corey: This episode is sponsored in part by ChaosSearch. As basically everyone knows, trying to do log analytics at scale with an ELK stack is expensive, unstable, time-sucking, demeaning, and just basically all-around horrible. So why are you still doing it—or even thinking about it—when there's ChaosSearch? ChaosSearch is a fully managed scalable log analysis service that lets you add new workloads in minutes, and easily retain weeks, months, or years of data. With ChaosSearch you store, connect, and analyze and you're done. The data lives and stays within your S3 buckets, which means no managing servers, no data movement, and you can save up to 80 percent versus running an ELK stack the old-fashioned way. It's why companies like Equifax, HubSpot, Klarna, Alert Logic, and many more have all turned to ChaosSearch. So if you're tired of your ELK stacks falling over before it suffers, or of having your log analytics data retention squeezed by the cost, then try ChaosSearch today and tell them I sent you. To learn more, visit chaossearch.io.Corey: The worst part is playing the ‘what is different here?' Game. You rack twelve servers: eleven come up fine and the twelfth doesn't.Mike: [laugh].Corey: It sounds like, okay, how hard could it be? Days. It can take days. In a cloud environment, you have one weird instance. Cool, you terminate it and start a new one and life goes on whereas, in a data center, you generally can't send back a $5,000 piece of hardware willy nilly, and you certainly can't do it same-day, so let's figure out what the problem is.Is that some sub-component in the system? Is it a dodgy cable? Is it, potentially, a dodgy switch port? Is there something going on with that node? Was there something weird about the way the install was done if you reimage the thing? Et cetera, et cetera. And it leads down rabbit holes super quickly.Mike: People that grew up in the era of computing that Corey and I did, you start learning tips and tricks, and they sound kind of silly these days, but things like, you never create your own cables. Even though both of us still remember how to wire a Cat 5 cable, we don't.Corey: My fingers started throbbing when you said that because some memories never fade.Mike: Right. You don't. Like, if you're working in a data center, you're buying premade cables because they've been tested professionally by high-end machines.Corey: And you still don't trust it. You have a relatively inexpensive cable tester in the data center, and when—I learned this when I was racking stuff the second time, it adds a bit of time, but every cable that we took out of the packaging before we plugged it in, and we tested on the cable tester just to remove that problem. And it still doesn't catch everything because, welcome to the world of intermittent cables that are marginal that, when you bend a certain way, stop working, and then when you look at them, start working again properly. Yes, it's as maddening as it sounds.Mike: Yeah. And then things like rack nuts. My fingers hurt just thinking about it.Corey: Think of them as nuts that bolts wind up screwing into but they're square and they have clips on them so they clip into the standard rack cabinets, so you can screw equipment into them. There are different sizes of them, and of course, they're not compatible with one another. And you have—they always pinch your finger and make you bleed because they're incredibly annoying to put in and out. Some vendors have quick rails, which are way nicer, but networking equipment is still stuck in the ‘90s in that context, and there's always something that winds up causing problems.Mike: If you were particularly lucky, the rack nuts that you had were pliable enough that you could pinch them and pull them out with your fingers, and hopefully didn't do too much damage. If you were particularly unlucky, you had to reach for a screwdriver to try to pry it out, and inevitably stab yourself.Corey: Or sometimes pulling it out with your fingers, it'll—like, those edges are sharp. It's not the most high-quality steel in some cases, and it's just you wind up having these problems. Oh, one other thing you learn super quickly, is first, always have a set of tools there because the one you need is the one you don't have, and the most valuable tool you'll have is a pair of wire cutters. And what you do when you find a bad cable is you cut it before throwing it away.Mike: Yep.Corey: Because otherwise someone who is very well-meaning but you will think of them as the freaking devil, will, “Oh, there's a perfectly good cable sitting here in the trash. I'll put it back with the spares.” So you think you have a failed cable you grab another one from the pile of spares—remember, this is two in the morning, invariably, and you're not thinking on all cylinders—and the problem is still there. Cut the cable when you throw it away.Mike: So, there are entire books that were written about these sorts of tips and tricks that everyone working [with 00:19:34] data center just remembers. They learned it all. And most of the stuff is completely moot now. Like, no one really thinks about it anymore. Some people are brought up in computing in such a way that they never even learned these things, which I think it's fantastic.Corey: Oh, I don't wish this on anyone. This used to be a prerequisite skill for anyone who called themselves a systems administrator, but I am astonished when I talk to my AWS friends, the remarkably senior engineers I talk to who have never been inside of an AWS data center.Mike: Yeah, absolutely.Corey: That's really cool. It also means you're completely divorced from the thing you're doing with code and the rest, and the thing that winds up keeping the hardware going. It also leads to a bit of a dichotomy where the people racking the hardware, in many cases, don't understand the workloads that are on there because if you have the programming insight, and ability, and can make those applications work effectively, you're probably going to go find a role that compensates far better than working in the data center.Mike: I [laugh] want to talk about supply chains. So, when you build a data center, you start planning about—let's say, I'm not Amazon. I'm just, like, any random company—and I want to put my stuff into a data center. If I'm going to lease someone else's data center—which you absolutely should—we're looking at about a 180-day lead time. And it's like, why? Like, that's a long time. What's—Corey: It takes that long to sign a real estate lease?Mike: Yeah.Corey: No. It takes that long to sign a real estate lease, wind up talking to your upstream provider, getting them to go ahead and run the thing—effectively—getting the hardware ordered and shipped in the right time window, doing the actual build-out once everything is in place, and I'm sure a few other things I'm missing.Mike: Yeah, absolutely. So yeah, you have all these things that have to happen, and all of them pay for-freaking-ever. Getting Windstream on the phone to begin with, to even take your call, can often take weeks at a time. And then to get them to actually put an order for you, and then do the turnup. The turnup alone might be 90 days, where I'm just, “Hey, I've bought bandwidth from you, and I just need you to come out and connect the [BLEEP] cables,” might be 90 days for them to do it.And that's ridiculous. But then you also have the hardware vendors. If you're ordering hardware from Dell, and you're like, “Hey, I need a couple servers.” Like, “Great. They'll be there next week.” Instead, if you're saying, “Hey, I need 500 servers,” they're like, “Ooh, uh, next year, maybe.” And this is even pre-pandemic sort of thing because they don't have all these sitting around.So, for you to get a large number of servers quickly, it's just not a thing that's possible. So, a lot of companies would have to buy well ahead of what they thought their needs would be, so they'd have massive amounts of unused capacity. Just racks upon racks of systems sitting there turned off, waiting for when they're needed, just because of the ordering lead time.Corey: That's what auto-scaling looks like in those environments because you need to have that stuff ready to go. If you have a sudden inrush of demand, you have to be able to scale up with things that are already racked, provisioned, and good to go. Sometimes you can have them halfway provisioned because you don't know what kind of system they're going to need to be in many cases, but that's some up-the-stack level thinking. And again, finding failed hard drives and swapping those out, make sure you pull the right or you just destroyed an array. And all these things that I just make Amazon's problem.It's kind of fun to look back at this and realize that we would get annoyed then with support tickets that took three weeks to get resolved in hardware, whereas now three hours in you and I are complaining about the slow responsiveness of the cloud vendor.Mike: Yeah, the amount of quick turnaround that we can have these days on cloud infrastructure that was just unthinkable, running in data centers. We don't run out of bandwidth now. Like, that's just not a concern that anyone has. But when you're running in a data center, and, “Oh, yeah. I've got an OC-3 line connected here. That's only going to get me”—Corey: Which is something like—what is an OC-3? That's something like, what, 20 gigabit, or—Mike: Yeah, something like that. It's—Corey: Don't quote me on that.Mike: Yeah. So, we're going to have to look that up. So, it's equivalent to a T-3, so I think that's a 45 megabit?Corey: Yeah, that sounds about reasonable, yeah.Mike: So, you've got a T-3 line sitting here in your data center. Like that's not terrible. And if you start maxing that out, well, you're maxed out. You need more? Again, we're back to the 90 to 180 day lead time to get new bandwidth.So, sucks to be you, which means you'd have to start planning your bandwidth ahead of time. And this is why we had issues like companies getting Slashdotted back in the day because when you capped the bandwidth out, well, you're capped out. That's it. That's the game.Corey: Now, you've made the front page of Slashdot, a bunch of people visited your site, and the site fell over. That was sort of the way of the world. CDNs weren't really a thing. Cloud wasn't a thing. And that was just, okay, you'd bookmark the thing and try and remember to check it later.We talked about bandwidth constraints. One thing that I think the cloud providers do—at least the tier ones—that are just basically magic is full line rate between any two instances almost always. Well, remember, you have a bunch of different racks, and at the top of every rack, there's usually a switch called—because we're bad at naming things—top-of-rack switches. And just because everything that you have plugged in can get one gigabit to that switch—or 10 gigabit or whatever it happens to be—there is a constraint in that top-of-rack switch. So yeah, one server can talk to another one in a different rack at one gigabit, but then you have 20 different servers in each rack all trying to do something like that and you start hitting constraints.You do not see that in the public cloud environments; it is subsumed away, you don't have to think about that level of nonsense. You just complain about what feels like the egregious data transfer charge.Mike: Right. Yeah. It was always frustrating when you had to order nice high-end switching gear from Cisco, or Arista, or take your pick of provider, and you got 48 ports in the top-of-rack, you got 48 servers all wired up to them—or 24 because we want redundancy on that—and that should be a gigabit for each connection, except when you start maxing it out, no, it's nowhere even near that because the switch can't handle it. And it's absolutely magical, that the cloud provider's like, “Oh, yeah. Of course, we handle that.”Corey: And you don't have to think about it at all. One other use case that I did want to hit because I know we'll get letters if we don't, where it does make sense to build out a data center, even today, is if you have regulatory requirements around data residency. And there's no cloud vendor in an area that suits. This generally does not apply to the United States, but there are a lot of countries that have data residency laws that do not yet have a cloud provider of their choice region, located in-country.Mike: Yeah, I'll agree with that, but I think that's a short-lived problem.Corey: In the fullness of time, there'll be regions everywhere. Every build—a chicken in every pot and an AWS availability zone on every corner.Mike: [laugh]. Yeah, I think it's going to be a fairly short-lived problem, which actually reminds me of even our clients that have data centers are often treating the data center as a cloud. So, a lot of them are using your favorite technology, Corey, Kubernetes, and they're treating Kubernetes as a cloud, running Kube in AWS, as well, and moving workloads between the two Kube clusters. And to them, a data center is actually not really data center; it's just a private cloud. I think that pattern works really well if you have a need to have a physical data center.Corey: And then they start doing a hybrid environment where they start expanding to a public cloud, but then they treat that cloud like just a place to run a bunch of VMs, which is expensive, and it solves a whole host of problems that we've already talked about. Like, we're bad at replacing hard drives, or our data center is located on a corner where people love to get drunk on the weekends and smash into the power pole and take out half of the racks here. Things like that great, yeah, cloud can solve that, but cloud could do a lot more. You're effectively worsening your cloud experience to improve your data center experience.Mike: Right. So, even when you have that approach, the piece of feedback that we give the client was, you have built such a thing where you have to cater to the lowest common denominator, which is the constraints that you have in the data center, which means you're not able to use AWS the way that you should be able to use it so it's just as expensive to run as a data center was. If they were to get rid of the data center, then the cloud would actually become cheaper for them and they would get more benefits from using it. So, that's kind of a business decision for how they've structured it, and I can't really fault them for it, but there are definitely some downsides to the approach.Corey: Mike, thank you so much for joining me here. If people want to learn more about what you're up to, where can they find you?Mike: You know, you can find me at duckbillgroup.com, and actually, you can also find Corey at duckbillgroup.com. We help companies lower their AWS bills. So, if you have a horrifying bill, you should chat.Corey: Mike, thank you so much for taking the time to join me here.Mike: Thanks for having me.Corey: Mike Julian, CEO of The Duckbill Group and my business partner. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and then challenge me to a cable-making competition.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Jesse Trucks is the Minister of Magic at Splunk, where he consults on security and compliance program designs and develops Splunk architectures for security use cases, among other things. He brings more than 20 years of experience in tech to this role, having previously worked as director of security and compliance at Peak Hosting, a staff member at freenode, a cybersecurity engineer at Oak Ridge National Laboratory, and a systems engineer at D.E. Shaw Research, among several other positions. Of course, Jesse is also the host of Meanwhile in Security, the podcast about better cloud security you're about to listen to.TranscriptJesse: Welcome to Meanwhile in Security where I, your host Jesse Trucks, guides you to better security in the cloud.Announcer: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the cloud: low effort, high visibility, and detection. To learn more, visit lacework.com. That's lacework.com.My recent experience prepping a commercial space for a state fire marshal office inspection and approval has me thinking about compliance and security and ever-present ‘temporary' fix for things. How many times have we said, “Oh, I'll just do this quick fix to get us by,” and that quick fix becomes the de facto supported production implementation? Repeat after me: all changes are permanent until replaced. All changes are permanent until replaced.Anything we alter at all, whether it in computing or in real life, is a permanent alteration until it is replaced by a new alteration, or by a natural corrective or evolutionary process, like decay. We cut our hair and it grows back. We weed our gardens and the weeds return. If you don't want temporary changes happening in your environment, then implement hard controls that will correct any aberrations that come up. Cloud-native architectures give us the tools to force this by making it seamless to close down and erased from existence anything that veers from your ideal. Take advantage of this now.Meanwhile, in the news. Password reset code brute force vulnerability in AWS Cognito. If you use this AWS service, you should read this one. Although it is now patched, it's good to understand how AWS Cognito works more closely, which is true for any other security service you rely upon that is hosted by your cloud provider or other vendor.Task force seeks to disrupt a ransomware payment. This is tangentially related to cloud security because both Amazon and Microsoft has joined up on this one, but I'm personally fascinated by strange frenemy combinations who work together on these things. I'm watching for either interesting things to happen with their recommendations that could have an impact on disclosure of ransomware incidents, or for it all to fizzle out to do nothing.Is your cloud raining sensitive data? Kubernetes generally needs securing like any other service. Time to stop ignoring your newest infrastructure and lock Kubernetes down. However, if you want real security for your Kubernetes clusters, you should look at a robust solution like Fairwinds Insights. I'm a big fan of outsourcing tool development to experts.Enterprise lift and shift to the public cloud requires a newer type of API and cloud security program to prevent data breaches. Ignoring some glaring editing mistakes, which is rather difficult for me to do, I'd like this easy-to-read case study of a traditional on-prem infrastructure going through a lift-and-shift cloud migration. This piece specifically addresses some of the serious security implications of doing this, and how your attack surface changes dramatically in the process.NOAA shifts some key environmental data processing to the cloud. This one is important to me personally. Years ago, when I was a security engineer for the United States Department of Energy Oak Ridge National Laboratory High-Performance Computing Group—boy, that's a mouthful—I helped ensure security for one of the National Oceanic and Atmospheric Administration—or NOAA—supercomputers doing climate research. NOAA moving any of its compute systems supporting global research is a very big deal, and this is a great example of why AWS GovCloud is helping the US federal government modernize and move to the cloud. Also, mixing an acronym-heavy industry with government work turns into a pile of TLS so fast. Also, as another aside, this was back when I met The Duckbill Group CEO, Mike Julian, in Knoxville, Tennessee.Announcer: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn't translate well to cloud or multi-cloud environments, and that's not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial. That's extrahop.com/trial.ClearDATA expands flagship solution to facilitate health care's adoption of containers and serverless tech. Speaking of outsourcing to experts, there are lots of compliance reporting options out there, and like my favorite, Qmulos. Full disclosure, remember I do work for Splunk. But there are less options for actively managing compliance in your cloud environment. Does anyone have experience with ClearDATA's Comply offering? Email me, I want to know more.Expanding security, visibility, and automation across AWS environments. I'm most interested in the AWS Graviton to ARM-based security in the asset discovery for AWS environments announcements in this piece. First, I love me some chip geekery, especially when security-related, and second, the thing most of us suck at is tracking your assets. Any help managing an asset list for our security tools is gravy.As Microsoft nears a $2 trillion market cap, Amazon is most likely to reach that level next. I'm always looking at economics and how that drives both behavior and technology. Also, looking at how markets move and companies grow and die tells us more about trends in technology decisions and spend than many other indicators. Stop and think about the implications of this: four of the world's five largest companies by market capitalization are us tech giants. Three of these are the parent companies of the three cloud giants: Microsoft, Amazon, and Alphabet or Google. It's a cloudy forecast for sure.Seven modern-day cybersecurity realities. None of these are earth-shattering news, but at least some of these will make you cringe when you consider your own environment. Feeling uncomfortable thinking about any of these is a good thing if you act on that feeling. Go forth and fix things.The challenge of securing non-people identities. Most of us wearily monitor people's account activity to ensure they aren't compromised. But the art and science behind monitoring accounts not tied to a person is more difficult to master. I argue some of the recent big security breaches shine light on these accounts being more critical to risk mitigation than human-used accounts.And now for the tip of the week. Turn off instances or containers or cloud services you aren't using. We turn off unused services on a system, right? Not using Postgres or MySQL? Shut it down. Not using the webserver? Shut it down.Leaving something answering on the network that isn't being actively used, or worse, not actively monitored, is an attack vector that can be easily leveraged by malware and bad actors. This is true for whole systems or cloud services that aren't actively part of your functional environment. If you aren't using your testing system, it should not be running at all. Leaving unused whole systems is far worse than leaving an extra service running because an intruder now has free reign over a whole machine that isn't in the spotlight, not just a corner of a well-used system. Given you can programmatically turn whole servers or containers on and off, there's no excuse for leaving them up when not in use. Turn those systems off. When in doubt, close the route.And that's a wrap for the week. This is Meanwhile in Security. Securely yours, Jesse Trucks.Jesse: Thanks for listening. Please subscribe and rate us on Apple and Google Podcast, Spotify, or wherever you listen to podcasts.Announcer: This has been a HumblePod production. Stay humble.

Mike Spear is a trial attorney and conservative scholar. Currently, he is an active member of the Ancient Order of Hibernians and a strong supporter of the Fraternal Order of Police. Previously, he played Division-1 College Football. ***TIMESTAMPS***   7:13 - Meat’s a father; The Trump Koi Pond Incident; Spear’s trip with Julian to NYC back in 2016; Why was Ben Carson in charge of HUD; Reviewing Bill Barr’s tenure as Attorney General 24:33 - The Jeffrey Epstein Conspiracy; Epstein’s ties to Bill Barr; Epstein’s ties to Bill Clinton; Ghislaine Maxwell, her father Robert Maxwell, and the Mossad Agent / Asset Conspiracy Theory 40:17 - Whether or not Presidents have gotten the real intel on Epstein; Obama & Drones; Who was Epstein? Epstein, blackmail, & His Infamous “Bill Clinton in a Dress” Painting 56:31 - Barack Obama & his speech against “Woke Culture”; Michelle Obama’s call to have Donald Trump banned from social platforms 1:12:19 - Providing context to the Black Unemployment statistics from both Obama & Trump’s Presidency; Trump’s actions on mass incarceration; Killer Mike, Louis Farrakhan, & Candace Owens; The 13th (Netflix Documentary) 1:26:16 - Ronald Reagan’s Presidency, The War on Drugs, & The AIDS Epidemic; American Exceptionalism; Mental Health in the Covid Pandemic; The complete correlation between crime and money 1:41:12 - Ilhan Omar & her criticism of President Biden’s stimulus check program compared with former President Trump’s 1:47:41 - Mike & Julian revisit Mike’s “tagging” every neighborhood in New York City back in 2016; Phish 1:51:51 - Meghan Markle & Prince Harry’s interview with Oprah; The racial issues with The Royal Family; The effects of Prince Harry’s experience with his own mother’s tragic death (Princess Diana) 2:15:35 - Piers Morgan hilarious interview of Alex Jones over gun control; Meghan Markle & Piers Morgan’s romantic history 2:30:51 - Governor Andrew Cuomo (NY) & his recent sexual assault allegations; Julian does Cuomo and Sebastian Maniscalco impressions; Governor Cuomo’s brother, Chris Cuomo and his naked escapade in front of his  wife’s yoga class; The Dr. Seuss controversy; How the Major News Networks are all profiting off of the division in the country. ~ YouTube EPISODES & CLIPS: https://www.youtube.com/channel/UC0A-v_DL-h76F75xik8h03Q  ~ Show Notes: https://www.trendifier.com/podcastnotes  TRENDIFIER Website: https://www.trendifier.com  Julian's Instagram: https://www.instagram.com/julianddorey  ~ Beat provided by: https://freebeats.io  Music Produced by White Hot

Join us for a very special episode all about poo! (We promise it's not quite as silly as it sounds, you may learn a thing or two...)In this episode, we cover pretty much everything from gorillas, wombats and hippos to unusual coffee, drunken cooks and thought-provoking short stories.P.S. We talk about COVID for a bit of this episode - If COVID is not your cup of tea at the moment and/or if you don't want to hear an excellent impression of UK Prime Minister Boris Johnson, please join us approximately 14mins in - enjoy!

Mike joins the boys to chat about how his vocal covers on YouTube helped him become the vocalist of Rival Town, how he feels about his Leafs going up against Columbus, and why he was called "the hockey jersey kid" in elementary school.Rival Town-Swear JarRival Town-A New NormalNorthstar-For Members OnlyHead to c2chockey.net and take 25% off with the coupon code BARDOWN

Join me as I continue the Whiteboard Confessional series with a look at multi-factor authentication (MFA) and the time my business partner Mike Julian lost his MFA device and needed to reset his Amazon MFA token and couldn’t figure out how. Among other things, I discuss why you shouldn’t make decisions or record podcasts when you’re angry, why you shouldn’t store MFA codes in your password manager, why your policies and procedures won’t matter if someone chooses to disregard them, how you should expect people to do the wrong thing and make it easy to do the right thing, why you shouldn’t incentivize people to hide mistakes, and more.

Rob talks to Mike Julian about how he found a designer on Folyo - including the questions he asked, the qualities he looked for, and the signals that won him over in the process.

Sponsors CacheFly Panel Nell Shamrell-Harrington Scott Nixon Episode Summary In this episode of the Adventures in DevOps podcast, panelists Nell Shamrell-Harrington and Scott Nixon talk about monitoring in the software world. They start the discussion by talking about the difference between monitoring and alerting. They discuss how logging comes into picture in monitoring, two main types of logs - structured and unstructured, log management in the DevOps environment, information storage, parsing logs and log aggregation. They list two major kinds of monitoring software - pull and push. Nell explains what they mean and how they work, and Scott gives examples of each including syslog, healthcheck, etc. They then talk about what it means by a "working" system, and consequently, when can something be considered to be non-functional. This leads to answering the important question of what exactly should be monitored. They explain to what extent should one go while monitoring and how to determine the significance level of the events in general. They discuss some concepts from Mike Julian's book "Practical Monitoring" including anti-patterns such as tool obsession, what not do do in monitoring and the fact that businesses need to customize their systems based on what works for them. They talk about the tool Nagios, benefits in using default monitoring tools provided by native cloud systems, using monitoring as a crutch, and manual configuration. They then discuss some good practices, namely composable monitoring, performance monitoring from the users' perspective, the mantra - 'buy not build', and continual improvements. They briefly touch on the topic of security in monitoring, and wrap up the episode with picks. Links Practical Monitoring-Mike Julian Accelerate: The State of DevOps with Dr. Nicole Forsgren High Output Management - Andrew S. Grove Picks Nell Shamrell-Harrington: Mauer Museum Tetris 99 Scott Nixon: Automated Reasoning Byron Cook, Amazon | AWS re:Inforce 2019 Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones The Obstacle Is the Way: The Timeless Art of Turning Trials into Triumph

Sponsors CacheFly Panel Nell Shamrell-Harrington Scott Nixon Episode Summary In this episode of the Adventures in DevOps podcast, panelists Nell Shamrell-Harrington and Scott Nixon talk about monitoring in the software world. They start the discussion by talking about the difference between monitoring and alerting. They discuss how logging comes into picture in monitoring, two main types of logs - structured and unstructured, log management in the DevOps environment, information storage, parsing logs and log aggregation. They list two major kinds of monitoring software - pull and push. Nell explains what they mean and how they work, and Scott gives examples of each including syslog, healthcheck, etc. They then talk about what it means by a "working" system, and consequently, when can something be considered to be non-functional. This leads to answering the important question of what exactly should be monitored. They explain to what extent should one go while monitoring and how to determine the significance level of the events in general. They discuss some concepts from Mike Julian's book "Practical Monitoring" including anti-patterns such as tool obsession, what not do do in monitoring and the fact that businesses need to customize their systems based on what works for them. They talk about the tool Nagios, benefits in using default monitoring tools provided by native cloud systems, using monitoring as a crutch, and manual configuration. They then discuss some good practices, namely composable monitoring, performance monitoring from the users' perspective, the mantra - 'buy not build', and continual improvements. They briefly touch on the topic of security in monitoring, and wrap up the episode with picks. Links Practical Monitoring-Mike Julian Accelerate: The State of DevOps with Dr. Nicole Forsgren High Output Management - Andrew S. Grove Picks Nell Shamrell-Harrington: Mauer Museum Tetris 99 Scott Nixon: Automated Reasoning Byron Cook, Amazon | AWS re:Inforce 2019 Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones The Obstacle Is the Way: The Timeless Art of Turning Trials into Triumph

When searching for new business, the last thing you want is to lose out to a competitor or find yourself working with a client who can’t afford your services.

This week I get a chance to speak to someone who just wants to save you some money on your cloud bills. Mike shares some great stories and gives insight to what he and Corey Quinn are working on at the Duckbill Group. Mike is the CEO of The Duckbill Group, a consultancy helping companies fix the horrifying AWS bill by both lowering the size of it and helping them understand where the money is going. Mike also hosts the Real World DevOps Podcast, is the author of O’Reilly’s Practical Monitoring, and editor/analyst at Monitoring Weekly. He was previously an SRE/DevOps Engineer/system administrator for companies such as Taos Consulting, Peak Hosting, Oak Ridge National Laboratory, and many more. Mike is originally from Knoxville, TN (Go Vols!) and currently resides in Portland, OR. Twitter: https://twitter.com/mike_julian https://www.duckbillgroup.com https://monitoring.love https://www.realworlddevops.com

In this episode I talk to Mike Julian of Monitoring Weekly about growing up and acting like a real business. We cover all the scary parts like lawyers, contracts, insurance, and more in a really non-scary way.

Wherein Corey Quinn and Mike Julian pontificate about the dangers of perfect infrastructure, why multi-cloud is (probably) a dumb idea, and that your biggest risk in a large-scale disaster is your entire team quitting to help your competitor for 10x more money.

Mike Julian is the Editor of Monitoring Weekly, author of Practical Monitoring for O'Reilly, and a consultant/analyst for the app and infrastructure monitoring world. He joined Mike Broberg to talk about systems monitoring and data, as well as what that means in terms of effective monitoring for security observability.

In the first episode of High Leverage, Heavybit General Partner and host Joe Ruscio is joined by Monitoring Weekly's Mike Julian for a discussion on observability tools, trends, and changing landscapes.

In the first episode of High Leverage, Heavybit General Partner and host Joe Ruscio is joined by Monitoring Weekly's Mike Julian for a discussion on observability tools, trends, and changing landscapes. The post Ep. #1, Monitoring Observability with Monitoring Weekly’s Mike Julian appeared first on Heavybit.

Links: https://weekly.monitoring.love/ https://www.practicalmonitoring.com/

Monitoring in the entire technical world is terrible and continues to be a giant, confusing mess. How do you monitor? Are you monitoring things the wrong way? Why not hire a monitoring consultant!          Today, we’re talking to monitoring consultant Mike Julian, who is the editor of the Monitoring Weekly newsletter and author of O’Reilly’s Practical Monitoring. He is the voice of monitoring. Some of the highlights of the show include: Observability comes from control theory and monitoring is for what we can anticipate Industry’s lack of interest and focus on monitoring When there’s an outage, why doesn’t monitoring catch it?” Unforeseen things. Cost and failure of running tools and systems that are obtuse to monitor Outsource monitoring instead of devoting time, energy, and personnel to it Outsourcing infrastructure means you give up some control; how you monitor and manage systems changes when on the Cloud CloudWatch: Where metrics go to die Distributed and Implemented Tracing: Tracing calls as they move through a system Serverless Functions: Difficulties experienced and techniques to use Warm vs. Cold Start: If a container isn't up and running, it has to set up database connections Monitoring can't fix a bad architecture; it can't fix anything; improve the application architecture Visibility of outages and pain perceived; different services have different availability levels Links: Mike Julian Monitoring Weekly Copy Construct on Twitter Baron Schwartz on Twitter Charity Majors on Twitter Redis Kubernetes Nagios Datadog New Relic Sumo Logic Prometheus Honeycomb Honeycomb Blog CloudWatch Zipkin X-Ray Lambda DynamoDB Pinboard Slack Digital Ocean

This week on Low Carb Conversations with Leah Williamson and Guests we are joined again by our special guest co-host Marty Kendall along with Alex Leaf, researcher at Examine.com and Mike Julian who has a passionate interest in nutrition and one of the Admins of Optimising Nutrition. Sit back and relax while Leah and Marty discuss the latest news headlines with special guests. This week on the show Leah, Marty, Alex and Julian discuss Roy Taylor's intensive weight loss diet for type 2 diabetics that totals about 825 calories a day.  Can this really work?  The next article brings up the topic again for protein.  We hear Alex and Mike's thoughts around this and around exogenous ketones.  Make sure you listen in to get all the facts.   12 Weeks on a Strict Liquid Diet Program Reverses Diabetes   How Too Much Protein is Bad for Ketosis     If you would like to keep the conversation flowing and share your thoughts on the discussion join us in the Low Carb Conversations Facebook Discussion Group.

On this special post Thanksgiving dinner edition pod Sam sits down with Mike & Julian from the Not That Serious podcast (@NTSPod) to recap the first month of NBA action and look ahead to what may be a special spring in Philly. Can the Rockets or Thunder see the Warriors in the West? Would you have faith in the Cavs even as a 6 seed?LeBron to Philly is an actual thing?Subscribe to the Pick N Pop Podcast on iTunes, Stitcher Radio & Google Play.Follow Sam on Twitter: @theesamdFollow the Not That Serious Podcast: @NTSPod See acast.com/privacy for privacy and opt-out information.

On this special post Thanksgiving dinner edition pod Sam sits down with Mike & Julian from the Not That Serious podcast (@NTSPod) to recap the first month of NBA action and look ahead to what may be a special spring in Philly. Can the Rockets or Thunder see the Warriors in the West? Would you have faith in the Cavs even as a 6 seed?LeBron to Philly is an actual thing?Subscribe to the Pick N Pop Podcast on iTunes, Stitcher Radio & Google Play.Follow Sam on Twitter: @theesamdFollow the Not That Serious Podcast: @NTSPod See acast.com/privacy for privacy and opt-out information.

You won’t become a recognizable name in your niche market overnight, but you can get a head start with these tips from Mike Julian of Aster Labs. His unique perspective on outreach and the way he uses social media can really put you at the forefront of your field and connect you with the right clients. Learn how to approach your niche the right way so you can start earning the income you need to make freelancing your full-time job. https://freelancetransformation.com/episode141 Today's episode is brought to you by Harpoon! Harpoon is the all-in-one financial management solution built specifically for freelancers like yourself. Harpoon is the first online invoicing and time tracking tool, at least that we've seen, that helps you actively predict if your freelance finances are on track to meet your goals, instead of just reporting them in hindsight. Try Harpoon completely risk-free for 14 days. And if you love it use the coupon code TRANSFORMATION to receive 20% off the first 3 months of your subscription.

Takeaways Effort + Execution + Empathy: It used to just be a numbers game. You put in the work, you made your calls, you sent your emails and it produced results. Then everyone started ramping up the volume. Today you have to not only put in the effort, but also apply strategic execution and have empathy for your buyer. Have you done your homework? Do you understand the real challenges they’re having? Do you actually care? Prospects buy from us because they believe we can get them to a place they can’t get to on their own. You Can’t Lose What You Don’t Have: Too many reps focus their energy on everything that could go wrong instead of what might go right. Very similar to Mike’s story, I didn’t come from much. So knowing that I’ve done without before and been fine, I have a different perspective on the world. I’m willing to take risks that others may not because I don’t have a false sense of security. Short Term Thinking vs Long Term Thinking: It’s the difference between “I want to sell to this company today” vs “I want to build a relationship that may carry me throughout my career.” Yes, I know you have a number to hit this week, month, and quarter, but if you put in the work today to build a solid pipeline, you can give yourself the freedom to build lasting value with your prospects. Full Notes https://www.salestuners.com/mike-julian/ Book Recommendation The Richest Man in Babylon by George S. Clason Sponsor What if every sales rep inherited the habits of your best rep? With Costello, they do.