Podcasts about announcer thank

Sometimes "reality" TV takes it one step too far.  Sometimes two steps.  Sometimes a flying leap. WARNING:  IMPLIED VIOLENCE AND TORTURE Written and Produced by Julie Hoverson Cast List Announcer  - Frankenvox Alison - Beverly Poole Bart - Michael Faigenblum Carl - Mike Campbell Debbie - E. Vickery Ms. Sheldon - Sharon Delong Tanya - Tanja Milojevic Mom - Shayla Conrad-Simms Dad - Reynaud LeBoeuf Son - Eli Nilsson Fred - Joel Harvey Bob - Glen Hallstrom Helen - Helen Edwards June - Shelbi McIntyre Kathy - Kim Poole Additional Voices - Russell Gold; Julie Hoverson Music by Brian Bochicchio (Seraphic Panoply) Show theme:  Kevin MacLeod (Incompetech.com) Editing and Sound:   Julie Hoverson Cover Design:  Brett Coulstock   "What kind of a place is it? Why it's right here, right now, can't you tell?" ************************************************************************ IDIOT BOX Cast: [Opening credits - Olivia] TV Announcer Alison, chipper Bart, sullen Carl, upbeat, hearty Debbie, nervous, angry underneath Ms. Sheldon, executive producer Tanya, in the sound booth Family - mom, dad, teenage son Bar - Fred, Bob, Helen Dorm - June, Kathy OLIVIA     Did you have any trouble finding it?  What do you mean, what kind of a place is it?  Why, it's right here, right now, can't you tell?  MUSIC SOUND     THEME MUSIC ANNOUNCER    last week, in the record-breaking debut of The Box, we were introduced to our four contestants: ALISON    [chipper] I'm Alison, from Santa Monica.  Hi, mom! CARL    [hearty] Carl, from Atlanta - home of the Cartoon Channel!!  DEBBIE    [nervous] Debbie, from Salem.  Uh, Oregon.  [quickly] Salem Oregon. BART    [sullen] Bart, Minneapolis [disgusted sigh]. ANNOUNCER    The rules are on the screen now for all you viewers out there, to cover the formalities.  They are also available on our website at [spelled out superfast] w-w-w-dot-s-k-i-n-n-e-r-i-d-i-o-t-b-o-x-dot-com.  AMB    FAMILY LIVING ROOM SOUND    CHIPS EATEN FROM BAG ANNOUNCER    [TV] And after this brief message, we'll show you the results of last week's voting. SOUND    CLICK OF REMOTE SOUND    POPCORN POPPING IN MICROWAVE MOM    [off] You better not have turned that off, hun! SOUND    MICROWAVE DINGS DAD    Just muted.  Sick of all these ads for freaking erectile dysfunction.  If anything's going to give a guy man-trouble, it's having to watch all those damn ads. SOUND    POURING POPCORN INTO BOWL SON    Ew, dad.  T-M-I. MOM    [coming in, munching popcorn] The one I hate is that smiling guy.  His wife just looks so scared all the time.  Almost as creepy as the King. SON    Am I adopted?  Please say yes. DAD    Ooops, back on! ANNOUNCER    [TV]  Did everyone vote? MOM    I certainly did! SON    Mom?  [disgusted noise] Why? ANNOUNCER    [TV] The voting is closed, the tabulations have been made, and the scores are coming up on the screen now. MOM    [over the announcer] Why not?  I want that nice young girl - the blonde - to win.  She's very wholesome. ANNOUNCER    [TV] And it looks like today Alison has been selected!  MOM    [satisfied] There! ANNOUNCER    We have Alison in the studio now - let's see how she takes it. SOUND     LIGHT MUSIC, ON THE TV SEGUES INTO REALITY ANNOUNCER    Hello Alison!  Say hi to everyone!  ALISON    Hi!  Hi mom!  Dad! ANNOUNCER    How's the first week been treating you? ALISON    This place is great! ANNOUNCER    Throughout the show, we'll be showing some of the fun you four have been having.  Now, why don't you tell me what you think of your new friends? ALISON    Oh, wow - everyone's really great.  ANNOUNCER    Don't you find Bart a bit... isolated? ALISON    He's just self-contained.  I'm sure he's a good guy, he just doesn't open up real easily. ANNOUNCER    And Debbie? ALISON    She's shy - a lot like my sister.  Hi Vickie!! ANNOUNCER    [chuckles] That's great. ALISON    And Carl - well, he's a blast.  He's always thinking up great stuff to do. ANNOUNCER    Yesterday you had sole access to the Dairy Dan Amusement park. ALISON    Oh, man - that was awesome!  They closed the gates and we got to ride all the rides all day long - no lines, no crowds!  Woo! ANNOUNCER    You've been chosen. ALISON    Woo!  [stumbles] I - What?  What? SOUND    CONTROL BOOTH ANNOUNCER    [TV]  Please step into the box. ALISON    [TV - gasp, then steels herself]  Right.  [somewhat bitter] Thanks America. SHELDON That's the shot - tight in on 2, now 3 - yes! Keep her face centered until she shuts the door. TANYA    Got it. SHELDON Okay, keep the volume low on that. It's early yet - don't want to wear out the viewers... SOUND    [TV] ELECTRIC SHOCK NOISE, SOMEWHAT BRIEF ALISON    [TV - short scream] ANNOUNCER    [TV] We'll be right back after the break to find out what today's challenge will be. AMB    DORM ROOM JUNE    Omigod!  Omigod!  Did you see that? KATHY    [distracted] Hmm?  No but I sure heard it - did they just do what I think they did? JUNE    They just shocked the crap out of the blonde chick! KATHY    Was there actually crap? JUNE    [duh] She was in the box.  Shh.  It's coming back on. SOUND    TV TURNS UP ANNOUNCER    [TV] We'll be right back with more of The Box after these messages. SOUND    SOUND DOWN AGAIN JUNE    I hate when they do that. KATHY    Shock someone? JUNE    No, have the logo come up and make you think the show is back on. KATHY    Yeah, that's much worse. JUNE    You know what I mean!  It was totally mean that they shocked her - she's the one who got the most votes! KATHY    Isn't that what everyone was voting for? JUNE    No!  At least, I don't think so - I mean, I thought it was voting for who would win something cool.  I ...voted for her. KATHY    You actually voted? JUNE    On the website, yeah. KATHY    Of course there's a website.  Maybe you should read the fine print. JUNE    Oh, oh!  It's back on!  Jeez, look at her poor hair! SOUND    TV UP ANNOUNCER    [TV] Back to the interview room, to hear from Alison. ANNOUNCER    [real] Before we go on, I need to point out, this is the only time you can choose to leave the show.  Are you prepared to stay? ALISON    [gulps, then quiet]  Yes.  [clears her throat, louder]  Yes.  [very shaky] That wasn't so bad. ANNOUNCER    Excellent.  Now I believe you recently graduated from college, Alison.  What did you get your degree in? AMB    BAR ALISON    [TV]  I'm a liberal arts major, with a minor in art history. FRED    So she's unemployed, eh? ANNOUNCER    [TV]  And you are engaged to be married? BOB    Too bad.  All the cute ones are taken.  Even with that weird hairdo. SOUND    TV SWITCHED TO SPORTS FRED    Hey, we were watching that! HELEN    Why?  It's awful, letting them mess with people on TV like that! FRED    [scornful] It's not real.  BOB    Course it is - it even has a website! HELEN    Puh-leez.  Lots of things have websites that aren't real. BOB    Name one. HELEN    Pamela Anderson's boobs. FRED    She got you there, pal. BOB    C'mon - just switch it back long enough to see what today's challenge is?  Please? HELEN    Ya big softie, you. SOUND    TV CHANGES BACK ANNOUNCER    [TV] Carl, you got the second most votes this week - Do you have anything to say to the viewers at home?  Obviously you're doing something right, to get so many votes. CARL    [TV]  I think it's just my sunny personality, Bob.  People like winners, and I am a winner. AMB    LIVING ROOM SON    Weiner. MOM    Language! SON    [dismissive noise] Doesn't that dipstick know that most votes gets zapped? DAD    Maybe he doesn't - they might not tell THEM everything, either.  Makes sense.  Why else would they be so excited? SON    But that sucks!  That sucks big time!  Here they are, trying to be all cool and get people to vote for them, and they're like masterminding their own torture or something. DAD    It's just a game,  No one really gets hurt. MOM    Well, I was kind of upset that Alicia-- SON    Alison. MOM    Yes, that she got shocked.  I didn't know that voting for her would do that.  I kind of feel bad now. SON    Well, don't vote for her next time. MOM    I certainly won't! ANNOUNCER    [on TV] Well, we've spoken to two of our four contestants, and the voting is open for the halftime winner.  Go on line now or text to-- SOUND    TV MUTES, AMB/DORM SOUND    COMPUTER KEYS KATHY    What are you doing? JUNE    Voting. KATHY    Vicious much? JUNE    No! I - I just don't want her to have to get shocked again.  Damn!  It only lets me choose one of those two - not the other guy. KATHY    So you want to see him get shocked? JUNE    Well, no, but I like him the least. KATHY    Just cause you don't think he's cute. SOUND    ONE LAST KEY JUNE    Um, there. KATHY    So who'd you vote for? JUNE    The guy - the nice one - of course.  I like him, too, but I don't want her to get shocked again. SOUND    TV UP AGAIN ANNOUNCER    [TV] Regular text messaging fees apply.  And now‑‑ SOUND    OMINOUS MUSIC ROLLS IN ANNOUNCER    [TV, ominous] The moment in the spotlight.  Will it be Alison or Carl?  The voting closes in three minutes, so hurry up and make your vote count - if the lines are overloaded, make sure and try back - but be quick.   [normal] While we wait, let's watch some clips from the preliminary interviews with the other two contestants. MUSIC ANNOUNCER    [TV] And what are you studying? DEBBIE    [TV] I'm - um - a poli sci major. FRED    So she's gonna end up unemployed too. BOB    Whatever happened to good old trade schools? FRED    They're still around - just the trades aren't.  You seen any cobblers in the U.S. of A recently?  Nope.  It's all farmed out to Pakistan and Koala Lumper. HELEN    Lumpur. FRED    Sez you. HELEN    I can turn it off, you know. BOB    Yeah - see now Helen here's got a job that can't be farmed out - long as there's guys like us, there's always gonna be bars, eh? FRED    Until they invent a mixology robot. BOB    Hey, the lights are flashing on the screen, must be something important. SOUND    TV TURNED UP. SOUND    OMINOUS MUSIC INTENSIFIES ANNOUNCER    [TV; evil "suspense" pacing]  And the one who got the most halftime votes.  Will it be Alison, our stoic liberal arts major? JUNE    Yes, yes - come on come on!!! ANNOUNCER    [TV] Or Carl, who tutors children with learning disabilities. MOM    Oh, that's awful! SON    Awful? That he works with retarded kids? MOM    [almost a whisper] That I voted for him. ANNOUNCER    [TV] And the one who got the most votes in the 8-minute half-time poll was-- SOUND    HEAVY DRUMBEAT ANNOUNCER    [TV] Was-- SOUND    HEAVY DRUMBEAT KATHY    Look at how much she's sweating! JUNE    You'd sweat too if you just got shocked! ANNOUNCER    [TV] is -Carl! JUNE    Whew! KATHY    Shh.  Let's see what happens. ANNOUNCER    [TV] This means that at the end of tonight's show, Carl will be up against the second half winner in a showdown to see who gets a million dollars sent to the charity of their choice. HELEN    Waitaminute - she gets shocked and he gets a chance to win big bucks?  That's so not fair! FRED    That's the way it is.  Women always getting the short stick. HELEN    Especially when they're dating you, eh? BOB    [laughs, tried to stop] FRED    Yeah, yeah - you can joke now, but I'll give you 70-30 odds that the other winner is that other guy. BOB    The grouch? FRED    Yup.  Is it a bet? BOB    Fifty bucks? FRED    Whoah, whoah!  Let's not get carried away here, now. MUSIC - OPENING THEME, PLAYS FOR A MOMENT ANNOUNCER    Entering week five of The Box, you can see the ratings posted for our four contenders.  [hushed] Last week, it looked as though Debbie had finally broken-- DEBBIE    [TV] I hate it!  I hate you all! You can all just go and-- SOUND    LONG SERIES OF BLEEPED WORDS SOUND    ZAPPING AND SCREAMING UNDER NEXT LINE ANNOUNCER    But after her trip to the box, she refused to cry off. DEBBIE    [TV] [breathing heavily and gulping] No [gasp] way! [gasp]  You don't [gasp] get rid of me [long shaky breath] that easily. [sob] ANNOUNCER    And now, a new week - and what was this week's challenge? STUDIO AUDIENCE Fasting! ANNOUNCER    Yes, fasting.  Whoever could go the longest without eating even a single bite of food got a free pass this week‑‑ ANNOUNCER    [TV] --and we'll find out who managed that in just a moment - after a few words from our sponsors. SOUND    CLICK, SOUND OFF JUNE    [urging] C'mon Debbie! KATHY    Debbie?  Hah.  She's got no body fat to start with.  Bart has a much better chance of surviving-- JUNE    Don't say that!  You just like him cause you know I don't! KATHY    I root for the underdog.  It's a principal. And no one likes that poor bastard. JUNE    If no one likes him, how come Debbie's the one always getting shocked, huh?  [almost a sob] Huh? ANNOUNCER    [TV]  Let's bring our four contestants out on stage to hear who's going to be free and clear for another week.  Alison-- SOUND    MUSIC UP, DOOR OPENS, SHAKY FOOTSTEPS ANNOUNCER    [real] Alison, how are you feeling? ALISON    [trying to be perky] Not too bad.  I made it almost three whole days on nothing but water.  ANNOUNCER    But then you lost it? ALISON    [heavy sigh] Yeah, I had to give in and get something.  [resigned] I figured fine - just put me in the box.  At least that eventually ends.  ANNOUNCER    Thank you, Alison.  Now go over to the isolation booth while we talk with each of your friends. ALISON    [venomous] Friends?  Hah! ANNOUNCER    [TV, confidential] She needs to learn to be careful about trading today's pain for tomorrow's - what she doesn't know is we've [ramping up] turned the voltage up another notch! AUDIENCE    [TV, CHEERS] HELEN    This just keeps getting worse.  It has to be against the law. BOB    Oh, come on.  They signed waivers, didn't they?  Plus, it's all fake - like wrestling.  Seriously.  Even if they did do this stuff, they have to have doctors and all on staff - make sure no one really gets hurt. SOUND    UNWRAPPING AND OPENING A FORTUNE COOKIE FRED    Hey, listen to this - "Those who cannot remember the past are condemned to repeat it." BOB    Figures the Chinese would think of that first. FRED    Nah.  The Chinese didn't make that up. HELEN    Then who did say it? FRED    [immediate] Thomas Jefferson. BOB    I don't think so. FRED    Yeah?  And who do you think it was? BOB    Some Greek philosopher or other.  [idea] Julius Caesar! HELEN    You guys make your bet, I'll call Jonesy on the next commercial and he can google it. SOUND    TV TURNS UP ANNOUNCER    [TV]  So Bart, you made it the longest without eating - you have any special tips for the viewers out there on how you did it? BART    [real] Huh? ANNOUNCER    Any tips?  We'll give you a minute - these moments of uncertainty are just further proof that our show is live and unedited.  While Bart ponders this, I'll recap - Alison gave into her craven need for food first, followed by Carl and Debbie - in a virtual photo finish, where Debbie held out for one millisecond longer than Carl.  Good going Debbie! BART    I hate you. ANNOUNCER    Hmm?  What's that? BART    I hate you and all you stand for.  ANNOUNCER    Do I hear an opt-out coming?  For those of you just tuning in, during this episode and this episode alone, any of our four contestants can opt out at any time - not just immediately following a trip into the Box.  So Bart, are you-- SOUND    A BEEP TRIES TO CUT HIM OFF ON THE FIRST WORD BART    Fuck you!  You can't get rid of me that easily.  BART    [TV] I don't care how many times you drug me and try to get me to bow down to the corporate machine!  You and all you people at home - you are sadistic bastards, but I'm here for the long haul - And when I finish, whether I win or not, I will be traveling around the country demanding the pound of flesh each and every one of you bastards owe me!!! KATHY    For god's sake, turn it off. JUNE    No, he's making a valid point.  We shouldn't be party to this. KATHY    The very act of watching it validates it. JUNE    No.  I'm only doing this to bear witness. KATHY    The advertisers don't care.  They just want to you to watch. JUNE    Well, I won't vote any more. KATHY    Then you can't complain when your favorite gets zapped. JUNE    [upset] Oh hell! ANNOUNCER    [TV]  Well, that was very enlightening.  Before you out there start emailing and phoning - please refer to clause 42 slash 8 slash F, subsection I-I-I, paragraph y, where it sets out the game's rules covering mental illness or defect.  Thank you, and good night! SOUND    TV TURNED OFF HELEN    Anyone checked out the big pools? FRED    What do you mean? HELEN    There's huge bets all over the place - everyone guessing who's gonna last the longest. BOB    Well, no one's washed out yet. FRED    They're a tough bunch of kids, but I bet I could make it on that show.  Age does bring wisdom. BOB    To who? FRED    You're too young to remember this, but I was a P-O-W in nam [rhymes with "ham"].  I been through it all.  Torture, deprivation, brain washing. HELEN    They sure got yours squeaky clean. SOUND    DRINKS WHOLE BEER DOWN. BOB    Ahhh. MUSIC    ANNOUNCER    This week, week 9 of The Box, we might just lose a second contestant.  ANNOUNCER     [TV] Alison, you've spent three days in this jacuzzi - brought to us courtesy of Big Joe's cut-rate pools and spas.  Now, people might think this was fun, but of course, you can't fall asleep or you might drown!  ALISON    [TV, parched, delirious]  You suck, Bob. FRED    Friend of yours? BOB    You wish. ALISON     [TV]  Get me out. ANNOUNCER    [TV] You do know that whomever leaves their jacuzzi first goes directly into the box? ALISON     [TV] No!  I want out!  OUT!  I can't - you can't make me stay here! JUNE    They can't, can they? KATHY     How much you wanna bet she signed something that says they can? JUNE    That's illegal! KATHY    Being stupid and greedy?  Nah.  They'd run out of prisons.  Unless you subscribe to the idea that our whole world is a prison. JUNE    [very upset] Don't talk like that - look at that poor girl!  They're just dragging her across the stage! KATHY    Wow.  I wouldn't'a thought it would take three guys to handle her, after all the crap she's been through. ALISON    [TV - screaming weakly and struggling] ANNOUNCER    [TV] It is understood, under the rules, that the clemency episode has run out and, once again, the only time you can opt out is right after a session in the box-- SON    If she's all wet, wouldn't that make the shock worse? DAD    At least her hair doesn't end up all weird since they shaved her head after that challenge last week-- SON    Three weeks ago. DAD    Really?  Anyway, they probably compensate somehow. MOM    Are you sure? DAD    [unsure] Well... They can't really hurt her - that would be... ANNOUNCER    [TV]     Oh, and - I've just got a word from the producer!  We've got a three minute vote - so grab your phones!  ANNOUNCER    [real] Now this will cost one dollar per vote, so make yours count!  Dial the studio number and hit 1 if you want us to let Allison forfeit and leave now, push 2 if you think we should hold her to the rules.  And voting opens [beat, then TV] Now! SHELDON Start the positive counter. TANYA    On it.  Running. NARRATOR     [TV] The positive votes will tally right here on the corner of the screen, and if, after the vote closes, there are more positive than negative votes, Alison will  immediately leave the studio - damper but wiser... BOB    Man, I wish I was in Vegas. FRED    Nah - you know what's going to happen.  The odd's'll be crap. HELEN    Course.  They'll let her go. FRED    You gotta lotta faith in people, babe.  Nah.  I'll give you 10 to 1 she's gonna ride the lightning. BOB     [incredulous] "Ride the lightning?" FRED    You know - old sparky. The electric chair?  Man where have you been? BOB    Considering no one's been executed in an electric chair in this state for - um - help me out Helen-- HELEN    50 years. BOB    50 years. FRED    Really? HELEN    How the hell'm I supposed to know? BOB    Well, whatever - a long time.  HELEN    Actually, I think this state always hanged people. FRED    Hung. BOB    The countdown! 5 - 4 - JUNE    3-2- MOM    [almost breathless] One. ANNOUNCER    [TV]  All votes are in, and as you can see, we had a regular landslide of support for our dear friend Allison here.  we have 4 million six hundred seventy two thousand, three hundred and forty-two votes for clemency.  Good for you everyone!  We'll show the other side, right after this-- SOUND    TV OFF DAD    No way! MOM    You can't    ! SON    I won't watch any more of this.  This is brutal. MOM    [angry] Don't you dare!  How can we not ... find out? SON    No. MOM    Just until they announce it - we don't have to watch ...if she... SON    Gets it? SOUND    REMOTE THROWN ONTO TABLE SON    You do what you want.  I'll be in the garage. SOUND    [after a moment] TV CLICKS ON COMMERCIAL    [something] KATHY    I bet the commercials for this cost top dollar.  Like superbowl ads. JUNE    How can you just be so snarky - that girl could die! KATHY    Nah.  They can't do that.  It would be illegal. JUNE    Not normally, but remember when that guy had a stroke on "Danger Island" last year?  The family sued, but the waiver made it perfectly legit.  KATHY    And that wasn't even that exciting. ANNOUNCER    [on TV]  For those just tuning in, we have perky little Allison in the Box, awaiting your verdict.  [continues under] Does she take the next shock, or have you tipped toward clemency for this poor girl? SHELDON Give them the split picture. TANYA    Before and after? SHELDON Uh-huh. [grim] Show them what they did. ANNOUNCER    [on TV] The negative votes have been tallied.  SOUND    DRUM ROLL, OMINOUS MUSIC ANNOUNCER    [ON TV] And we had 4 million six hundred seventy two thousand, three hundred and forty-two votes to let her go.  BOB    I'm still saying they'll let her off.  FRED    Nope.  You already lost that twenty, pal. HELEN    Shh! ANNOUNCER    [TV] The negative count is seven million three hundred-- SOUND    TV OFF KATHY    Did you vote? JUNE    Yes.  [beat]  Twenty times. KATHY    [shrug] You can't beat the bastards. JUNE    But if everyone just voted a few more times... KATHY    Three million more times. JUNE    How can people be so horrible? SOUND    [NEXT DOOR TV] SCREAMING PEOPLE    [laughing] SOUND    POUNDING ON WALL JUNE    [yelling at them] How can you be so horrible?? KATHY    They're drunk.  Didn't you see the sign? JUNE    [half a sob] Sign? KATHY    The one that said "come to gary's room, get drunk and watch The Box"? JUNE    [down] No. KATHY    Look, turn it on.  You'll see she's not dead or anything, then you'll feel better. JUNE    But what if she's not?  I mean, what if she is?  I mean-- KATHY    [sigh] Then you'll know. SOUND    [beat, then] TV TURNS ON SOUND    [on TV] AMBULANCE SIRENS JUNE    [sob] MOM    [sob] Her poor parents! DAD    Don't worry so much - she's not dead. MOM    She was for 43 seconds. DAD    That doesn't even count these days - happens all the time on House. MOM    [very upset] But this is real! SOUND    [on tv] MUSIC UP ANNOUNCER    [tv]  And we'll be checking in with Allison as soon as she regains consciousness to confirm her wish to opt out.  For now, the game comes down to Bart and Carl.  ANNOUNCER    Don't forget - no matter what happens, the game's big final episode is in two weeks.  SOUND    CAMERA OFF SHELDON Nicely done. ANNOUNCER    It's really wearing me thin. SHELDON Almost over. And after today's vote, there's no way the station can afford to cancel us. ANNOUNCER    [sigh, then grudging] Two more shows.  SHELDON [with meaning] And then we announce the results. MUSIC - OPENING THEME, PLAYS FOR A MOMENT AMB     NOISY BAR BOB    [ordering] Another one. FRED    Packed tonight. SOUND    DRINK SET DOWN HELEN    It's the finale. FRED    [tired] Oh, yeah.  That. BOB    Bottom's up! HELEN    Slow down, or I'm gonna have to pour you into a cab. SOUND    CAR KEYS SLAPPED ONTO THE BAR, SCOOPED UP SOUND    GLASS SET DOWN HARD BOB    Ahhh. CROWD    ROAR OF EXCITEMENT HELEN    Hold on!  I'll get it. SOUND    TV SOUND UP MUSIC    FANFARE ANNOUNCER    It's the night we've all been waiting for.  The night the final results are announced.  And we will have an ultimate winner.  Let's recap what the winner will walk away with.  SOUND    VOLUME DOWN SOUND    DOOR OPENS KATHY    Oh, you're not watching that, are you?  [sneer] I thought you decided it wasn't worth it! JUNE    [shell shocked] I can't not watch!  I have to know! KATHY    Look, let's go to the library or something. JUNE    No!  I would die of suspense! KATHY    It's not-- SOUND    TV VOLUME COMES UP KATHY    [sigh] I'm not staying. SOUND    DOOR CLOSES ANNOUNCER    And the contest comes down to our two finalists, Bart and Carl.  They have endured amazing hardship to make it this far.  Do you have anything you want to say to the people at home, Bart? BART    You still suck and you always will.  Every single one of you!  Every person who just sits by and supports this shit! ANNOUNCER    [still jovial]  And yet, you have continued to play our sick little games - as you call them - despite being offered chance after chance to leave. BART    Hah!  I don't plan to fucking let you win, you scumbags! ANNOUNCER    Well said.  And you, Carl, do you have anything for the audience? CARL    [mumbles] ANNOUNCER    Speak up? CARL    [vague, reciting] We are the music makers, and we are the dreamers of dreams.  Sitting by lone sea-- lone sea....  the sea.  The sea.  See see oh playmate, come out and play with me.... [fades out] ANNOUNCER    There you have it, folks.  And now we go to our man in the street interviewer, Tanya.  Take it away! TANYA    Thank you.  I'm in a major metropolitan center here, asking people on the street what they think of the Box. ANNOUNCER    If they're outside right now, instead of glued to their sets, they must not think much of it. BOTH    [fake laugh] SOUND    TV OFF SOUND    EATING MOM    What?  Don't you dare! DAD    Hey, we were watching that! SON    Are you enjoying this? MOM    Enjoying? DAD    What do you mean? SON    All this shit they've put those people through!  You can barely tell them apart now, after they've been starved and had their heads shaved.  They look like concentration camp victims! MOM    But - but this is the last show! DAD    What does it matter if we watch or don't watch? SOUND    THROWING DOWN A REMOTE SON    Do what you want.  I'll just hope for a six-car pileup.  Maybe you'll trade up. SOUND    DOOR OPENS AND SHUT SOUND    REMOTE TAKEN, TV ON ANNOUNCER    And for tonight, the big surprise is-- SOUND    DRUM ROLL, OMINOUS MUSIC ANNOUNCER    Two boxes! SOUND    CANNED CHEERING ANNOUNCER    One for each of you.  While we get them all set, here's a word from our sponsor! AMB    BAR CROWD    Buzzing "two boxes?" BOB    [slurry] Whaddaya think they've got up their shleeves? FRED    They're gonna kill one of those boys. HELEN    [confidential] I heard that girl Allison is in a private clinic, barely alive. FRED    Where'd you--? HELEN    Internet.  BOB    [sarcastic]  Yeah.  Then it's probably true. SOMEONE    Turn it up! HELEN    Got it! SOUND    TV UP ANNOUNCER    And now.  The moment of truth!  All the votes have been tallied.  As you can see, we have Bart over here in the red box-- SOUND    CANNED APPLAUSE ANNOUNCER    [tv] --and Carl over there in the blue. SHELDON close up on Bart, camera 2. Yeah, baby, clench that jaw.  Now cut to that trickle of sweat on Carl's face.  Nice. TANYA    Back to the announcer? SHELDON One more second, and - yes! ANNOUNCER    [tv] And now, with the votes tallied, we will find out who you out there have selected as the big winner, and who has to take the big penalty.  ANNOUNCER    [real] But first, we caught each of our contestants here on secret camera last night.  Let's see what they were doing on the penultimate night. SOUND    QUICK JAB OF STATIC VOICE    [tv] ...need to get out now.  You don't understand what they have planned for tomorrow.  It's so much worse! AMB    BAR BOB    Who the hell izzat? BART    [TV] [scoff] Worse?  Worse how? HELEN    Don't know.  FRED    Look at that announcer fellow - he's surprised too.  HELEN    [half a chuckle]  Serves him right. ANNOUNCER    [tv] Sorry - we should have screened that clip before playing it.  Let's go over to Carl's shot. CARL    [tv] Yea though I walk through the valley of the shadow of death, I shall--[cuts out suddenly] ANNOUNCER    [tv] And that's all the time we have for that.  And now the moment of truth.  Carl or Bart?  You held their fate in your hands. SOUND    COMMERCIAL COMES ON UNDER MOM    [coming in]  Where's Kyle?  Have you seen Kyle? DAD    [mesmerized]  He'll be back.  Just ... went out to a friend's house.  Probably. MOM    You should turn that off and find him! DAD    We can look in ten minutes just as easily as we can look now! MOM    This is our son! DAD    It's almost over! SOUND    OMINOUS MUSIC ON TV ANNOUNCER    [tv] And now.  The final countdown. MOM    Five minutes. SOUND    SHE SITS ANNOUNCER     [tv]  This has been quite a journey for everyone - and we would like to thank you all for your support and participation. BOB    Support?  I'd shoot that stupid bastard if I had a chance.  And a gun. HELEN    You're not the only one, but a lot of people paid a lot of money into that damn show. ANNOUNCER    [tv] --making us the highest rated network series ever-- FRED    yeah, and even WE count for ratings, since we happen to be watching it. BOB    [steaming into an alcoholic rage] Then let's not watch it! SOUND    SLAMS GLASS ON BAR, LIQUID SLOSHES FRED    Calm down, pal. BOB    No! Is this what our world has come to?  This crap?? SOUND    THROWS BEER GLASS AT TV, TV DIES, BUT OTHER SET PLAYS ON IN THE BACKGROUND CROWD    [Shocked silence] FRED    Great, one down, only seven hundred million TV sets to go. HELEN    I'll put it on your tab. CROWD     [chatter begins again] ANNOUNCER    [tv] --will definitely be returning for a second season, starting next fall-- SOUND    DOOR OPENS ANNOUNCER    [tv] --and we're looking at celebrity contestants.  TANYA    [tv]  That will be a whole new ballgame. KATHY    Sorry, didn't know it was still on. JUNE    [distraught] Stay.  Please. KATHY    Ugh.  Why? JUNE    Because I don't think I'll make it otherwise. KATHY    Make what? ANNOUNCER    [tv] And now for the final outcome.  MOM    Yes? DAD    About time. ANNOUNCER    [tv]  the final results. FRED    Don't call the police.  I'll get him home. HELEN    Yeah.  This time. ANNOUNCER    [tv] What we've all been working toward. JUNE    [crying] Can't they just say it? TV, MUSIC SWELLS, THEN CUTS OUT SUDDENLY JUNE    What? HELEN    Shit, must have blown the circuit. DAD    The electricity's still on! KATHY    Is there something wrong with your TV? MOM    No!  It's practically new! FRED    Come on.  Quitting time, pal. SOUND    TEST PATTERN NOISE, THEN MUSIC SUDDENLY CUTS BACK IN ANNOUNCER    Thank you all for participating in our experiment.  MOM    [gasp] ANNOUNCER    As you can see, all of our actors are in perfect health. JUNE    [sob]  How could they--? KATHY    Bastards. ANNOUNCER    We would love to hear your reactions to this show.  Please feel free to leave us a message at www-dot- SOUND    TV SWITCHES OFF HELEN     [last call voice] Allright.  That's it. CLOSER   [NOTE:  George Santayana, author of the quote.]

Show Notes 00:00 Introduction 00:22 Medicare part A 06:39 Medicare part B 15:35 Medicare part D Links Referenced: medicare.gov: https://medicare.gov Zach's email: mailto:zach@getsbi.com Nick's email: mailto:nick@getsbi.com Facebook: https://www.facebook.com/seniorbenefitinc Webpage: https://seniors-livinghealthy.com/ TranscriptAnnouncer: Welcome to our fireside chat with Seniors Living Healthy, the podcast that helps prepare and educate you as you enter and live out your golden years. With over 10 years of experience, Nick and Zach are experts in the senior market and are here to help you live a healthy, full life. And now fireside with your hosts, Nick Keene, and Zach Haire.Nick: Hello, and welcome to season two of Seniors Living Healthy, episode one. I'm Nick. And I have Zach, our co-host with us.Zach: Hey, folks.Nick: And for episode one of season two, we want to cover parts A, B, and D of Medicare, and the changes for 2022. So Zach, let's jump right in.Zach: Sounds good. So, kind of start off there from the top, Part A, just like in the alphabet, starting out with the first letter there, you know, that is our hospitalization, sir. You know, Nick, what are some common things that Part A covers?Nick: Yeah, so Part A kicks in when individuals are admitted to the hospital. It's worth mentioning, Zach, that they're admitted because we are seeing more commonly that people are being put in the hospital under observation. And that is actually covered under Part B. So, very simply, anytime someone is admitted to the hospital, not under observation, Part A kicks in.Zach: Got you. So, let's say, you know, I'm getting ready to turn 65 in a few months. I'm still working things like that, how do I get Part A? What do I have to do to qualify for it?Nick: Great question, Zach. We do get this question quite frequently. So, the most common way to qualify for Medicare is those individuals that have worked 40 quarters or ten years and paid into Medicare via payroll taxes, right? Those individuals get Medicare the month of their 65th birthday.Zach: Got you. So, no matter what, they're going to get Part A. I know you said you paid into it while you're working. Is there any additional costs added to that?Nick: Right. So, great question there, Zach, and worth mentioning here as well. For those individuals that qualify traditionally for Medicare, they worked 40 quarters, ten years, and paid in, Part A is premium-free, think of it as prepaid. But also you have those individuals that may qualify based on their spouse's, right? Their spouses may have worked 40 quarters or ten years, they also qualify for Medicare Part A the month of their 65th birthday.Then the third situation, there is a cost. And those individuals that don't have a spouse that qualifies for Medicare they can draw off of and don't have the credits themselves, depending on how much they have worked and paid in, Part A can be purchased.Zach: Yeah. So, you do still have the ability to get Part A, if you don't ‘qualify', you can always pay for that and pick it up.Nick: Absolutely.Zach: So, we know that in most cases, there's no additional premium; you've paid into it as you were working. Are there any other, you know, common costs associated with using Part A, whether it be a deductible, whether it be you know, skilled facility care, things such as that?Nick: Absolutely. So, yeah. So, basically with Part A, the way it works is it's designed with what we call a Medicare period of care, right? So, when those individuals that have Part A are admitted to the hospital, they are immediately responsible for a $1,556 deductible in the year 2022 that covers their first 60 days in their period of care, right? So, for those individuals, they go in, they pay that $1556 deductible, they're covered for the first 60 days, right?But it's worth mentioning that if they go beyond day 60 they do have additional cost, right? And that period of care doesn't end until they go a continuous 60 days without accessing care under Part A. So, assuming their period of care extends, day 61 through 90, those individuals are going to be responsible for $389 a day that they're in the hospital, and day 91 and beyond using those 60 lifetime reserve days, they're going to be responsible for $778 a day. You know, and the other thing to touch on here, Zach, that you mentioned is skilled facility care, right? So, we've seen a major transition in our market over the last five to ten years.You can recall when we were little, people had extended stays in the hospital, you know, people were in their one, two, four, six months. That doesn't happen really anymore, right? What we're seeing, the trend is individuals are being admitted to the hospital, they're being stabilized, and they're being shipped off to skilled facility care centers, right? And you know, whether that's for a hip replacement or a knee replacement, they fell and they broke something, speech, occupational therapy, whatever it may be, these individuals are staying at the skilled facility care centers for extended periods of time, not in the hospital. So, to qualify for Medicare to cover skilled facility care, they have to be in the hospital for at least three days and be admitted to the skilled facility care center within 30 days of being discharged. If those criteria are met, Medicare will cover day 1 through 20, one hundred percent, and then day 21 through 100, the individual is responsible for $194.50 per day.Zach: Got you there. So, you know, once someone is on Part A [everything 00:05:16], is there any limits where they can go, networks, anything like that?Nick: Yeah, one of the beauties of Medicare, Zach, and you know, we tell clients this all the time is Medicare's a nationwide program, right? California, North Carolina, Michigan, to Florida, and everywhere in between. They can access care, right? And that's one of the great things about Medicare is almost all facilities, almost all doctor's office accept Medicare. So, they have no restrictions, they can go just about anywhere they want.Zach: Got you. So, kind of wrapping up Part A there is, anyone can get that as long as you've worked 40 quarters or your spouse has worked 40 quarters. You're able to get that the month you turn 65, the first day of the month.Nick: Absolutely.Zach: And no matter whether you're continuing working or what you've got Part A?Nick: Yep.Zach: And with Part A alone, there was a $1,556 deductible on that they'd be responsible for but then, you know, it does help you in the skilled facility care things such as that, along with your 60-day continuous window of care. And again, no network so you can go wherever you want to go if you've got that Part A; pretty much every hospital, I'd say, in America takes Medicare.Nick: Absolutely, Zach. And just to wrap up on Part A, you know, one of the things that people need to remember is Part A is just hospital admittance insurance. Most of your typical services that are everyday needs are happening on outpatient care, or Part B, which we will be covering shortly.Zach: All right, so now we're going to roll into Part B, again, following our alphabet here, B comes right after A. So B, if you look at your red, white, and blue Medicare card, it is going to say medical, but we refer to it as outpatient.Nick: Absolutely, yeah. Yeah. And, you know, we try to eliminate confusion there because the Medicare card says ‘hospital' for Part A and ‘medical' for Part B, but we kind of feel both of those are medical, right? So, we like to explain Part B as anything that is outpatient care, or care that is not admitted into the hospital.Zach: Exactly, yeah. So, kind of got that cleared up. What exactly does it cover when it comes to different things?Nick: Yeah. So, Part B is by far the most common Medicare part, right? It's the most common used, and it literally covers any Medicare-approved charge outside of being admitted to the hospital, right? So, that could be hospital admittance under observation, that could be lab work, physical therapy, CAT scans, MRIs, doctor visits, primary care, or specialists, durable medical equipment, diabetic testing supplies, all those things encompass Part B.Zach: So, we know in Part A you get that automatically when you turn 65. Part B work the same way, or is there a few more hoops to jump through for that?Nick: Yes. So, for Part B, you know, that individual that qualifies for Medicare, either off their work experience or off of a spouse's work experience, they still are eligible to get Part B the month of their 65th birthday, right? However, with Part B, there is a premium, so Medicare does allow it to be elective.Zach: So, with it being elective, how does that situation play out? Do I have to take Part B when I turned 65? If I have creditable coverage, am I fine? You know, if I don't take am I going to get penalized? How does that work?Nick: Yeah, so we're seeing this question come across our desk more and more, Zach. It seems like in this day and age, more and more people are working post-65. We didn't run into this a lot five years ago. But basically, the way it's working is for those individuals that are Medicare-eligible, turning 65, they qualify for Medicare, they can still take Part B the month of their 65th birthday, but if they're still working and have credible coverage, right, which is defined as coverage, at least equivalent to Medicare, they do not have to take Part B. They can postpone it without penalty, assuming they have credible coverage.Zach: Got you. So, you said, you know, 2022, that average premium is $170.10.Nick: Yep.Zach: Which leads you to say if that's the average, there can be some outliers. Is there a way to make that cost lower?Nick: Yeah. So, you know, for a lot of individuals out there, they qualify for what's called Medicare Savings Programs, right? And we know those different programs, whether that's QMB, SLMB, Extra Help those types of things, those programs are designed to reduce or eliminate the premiums, deductibles, and copays associated with Part B, right? So, there are individuals that pay less, there are individuals that pay nothing if they qualify for those Medicare Savings Programs. And it's worth mentioning, to qualify for those programs, you need to reach out to Medicare, the Social Security office that goes through them.Zach: I'd be willing to bet it works the other way, too. I bet they can get a multiplier on you also.Nick: Yep, yep. So, what we see—you know, and once again, we're seeing it more and more as people are coming out of the workforce later in life—those individuals have what's called an IRMA, right, Income-Related Medicare Adjustment. So, if you have income levels above certain thresholds, Medicare is actually going to charge a multiplier, right, you're going to pay more than that $170.10 in 2022. medicare.gov is a great resource, they have the chart right there on the website, showing what those brackets are to get higher Part B costs.So, we certainly encourage people that think they may fall into that bracket, get on medicare.gov, reach out to us, you know, we can ask a couple questions and tell them what they would be looking at.Zach: Got you. So, kind of how we're on the cost of Part B—Nick: Sure.Zach: You know, if someone doesn't have credible coverage and they don't take Part B, then down the road they take Part B, what kind of penalty are they looking at?Nick: Yeah, so the government is penalizing those individuals that don't have Medicare and don't have credible coverage, right? And the penalty that they impose is 10% of the cost of Part B, per full year not covered either via Part B or creditable coverage, right? And it's worth mentioning, if they try to apply for Part B down the road, they're still going to pay that standard premium, they're going to pay that penalty on top of it, and unless they qualify for one of those Medicare Savings Programs like we were talking about, that's never going away.Zach: Yep so looking there, at you know—there are different times to enroll, in that, you know, when people do turn 65, a lot of times they take A and B at the same time.Nick: Yep.Zach: You can delay Part B, as we've talked about. What are those—that situation look like? If someone delays Part B, does that vary from when they turn 65?Nick: Absolutely. So, for individuals that are taking Original Medicare when they're turning 65, those individuals, you know, they get it the month of their 65th birthday. But for those individuals that are delaying Medicare, right, there's two different groups that it's worth mentioning here. For those people that have credible coverage that are still working, you know, they can take Part B anytime concurrent with their loss of coverage, or retirement, right, they have what's called a special election period. But the thing to mention is for those individuals that delay Part B that don't have credible coverage, they can only apply for Part B at certain times throughout the year, right?And that's what's called the general election period. Zach, right? And basically what that is a period from January 1st through March 31st each year that they can apply for Medicare Part B to go into effect 7/1 of that year.Zach: Right. So, you know, kind of look at you have your annual enrollment period, which is every year, October 15th, December 7th, which doesn't really play into this, but then you have your initial enrollment, which people might hear a lot about when they first turned 65, or take their Part B of Medicare. So, looking at, you know, we've kind of we've gone over what the premium can be as well as what possibly the penalty could be. As a whole, what does Part B have? What is it going to cover? What's going to be your out-of-pocket with that?Nick: Yeah, so you know, back to what we kind of mentioned earlier, just to kind of recap this is, Part B is going to cover anything that's not admitted into the hospital, right? So, you know, once again, that's hospitalization under observation; CAT scans; MRIs; lab work; physical therapy; doctor's visits, whether primary care or specialists; diabetic testing supplies; durable medical equipment. And the way Part B is designed, it's an 80/20 coinsurance, right? So, Medicare's covering 80%, the client is responsible for the remaining 20%, plus the Part B deductible, which is, in the year 2022, $233, right? So, it's worth mentioning here—and we tell this to people all the time, this is why we encourage people to get supplemental policies—that 20% that we speak of is uncapped.Now, if you're going to the doctor once a year, that's not a big deal, right? But if you're going through cancer treatments, if you're going through some sort of outpatient surgery, you got to pay 20% of all of that cost, which certainly leaves people with some exposure, right?Zach: Got you. So, you know, no max out of pocket; you know, you're going to keep paying that 20—Nick: Absolutely.Zach: —until—and again, Part B is very similar to Part A, there's no networks.Nick: Absolutely.Zach: They take Medicare, they're going to take in. As long as you may have been doing this, I don't think I've ran into a doctor's office that doesn't take Medicare, yet.Nick: Yeah. In ten years, I've ran into one facility that didn't accept Medicare.Zach: Yep. So, kind of wrapping up Part B there. Know no, it is, in a sense, elective; when you turn 65 or retire from work losing credible coverage, you can pick up Part B at that time. If you don't pick up Part B without credible coverage, they are going to give you a nice little permanent penalty to add onto that, which for 2022 is $170.10. Probably going to see an increase in that down the road.Nick: Mm-hm. Absolutely.Zach: It's going to cover everything for you 80/20, whether that be durable medical equipment, diabetic testing, outpatient surgery, or anything like that. But that 20% is not going to be capped.Nick: Yep, absolutely.Zach: All right. And kind of moving on down the line. Here we've done A, we've done B. We're going to skip over C, so we're going to hit in Part D of Medicare. Easy to remember what it covers because covers your drugs. Part D: Drugs, easy to keep up with there. So, we have talked about, you know, in Part A and Part B, how you get it, what you qualify for. How does that work with Part D?Nick: Yes, so Part D, you know, it's worth mentioning, Unlike Supplemental Coverage, or Medicare Advantage coverage, which we will be covering in next episode, With Part D, the individual only has to have a minimum of Part A or B of Medicare, although most people have A and B, right? But it's worth noting for those individuals that are still working that are delaying Part B, just having Part A is enough to purchase Part D. And it's also worth mentioning, you have to live in the plan's service area, right? Part D drug plans are network-based, so you have to have a minimum of A and/or B, and live in the plan's service area to purchase a drug plan it.Zach: So, also we've talked about cost. When it comes to cost, A and B for the most part, are standardized. Is Part D the same way, or you know, what is its cost?Nick: Yes. So, one of the things that, you know, we're always telling people as we're speaking with them is all prescription drug plans are different, right? And, you know, we see drug plans anywhere from $6.50 a month in premium in the year 2022 All the way north of $100 a month, right? And, you know, it's like we say, if one plan was the best for everybody, right, they would put the rest out of the business.So, as far as costs, it certainly has a wide range, and that all depends on what the scripts, what medications those individuals are taking, right? But it's also worth mentioning, just like Part B of Medicare, right? Medicare Savings Programs can cover some or all of the costs of the drug plans and can also either reduce or completely eliminate the cost of those medications people are taking as well, right? So, it can come down. And it's also worth mentioning, IRMA coming back into play here, right, that Income-Related Medicare Adjustment, for those individuals that are higher-level earners, right, they have a multiplier on that Part D premium, so they would pay that multiplier on top of the standard premium for Part D.Zach: Pretty easy to see why Part D is the most complicated part of our job—Nick: Absolutely.Zach: When it comes there. So, you know, kind of covered, premiums are going to vary, and then on top of that you could get help through Medicare, or you could get a multiplier on Medicare there. So, what does it take to qualify for Part D? I know you said yet to have Part A and/or Part B, one or the other, but what if I'm-you know, what, if I'm in that boat where I'm still working? Do I have to take Part D if I have Part A, or can I forgo it?Nick: Yeah. So, very similar to Part B, Part D is elective right? Now, you have to have credible coverage to not be penalized, but you can delay it. So, if you're 65, you're becoming Medicare eligible, you're still working, or maybe you're retired and you're still carrying group insurance, you don't have to take a drug plan as long as your coverage is credible. And once again, credible [unintelligible 00:18:59] coverage is defined as coverage at least equivalent to Medicare's basic coverage, right?So, for those individuals that are still working, they are not needing Medicare Part D, they will not be penalized for not taking a Medicare prescription drug plan.Zach: So, you said they—you know, if they have credible coverage, they're not going to be penalized, which therefore means there's a penalty.Nick: Yep.Zach: What is that penalty?Nick: Yeah. So, it's a little bit different than the way Part B works. So, for Part D, the average cost of a per prescription drug plan in 2022 is approximately $34. So, every full month that they go without credible coverage, or coverage, they are going to be penalized 1% of that $34 premium in the year 2022, times the amount of full months they went without coverage. Now, it's worth noting that average premium costs switches year-to-year, right? We've watched that steadily creep up over the last few years.So, you know, it's very hard for us to be able to give people an exact penalty, what they would be looking at. Medicare is who's going to determine those, Medicare is who's going to issue those, so we can give people an idea, but ultimately that information has to come from Medicare, right?Zach: Got you there. So, you know, we know when you first turn 65 going into Medicare, you can get Part D, if you go that route.Nick: Yep.Zach: What if I've been 65 for a while and I get some new prescriptions, it's not covered well on my plan, when can I make changes to those?Nick: Yes. So, for those individuals that are new to Medicare, they're in that initial enrollment period, right? That window runs three months before their effective date up to three months after. Once that period ends, right, they're very limited in the ways that they can make changes, right, the most common is annual enrollment period, right? Anybody that's been in this business, knows anything about it, they get bombarded, you know, in that timeframe.But from October 15th through December 7th, those individuals can make changes, as many as they want, and when the sun goes down December 7th, the last application that was signed and turned in becomes effective 1/1, right? But now over the last few years, you know, Medicare introduced the Medicare Advantage open enrollment period, right, which is now running January 1st through March 31, and during that timeframe, individuals that are on Medicare Advantage plans can make a change to their drug coverage in two different forms, right? So, they can change from one Medicare Advantage plan to another Medicare Advantage plan, or if they so choose, they can drop Medicare Advantage back to Original Medicare and pick up a prescription drug plan. But outside of those two windows, Zach, the only other situation, typically, that we see people can make changes is they have a special election period, right? And in our business, what that means is, A, they're moving, right?In our area, we see people coming down from the north moving here, or maybe they're snowbirds, they're moving from here or the north down to Florida. Those individuals get a special election period because they're moving out of that plan's service area, right? And then the other caveat would be those individuals that are post-65 that are still working, that are still carrying group insurance, those individuals have a special election period when they retire and/or lose coverage that they can make a change to their drug coverage as well.Zach: So, kind of off that point, there are networks on these drug plans that does give you the ability to change if you do move because you would be out of your network service area—Nick: Absolutely.Zach: There. Yep. So, you know, we talk to people all the time, especially [AEP 00:22:44] about prescription plans. When you're talking to us, talking to your agent, whoever, when you're going through this, one, you know, what are some things you need to make sure you have handy to make our lives easier as an agent, but then what—tell them on our end what we're looking at, to help them make a decision?Nick: Yeah, so I'm going to answer that question backwards, Zach, okay? I'm going to answer your second question first, and we'll fire away on the second one. So, for those individuals that are looking, right, to get prescription drug coverage, there's several things that they need to understand about a plan, or at least grasp, right, to know why it is what we're doing, right? It's easy for us to recommend a solution, but we feel—I know, we've always discussed this—we feel that ultimately, you know, it's our job to educate people, but it is ultimately their decision, right?So, for us, you know, what we're looking at, you know, in the grand scheme here is overall cost, right? I mean, you know, that's what I want to know, what are these plans going to cost you, whether that's in the form of a premium, whether that's in the form of a deductible on your plan, whether that's in the form of the copays you pay to fill your script each year, we're looking at that aggregate annual cost, right? Now, as far as what we need to be effective as a tool for them in searching plans, you know, all plans are different, Zach, as we know. The premium is different, some plans have deductibles, some don't, some offers zero copay on tier one, tier two, some don't, right?So, what we ask of clients to be effective in this manner is we need a list of your prescriptions, we need to know the dosages of each one of your prescriptions, and then we ultimately need to know the frequency that you're taking them or filling them, and we have the ability to plug in and pull all options in their area and discuss those costs with them.Zach: Yeah, definitely. So, kind of wrapping up Part D, put a bow on it there. It is similar to B, it's elective—Nick: Sure.Zach: —in a sense. As long as you've got credible coverage elsewhere, you don't have to take Part D at the time you turn 65. As long as you have A or B, you are eligible for it. And plans vary. This is a plan that you definitely need to reach out to your agent, reach out to us—Nick: We'd prefer if it was us, Zach.Zach: Yeah. [laugh]. Oh, yeah. And so, you know—because they do vary so much by premium, deductibles, copays, networks, things like that, but they will cover your prescriptions; there are ways out there to work that.Nick: Yeah. Just to add, wrapping up here, Zach, you know, one of the things that we always preach to our agents and we always tell our clients is, this is the basics of everything that has to do with Medicare, right? So, we feel that these are important, people need to have a grasp of the way that Original Medicare and prescription drug coverage works before they're really ever going to have a chance, right, to know how that secondary or that Medicare Advantage plan works.So, as you're listening to this, we've kind of been generic, right? We're covering the highlights. For those individuals that have more questions that maybe have a specific question, you know, reach out to us, 844-437-4253. We're here, we're ready to answer your questions, and we'd certainly love to hear from you.Zach: All right, folks. So, this kind of wraps up episode one here. We covered Parts A, B, and D of Medicare. We hope that that helped you out there, answered some questions for you. We tried to cover some of the real basic questions we get on a daily basis.You know, but if you do have more questions or want more information, you know, ready to sign up and looking for help, we'd be more than happy to help. You know, as Nick stated earlier, you can always give us a call at 844-437-4253, or we can always be reached by email zach@getsbi.com or nick@getsbi.com. We hope you found this episode informational and helpful, and as always, we'll catch you guys next time.Announcer: Thank you for listening, and we hope you found this episode informative. If we answered your questions, odds are you aren't the only one wanting to know, so please share this episode with your friends and family. If you enjoyed this episode, please subscribe and rate our show on Apple Podcasts, or wherever you listen to podcasts to catch all of our episodes. If you want more information, or want to talk directly with Nick and Zach, you can call them at 1-844-437-4253. You can also find them on Facebook at facebook.com/seniorbenefitinc or on their website. seniors-livinghealthy.com. Thanks for listening, and have a great day.

In part one of a two-part conversation, Oncology, Etc. hosts Patrick Loehrer (Indiana University) and David Johnson (University of Texas) have a blast speaking with two physician astronauts. Hear the incredible stories of Drs. Robert L. Satcher (MD Anderson), Ellen Baker (MD Anderson), and their lives on and off this planet. If you liked this episode, please subscribe. Learn more at https://education.asco.org, or email us at education@asco.org. TRANSCRIPT Patrick Loehrer:  Hi, I'm Patrick Loehrer. I'm Director of Global Oncology and Health Equities at Indiana University. Dave Johnson: My name's Dave Johnson. I'm at UT Southwestern in Dallas. Patrick, we're excited to be back for another segment of ASCO's Educational Podcast, Oncology, Etc. We have two very special guests today, Drs. Ellen Baker and Robert Satcher, oncologist and former astronauts. So I predict this will be an out of this world segment. Patrick Loehrer: It's starting out pretty slow right now, with that one. Thank you though.  Dave Johnson:  Well, listen, this should be a great segment. But before we get started, do you have a favorite astronaut book?  Patrick Loehrer:  Well, the one I read this summer was called American Moonshot by Douglas Brinkley, and it basically took the story of John F. Kennedy and how the space race happened from 1960 and actually earlier than that, into getting onto the moon. It was really, I thought very extraordinary.  Dave Johnson:  So, I haven't read that yet. My favorite would be Rocket Boys by Homer Hickam. I thought that was a fantastic book about his life growing up in West Virginia and ultimately, becoming a NASA engineer and rocket scientist. So that was really great. Patrick Loehrer:  I read that several years ago after you recommended it. They made a movie out of that.  Dave Johnson:  Yeah, I think October Sky was the name of the movie. Yeah. >Patrick Loehrer: Yeah, it's a terrific book. Dave Johnson: Well, why don't we introduce our guests? You want to start with Dr. Baker?  Patrick Loehrer: Oh, sure. It's my pleasure to introduce Dr. Ellen Baker. I have known Ellen over the years through interactions in global oncology. She was born in Fayetteville, North Carolina, graduated from Bayside High School in New York, got a bachelor of arts degree in geology from the State University at Buffalo. A doctorate in medicine for Cornell, masters in public health at UT Public Health, and then trained in internal medicine at UT San Antonio. And during her residency, she decided to join in the NASA program as a medical officer. And it was actually, she did a residency around the same time I did, and I remember the space call for astronauts at that time. She was selected as an astronaut in may of 1984, had a variety of jobs. She's logged more than 680 hours in space and has been a mission specialist. She retired from NASA in 2011 and is Director of the MD Anderson project, Echo Program in which he does projects in rural Texas, as well as Zambia, Mozambique, in Central and South America. It's such a great pleasure to have you here today, Ellen.  Dr. Ellen Baker:  Thank you, Patrick.  Dave Johnson:  Our other guest is Dr. Robert Satcher, currently an Associate Professor in the Department of Orthopedic Oncology Division of Surgery at MD Anderson, as well. Dr. Satcher grew up in Hampton, Virginia. He's the son of a university professor and English teacher, so I'm sure they made him do all of his homework. He received a bachelor degree, as well as a PhD in chemical engineering from the Massachusetts Institute of Technology and received his MD degree from Harvard. By the way Patrick, Dr. Satcher enrolled at MIT at age 16 and graduated at the top of his class. Later, we'll find out what you were doing at age 16? Patrick Loehrer: I do have to interject that I was a mechanical engineer at Purdue, and it's like being in high school compared to chemical engineers. Dave Johnson: Yeah, there's a hierarchy in the engineering world for sure. Dr. Satcher's medical degree was followed by internship and residency in orthopedic surgery at the University of California, San Francisco. In addition, he did a post doc research fellowship again at MIT and University of California, Berkeley. And completed a fellowship in muscular skeletal oncology at the University of Florida, before joining the faculty at Northwestern in Chicago. You guys have a lot of connections it seems? Not satisfied with that massive amount of education and training, Dr. Satcher went on to NASA to train as an astronaut, culminating in a flight aboard the space shuttle Atlantis in November of 2009. Apparently, he got a little bored cooped up in the shuttle because he took a couple of space walks, which I'd like to know more about. In fact, I read on NASA's website that he did some lubricating on one of his walks. I'm not sure what that's about, but I assume there's not a lot of gas stations in space. I also read that he repaired one of the robotic arms, which seems perfect appropriate for an orthopedic surgeon. You'll have to tell us more. Anyway, while these achievements would be enough for most people, it only begins to scratch the surface of the many accomplishments of Dr. Satcher's career to date.  I really can't cover it all, but our audience should know that Dr. Satcher's has been involved in a number of community activities, as well, ranging from his involvement in Big Brother for Youths at Risk, counseling program, to serving as a lay minister in his church. He's also undertaken a number of medical mission trips to underserved areas in countries, such as Nicaragua, Venezuela, Nigeria, and Gabon, where he once served as a Schweitzer fellow at the Albert Schweitzer Hospital in Lambaréné. Dr. Satcher, welcome to Oncology, Etc., we very much appreciate your joining us. Dr. Robert Satcher: Thanks Dave. Glad to be here. Dave Johnson: Perhaps I'll start by asking the two of you, if you could give us just a little bit about your background, other than what we've stated and what got you interested in medicine and how you got interested in space? Dr. Baker, maybe we'll start with you? Dr. Ellen Baker: I've always been interested in space. I was a child of the '60s, and I think one of the very memorable parts of the '60s was the US Space Program. It was a fairly tumultuous decade in the US, I think otherwise, and the space program was really literally a shining star. However, there were no girls in the space program at that time. Right? There were no girl astronauts and so I thought about it, but it didn't seem like it was even possible. I come from a family of medical people. My mother was a nurse, my father was a doctor, my brother is a doctor, and so that had a lot of influence, I think, in my choosing to go into medicine. And in fact, my brother is an oncologist and though I am actually not an oncologist, I've been hanging around with him long enough that some people think I'm an oncologist. So I think that's what got me pointed in that direction. Dave Johnson: Your brother is Larry Schulman, by the way, right? Humanitarian of the Year Award from ASCO, a couple years ago. Dr. Ellen Baker: It is, yes, and he is a great humanitarian. I was actually at my last year of medical school when I saw a little article in the New York Times that NASA was selecting new astronauts and women and minorities were urged to apply. And I kind of filed that away and went off to do my medicine residency and at the end of my residency, I applied to work for NASA as a physician. And that's how I got to Houston and to Johnson Space Center. Dave Johnson: Dr. Satcher, what about yourself? Dr. Robert Satcher: I got interested in space and being an astronaut when I was a kid too, and I was watching astronauts land on the moon. And I was sitting there listening what Ellen was saying and yeah, it was a time where all the astronauts were of course, white males and I didn't see myself there walking on the moon, but I knew it was something that I would really like to do, but didn't really see a way either, at the time. So it really was with the shuttle era that women, people from underrepresented groups finally had the opportunity to become astronauts. And right around the same time too, was when I was starting as an undergraduate at MIT and had the good fortune of meeting Ron McNair, who got his PhD from MIT. And that was the first time I ever saw someone who looked like me, who was an astronaut and that planted the seed that maybe I could do that one day. I also come from, as you mentioned, a family that values education and we've got a number of physicians in our family. Most famously my uncle, Dr. David Satcher, who was Surgeon General of the Clinton Administration. And I always remember, this is a funny story I like to tell if he's listening. When I was going around getting advice on whether or not I should stick with medicine or think about applying to become an astronaut, he advised me just to stick with medicine. So didn't take his advice, this is the one time I didn't, but I did take his advice, in terms of going to medical school. And I wound up applying to become an astronaut, right actually, before the Columbia accident and so they actually paused taking astronauts for a number of years. And in those years, actually that's when I went to Northwestern and was on faculty there. So they opened it up again and that's when I submitted my application and I was in Chicago and was selected from there. Dave Johnson: There's a number of questions that spill off from that. You guys may know David Wolf, who's an astronaut, who's from Indiana University. I had him in my clinic when he was a medical student and as I always do, I basically say, "What do you want to do when you grow up?" And he said, "I want to be an astronaut." And I just smiled and I don't think I ever went back in the room with him, I just figured, "This guy is a flake, what's going on with him?" And I think he's logged more space miles than just about anybody, he's an incredible human being. And again, if he's listening, you're terrific. David. Patrick Loehrer: I just have to ask this as a boyhood thing, because I did contemplate being an astronaut because I was an engineer and MD just like you were Ellen and so were you Bobby. But it hit me, one is, you have not to be afraid of heights, I think. And then, you have to be willing to go around in a circle a thousand times a minute and not throw up. But in the training there, did any of that stuff bother you or was this no big deal? Dr. Ellen Baker: That sounds a little like Hollywood and not so much like the real thing, actually. In the early days of the space program, they did put astronauts in rotating chairs. I'm not exactly sure why? But we don't do that anymore. I would say that the training is rigorous, but it's not unreasonable. I don't know what you think, Bobby? It's fairly predictable. And I will put a little shout out for Purdue, which maybe has one of the highest number of astronauts of any university outside the military academies. I don't know if that's true now, but 10 or 15 years ago, we just had a large number of engineers from Purdue who were in the astronaut office. Dr. Robert Satcher: I agree, the training is not quite how it's portrayed sometimes in the media. The question I get asked from my doctor friends a lot, "I guess you can't be claustrophobic going into space because you're in the space capsule." And that really does deter a significant number of people, probably more than myself or Ellen appreciate because I'm always surprised when I get that question. Dave Johnson: But I'm 100% sure I couldn't be an astronaut if claustrophobia is an issue because I can't stand to be in an elevator. Dr. Ellen Baker: Let me step back a little, I don't want anybody to get discouraged. But you'd be surprised at how many pilots have a fear of heights if they're not encased in an airplane. So fear of heights is not necessarily something that you might worry about if you're an astronaut. And I disagree with you Bobby, I don't think space vehicles are claustrophobic. It's not like being in a submarine, which I would find very claustrophobic. Particularly in the shuttle because we actually had great windows and great views and we didn't feel confined. I don't know what you think Bobby, but I didn't think it was like being in a cave or closet or a submarine. Dr. Robert Satcher: Yeah, I definitely agree with that, the views are spectacular. Patrick Loehrer: So William Shatner just went up and had a brief stint in space and is now considered an astronaut, but he came back with such a sense of awe. Can you describe a little bit what it's like, the view from up there and how does it leave you, particularly your first time up in space? Dr. Robert Satcher: The words don't completely capture it. The colors are so vivid, the view is so breathtaking. It's really impressive seeing the Earth, not only at daytime, but at nighttime and it goes beyond. It's one of a few things I always tell people, that really exceeded my expectation, in terms of what my preconceived notion of what it would be. And ever since going, that's what's so addictive about going into space, it makes you want to go back just to see that again and experience it again. The closest that I've come to seeing anything like that is, they have these new virtual reality experiences, where they actually got footage that they shot in high-def on the space station, outside the space station. It comes close, in terms of the look but it still doesn't fully capture it because the colors and everything, aren't as clear and crisp as it is when you're there. Because you're in the vacuum of space and there's nothing interfering with it and it's just spectacular. Dr. Ellen Baker: Yeah. I'll jump in here too and I absolutely agree with Bobby. And the other thing is, you orbit the Earth every 90 minutes, so you cover a lot of ground and you see a lot of the planet and in that one orbit, you get a sunrise and a sunset. So you orbit the Earth 16 times a day and get 16 sunrises and sunsets, so every 45 minutes the sun is either coming up or going down. It's really hard to know what time of day it is, in terms of your work day in orbit, but it's just magnificent. And it did give me a sense that we're all from the same planet and of course, it makes me wonder why we can't all get along better? Patrick Loehrer: I think people our age remember a few things, we remember the day Kennedy was shot. We remember the landing on the moon with Neil Armstrong. We also sadly, also remember the Challenger disaster and where we all were at that point in time. Ellen, you were an astronaut at that time. Dr. Ellen Baker: I was. Patrick Loehrer:> Can you reflect a little bit about that, if you can? I'm not sure I could do it without getting emotional, if I was there. Dr. Ellen Baker: Yeah, I was actually at work of course, and watching the launch and it was just devastating. Obviously, I knew everybody on board, I knew their families very well. And I have to say, we knew flying in space was risky, but we were still pretty unprepared for Challenger and for Columbia. It is something that I still think about. As weird as it might sound, I actually rescued the plant from Dick Scobee's office. He was the Commander of Challenger and I still have it growing. So it's a reminder that he was a good friend and a good colleague, it was a terrible blow, as was the Columbia accident. In that same tradition, I have African violets by my sink that were in Laurel Clark's office and Laurel was a physician who was aboard the Columbia, as well as Dave Brown, who was also a physician on the Columbia. Dave Johnson: That's an amazing tribute to both of your colleagues. It's very touching to hear that. I wonder if the two of you could share with us, what you actually did on your missions scientifically? How did you use your medical knowledge, if at all? And Robert, what about your oncologic knowledge, was that useful at all? Dr. Robert Satcher: Fortunately, no use for the oncology training in space. I got to use the orthopedic training. One of the most common problems is back pain the first few days when you're in space. We attribute it to, you're going into weightlessness and all of a sudden, there's these fluid shifts, you get more fluid in your inner discs, everybody gets taller. Dave Johnson: Well, let's send Patrick to space. Patrick Loehrer: We need to get up there soon, Dave. Dr. Robert Satcher: Yeah. Actually, during our visit, we measured everybody's height just so that we know what that change was and we did do a experiment actually, that my background came very useful for. We were looking at the effects of bisphosphonate on bone loss. So bisphosphonates, well, everybody knows, this is ASCO, a class of drugs, anti-absorptives, used in patients with bone metastasis. It was actually being tested in rats, thinking about countermeasures for keeping people from losing bone in long duration, space trips. Unfortunately, bisphosphonates come with a lot of effects. But it did seem to work, in terms of the bone mass, which is induced by weightlessness in rats. And then finally, I was the crew medical officer for my group because I was the only MD up there in our crew. Ellen probably was the same in hers. So, what that involves is, NASA was one of the first places to pioneer use of telemedicine. So, you're basically, communicating with the flight surgeons on the ground every day, talking about what sort of medical issues there are with the crew and getting their recommendations for dealing with whatever it is. So yeah, being a doctor actually came very much handy. Dave Johnson: I was going to ask both of you. During your flights, did you encounter any medical emergencies without violating HIPAA? I mean, did you have to do an appendectomy? Dr. Ellen Baker: No appendectomies and fortunately, no emergencies, astronauts are pretty well scrutinized and examined before flights. And I think the best insurance against a medical problem in space is to send crews that are really fit and really healthy. And of course, there's always the risk of things like trauma, but the risk is actually fairly low. So no emergencies on my flight. Patrick Loehrer: There's no RVUs up there are there, that you have to worry about billing or no? Dr. Robert Satcher: No, we didn't have to deal with third-party payers. Patrick Loehrer: This is looking more and more like a job I want to pursue here, I'm going to be taller and there's no billing. This is great. Dave Johnson: I want to hear from Ellen, what sort of experiments she was involved in and how her medical background helped her assist with that? Dr. Ellen Baker: Yeah. People think, "Gee, if you're a doctor and you're an astronaut, they selected you to be a doctor in space." But in fact, astronauts are selected sort of generically. So once you come into the office, you're trained as an astronaut and Bobby and I were both mission specialists, as opposed to astronaut pilots. And so we received training on all sorts of different things. Bobby did a space walk. I trained for space walks, but never did one. We trained on the remote arm. We trained how to fix things when they were broken. I was actually a flight engineer on one of my flights. But that aside, I was also the crew medical officer and on every flight astronauts participate in medical experiments, because we are trying to gather lots of information from very few people on the physiologic effects of space flight, particularly looking towards longer flights, perhaps back to the moon or to Mars. So I did participate as a subject, I was an operator in a few medical experiments, but the bulk of my responsibilities actually were not medical. And I think Bobby, maybe you would agree with that? We did have some flights that were designated as what I would call, space life sciences flights, but there was only a handful of them and I was not a crew member on any of those flights. Patrick Loehrer: Boy, I could spend another three hours on this and would love to hear more stories. Dave Johnson: This concludes part one of our interview with former NASA astronauts, Drs. Ellen Baker and Robert Satcher. Please be sure to tune in to part two of the interview, where we will learn more about the incredible work they're doing in their post NASA careers. Thank you for tuning in to Oncology, Etc., an ASCO Educational Podcast. If you have an idea for a topic or guest you would like us to interview, please email your suggestion to education@asco.org. Thank you for making Oncology, Etc. a part of your day. Announcer: Thank you for listening to the ASCO Education Podcast. To stay up to date with the latest episodes, please click subscribe, let us know what you think by leaving a review. For more information, visit the Comprehensive Education Center at education.asco.org. Announcer 2: The purpose of the as podcast is to educate and to inform. This is not a substitute for professional medical care and is not intended for use in the diagnosis or treatment of individual conditions. Guests on this podcast express their own opinions, experience and conclusions. Guests statements on the podcast do not express the opinions of ASCO. The mention of any product, service, organization, activity or therapy, should not be construed as an ASCO endorsement.

ANNOUNCER Thank you for listening. Today's episode was written by Nicole Tuttle and edited by Mariah Clawson, and featured the vocal talents of Nicole Tuttle, Micheal Porteus, Emily Buza, Ace Corsaira, Wes Haas, Daniel Young, Daniel Santoy, Griffin Coldiron, Zachary Robb, Liza Holmes, Tal Minear and Rin Moran. This episode was edited by Brad Colbroock, Mariah Clawson and Griffin Coldiron and featured the music of John Bartmann. For audio experiences that keep people listening, visit https://johnbartmann.com, and our announcer Sophia DeRise. Want to stay up to date on all things The LaFresian Chronicles: Arsen? Follow our social media for more information about upcoming episodes @lafresiaaudio on Twitter. If you enjoyed today's episode, subscribe to our show on your favorite podcasting site and leave a review. Until next time listeners. Transcript: https://docs.google.com/document/d/14Md9738_d8-ea4y8Y0ASi1ydIXK0iTlJrL8y1hpMm0Q/edit Find out more at https://arsen.pinecast.co

ANNOUNCER Thank you for listening. Today's episode was written by Nicole Tuttle and copy edited by Mariah Clawson, and featured the vocal talents of Nicole Tuttle, Micheal Porteus, Emily Buza, Ace Corsaira, Wes Haas, Daniel Young , Mariah Clawson, Daniel Santoy and Griffin Coldiron. This episode was edited by Brad Colbroock, Mariah Clawson and Griffin Coldiron and featured the music of John Bartmann. For audio experiences that keep people listening, visit https://johnbartmann.com And our announcer, Sophia DeRise. Want to stay up to date on all things The LaFresian Chronicles: Arsen? Follow our social media for more information about upcoming episodes @lafresiaaudio on Twitter. If you enjoyed today's episode, subscribe to our show on your favorite podcasting site and leave a review. Until next time listeners. Transcript: https://docs.google.com/document/d/1Z3ebYxQ9awgUxnw9djPZ8Jsq7bsRTmUedMiUwHjJQkc/edit Find out more at https://arsen.pinecast.co

ANNOUNCER Thank you for listening. Today's episode was written by Nicole Tuttle and edited by Mariah Clawson, and featured the vocal talents of Nicole Tuttle, Micheal Porteus, Emily Buza, Ace Corsaira, Wes Haas, Mike Crawford, Mariah Clawson, Griffin Coldiron, Zachary Robb and Liza Holmes. This episode was edited by Brad Colbroock, Mariah Clawson and Griffin Coldiron and featured the music of John Bartman. For audio experiences that keep people listening, visit https://johnbartmann.com. And our announcer Sophia DeRise. Want to stay up to date on all things The LaFresian Chronicles: Arsen? Follow our social media for more information about upcoming episodes @lafresiaaudio on Twitter. If you enjoyed today's episode, subscribe to our show on your favorite podcasting site and leave a review. Until next time listeners. Transcript: https://docs.google.com/document/d/11-rzrpXsHSdtyGp_BRa3JSVL-j8sFvuUuPyJu1nTUVc/edit Find out more at https://arsen.pinecast.co

Paul Casey: A good rule of thumb is before you speak, ask yourself, "Is what I'm about to say true, necessary, and kind?" T-N-K. Speaker 2: Raising the water level of leadership in the Tri-Cities of Eastern Washington, it's the Tri-Cities Influencer Podcast. Welcome to the TCI Podcast, where local leadership and self-leadership expert, Paul Casey, interviews local CEOs, entrepreneurs, and non-profit executives, to hear how they lead themselves and their teams so we can all benefit from their wisdom and experience. Here's your host, Paul Casey, of Growing Forward Services, coaching and equipping individuals and teams to spark breakthrough success. Paul Casey: It's a great day to Grow Forward. Thanks for joining me for today's episode with Ken Gosney. Ken is the Executive Director of Goodwill of the Columbia. And a fun fact about Ken is he really tries to be hip at home, but his family's like, "Not so much." So Ken, tell us a little bit about that. Ken Gosney: Well, I try to keep it cool with the kids, and the other day my wife told a story, and after she was done, I said, "Cool story, bro." I thought it was quite funny, and my kids just ripped on me and told me that was three years old and nobody says that anymore. It was a strong effort, but another failure. Paul Casey: Thanks for trying to be relevant. Ken Gosney: Yeah, keeping it real. Paul Casey: Well, we'll dive in after checking in with our Tri-Cities Influencer sponsor. Speaker 4: It's easy to delay answering uncomfortable questions like, "What happens to my assets and my loved ones when I die?" So it's no surprise that nearly 50% of Americans don't have a will, and even fewer have an estate plan. Many disabled clients worry that they don't have enough assets to set up an estate plan, but there are important options available to ensure that you have a voice in your medical and financial decision making even if your health takes a turn for the worst. Estate planning fives you a voice when your health deteriorates or after you're gone. Maren Miller Bam, Attorney at Law, is currently providing free consultations. To find out more about estate planning or to book an appointment, call Maren at 206-485-4066, or visit Salus, that's S-A-L-U-S, dash Law.com today. Speaker 4: Thank you for your support of leadership development in the tri-cities. Well, welcome Ken. I was privileged to meet you, we're thinking it's about nine years ago. My son was a freshman at Hanford High, you were the principal there, and of course I always want to get to know my kid's principal because I was a principal many years before that. I remember being in a parent-teacher meeting where we were talking about the regulations of the upcoming dance. And I realized I don't want to be in these meetings. Ken Gosney: Yeah, those were interesting meetings, what's appropriate at a school dance. Paul Casey: Yikes. So that our Tri-City Influencers can get to know you, take us through a couple of career highlights that led you to your current position. Ken Gosney: Well, when I first entered education, I was an English teacher out of Prosser High School. Paul Casey: Yeah, English. Ken Gosney: Yes. Paul Casey: That's my minor. Ken Gosney: There you go. Loved it, was teaching, and then coaching basketball and golf after school. My principal convinced me that administration might be a good thing to do, a good career. I was in the middle of getting my master's degree at that point, so I took a few extra classes, got my credentials. Low and behold, the athletic director/assistant principal blew a hole in his esophagus when he was eating Cheerios, he choked. Paul Casey: No. Ken Gosney: And so, immediately I was pulled out of the classroom, and for the next four months was an administrator at the high school, and loved it. So then took the next step of applying for jobs and was hired at Hanford High School, was there for 12 years. I then made the natural transition to Goodwill. Paul Casey: There was a little sarcasm in natural transition, right. Ken Gosney: Yeah. No, so I knew actually the former executive director at Goodwill, so when he left, the job came open, and I thought that that seems like a great opportunity and still kind of satisfied my desire to have a job where I can look in the mirror and say I'm trying to give back to the community, I'm trying to make a difference. So went for it, and yeah, the board selected me. It's been five-and-a-half years now. Paul Casey: Okay, and why do you love what you do? Ken Gosney: Well, when I was a principal, I worked heavily in the special education department and loved it. I loved working with that population of kids. You'll never meet better human beings than those kids. I still see those kids still now because Goodwill of course works with a lot of employees with disabilities or other barriers. It's great to see them still progressing, getting to see how they're doing. And I really just valued a mission that Goodwill has of changing lives through the power of work. We're able to, really on a daily basis, impact people's lives, and the lives of their family, and get out of that cycle of not having a job, and poverty, and helping people achieve some goals. It's really cool. Yeah, I can't ask for a better job, I love what I do. Paul Casey: What a great mission. I love the thrift stores too. We're thrifters, my wife and I, and my mom. I think these jeans probably were purchased at Goodwill. Ken Gosney: I hope you got a good deal on them. Paul Casey: Always, always. There was probably that moment of that decision to switch from being a principal to executive director, what was going through your mind during that? Did you do pros and cons? Ken Gosney: Absolutely. Yeah, actually I would take a lot of walks at night, I'd take the dog out, and walk for miles just going through my mind if I really wanted to leave a job I loved. I loved being the principal at Hanford High School. The high school was really humming along. I had been there long enough that I had hired the administrative team, the leadership team, and a lot of the teachers. Things were going really well, I loved it, so making that kind of a change was pretty dramatic. I had a lot of pros and cons lists I guess going in my head, a lot of talks with my wife about what we thought would be the best for the family. Really, I mean, not to get dark here, but when my youngest was six, he had a brain tumor. We were really lucky it was in a good location, the surgery was successful, but it was really scary. Ken Gosney: We were in pediatric ICU for two weeks with him. So at that point, I really kind of analyzed where I was as a parent, and I saw other people's kids more than I saw my own kids. At that point, I started looking around, but I still wanted to make sure it was a job that fulfilled my need of wanting to be a positive impact in the community. So when this job came open, I was like, "That's the one," but still it took a lot of thought to get there. But it allowed me to spend more time with my family than a high school principal position, which is just crazy hours. I mean it really is. Paul Casey: Yeah, yeah. So work-life balance was a huge, almost number one... Ken Gosney: Right, yeah. Absolutely. Paul Casey: ... on the list for that decision. Just for our listeners who might have a critical decision to make in their life, what do you tell people who are that crossroads of decision making. It could be career or it could just be another big decision in their life. What advice would you give? Ken Gosney: Well, I think you really need to analyze why you're considering the change. I could throw out the old cliché, which I truly believe, you follow your heart. But really make sure you understand why you're wanting to make that change and if it truly is right for you. I mean, money is a factor, there's no question. There's lifestyle, what are the hours? Are you working weekends? Is it straight 9:00 to 5:00? I think all those things come into play, and also where you are in your life. If you have kids, a wife, all those decisions of course have a direct impact on your family. But I think at the end of the day you do make that pros and cons list and try to make the best decision you can make with all those factors being included. Ken Gosney: Sometimes jobs I think look really attractive on the outside, and then when you really start to dive in, is this the best move? Maybe it's not. I would just say put a lot of thought into it and why you want to go into that position, what are you hoping to gain or accomplish by making that move. Paul Casey: What's the most rewarding part of being a leader for you? And then how do you stay focused on that and not the hassles, the disappointments, and the other junk of being a leader? You probably had to do that as a principal and also now. Ken Gosney: Yeah. It's really easy to get dragged down into the muck of it. And sometimes I have to remind myself I'm not perfect at it. I have to remind myself that looking at all the positive things is really important. It is important to acknowledge what you're doing well as a leader, as an organization. Not to rest on your laurels with it, but to acknowledge that there are things we're doing really well and there are things we need to improve on. But sometimes you get really lost in the negative, whether it's employee situations, you name it, right? Just like the lockdown when we were shut down for three-and-a-half months, we had to make some really difficult decisions in there and it was very frustrating because of the limited control we had on the situation. Ken Gosney: Yeah, I just think it's important to acknowledge what is going well, not to dwell on it too much, but to remember, yeah we got these significant challenges or frustrations right now, but we're doing some things that are going really well too, and we're having some really positive impacts. But again, I have to remind myself to do that because I think human nature sometimes it to just focus on. And sometimes like that five percent of the job that really exhausts you mentally, physically, whatever, it's easy to overlook the other 95% that's going fairly well because you're just focused on that 5%. And so I think sometimes it's just good to take a break and step back and say, "Okay, we're doing this and we're going to be able to handle this challenge." But and also I think surrounding yourself with people that embrace challenges and are positive themselves, you have people that are energy givers and you have people that are energy suckers and who you surround yourself with, I think is really important. Paul Casey: How do you intentionally try to celebrate those wins? Do you do it in staff meetings? Do you do it in one to ones? Is there retreats, where do you try to capture those wins and those stories? Ken Gosney: I think all of the above, I mean, we have like with our store managers, we have weekly meetings and in those meetings, there's an agenda and some of those are things we have to work on or things that aren't going well, that we have to make adjustments. We always talk about things that people are doing really well. And sometimes we actually single out a manager and said, "Hey, this manager was able to accomplish this." And acknowledge the things that our people are doing really well. Ken Gosney: And with my direct leadership team, I have five direct reports. I was acknowledged with my board. I have a board of 14, so I have 14 bosses. And they're great. They're all volunteers in the position. They really value what we do, but I always make sure to acknowledge the work that my team has done, whether individually or as a group because it's a collaborative process. And so it's easy sometimes for a leader to sometimes say, well, this is what I've done. And I try to never say that, I would say, this is what we've done, or this is what Paul did to make things work better for us. And I think that's really important that people that are working hard and being part of your team, understand that you value what they bring to the team and there's ways to acknowledge that. But I think one of the most important things is to collaborate and say, Paul, what do you think about this situation? Paul Casey: Yeah. Ken Gosney: Now at the end of the day, I have the final say, but you should be able to foster an environment of just, "Hey, let's, let's get after it here. And let's disagree." I don't want, yes men. I guess I should say yes people, but I want people to disagree with me and that's a hard environment to foster sometimes because as a leader, you're making yourself vulnerable to, "Hey, my idea, wasn't the best idea." But it also helps you come up with the best solutions. Paul Casey: Yeah. Good stuff. When I teach positive culture, I talk about acknowledging people, getting their input and communication and you hit two or three of those just right there. What makes people feel valued? Ken Gosney: Yeah. Also gives opportunities, I think for growth. I was really proud of when I was at Hanford High School, three or four of the assistant principals, I had went on to take their own buildings as lead principals. And in order to do that they have to have experience and it's a different gig going from assistant principal to a lead principal just like any leadership, right. When you take that step up and suddenly you're in charge of all of it. It's a different gig. And so providing them opportunities to learn, to grow, to fail and fail safely but learn from it. I think it's really important. Paul Casey: How are you growing these days? You mentioned growth and how have you matured as a leader just in recent years? Ken Gosney: Well, maturity is not typically a word that people use with me. But how have I grown as a leader? I think it's really important that you are always as a leader open-minded and I think that's where the most growth comes from. And I think it's important to read books on leadership and all that type of stuff. When it comes down to actually executing it, that collaborative environment allows you as a leader to really get the best information from the best people. If you hire great people and you need to let them be a part of your team and really collaborate in that environment, you learn from each other all the time. And you learn to function as a team in a high level team and that's work, doing that as a work because people have to trust each other that when they throw out an idea, even if it's disagreed with, but it's going to be done respectfully and not in a personal attack type mode. And that takes work to get there. And I think our team is there right now. We're really happy with where we are. But it took some time to get there. Paul Casey: Yeah, so one-liner on a job description of staying open-minded or creating psychological safety collaboration, but it could take, it does take a lot of work to get that. Well, you probably have a lot on your to-do list, like all leaders do, and it's probably greater than the time you have to do it. So how do you triage your tasks? How do you know what to delegate? How do you know what to focus on all that? Ken Gosney: So I'll go back to when I was a principal. I was horrible at delegating and I mean, horrible at delegating my first, probably two or three years as a lead principal. Because I was so, anxiety ridden about making sure everything was done in my world of right, right. So when my kid had that tumor and of course I was out for a month with him, I really had to let go and let my assistants and my leadership team run the show and you know what happened? Paul Casey: Nothing fell apart. Ken Gosney: Nothing fell apart. As a matter of fact, it went really smoothly and it opened my highest, a little bit of, okay, have I been, what have I been shouldering too much and too, have I not been giving opportunities to people to grow and show and develop their skillset. Paul Casey: Yeah. Ken Gosney: And so that was really an eye-opener for me that, okay, you know what, I need to trust my team to do their job and understand that they can do it and they can do it really well. And in fact, sometimes better than me. And so that was a real eye-opener and that really changed I think my perspective of how to work with my team, Paul Casey: Any delegation tips that you'd pass on to our listeners? Ken Gosney: I would say, whatever you... you know first off, if you hire, like in my current position, I have a director of finance. I'm not going to handle finance stuff. I'm going to delegate finance stuff to her, one because that's her job and two that's her training. I should not. I'm an English teacher by trade. I shouldn't be handling finances. So I delegate, if it's within their realm, I think they should be handling it. And so it's easy for me to say, "Okay, I'm going to really control this budget." But really she's part of my team and that's her expertise. So she's going to be right there with me and the rest of the team saying, "Here's where we are. This is what we can do. This is what we can't do." So I would say one of the important things with delegating is to let people do their job. If I have an athletic director as a principal, I should let the athletic director handle athletics and all the issues that can come with that. If I have one that's in charge of discipline, I should let them handle discipline. Yeah, that doesn't mean I'm not involved, but I think you let people do their jobs. Paul Casey: Yeah. Don't pull the rope back. Ken Gosney: Yeah. Paul Casey: Give them authority and responsibility. Right? Ken Gosney: Yeah. Paul Casey: Good stuff. Well, before we head into our next question on relationship building, a shout out to our sponsor. Speaker 4: Located in the Parkway, you'll find motivation, new friends and your new coworking space at Fuse. Whether you're a student just starting out or a seasoned professional, come discover all the reasons to love coworking at Fuse. Come co-work at Fuse for free on Fridays in February, enjoy free coffee or tea, Wifi, printing conference rooms, and more, and bring a friend. Fuse is where individuals and small teams come together in a thoughtfully designed resource, rich environment to get work done and grow their ideas. Comprised of professionals from varying disciplines and backgrounds. Fuse is built for hardworking, fun loving humans. Learn more about us at fusespc.com or stop by 723 The Parkway in Richland Washington. Speaker 4: So can you probably believe, like I do that leadership is relationships. Talk about what relationships are key to your success and how do you intentionally develop those? Speaker 4: Well, I would say the relationship with my, especially my direct leadership team is very important. Like you can't really overstate the importance of that relationships and the way I develop it now, of course, this is the world, according to God's name and I'm sure there's many ways to do this. I just try to make time to talk. And so one thing we frequently do we haven't released since the pandemic because that's kind of screwed up all of our scheduled regular scheduled meetings, but we would start out every meeting our weekly meeting with how was your weekend and we'd go around the table and everybody would take five, 10 minutes and talk about their weekend. Paul Casey: Right now. Ken Gosney: So you think about that, well, that's an hour to an hour and a half of our meeting. By the time you have five or six people reporting. Right? Paul Casey: Yeah. Ken Gosney: And of course it leads to questions from the other, in an engaging conversation. And you get to learn about people, you get to learn about their families, what their interests are. And so that helps to break down walls because all of a sudden now I'm viewing Paul, not just as a work colleague, but kind of a friend and somebody I trust. And so now we can have conversations that maybe a month ago we couldn't have, because I wasn't willing to trust that you were going to be okay with my ideas, whether you liked them or not. And so yeah, I think time is the biggest factor. Ken Gosney: And it's really difficult sometimes because trust me, there are days I come in and it's like, okay, I got a full list of things to do and somebody walks in my office and they want to chat. And I have to, I really have to just push that aside and focus on them because that's important. That relationship is really, really important. And it's important for them to get my attention at that moment because they've come in looking for it. And for me to shut it down and say, I don't have time would not be healthy and any way, shape or form. So yeah, it takes a lot of time. But I don't know that you could do anything more important than develop those relationships, if you really want to have a collaborative environment, Paul Casey: You said everything changed in COVID. Do you, do not check in via zoom or whatever you're doing now, or just less of it? Ken Gosney: So we still have weekly meetings, but we've changed from, now we have about 10 or 11 people in there and we're all spread out through the conference room. And we brought in people, extra people because of safety, we've got a safety person, who's making sure we're following all the safety protocols. And then, so it's almost become too big of a, "Hey, let's take 10 minutes because now we're going to be here for three hours." And really people is we've been really busy. And so now it's trying to find that balance between talking and honoring their time. And so yeah, it's become a little bit bigger, but I think we've been able to handle it well because we've already had those established relationships. And so it hasn't been as big of a deal, but we do miss it, but there, you know, I just was talking to my HR director, which I haven't seen in a couple of weeks because I have been out and she's went out. And so we took 20, 25 minutes and taught and just shooting the breeze. And so those pockets of time are still happening as they can, but it's been difficult to find the time to just say, "Hey, this is, we're going to work as a team and build relationships at this moment." So. Paul Casey: Okay. Well, self-care, essential to mental health to top performance, especially now when you made a career altering decision because of self-care and family work-life balance. So what recharges your batteries? Ken Gosney: Well, I think really at this point in this environment I enjoy my family and really enjoy going to work, especially after three and a half months where we really couldn't go to work or it was very limited in what we could do and who we could see and all that. I really came to value more than ever what we do. So I worked, does recharge my batteries. I know that's a, maybe not something you hear every day, but I do love going to work and I miss it when I'm not there. I've missed the routine of being there every day with people. But really at this point my wife and I are kind of getting two of our three kids are gone to college and then we got a sophomore. Ken Gosney: And so a lot of times it's just, my wife and I we're sitting there watching TV and we've settled into this, watching Wheel of Fortune and Jeopardy and- Paul Casey: No, you haven't gone there. Ken Gosney: And then we have our cream corn and go to bed. We enjoy each other's company and it's been good just to at night, we're not running kids all over the place anymore. And I know sometimes that's scary when couples hit that phase, but Michael, we haven't been able to run him anywhere because all of the stuff has been canceled. And so really we've been able to just enjoy each other's company. And so that's been good at night be able to sit there and relax and just talk or watch TV together or whatever. A lot of Netflix documentaries. Paul Casey: Yes. Well, you brought up your finance director earlier when we talk about finances are really, one of those big things that you have to do as an executive director to sort of keep just an overall perspective on that your board probably of course would appreciate that you're doing that. So what does that evaluation look like in your position? Ken Gosney: The evaluation of finances? Paul Casey: Yeah. Budget commences, all that kinds of stuff. Ken Gosney: So, like I said, we have 14 board members and they come from a variety of walks of life. Some of them heavily with finance background or banking, business banking. And so we meet monthly and we have to report our finances to the board on a monthly basis. And Goodwill as a nonprofit is an interesting setup, in that we have our mission side of the house and we also have our retail side of the house. And so what's interesting is we have to run the retail side like any business would run. We have all the same bills, all the same problems that any business would have. And how well we run that business is how well we can take care of our mission side of the house. Because the more money we make on the retail side, the more money we have to spend on mission. Ken Gosney: So it's really important, but it's this weird balance between we're, going to be really business savvy and we're going to be really mission savvy and compassionate savvy. And sometimes that's almost like it doesn't go together. So, it's like a compassionate way to run a business, which sometimes, I think some people would say that doesn't work. It does, but we really have to do well on the retail side for our mission side to do well. And so we have to report the finances and the finances, if they're not looking good or they're not sustainable, we would go bankrupt like any other business. Ken Gosney: So my board, yeah, they hold us accountable to making sure the finances are looking good. And if there are issues, we better have an answer for the board as to why something's happening within the finances or expenses and what our plan is to move forward with that. Now the board has been very supportive. Like I said, they're great people, they're very passionate about our mission. But they also understand that mission is funded by our retail. But they hold our feet to the fire and like I said, they know their stuff. And so there's no trying to pull the wool over the eyes of that group. They're too sharp for it. Not that we would try it anyway, but anyway, as a nonprofit, I think, especially if it's a well-run non-profit their finances are in order. If their finances aren't in order, that may spell trouble down the road for them. Paul Casey: Did you have to make any big changes when you came in? I noticed that you had some storefronts and then no longer have as many storefronts for receiving donations. I don't know if that was just an observation I made. Did you have to do some changes of what works, what doesn't work? Ken Gosney: We did collapse two stores in Kennewick into one. We built a brand new one. And we have moved or eliminated some of the donation centers where they're this standalone trailers. And so what's really interesting about that is we use the same process that Walmart would use when placing a store. So Walmart doesn't just come in and buy any old place land. They do research on demographics and what store is going to perform the best and where. We do the same thing. And because it's the business side of the house, we have to make sure that our expenses are under control and we're able to maximize our profits out of there. So yeah, sometimes we were in the wrong spot or needed to make some efficiency changes, and that's what we came up with. So that was a steep learning curve for me, by the way. Ken Gosney: Because a budget as a high school principal, I mean, basically the district gives you, "Here's some money, make sure you don't spend more than you have." You're not in charge of generating revenue. And so now being in charge of generating revenue and so minimum wage went from $9.47 when I first started and now it's $13.69. We pay $14 an hour, is our minimum. So that's a dramatic, huge, especially when you're talking about hundreds of employees, we have about 300 employees. That's a huge impact on your budget. And it's a good thing. I'm not complaining about it, but we had to make some adjustments to make sure we could handle that increase in wages. Which we've done. But if we're not looking to the future and trying to be more efficient and always on, I guess cutting edge with what we're doing, those expenses will overtake you. Paul Casey: Yeah. Well, let me just follow that rabbit trail. So strategic planning, what does that look like then say, you're always looking forward. Ken Gosney: So yeah, our strategic plan, we're looking at, how do we handle expenses, including wages benefits. We have an excellent benefits package for our employees. It's very reasonably priced and that's very important to us that they can be covered and not break the bank while doing it. Which means we cover a huge portion of that. So all those things were taken to account. Capital projects. So for instance, the Kennewick store that we built a year ago, year and a half ago, that was simply, we were leasing two facilities and it became, lease are expensive. And then when you have to fix a 25,000 HVAC unit, you're actually fixing somebody else's HVAC unit. You're putting in a brand new one and you're just paying for all of it and you don't really, it's not yours. Ken Gosney: So we started looking at building and building turned out to be cheaper substantially than leasing the two buildings. And at the end of the day, once we have it paid for, it's ours. And then it frees up all that extra money to go into mission. So yeah, I mean, we're talking five, 10, 15 years down the road is how we're looking at things. Your infrastructure has got to be healthy. I mean, we had trucks that weren't running when I first got there. So we looked at ways to get new trucks for our guys to drive. The battery start. So yeah, it's strategic planning involves our board, it involves employees, our leadership team. And we do that, it's typically a three or four year document, but it also is subject to change if COVID happens or some other situation comes up. Paul Casey: So you've got the building, the new buildings on Columbia Center Boulevard. Ken Gosney: It is. Next to Fred's Appliance. Paul Casey: And then the other building in Pasco? Ken Gosney: Yeah. The old K-mart for those that have been around for awhile. Paul Casey: Yes, Kmart. Ken Gosney: Remember that Kmart there? Paul Casey: Great. Just want to give that info. Where everybody needs to go and- Ken Gosney: And then next to Fred Meyer and Richland. Paul Casey: Next to Fred Meyer. There we go. Those are the three. Well, finally, Ken, what advice would you give to new leaders or anyone who wants to keep growing and gaining more influence? Ken Gosney: Well, I would just say don't, don't be afraid to fail. Because you're going to. And you're going to make mistakes and it's okay to say, "I made a mistake." And I go back to when I was first made a principal, the lead principal, my very first staff meeting, it was in this summer, summer was just ending and school was about to start. And one of my assistant principals was in charge, I think it had to do with lockers. I can't really remember. But he was up presenting and something had happened and we had a blow up with some of the staff. They were furious and I totally got it. And as they were talking about it, I was like, "Oh no, we missed that. We didn't catch that. That was going to be a problem." And it was too late. Ken Gosney: And so I had a to make right then and there, because I was off to the side and my assistant was up presenting this part. And so I just stepped up and said, "Hey this is my mistake. I did not consider that and I apologize." And I was ready for this backlash to now come towards me, and I really wanted it to come towards me and off the... Because really I'm the guy in charge. And what happened was the exact opposite. The anger left that room in like three seconds and it turned almost towards compassion where they're like, "We can handle that." And it was never brought up again. It was like, "Okay, we can fix this semester. This is what we'll do to handle it now." And that was a real lesson for me to learn that, you know what? You can't get up and say, "I screwed up," every day, but when you do make a mistake, I think it's important to acknowledge it. I don't think people expect leaders to be perfect. Paul Casey: That's right, yeah. Ken Gosney: And if they do, you're never going to make them happy anyway. But I think most people just want an authentic leader. And I think my advice to any leader, new, young, old whatever is to be yourself. You can steal ideas from other people, but don't try to be other people, because people will see through that. Be authentic, be who you are always lead with integrity, and I think you'll be okay. Paul Casey: Yeah. And apologies build trust. That's a good lesson. That's a good lesson. How can our listeners best connect with you? Ken Gosney: Well, through the Goodwill Industries of the Columbia website. We have email addresses listed on there for all the leadership team. And of course, we live in a great community and we've been super happy with people kept their donations for those three and a half months. And so when we were allowed to reopen, we've always appreciated the support of the community. And we've been here for over 50 years and just really looking forward to many years of being here. It's been awesome. Paul Casey: Yeah. It's still, there's a lot of great thrift stores in town, but I think the common phrase is people like, "Yeah, I'm just going to donate that to Goodwill." That's a great thing to say. Ken Gosney: And if I could just plug real quick. In 2019, we don't have the 2020 numbers yet, but in 2019 donations helped us serve over 5,000 people locally and place 900 people into jobs locally. Paul Casey: That's incredible. Ken Gosney: So those donations make a huge difference. And so again, we've really always appreciated the support, but just know that it does make a huge difference. Paul Casey: Awesome. Well, thanks for all you do to make Tri-Cities a great place, and keep leading well. Ken Gosney: All right, well, thanks for having me. Paul Casey: Let me wrap up our podcast today with a leadership resource to recommend. If you're looking for something to motivate your employees, and you're probably working remotely, I want to put you on to a YouTube site. Well, I found it on YouTube. It's from snacknation.com. It's 11 insanely powerful and motivational videos for employees. So they've just taken some of these best motivational talks by Brendon Burchard, Daniel Pink, Les Brown, Shawn ACOR, and they've put them in these little bite-sized nuggets that you could play at the beginning of a staff meeting. Paul Casey: So again, it's snacknation.com, 11 insanely powerful and motivational videos for your employees. Again, this is Paul Casey. I want to thank my guest, Ken Gosney from Goodwill of the Columbia for being here today on the Tri-Cities Influencer Podcast. And we want to thank our TCI sponsor and invite you to support them. We appreciate you making this possible so we can collaborate to inspire leaders in our community. Finally, one more leadership tidbit for the road to help you make a difference in your circle of influence. As a leader, you are a stage procurer, not a perfumer. Until next time, TGF! Keep Growing Forward. Announcer: Thank you to our listeners for tuning in to today's show. Paul Casey is on a mission to add value to leaders by providing practical and strategies that reduce stress in their lives and on their teams, so that they can enjoy life and leadership and experience their key desired results. If you'd like more help from Paul and your leadership development, connect with him at growingforward@paulcasey.org, for a consultation that can help you move past your current challenges and create a strategy for growing your life or your team forward. Announcer: Paul would also like to help you restore your sanity to your crazy schedule and getting your priorities done every day by offering you his free control-my-calendar checklist. Go to www.takebackmycalendar.com for that productivity tool or open a text message to 72000, and type the word Grown. Paul Casey: Tri-Cities Influencer Podcast was recorded at Fuse SPC by Bill Wagner of Safe Strategies.

ANNOUNCER Thank you for listening. Today’s episode was written by Nicole Tuttle and edited by Mariah Clawson, and featured the vocal talents of Nicole Tuttle, Micheal Porteus, Emily Buza, Ace Corsaira, Wes Haas, Mike Crawford, Daniel Santoy, Mimi Brown and Rin Moran. This episode was edited by Brad Colbroock, Mariah Clawson and Griffin Coldiron and featured the music of John Bartman and our announcer Sophia DeRise. Want to stay up to date on all things Arsen? Follow our social media for more information about upcoming episodes @arsenaudio on Twitter. If you enjoyed today's episode, subscribe to our show on your favorite podcasting site and leave a review. Until next time listeners. transcript: https://docs.google.com/document/d/1-XOluGWfZ9Zr5qIbsNjdZeyR6MzK_O3GZNT-lZ2JBUo/edit?usp=sharing Find out more at https://arsen.pinecast.co

Matt Browner Hamlin is an activist, organizer, and writer residing in Washington DC. He is the Head of Engagement Strategy & Planning for Greenpeace International. He has previously worked as He is the Managing Director for Digital Strategy of Ethical Electric, a Senior Economic Strategist at Citizen Engagement Lab and an organizer of http://occupyourhomes.org/ (OccupyOurHomes.org). He was the Deputy New Media Director of SEIU, the Deputy Internet Director on Chris Dodd's presidential campaign, and the Internet Director on Mark Begich's US Senate campaign in Alaska. Matt blogs about politics at http://holdfastblog.com/ (Hold Fast), cocktails, bars, and spirits at http://ajiggerofblog.com/ (A Jigger of Blog), and travel and technology at http://bloggerhamlin.wordpress.com/ (Blogger Hamlin). He has also been a contributor at http://www.americablog.com/ (AMERICAblog). The views and opinions presented in Matt's posts are his alone. They are not representative of the policies or opinions of his employer or past clients. [INTRODUCTION] [00:00:07] ANNOUNCER: Thank you for joining the conversation on Colloquium. This episode is brought to you by Excelsior Capital, an investment platform focused on democratizing private equity by providing individuals access to direct opportunities. To learn more about the firm, please visit excelsiorgp.com and connect with Bryan on LinkedIn. [INTERVIEW] [00:00:07] BA: Welcome to the conversation on Colloquium. Today I’ve got an old friend of mine, Matt Browner-Hamlin with me. Matt, how are you today? [00:00:37] MBH: I’m doing great, Bryan. How are you? [00:00:38] BA: Good. I’m going to give a little bit of kind of the bio and then we'll get into some of the work that you've been doing. And as context, Matt and I went to college together a long time ago and was a good friend of mine in school. And then as many of us in our kind of pre-Facebook generation, just kind of lost track of each other frankly. And it's been probably a good 10 years since we last spoke. But it's fun to get reconnected. And you've been doing some really interesting things. So I’m excited to get into it. So his background, Matt is an accomplished technology strategist, data-driven campaigner, digital marketer and writer. He is currently the interim chief technology officer at Greenpeace International. Greenpeace is a global independent campaigning organization with offices in 55 countries. Matt has previously worked for a direct consumer renewable energy startup, international human rights, labor economic justice campaigning context, as well as in American politics and presidential incentive campaigns, as well as some other interesting blogging and content creation that you've made over the years, which we'll probably get into.  But, first, I really want to dig into kind of Greenpeace itself. I think a lot of people in our minds think of the efforts they had against Japanese whaling and the kind of images of television. But it's actually a much older organization. Can you maybe talk a little bit about how you found yourself at Greenpeace and give folks a little bit more context about the originations of the organization and what the focus is today? [00:02:11] MBH: Yeah, sure thing. And, first, thanks for having me on, Bryan, and it's great to reconnect on this occasion. It's been a long time. And I think a lot of us are using the COVID times to build back old bridges and have conversations that are probably long overdue, but easily slip past when it's hard in our regular life. So thanks for having me on. [00:02:28] BA: Yeah. Absolutely, man. [00:02:29] MBH: So Greenpeace is about to turn 50. I think in 2021 is our 50th anniversary. And the organization has its roots in anti-nuclear campaigning. And the story of its founding actually goes back to a bunch of activists in Canada who were really opposed to ongoing U.S. nuclear testing in...

Tara Jaraysi Kenning: "Teamwork makes the dream work." John C. Maxwell. I'm Tara Jaraysi Kenning, and I'm a Tri-Cities influencer. Paul Casey: So to be a go-to guy or a go-to girl, you must push through your fear of failure. Announcer: Raising the water level of leadership in the Tri-Cities of Eastern Washington, it's the Tri-Cities Influencer Podcast. Welcome to the TCI podcast, where local leadership and self-leadership expert, Paul Casey, interviews local CEOs, entrepreneurs, and nonprofit executives to hear how they lead themselves and their teams so we can all benefit from their wisdom and experience. Here's your host, Paul Casey of Growing Forward Services, coaching and equipping individuals and teams to spark breakthrough success. Paul Casey: It's a great day to grow forward! Thanks for joining me for today's episode with Michelle Whitney. She's the superintendent of the Pasco School District. And fun fact about her, she's sort of a driver. So she said she can do a two-minute meditation a lot faster! Michelle, tell us about that. Michelle Whitney: Well, I just think it's about efficiency. You've got a lot to do. Two-minute meditation, we do it in 30 seconds. We move on to the work. We have things to do. I don't have time for that. Paul Casey: And her staff teases her about how fast she walks. Michelle Whitney: Right. Paul Casey: It's with intention. Well, we're going to dive in after checking in with our Tri-City Influencer sponsor, Mario Martinez, Northwestern Mutual. Mario what types of services do you offer? Mario Martinez: Hey, Paul. Thank you for letting me be on here. We run bifurcated practices in that we focus in two areas of financial plan. The first one is we do protection pieces, which include life insurance, disability insurance, long-term care insurance, really the things that people should be focused on to protect their families, their businesses. On the other side of our practices, we do investment services. On the investment platforms, we do both the brokerage platform, and we do the advisory level services. So depending on what someone's looking for as far as guidance on their investment strategies, we can curtail and build a strategy for them that makes sense. Paul Casey: Mario, how can people get in touch with you? Mario Martinez: The easiest way, you can reach out to me directly on my business cell phone is 509-591-5301. You can send me an email at mario.martinez@nm.com. Or you can reach out to us on our social media platforms, the easiest one being Mario Martinez Northwestern Mutual on Facebook. Paul Casey: Thank you for your support of leadership development in the Tri-Cities. Well, welcome, Michelle! I was privileged to meet you seven years ago in Leadership Tri-Cities. You were class- Michelle Whitney: 18. Paul Casey: And another one of the best classes ever, right? Michelle Whitney: That's right. We were actually the best class. Paul Casey: Oh, okay. I see. As opposed to number 11, which really was. Yes, and you had a different job at that time in the school district. You had gone from being a middle school principal into HR. Michelle Whitney: Yeah. Everyone was glad to see me when I was a middle school principal, and that changed drastically when I became director of HR. So it was tough transition. Paul Casey: HR managers, we love you! Thanks for listening. So Michelle, tell us your career highlights that got you to where you are today so our Tri-City influencers can get to know you and why you love what you do. Michelle Whitney: Well, I appreciate that question. I have a huge commitment to public service. My grandparents were both public servants. My grandfather was a police officer and chief of police of Pasco. My great-grandmother was a nurse at Our Lady of Lourdes, so I really come from a foundation in my family of civic service, and in particular, civic service to the community of Pasco. So I always wanted to be a teacher, and it was only right for me to tailor my education to be able to come back to a community that I loved and that my family had served in such amazing roles. And to be important in the lives of the students in Pasco has just been a gift. Michelle Whitney: And I've been extraordinarily blessed to serve in a variety of roles. I was a kindergarten teacher. I taught fourth grade. I was a counselor. I was a technology facilitator, a librarian. Paul Casey: Wow. Michelle Whitney: And it was always really just about offering my unique skills and talents to the organization so that I could be of best use to the organization and the kids of Pasco. Michelle Whitney: And then I really started a leadership journey, which I never intended to end up in a leadership role, but I was invited to the leadership table, if you will, by a mentor of mine. And I just feel a lot of gratitude towards the amazing mentors I've had over my career. I never saw myself that way, and it was by someone else recognizing those leadership talents in me and encouraging me in that direction that I took the risk to do an administrative internship program, became an assistant principal and then principal at the middle school. Then that started the trajectory into the district office with director of HR, which, having been a middle school principal and a counseling background, I was really uniquely prepared to do that job. Paul Casey: Yeah! Michelle Whitney: And I actually really loved that work. People joke with HR managers, but I do believe that in any organization, being able to be at the front line of hiring talent into an organization is just an extraordinary opportunity and gift. So that was amazing, and then I stepped into some other district office jobs. Michelle Whitney: And again, I would love to tell you I had this trajectory. I was going to teach kindergarten and then be the superintendent, but it wasn't really like that. And the opportunity to apply for the superintendency became open, and it really was about me throwing my name into the hat for a leadership position in a district I loved my entire life. And to be awarded that position was one of the greatest moments and has continued to be great moments since I was awarded the position. Michelle Whitney: I'm going into my fifth year. Every single year has had its complications. Every single year, I've been proud to be on the team and honored to do my part. So I have the best job ever, and the best part of my job is the students for sure. So sorry adults, you're second. Kids will always be first. Paul Casey: And I heard you tear up pretty easily when you think about those wonderful students. Michelle Whitney: I do. I do. In our organization, it's not uncommon for me to stand in front of a group of people, and I say, "We are," and the response is, "Pasco." It really for us is an outward commitment to our value of standing in the gap and bridging the divide for the students who need us the most. So when I'm in front of students or I'm in front of staff, it is not uncommon for me to be emotional about it because it isn't just a job for me. It truly me living that outward commitment every day. Michelle Whitney: And even when it's hard, I feel so extraordinarily blessed to be able to be part of what we're doing. And then when it's great and there's a success, those successes are just that much sweeter, and they really do truly move me to tears. Even talking to you about it, I get goosebumps. I live my purpose every day. And they say if you live your purpose, you never work a day in your life, and I truly feel like that I'm lucky in that way. Paul Casey: I think we're done here. That was an amazing- Michelle Whitney: Well, there you go. See? Efficiency. Paul Casey: ... story. Michelle Whitney: I told you. Paul Casey: That was an amazing story. Now I want to cry too. But living your purpose, so huge. Obviously, that's why I do what I do as a coach is to help people do exactly what you are feeling right now, so that's awesome. Paul Casey: Let's go back to that crossroads where you are going to take the jump into leadership or not, and you decided to move that direction. What helped you make that decision, and also what advice would you give someone else who might also be at a crossroads? "Should I take the jump into leadership, or should I just stay as a individual performer that I'm doing really well at?" Michelle Whitney: Right. So I think what helped me make that jump and take that risk was the courage of the support of the mentorship I had. So had it not been for Jean Carlton, who was the person who very first invited me into a leadership role, she really stood shoulder-to-shoulder with me in those early years in making sense of who I would be as a leader. And without that personal connection with her, I'm not sure I would have had the courage to take that first step. So I think that, as a leader now, I find that one of my purposes is to recognize other leaders and invite them in and stand shoulder-to-shoulder with them. I don't think you can ever underestimate the power of somebody with that invitation and that offer of support.... I could give you a list of 100 people that have been amazing mentors to me. And I just I think it's that support from trusted mentors that gave me the courage to do it. Michelle Whitney: Really as I reflect back on it now as having had some experience and I hope some wisdom, there's really, all of us, no matter what role you play, you're a leader in your own right. So there's never really a downside to leaning in to the desire to explore that leadership characteristics of yourself. The worst thing that's going to happen is you're going to learn some great skills and learn some things about yourself that will make you even better at what you're currently doing. But if you take that little bit of a risk, and maybe it's just a toe dip, it doesn't have to be a full jump at first, you'll likely find that it's a good fit for you. And then that success breeds success, and the more you try it out a little, and it's like you put the sweater on and it fits pretty good so you wear it around a little bit. But I would just encourage people, especially if you're nervous, find a trusted mentor, start slow, start small, but just continue to take those steps, and it's only going to make you better at whatever you choose to do. Michelle Whitney: You may never choose to be the person that is the front of a large organization, but leadership skills, regardless of what you choose to do, will always just make you a better contributor. And that's really what we are as leaders is contributors. So I would encourage anyone to take the risk if you're thinking about it, and again, find a trusted mentor that you can lean on because there are times where it's challenging, and you'll have self-doubts and having that mentor you can go to and be vulnerable about that is very important. Paul Casey: Fantastic answer because leadership is influence wherever you're at, and it will fill up your game no matter what you're doing. And I love how you said mentorship was a courage builder for you, even up to 100 people, which is probably true. It's probably not hyperbole. There's just a ton of people that we would not have taken that extra step had it not been for someone giving us that boost and saying you can do it. Maybe you're further along your journey, and you're like you need to turn around and bring somebody with you and mentor them, even if it's informal and you don't call it mentoring, but you want to help somebody along their journey. Paul Casey: Like you said, in five years, you've had a lot of issues to deal with in your position. You're smiling still! That's good. A lot of hassles, a lot of disappointments, a lot of things that get in the news making some people choose one side of an issue or another. But then there's the rewarding part of the job. We're not going to go into those other places. Here's the rewarding question, What allows you to focus then on those most rewarding things, and what is actually the most rewarding part of your job? Michelle Whitney: Well, I was very serious when I said the most rewarding part of my job is students. And, from the very first day that I started as superintendent and actually even prior during my successor year, I had a transition year, which was gift, I prioritized being in classrooms. There is nothing more magical than the relationship between students and their teachers. So I scheduled on the calendar Mondays, Wednesdays, and Thursdays every morning when we were in person to start in classroom. Michelle Whitney: Most days I would get there. Some days I wouldn't. I got there more than I would if I didn't schedule it in. Some days I could stay a long time. Some days I could only be in one or two classrooms, but I always prioritized that. And it does a lot of different things. Michelle Whitney: One, it keeps me connected to what teaching means now. It's been a while since I've been in a classroom, and teaching is not the same as when I was in a classroom. So I think that's very important to stay in touch with those you're in service to so you can do right by them in your decision-making. Michelle Whitney: But there is nothing that feeds my soul more than students. So in this pandemic environment, I've done the same thing in that I am scheduled to go into Zoom classrooms. And one morning, I was in a kindergarten classroom, and they were doing this good morning routine where they would say good morning to one another. And of course, I'm in tears. There's nothing more sweet than kindergartners saying good morning to each other by name. And it's those moments where you can really connect with why we do our work, the sweetest kindergarten doing the most genuine thing by saying good morning to one another reminds you that those hard moments are worth it, that there's students depending on you, and that there's this kindness in our system. That's what makes the hard moments for me worth it is I stay connected to those things that are closest to students, students, teachers in the classrooms. Michelle Whitney: And I have to tell you, that's what's made this pandemic so challenging is we're thrust into the really hard parts of our job for most of our job, and you have to be a lot more intentional about getting out and participating in those things that the really feel your soul, feed your bucket, whatever those things are that you say. For me, I have never lost sight of what it means to be a teacher. I walk around with a teacher's heart, and I think that's why I cry is it touches that part of me that is so important. And like I said, I stay really closely connected to students, and that for me is the difference in those dark moments. Paul Casey: So cool that you kept your goal of being physically present, even, well, Zoom is not physical, but you still kept that goal alive. You found a way to still be in classrooms, even though it's online in order to do that. That's pretty neat. Michelle Whitney: Yeah. It's incredible. I get to read stories, and the kids bring their puppies and baby sisters to the Zoom. So in so many ways, you get to experience even more of a student's life. So it's been a gift really. And while in-person education is what we're about and who we're about, there really truly have been some silver linings to this environment. Paul Casey: Mm-hmm (affirmative). And I was an elementary principal so I totally get teacher, vice principal, principal, the journey. And playing with the kids at recess was a bunch of fun. I even sprained my ankle one year playing freeze tag. Michelle Whitney: Mine was flag football. Paul Casey: Was it? Michelle Whitney: So yes. I, yeah. Paul Casey: It's like the scar of courage- Michelle Whitney: Yeah, it's a rite of passage. Yeah, uh-huh (affirmative). Paul Casey: But that is where the joy was for sure. So leaders must keep growing or they become irrelevant. How have you matured as a leader, I'll just say in these five years of being superintendent? Michelle Whitney: Oh, that's a great question. I think the number one way that I've matured as a leader is by listening. I do a lot of listening to the people that I'm in service to. So I meet with parents a lot. I've done lots of different ways of doing that. Of course, pre-pandemic I did community coffee events. I've done Soup with the Superintendent. Since the pandemic, I've done some virtual town hall meetings. You go out and listen, but it's listen with intention. And it's really listening to understand those you're in service to, what they need from you, and reflecting on those decisions that you need to make and the kind of leader you need to be on behalf of those you're in service to. Michelle Whitney: I also do a lot of listening to teachers and staff in a lot of the same ways by being present, and I have the most amazing thing. I have a superintendent student advisory council. So I have 30 high school students that meet with me six times a year, and we tackle really difficult issues together, whether it's planning for a new high school or changing boundaries or overcrowding issues or social, emotional, health issues. Again, it's about being present with them and listening to them and tailoring my leadership and the way that I make decisions and what I prioritize and focus on in a way that they need me to do right by them on their behalf. Michelle Whitney: I would love to tell you it was some professional development training I went to or some class I took, but it really wasn't. It was being present in my system and listening to those that I've made a commitment to represent and learning from them how I can be better and different in order to serve them in the best possible way. Michelle Whitney: High school kids know what they need from us, and know what they need from us around very complex issues. The most diligent group of people that I saw work on a set of boundaries in a couple of facilities plan was that superintendent student advisory council. Michelle Whitney: So I joke that really we should let the students run the place. I just need to buy lunch and give them a ride. They truly do know what they need from us, and we just have to create opportunity for them to participate. And it's in those moments where I feel like I'm my best self and I'm my best leader for them. And I feel like it's a gift. Paul Casey: Learning by listening. Yeah, I was part of a group in Richland School District. I feel like it was called Focus years ago. I don't know if it still exists today. I was part of the faith community then, too, to come in, and it was all the folks that cared about students and could provide auxiliary services. Plus, the principals were in there. Plus, these cream of the crop students were in there, and they blew me out of the water! The maturity of speaking in front of these scary adults, and they're holding their own and saying, "These are the programs that we got growing. These are the thing we needed." I could totally see how that would be an energizing meeting for you. Michelle Whitney: Oh, it's incredible. Paul Casey: Just to further validate the whole listening post, I had a boss who do a listening post, he would call it. He would invite a dozen of the constituent monthly, and he would ask the same set of questions to each one to hear on the ground level of how to serve them in a nonprofit. And then just yesterday, I'm part of the National Speaker Association, and a board member called me from the Northwest Chapter and just said, "I'm a board member, and I'm just trying to get ground-level intel of how we can best serve our constituency." And I was like, "Wow!" He followed up with an email, and we even played with a couple of ideas for the pandemic of how to speak virtually. I'm like, I think it's always a great move when a leader gets down on the ground with the frontline people, who know all the answers, like you said, they how to run the school, and listen to them. Paul Casey: Well, as a superintendent, it must be hard, with a huge to-do list and probably a billion emails coming in, to know how to spend your time, how to triage tasks, how to know what to delegate and what you have to own. How do you sort how to spend your time? Michelle Whitney: Yeah. That's a great question, and I'm going to be really honest with you because if I'm not, the people that know me will call me on that. That probably is my biggest weakness, because I believe so deeply in every single person that I'm in service to, and we have a large organization, almost 20,000 students, which represent 40 to 60,000 parents, 2,200 employees, five board members. And every single one of those interactions is important to me. So the whole delegation and all of that is absolutely a work in progress. Michelle Whitney: For me, I think one of the biggest pieces that I'm blessed with, and this may sound cliché, but it's absolutely critical is to have an executive assistant that you trust. My assistant is absolutely incredible. She knows what I need to work well. She knows what I need as a person to function well. She's my number one fan. I'm fairly certain I'm her favorite person in the universe above and beyond everyone else. And on those hard days, I know for sure that my assistant still likes me. That's just that, not only does she help me organize myself professionally, she makes sure I'm where I'm at with what I need, my calendar is organized and squared away, but she's also that person that's there to bolster and support on the days when days are tough. Michelle Whitney: And I think that executive assistant and, for me, superintendent of management relationship is so important to have a person that you can go, "Oh my gosh. I don't know how to... I'm not going to be able to get all this done," and something that you can delegate and help follow through with those priorities. Michelle Whitney: The other piece is I have a great team. And we're learning about the strengths of the team together over the last five years. We've utilized a book called StrengthsFinder 2.0. And that was really a game changer in terms of knowing and understanding where people fell out around those strength characteristics and really now trying to organize work function up against and those and trying to collapse some of the siloed nature of the way work is typically done in a large organization to be able to align work tasks with people's strengths. Michelle Whitney: So I'm not a great executor. That's not my strength. I'm big vision, relationships, but the details of beginning to end is not my strength, but I know I have a colleague and a teammate that that is her strength. So when I need help with that task, I go to that person. I think knowing the strengths and talents of the people around you is a real efficiency builder, and then having someone like an assistant like I have, Jenny is amazing, that really can help you prioritize, especially for someone like me that everything is equally important all the time. And that's just not viable in the long term. You can do that for a short period of time, but at some point, you need some help in making sure that you're keeping the right things up front all the time. Paul Casey: What do those meetings with Jenny look like? How do you sort, prioritize? What do you discuss? Is it regimented? Is it ad hoc? What does it look like? Michelle Whitney: Yeah. That's a great question. I think with Jenny and I, it's evolved over time. We started out with it being scheduled that if I get to go to a kid thing or meet with Jenny on virtual. Paul Casey: That's going to bump it. Yeah. Michelle Whitney: Yeah. So Jenny was getting bumped all the time. Then it became more ad hoc. Lately, what's been amazing about this virtual environment is she just comes to the meeting, whatever meeting I'm in that I feel like I'm going to need her in, she is in there virtually, and she can listen to the meeting and help me prioritize that way. My thinking post-pandemic, when we're back to a more normal environment, I think that organization works best for us. So those meetings that she can be at with me or helping facilitate or there as a note-taker, I think that, we've landed on that arrangement for us probably works better. Michelle Whitney: But I think the key to all of that is you have to figure it out for yourself. That management system with your assistant is going to be very assistant-and-manager-specific in terms of how your personality works. So the key is finding something that works. I don't think it has to look any single way, but it has to work for both of you. Paul Casey: Sounds good. Well, before we head into our next question on external relationships, a shout-out to our sponsor. Mario Martinez, Northwestern Mutual. Mario, why should people work with a financial advisor? Mario Martinez: Hey, Paul. That's a great question. Really, I think there's two types of people who should be seeking out a financial professional. The one person is somebody who has very limited access to financial guidance. Maybe they're a younger professional or somebody who just hasn't had an introduction to a financial professional yet. And the other type of person is really someone who has a lot of different exposure to different professionals. They just haven't found the one person that they really trust to take guidance from. So there's really an over-information in that sense. Those are really the two types of people that should be looking to be introduced to a financial professional. Paul Casey: Fantastic! So Mario, how can people get in touch with you? Mario Martinez: The easiest way is to reach out to me directly on my business cell phone, which is 509-591-5301. You can send an email to mario.martinez@nm.com, or you can find us on our business Facebook page, which is Mario Martinez Northwestern Mutual. Paul Casey: So it's been very clear, Michelle, that leadership is relationships for you, that you believe that like I do. You've talked about a lot of internal relationships within the school and school system. Now you've got this community around you. So how do you intentionally develop relationships with the City of Pasco and beyond? Michelle Whitney: That's a great question. I feel very lucky. The City of Pasco, the city manager, Dave Zabell in Pasco, we have what's called a Create Group. It's the port, the city, the public utility, the county. I'm sure I'm leaving someone out, but it's the leadership of the infrastructure of Pasco. And we meet monthly, and we keep each other updated on those core functions that would overlap. So for me, I often report out on facilities or potential construction projects. They're also very interested in our educational programming. So we give updates there. That, I think, is an incredible unique opportunity for us to partner as a collective for the good of our community. Michelle Whitney: I'm also a Kiwanian, and we have representation at all of the local groups like the Chamber of Commerce and so on. I think that's very important. Maybe it's not always me that's at those tables, but there's someone from our organization at those tables that can act as a liaison. Michelle Whitney: The other piece, though, for me is, like I mentioned earlier, those community coffees, that I do them monthly. I've done that almost consistently in at least the last four years if not five in some variation, whether it's at the Starbucks, or we did Soup with the Supe at the Booth Building one time. But that really is just an open invitation to anyone who wants to come and sit knee-to-knee, shoulder-to-shoulder with me. And we talk about whatever they want. It's not a pre-canned presentation that I do. They just bring topics and discussion and we talk about, and I answer whatever questions there are. I think those are the relationships, too. Michelle Whitney: So there's the organized groups that you would expect an organizational leadership to be interacting in. Certainly we participate in all of those, but there are people out there who want to build relationship or want to be in a relationship with the school district and don't maybe know how, or maybe don't know what they want to do. That's where those more informal opportunities like a community coffee come in really handy for people who just come and say, "Hey, here's a skill or talent I have. I want to get involved. How could that look?" And then I just am a big connector at that point, and I'm like, "Hey, you need to get in contact with that person." The night Jenny's there, and I say, "Hey, Jenny, will you get them in contact?" Then she takes care of making those connections. But it's that central location at the Starbucks community coffee that they know the superintendent will be there and listen. Michelle Whitney: Then over the last year or so, I've started to invite some of my executive team there, too, so that there's other people other than just me there that I can connect folks to. So that's been a great relationship builder. Then we have phenomenal programs like our PEAK! Partnership Program where I have a staff who go out and talk to organizations in the community that want to partner directly with schools. So we have some real innovative organized efforts like that as well. Michelle Whitney: We don't do the work for educating kids on our own. It's truly a system-wide and a community-wide effort to build that network of support and lift all of our students. And I'm proud to work shoulder-to-shoulder with the best in the community. Paul Casey: Well, Tri-City Influencer listeners, Soup with the Supe, so you could steal that idea if you're a supervisor- Michelle Whitney: There you go! Paul Casey: ... not just a superintendent. Michelle Whitney: That's right! Paul Casey: And post-COVID you can have with you. I love it! So we rarely talk about money on this podcast, but every leader has to know their organization's financials. I know you've got a whole finance department, I would assume, within the district. But what do you have to stay accountable to? What does evaluation of finances look like in your position? Michelle Whitney: Yeah, that's a great question. Education finance and funding has been in a state of flux over the last three or four years…Probably longer than that, but since I've been in the superintendency, it's been in a flux and change post-McCleary. So I've had to stay very in tune with learning all the new legislation and how the laws are impacted because it was a complete shift in the way things were done. One, it was legislative advocacy around the McCleary decision, pre-McCleary decision, and then post-McCleary, it was about learning and understanding those aspects of the change in the way education funding was done. Michelle Whitney: But then it's really about my interfacing with the school board to build some benchmarks and goals and priorities for a philosophy about the way we spend our dollars. So my interface with the school board to ensure that we're building budgets that are in alignment with their philosophy and their goals and to meet our strategic planning priorities. So that's really the level that my leadership and influence is at is making sure that the way that we're budgeting and prioritizing dollars is in alignment with our strategic plan and our board's vision for our district. Michelle Whitney: And then I work very closely with our business office. I typically supervise directly assistant superintendents. My business officer is an executive director, but I directly supervise him because he is in charge of the district finances. And that was personal decision that I made. Not all superintendents or supervisors do it that way, but one, he was new. The funding was new, and I was new. So we all are learning together, and it's again, that spending time together, knee-to-knee, shoulder-to-shoulder learning is the best way, I think, to build team and accountability together, especially around something as important as finances. Michelle Whitney: We also do a fair amount, or I do a fair amount of interfacing with the community around things like bond planning and levy planning. And we do that through community task forces. And I had an assistant superintendent who did an extraordinary job in interfacing with a group called the Community Builders who helped us plan out a district strategic plan around our facilities. And then of course, layered on top of that is bonds and how those cycles will work. So it's really a multi-tiered effort as a superintendent in a school district, and there's a variety of stakeholders who are involved in different ways in education and finance decision-making. I think it's a really fascinating part of the work. It's a very complicated part of the work. Michelle Whitney: Recently, in the last couple of years, we've had to make some adjustments because the McCleary funding that would require us to repurpose dollars and really there were some areas that we were going to need to reduce. And we got teachers very involved, staff, building-level staff very involved in that. I went out and did a budget presentation, actually two presentations at every single building, and then invited people to come to the table to help us build some budget efficiencies. So we're really working hard to broaden the ownership and leadership around educational finance in Pasco. And we just started to get traction around that work pre-COVID, and then of course, COVID hit. So once we get back to something that appears a little more predictable, we'll get back to that. Michelle Whitney: But I felt like that was an extraordinary opportunity to peel back the curtain of finance of any big organization is complicated. I'm really excited about picking that work back up. Paul Casey: You mentioned strategic planning. You also mentioned in your StrengthsFinder. Vision is huge for you, by the way, huge StrengthsFinder fan. It's called CliftonStrengths now, but Tri-City Influencer listeners, please look up StrengthsFinder 2.0 the book or CliftonStrengths. There's a $19.99 version which will give you your top five. You'll be like, "Someone's reading my mail when you look at it." You're like, "Wow! This is what I love." And if you can be doing that 80% of your work day, you're going to love what you do. And talk to your supervisor about that. You can do it as a whole team. Paul Casey: I can facilitate that for you. It would just be a super fun thing to realize, "Wow, you're so good at that. You should be doing more of that!" And, "Wow, you don't do good at that? What's a way that we can move that around on a team?" Paul Casey: So when you think about the next hill to climb as a district, and you think about continuous improvement, what's your process for that? Michelle Whitney: Well, right now our focus really was derailed in the face of and the impacts of COVID. My eye now is on transitioning our students back into some kind of in-person learning that's safe for them over the course of the next few months and then really getting specific and intentional about how we bridge the impact of the disruption to their educational experience. Michelle Whitney: So we had a really nice trajectory of focus starting in my first year with identifying some outrageous outcomes and strategic plans, and we just refreshed and got a brand new strategic plan approved in January before COVID. So those things really are push-pinned on the bulletin board right now as we're getting really hyper-focused on how do we met the needs of our kids during a pandemic in this just extraordinarily complicated environment. Michelle Whitney: We will need to get back to that. Matter of fact, in January, we'll start again with the board re-calibrating our expectations of the strategic plan, and it will then be focusing on what is life after, post-COVID, or as we start to get kids back. It will be about how do we fill those gaps that have been created by a disruption in the traditional education environment. So those processes for me are done in coordination and collaboration with stakeholders that are closest to the work. So we involve our teachers and our students, and there's a lot of listening that happens and serving that happens and focus groups and task force that happen. So I would envision those things starting to occur. Michelle Whitney: Once we get back to something that's a little bit more predictable, right now, people's priority and focus is dealing with the current crisis. And it truly is still a crisis for us. We've been in a crisis mode since March 13th, so it's really difficult to get people to, and really probably inappropriate to try to get people to think about something more long range. It's like you wouldn't be thinking of building your next house while your current house is on fire. So we really just need to honor where we are right now and know that there'll be work to be done when we get back to something that's a little bit more normal. Michelle Whitney: But my process is really around valuing those closest to the work. They know what they need from us, empowering them in the decision-making, and then being really specific and strategic and intentional about a few key priorities, and I think that was mistake I made early on as a leader is taking on too many things. I'm a person that likes to do too many things, but organizations don't. And I don't mean any single person in an organization. Just systems don't. So when you think about a system like a rubber band, if you stretch it too tight, it'll break, and systems are the same way. So that's been a huge learning piece for me as an individual leader, that you really have to be keyed into systems don't work the same way as you do as an individual leader. Michelle Whitney: So part my leadership responsibility is to help narrow people's focus so that we can get really good at a few things versus having our attention spread across a lot of different things. And I'll tell you, we're not quite there yet as an organization, but we certainly have our eye on that. And this strategic plan that we had in place pre-COVID puts us in a nice spot for that. So we'll be able to pick that up post-COVID and move forward. Paul Casey: Great. Yeah, that makes total sense. I like to say too much change too fast kills change, and it kills you! Michelle Whitney: It's true. It's true. Paul Casey: Well, finally, Michelle, what advice would you give to new leaders or anyone who wants to keep growing and gaining more influence? Michelle Whitney: So, new leaders, I would just say, be kind and gentle with yourself. Find a network of people that you can talk to and that you trust, and be vulnerable with them about what you don't know. Maybe don't be vulnerable with everybody about what you don't know, but find a few people that you really, really trust because none of us as leaders know everything. There is such great learning in vulnerability. So that would be a huge piece of advice for new leaders. Michelle Whitney: And like I said, be kind and gentle with yourself. There are things looking back now, I wish I would have done differently in my first couple years, but I learned from that. We make changes, and we move forward from there. Michelle Whitney: Continuing to evolve in your own leadership, while I talked about listening as learning, that's one part of learning. But there really is the piece about extending yourself beyond your current knowledge base and whether it's through a professional organization that you're involved in or a group or a network of people, reaching out and maybe targeting one key professional development that you want to extend, one in a year, and just being really thoughtful and strategic and gentle about that choice. I think that's always a good thing, to pick one thing that you're going to really extend, maybe beyond your own comfort zone and lean into something that maybe scares you a little every year, and making a commitment to doing that. And either doing it as an individual leader or as a leadership team, I think is always also a really a great thing to keep in mind. Michelle Whitney: Because it's easy in the busyness of our lives to forget about extending ourself in some formal professional development, too. So there's the informal listening and learning to be a better person and a better professional, but there's certainly that formalized professional development that also meets a need for us as leaders and influencers. Paul Casey: Well, Michelle, how can our listeners best connect with you? Michelle Whitney: Well, email is always the best way to connect with me. I'm on that thing all the time. Just ask my husband. He'll tell you. But I do take great joy in interfacing and being a support and assistance to fellow leaders. People always say, "Oh, but you're so busy. We don't want to bother you." It is never a bother to sit with someone who needs me to be a good listener. Email's always a great way to get in contact with me, and I'm always happy to help and be of support and assistance. Paul Casey: Well, thanks again for all you do to make the Tri-Cities a great place and keep leading well! Michelle Whitney: Thank you. Paul Casey: Let me wrap up our podcast today with a leadership resource to recommend. Michelle was just talking about getting more leadership proficiencies in your professional development, especially if you're an emerging leader or a young professional. I offer a program called Leader Launcher. Paul Casey: Leader Launcher is a Tri-City program for young professionals and emerging leaders where I do a training two hours every month on a leadership proficiency. You'll turn that into an action plan, be able to bring that back to your workplace and use it right away in your workplace. It'll be professional growth or leadership, and you can go to leader-launcher.com to sign up. For a full-year program, you'll get 24 hours of training that you'll be able to then apply right on the ground there at work. So leader-launcher.com. Paul Casey: Again this is Paul Casey. I want to thank my guest, Michelle Whitney from Pasco School District for being here today on the Tri-Cities Influencer Podcast. We also want to thank our TCI sponsor and invite you to support them. We appreciate you making this possible so we can collaborate to inspire leaders in our community. Paul Casey: Finally, one more leadership tidbit for the road to help you to make a difference in your circle of influence. It's a quote from Zig Ziglar. He said, "What you get by achieving your goals is not as important as what you become by achieving your goals." Until next time, KGF, keep growing forward! Announcer: Thank you to our listeners for tuning in to today's show! Paul Casey is on a mission to add value to leaders by providing practical tools and strategies that reduce stress in their lives and on their teams, so that they can enjoy life and leadership, and experience their key desired results. If you'd like more help from Paul in your leadership development, connect with him at growingforward@paulcasey.org for a consultation that can help you move past your current challenges and create a strategy for growing your life or your team forward. Announcer: Paul would also like to help you restore your sanity to your crazy schedule and getting your priorities done every day by offering you his free Control My Calendar checklist. Go to www.takebackmycalendar.com for that productivity tool, or open a text message to 72000 and type the word Growing. Paul Casey: Tri-Cities Influencer Podcast was recorded at Fuse SPC by Bill Wagner of Safe Strategies.

ANNOUNCER Thank you for listening. Today’s episode was written by Nicole Tuttle and edited by Mariah Clawson, and featured the vocal talents of Nicole Tuttle, Wes Lacy, Emily Buza, Ace Corsaira, Wes Haas, Mike Crawford, Chase Greenlee, Abbey Casino, Tim Lowe, Cait Gallagher, Cody Page and Liza Holmes. This episode was edited by Brad Colbroock, Mariah Clawson and Griffin Coldiron and featured the music of John Bartman and our announcer Sophia DeRise. Want to stay up to date on all things Arsen? Follow our social media for more information about upcoming episodes @arsenaudio on Twitter. If you enjoyed today's episode, subscribe to our show on your favorite podcasting site and leave a review. Until next time listeners. transcript; https://docs.google.com/document/d/1fh5tA4AwT7wij9FHEDLf-KFq6MDnfvVo3NvGuL8jNVM/edit?usp=sharing Find out more at https://arsen.pinecast.co

In ASCO eLearning’s first Social Determinants of Health (SDOH) series episode, Dr. Ramy Sedhom, MD moderates a discussion with ASCO President, Lori Pierce, MD, FASTRO, FASCO; Abenaa Brewster, MD, MHS; and Katie Reeder-Hayes, MD, MBA, MS on why understanding SDOH’s impact on patients is critical to providing equitable care. We hope you enjoy this episode. Subscribe: Apple Podcasts, Google Play | Additional resources: elearning.asco.org | Contact Us ANNOUNCER: The purpose of this podcast is to educate and inform. This is not a substitute for medical care, and is not intended for use in the diagnosis or treatment of individual conditions. Guests on this podcast express their own opinions, experience, and conclusions. The mention of any product, service, organization, activity, or therapy should not be construed as an ASCO endorsement. RAMY SEDHOM: Hello, and welcome to ASCO's newest podcast focused on the social determinants of health. My name is Ramy Sedhom, and I'm a medical oncology fellow at Johns Hopkins. Today, I am joined by ASCO president Dr. Lori Pierce, professor of radiation oncology at the University of Michigan. We are also joined by Dr. Katherine Reeder-Hayes, associate professor of oncology and health services researcher from UNC Chapel Hill, also chair of the ASCO Health Equity Committee. And finally, we are also joined by Dr. Abenaa Brewster, chair of the ASCO Prevention Committee, medical oncologist, and professor of epidemiology at MD Anderson. All our distinguished health researchers focused on disparities work. We are excited to launch this theme in light of the theme as highlighted by Dr. Pierce, equity, every patient, every day, everywhere. Dr. Pierce, why are the social determinants of health relevant? How do they relate to our work at ASCO and to the theme of equity? LORI PIERCE: Thanks, Dr. Sedhom. Yes, the ASCO theme this year is based on equity, and treating patients with equitable care is at the very heart of what ASCO does. It's what ASCO stands for. But we know there are factors, including social determinants of health, that can significantly affect the quality of health care that our patients receive, so we're hoping that this series of talks will increase awareness to many of those factors so providers can understand even better what some of the barriers are that our patients are experiencing. So we feel that understanding what they are, that's an important first step in terms of determining meaningful actions that we can take. RAMY SEDHOM: Thank you, Dr. Pierce. Dr. Reeder-Hayes, any additional comments? KATHERINE REEDER-HAYES: So I would just add that the reason I think oncologists inherently do care about the social determinants of health is that we care who gets cancer, and we care about the survival of that cancer. That's what we do as physicians. And as we follow cancer outcomes and cancer registries over time, what we realized is that there are a lot of biological things about cancers that determine their outcomes, but there are also many things that determine that outcome of a person's cancer that aren't biologic, and we have put those things together in a large group and use labels like the social determinants of health to understand what those non-biological factors are. But as physicians, as people who want to cure cancer, I think that it's just as important for us to have a good understanding of those non-biologic factors and how they're going to influence our patient's chance of the best outcome from their disease as it is for us to have a good grasp on the biologic determinants that are going to influence our patient's outcome. RAMY SEDHOM: Thank you. And I think this is a really good time to emphasize that the series is at the intersection of work through the Health Equity Committee and the Prevention Committee as being put forth by ASCO, and the ASCO Education Committee as well. And Dr. Reeder-Hayes, how would you importantly define the social determinants of health? KATHERINE REEDER-HAYES: So that's a pretty broad range of definitions, because there are a lot of people in health care who have talked about this concept of non-biologic determinants of how patients do for a long time, both in cancer care, and in other fields, like primary care, diabetes, and hypertension research. And so there are several organizations that have put forth definitions. The first one that I would point people to as the World Health Organization. So these patterns occur globally, not just in the United States. And the WHO describes the social determinants of health as the circumstances in which people are born, grow up, live, work, and age, and the systems that are put in place to deal with illness. So there's this idea that the circumstances that people find themselves in, as well as the structure of the health care system are part of this conversation about social determinants of health. The Robert Wood Johnson Foundation has done a lot of work to help put these concepts into more understandable and everyday language. And so the definition they use is that health begins where we live, learn, work, and play. So the social determinants of health include factors that operate on a few different levels. So there's the level of societal conditions, social conditions, economic conditions, physical conditions of where and how people live, and where and how people work, and then there are also psychosocial levels, like the patient-level psychosocial factors. And Centers for Disease Control also focus a lot of their work on social determinants at this idea of psychosocial factors at the patient level. So even though the concept is simple, it's actually also very broad. And when people talk about social determinants of health, they can be talking about things that are operating at several different levels, from the structure of the health care system, on down to the structure of the apartment complex where your patient lives. And so you have to think broadly, I think, when you're thinking about analyses for research or how to talk to your patients about social determinants of health. But the American Cancer Society has recently also put out a paper that tried to help people put these social determinants into appropriate buckets, particularly the ones that really seem to impact cancer care. And so the three categories or domains that they've suggested are structural inequities, so things that happen in a larger society and in the structure of the health care system that impact our patients and how they access care. Secondly, institutional environments. What do the places where we treat cancer look like? What are their structures? And how does that impact our patients outcomes? And then, finally, living environments. Where do our patients go back to when they go away from the health care setting? So those are some of the ways that we can think about the finding and talking about the social determinants of health. RAMY SEDHOM: Thank you for sharing that. That's a lot to think about and to consider. Dr. Brewster can you comment on the impact of those social determinants of health in disparate groups beyond race and ethnicity? ABENAA BREWSTER: So I agree that we're really used to thinking about social determinants of health as explaining some of the race-ethnic disparities that we see in cancer incidence and outcome. But we know that there are other groups that also experience differences in outcomes based on these determinants. For example, characteristics of sex, gender identity, sexual orientation, geography, immigration status, these are all characteristics that are impacted by social determinants of health. An example that I like to give for the race-ethnic disparity is the 40% increase in mortality that African-American women experience who are diagnosed with breast cancer compared to white women, and that, for a large part, is due to socioeconomic poverty and insurance differences. And in fact, then those characteristics then explain decreased screening, later stage of diagnoses, and then leads to disparate delays and receipt of treatment. There was a really nice paper published in JCO by Jamal and others which showed that insurance alone accounted for about 37% of that 40% increase in mortality that Black women experience who are diagnosed with cancer. And so you can see the great opportunities that are there to make care more equal if you're able to improve access. And so examples of other groups, for example, we have registry data showing that urban populations, although they have higher cancer incidence, the rural populations actually have a higher incidence of the cancers that are related to tobacco use or HPV. And we also know that rural populations also have higher mortality than urban populations when it comes to cancer, and that's largely driven by poverty, under insurance, socioeconomic status, isolation. And so that's another group that is impacted by those social determinants of health as outlined by Dr. Reeder-Hayes. And then, lastly, the LGBT community. Although there haven't been a lot of large studies looking at the impact of cancer incidence and outcomes in that community, there have been lots of survey studies that have shown that these determinants of health tend to cluster within those communities. Once again, social isolation, negative patient-provider relationships, under insurance, poverty, and so that's another group where this becomes very meaningful. And I think kind of what cuts across all of these groups are really characteristics that have been linked to historical discrimination. And so that's something that we see very clearly. And then, we also see it reflected in our oncology population, right? Only about 2% of oncologists are African-American. And so these social determinants of health not only impact our patients, but they impact what our workforce looks like, and then, that, in turn, leads to differences in terms of the patient-provider relationships. RAMY SEDHOM: Absolutely. And this is a great question to direct towards you as the chair of the Prevention Committee. How do the social determinants of health-- how are they relevant for our cancer patients and our cancer survivors? ABENAA BREWSTER: Social determinants affect the trajectory of cancer care. And so we've talked about the impact of social determinants of health on cancer mortality and cancer outcomes, receipt of treatment, timing of treatment. So those are all impacts that our oncology patients face. But I would say, in terms of the field of cancer prevention, over the past four decades, there have been significant advances in cancer prevention, vaccine, preventive therapies, high-quality screening. And unfortunately, there are populations that are being left out of those advances that we've seen in cancer prevention. We also know that maintaining a healthy weight, having a physically active lifestyle, having a nutritious diet with fruits and vegetables also play a key role in cancer prevention. But those are impacted by where our patients live. And so patients who live in disadvantaged neighborhoods may have less availability for these nutritious food choices, and they may have less safe spaces to engage in physical activity. And so while we advance our prevention knowledge and our prevention strategies, we are leaving behind groups of individuals who, based on where they live, and where they work, and how they're educated, are not being able to access those types of advances in cancer prevention, and as a result, we see increasing cancer incidence. RAMY SEDHOM: And Dr. Pierce, how does having this context or understanding make us better researchers, clinicians, and administrators? And also, how does it benefit our interactions with our patients and our colleagues? LORI PIERCE: So understanding the social determinants of health, it will make us better researchers, then clinicians, and just better doctors, because it allow us to see how the cancer world looks to our patients, kind of look at the cancer world through their lens. And an example, we have all these groundbreaking therapies, but if the patients don't have transportation and they can't come in, then those therapies are for naught. And I'm going to echo what Dr. Brewster said a few minutes ago. Another example is if we're advocating for patients to maintain a healthy weight, and a part of that is to do exercise, but they can't go out in their communities because they're not in a safe community, then we need to come up with other strategies. So I think it's very important that we understand these issues so we can advise, we can advocate, and we can act once we truly understand the barriers that our patients are facing everyday. RAMY SEDHOM: And Dr. Reeder-Hayes, can you comment on how the Health Equity Committee is seeking to understand the social determinants of health, their effects on populations, and the actions that can be taken to improve cancer care? KATHERINE REEDER-HAYES: Sure, so one of the things that the Health Equity Committee has been really excited about and working with Dr. Pierce on, her presidential theme, and also, in entering this conversation about the social determinants of health, is getting more of our members engaged with the idea of knowing our patients and their non-biologic characteristics as a way of being better doctors, as Dr. Pierce said. So the most simple way I can explain the concept is that every patient has a backstory. We know this as doctors. Every patient brings this unique story into their cancer care experience. And we know intuitively that that story that they bring with them into cancer care is going to impact what their cancer journey looks like. But we also need to acknowledge that it's going to impact their cancer outcome. We already know that it's going to impact how we interact together as a team, patient and provider, but we need to raise awareness that it's also going to impact how the cancer turns out. Because I think that's something our members naturally care about, and I think our members want to understand their patients' cancer care stories, So as researchers, and as folks on the Health Equity Committee, we would like to help people understand how to obtain the most relevant information about their patient's backstory in the most respectful and efficient and effective way, and then to integrate that well into how they care for their patients, as well as how they shape their research and their administrative roles. So we're excited to be doing that. Now, some of these circumstances are not immediately changeable, and they certainly aren't within the patient's individual power to change, but some of them, at least in terms of how they impact our patient's access to care, are modifiable, either by us as their physician, or by someone else in the health care system, or by someone in an organization that comes alongside the health care system to help, like a private foundation or a philanthropic organization. If our patient has a barrier to care because of transportation, like the example that Dr. Pierce gave, and they can't get a ride to cancer care, there are actually some ways that we can intervene on that. And there are even people in our health care system who are expert on how to intervene on that, as well as organizations that have volunteers that can help intervene on that. But if we don't obtain the information, we're not going to know, and then we can't bring the strategy to bear to get our patient to that groundbreaking therapy that they get. If they could make it to us, or if we could make it out to them, thinking about innovative ways to deliver care. So those are some of the things that we're excited about in the Health Equity Committee. RAMY SEDHOM: Thank you for sharing with us a lot of the great work that you're doing, and the ASCO team. On a similar note, Dr. Brewster, can you review with us some of the key contributions related to the social determinants of health from the Prevention Committee? ABENAA BREWSTER: Sure. Well, first of all, I'm so proud of the ASCO Cancer Prevention Committee for the work that they've done over the years to raise awareness, not only about cancer prevention to the public, but also within the oncology community. And ASCO's Cancer Prevention Committee has really spearheaded ASCO's work in tobacco control and cessation over the past 20 years, including supportive position statements on tobacco cessation and control, and also, for the electronic nicotine delivery device systems. And the committee continues to educate providers on the importance of addressing smoking cessation among their patients before and after a diagnosis of cancer. Also, within the area of obesity and energy balance, the committee has issued a position statement, because, actually, the majority of Americans are not aware of the impact that obesity has on cancer risk and cancer prognosis, and just raising the awareness of that issue, particularly around underserved populations, is important, because those are the populations that actually have some of the higher rates, but also lack that knowledge. And so the committee has conducted surveys of not just oncologists, but also patients, to try to really understand how that information is being disseminated, and then what are the barriers to be able to address the issue of obesity? And so that's important work that's going on. And a few years ago, the Prevention Committee also issued a statement on the association between alcohol and cancer risk and outcomes, and that's kind of all still a risk factor that has very little public awareness of its importance, and that's really driven a lot of the media and other publications that have come out really showing that association. And so raising that awareness is important. And the good news is that there are strategies and interventions and policy changes that can be brought to bear to impact some of these modifiable risk factors. And so that's a very exciting avenue for research and practical applications. RAMY SEDHOM: Thank you, Dr. Brewster. That's actually a wonderful shift toward policy and advocacy. And I, myself, participated on the Hill with ASCO. And Dr. Pierce, can you share with us or discuss some of the advocacy efforts from ASCO for our patients and lawmakers who can help in critical ways as it relates to the social determinants of health? LORI PIERCE: Thanks for the question. It's a great question. Advocacy is so important. So important that we advocate for our patients, because they often can't advocate for themselves. And we advocate to lawmakers, that's on a local level, on state level, and a national level, because it really can highlight to our lawmakers some of the barriers that our patients face and things that they can do to remove those barriers. And I'll give you a great example, HR 913, the Clinical Treatment Act. For those of you who don't know what that is, it's the following. Medicaid is the only insurer that does not cover routine care costs for patients, Medicaid patients who go on clinical trials. They're the only insurer that doesn't do that. And you talk about routine costs, we're talking about doctor visits, we're talking about x-ray tests, routine blood tests. The irony is Medicaid will cover these same costs if patients were not on a clinical trial, but they won't cover it on a clinical trial. And of course, we know there are a lot of minority patients that are on Medicaid. And so this is the disincentive for minority patients to go on clinical trials, which is exactly what we do not want to see. And so ASCO has been spending quite a bit of time advocating to the legislation about how important it is to cover these costs in clinical trials. And so the House has legislation, and just last week, the Senate now has parallel legislation. So we are moving the needle. We are going in the right direction. And it's a very important example of just what advocacy can do in terms of improving care for our patients. RAMY SEDHOM: That's great to hear, and we hope things continue to move forward. Dr. Pierce, also would like to direct this question to you. What is your vision for the future of this podcast series? And what do we really want our listeners to take home from this? LORI PIERCE: So I'm so excited about this series, and I am so appreciative to everyone, those who are on the series, and everyone in the background for getting this going. It's my hope that the series of broadcasts will be an important educational tool to really understanding the social determinants of health and how they impact our patients and society at large. And so this series will contain a lot of aspects, a lot of podcasts and videos on social determinants of health that our task force, which you are a key member of, is helping us to design. So our task force is made up of fellows and junior faculty, the future leaders of ASCO, and the future leaders of oncology in general. So thanks to your enthusiasm, thanks to your feedback, we are launching this, and I am super, super excited. RAMY SEDHOM: Thank you, again, everybody. The conversations today are really a reminder of all of our shared experiences. I remember as an oncology fellow, oftentimes, the hardest part of care in the clinic was not the science or the biology of cancer, but actually, all of the things discussed today, especially the social determinants and how they impact our patients. Again, we want to thank all of our wonderful faculty for serving as change agents. The planning of this podcast series is a joint effort through all of the wonderful ASCO volunteers who are fellows, junior faculty, ASCO leadership, and importantly, cancer survivors and ASCO staff. We do want everyone to look forward to next month's episode, where we will discuss and look at what ASCO can do as a professional society to address equity as a part of its mission. Again, thank you to all of our listeners, both locally and abroad, for joining us for this episode of the ASCO podcast series on the social determinants of health. To keep up with the latest episodes, please be sure to subscribe. Let us know what you think about this series by leaving a review or by emailing us at professionaldevelopment@asco.org. Again, thank you, and we look forward to seeing you in the next episode. ANNOUNCER: Thank you for listening to this week's episode of the ASCO eLearning weekly podcast. To make us part of your weekly routine, click Subscribe. Let us know what you think by leaving a review. For more information, visit the Comprehensive eLearning Center at elearning.asco.org.

Tara Jaraysi Kenning: "Teamwork makes the dream work," John C. Maxwell. I'm Tara Jaraysi Kenning, and I'm a Tri-Cities influencer. Paul Casey: A good rule of thumb is before you speak, ask yourself, is what I'm about to say true, necessary and kind? TNK Announcer: Raising the water level of leadership in the Tri-Cities of Eastern Washington. It's the Tri-cities Influencer podcast. Welcome to the TCI podcast, where local leadership and self leadership expert Paul Casey interviews, local CEOs, entrepreneurs, and nonprofit executives to hear how they lead themselves and their teams, so we can all benefit from their wisdom and experience. Here's your host, Paul Casey of Growing Forward Services, coaching, and equipping individuals and teams to spark breakthrough success. Paul Casey: It's a great day to grow forward. Thanks for joining me for today's episode with Tony Howard. He is the assistant superintendent of human resources for the Richland School District. I asked him for a fun fact about him. And he said, "Well, during this Corona crisis, the family has decided to watch Survivor reruns." And I said, "You know what? We are too." So I think we're on season 23 going back. And we both said we would never do that in a normal time, but what are we going to do? It'll be our memory for the Corona season is watching survivor reruns. Paul Casey: Well, we're going to dive in with Tony after checking in with our Tri-City Influencer sponsors. Preston House: Hi, my name is Preston House, and I'm the local owner of Papa John's Pizza right here in Tri-cities. Jesus Melendez: I'm Jesus Melendez, vice president and commercial lender with Community First Bank and HFG trust. Preston House: When I moved here in 2009 with my family from Boise, Idaho, I knew I wanted to move from a franchise to a local business owner. I'd been working with Papa John's since I was 16 years old, so when it came time to open my own location here in my own community, I knew I needed some financial guidance from an organization who understood my needs as a small business owner. Jesus Melendez: Small business owners often have a lot on their plate: employment retirement plans, payroll bills. Our mission is to become your financial partner for life, and is motivated by providing people in our community, like Preston, with all the information and support they need all under one roof. Preston House: It's really simple. No matter what I need, all it takes is one phone call. No automated prompts, no call waiting. It's just a local business serving another local business. Jesus Melendez: For more information on how Community First bank and HFG trust can help you get back on track. Visit www.community1st.com. That's www.community1st.com. Paul Casey: Thank you for your support of leadership development in the Tri-Cities. Well, welcome, Tony. I was privileged to meet you years ago. Boy, how many years ago would that be? Tony Howard: I think in 2007, '08, '09? Maybe in there somewhere. Paul Casey: That sounds right, yeah. Yes. My kids were in middle school at Enterprise Middle School. And you were one of the administrators there. Since I was an administrator, I thought, I like to bond with my kids' teachers and principals, so I invited you to lunch or coffee, and you said yes. So that was great. We could talk shop together and I've watched as you continued to move up the chain since then in the district office. So thanks for all you do for our school system. Tony Howard: Well, thank you. I'm really glad to be here today. I appreciate the invitation. Paul Casey: Absolutely. So let's let our listeners get to know you a little bit. Take us through a couple of career highlights that led you to what you're doing now, and also throw in there why you love what you do. Tony Howard: Well, thanks again. I'm really proud to be a lifer in K-12 public ed. I started as a classroom teacher in the early 90s. This is my 27th year. Paul Casey: Wow. Tony Howard: Which I don't know where the time goes. The last 13 of it in Richland, this'll be the start of year 14 in Richland. Worked on both sides of the mountains in the state of Washington. I was a teacher and assistant principal, then principal. What really stands out to me is just the people that I've known along the way and the relationships I've made and people that have taken a 23 year old know-it-all under their wing and let me learn my lessons the hard way, but in a structure of support. I really did imprint on some of that and have tried to model that in my own leadership career with folks. Tony Howard: And now, here we are 27 years later in the middle of trying to redefine public ed on the fly in this brave new world. It's just been an exciting time. I've been in HR for 10 years, as a principal for eight prior to that. Just at every stop, I've had just a wonderful opportunity to work with kids and parents and teachers and staff members and community folks, and coffee with people like you, 12, 13 years ago, that paid off today. That's really been the career highlight for me, is being able to pay all that back to the people that have helped me. Paul Casey: Wow. That's fantastic. Let me ask you this. Was there a time ever in that 27 years that you just said, "Maybe I should think about doing something else?" Did you ever have those days? Tony Howard: I think that 99% of all the days are good days. Like in any career, there are days where you go, "Oh, wow." But I've heard this quote once, "A setback is an opportunity for a comeback." Something like that. Paul Casey: I love that. Tony Howard: Something like that. So, no, I am really proud of the work that I've been able to do with kids and parents and families and communities over the last 27 years, and I'm a pretty happy K-12 guy. Paul Casey: So throughout the journey, you probably hit some obstacles to success, even though you didn't consider making the jump out of there. So what's one of the biggest hurdles you overcame in your career? Tony Howard: It took me some time as a young administrator ... this is 2003, four or five ... to learn the concept of I versus we. I learned some hard, painful lessons about I versus we, and the damage that I can cause, and the power that we can cause. Learning from those experiences really helped my career as an administrator take off because I learned through some speed bumps that it's not about me at all. Leadership is, in part, checking your ego at the door and looking for ways to move the collective forward in a way that's positive for the system. Tony Howard: Has nothing to do with Tony being brilliant or not brilliant on a particular topic. And that took some time. I don't have to have all the right answers. I don't have to always have to insert my opinion everywhere. It helps build trust with the people who work with me and for me, both subordinate in my department and within the organization. But at the end of the day, my role is to facilitate the growth for kids in the public school setting. It's not about what I want, necessarily. That was hard. I was young and pretty headstrong and thought I had it all figured out and really didn't. And like I said, in the open, had some very special people take a liking to the potential that I had and worked me through growing up a little bit. Paul Casey: Wow. So if you're willing to be vulnerable, what was the lesson you learned the hard way? With the whole damage of I. By the way, love that Tri-City influencers. If you missed it, he said, "The damage of I and the power of we." That's a keeper. That's a tweetable moment right there. Tony Howard: I think my first year as an assistant principal way back when, on the other side of the mountains, we had a hearing or something going on with the principal. I was the assistant principal and got a call, "Bring over some stuff to the hearing." I brought it over and made a passing offhand comment, and kind of joking to the district secretary when I came in the door, "I'm here to save the principal." It was a total in jest kind of comment. It landed poorly. "I'm here to save," That's not going to work in terms of what leadership is. Tony Howard: While I maybe didn't appreciate it at the time, rightfully so, got my bell rung pretty good about the fact that, first of all, I don't do anything. Second of all, we don't say that to each other. And third of all, it's time for you to grow up and sit in the chair that you need to sit in. I remember, it was done at four o'clock on a Friday afternoon, which was always tactically brilliant when you're trying to deliver lessons of love, you know? Tony Howard: I got to chew on it all weekend and decided he was right and I was going to make some changes. Within 15 months, I got moved from an assistant principal to a principal, and we're off to the races. So that's that tough love, that invest in potential, that I think is important for leaders to see. While I didn't appreciate getting chewed out, it was the greater good and he was right. Paul Casey: Yeah. You mentioned that some people took you under their wing and maybe taught you some of those lessons. Were those some mentors in your life? Were those other educators? Who were those folks in your life along the way that you really respect them speaking into your life? Tony Howard: I think all was the answer to that. This particular example was a supervisor. The superintendent brought me in and let me have it pretty good. But I never once thought ... and I think this is important in leadership ... that he was attacking me the person. He was getting after a bad choice in an effort of growth. Because he could fire me if you wanted to. That's the way it goes. But he chose to invest. Tony Howard: So I've had really good experiences with superintendents and principal, administrative colleagues over the years. The strongest bonds I have are with folks that aren't afraid to question, or aren't afraid to push on me a little bit. I appreciate intelligent discourse. I work in human resources. All we do is manage conflict. Paul Casey: Whoo hoo! Tony Howard: Nobody calls us up and says, "Hey, great job." We're always working on some sort of thing. So I appreciate folks that, with the right intention, ask hard questions and we're looking for the common right answer. I wouldn't be who I am professionally without my wife, as a counterpoint. She's got insight. She's not an administrator K-12 and she doesn't run in the world I run in professionally every day, but there are times where she has insight into who I am. She knows me pretty well, and will bring me up and have me think about a couple of things, which I like. Because like I said, it's not about me being right. It's about what's right for the system. We all get stuck in our thinking once in a while and really need those people to trust, to kind of push you. Paul Casey: That's right. Receiving feedback is difficult for many people, and I think if you do approach like you do with that, let's move ego and put it to the side, because ego usually leads to defensiveness. Somebody said, "Ego is edging greatness out." That's a good little acronym for that. Tony Howard: Yeah. Paul Casey: That way, you can receive it better. So leadership is difficult. Tony, what's your biggest ongoing challenge as a leader? What's really stretching you either in a positive way stretching, but it's uncomfortable? Tony Howard: Well, we'll set COVID aside. Paul Casey: Yes. That's its own bailiwick. Tony Howard: That's a whole other podcast, I think probably, in terms of what the leadership challenges that are. For me, it has been ... and I have this discussion a lot with paying it forward to newer administrators and aspiring leaders ... is the difference between responsibility and authority. I am the assistant superintendent of HR. My job is to assist the superintendent and the school board in moving forward labor and human resources kinds of issues. I have significant responsibility in the org chart to bargain with unions and solve problems and address personnel, and all those things that happen in HR. But I don't supervise anyone. I don't have, necessarily, the authority to walk in and say, "You, principal or you, teacher have to do this." Tony Howard: I supervise my department, but I have all of this systemic responsibility and my authority while it's there. If someone's doing something illegal, I can say, "Knock it off." But much more, it's lead by influence because I don't have the supervisory direct relationship with 2000 employees in the school district. I like that. I liked being able to be a problem solver without the threat of evaluation and over the top of a conversation, or being able to coach because I'm not the one that's going to write your evaluation. Tony Howard: I find that causes different conversations sometimes with folks. But you have lots of responsibility, not a ton of authority when it comes to just, "You shall do this." The leadership art of that is being able to work with people to move an agenda forward without just telling them to do it. Paul Casey: That's right. Inspire them forward. Tony Howard: Exactly. Paul Casey: So, if you had a leadership philosophy that would be put front and center on a bulletin board in your office for all to see, what would some of those messages say? Tony Howard: I do presentations all the time for aspiring teachers. How do you get your first job? How do I interview? I think we'll get to the answer here. And I always talk about, don't throw away the easy points. Don't ... How do I say this? ... give up things that are easy, that you cannot be taught. And the examples I use is, I can't teach you to have a good work ethic. You can't do it. I can't teach you to be nice to each other in the workplace. I can't teach you to like kids, which would seem like a no brainer in our profession, but it happens. Tony Howard: I can teach you curriculum. I can teach you an assessment strategy. If there's some classroom management kinds of issues, there's a billion different strategies for that. But I can't teach you to fundamentally love the game. I think that has always transferred well in terms of a leadership philosophy for me: work hard, be honest, keep an eye on the big picture. I use a balcony example. I think I read it in a book once. Get on the balcony and look out over the organization, because your decisions sometimes impact in ways you don't even understand. Tony Howard: Be open to feedback, and don't be rigid in your thinking. There are times where you have to be. You did this and I have to fire you. Sometimes it's that simple in our discussions, but rarely. Most of our discussions are, if you've got a better way to build a mouse trap, I want you to be open to share it. Whether we can do it or not, the journey is sometimes more important than how you end up in an issue. But I always look for, those are the easy points. If I'm a principal and I'm looking to hire a teacher and the teacher is going to be combative with their team, with their parents, with me, with the office staff? Life is too short. Those are easy points. Remember that we're all in this together. Paul Casey: Yeah. It sounds like you're talking about the inside game, like the stuff within that you can work on, your personal development. Tony Howard: Right. Paul Casey: Which will then bleed over into your professional life. Tony Howard: Sure. Paul Casey: Most influencers I know have a bit of visionary inside them in order to take that next hill. So where do you take time to dream about the future? What does that look like for you? Tony Howard: There are days when that's hard because in the here and now, there's a lot going on in the here and now, even more so now. There's all sorts of things that aren't in any manual in terms of how to approach them from a leadership perspective. My dreams about the future are being able to look back with satisfaction that I was true to myself, and that at the end of the day, my 27 to 40 years of experience with K-12 helped kids. Tony Howard: My HR job is not necessarily a kid directed position. I don't work with parents very often anymore. I don't work with kids almost ever, but I was a principal and a teacher and a lifer in the system. I think that experience matters. If I'm making human resources decisions in the system that aren't front and center towards kids and staff in the school system, then what am I doing? I think that's what I look forward to in the future. I call it with my folks, the human side of human resources. We manage resources pretty well, whether it's a contractor, a stipend, or whatever it is. But there are times when just the volume of human, everyone that contacts us, almost, has got some sort of crisis. Like I said, no one rarely walks into HR and said, "I'm having a great day, everybody." Paul Casey: HR, grumble, grumble. Tony Howard: Yeah, exactly. Right. So, in that customer service, human side of things, we see people at some of their most stressed moments, and we need to make sure we never forget that. Paul Casey: Yeah. The best HR professionals I see are those that keep the human first over the resources. Tony Howard: Right. We have an employee will come in and a spouse is terminally ill, and all they need from us is the love to support them. They don't know, and they're terrified, and they don't need us closing off opportunities to help them. They need us to work with them. While those conversations are emotional, they're very important. If you don't have them, your system doesn't work. Paul Casey: It's right. Well, before we head into our next question on Tony's morning routine, a shout out to our sponsors. Preston House: Hi, my name is Preston House, and I'm the local owner of Papa John's pizza right here in tri-cities. Jesus Melendez: I'm Jesus Melendez, vice president and commercial lender with Community First bank and HFG trust. Preston House: When I moved here in 2009 with my family from Boise, Idaho, I knew I wanted to move from a franchise to a local business owner. I'd been working with Papa John's since I was 16 years old, so when it came time to open my own location here in my own community, I knew I needed some financial guidance from an organization who understood my needs as a small business owner. Jesus Melendez: Small business owners often have a lot on their plate: employment retirement plans, payroll, bills. Our mission is to become your financial partner for life and is motivated by providing people in our community like Preston with all the information and support they need, all under one roof. Jesus Melendez: It's really simple. No matter what I need, all it takes is one phone call. No automated prompts, no call waiting. It's just a local business serving another local business. Preston House: For more information on how Community First Bank and HFG trust can help you get back on track. Visit www.community1st.com. That's www.community1st.com. Paul Casey: So Tony, what's your typical morning routine? Before work, once you arrive at work, any rituals to help you start your day strong? Tony Howard: It's called coffee. Let me start there, I kind of have a tongue in cheek running joke with my assistant that no one can yell at me before the first cup of coffee is drank. Paul Casey: I like that. Tony Howard: It doesn't happen sometimes, but it's been kind of a running joke for years. I get up and get going in the morning. I like to take some time in the morning to review the overnights, what comes in, in the email overnight, if there were late stuff that didn't get finished, put a bow on the day before I go onto the next one. I'm pretty meticulous with things like a phone log, because it's been useful for me over the years to log what I do and when, so I want to make sure that's current, and just kind of the nuts and bolts of administrivia, of being an office guy. Tony Howard: It's a chance for me in the morning to connect with department staff if I need to, or look to see where the hotspots are or the day or what's in the calendar or whatnot. But it all circles around coffee. I'm not much good to anybody without that first cup of coffee. Paul Casey: When you say hotspots of the day, sort of previewing that, what would be some examples of that? Tony Howard: Well, do I have a contentious personnel meeting in the day? Do we have an investigatory issue? Do we have a grievance hearing? Some of those things are multi-day planning, but do I have an employee that's fired up about a particular issue and needs a little face time? Things drop into my calendar that I don't know what the context of them are. Sometimes you're going to meet with X and, well, why? And try to get my head wrapped around that. Tony Howard: It's also a good chance for me to check in with my staff. I have eight folks report directly to me and they do a variety of technical things for me. If you've been in the district HR's office, which we're moving out of happily here in a couple of months, but it's about as big as this room. No, it's a little bigger than that. But it's not a very big space and just making sure we're fine because if the staff's not fed and we're not taking care of our people, then it's hard for them to feel like they're part of the bigger picture, and they don't get to know everything I see. So just making sure we have that connection. Paul Casey: That's good. How do you deal with the everyday grind of your work without burning out, especially in this intense people, intense conflict job that you live and work in? Tony Howard: I have, more so as an HR director than I did as a principal. As a principal, you get knocks on the door… We're talking direct kid line issues. A mom is distraught or a dad is upset about A, B, or C. And those are real time emotional issues. The kids can't sit, and so they happen a lot. In HR, less so. My meetings are sometimes more formal, and I've been able to kind of balance out the two things. And I tried really hard ... like I said, we manage a ton of conflict and there's a ton of drama that comes through, 2000 employees and all their different needs and whatnot ... to keep it separate. My kids, I've always told myself, I'd never gave my writeup to raise my own kids and be a dad and a spouse in order to work. Paul Casey: That's good. Tony Howard: It feels like that sometimes, but that's the nature of the game. But I have always made conscious and tried to leave the work at the door, and I'm not perfect at that. COVID has messed it all up. It's just messed it all up. In fact, I don't know, June-ish, I started just coming into the office during the day. Not because I can't work at home, but I'm less efficient and it's just stuff I don't- Paul Casey: You needed that break. Tony Howard: It's a mental thing for me. Some of the conflict I didn't want in the house where kids are running in and out of the room and doing their thing, or my wife's coming through. We're a house, and all the kids are home, home learning too, so we're all trying to do that. For me, it was more appropriate to have a little bit of separation between those two things. I will use the drive home as the unplugged time, whether that's an audible or a podcast or whatever I've got, or some music. I like to drive and it's a few minutes just to physically purge the day and go home. Paul Casey: Yeah, it's like emotional white space, right? Tony Howard: Yeah, a little bit. A little bit. So I can be good to my kids and good to my wife and all those things that are important to me, because they're going to be with me far longer than the career, and it's not fair to them to overbalance. It's hard, though. Paul Casey: Yeah. There's a YouTube video. I'm trying to think what the name of it is, but it's like, we all blasted off of earth on these little spaceships, and COVID hits, and we have to make ... It's called maintain the vessel, I think. So we have to maintain our vessel, which is like our body, and each section of the spaceship, one is for work, one is for sleep, one's for eat, we shouldn't cross over. But it's like when you cross the threshold, that's all you do is work in the work room. So you've had to do that, and a lot of people have had to do that, when they go into the office, is create that separation so work stays work. Tony Howard: Well, there's an exercise to getting up and getting a shower and shaving and getting dressed and putting on your work clothes and going and doing that. Besides, I'm not going to be on camera for a Zoom at eight o'clock, so I'm going to roll out of bed and plop out in the recliner. It just didn't work for me. It just didn't. While I love being home, I want to be home and attentive, not home and, "Get off the internet, because I need to get on." It just didn't work for me. Paul Casey: Well, family is a big deal to you. It's a big deal to most people. I know it's difficult now, like you said, how do you prioritize family time, yet still be a high performer at work? So you mentioned already, trying to leave work at the door, not letting that creep into the family as much as possible. Any other tips you have? Tony Howard: You asked me earlier about it. I never thought about getting out of K-12, and this is one of the reasons why I'm not: because the fit is good. In my job, I flex time as I need to. If I've got a kid with a soccer game, I can go. I've got a cell phone and a laptop, and I will keep up and we'll communicate with that. But it's important. Our kids are in Richland in the Richland school district. I work in the Richland school district. My wife works in the Richland school district. We're invested in that time for them, and they're here and it's something that we can do. Tony Howard: Public is a good fit in that regard, which is another reason why I'm a K-12 lifer. It was very important to us that our kids were involved in the system that we were trying to lead. Otherwise, what's the point of all of it? If it's not good enough for my kids, then that's a moral bar for me. That is a good checks and balance. But there's a benefit to that. I have a ninth grader to be, whose life's ambition is to play Bomber soccer and while it's being delayed a little bit as we try to navigate all of these restrictions going to Bomber soccer games is just fine. Tony Howard: My oldest played at Hanford for a couple of years and that's great. Being able to be part of the system and be involved in those because I'm leading in the system, is rewarding to me. Paul Casey: Incidentally, how much vacation do you take, just to refresh? Tony Howard: Oh, four or five weeks a year. We get five weeks as administrators. I try to choreograph that around kid breaks. I'll take a chunk of time at Christmas and spring break. We just got back from doing some summer stuff. I try to balance that. I don't take all the five weeks some years. It just kind of depends. But the basic breaks like everybody else, I figure if they're in school and my wife's working, I'd probably be working too. Paul Casey: Do you try to stay disconnected during vacation? Like, like be "done" done? I'm asking because I've asked my clients, so do you check email and not respond? Do stay disconnected? Tony Howard: I've done all the different models that I think there are. I try really hard in July. Around the 4th of July, we'd like to do some camping and whatnot, that I unplug completely. Just for that seven to 10 days, don't check a thing. It's a dead time for us anyway. Paul Casey: Sure. Tony Howard: Buildings are closed. Kids are gone. It's relatively quiet as things get in the office. This year, I really wasn't able to do that because there was just too much going on. Well, I'm trying to, from a campground, Zoom into a school board meeting so I can keep an eye on what's going on, thinking this is just surreal trying to navigate all that. But I do try to unplug a little bit at Christmas, the in between Christmas and New Years, I'll try to unplug for three or four days. Tony Howard: The rest of it, I think the world we live in and the role that I have requires some monitoring. I'll keep an eye on things. I'll respond if I need to. Some of it I'll keep and I'll just flag it for when I get back to work. But where I sit in the org chart for the district and the emergent needs, I have to be able to respond to a problem with it comes up. I think it's just a leadership thing. I think I need to do that. And if not, then who? Is kind of the question that I have. Paul Casey: Sure. Tony Howard: That's kind of how I approach that. I try to unplug a little bit. I think it's healthy for me and it's healthy for the family. This year, we weren't able to unplug as much, but there are extenuating circumstances. Paul Casey: Yes. Work and life got mushed together. Tony Howard: They sure did. Paul Casey: Well, influencers aren't know it alls. They are learners. Where do you go, Tony, for the wisest advice? That can be live people here in town or they could be authors and motivators, education professionals. Where do you go? Tony Howard: I go to ... I think we talked about it in the open a little bit ... the people that I trust. It starts there. There has to be some degree of safety in order to be vulnerable to ask that question. I'm lucky in the sense that I spent all 13 plus years in the RSD and have a network of folks that I can reach out to and say, "I am missing something here." Or, "I'm thinking this, and I'm pretty sure there's a bucket there I'm going to step in. Can you tell me what I'm not seeing?" I do some of that with people that I trust. Tony Howard: I do some of the conference. All that whole world is going to change, the onsite stuff and whatnot. I find, it may be a function of just maturing a little bit, my attention span to irrelevant is less than it used to be. If I take a class or read a book or go to a session and it's just not clicking with what I need to do, it's hard for me to keep engaged to that. I find it's 50/50. There's a lot of influencers, as you put out there, that have a lot of good content, but it's just not possible to soak all that in. Tony Howard: I like to do some personal reflection, that driving stuff like we talked about and whatnot. I'll just chew on, all right, what's the larger issue that I'm not seeing? It's a question I ask myself a lot. Where's my blind spot? So whether it's a negotiation or a labor issue or a personnel issue, is where are we not quite right? Tony Howard: Our attorney works next to us, works in the district office. He really pushed on me, when I started 10 years ago in HR and to write better. Because now I'm writing for real issues. And I wasn't used to having him edit a letter and getting back three pages of red: change this and change that, and do this and do that. He was great. He's a terrific asset and a good friend. I tease that the highest praise I've gotten from anybody in the district in the past nine plus years is a letter he sent back to me a couple of years ago that says, "Looks good." That was like, I finally made it, a little bit. But I've got a network and I tend to lean towards the network of people that I know and trust more so than industry standard kinds of things. It's just my comfort level. Paul Casey: Sure, sure. I love that question: what's the issue I'm not seeing here? What's missing? What's the blind spot? These are questions to ask that open up creativity. They're good ones. Tony Howard: Yep. Paul Casey: Well finally, what advice would you give to new leaders or anyone who wants to keep growing and gaining more influence? Tony Howard: Be a sponge, good or bad. I've had this discussion with leaders before, principals before. You learn some times as much from a bad or a negative classroom experience as a student than you do from the great ones. I've done exercises before: think through your three most powerful teachers that you've ever had. Now let's think through the three most traumatic, poor experiences you've had in a classroom. People can do that. They can remember real quick, the good and the bad. Tony Howard: Then I'll ask them to think through somebody in the middle, and they have a harder time, because good and bad ... or good and negative, I guess, is maybe a fair way to say that, are telling learning opportunities. So be a sponge. Just because someone has a different style or you don't agree with their tactics, you can still learn from them. The other part of this that I would tell young leaders is not to worry about errors of effort. While there are always exceptions and there are errors we just, can't not deal with, for young administrators, you're going to mess up. For young leaders, you're going to make mistakes. I've been doing this a long time, I still mess up. If someone's not mad at me at least once a day, I'm probably not doing my job right. Paul Casey: That's right. Tony Howard: Errors of effort are correctable and coachable. Errors of apathy aren't. If you're just making the same mistake twice because you're just too lazy to change, or unwilling to take feedback in a positive way, that's where I start to get concerned about leadership being a failure for somebody. If you botch it: eat it, fix it. It happens. I've gotten up in front of staffs and said, "Well, that didn't work. Here's what we've tried to do. And now we're going to step back and readjust." And I have found over the years that adults respond to that. Paul Casey: Yeah. They respect you more, right? Tony Howard: I screwed up. Because I've always tried to work from, I'll take the hit if the building messes up. You guys are going to celebrate. It's not the I and we stuff again. But if you're wrong, you're wrong. I've seen leaders really blow up a room just because they're not willing to accept that, whether it may or may not be their fault, it's still them. Paul Casey: Yep. Tony Howard: I didn't make that teacher yell at that kid or throw that book or whatever happened in the classroom, but that's still a teacher under my supervision, and that reflects on all of us, and I'm the leader of the building. So, errors of effort, I can live with those. Errors of apathy or errors of stubbornness, I have less patience. Paul Casey: Wow. So good. So, Tri-Cities influencers, be a sponge and don't worry as much about errors of effort. Really be concerned about errors of apathy. Well Tony, how can our listeners connect with you if they wanted to reach out? Tony Howard: I'm on the district website, and email's up there and all of that. My LinkedIn profile is active. I think that's how you saw me. Paul Casey: I did. We reconnected through that, yes. Tony Howard: We reconnected here just not too long ago. But I'm not hard to find. I'm just at the district website in the HR department. I'm open to email and contact. The art of leadership is a career interest of mine. Paul Casey: Yes, indeed. A passion we share. Tony Howard: Well, I appreciate you inviting me in. Paul Casey: Yes. Thanks for all you do to make the Tri-Cities a great place, and keep leading well. Tony Howard: Aw, thank you. Paul Casey: Let me wrap up our podcast today with a leadership resource to recommend. The website is arealme.com. Arealme.com. These are really fun quizzes to take. You've probably seen some on Facebook, like what kind of animal are you? What kind of Disney princess are you? What's your super power? What chemical element are you? If you just want a fun little website of surveys, they'll even create some for you, arealme.com. Paul Casey: Again, this is Paul Casey. I want to thank my guest Tony Howard from the Richland school district for being here today on the Tri-city influencer podcast. We also want to thank our TCI sponsor and invite you to support them, appreciating that they make this possible so we can collaborate to help inspire leaders in our community. Finally, one more leadership tidbit for the road to help you make a difference in your circle of influence. Sade said, "Have patience. All things are difficult before they become easy." Until next time, KGF. Keep growing forward. Announcer: Thank you to our listeners for tuning in to today's show. Paul Casey is on a mission to add value to leaders by providing practical tools and strategies that reduce stress in their lives and on their teams, so that they can enjoy life and leadership and experience their key desired results. Announcer: If you'd like more help from Paul in your leadership development, connect with him at growingforward@paulcasey.org, for a consultation that can help you move past your current challenges and create a strategy for growing your life or your team forward. Announcer: Paul would also like to help you restore your sanity to your crazy schedule and getting your priorities done every day by offering you his free control my calendar checklist. Go to www.takebackmycalendar.com for that productivity tool, or open a text message to 72000, and type the word "growing." Paul Casey: The Tri-Cities influencer podcast was recorded at Fuse SPC by Bill Wagner of Safe Strategies.

Some highlights of the show include: The company's cloud native journey, which accelerated with the acquisition of Uswitch.  How the company assessed risk prior to their migration, and why they ultimately decided the task was worth the gamble. Uswitch's transformation into a profitable company resulting from their cloud native migration. The role that multidisciplinary, collaborative teams played in solving problems and moving projects forward. Paul also offers commentary on some of the tensions that resulted between different teams. Key influencing factors that caused the company to adopt containerization and Kubernetes. Paul goes into detail about their migration to Kubernetes, and the problems that it addressed.  Paul's thoughts on management and prioritization as CTO. He also explains his favorite engineering tool, which may come as a surprise.  Links: RVU Website: https://www.rvu.co.uk/ Uswitch Website: https://www.uswitch.com/ Twitter: https://twitter.com/pingles GitHub: https://github.com/pingles TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast, where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I'm your host, Emily Omier, and today I am chatting with Paul Ingles. Paul, thank you so much for joining me.Paul: Thank you for having me.Emily: Could you just introduce yourself: where do you work? What do you do? And include, sort of, some specifics. We all have a job title, but it doesn't always reflect what our actual day-to-day is.Paul: I am the CTO at a company called RVU in London. We run a couple of reasonably big-ish price comparison, aggregator type sites. So, we help consumers figure out and compare prices on broadband products, mobile phones, energy—so in the UK, energy is something which is provided through a bunch of different private companies, so you've got a fair amount of choice on kind of that thing. So, we tried to make it easier and simpler for people to make better decisions on the household choices that they have. I've been there for about 10 years, so I've had a few different roles. So, as CTO now, I sit on the exec team and try to help inform the business and technology strategy. But I've come through a bunch of teams. So, I've worked on some of the early energy price comparison stuff, some data infrastructure work a while ago, and then some underlying DevOps type automation and Kubernetes work a couple of years ago.Emily: So, when you get in to work in the morning, what types of things are usually on your plate?Paul: So, I keep a journal. I use bullet journalling quite extensively. So, I try to track everything that I've got to keep on top of. Generally, what I would try to do each day is catch up with anybody that I specifically need to follow up with. So, at the start of the week, I make a list of every day, and then I also keep a separate column for just general priorities. So, things that are particularly important for the week, themes of work going on, like, technology changes, or things that we're trying to launch, et cetera. And then I will prioritize speaking to people based on those things. So, I'll try and make sure that I'm focusing on the most important thing. I do a weekly meeting with the team. So, we have a few directors that look after different aspects of the business, and so we do a weekly meeting to just run through everything that's going on and sharing the problems. We use the three P's model: so, sharing progress problems and plans. And we use that to try and steer on what we do. And we also look at some other team health metrics. Yeah, it's interesting actually. I think when I switched from working in one of the teams to being in the CTO role, things change quite substantially. That list of things that I had to care about increase hugely, to the point where it far exceeded how much time I had to spend on anything. So, nowadays, I find that I'm much more likely for some things to drop off. And so it's unfortunate, and you can't please everybody, so you just have to say, “I'm really sorry, but this thing is not high on the list of priorities, so I can't spend any time on it this week, but if it's still a problem in a couple of weeks time, then we'll come back to it.” But yeah, it can vary quite a lot.Emily: Hmm, interesting. I might ask you more questions about that later. For now, let's sort of dive into the cloud-native journey. What made RVU decide that containerization was a good idea and that Kubernetes was a good idea? What were the motivations and who was pushing for it?Paul: That's a really good question. So, I got involved about 10 years ago. So, I worked for a search marketing startup that was in London called Forward Internet Group, and they acquired USwitch in 2010. And prior to working at Forward, I'd worked as a consultant at ThoughtWorks in London, so I spent a lot of time working in banks on continuous delivery and things like that. And so when Uswitch came along, there were a few issues around the software release process. Although there was a ton of automation, it was still quite slow to actually get releases out. We were only doing a release every fortnight. And we also had a few issues with the scalability of data. So, it was a monolithic Windows Microsoft stack. So, there was SQL Server databases, and .NET app servers, and things like that. And our traffic can be quite spiky, so when companies are in the news, or there's policy changes and things like that, we would suddenly get an increase in traffic, and the Microsoft solution would just generally kind of fall apart as soon as we hit some kind of threshold. So, I got involved, partly to try and improve some of the automation and release practices because at the search start-up, we were releasing experiments every couple of hours, even. And so we wanted to try and take a bit of that ethos over to Uswitch, and also to try and solve some of the data scalability and system scalability problems. And when we got started doing that, a lot of it was—so that was in the early heyday of AWS, so this was about 2008, that I was at the search startup. And we were used to using EC2 to try and spin up Hadoop clusters and a few other bits and pieces that we were playing around with. And when we acquired Uswitch, we felt like it was quickest for us to just create a different environment, stick it under the load balancer so end users wouldn't realize that some requests was being served off of the AWS infrastructure instead, and then just gradually go from there. We found that that was just the fastest way to move. So, I think it was interesting, and it was both a deliberate move, but it was also I think the degree to which we followed through on it, I don't think we'd really anticipated quite how quickly we would shift everything. And so when Forward made the acquisition, I joined summer of 2010, and myself and a colleague wrote a little two-pager on, here are the problems we see, here are the things that we think we can help with and the ways that technology approach that we'd applied at Forward would carry across, and what benefits we thought it would bring. Unfortunately because Forward was a privately held business—we were relatively small but profitable—and the owner of that business was quite risk-affine. He was quite keen on playing blackjack and other stuff. So, he was pretty happy with talking about probabilities of success.And so we just said, we think there's a future in it if we can get the wheels turning a bit better. And he was up for it. He backed us and we just took it from there. And so we replaced everything from self-hosted physical infrastructure running on top of .NET to all AWS hosted, running a mix of Ruby, and Closure, and other bits and pieces in about two years. And that's just continued from there. So, the move to Kubernetes was a relatively recent one; that was only within the last—I say ‘recent.' it was about two years ago, we started moving things in earnest. And then you asked what was the rationale for switching to Kubernetes—Emily: Let me first ask you, when you were talking with the owner, what were the odds that you gave him for success?Paul: [laughs]. That's a good question. I actually don't know. I think we always knew that there was a big impact to be had. I don't think we knew the scale of the upside. So, I don't think we—I mean, at the time, Uswitch was just about breaking even, so we didn't realize that there was an opportunity to radically change that. I think we underestimated how long it would take to do. So, I think we'd originally thought that we could replace, I think maybe most of the stuff that we needed replaced within six months. We had an early prototype out within two weeks, two or three weeks because we'd always placed a big emphasis on releasing early, experimenting, iterative delivery, A/B testing, that kind of thing. So, I think it was almost like that middle term that was the harder piece. And there was definitely a point where… I don't know, I think it was this classic situation of pulling on a ball of string where it was like, what wanted to do was to focus on improving the end-user experience because our original belief was that, aside from the scalability issues, that the existing site just didn't solve the problem sufficiently well, that it needed an overhaul to simplify the journeys, and simplify the process, and improve the experience for people. We were focusing on that and we didn't want to get drawn into replacing a lot of the back office and integration type systems partly because there was a lot of complexity there. But also because you then have to engage with QA environments, and test environments, and sign-offs with the various people that we integrate with. But it was, as I said, it was this kind of tugging on a ball of string where every improvement that we made in the end-user experience—so we would increase conversion rate by 10 percent but through doing that, we would introduce downstream error in the ways that those systems would integrate, and so we gradually just ended up having to pull in slightly more and more pieces to make it work. I don't think we ever gave odds of success. I think we underestimated how long that middle piece would take. I don't think we really anticipated the degree of upside that we would get as a consequence, through nothing other than just making releases quicker, being able to test and move faster, and focusing on end-user experience was definitely the right thing to focus on.Emily: Do you think though, that everybody perceived it as a risk? I'm just asking because you mentioned the blackjack, was this a risk that could fail?Paul: Well, I think the interesting thing about it was that we knew it was the right thing to do. So, again, I think our experience as consultants at ThoughtWorks was on applying continuous delivery, what we would today call DevOps, applying those practices to software delivery. And so we'd worked on systems where there weren't continuous integration servers and where people weren't releasing every day, and then we'd worked in environments where we were releasing every couple of hours, and we were very quickly able to hone in on what worked and discard things that didn't. And so I think because we've been able to demonstrate that success within the search business, I think that carried a great deal of trust. And so when it came to talking about things we could potentially do, we were totally convinced that there were things that we could improve. I think it was a combination of, there was a ton of potential, we knew that there was a new confluence of technologies and approaches that could be successful if we were able to just start over. And then I think also probably a healthy degree of, like, naive, probably overconfidence in what we could do that we would just throw ourselves into it. So, it's hard work, but yeah, it was ultimately highly successful. So, it's something I'm exceedingly proud of today.Emily: You said something really interesting, which is that Uswitch was barely profitable. And if I understand correctly, that changed for the better. Can you talk about how this is related?Paul: Yeah, sure. I think the interesting thing about it was that we knew that there was something we could do better, but we weren't sure what it was. And so the focus was always on being able to release as frequently as we possibly could to try and understand what that was, as well as trying to just simplify and pay back some of the technical debt. Well, so, trying to overcome some of the artificial constraints that existed because of the technology choices that people have made—perfectly decent decisions on, back in the day, but platforms like AWS offered better alternatives, now. So, we just focused on being able to deliver iteratively, and just keep focusing on continual improvement, releasing, understanding what the problems were, and then getting rid of those little niggly things. The manager I had at Forward was this super—I don't know, he just had the perfect ethos, and he was driven—so we were a team that were focused on doing daily experiments. And so we would rely on data on our spend and data on our revenue. And that would come in on a daily cycle. So, a lot of the rhythm of the team was driven off of that cycle. And so as we could run experiments and measure their profitability, we could then inform what we would do on the day. And so, we have a handful of long-running technology things that we were doing, and then we would also have other tactical things that he would have ideas on, he would have some hypothesis of, well, “Maybe this is the reason that this is happening, let's come up with a test that we can use to try and figure out whether that's true.” We would build something quickly to throw it together to help us either disprove it or support it, and we would put it live, see what happened, and then move on to the next thing. And so I think a lot of the—what we wanted to do is to instill a bit of that environment in Uswitch. And so a lot of it was being able to release quickly, making sure that people had good data in front of them. I mean, even tools like Google Analytics were something which we were quite au fait with using but didn't have broad adoption at the time. And so we were using that to look at site behavior and what was going on and reason about what was happening. So, we just tried to make sure that people were directly using that, rather than just making changes on a longer cycle without data at all.Emily: And can you describe how you were working with the business side, and how you were communicating, what the sort of working relationship was like? If there was any misunderstandings on either side.Paul: Yeah, it's a good question. So, when I started at Uswitch, the organizational structure was, I guess, relatively classical. So, you had a pooled engineering team. So, it was a monolithic system, deployed onto physical infrastructure. So, there was an engineering team, there was an operations team, and then there were a handful of people that were business specific in the different markets that we operated in. So, there was a couple of people that focused on, like, the credit card market; a couple of people that focused on energy, for example. And, I used to call it the stand-up swarm: so, in the morning, we would sit on our desks and you would see almost the entire office moved from the different card walls that were based around the office. Although there was a high degree of interaction between the business stakeholders, the engineers, designers, and other people, it always felt slightly weird that you would have almost all of the company interested in almost everything that was going on, and so I think the intuition we had was that a lot of the ways that we would think about structuring software around loosely-coupled but highly cohesive, those same principles should or could apply to the organization itself. And so what we tried to do is to make sure that we had multidisciplinary teams that had the people in them to do the work. So, for the early days of the energy work, there was only a couple of us that were in it. So, we had a couple of engineers, and we had a lady called Emma, who was the product owner. She used to work in the production operations team, so she used to be focused on data entry from the products that different energy providers would send us, but she had the strongest insight into the domain problem, what problem consumers were trying to overcome, and what ways that we could react to it. And so, when we got involved, she had a couple of ideas that she'd been trying to get traction on, that she'd been unable to. And so what we—we had a, I don't know, probably a, I think a half-day session in an office. So, we took over the boardroom at the office and just said, “Look, we could really do with a separate space away from everybody to be able to focus on it. And we just want to prove something out for a couple of weeks. And we want to make sure that we've got space for people to focus.” And so we had a half-day in there, we had a conversation about, “Okay, well, what's the problem? What's the technical complexity of going after any of these things?” And there's a few nuances, too. Like, if you choose option A, then we have to get all of the historical information around it, as well as the current products and market. Whereas if we choose option B, then we can simplify it down, and we don't need to do all of that work, and we can try and experiment with something sooner. So, we wanted it to be as collaborative as possible because we knew that the way that we would be successful is by trying to execute on ideas faster than we'd been able to before. And at the same time, we also wanted to make sure that there was a feeling of momentum and that we would—I think there was probably a healthy degree of slight overconfidence, but we were also very keen to be able to show off what we could do. And so we genuinely wanted to try and improve the environment for people so that we could focus on solving problems quicker, trying out more experiments, being less hung up on whether it was absolutely the right thing to do, and instead just focus on testing it. So, were there tensions? I think there were definitely tensions; I don't think there weren't tensions so much on the technical side; we were very lucky that most of the engineers that already worked there were quite keen on doing something different, and so we would have conversations with them and just say, “Look, we'll try everything we can to try and remove as many of the constraints that exist today.” I think a lot of the disagreement or tension was whether or not it was the right problem to be going after. So, again, the search business that we worked in was doing a decent amount of money for the number of people that were there, and we knew there was a problem we could fix, but we didn't know how much runway it would have. And so there was a lot of tension on whether we should be pulling people into focusing on extending the search business, or whether we needed to focus on fixing Uswitch. So, there was a fair amount of back and forth about whether or not we needed to move people from one part of the business to another and that kind of thing.Emily: Let's talk a little bit about Kubernetes, and how Uswitch decided to use Kubernetes, what problem it solved, and who was behind the decision, who was really making the push.Paul: Yeah, interesting. So, I think containers was something that we'd been experimenting with for a little while. So, as I think a lot of the culture was, we were quite risk-affine. So, we were quite keen to be trying out new technologies, and we'd been using modern languages and platforms like Closure since the early days of them being available. We'd been playing around with containers for a while, and I think we knew there was something in it, but we weren't quite sure what it was. So, I think, although we were playing around with it quite early, I think we were quite slow to choose one platform or another. I think in the end, we—in the intervening period, I guess, between when we went from the more classical way of running Puppet across a bunch of EC2 instances that run a version of your application, the next step after that was switching over to using ECS. So, Amazon's container service. And I guess the thing that prompted a bit more curiosity into Kubernetes was that—I forgot the projects I was working on, but I was working on a team for a little while, and then I switched to go do something else. And I needed to put a new service up, and rather than just doing the thing that I knew, I thought, “Well, I'll go talk to the other teams.” I'll talk to some other people around the company, and find out what's the way that I ought to be doing this today, and there was a lot of work around standardizing the way that you would stand up an ECS cluster. But I think even then, it always felt like we were sharing things in the wrong way. So, if you were working on a team, you had to understand a great deal of Amazon to be able to make progress. And so, back when I got started at Uswitch, when I talk about doing the work about the energy migration, AWS at the time really only offered EC2, load balancers, firewalling, and then eventually relational databases, and so back then the amounts of complexity to stand up something was relatively small. And then come to a couple of years ago. You have to appreciate and understand routing tables, VPCs, the security rules that would permit traffic to flow between those, it was one of those—it was just relatively non-trivial to do something that was so core to what we needed to be able to do. And I think the thing that prompted Kubernetes was that, on the Kubernetes project side, we'd seen a gradual growth and evolution of the concepts, and abstractions, and APIs that it offered. And so there was a differentiation between ECS or—I actually forget what CoreOS's equivalent was. I think maybe it was just called CoreOS. But there are a few alternative offerings for running containerized, clustered services, and Kubernetes seems to take a slightly different approach that it was more focused on end-user abstractions. So, you had a notion of making a deployment: that would contain replicas of a container, and you would run multiple instances of your application, and then that would become a service, and you could then expose that via Ingress. So, there was a language that you could use to talk about your application and your system that was available to you in the environment that you're actually using. Whereas AWS, I think, would take the view that, “Well, we've already got these building blocks, so what we want our users to do is assemble the building blocks that already exist.” So, you still have to understand load balancers, you still have to understand security groups, you have to understand a great deal more at a slightly lower level of abstraction. And I think the thing that seemed exciting, or that seems—the potential about Kubernetes was that if we chose something that offered better concepts, then you could reasonably have a team that would run some kind of underlying platform, and then have teams build upon that platform without having to understand a great deal about what was going on inside. They could focus more on the applications and the systems that they were hoping to build. And that would be slightly harder on the alternative. So, I think at the time, again, it was one of those fortunate things where I was just coming to the end of another project and was in the fortunate position where I was just looking around at the various different things that we were doing as a business, and what opportunity there was to do something that would help push things on. And Kubernetes was one of those things which a couple of us had been talking about, and thinking, “Oh, maybe now is the time to give it a go. There's enough stability and maturity in it; we're starting to hit the problems that it's designed to address. Maybe there's a bit more appetite to do something different.” So, I think we just gave it a go. Built a proof of concept, showed that could run the most complex system that we had, and I think also did a couple of early experiments on the ways in which Kubernetes had support for horizontal scaling and other things which were slightly harder to put into practice in AWS. And so we did all that, I think gradually it just kind of growed out from there, just took the proof of concepts to other teams that were building products and services. We found a team that were struggling to keep their systems running because they were a tiny team. They only had, like, two or three engineers in. They had some stability problems over a weekend because the server ran out of hard disk space, and we just said, “Right. Well, look, if you use this, we'll take on that problem. You can just focus on the application.” It kind of just grew and grew from there.Emily: Was there anything that was a lot harder than you expected? So, I'm looking for surprises as you're adopting Kubernetes.Paul: Oh, surprises. I think there was a non-trivial amount that we had to learn about running it. And again, I think at the point at which we'd picked it up, it was, kind of, early days for automation, so there was—I think maybe Google had just launched Google Kubernetes engine on Google Cloud. Amazon certainly hadn't even announced that hosted Kubernetes would be an option. There was an early project within Kubernetes, called kops that you could use to create a cluster, but even then it didn't fit our network topology because it wouldn't work with the VPC networking that we needed and expected within our production infrastructure. So, there was a lot of that kind of work in the early days, to try and make something work, you had to understand in quite a level of detail what each component of Kubernetes was doing. As we were gradually rolling it out, I think the things that were most surprising were that, for a lot of people, it solved a lot of problems that meant they could move on, and I think people were actually slightly surprised by that. Which, [laughs], it sounds like quite a weird turn of phrase, but I think people were positively surprised at the amount of stuff that they didn't have to do for solving a fair few number of problems that they had. There was a couple of teams that were doing things that are slightly larger scale that we had to spend a bit more time on improving the performance of our setup. So, in particular, there was a team that had a reasonably strong requirement on the latency overheads of Ingress. So, they wanted their application to respond within, I don't know, I think it was maybe 200 milliseconds or something. And we, through setting up the monitoring and other bits and pieces that we had, we realized that Ingress currently was doing all right, but there was a fair amount of additional latency that was added at the tail that was a consequence of a couple of bugs or other things that existed in the infrastructure. So, there was definitely a lot of little niggly things that came up as we were going, but we were always confident that we could overcome it. And, as I said, I think that a lot of teams saw benefits very early on. And I think the other teams that were perhaps a little bit more skeptical because they got their own infrastructure already, they knew how to operate it, it was highly tested, they'd already run capacity and load tests on it, they were convinced that it was the most efficient thing that they could possibly run. I think even over the long run, I think they realized that there was more work that they needed to do than they should be focusing on, and so they were quite happy to ultimately switch over to the shared platform and infrastructure that the cloud infrastructure team run.Emily: As we wrap up, there's actually a question I want to go back to, which is how you were talking about the shifting priorities now that you've become CTO. Do you have any sort of examples of, like, what are the top three things that you will always care about, that you will always have the energy to think about? And then I'm curious to have some examples of things that you can't deal with, you can't think about. The things that tend to drop off.Paul: The top three things that I always think about. So, I think, actually, what's interesting about being CTO, that I perhaps wasn't expecting is that you're ever so slightly removed from the work, that you can't rely on the same signals or information to be able to make a decision on things, and so when I give the Kubernetes story, it's one of those, like, because I'd moved from one system to another, and I was starting a new project, I experienced some pain. It's like, “Right. Okay, I've got to go do something to fix this. I've had enough.” And I think the thing that I'm always paying attention to now, is trying to understand where that pain is next, and trying to make sure that I've got a mechanism for being able to appreciate that. So, I think a lot of the things I try to spend time on are things to help me keep track of what's going on, and then help me make decisions off the back of it. So, I think the things that I always spend time on are generally things trying to optimize some process or invest in automation. So, a good example at the moment is, we're talking about starting to do canary deployments. So, starting to automate the actual rollout of some new release, and being able to automate a comparison against the existing service, looking at latency, or some kind of transactional metrics to understand whether it's performing as well or different than something historical. So, I think the things that I tend to spend time on are process-oriented or are things to try and help us go quicker. One of the books that I read that changed my opinion of management was Andy Grove's, High Output Management. And I forget who recommended it to me, but somebody recommended it to me, and it completely altered my opinion of what value a manager can add. So, one of the lenses I try to apply to anything is of everything that's going on, what's the handful of things that are going to have the most impact or leverage across the organization, and try and spend my time on those. I think where it gets tricky is that you have to go broad and deep. So, as much as there are broad things that have a high consequence on the organization as a whole, you also need an appreciation of what's going on in the detail, and I think that's always tricky to manage. I'm sorry, I forgot what was the second part of your question.Emily: The second part was, do you have any examples of the things that you tend to not care about? That presumably someone is asking you to care about, and you don't?Paul: [laughs]. Yeah, it's a good question. I don't think it's that I don't care about it. I think it's that there are some questions that come my way that I know that I can defer, or they're things which are easy to hand off. So, I think the… that is a good question. I think one of the things that I think are always tricky to prioritize, are things which feel high consequence but are potentially also very close to bikeshedding. And I think that is something which is fair—I'd be interested to hear what other people said. So, a good example is, like, choice of tooling. And so when I was working on a team, or on a problem, we would focus on choosing the right tool for the job, and we would bias towards experimenting with tools early, and figuring out what worked, and I think now you have to view the same thing through a different lens. So, there's a degree to which you also incur an organizational cost as a consequence of having high variability in the programming languages that you choose to use. And so I don't think it's something I don't care about, but I think it's something which is interesting that I think it's something which, over the time I've been doing this role, I've gradually learned to let go of things that I would otherwise have previously thoroughly enjoyed getting involved in. And so you have to step back and say, “Well, actually I'm not the right person to be making a decision about which technology this team should be using. I should be trusting the team to make that decision.” And you have to kind of—I think that over the time I've been doing the role, you kind of learn which are the decisions that are high consequence that you should be involved in and which are the ones that you have to step back from. And you just have to say, look, I've got two hours of unblocked time this week where I can focus on something, so of the things on my priority list—the things that I've written in my journal that I want to get done this month—which of those things am I going to focus on, and which of the other things can I leave other people to get on with, and trust that things will work out all right?Emily: That's actually a very good segue into my final question, which is the same for everyone. And that is, what is an engineering tool that you can't live without—your favorite?Paul: Oh, that's a good question. So, I don't know if this is a cop-out by not mentioning something engineering-related, but I think the tool and technique which has helped me the most as I had more and more management responsibility and trying to keep track of things, is bullet journaling. So, I think, up until, I don't know, maybe five years ago, probably, I'd focus on using either iOS apps or note tools in both my laptop, and phone, and so on, and it never really stuck. And bullet journaling, through using a pen and a notepad, it forced me to go a bit slower. So, it forced me to write things down, to think through what was going on, and there is something about it being physical which makes me treat it slightly differently. So, I think bullet journaling is one of the things which has had the—yeah, it's really helped me deal with keeping track of what's going on, and then giving me the ability to then look back over the week, figure out what were the things that frustrated me, what can I change going into next week, one of the suggestions that the person that came up with bullet journaling recommended, is this idea of an end of week reflection. And so, one of the things I try to do—it's been harder doing it now that I'm working at home—is to spend just 15 minutes at the end of the week thinking of, what are the things that I'm really proud of? What are some good achievements that I should feel really good about going into next week? And so I think a lot of the activities that stem from bullet journaling have been really helpful. Yeah, it feels like a bit of a cop-out because it's not specifically technology related, but bullet journaling is something which has made a big difference to me.Emily: Not at all. That's totally fair. I think you are the first person who's had a completely non-technological answer, but I think I've had someone answer Slack, something along those lines.Paul: Yeah, I think what's interesting is there there are loads of those tools that we use all the time. Like Google Docs is something I can't live without. So, I think there's a ton of things that I use day-to-day that are hard to let go off, but I think the I think that the things that have made the most impact on my ability to deal with a stressful job, and give you the ability to manage yourself a little bit, I think yeah, it's been one of the most interesting things I've done.Emily: And where could listeners connect with you or follow you?Paul: Cool. So, I am @pingles on Twitter. My DMs are open, so if anybody wants to talk on that, I'm happy to. I'm also on GitHub under pingles, as well. So, @pingles, generally in most places will get you to me.Emily: Well, thank you so much for joining me.Paul: Thank you for talking. It's been good fun.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.

Some of the highlights include:  Why Vodafone moved to a cloud native architecture. As Tom explains, the company was struggling to manage operations across more than 20 markets. They also needed to improve the customer experience, and foster customer loyalty.  Why their business and engineering teams were both in favor of cloud native. The benefits of deploying daily operational activities around a single cloud native platform.   An overview of where Vodavone currently is in their overall cloud native journey. Tom also explains how cloud native conversations have changed inside of the company throughout their journey, as various business units have caught on to the benefits of the cloud. Vodafaone's transition from outsourcing roughly 97 percent of their operations, to bringing 95 percent in house. Tom explains how this has improved efficiency and expedited time to market. The challenge that Vodafone faced in trying to apply legacy network security solutions to distributed and dynamic systems.  Tom's thoughts on why Vodafone's cloud native transition and modernization efforts have been crucial to their success over the last five years.  Links: Vodafone Group: https://www.vodafone.com/ Connect with Tom on LinkedIn: https://uk.linkedin.com/in/tom-kivlin-5b469321 The Business of Cloud Native: http://thebusinessofcloudnative.com  Tom's Twitter: https://twitter.com/tomkivlin CNCF GitHub: https://github.com/cncf CNCF Slack: https://slack.cncf.io/ Kubernetes Slack: http://slack.kubernetes.io/ TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast, where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I'm Emily Omier, your host, and today I am chatting with Tom Kivlin. Tom, thank you so much for joining us.Tom: You're welcome. No problem.Emily: Let's just start out with having you introduce yourself. What do you do? Where do you work, and what do you actually do during your workday?Tom: Sure. So, I'm a principal cloud orchestration architect at Vodafone Group. I work in the UK. And my day job consists of providing guidance and strategy and architectural blueprints for cloud-native platforms within Vodafone. So, that's around providing guidance to the software domains that are looking to adopt cloud-native architectures and methodologies and also to the more traditional infrastructure domains to try and help them provide their services in a more cloud-native manner to those modern teams.Emily: And what does that mean when you go into the office—or your home office, go into your dining room where your laptop is, I don't know—what do you actually do? What does an average day look like?Tom: It can vary. So, depending on the activity at the time, it could be anything from preparing a global policy that needs to go through the senior technology leadership team, to preparing some extremely detailed requirements for selection process or creating some infrastructures code, or the code artifacts for the deployment of cloud-native services, whether that's in our lab, or to help our services teams within Vodafone.Emily: Tell me a little bit more about what pain made Vodafone think about moving to cloud-native and Kubernetes.Tom: Primarily, it was the challenge of having 25 different markets, or 23 now. We launched a digital strategy to—so back in 2015, we launched a five-year strategy, which we wanted to massively increase the rollout of 4G, of converged network offerings, of improved customer experience. And we found that the traditional way of managing software was not supportive enough in our ambition. And so, having to choose cloud-native technologies, things like Kubernetes, but also the modern operating models, that was the driver: it was to improve our customer experience, and our customer-affecting KPIs, really.Emily: And when you say it wasn't supportive enough, what do you mean specifically?Tom: So, things like time to market, for example. So, if we wanted to offer a new service—so one of the things that 4G started the drive towards was a more granulated service offering to consumers, and so lots of different things could be offered. And if it took you six months to think of an idea and then have to go through—or even longer than six months to get to the point where that could be offered to customers, even if it was just a very minor feature within an existing product, then that's not going to engender customer loyalty. And so, things like the cloud-native mindset, where there's a much closer link between the engineering teams and the customer, there are much shorter periods of time between ideas coming in from the customers and then being delivered back to the customers as product features, that sort of time to market was really enabled by cloud-native technologies and mindsets.Emily: And how does having two dozen, more or less, different markets, how does that play into the decision A) to move to cloud-native in general, and managing the IT infrastructure?Tom: So, one of the things that's really driven it is trying to simplify and reuse artifacts. So, if you've got 23 markets all doing a different thing, then there's obviously a lot of duplication happening across the group, whereas if everyone's using the same technology in the same platforms—take Kubernetes as the example—everyone can write their software for that platform. Everyone can write their operational ecosystem around that platform. So, the deployment artifacts, the pipelines, the day two operational activities, they can all be based around that single cloud-native platform. And so, that enables a huge amount of efficiency from the operational side. And that in turn allows those engineering teams to focus on things that are adding value to the business and the customer instead of having to focus on fairly low-level tasks that are just keeping the lights on, if you like.Emily: What's different for each one of those markets?Tom: So, it might be something like language, it might be something as simple as that. It may be that the offerings are slightly tweaked. So, rather than, I don't know, as an example, rather than Spotify being included as a kind of add on, it might be some other service that's more relevant to that market. It may be that there are particular regulatory requirements that are specific to a market that needs to be considered within the product design and the engineering of it. And so, having a cloud-native response allows sharing and reuse of artifacts where we can, but still allows for that customization where it's required.Emily: Where would you say Vodafone is in the cloud-native journey? Do you feel like you've, mission accomplished?Tom: So, mission accomplished, as in the first step, yeah. So, we set out a goal in 2015, to get a certain number of our applications to the Cloud, and that's largely been reached, I think, especially with our customer channels, so that the kind of points of interaction with the customer, the huge number of those are cloud-native today. And things like automated customer interaction with chatbots, and the like, that's all added to the cloud-nativeness of the interaction. As part of our next iteration, we'll be looking for more cloud-native software and cloud-native platforms, and that will start extending into the network systems themselves, as well as the more digital and easily modernizable layers, if you like.Emily: What sort of business value do you feel like you're looking for as you move to the next step?Tom: So, primarily, it's going to be driven by customer satisfaction and customer affecting KPIs, like I said before. That's always what's driven the business metrics anyway. So, things like being able to support the demand of the customer. So, whether that's the new 5G services, for increased bandwidth. So, obviously, if our network systems themselves are cloud-native, then taking advantage of the auto-scaling, and the auto-healing, and the autonomic nature, then the customer experience, and the customer satisfaction will increase. Improving time to market, so again, part of 5G is that the whole notion of creating more differentiating services, and so if we can do that through the cloud-native mindset with product owners being much more closely engaged with customers, then that improves our product offerings. And we can optimize our network profitability by using cloud-native features like modern big data analytics, and even AI and automation to improve the operations of the network. At the end of the day, the business value is improved customer satisfaction, which improves our financial performance, obviously.Emily: And when you started out in 2015, who was pushing for moving to cloud-native? Was this the business saying, “Hey, how do we improve customer satisfaction?” Was it engineering saying, “Hey, here's an idea for something that could help us move faster?” Who was behind that?Tom: That's a good question. I think it's probably an element of both. It was the opposite of the push me, pull you, I guess. So, there was engineering pushing on an open door, I suppose you could say. So, Cloud was a bit of a buzzword around that time anyway, but I think it's fair to say the concepts of improved time to market, improved stability, the potential for improved security, improved automation, and repeatability, they were all relatively easy sells to product teams who want to be able to sell products to customers. And once you're able to explain what problems those concepts solve, I think it became a bit of a, like I say, pushing on an open door.Emily: Can you tell me a little bit about the process of explaining what problems these things solve? Was there anything that was getting lost in translation?Tom: Yeah. I think the biggest thing that I can recall—obviously, it's a company-wide thing. I'm never going to be aware of everything that happens—certainly, it's critical to try and understand what the target operating model is before trying to say, “Here's the technology solution to it.” So, I think some of the lessons that were learnt in the early stages were, rather than trying to say, “Here's the technology answer to a modern way of working that hasn't been agreed or adopted or even understood yet,” let's do that part first, so people understand how they need to work in this modern kind of culture. And then the technology answers then make a bit more sense to people because they're able to say, “Okay, I understand the problems that's solving now because I'm now working in that way of working.” So, that's probably the biggest learning point I would take from the previous five years.Emily: Do you feel like the conversation, how did it evolve from the first conversations over the course of the past five years, and then what's it like now?Tom: It's very different now. The concept of Cloud and cloud-native has become a given and very well understood across the business, even outside of technology. So, we talked to other business units, and they're quite comfortable in understanding the benefits of Cloud. And it's now about when they mature into cloud-native, and when they mature operating models, rather than if. And it's now talking and giving guidance about how to do it, rather than trying to sell the concept itself. So, it just feels like you're at that next stage of not having to sell the idea anymore, and more into the detail of how to implement that idea.Emily: What would you say were some of the biggest surprises? And let's start with thinking about some of the biggest surprises, not necessarily technically but organizationally, in how engineering was talking with the business, how people were working together. Was there anything about this journey that was unexpected?Tom: Not particularly. I think the biggest change that happened, which was possibly unexpected when we started, was the level of insourcing that we have undertaken to support the cloud-native operating models, the time to market, and the modern engineering teams. So, we used to be around 97 percent outsourced or something like that, in terms of building software that wasn't just vendor supplied. And for all that software now, we're more like 95 percent in-house. And so, that's quite a big change, and I think that probably surprised people that A) we needed to do it, and B) that we have done it, and relatively successfully got pretty wide-scale digital engineering functions across many markets now.Emily: And why do you think that matters?Tom: Because it gives us control of the roadmap, it gives us control of that time to market cadence, and it allows us to use the data that our teams understand and know about, and to share that with other markets. So, as I say, even though an engineering team might be in the UK, they can share what they've done, they can share the artifacts, they can share the data that's driven decisions and software activity with other markets within Vodafone. And that just improves that efficiency, again.Emily: Do you think insourcing also improves customer satisfaction KPIs?Tom: Certainly we've seen that. So, whether that's a correlation or causation is kind of for someone with more access to more data than I've got. But certainly, we've seen an increase in online sales, and our digital marketing is more data-driven. And that has happened in correlation with the in-sourcing of software engineering skillset, yeah.Emily: Do you have any specific examples that come to mind in, maybe you are able to react in a way that wouldn't have been possible if you'd been using the old system?Tom: I'm not aware of any specific examples, unfortunately.Emily: Was there anything about the move to Kubernetes, to cloud-native, that you expected to be difficult, and wasn't. So, that was easier than you expected?Tom: That's a good question. I suspect the provision of multiple clusters. Kubernetes is difficult. It's a complex system, hence why there are so many cluster management vendor offerings available. And I think we chose a couple of partners early on in the journey to help us with that, and I think that really helped, and it made Kubernetes a little less scary for the software teams who were using it. So certainly, I've heard feedback—this is anecdotal, rather than anything that's evidence-driven—actually, just being able to create clusters and deploy into them was easier than people had thought when they were learning about Kubernetes through the quick start tutorials and the like.Emily: Was there anything that sticks out as being far more difficult than expected? The more unpleasant surprises?Tom: I wouldn't necessarily call them unpleasant, but obviously there's going to be a transition period—which we're in—between the traditional data-center-centric networking and network security policies and concepts, and those that work with Cloud and cloud-native platforms like Kubernetes. And there have definitely been challenges in trying to apply the legacy approach to network security with a distributed and dynamic system like Kubernetes, where you can't give everything a static IP address or even have separate subnets within a cluster for segregation, for example. It has to be done in a different way. You can still apply the same controls, they just have to be done in a different way. So, I think that's one of a few challenges that we found that we've had to work through with different vendors, with engineering teams, and with our internal teams to try and update our guidance on how to apply those controls.Emily: And to what extent have there been organizational challenges, and how have you gotten over those?Tom: That's a tricky one to answer, really. I think it all comes down to the balance between understanding and buying into a strategy, but then applying that to application lifecycle and investment lifecycles. So, I think this is probably true for any company: just because a strategy says this is the thing to do, you got a roadmap for your portfolio of applications and services that you need to balance a limited budget. And so, that's been the biggest challenge, is to try and identify how much of each budget at various levels can be spent on strategic activity, and then for which services, and trying to keep that balance, and bearing in mind that there are lots of different things pulling on that same pot of money.Emily: And what have you learned about managing that?Tom: I think primarily that there needs to be a holistic view of strategic projects. It's quite difficult to put the onus on a local budget, to spend the money to do something strategic when the benefits are probably—and the business case is probably seen more widely than the individual budget area. But I think it differs between situations, and between markets, and what's happening. I think the primary thing is to understand the costs of the strategy upfront, and try and work those costs into whatever needs doing over the period.Emily: A slightly different question, which is, is there anything you feel like in the cloud-native journey that you're still working on solving, that you haven't really figured out yet?Tom: I'm not sure whether we haven't figured this out yet, but one of the things we're putting a lot of effort in at the moment, is the use of advanced data and analytics platforms to try and drive even more network automation, and network planning efficiencies. So, I think it was last year at Google Next, we announced a partnership with Google to make use of their data services. And there's a few projects ongoing within Vodafone to try and drive the amount of knowledge and useful information we can gather from the vast quantities of data we have about our services and the customers that use them because the more we can use that data, the more we can respond to customer need in a timely manner, whether that's reactively in terms of operational response or whether that's proactively in seeing trends that we can then meet a need that may be unsaid yet.Emily: And if you were to talk to another engineering leader who was trying to push through the open door as you were saying, what advice would you give them?Tom: The biggest bit of advice is to understand the current way of working for whichever area you're—is on the other side of the door, and understand their pain points because it's not always the same answer. So, generalizing, it may be that one area is more than happy to have a centralized global platform offering, whether that's within our data centers, or public cloud, or both. Another area, just the way it's managed, may require a more distributed model, where the services are offered on a more market specific level. And so, I think that that's the main thing, is to understand the specifics of that area that you're talking to because it will affect how you want to architect and onwardly deploy and manage that technology.Emily: It would affect not just how you want to architect the technology, but also how you want to communicate what your plan is, right?Tom: Absolutely. Yeah. So, in the first of the examples I gave, where an area might be happy with a centralized service, that probably means they're already using one. The way you would communicate that would be via that existing channel, if you like. Whereas on the flip side, that kind of channel may not exist, and therefore running the project or projects and communicating with stakeholders would be much more distributed.Emily: At Vodafone was there ever any challenge selling it, not just over to the business side, but also selling internally inside engineering teams? Or was everyone pretty gung ho to do this?Tom: No, there's always challenges. I think again, it goes back to understanding the pain points of an area and understanding why things are the kind of as they are today, which I guess is general for things outside of technology and outside of Vodafone generally is. If you understand the position of the person you're debating with, then you're more likely to reach a common understanding than if you go into it with your own point of view and being unwilling to listen. So, I think that's the main thing is just being willing to listen, to understand pain points, and to be able to react to those within a strategy. You'd hope that it's flexible enough to be able to meet a wide range of needs without needing to necessarily change the overall vision.Emily: How important do you think this cloud-native transition has been for Vodafone?Tom: I think it's been crucial. I think we couldn't have done what we've done in the last five years without it. So, there's a video that our group CTO has posted on LinkedIn recently which highlighted a few things around improved mobile KPIs, we've got 4G in 21 markets, we've got the largest 5G in Europe, and all of those improvements from time to market I've already mentioned, we simply couldn't have done that without a modernization program to move to cloud-native across a number of our systems. So, yes, that's partly a technology thing, but also, it is such a cultural thing, and having that modern way of working where you have your modern engineering teams who are closer to the customer, but they're also—the different mindset of a modern engineering company where you're not afraid to try new things, and if you fail, you learn from them. And I think that's all part of what I would class as cloud-native, and that has been, like I say, it's been crucial for us to be able to get where we have been.Emily: It's interesting to think cloud-native means if you fail, you learn from it. That's a fairly basic concept, and yet true. I can see how that is, sort of, part of being cloud-native.Tom: Yeah, it's one of those things is quite a basic thing, but I think in traditional ways of working, the focus on the availability of systems and the performance of systems can blind everyone to the possibilities outside of that particular area of focus. And it puts pressure on people at all levels to try and minimize periods of downtime or periods of low performance. And over time, people become less and less willing to be able to try new things, through fear of failing because just the way people work it's difficult to learn from those failings because it affects customers. And so, what cloud-native technologies enable because of the way things are orchestrated—things are dynamic, things are repeatable—it's very easy to try new things, and not affect all customers. Now, obviously, good software engineering practices help as well. But I think the cloud-native technologies and the ways of working really do support the whole “learn by failing” premise.Emily: Do you think it would have been possible to get the customer satisfaction KPIs that you did, without moving to cloud-native, in any other way?Tom: I think the only way you could have done is by a huge investment in people and the traditional technologies. It would have been a much more expensive and slower journey, in my opinion.Emily: Anything else that you want to add about your experience moving to cloud-native?Tom: No, I don't think so. I think one of the things—like I said before, the increase in automation, the increase in the modern technologies is just really helped with those customer affecting KPIs, and that has to be the drive for why you're doing it.Emily: All right, just a couple more questions, then. What is your can't-live-without engineering tool?Tom: Oh, that's a good question. Probably Python. I think so many people use it either as a cross-platform scripting tool to be able to automate things and get on the first step towards cloud-native, or it's such a key part of many cloud-native tools like things like Ansible and other tools, and it's used hugely within our data analytics domain to try and drive the usefulness of the data. So, yeah, that's probably the one I'd choose.Emily: And then this actually is the last question which is, how can listeners follow you or connect with you?Tom: So, I'm on Twitter at @tomkivlin. I'm also on LinkedIn. So, I'm Tom Kivlin, working for Vodafone Group. I am a member of the telecom user group within the CNCF. So, you can find them on GitHub and also in the… I think it's the CNCF or the Kubernetes Slack. And yeah, happy to share experiences and keep learning.Emily: Well, thank you so much. Again, this is Tom Kivlin, and we'll go ahead and wrap it up there. Thank you so much, Tom.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

This conversation covers:  Why many businesses are shifting away from analyzing total cloud spend (CapEX vs. OpEX) and are now forecasting spend based around usage patterns. The difference between cloud-native, cloud computing, and operating in the cloud.  The delta that often exists between engineering teams and business stakeholders regarding costs. Travis also offers tips for aligning both parties earlier in the project lifecycle. Common misconceptions that exist around cost management, for both engineers and business stakeholders. For example, Travis talks about how engineers often assume that business teams manage purely to dollars and cents, when they are often very open to extending budgets when it's necessary. Tips for predicting cloud spend, and why teams usually fall short in their projections. Why conducting cloud cost management too early in a project can be detrimental.  Comparing the cost of the cloud to a private data center.  The growing reliance on multi-cloud among large enterprises. Travis also explains why it's important to have the right processes in place, to identify cross-cloud saving opportunities.  How IT has transitioned from a business enabler to a business driver in recent years, and is now arguably the most important component for the average company. Links: Twitter: https://twitter.com/TravisWRehl LinkedIn: https://www.linkedin.com/in/travis-rehl-tech/ Main Company Site: https://cloudcheckr.com  CloudCheckr All Stars: https://cloudchecker.com/allstars  TranscriptAnnouncer: Welcome to The Business of cloud-native podcast, where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to the Business of cloud-native. I'm your host, Emily Omier, and I'm here today with Travis Rehl, who is the director of product at CloudCheckr. Travis, I just wanted to start out, first of all, by saying thank you for joining me on the show. And second of all, if you could just start off by introducing yourself. What you do, and by that I mean, what does an actual day look like? And some of your background?Travis: Yeah. Well, thanks for having me. So yeah, I'm Travis Rehl, director of product here at CloudCheckr. What that really means is, I have the fun job of figuring out what should the business do next in relation to our product offering here at the business. That means roadmap, looking at the market, what are customers doing differently now, or planning to do differently over the next year, two years or so, on cloud? What their cost strategies are, what their invoicing and chargeback strategies are, all that type of fun stuff, and how we can help accommodate those particular strategies using our product offering. Sort of, day to day, though, I would say that a bunch of my time during the day is spent talking to customers, figuring out where they are in their cloud journey, if you will, what programs or projects they may have in flight that are interesting, or complicated, or they need help on. Especially making any sort of analysis help in particular, and then lastly, taking all that information and packaging it up neatly, so that the business can make a decision to add functionality to our product in some way that can assist them move forward.Emily: The first question I wanted to ask is actually if you could talk just a little bit about the distinction between cloud-native, and cloud computing, and operating in the cloud. What do all of those things actually mean, and where's the delta between them?Travis: Sure. Yeah so, it's actually kind of interesting, and you'll hear it a little bit differently from different people. In my background, in particular—I used to run an engineering department for a managed service provider. And so we used to do a lot of project planning of companies as to what's their strategy for their software deployment of some kind on cloud. And typically the two you see for, say, cloud-native versus operating in the cloud, operating on the cloud is very atypical. You'd associate that to something like lift and shift—probably hear about a lot—the concept of taking your on-prem workload and simply cloning it, or taking it in some way and copying in some way, on to the cloud-native vendor in particular. So, literally just standing up servers of clones of hard drives and so forth, and emulating what you had on-prem, but on the cloud. That's a great technique for moving quickly to cloud. That's not a great technique if you want to be cloud-native. So, that's really the big segue for cloud-native, in particular, is you want to build a software solution that takes advantage of cloud-only technology, meaning serverless compute resources, meaning auto-scaling different types of services themselves, stuff you probably didn't have when you're on-prem originally, that you now have, you can take advantage of on the cloud. That's almost like a redesign, or reimplementation around those models that cloud itself provides to you. That, to me, is the big difference. And I see oftentimes that gap-wise, many companies who are starting on-prem, they will do the migration to cloud first, the lift and shift model, and then they will decide, “Hey, I want to redesign pieces of it to make it more cloud-native.” And then you'll see startups who don't have on-prem at all, they will just go into cloud-native from the get-go.Emily: Of course, CloudCheckr specializes in helping with costs among some other things, but how do costs fit into this journey, and what sort of cost-related concerns do companies have as they're on this cloud journey?Travis: Yeah, so there's a few. I would actually say that years ago—just to clarify, the discussion has changed over the last few years—but years ago, it started with CapEx versus OpEx costs, specifically for purchasing of your IT services. On-prem, you'd probably purchase up-front a bulk number of VMs or servers or otherwise, for a number of years, and so be a CapEx cost. When you moved over to cloud and more of this, usage-based, model kind of threw a lot of people for a loop when it came to more OpEx usage space models. AWS, Azure, GCP have helped in that regard with things like reserved instances for companies who are more CapEx oriented as well, but in terms of the initial years ago, a big hurdle was communicating that difference and how the business may pay for these services. And a lot of people were very interested in moving to OpEx back then, in particular. When it came to how do you take into account all these cost-related changes the business may go through, one of the big ones that I see most recently is around the transference and storage of data. In the past, it would have been about how much money total am I going to spend on the cloud itself. Now, it's about what am I forecasting to spend based off of those usage patterns. It's a bit easier to forecast those things when you have servers that run for a period of time, but when you have usage patterns for data ingestion, for data transfer, for servers spinning up and spinning down and scaling out horizontally, that pattern becomes a bit more fluid. So, that's typically a conversation that comes up quite often. That's the type of thing that CloudCheckr and products like us can help with.Emily: Do you think there's any delta between how engineering teams think about and understand cost, and business stakeholders think about and understand costs?Travis: I would say yeah, there's a fairly significant difference there. I would say that engineers initially care a little bit less about cost. They have an objective. They're trying to solve for a project or goal they're trying to achieve. And so as a result, they're saying, “How can I achieve that quickly?” And that changes over time as the product becomes closer to production-ready. They may say, “Okay, now I want to optimize a bit more what I built.” Whereas the business is more thinking about, “I have this project plan in front of me, how do I go as quickly as possible, without incurring so much costs that it goes over budget, or I don't foresee a particular budgetary circumstance occurring?” It's slightly different mentalities, to be quite honest with you. What I see the most of is that they do align towards the end of the project, or the steady-state of the project, when the team has delivered the thing, whatever that may be, but need to make a decision quite quickly as to how do they want to either cap those costs moving forward or come up with an appropriate sort of budgetary model that scales linearly or otherwise, that both sides can agree upon?Emily: And are there any best practices that you can think of for how that gap can be bridged earlier in the process?Travis: Yeah. What's interesting, you're seeing, probably in the last couple years now—as you've probably heard, “Cloud Centers Of Excellence.” I'm assuming you've heard that term?Emily: Mm-hm. Yep.Emily: Yeah. So, one of the big amplifiers of that is bringing in your financial team to that planning model for how you want to deploy and manage cloud resources, so they have a say early on in the process, but also, it's more about communicating what the plan needs to be. Typically, I see the engineering team—the product manager, right—going off and saying, “We need a build and go fast.” And then it's the financials—accounting or otherwise—that says, “Hey, what happened to this project once you delivered it?” To bring them in earlier to that conversation via a CCOE sort of methodology—Cloud Center Of Excellence methodology—I see helps the most. It's really about communicating first. The second, though, is about setting corporate guidelines for what people are allowed or not allowed to do. Shadow IT has been a problem for a very long time for businesses. Cloud actually, in my opinion, can amplify that problem because of how easy it can be to start spinning up resources, and doing projects, or otherwise. So, it's pretty important for central IT, or for that Center Of Excellence group to set the standards of how the business needs to operate from that point forward, but then allowing different organizations, business units, or groups to deploy and manage at the cadence they need.Emily: What do you think are some misconceptions on both sides about cost management? So, misconceptions from engineers and misconceptions from business stakeholders?Travis: I think, as a product person, I have the fun job of being someone in the middle of it all. I think engineers have a misconception that the business side is managing to dollars and cents. That can often be the case. So, if you're an engineer, and you are a creative type, and you want to build something, and you have someone out there saying, “No, no, no, you can't do it because of XYZ thing,” cost or [unintelligible], or otherwise it can create this barrier between the two of them. But what I have learned, though, is that from an engineer communicating to a business person, especially on this subject, documenting and communicating the impact, the positive impact, and sometimes negative impact, but positive impact I'd recommend, on the business for why the project will enable your teams, or get you into a market faster, or solve a corporate problem that's been around for a while. The business side is very open, I found over the years, to hearing out those reasons, and agreeing to extending costs, or being more lenient on the cost model of a project so it can go faster. Likewise, though, on the business side, when they're working with engineers, it can be a little bit fraught when they think of, say if you've got a project moving and a business team says, “You have a budget for this and a line item,” and they say, “What's the project plan? When are you going to deliver? What's the velocity for delivering?” Cloud lets engineers do a bunch of new things they probably haven't had in the past, like faster deployment times to the product or a project in question; being able to react, or be proactive, however way you want to manage to it, for a customer requests, or otherwise. And as a result, it can be easy for that business person to expect a very linear approach to product development or delivery of services. And so for them, it's more about seeing and working with engineers to see how, “Hey, maybe the business can be improved by doing things a little bit differently than they had in the past.”Emily: Yeah. So, basically, what you're saying is, as long as the business is understanding this expense as an investment in something, usually it's not a big problem.Travis: Yeah. But it really comes down to communicating that. It can be so hard for an engineer, to say, “Hey, if I deliver this new pipeline, then I can deliver code, converge code, deploy code, test code faster.” That may not translate really well to a business person, and that's kind of the, I would hope, the role of product within your organization, or their organization, to help do that translation because if you're able to deploy quicker, that means you can solve customer problems faster, your time to market's faster, you can make changes to your cost model faster, with very little risk. But that's the communication gap that typically resides.Emily: And what do you think are some surprises that come up on the cloud journey regarding—we've already talked about how this is a shift from CapEx to OpEx, but what do people not fully understand about what those OpEx costs are going to be for example, or how it's going to work at the beginning of the journey versus at the end?Travis: Yeah, so at the beginning, I think there's an interpretation that—let's say you're spinning up a project for a very first time, and you're typically used to that CapEx model, and the team delivers their budget to you, the business person, and they're like, “It's going to cost us, round numbers, like$1,000 this month to get started.” And then by month two, they've scaled up, and they're two grand or three grand or something, whatever that number needs to be. The business is typically not prepared or is ready to be prepared to understand that it will change; the costs will change. It never is exactly what you ever thought it was going to be at to start. You can estimate all you want, there's tons of different calculators out there that helps you get close enough, but it's never going to be the number you always envisioned it to be. And that's just something you have to live with and get used to. It's more about the later on in the project that matters more. So, I think up front, it's very useful for project leaders to also communicate to the business that this is an estimate. We think it's going to start here, but we expect it to grow. And then, have good caps for yourself, almost like milestones to say, if you're getting closer to a particular cap, to a particular budget, maybe you should sit down for a moment and say, “Hey, are we doing this the wrong way, or can we do it a little bit differently?” Later on in the project, though, it gets kind of interesting. So, in my past life, I worked at a company where we did a lot of eCommerce implementations for other vendors, for other businesses, and one of the things always surprised people the most was data transfer costs. So, when we will be scoping a project, we would say, “Okay, here's the traffic we would expect to hit your new eCommerce solution, here's the type of things we expect them to do, here is the type of content, they need to load from the system, and thus, here's how much you're going to spend in transfer rates from the servers or otherwise—or CDN or otherwise—to the end-user.” And then what always hit us every single time is, like, Black Friday, or some kind of promotional event a marketing team would do, where we see a ton more traffic coming onto the system. And suddenly, all those atypical usage statistics would spike, you would be sending more images out to users, you'd be ingesting more traffic than you're anticipating, and you'd have more storage of their user information or otherwise, on your back end. And so, that sort of scaling model of usage patterns becomes more common over time because you see it more often. I like to put sort of, like, buffers on those particular areas of the budget, that say, “Hey, here's how much I think I'm going to spend on EC2, or computes, or VMs, or what have you, but I know that we ingest a lot of images from end-users, and we have to take in those images all the time, and I know we're going to run four promotions in a given year, and I should expect a 40 percent increase on those time periods. So, maybe my OpEx costs should be increased by a certain budget to accommodate that.” That type of planning has to start, has to become normal to happen between the project team and the accounting group, or financial group of some kind. So, there's some more of a clear agreement as to what realistically the budget needs to be long term.Emily: Do actual cloud costs ever come in less than estimated?Travis: I would say it's more often over what's estimated. The only times I see it coming in less is when the team delivering the solution implements a really sophisticated sort of data transfer strategy, or storage and archival strategy than expected. Maybe usage is less than you expect as well. Or, everybody should always be planning for some buffer, so if you've got a project that you think it's going to cost you $800 a month, maybe you should tell the business it's $1000. [laughs]. Sort of overachieve where you think it's appropriate.In that case, you're always going to come in lower than you expect. I actually think that cloud costs will always increase. That's just natural to do so. As you do more on cloud, you store more, you transfer more, you need more compute; it will always increase. I think it's more important to analyze the rates in which you are increasing. If you are spiking and it continues to spike, then there is a problem. If you're steadily growing, but your user account is growing, your revenue is increasing as well, and it's linear in some fashion, in that regard, that's a good thing. That doesn't mean something's wrong. But if you're spiking and things are consistently spiking, then you have a problem.Emily: Just like any other part of your business. If you're hiring more people, hopefully, it's because it means your business is growing. But you're also spending more—Travis: Exactly.Emily:—obviously. You were also saying sometimes the costs end up being less than expected because the company has invested in cost management strategies. Why—Travis: Yeah—Emily: Oh, go ahead.Travis: I was just going to say, I would say that companies don't typically start there. At least in the past, they haven't started first food forward, where they're saying, “I want to keep costs in check first, prior to doing the project.” Typically, they do a project, then they find that oh, they did too much, and now we need help. That's the normal I see.Emily: Yeah, in fact, I was going to ask, are there disadvantages to doing cloud cost management?Travis: In general?Emily: Yes, or scenarios where you would say, “Maybe this isn't what you're supposed to be focused on?”Travis: Yeah, I think it depends on the stage of either the entire company or the individual business unit or project. And what I mean by that is, if you're Google, you're probably managing a very small team, like they're a startup, or your startup yourself. And so, in the very early onset of any product delivery, cloud or not, it's typically more about volume and usage patterns of your end-users on the system. And you will spend money to get there, you will put in long hours to make it happen. Don't create barriers yourself, to achieve the goal that you have in mind for your team, your project, or otherwise. But then as the project matures, you typically see people sort of step back and say, “Okay, we made a great lot of great strides. We've hit a lot of our milestones, we have the user base we need, how do we tailor back a bit so that things become a bit more normalized and consistent on the delivery?” And so early on, I don't see a lot of people putting a lot of effort into cost savings, cost optimizations, and recommendations, just because things are fluid, dynamic, chaotic, whatever word choice you want to use for yourself at the time. But at some point in any business, you will come to the conclusion of, “Okay, we've done a lot of things; a lot of stuff. How do you then efficiently optimize what you have already delivered?” And that's where tools like CloudCheckr and others come into play to help you figure out that cost optimization and that strategy.Emily: Yeah. And can you think of any examples where you think, maybe this company would have paid less if they had done cost management, but maybe they wouldn't have accomplished x?Travis: Yeah. Without using a customer name, but to be honest, I forget their name offhand; it's about a year ago, we had a particular scenario where they're moving from on-prem. They were a video provider, and they were moving from on-prem to cloud. And in the move they had, it was something like $400,000 per month in wasted resources. Meaning they had IP addresses not attached to particular servers, or load balancers, they had storage volumes that are no longer in use, not even being backed up or anything, just kind of live off to the side, that people kind of forgot about. It was like $400,000 a month in that stuff. It was a lot of money. And they actually made a decision in the middle of the project, to back up and slow down for a moment and clean up all those things, and it probably cost them a couple months or so—because of the size and volume they're driving at—it probably cost them a couple months of their internal resource time, or otherwise to remediate those loss cost savings, and then continue the project forward with better guardrails. If I remember correctly, they actually missed a significant event day. Like, it wasn't Black Friday, it was some media day they had at the company they wanted to leverage a new system for so that it had a more performant experience with their customers to see videos and content they were delivering. And they had to leverage their older system at the time, which actually cost them more money at the moment because they had to spin up additional resources on their old system to handle that load, and it just wasn't optimal experience for themselves to do that. I think they broke even at the end of the day. I would have just gone faster on the new thing in that scenario, but that's how it played out for them.Emily: So, basically, what you're saying is that it's not always a really simple decision. There's pluses and minuses, and everybody has a limited amount of time, resources, and focus, so you have to decide what's most important at this moment for your organization.Travis: That particular daily experience is near and dear to my heart because that's the majority of my job. There are a lot of opportunities in front of you, a lot of bad things you could do, too, sometimes you're not going to get everybody happy, either. You need someone who's going to make a decision and move quickly, especially if you're a business that's on cloud in a competitive market that's trying to grow. You don't have months or years—sometimes even weeks—to make that type of decision. You kind of have to have all the data in front of you and say, “Here's the best one.” Cloud makes that harder to be quite honest with you. The speed at which you can deliver functionality, and your competitors themselves, the speed at which they can change means that you need to be very confident in what you can do and what your team can do, as well as understand what the risks are in front of you so you can make a decision quickly. If you're not ready to do that, then there may be problems ahead for you because as the market continues to go in that direction, that's just going to be a soft skill required moving forward for a lot of people.Emily: Do you think that the cloud is less expensive than data center? Do you think that cloud-native is less expensive than say someone doing a lift and shift?Travis: Yeah, so it depends a little bit. So, I can start with cloud-native and then work my way back to private cloud. It will be more expensive upfront, to go cloud-native typically, because you're building it from scratch, or redesigning a significant portion most likely of an existing application stack. So, the upfront overhead to go cloud-native is always going to be higher, especially when you factor in things like labor costs. However, though, once you have made that transition, costs can be tremendously lower. So, it's really more so about, is the business willing to take on the effort to make that leap upfront, or is there only so much that's palatable [laughs] at the time to move forward.But typically, it's a bigger spike upfront with much cheaper experience late game. When it comes to things like on-prem, or private cloud, or even hybrid cloud to some extent, I actually think if you have workloads that are very consistent, if you know exactly how much storage you're going to need, how much compute power you need, it's very stable, very consistent, it's probably cheaper to go to your private data center because you can purchase in bulk at the best rates for the time period you need for that specific thing, because you know exactly what it's going to be. That is only the case for a small subset of businesses in the world, who even have the analytics to come up with that data. But if you do, it's completely viable. Was it Dropbox, I think, recently—not recently. A couple years ago now, actually—they went off of their cloud provider. I think they're on AWS at a time, offhand. And they went to their own private data center. That could be butchered in this, slightly, but you get the point, is they knew exactly how much storage they need. And they found it to be cheaper to do it themselves on their own private data center. So, they did. Not a lot of companies can do that. But it's completely an option out there.Emily: What do you think the stakes are? And by which I mean, how big a piece of the pie is IT budgets?Travis: So, what funny is five, six years ago, if you were to ask me that question—this is back when even I worked more true to form IT—IT was not a business driver. It was overhead. It was, make sure my email is still on, and I have computers, and I can do these things, and connect the networks together. And it wasn't driving business decisions, type of a thing. That has radically changed over the last few years. I would actually say that IT is probably the most important aspect of your business now. Without a really strong IT department, you can't move fast on product development. Without a strong IT department, your engineering teams are—the velocity of the deliverables is slower. You can't leverage all these great new stuff that's coming out. So, IT now, to me, is probably the most important thing a business can truly manage appropriately. When it comes to the total budget, it has greatly increased because more responsibilities have also been placed upon IT.They now have to manage to different usage patterns from forecast, different budgets. They have to manage to new skill sets requirements for cloud in particular, and other different types of niche technologies that your business may need. That means that IT themselves has to either staff up or purchase technology solutions like CloudCheckr to accommodate that growth in the business requirement. So, the IT budget is greatly increasing, but for good reasons. It's because the business demands nimbleness, and only a strong IT department can deliver that.Emily: Does that mean though, that it's more important than ever to figure out where you're wasting resources, or at least be strategic about where you're spending those IT resources?Travis: I one hundred percent believe in the strategy of identifying and managing to cloud waste should be in the forefront of pretty much every conversation, among other items. When it comes to focusing entirely on cloud waste, I don't think that's the right decision. I think there's a way that manage to that, that the business will accept while allowing for the right velocity of improvements to products or services, but when it comes to defining—really this comes down to me is the Center Of Excellence, at the end of the day. The Center Of Excellence should be that IT is either the owner of, or is a prominent member of. And they should be defining what's the atypical cost-saving strategy? What is the security profiling? What is the particular way you want to tag your resources, or otherwise, so that you can allocate them to the right business unit, or project code, or etcetera, for that cost analysis? That is the objective of the Center Of Excellence is defining the strategy. Cost savings is a piece of—a significant piece of—but a piece of the overall strategy that you should have an answer to, but it should not be the primary driver.Emily: Anything else that you'd like to add?Travis: In relation to Centers Of Excellence?Emily: Just in relation, actually, to this whole discussion about cloud-native, costs, etcetera?Travis: Yeah. Really, I want to speak to more on the large enterprise side, and/or smaller companies who are maybe being acquired, and having to manage themselves inside of a larger organization all of a sudden. It's becoming fairly normal to see two cloud providers inside of a larger organization. And the normalcy that I've seen is acquisitions: a larger business purchasing a smaller one, maybe the larger ones on AWS, but a smaller one has chosen GCP. And no company is going to tell the new acquisition to refactor themselves and shift over to Amazon. That's a—millions of dollars will be spent [laughs], and lots of time spent to accommodate that request, so no one's really going to answer it. Instead, what they will, though, is central IT will be asked to manage both. So, multi-cloud is becoming a thing. It's becoming more and more important to have the same business strategy applicable to multiple cloud vendors at once. It's important to have the right tooling in place so that as you onboard different cloud vendors than you're normally used to, at least the terminology stays the same, the functionality stays the same because you have the right tools that can help make that translation for you. Without that right process in place, it will mean that your organization will have to incur overhead to help manage always to different costs, different ways to optimize your costs, or otherwise. And so multi-cloud tooling is becoming more and more important for larger orgs, especially as your atypical business processes pan out.Emily: How exactly, actually, does multi-cloud impact to this sort of cost equation?Travis: Yeah, so there's two trains of thought when it comes to how multi-cloud impacts a project and then how the project [eventual] cost savings. So, there's the one that I personally believe does not happen that often, but it's worth mentioning, where you have a project that's deployed—you know, software deployed, application stack deployed—and that's leveraging a small piece of it from a different cloud vendor. So, maybe 90 percent of the stack is deployed on AWS but uses Google Cloud for some analysis engine they may have, or some particular tool. So, now you've got to combine these two things together. That happens, but rarely. The alternative to that is you have discrete projects who have their own cloud vendors that they use 100 percent of but you now need to manage the business unit, owns both those projects. And how do you then combine those two things, but there is no project dependency between the two? In both cases, there needs to be a strategy employed for identifying the individual resources. So, in AWS or otherwise, you have a tagging strategy based off of project code, or business unit, or reporting lines, or however the business needs to manage to it. The second is, you need to come up with and identify consistent ways you want to save money. Every cloud vendor may have a slightly different functionality because they need to differentiate among themselves; that's normal. But everybody should be turning off servers from 7 p.m. at night until 6 a.m. the next day, when the workforce is typically not working—the internal systems, pre-production type systems—if you can. And that's true to form, regardless if you're Amazon, or Azure, or GCP. And so you need tooling and the right process in place to identify those cross-cloud cost-saving strategy options available to you, and need to normalize the way you want to implement on it. If you don't do that, what typically will end up happening is you're going to be working out of multiple consoles with different terminology, and different settings you're not used to, and different ways to implement it, and somewhere something will fall through the cracks, and you will not have consistency in your cost-saving strategy.Emily: All right, one more question. What's your favorite can't live without engineering tool?Travis: Oh, I can't say that. Because I'm going to get yelled at. [laughs]. If I had to take off my CloudCheckr hat and put my personal hat on for a brief moment, I would say log analysis tools like Sumo Logic, like Splunk, is my favorite thing, period. Back when I was more in engineering side of the house, before I was in product, I would spend hours looking through log analytics. There's just so much you can do with those tools nowadays to identify, come up with really cool analysis products. That's actually one of the guiding principles I see for CloudCheckr is people really want to have fun tools they can play with, and see, and get their hands on, and come up with new conclusions to answers they may not have had in the past. That's what I see those log analytics products doing. CloudCheckr is following a very similar route, and that's the ethos that I'm instilling on our product team is our users, our customers, they should have a fun product to be able to get into and play with their data, and use in different ways to come to new conclusions they've never seen before, especially on cost analysis. So, those are my favorite products and how we integrate them into our product decisions.Emily: Where can listeners connect with you?Travis: Yeah, so I'm on LinkedIn. Just Google my name Travis Rehl, CloudCheckr, feel free to contact me; happy to chat. CloudCheckr, if you're a CloudChecker customer, we have an All-Stars program at cloudchecker.com/allstars, which enables you to have a Slack workspace with us, which myself and my team run so you're able to chat, have conversations like this, or otherwise. I also post on Twitter, but be honest with you not that often. So, if you're looking to reach out, I'd say LinkedIn is probably your best bet.Emily: Thank you again. This is Travis Rehl. And thank you for joining us on the Business of cloud-native.Travis: Sure. Thanks for having me.Announcer: Thank you for listening to The Business of cloud-native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

Some of the highlights of the show include:  The difference between cloud computing and cloud native. Why operations teams often struggle to keep up with development teams, and the problems that this creates for businesses. How Dave works with operations teams and trains them how to approach cloud native so they can keep up with developers, instead of being a drag on the organization.  Dave's philosophy on introducing processes, and why he prefers to use as few as possible for as long as possible and implement them only when problems arise.  Why executives should strive to keep developers happy, productive, and empowered.  Why operations teams need to stop thinking about themselves as people who merely complete ticket requests, and start viewing themselves as key enablers who help the organization move faster.  Viewing wait time as waste.  The importance of aligning operations and development teams, and having them work towards the same goal. This also requires using the same reporting structure.  Links: Company site: https://www.mangoteque.com/ LinkedIn: https://www.linkedin.com/in/dmangot/ Twitter: https://twitter.com/DaveMangot CIO Author page: https://www.cio.com/author/Dave-Mangot/ TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast, where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I'm your host, Emily Omier, and today I am chatting with Dave Mangot. And Dave is a consultant who works with companies on improving their web operations. He has experience working with a variety of companies making the transition to cloud-native and in various stages of their cloud computing journey. So, Dave, my first question is, can you go into detail about, sort of, the nitty-gritty of what you do?Dave: Sure. I've spent my whole technical professional career mostly in Silicon Valley, after moving out to California from Maryland. And really, I got early into web operations working in Unix systems administration as a sysadmin, and then we all changed the names of all those things over the years from sysadmin to Technical Infrastructure Engineer, and then Site Reliability Engineer, and all the other fun stuff. But I've been involved in the DevOps movement, kind of, since the beginning, and I've been involved in cloud computing, kind of, since the beginning. And so I'm lucky enough in my day job to be able to work with companies on their, like you said, transitions into Cloud, but really I'm helping companies, at least for their cloud stuff, think about what does cloud computing even mean? What does it mean to operate in a cloud computing manner? It's one thing to say, “We're going to move all of our stuff from the data center into Cloud,” but most people you'll hear talk about lift and shift; does that really the best way? And obviously, it's not. I think most of the studies will prove that and things like the State of DevOps report, and those other things, but really love working with companies on, like, what is so unique about the Cloud, and what advantages does that give, and how do we think about these problems in order to be able to take the best advantage that we can?Emily: Dive into a little bit more. What is the difference between cloud computing and cloud-native? And where does some confusion sometimes seep in there?Dave: I think cloud-native is just really talking about the fact that something was designed specifically for running in a cloud computing environment. To me, I don't really get hung up on those differences because, ultimately, I don't think they matter all that much. You can take memcached, which was designed to run in the data center, and you can buy that as a service on AWS. So, does that mean because it wasn't designed for the Cloud from the beginning, that it's not going to work? No, you're buying that as a service from AWS. I think cloud-native is really referring to these tools that were designed with that as a first-class citizen. And there's times where that really matters. I remember, we did an analysis of the configuration management tools years back, and what would work best on AWS and things like that, and it was pretty obvious that some of those tools were not designed for the Cloud. They were not cloud-native. They really had this distinct feel that their cloud capabilities were bolted on much later, and it was clunky, and it was hard to work with, whereas some of the other tools, really felt like that was a very natural fit, like that was the way that they had been created. But ultimately, I think the differences aren't all that great, it just, really, matters how you're going to take advantage of those tools.Emily: And with the companies that you work with, what is the problem or problems that they are usually facing that lead them to hire you?Dave: Generally the question, or the statement, I guess, that I get from the CIOs and CTOs, and CEOs is, “My production web operations team can't keep up with my development teams.” And there's a lot of reasons why those kinds of things can happen, but with the dawn of all these cloud-native type things, which is pretty cool, like containers, and all this other stuff, and CI/CD is a big popular thing now, and all kinds of other stuff. What happens, tends to be is the developers are really able to take advantage of these things, and consume them, and use them because look at AWS. AWS is API, API, API. Make an API call for this, make an API call for that. And for developers, they're really comfortable in that environment. Making an API call is kind of a no brainer. And then a lot of the operations teams are struggling because that's not normal for them. Maybe they were used to clicking around in a VMware console, and now that's not a thing because everything's API, API, API. And so what happens is the development teams start to rocket ahead of the operations teams, and the operations teams are running around struggling to keep up because they're kind of in a brand new world that the developers are dragging them into, and they have to figure out how they're going to swim in that world. And so I tend to work with operations teams to help them get to a point where they're way more comfortable, and they're thinking about the problems differently, and they're really enabling development to go as quickly as development wants to go. Which, you know, that's going to be pretty fast, especially when you're working with cloud-native stuff. But I mean, kind of to the point earlier, we built—at one of the companies I worked at years ago—what I would say, like, a cloud environment in a data center, where everything was API first, and you didn't have to run around, and click in consoles, and try to find information, and manually specify things, and stuff like that; it just worked. Just like if you make a call for VM in AWS, an EC2 instance. And so, really, it's much more about the way that we look at the problems, then it is about where this thing happens to be located because obviously cloud-native is going to be Azure, it's going to be GCP, it's going to be all those things. There's not one way to do it specifically.Emily: What's the business pain that happens if the operations team can't keep up with the developers? What happens? Why is that bad?Dave: That's a great question. It really comes down to this idea of an impedance mismatch. If the operations teams can't keep up with the development teams, then the operations teams become a drag on the business. There's so much—if you read the state of DevOps reports that are put out by DORA research and—I guess, Google now, now that they bought it—but they show that these organizations that are able to go quickly: the organizations that are able to do deploys-on-demand, the organizations that are able to remediate outages faster, all those things play into your business's success. So, the businesses that can do that have higher market capitalization, they have happier employees, they have all kinds of fantastic business outcomes that come from those abilities, and so you don't want your operations team to be a drag on your organization because that speed of business, that ability to do things a lot more easily, let's even call it like a lot more cloud-native if you want, that has real market effects. That has real business performance impacts. And so, if you look at the DevOps way of looking at this—like I said, I've been pretty involved in the DevOps movement—really the DevOps is about all the different parts of the organization working together in concert to be able to make the organization a success. And the first way of DevOps, you're talking about systems thinking, you're looking at the overall flow of work through the system, and you want to optimize that because the faster we can get work flowing through the system, the faster we can deliver new features to our customers, bug fixes to our customers, all the things that our customers want, all the things that our customers love. And so if you're going to optimize flow of work through the system, you definitely don't want work slowing down inside the operations part of the system. That's bad for the business, and that's bad for your business outcomes.Emily: And how do you think companies realize that this is a problem? I mean, is it obvious or not?Dave: I think it's one of those things like I always talk to people about process right? When do we want to introduce process? a lot of startups are like, “We need more process here, we need more process there.” And my advice to everybody is always, use as little process as possible for as long as possible, and when you need that process, it will make itself known. The pain will be so obvious that you'll be like, “Okay, we can't do this anymore this way. We've run this all the way to the end, and now we have to change things, and now we have to introduce process here.” And I think that it becomes pretty obvious to certainly the companies that I work with. At one point where they're like, “This isn't working, I'm getting my development leadership coming to be saying ‘I'm waiting for this, I'm waiting for that. I'm waiting for this. I'm waiting for that. I don't have permission to do this. We're being blocked here.'” all the things that you don't want to be hearing from your development leaders because what they're expressing is their pain of being inhibited; their pain of being slowed down. And I think it's just, like, with the process thing, I think at some point, the pain becomes obvious enough that people say, “We have to do something.” I remember talking to one company, and I was like, “Well, what do you want out of this engagement? What's your end goal?” And they said, “We'd like it for our developers to show up every day and be really happy with the environment that they're working in.” And so, you can hear it there, right? Their developers are not happy. People are coming from other companies, and they're going to this company—and I certainly won't name who they are—but they're going to this company, and they're saying, “Hey, when I worked at this other place, I didn't have all this. I didn't have all these things stopping me. I didn't have all these things inhibiting me.” And so that's why, what I said in the beginning, it's things that I'm hearing from CEOs, and CTOs, and people in those positions because at some point that stuff is just bubbling up and bubbling up, and the amount of frustration just really makes itself known.Emily: Why do you think COOs care how happy their developers are?Dave: Well, I mean, there's tons of studies that show the happier your developers are, the more productive they are. I mean, look at the Google rework stuff about psychological safety. Google discovered after hiring a professional psychology researcher to determine who were their highest performing teams, their highest performing teams weren't the teams that had the most talented engineers; it wasn't the people who went to MIT; their highest performing teams weren't the ones who had the best boss, or the coolest scrum master, or anything like that. Their highest performing teams were the teams that had the most psychological safety. People who were able to operate in an environment where they felt free to talk about the things that maybe weren't going well, things that could be improved, crazy ideas they had to make improvements, stuff like that. And I don't think that you can be on a team that's unhappy and feel like there's a lot of psychological safety there. And so, I think those things are highly correlated to one another. So, I mean, obviously, the environment that's necessary for psychological safety goes far beyond whether or not my Kubernetes cluster is automatically deploying my Docker containers; that's certainly not the case. But I think it's important to recognize that if developers are in an environment where they feel empowered, and they're not being inhibited, and they can really focus on their work, and improving things, and making things better, that they're going to produce better work, and that's going to be better for companies and certainly their business outcomes.Emily: And bringing it back to cloud-native a little bit. Can you connect for me how a cloud-native type architecture helps bring operation teams up to speed or helps remove these roadblocks?Dave: Yeah. Well, I think it's a little bit of the reverse, right? I think that the successful operations teams are the ones who are enabling these cloud-native ways of looking at the world. I think there used to be this notion of, if you want something from operations, you open up a ticket, and then operations goes and they do the ticket, and then they come back to you and say, “It's done.” And then, this never-ending cycle of sending off something and then waiting, and then sending off something, and waiting. And in cloud-native environments, we don't have that. In cloud-native environments, people are empowered and enabled, to go off and deploy things, and test things, and remediate things, and do dark launching, and have feature flags, and all these other things that, even though we're moving quickly, we can do that safely. And I think that's part of the mind shift that has to happen for these operations teams, is they need to stop thinking about themselves as people who get things done, and they need to start thinking about themselves as people who are enabling the whole organization to go faster, easier, better. I always talked to my SREs—Site Reliability Engineers—who used to work for me, and I'd say, “You have two responsibilities and that's it. And this is, in order, your first responsibility is to keep the site up.” That sounds pretty normal, right? That's what most operations teams feel like they've been tasked with. And I'm like, “Your second responsibility is to keep the developers moving as fast as possible.” And so really, when you start taking that to heart, keeping developers moving as fast as possible, that's not closing tickets as fast as you can. That's not keeping the developer moving as fast as possible, that's enabling developers to have self-service tools, and have things where they want to get something done and it's very painless for them to do that. We used to launch EC2 instances at one of the companies I was working with, where we had gotten it down to a point where you just said what kind of machine you wanted, and then that was it; and you were done. And everything else got taken care of for you: all the DNS, all the security groups, routing, networking, DNS, like, everything was all taken care of, all the software was loaded. There wasn't anything to do but say what you wanted. And we actually were able to turn that tool over to the developers so they could launch all their own stuff. They didn't need us anymore. And I think that's really creating these cloud-native ideas. Certainly, a lot of that stuff is part of the cloud-native tooling, now. This was a few years ago, but really it's enabling the developers to go as fast as possible. We could have said, “Hey, you want a machine? Open up a ticket, and it's so easy for us to spin up a machine.” But we didn't do that. We took it to the next level, and we empowered them, and we allowed them to go quickly. And that's really the sort of mental shift that the operations teams have to make. How do we do that?Emily: I have to say, I have never been a developer, but whenever anyone talks about this process of submitting a ticket and waiting for it to get addressed, it just sounds like hell.Dave: Yeah. Well, if you look at it from a lean manufacturing, Toyota kind of thing, all that wait time is waste. In lean, they call that waste. It's a handoff: there's no work being accomplished during that time, and so it's waste in the system. And so Toyota is always trying to move towards—I can't remember they call. It, I think it was, like, one piece flow, or something like that where basically you want work to be happening at all times in the system, and you certainly don't want things sitting around. And so, developers don't want that either. Developers want to put things out there. They want to see, does this work? Does this not work? And when you enable developers to have that kind of power and have that ability to go really fast, there's all kinds of like things that we can enable for the business that help cost savings, better security, all kinds of stuff far beyond just simple, “Hey, here's more features. Here's more features.”Emily: How easy do you think it is for operations teams to sort of shift to think, like, “Our job is to make things as easy as possible for developers?”Dave: I don't think it's that hard, actually, mostly because if we're starting to look at things from a DevOps mindset, we're understanding that the whole goal is to optimize the entire system; it's not to optimize a single point in the system. And I always advocate that operations teams report up through the same reporting structure as the engineering teams do. The worst thing you can do is silo it off so all the operations teams report to the COO, and all the engineering teams report to the CTO. Like, that's awful because what you want to do is you want to align the outcome so that everybody's working towards the same goal, and now we can start to partner up together in order to be able to achieve those goals. And so, one of my favorite examples of this from, like, enabling the developers to go fast, and doing that in partnership with operations was, I worked at a company, and we had a storage system, and we were storing all this stuff in a database, and we were paying a lot of money to store all this data for our customers. And that's what the customers were paying us for, was to store their data. And so the developers had this idea that they wanted to try this other way of storing the data. And so, we worked with them—the operations teams work with them, “How do you want to do this? What kinds of things do you need? What's going to work best? Is this going to work best? Is that going to work best?” And we had a lot of collaboration, and, “Here's where we're going to launch these new things, and we're going to try them out. And this is how we're going to try them out.”And it wasn't a process that happened overnight: from beginning to end of this project, it probably took, I don't know, a year and a half or something like that, of iterating, and trying, and testing, and making sure it's safe, and all these other stuff. But in the end, we wound up shutting off the old database system and talking to the engineers about what that meant for the business. They said a conservative estimate would be that we saved the company 75 percent on storage costs. That's the conservative estimate. I mean, that's insane, right? 75 percent for their biggest cost. That was the biggest cost of the company, and we knocked it down by 75 percent, at minimum.And so this idea of enabling this cloud approach of going quickly, and taking advantage of all these resources, and moving fast without impediments, that can have some major impact. And it's not operations teams doing that; it's not development teams doing that; it's operations and development teams doing that together in partnership to achieve those pretty awesome business outcomes.Emily: In that particular case, who had the initial push? Who had this initial idea that let's figure out a better way to approach storage?Dave: Well, I mean, we got a challenge from the business. The business said, look at our costs. Look at what we're doing. Are there ways that we can improve this so that we can improve our profitability? And so it was a challenge. And I think the best thing about that is, it wasn't the business telling us how to do it; it wasn't people saying, here's what you should do. The business is saying, “If this is a problem, how do you solve it?” And then they, kind of, got out of our way and said, “Let the engineers do their engineering.” And I think that was kind of fantastic because the results were exactly what they wanted. But the business is going to look at the problems from a business perspective, and I think it's important that as engineers, we look at the problems from a business perspective as well. We're not showing up for work to have fun and play with computers. We're showing up at work to achieve an objective. That's why we get paid. If you want to hobby around with your computers, you can hobby around at home, but we're getting paid at work to achieve the goals of the business. And so, that was the way that they were looking at the problem, and that's the way that we wound up looking at the problem. Which is the correct way?Emily: Do you have any other notable examples that come to mind?Dave: Yeah, I mean, this idea of cloud and being able to go quickly, we had this one problem with that—actually, with that same database engine, which is hilarious, before we wound up replacing it, where we were upgrading the software from one version to another, and we're making a pretty big jump. And so, we spun up the new version of the software; we loaded the data on, and we started seeing how their performance was. And the performance was terrible. I mean, not just, we would have trouble with it; it was unusable. There was no way we could run the business with that level of performance. And we're like, “What happened? [laughs]. What did we do here?” And so, we went and looked in GitHub at the differences between the old version of the software, and the new version of software. And there was, like, 5000 commits that had happened between the old version and the new version. And so all we had to do was find out which of those 5000 commits was the problem. [laughs]. Which, that's a daunting task or whatever. But the operations team got down to it, and we built a bunch of tooling, and we started changing some things and making improvements so that we were able to spin up clusters of this software and run a full test suite to determine whether this problem still existed. And that was something we could do in 20 minutes. And so then we started doing what's called git bisecting, but we started searching in a certain kind of pattern, which I won't get into, for which of these—where was the problem? So, we would look, say in the middle, and then if the problem wasn't there in the middle, then we would look between the middle and the right. If it was there, then we would look between the middle and the left. And we kept doing this bisect, and within two days, we had found the exact commit that had caused the problem. And it was them subtracting, like, a nano from a milli, or something like that. But going back and talking to the CTO afterwards, I said, “You know, if we hadn't built these tools, and we hadn't had this ability to really iterate super quickly in the Cloud, what would you have done?” And he was like, “I have no idea.” He's like, “Maybe we would have spent a couple of days and then given up, maybe we would have just gone in a completely different direction.” But that ability to be able to work so effectively with these cloud tools, and so easily with these cloud tools, enabled us to do something that the business just would just not have had the opportunity to take advantage of at all. And so that was a major win for being able to have operations teams that think about these problems in a completely different way.Emily: It sounds like, in this particular company, the engineering teams and the business leaders were fairly well-aligned, and able to communicate pretty well about what the end goals are. How common do you find that is with your clients?Dave: I don't know. I think it's pretty variable. It depends on the organization. I think that is one of the things that I emphasize when I'm working with my clients is how important that alignment is. I sort of talked about it a little bit earlier, when I said you shouldn't have one group reporting to the CTO and another group reporting to the COO. But also, it's really important for leadership to be communicating this stuff in the proper way. One of the things I loved most about my experience working at Salesforce was, Marc Benioff was the CEO and he would publish what they call V2MOMs, which is like—oh boy, vision, values, metrics, obstacles, and measures, or something. I don't remember what the last thing was. But he would publish his V2MOM, which was basically his objectives for the next time period, whether it was quarterly, or yearly, I don't really remember. But then what would happen was the people that worked for him would look at his V2MOM, and they would write theirs about what they wanted to get accomplished, but showing how what they were doing was in support of what he wanted. And then the people below them would do the same thing. And the people below them would do the same thing. And what you were able to create was this incredible amount of alignment at a 16,000 person company, which is crazy, up and down the ladder so that everybody understands what they're doing, and how it fits into the larger picture, and what they're doing in support of the goals of the business, and the objectives of the business, and that goes all the way down to the most junior engineer. And I think having that kind of alignment is, I mean, it's obviously incredibly powerful. I mean, Salesforce is a rocket ship and has been for a long, long time. And Google does this for their OKRs, and that there was that thing that was popularized by Intel as well; there's a whole bunch of these things. But that alignment is phenomenal if you want to have a lasting, high performing organization.Emily: When you see companies that don't have that alignment, or even just, it seems like the engineering team maybe doesn't entirely understand where the business is going, or even the business doesn't understand what the engineering team is doing, what happens and where is the communication going wrong?Dave: I mean, you see the frustration. You see the fracturing. You see the silos. You see a lot of finger-pointing. I've definitely worked with some clients where the ops team hates the dev team; the dev team hates the ops team. I remember the dev team saying, “Ops doesn't actually want to do any work. They just want to invent stuff for themselves to work on. And that's how they want to spend their day.” And the ops team is saying, “The developers don't even understand anything about what we're doing, and they just want to go o—” you know, there's all these crazy, awful made up stories. And if you've ever read the book Crucial Conversations—they also have a course, or whatever—one of the things they talk about is you need to establish mutual purpose in order to have a difficult conversation. And I think that's really important for what we're talking about in the business: we need to establish mutual purpose, just we talked about with DevOps, there's only one goal. And the other thing that they say in that class, or in that book, is that when we are going to have a conversation with somebody that we are not getting along with, we invent a story that explains why they behave the way that they do, and every time we see something that validates that story, then it's even more evidence that that story is actually correct. The problem is, is it's a story. Like it's not real. It may seem real to us; it may feel real to us. But it's a story. It's something that we made up. And so, that's the kind of outcomes that you get when you have this fracturing, where you don't have this alignment up and down. You have people telling these stories like, “Operations doesn't want to do any real work. They just want to make stuff up for themselves to work on.” Which is, if you're not in that environment, if you're someone like you and me looking from the outside, that's absurd. But I can certainly see how you can make up a story that gets continually validated by what you see because you're looking for evidence that supports your story. That's part of what makes you think that you're right, is that you're always searching for this evidence. And so obviously, those are not going to be high performing organizations. That's why it's so important to get that kind of alignment.Emily: Going back to this idea of sort of moving to cloud-native, what do you think are some of the surprises or misconceptions that come up when teams are moving to more cloud-native approaches?Dave: I feel like my clients generally are not terribly surprised. I think by the time that someone's reaching out to me, they are feeling a lot of pain, and they know that things have to change, and they are looking for what are the ways that things have to change? I don't ever have to go into a client and convince them that they need to do it better. The clients that are coming to me, recognizing that they're having a problem, and so it's really just getting them to stop focusing on what we call an SRE toil, which is popularized by Google, which is—I don't remember the exact definition, but it's basically manual work that's devoid of enduring value, that's repetitive, it's automatable, it's just repeat, repeat, repeat, repeat: we're not making improvements anymore. And so once we start to have this Kaizen mindset, this idea of continual improvement at all times, instead of just trying to keep the business running, that starts to enable all this kinds of stuff. And that's why we talked about building things in sort of a cloud-native manner. We're talking about that ability to go fast. We're talking about that ability to enable things, and part of that is this idea of continual improvement; this idea of always making things better. A lot of this comes out of Agile as well. I always talk to people about their sprint retrospectives, and I say, “It's your opportunity to make your team better. It's your opportunity to make your environment better. It's your opportunity to make your company better.” And I was like, “The worst thing that you could do in Agile is if in January of last year, and in January of this year, your team is just as good as it was.” That's terrible. Your team needs to be much better than it was. And so enabling developers to go quickly and all that other stuff. And putting all those things in place is a big part of that.Emily: Anything else that you want to add about this topic that I didn't think to ask?Dave: I mean, I think embracing these principles is really important. I think that if you look at the companies who are trying to go fast, and don't embrace these principles, these cloud-native ideas, or just even these cloud computing ideas, it basically becomes technical debt that keeps building up, and building up, and building up. And everybody knows accumulating tons of technical debt is not going to help your organization to move faster; it's not going to help you achieve all those great business outcomes that you want to get out of the State of DevOps report. And so I've seen situations where they have not been able to make that transition into this way of looking at the world, and the environment becomes really fragile; it becomes really brittle; it becomes really hard to make changes, and the only way is to make changes is to double down on the technical debt, and accumulate more of it, to the point where eventually they wind up spinning up an entire team whose sole purpose is to try to undo the mess that's been created. And you don't want that. You don't want to allocate a team to start unpacking your technical debt. You'd rather just not accumulate that technical debt as you're going along. And so I think it's really crucial for businesses that want to be successful in the long term that they start to embrace these ideas early. And obviously, if I'm a startup and I want to embrace a lot of these cloud-native things, that's a lot easier than if I'm a well-established company. I, in my consulting practice, I don't really work with startups because they don't tend to have these problems. They don't tend to accumulate a lot of technical debt because they are founded with this idea of going quickly and being able to empower developers and enable people to go quick. To your point earlier, the companies that I'm working with are the ones who are making this transition, where they've been running in the data center, or maybe they built an environment in the Cloud, but it's just not operating the way that they expected, and they're paying ridiculous amounts of money [laughs] to run stuff in AWS, where we thought, “Hey, what's going on? This isn't supposed to be this way.” But startups have the ability to do this much easier because they're unencumbered. And then as they grow, and they start to introduce more process because that stuff is inevitable that we're going to need to do that, that's when these things become even more important that we make sure that we're keeping them in mind and we're doubling down on them, and we're not introducing lean waste into the system and stuff like that, that will ultimately catch up with us.Emily: It's so true. All right, just a couple more questions. What is your favorite engineering tool?Dave: Ah. I mean, I'm supposed to give some kind of DevOps-y, it's not about the tools answer, but this week I think it was, on Twitter, I saw somebody else put up a SmokePing graph. And most people are not going to have heard of SmokePing. I worked at multiple ISPs in my career already, so the networking stuff is important to me. But wow, I love a SmokePing graph. And it's basically just a bunch of pings that are sent to some target, and then they're graphed when they come back, but instead of saying, “I sent one ping, and I came back with 20 milliseconds,” it sends 20, and then it graphs them all at that time point, so you can actually see density. It's basically before everybody came up with the idea of heat maps, this was one of the original heat map tools, and I still run SmokePing in my house just to see the performance of my home network going out to different parts of the internet, and that's definitely my favorite tool.Emily: Where can listeners connect with you? Website?Dave: Yeah, yeah, that's a great question. So, if people are interested in my business, I'm at mangoteque.com, M-A-N-G-O-T-E-Q-U-E. That was a fun name invented by Corey Quinn of The Duckbill Group and I loved it, and so I wound up using it. And they could also find me on LinkedIn obviously, or Twitter at @DaveMangot. M-A-N-G-O-T, and I post a lot on there about things that I've observed. I post a lot on there about DevOps. I post a lot on there about taking a scientific approach to a lot of the things we're doing, not just in terms of the scientific method, but like in terms of cognitive neuroscience, and things like that. And I also write a monthly column for CIO.com.Emily: Well, Dave, thank you so much for joining me.Dave: Thank you for having me, Emily, this was really fun.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

This conversation covers: How Frame.io was faced with the decision to be cloud native or cloud-enabled — and the business and technical reasons why Frame.io chose to be cloud native.  How Abhinav successfully built a world class cloud-native security program from the ground up to protect Frame.io users' sensitive video content. Abhinav also talks about the special security considerations for truly cloud native applications.  Cloud native as a “journey without a destination.” In other words, there is no end point with cloud native transitions, because new technologies are always being developed. Why Abhinav is a firm believer in both ISEs and GitOps, and why he thinks the industry should embrace both of these strategies. The challenge of not only maintaining security in this type of environment, but also communicating security issues to various stakeholders with different priorities. Abinhav also talks about the role that specialists like AWS and machine learning experts can play in furthering security agendas. Common misconceptions about cloud native security. Frame.io's decision to roll out Kubernetes, and why they are also considering adding chaos engineering to fortify against unexpected issues. Tool and vendor overload, and the importance of trying to find the right tools that fit your infrastructure.  Links: Frame.io: https://frame.io/ Connect with Abhinav on LinkedIn: https://www.linkedin.com/in/absri/ The Business of Cloud Native: http://thebusinessofcloudnative.com  TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I'm Emily Omier, your host, and today I am chatting with Abhinav Srivastava. Abhinav, can you go ahead and introduce yourself and tell us about where you work, and what you do.Abhinav: Thanks for having me, Emily. Hello, everyone. My name is Avinash Srivastava. I'm a VP and the head of information security and infrastructure at Frame.io. At Frame, I am building the security and infrastructure programs from ground up, making sure that we are secured and compliant, and our services are available and reliable. Before joining Frame.io, I spent a number of years in AT&T Research. There I worked on various cloud and security technologies, wrote numerous research papers, and filed patents. And before joining AT&T, I spent five great years in Georgia Tech on a Ph.D. in computer science. My dissertation was on cloud and virtualization security.Emily: And what do you do? What does an average day look like?Abhinav: Right. So, just to tell you where I answer the question where I work: so I work at Frame.io, and Frame.io is a cloud-based video review and collaboration startup that allows users to securely upload their video contents to our platform, and then invite teams and clients to collaborate on those uploaded assets. We are essentially building the video cloud, so you can think of us as a GitHub for videos. What I do when I get to office—apart from getting my morning coffee—as soon as I arrive at my desk, I check my calendar to see how's my day looking; I check my emails and slack messages. We use slack primarily within the company doing for communication. And then I do my daily standup with my teams. We follow a two-week sprint across all departments that I oversee. So, a standup gives me a good picture on the current priorities and any blockers.Emily: Tell me a little bit about the cloud-native journey at Frame.io? How did the company get started with containers, and what are you using to orchestrate now? How have you moved along in the cloud-native journey?Abhinav: We are born in the cloud, kind of, company. So, we are hosted in Amazon AWS since day one. So, we are in the cloud from the get-go. And once you are in the cloud, it is hard not to use tools and technologies that are offered, because our goal has always been to build secure, reliable, and available infrastructure. So, we were very, very mindful from the get-go that while we are in the cloud, we can choose to be cloud-native or just cloud-enabled. Means use tools, just virtual machines, or heavyweight virtual machines, and not to use container and just host our entire workload within that. But we chose to be cloud-native because, again, they wanted to boot up or spin up new containers very fast. As a platform we, as I mentioned, we allow users to upload videos, and once the videos are uploaded, we have to transcode those videos to generate different low-resolution videos. And that use case fits with the lightweight container model. So, from the get-go, we started using containerized microservices; orchestration layer; From AWS, their auto-scaling; automation infrastructure as a code; monitoring. so all those things were, kind of, no brainer for us to use because given our use case and given the way we wanted to be a very fast uploader and transcoder for all of our customers.Emily: This actually leads me to another question: have you guys seen a lot of scaling recently as a result of stay-at-home orders and work from home?Abhinav: Right. So, we are seeing a lot more people moving towards remote collaboration tools who are actually working in the production house since they have to work from home now. So, they are now moving to these kind of tools such as Frame.io. And we do see a lot more customers joining our platform because of that. From the traffic perspective, we did not see much increase in the web traffic or load our infrastructure, because we have always set up the auto-scaling and our infrastructure can always meet these peak demands. So, we didn't see any adverse effect on our infrastructure from these remote situations.Emily: What were some of the other advantages? Like you were talking about that you had the choice to be either cloud-enabled or truly cloud-native? What were the biggest, you know—and I'm interested, obviously in business rationale to the extent you can talk about it—for being truly cloud-native?Abhinav: So, from business perspective, again, a goal was to [basic] secure available and reliable production infrastructure to offer Frame.io services. But cloud-native actually helped us to faster time to market because our developers are just focusing on the business logic, deploying code. They were not worried about the infrastructure aspect, which is good. Then we're rolling out bug fixes very quickly through CI/CD platform, so that, again, we offer the better [good] services to our customer. Cloud-native helped us to meet our SLA and uptime so that our customer can access their content whenever they would like to. It also helped us securing our infrastructure and services, and our cost also went down because we were scaling up and down based on the peak demand, and we don't have to provide dedicated resources, so that's good there. And it also allowed us to faster onboard developers to our platform because we are using a lot of open source technologies, and so the developers can learn quickly—there are a lot more resources out there for them to learn. And it also helped us avoid vendor lock-in. We are relying on more and more open-source projects, CNCF [unintelligible] projects, so that has helped us. And more importantly, it is helping us stay competitive because in this industry—in this time—we would like to be available, we would like to be secure. So, for our customers to stay doing their job that they used to do in an office setting or in a non-remote setting, and we can continue providing help that they need.Emily: How has this changed the security story?Abhinav: So, obviously, security story is same what we have before because, I mean, we allow people to have upload their media content to our platform. So, that's very sensitive content. So, we always wanted to make sure that they stay secure. And for that, we have built a world-class security program from ground up, with emphasis on product security, cloud security, security data science, and also compliance and privacy program. So, we are doing what we used to do: making sure that content is still secure, our infrastructure follows the AWS security best practices, we can identify vulnerability within our application and fix it. So, again, as I said, that it hasn't changed much from security perspective, as far as Frame.io's daily operations are concerned.Emily: How does having a truly cloud-native application, how is that different from a security perspective from something that isn't cloud-native?Abhinav: So, security is very important whether you are cloud-enabled or cloud-native. So, security is very important for all the services. Being in the world of microservices and in the container, actually, it helped us to model the application behavior. For example, if you have one very big monolithic application, it does so many things, so it's really hard for you to know to find out what's the normal execution pattern. And when this application is going to—if it attacked, how it's going to behave, how is abnormal execution look like? But in the microservices world, since each application, each microservices is getting one job. So, you can create a good model of behavior of that container. Or even if you are monitoring their runtime behavior, you know that what kind of processes are going to be invoked from that container? What kind of network connections are going to be made? What are the files are going to be accessed by the services within the host, or within S3, or other resources? So, you know their interaction pattern—execution pattern, and that, you can qualify, both in terms of your security rules that you want to create on the infrastructure for those services, or you can create a better anomaly detection or machine learning models for those behavior. And we did both in our infrastructure to keep them secure.Emily: And how do conversations about security go when you talk with different stakeholders. I'm curious to know if there's any sort of miscommunications, or things that are lost in translation when you're talking about security with, say, the development team; with the business stakeholders; with platform engineers. What are some of the things—anything that gets lost in translation?Abhinav: So, there are two parts of this question. In general, having a discussion around cloud-native services and the security of cloud-native services. Because there are various ways you can deploy a service in the cloud, you can have a service deployed in the cloud just by running a bunch of VMs, or you can deploy it using cloud-native architecture where you have doing all those things. But the cloud-native architecture requires you to think of all the stages of the services. For example, how will SLAs, SLOs, SLIs look like for this service? Or, how do you monitor the service when it execute? How will you protect these services when you deploy them? What kind of resources are going to be accessed by this service? How will create their identity and management rules there? How would you deploy it and how would you create network rules for that so that you can do it in a principle of least privileged fashion, you can execute these services?So, you need to do proper planning that how would a new service going to interact with other services in the infrastructure. And these non-functional requirements are, many times, described poorly or not written at all because as a developer, you would like to create service and deploy service, and so that customer can use it. And these are the things behind the scenes we have to think about it. And we, as a team are working very actively to bridge this knowledge and semantic gap so that these things don't get lost in the translation when you're thinking about the service.Emily: What about when you talk to say, business stakeholders? Is there anything that gets lost in the translation?Abhinav: So, I mean, in the business sense, we always have to keep the discussion at a very high level. That, what's a use of service? Or, where we should deploy? Who are going to be the users? So, at that time, we don't want to talk about those underlying infrastructure-related issues because at the business level, we would like to know that how the service is going to function, and mostly functional requirements. But at the low level, we would like to think about that when we are about design these services, what are the things we have to worry about in order for that service to deploy securely and reliably?Emily: How important is security to Frame.io? Not every company thinks the same about security, I should say.Abhinav: And that's a great question. I think for us, security is very important. I know every company says that, but I think we truly mean that. So, we are close to 150 employees, but I was hired around when I was a [00:12:31 unintelligible] employee as a head of security. So, that shows that we care about security. And I have been building security from ground up. We got our SOC 2 Type II compliance when we was around 70 employees. And there are companies out there who are doing SOC 2, and they are thousand employees. So, we are GDPR compliant; we are working towards our CCPA compliance, and we are TPN compliant as well. TPN stand for Trusted Partner Network, which is the [same world] media, and entertainment companies, and industry users. And we were the first few companies who got that certification, also. So, we care about security very much because we allow users to upload their contents in our cloud and we make sure that those contents remain secure.Emily: And so, is there any tension that you feel between talking about security or making things as secure as possible, and either business stakeholders or other parts of the IT team?Abhinav: So, there is definitely attention. [laughs]. If I say no, then I would be lying because our goal—engineers or developers or service creators, they want to deploy the service. They will get satisfaction once the customer start using those services. And our job is to make sure to—we put some guardrails in place—or barriers in place so that we can vet the application, we can vet the service, we can do the proper testing, we can make sure that by deploying the service, we don't increase our exploitable surface. So, that kind of tension will always be there because, by nature, security's job is to make sure that whatever is deployed is secure. Our infrastructure is secure and the service owner's job is to deploy the service. But I think what we are trying to do in the organization, we are trying to take a risk-based approach because security is just another business function. The way sales is important, the way engineering is important, the same way that security is important. And there's a risk in this environment of not meeting sales targets, same way there's a risk of getting breached. So, how do we provide a risk-based methodology so that when we talk about security, we talk in terms of risk; we talk in terms of probabilities versus possibilities? Because there is always possibility of something going wrong, but what's the probability of something happening? And that basically gives us some way of talking to other business-holders saying that, “Okay, if you deploy the service the risk is high. But the risk is high because the likelihood of getting breached is high, but impact would be very low. So, since risk is the product of impact and likelihood, overall the risk is low.” But sometimes the risk is that chance of getting attacked is very low, but the impact could be very high. Again, you will have risk low because probability of actually happening that event is low. So, that basically gives us some common language we can use to talk to other business-holders because risk is being used as a language across other departments. We try to use the same language to convey cybersecurity risk as well.Emily: Since starting with Frame.io and building this security program from the ground up, what surprises have you encountered?Abhinav: I would say there were many surprises. First of all, I had those surprises because I come from a background from research and development. There, goal was to develop services, goal was to think about new security product, and goal was to think of attack and coming up with defenses for them. Having the responsibility of building the security program from ground up, or having to adjust this risk-based mentality was a big surprise because it's not that just because there is a bug, engineering is going to fix it. You have to show the impact of that bug. You should have a proper [unintelligible] associated with that. You have to show that what are the ways that bug can be launched. So, it means, just because you care about security, doesn't mean that everybody else cares about security. So, you have to keep the communication on. You have to always talking, you have to always adjusting, and you have to use the right language to the right person that you are talking to.Emily: What tips do you have about adjusting your language for different audiences and getting them to understand what you're talking about?Abhinav: So, one thing is to use risk-based methodology. That is saying that, “Oh, we have a bug, or we have a high priority bug.” I think saying that, “What is the impact of that bug? How would that bug be exploited in a real setting?” I think those things are important because people care about security, but then they have hundred other things to do, as well. So, how do you talk to their language? And also building the right team, as well. So, if you want to target product security, you have to have a product security specialist, who can understand these nuances; who can understand what are the different attacks. Some companies build a security team with many generalists. I took an approach where I'm building team with the specialists. So, for product security, I have two core product security engineers who have done this thing many times before. For cloud security, I have a specialist who knows about AWS Cloud and everything. For security data science, I have a machine learning expert. So, for each of those roles that you have mined, you try to fill the position with the right set of people. And coming back to this cloud-native security. I think one thing is very important in the cloud-native world, as I have realized lately, that infrastructure as a goal is very important piece for securing your cloud. It's not that I or the team don't know about it, but the temptation to do things quickly sometimes resorting to manual work instead of writing your Terraform or CloudFormation. So, you can do things quickly, but then the chances of you making error are also high. Because if you go to Terraform, you can follow the regular CI/CD process, you can have your pull request approved by somebody, and chances of finding a error quickly is high. And for security purposes, infrastructure code is a blessing. Because you can put proper guard rails in place to make sure that nobody does manual operation in the infrastructure, and everything goes through proper approval process, and that will—as a head of security if you know that if somebody wants to do anything or open any port in the infrastructure, two people are going to look at it and then they're going to have a dialogue with each other, and they're going to find out the real need for opening that port. Your life will be a lot simpler. Emily: What do you think are some misconceptions about cloud-native security, both inside the engineering department—so developers, for example—and then outside in the rest of the company?Abhinav: I think misconception that I view—and it's my opinion—is that the only thing that is important is deploying fast, or moving to production very fast. I think there are so many things has to be done behind the scene in order for you to move fast. And if you don't do those things, then it means that either you're going to break your application, or you're going to make your infrastructure insecure. So, for example, if you have a CI/CD set up and you want to deploy a business logic, and you think that, “Oh, I can code that thing in AWS Lambda functions.” AWS Lambda function is completely managed service. You went ahead and coded in Python, and your service is up and running. But now in doing so, what you did quickly that you forgot to follow the best practices that Lambda function has to be within the VPC; you need to generate an IAM role that has restricted permission; you have to make sure that proper security groups has to be attached to Lambda functions so that it is not open to www. And those things are part of misconception that, “Oh, if I have to do something, AWS allows that we can do it quickly.” That's what we are trying to do. We are trying to come up with a set of best practices for each of those resources as a team, writing documents, sharing with engineering that, “Okay, you want to do it? Sure, go ahead, do it, but just follow these best practices.” So, that even if you SAM or Terraform, whatever you want to use to deploy your application, make sure that best practices are always followed.Emily: Can you think of any misconceptions about cloud-native security that, say, somebody might have if they're coming from a legacy environment: managing security but in a very different type of environment.Abhinav: So, I mean, cloud-native security is all about making sure that your microservices are secure, the kind of access pattern they have, kind of network pattern they have. So, I think one misconception is that—you can think of misconception is, if you are coming from a monolithic world, where you have logged on your services, but just by assuming that you have a parameter between outside world and inside world, so your firewall rules are just like that between in and out. But that parameter is blurred now. There is no such thing as a “them versus us.” It's all blurred now. So, in the microservices world, instead of North/South traffic going up and down. You have to think about East/West traffic as well. So, making sure that your service communication are secure as well: you make sure you use proper cryptography, make sure your endpoints are authenticated so that your services are not compromised. Because if one service compromised, if you don't use proper control among those services, then your other services can be compromised very quickly. And that's the problem when we go from monolithic application to microservices.Emily: Do you think that people outside of the security team understand that distinction?Abhinav: I would say they do, to the extent that they know about it, but then when we have to actually implement it, there are always some concerns that it is going to slow down our application, it is going to introduce latency in the application. So, people do understand that okay parameter is going away, but to the extent that they know about it, but when you—again, when we start implementing it, there is always concern that how it's going to play out.Emily: Do you think Frame.io is fully cloud-native? Do you think there's anything that you could do to be more quote-unquote, “cloud native.”Abhinav: So, in my opinion, it is a journey without any destination. Just like security, you can never say, “I'm secure.” You will have to adjust your control based on the threats or attacks going on. In the same way, there is no end to transition to cloud-native because new technologies are coming, and we will have to evaluate new tools that can help us realize our business goals effectively. So, we are cloud-native, but still, we can do a lot more things, given time and resources. So, in some concrete world that we are doing right now, that we are creating more tools for developers to perform tasks themselves. So, creating more self-serve culture. As I said that moving towards more [IFC] model, and so on. And for that, we are setting up guardrails so that they can perform those operations within those boundaries without impacting security and reliability. We are also looking into ways to extend Kubernetes. Because Kubernetes is in itself a full cloud platform with a lot of possibilities. So, we are interested in making it more programmable for our environment. But these are ongoing things that we'll have to continue doing it.Emily: Do you have any other next steps that you could share? What's next in your journey?Abhinav: So, we rolled out Kubernetes in our infrastructure last December, and that move paid us off. So, we are building more tools on Kubernetes. As I said, that we are going towards more self-service style of architecture where developers can do a lot more things within those guardrails and we are also looking into ways to introduce chaos engineering in our environment because we do things fast, but we break things fast as well. [laughs]. So, one small configuration error can create severity zero alert. So, what we need is a good chaos engineering practices to simulate these areas, so that everybody can train on these events and know how to prevent and respond to such problems. That will reduce our incident resolution time as well.Emily: When—sort of last question: anything else that you would like to add?Abhinav: Two things, I think. One thing is we all should be going towards IFC and GitOps; infrastructure code and GitOps. If this is the one takeaway from this podcast, is that that's the way to go. I know manually doing work is tempting, but that creates problem down the road. So, life will be a lot simpler if we go with the IFC and GitOps. Second thing is that I feel this pain, and many other people are facing the same way, that there are too many tools and vendors out there. So, it's really hard to choose from what is going to work in your environment. CNCF is helping us by highlighting some of these projects by assigning proper maturity levels, like sandbox incubation, and graduated project, so on, but it still is very challenging to find the right tooling that fits your infrastructure. So, always make sure that when you choose a new technology, see how it's going to be working with your existing technologies because it's not that easy to throw away an existing thing because all these things that the tool that you try, it also complicates your security as well because you just do not know how it's going to play out when you deploy this new technology in your environment where the other tools and services are running. So, I think we have to evaluate all tools carefully to make sure that we understand its a security and reliability impact on our existing infrastructure.Emily: What is your can't live without engineering tool or security tool?Abhinav: Huh, that's a good question. Right now, one tool that I cannot live without is Falco. That is a runtime container monitoring solution. We invested a lot on it, and it is paying off in terms of the kind of alert it is generating, kind of visibility it is providing in our infrastructure. And one tool I can't leave off from both from security infrastructure perspective is Slack because we have done a lot of automation to bring all these alerts through Slack. So, all of our ops happen via Slack. So, I think these are the two tools I'm relying a lot in terms of visibility and in terms of response.Emily: Well, thank you so much for joining me.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

In this episode of the Business Cloud Native, host Emily Omier talks with Jon Tirsen, who is engineering lead for storage at Cash App. This conversation focuses on Cash App's cloud native journey, and how they are working to build an application that is more scalable, flexible, and easier to manage.The conversation covers: How the need for hybrid cloud services and uniform program models led Cash App to Kubernetes.  Some of the major scaling issues that Cash App was facing. For example, the company needed to increase user capacity, and add new product lines.  The process of trying to scale Cash App's MySQL database, and the decision to split up their dataset into smaller parts that could run on different databases. Cash App's monolithic application, which contains hundreds of thousands of lines of code — and why it's becoming increasingly difficult to manage and grow.  How Jon's team is trying to balance product/ business and technical needs, and deliver value while rearchitecting their system to scale their operations. Why Cash App is working to build small, product-oriented teams, and a system where products can be executed and deployed at their own pace through the cloud. Jon also discusses some of the challenges that are preventing this from happening. How Cash App was able to help during the pandemic, by facilitating easy stimulus transfers through their service — and why it wouldn't have been possible without a cloud native architecture.  Links: Cash App: https://cash.app/ Square: https://squareup.com/us/en Jon on Twitter: https://twitter.com/tirsen?lang=en Connect with Jon on LinkedIn: https://www.linkedin.com/in/tirsen/?originalSubdomain=au The Business of Cloud Native: http://thebusinessofcloudnative.com  TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. My name is Emily Omier, I'm here chatting with Jon Tirsen.Jon: Happy to be here. My name is, as you said, Jon Tirsen, and I work as the engineering lead of storage here at Cash App. I've been at Cash for maybe four or five years now. So, I've been with it from the very early days. And before Cash, I was doing a startup, that failed, for five years. So, it's a travel guide in the mobile phone startup. And before that, I was at Google working on another failed product called the Google Wave, which you might remember, and before that, it was a company called ThoughtWorks, which some of you probably know about as well.Emily: And in case people don't know, the Cash App is part of Square, right?Jon: Yes. Cash App is where we're separating all the different products quite a lot these days. So, it used to be called just Square Cash, but now it has its own branding and its own identity, and its own leadership, and everything. So, we're trying to call it an ecosystem of startups. So, each product line can run its business the way it wants to, to a large degree.Emily: And so, what do you actually spend your day doing?Jon: Most of my days, I'm still code, and doing various operational tasks, and setting up systems, and testing, and that sort of thing. I also, maybe about half my day, I spend on more management tasks, which is reviewing documents, writing documents, and talking to people trying to figure out our strategy and so on. So, maybe about half my time, I do real technical things, and then the other half I do more management stuff.Emily: Where would you say the cloud-native journey started for you?Jon: Well, so a lot of Square used to run on-premises. So, we had our own data centers and things. But especially for Cash App, since we've grown so quickly, it started getting slightly out of control. We were basically outgrowing—we could not physically put more machines into our data centers. So, we've started moving a lot of our services over to Amazon in this case, and we want to have a shared way of building services that would work both in the Cloud and also in our data centers. So, something like Kubernetes and all the tools around that would give us a more uniform programming model that we could use to deploy apps in both of these environments. We started that, two, three years ago. We started looking at moving our workload out of our data centers.Emily: What were the issues that you were encountering? Give me a little bit more details about the scaling issues that we were talking about.Jon: There two dimensions that we needed to scale out the Cash App, sort of, system slash [unintelligible] architecture. So, one thing was that we just grew so quickly that we needed to be able to increase capacity. So, that was across the board. So, from databases to application servers, and bandwidth, everywhere. We need to just be able to increase our capacity of handling more users, but also we were trying to grow our product as well. So, at the same time, we also want to build and be able to add new features at an increased pace. So, we want to be able to add new product lines in the Cash App. So, for example, we built the Cash Card, which is a way you can keep your money in the Cash App bank accounts, and then you can spend that money using a separate card, and then we add a new functionality around that card, and so on. So, we also needed to be able to scale out the team to be able to have more people working on the team to build new products for our users, for our customers. Those are the two dimensions: we needed to scale out the system, but we also needed to have more people be able to work productively. So, that's why we started trying to chop up—we have this big monolith as most companies probably do, which that's I don't know how many hundreds of thousands of lines of code in there. But we also wanted to move things out of that, to be able to have more people contribute productively.Emily: And where are you in that process?Jon: Well, [laughs], we're probably adding still adding code at an exponential rate to the monolith. We're also adding code at an exponential rate outside of the monolith, but it just feels so much easier to just build some code in the monolith than it is outside of it, unfortunately, which something we're trying to fix, but it's very hard. And it is getting a little bit out of hand, this monolith now. So, we have, sort of, a moratorium on adding new code to the monolith now, and I'm not sure how much of an effect that has made. But the monolith is still growing, as well as our non-monolith services as well, of course. Emily: When you were faced with this scaling issue, what were the conversations happening between the technical side and the business owners? And how is this decision made about the best way to solve this problem is x, is the Cloud, is cloud-native architecture?Jon: I think the business side—the product owners, product managers—they trust us to make the right decision. So, it was largely a decision made on the technical side. They do still want us to build functionality, and to add new features, and fix bugs, and so on. So, they want us to do that, but they don't really have strong influence on the technical choices we've made. I think that's something we have to balance out. So, how can we keep on giving the product side and the business side what they need? So, to keep on delivering value to them while we try to rearchitect our system so that we can scale out our operations on our side. So, it's a very tricky balance to find there. And I think so far, maybe we've erred on the side of keep on delivering functionality, and maybe we need to do more on the rearchitecting things. But yeah, that's always a constant rebalancing act we're always dealing with. Emily: Do you think that you have gotten the increased scalability? How far along are you on reaching the goals that you originally had?Jon: I think we have a pretty scalable system now, in terms of the amount of customers we can service. So, we can add capacity. If we can keep on adding hardware to it, we can grow very far. We've actually noticed that the last few weeks, we've had an almost unprecedented growth, especially with the Coronavirus crisis. Every single day, it's almost a record. I mean, there's still issues, of course, and we're constantly trying to stay on top of that growth, but we have a reasonably good architecture there. What I think is probably our larger problem is the other side, so the human side. As I said, we are still adding code to this monolith, which is getting completely out of hand to work with. And we're not growing our smaller services fast enough. It's probably time to spend more effort on rearchitecting that side of things as well.Emily: What are some of the organizational, or people challenges that you've run into?Jon: Yeah. So, we want to build smaller teams oriented around products. We see ourselves more of a platform on products these days: we're not just a single product. And we want to build smaller teams. That is, maybe we have one team that is around our card, and one team around our [unintelligible] trading and so on. And we want to have the smaller teams, and we want them to be able to execute independently. So, we want to be able to put together a cross-functional team of some engineers, and some UX people, and some product people, and some business people, and then they should be able to execute independently and have their own services running in our cloud infrastructure, and not have to coordinate too much with all of the other teams that are also trying to execute independently. So, each product can do its own thing, and own their own services, and deploy at their own pace, and so on. That's what we're trying to achieve, but as long as they still have to do a lot of work inside of our big monolith, then they can't really execute independently. So, one team might build something that actually causes issues with another team's products, and so on, and that becomes very complicated to deal with. So, we tried to move away from that, and move towards a model where a team has a couple of services that they own, and they can do most of their work inside of those services.Emily: What do you think is preventing you from being farther along than you are? Farther along towards this idea of teams being totally self-sufficient?Jon: Yeah, I think it's the million-dollar question, really. Why are we still seeing exponential growth in code size in our monolith, and not in our services? And I think it's a combination of many, many things. One thing I think, we don't have all of the infrastructure available to us in our cloud, in our smaller services. So, say you want to build a little feature, you want to add a little button that does something, and if you want to do that inside our monolith, that might take you two, three days. Whereas if you want to pull up a completely new service—I think we've solved it at an infrastructural layer, it's very quick and easy to just pull up a new service, and have it run, and be able to take traffic, and so on—but it's more of the domain-specific infrastructures of being able to access all the different data sets that you need to be able to access, and be able to shift information back to the mobile device. And all these things, it's very easy to do inside a monolith, but it's much harder to do outside of the monolith. So, we have to replicate a big set of what we call product platforms. So, instead of infrastructural platform is more product specific platform features like customer information, and be able to send information back to the client, and so on. And all those things have to be rebuilt for cloud services. We haven't really gotten all the way there yet.Emily: If I understood correctly from the case study with the CNCF, you sort of started the cloud-native journey with your databases.Jon: Yes, that was the thing that was on fire. Cash App was initially built as a hack week project, and it was never really designed to scale. So, it was just running on a single MySQL database for a really long time. And we actually literally put a piece of hardware on fire with that database. We managed to roll it, roll it off, of course, didn't take down our service, but it was actually smoking in our [laughs] data centers. It melted the service around it in its chassis. So, that was a big problem, and we needed to solve that very quickly. So, that's where we started.Emily: Could you actually go into that just a little bit more? I read the case study, but probably most listeners haven't. Why was the database such a big problem? And how did you solve it?Jon: Yeah, as I said, so we only had a single MySQL database. And as most people know, it's very hard to keep on scaling that, so we bought more and more expensive hardware. And since we were a mobile app, we don't get all the benefits from caching and replica reads, so most of the time, the user is actually accessing data that is already on the device, so they don't actually make any calls out to our back end to read the data. Usually, you scale out a database by adding replicas, and caching, and that sort of stuff, but that wasn't our bottleneck. Our bottleneck was that we simply could not write to the database, we couldn't update the database fast enough, or with enough capacity. So, we needed to shard it, and split up the data set into smaller parts that we could run on separate databases. And we used the thing called Vitess for that, which is a Cloud Native Foundation member, a product and [unintelligible] CNCF. And with Vitess, we were able to split up the database into smaller parts. It was quite a large project, and especially back then, Vitess was—it was quite early days. So, the Vitess was used to scale out YouTube and then it was open-sourced. And then, we started using it. I think, not long after that, it was also used by Slack. So now, currently Slack uses it for most of its data. And we started using it very early, so it was still kind of early days, and we had to build a lot of new functionality in there, and we had to port [00:15:20 unintelligible] make sure all of our queries worked with the Vitess. But then we were able to do shard splitting. So, without having to restart or have downtime in our app, we could split up the database into smaller parts, and then the Vitess would handle the routing of queries, and so on.Emily: If at all, how did that serve as the gateway to then starting to think about changing more of the application, or moving more into services as opposed to a monolith?Jon: Yeah, I think that was kind of orthogonal in some ways. So, while we scaled out the database layer, we also realized that we needed to scale out the human side of it. So, we have multiple teams being able to work independently. And that is something we haven't I think we haven't really gotten to completely, yet. So, while we've scaled out the database layer, we're not quite there from the human side of things.Emily: Why is it important to scale any of this out? I understand the database, but why is it important to get the scaling for the teams?Jon: Yeah, I mean, it's a very competitive space, what we're trying to do. We have a very formidable competitors, both from other apps and also from the big banks, and for us to be able to keep on delivering new features for our customers at a high pace, and be able to change those features to react to changing customer demands or, like during this crisis we are in now, and being able to respond to what our competitors are doing. I mean, that just makes us a more effective business. And we don't always know when we start a new product line where it's exactly going to lead us, we sort of look at what our customers are using it and where that takes us, and being able to respond to that quickly, that's something that is very hard if you have a big monolith that has a million lines of code and takes you several hours to compile, then it's going to be very hard for you to deliver functionality and make changes to functionality in a good time.Emily: Can you think of any examples where you're able to respond really quickly to something like this current crisis in a way that wouldn't have been possible with the old models?Jon: I don't actually know the details here. I live currently in Australia, so I don't know. But the US government is handing out these checks, right? So, you get some kind of a subsidy. And apparently, they were going to mail those out to a lot of people, but we actually stepped up and said, look, you can just Cash App them out to people. So, people sign up for a Cash App account, and then they can receive their subsidies directly into the Cash App accounts, or into their bank accounts via our payment rails. And we were able to execute on that very quickly, and I think we are now an official way to get that subsidy from the US government. So, that's something that we probably wouldn't have been able to do unless we've invested more to be able to respond to that so quickly, within just weeks, I think.Emily: And as Cash App has moved to increasingly service-oriented architectures and increasingly cloud-native, what has been surprisingly easy?Jon: Surprisingly easy. I don't think I've been surprised by anything being easy, to my recollection. I think most things have been surprisingly hard. [laughs]. I think we are still somewhat in the early days of this infrastructure, and there are so many issues; there's so many bugs; there's so many unknowns. And when you start digging into things, it just surprises you how hard. So, I work in the infrastructure team, and we try to provide a curated experience for our product teams, the product engineering teams, so we deal with that pain directly where we have to figure out how all these products work together, and how to build functionality on top of them. I think we deal with that pain for our product engineers. But of course, they are also running into things all the time. So, no, it is surprisingly hard sometimes, but it's all right.Emily: What do you think has been surprisingly challenging, unexpectedly challenging?Jon: Maybe I shouldn't be, but I am somewhat surprised how immature things still are. Just as an example, how hard it is, if you run a pod, in a EKS—Amazon Kubernetes cluster, and you just want to authenticate to be able to use other Amazon products like Dynamo, or S3, or something, this is still something that is incredibly hard to do. So, you would think that just having two products from the same vendor inside of the same ecosystem, you would think that that would be a no-brainer: that they would just work together, but no. I think we'll figure it out eventually, but currently, it's still a lot of work to get things to play well together.Emily: If you had a top-three wish list of things for the community to address, what do you think they would be?Jon: Yeah, I guess the out-of-the-box experience with all of these tools, so that they just work together really well, without having to manually set up a lot of different things, that'd be nice. I think I also, maybe this all exists, we haven't integrated all these tools, but something that struck me the other day, I was debugging some production issue—it wasn't a major issue, but it was an issue that had been an ongoing thing for two weeks—and I just wanted to see what change happened those two weeks ago. What was the delta? What made that change happen? And being able to get that information out of Kubernetes and Amazon—and maybe there's some audit logging tools and all this stuff, but it's not entirely clear how to use them, or how to turn them on, and so on. So, that's a really nice, user friendly, and easy to use kind of auditing, and audit trail tools would be really nice. So, that's one wish, I guess, in general: having a curated experience. So, if you start from scratch, and you want to get all of the best practice tools, and you want to get all the functionality out of a cloud infrastructure, there's still a lot of choices to make, and there's a lot of different tools that you need to set up to make them work together, Prometheus, and Grafana, and Kubernetes, and so on. And having a curated out-of-the-box experience that just makes everything work, and you don't have to think about everything, that would be quite nice. So, Kubernetes operators are great, and these CRDs, this metadata you can store and work with inside of Kubernetes is great, but unfortunately they don't play well with the rest of the cloud infrastructure at Amazon, at AWS. Amazon was working on this Amazon operator, which you would be able to configure other AWS resources from inside of the Kubernetes cluster. So, you could have a CRD for an S3 bucket, so you wouldn't need a Terraform. So right now, you can have Helm Charts and similar to manage the Kubernetes side of things, but then you also need Terraform stuff to manage the AWS side of things, but just something thing that unifies this, so you can have a single place for all your infrastructural metadata. That would be nice. And Amazon is working on this, and they open-sourced something like an AWS operator, but I think they actually withdrew it and they are doing something closed-source. I don't know where that project is going. But that would be really nice.Emily: Go back again to this idea of the business of cloud-native. To what extent do you have to talk about this with business stakeholders? What are those conversations look like?Jon: A Cash App, we usually do not pull in product and business people in these conversations, I think, except when it comes to cost [laughs] and budgeting. But they think more in terms of features and being able to deliver and have teams be able to execute independently, and so on. And our hope is that we can construct an infrastructure that provides these capabilities to our business side. So, it's almost like a black box. They don't know what's inside. We are responsible for figuring out how to give it to them, but they don't always know exactly what's inside of the box.Emily: Excellent. The last question is if there's an engineering tool you can't live without?Jon: I would say all of the JetBrains IDEs for development. I've been using those for maybe 20 years, and they keep on delivering new tools, and I just love them all.Emily: Well, thank you so much for joining.Jon: Thanks for inviting me to speak on the podcast.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

Emily and Dejan cover the following points: 8x8's journey to a leading cloud technology provider. Why 8x8 decided to migrate to Kubernetes, a move that gave them the flexibility to run workloads wherever they want. Dejan's thoughts on the Kubernetes migration, and how it's helped the company improve its operations. For example, Kubernetes has helped 8x8 migrate away from several legacy systems. The biggest challenges and surprises that the 8x8 team experienced during their migration journey, such as getting engineering teams to embrace a culture built around monitoring, observability, and documentation. How 8x8 has avoided “feature bloat” and maintained a product that performs at a high level, while staying true to the features that are important for its core customer base.  The strategy of obtaining buy-in from stakeholders and fellow executives by focusing on business problems, instead of technical issues. This included cost, velocity of innovation, global scale, and so on. How 8x8's cloud-native architecture has made it faster and easier to scale.  TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I'm Emily Omier, and I am talking with Dejan Deklich, from 8x8.Dejan: So, I'm the Chief Product Officer at 8x8. To give you an idea, 8x8 is now 16 or 1700 employees worldwide, 450 million in revenue, give or take, offices all over the world, customers all over the world. I'm responsible for all product management, engineering, QA, project management operations for all the products worldwide for 8x8.Emily: Can you give me a little bit of an idea of 8x8's history in the Cloud?Dejan: So, 8x8 has been around, probably, a lot longer than most companies you're talking about. We've been public 30 years, give or take. We have been in the business of communication and collaboration since early 2000s. As you can imagine, we have gone through so many different tech stacks, architectures, and so on, that it is pretty amazing. We have, in the last several years, done a massive cleanup and rebuild of our software stack. We rebuilt pretty much all of the mobile apps, desktop apps, web apps. We rebuilt the platform starting with billing and provisioning all the way down to how the voice traverses the world. So, it's been a incredible couple of years, incredible journey where I would argue we have gone from the early versions of hosted service to early versions of Cloud, maybe 10 years ago, and we are now what I would like to call a proper cloud technology company. And it's been a very interesting, difficult journey. We learned a lot. We messed up a lot of things, then we learned some more than they did it correctly.Emily: When you first moved to Kubernetes, and the modern public cloud, what was the rationale? What were their business reasons?Dejan: Those multiple steps there. We moved to public cloud I don't know, five, six, seven years ago. We ran a lot of things in Amazon. And to be fair, we still also have data centers around the world. So, let me explain quickly what we actually running because I think it's important. So, we have, I think 16 data centers around the world, and then we run in pretty much every region of Amazon, we use Google Cloud extensively, and we have now shifted a lot of workloads to Oracle Cloud. At the same time, business is threatening me with Alibaba Cloud and Tencent Cloud as something that might be coming our way in the next couple of quarters. So, data centers are there because on the networking layer, the Cloud does not yet give us what we need for the realtime voice and video transmission. We actually are the best voice provider in the industry. We have proven that, and that's where your milliseconds really matter, therefore networking still sits in data centers. As soon as the backbone can be moved into Amazon, and we are told that could happen in the next three to four years, we will move likely everything to the Cloud. So, what we have generally in the Cloud are different applications, and the reason for that is simply the velocity of deploying and scaling them. So, what matters to us is, on one hand, the global reach: we have customers in 150 countries around the world. We have to have data centers close to the customers. And the applications need to be as close to the customer as possible, therefore all the different regions of Amazon, and Google, and whatnot. So, as you can imagine, managing all of that, monitoring all of that is a non-trivial exercise. So, we moved to Kubernetes, in large reason, simply because it is one underlying framework that allows us to run workloads wherever we want. So, to give you an idea, we launched a video meetings product to compete with Zoom. We had, on launch, a couple of hundred thousand users, nothing really. And then, this COVID-19 happened, and within a period of weeks, we now hit 15 million users. The only way you can scale a system like that is if you have a properly built underlying architecture, everything horizontally scalable. I was blown away, everything really worked. People were super busy, but by having proper cloud architecture, we were able to actually scale, and fulfill the demand that we have seen worldwide. Now, the nice thing is, as you put more and more workloads on top of Kubernetes, you can shift them between clouds as you want, or data centers as you want. And I think that's number one reason why we went with Kubernetes. I love Amazon, I love Google, and nothing makes me happier than writing them a million-dollar checks, but I also want to be able to move the workloads wherever I can run them cheaply. And, to me, that's very important. I don't have unlimited budget; I have to be able to play the game and get the most compute and the most bandwidth for the lowest cost that I can, and Kubernetes lets me do that.Emily: And would you say that Kubernetes was a technical decision or a business decision or both?Dejan: That's a good question. I think normally, the way we operate at 8x8, you start with the business problem. The business problem was we don't want to be locked into one cloud. We want to be able to run wherever we want to run, and on top of that, we have customers in Europe who are not very friendly towards Amazon, and want us to run on other clouds. And then, we took a peek: what can we do? What's the fastest and easiest way to do it? Turned out it was Kubernetes, so that's the way we went.Emily: What did the move to Kubernetes, what was it like? What were some of the surprises?Dejan: It was very interesting. It is still very interesting. So, on one hand, the good thing was we have already broken the monoliths in the past God knows how many years, into services. But to get things running properly in Kubernetes, you have to go a bit deeper, you actually have to really clean up your code, and so on, and so on. So, one thing that I thought was incredibly useful was this allowed us to, for the first time in 8x8 history, create a proper template for a service where all your monitoring, logging, debugging, all of your stuff is standard for all the services. So, as you can imagine, when you have a company that's been around for a long time, and you have a lot of services, and through various acquisition, you end up with different languages, different platforms, and so on, this now allowed us to clean up a lot of legacy and actually get to what I consider a really impressive engineering velocity. So, super challenging in many ways, because people had to say goodbye to some of the legacy tools, or ways of thinking, and so on. On the other hand, super-positive from the velocity we are now seeing. And honestly, if we haven't done some of these things, I doubt our platform would have survived the spike of the last two months.Emily: What were some surprises in moving to Kubernetes? And you mentioned that you made some mistakes and learned a lot. What were some of those?Dejan: It's funny, the good old story of automation and orchestration comes to bite you in the rear end every time, again. It's all nice when you have 10, 20 services. People can keep things in their mind. Once you are getting to hundreds or thousands of services that are running all over the world, things that you normally never want to do, such as documentation, such as fully automated CD/CI really become an absolute necessity. They stop being something that you learned about in your software engineering 101, and they become simply something that you really have to hardcore do. And getting the engineering teams to accept that and changing the culture towards more documentation, more monitoring, more observability, was the key. It was also really interesting to see all these different monitoring tools, how they either couldn't scale, or if they were able to scale the cost would explode, or getting the insight out of the data that was being collected was next to impossible, or you needed somebody who spent their whole life looking into Splunk or something like that in order to figure out what's really going on. The release process had to be completely changed. We completely rebuilt—we blew up the CI/CD pipelines completely, rebuilt everything cleanly, and so on, and so on. So, lots of very difficult engineering challenges, and I would argue the hardest thing was selling some of that work to the business. Business is always coming to you and saying, “Hey, we need feature XYZ, we need this product, we need that product.” And we would have to go back and say, “Well, actually, what we need is the availability, and the uptime, and stability, and we need to know what is really going on with the system so we can continue scaling it.” And looking back, I am really glad we did what we did because if we haven't started down this way, the 15 million users would not have happened.Emily: Tell me a little bit more about making that sale to the business. Was there anything that's lost in translation, more about that conversation with the business about why this needed to happen?Dejan: Right. So, it is very, very interesting. Most—and I have seen that pretty much in every company I've ever worked. There is always this feature bloat, and we are all guilty of that. And for sales, it is very easy to go to a prospect and go, “Hey, how do you like this?” Then prospects go, “Well, how about this and that feature?” sales says, “Absolutely.” And you end up with a roadmap which becomes this incredible sea-urchin-like monstrosity that goes in every possible dimension, and the cleanliness of the original product vision disappears over the years. You end up with this extremely complex enterprise product, which does everything for everybody, and does everything badly for everybody. So, I don't believe that's the right way to do business. And I was lucky enough that my CEO and the board fully agreed with my view that we need to go back to the core; we need to figure out what's really important to our customers, and as bad as this COVID-19 is, and as much chaos as this caused around the world, it was really gratifying for us inside 8x8 to see that all the work we have done in the last few years was actually done correctly with a correct view of the future. Think about it, a few years ago, everybody was always debating, “Oh, should I have engineers working from home, or not? Are people more efficient working from home or not?” Now, everybody's working from home. There is no more choice. There is no more talking about that topic. You have to be able to function from home. 8x8 builds the tools that ultimately allow enterprises to have their people function from home, both on the unified communication, and the contact center, and the video meeting side. So, you go, this now opens up the whole universe; I can hire people in anywhere around the world, we can do business anywhere around the world. And to me, that's the right way to go forward, but my job was to articulate what might happen, and what is likely to happen in the world and the economy in the coming five to 10 years, and this COVID-19 thing just pulled all of this work from home thing into now and not into next two, three, five years.Emily: When you were talking to business stakeholders about moving to Kubernetes and why it was important. Was there anything that you felt like was lost in translation?Dejan: They don't care about Kubernetes at all, and I never talked to them about Kubernetes. I talked to them about the scale and performance. Your CEO, your CFO, your Chairman of the Board, they don't care about Kubernetes or any technology. They care about the business problem. And the business problem is cost, velocity of innovation, global scale, global delivery, size of the engineering team, funding for R&D, you have to articulate what you want to do in the terms that matter to the business. Technology is one of many ways to solve the problem. If you could have trained monkeys do something super fast and super cheap, it might be a valid way to solve a problem. As we are lacking trained, cheap monkeys, we have to use technology. And for a lot of us technology is a great, great toy that we love to play with, and we enjoyed dealing with it, so it is sort of a way we solve the problem. The way a business solves the problem is effectively through the P&L. So, this translation between technical and business has to happen, and if you can't tell the story correctly, you will not get the funding you need, you will not get the project that you need, and ultimately the business will fail. And I think that's where the challenge lies. I see a lot of engineers and product people talk about the features, about how cool this or that would be, but they fail to forget that in the end, in a publicly-traded company, CFO and CEO go on the stage quarterly, and say, “This is the money we made. This is the money we spent.” And if you can help tell your story and the story of a product and engineering in the terms that they can use with the outside world, everybody's life becomes much easier.Emily: How did that conversation progress as you progressed on the technical transition?Dejan: So, it's like every other conversation. First time I said, “Yo, guys, we need to stop the feature work.” Obviously I got completely blank looks from everybody along the lines of, “Why would you ever want to do that?” after a couple of quarters of improving the engineering velocity, improving the product quality, stability, and so on, and so on. I think the whole company is now on the bandwagon. I think everybody by now understood why we are doing what we are doing. Which is great. If initially, there was some pull back from Sales—Sales loves more features, it makes their life easier, but there is also, I have to tell you now, a very visible and noticeable pride in the whole company about what we were able to do, about the scale we have achieved, and how quickly and painlessly we actually got there. So, it was not easy, first few months, I have to tell you that.Emily: What would you say were the biggest challenges?Dejan: Changing the culture. Changing the culture for the whole company to stop chasing features and start chasing elegance, and speed, and stability, and uptime, and global scale, on one hand; on the engineering side, it was about, stop committing to featurettes, and start thinking about larger architectural projects that will massively improve the company; on the sales side, it was about how do I stop talking about the future roadmap and start talking about the vision of the company and the vision of the product. Those are very noticeable and hard to pull off changes, and it took a lot of effort from all the teams to actually get on the bandwagon.Emily: Was there anything that was much easier than you expected?Dejan: [laughs]. Probably the core technical parts. The nice thing with good engineers is that they always love to try the latest and greatest. So, I was worried that people here will have objections to changing some of the frameworks and so on. Turned out that that was absolutely not the problem. I was delighted that the engineers and PMs were able to jump on the latest and greatest and just go run with it, learn what they had to learn, then go forward. In previous companies, I have seen that there's much more of a challenge than at 8x8.Emily: And then, from a technical perspective, what was more challenging than you expected, or what was a pain point?Dejan: The CI/CD pipeline. It was really interesting how—it's almost once you have gone down the cloud way, you have to clean up how you build and release software. And most companies get to this sort of in-between state where there is a part of automation, but not everything is fully automated, and so on, and so on. You have to really talk to the people at really big companies to get to the point where there is proper really end-to-end CI/CD. So, we were also in this in-between world. Yes, it's automated, yes it's CI/CD, but not really. And we spent a tremendous amount of time and energy into cleaning that up and finding all the edge cases. And honestly, I don't think that work will ever end. It seems to be as soon as we finish one project, we find other things that can be improved and cleaned up, and then we go and do that. So, by far, the biggest challenge was the Continuous Integration. Getting that to the point where it really works is a lot, a lot of work.Emily: Would you consider that among your biggest continuing challenges? Are there any other sort of challenges that you haven't quite figured out yet?Dejan: I think this is one of the bigger ones because we are not the only AWS, or only Google Cloud, or only data center. We have a mix of everything, and a mix of different applications and platform components that have to reside all over the world. And it is stunning how much work goes into this. The other very interesting set of problems that we are encountering is on the data side, and figuring out—I would argue we haven't yet really fully figured out the data storage requirements. So, to give you an idea—and it's not really so much even us figuring it out, it's the world figuring it out—every country has its own set of compliance regulations, security directives, they're all good, but how do you actually comply with all of them if you are providing service in 150 countries is a different problem. It almost feels like somebody on sand hill should invest in a startup which builds legal-as-a-service, something where the security and compliance will be done at the file system level so my engineers don't have to worry about it. To give you an idea, say you have a conversation, you and me: one of us sits in Europe, one of us sits in US. Who's data privacy laws win? Where can the data reside? How do you store the data? What happens if you have an employee of a US company which resides in Europe? How is that data handled? So, there's a lot of these unknowns on the data security side, which are really interesting, which are really complicated. And I think if somebody manages to build a service around that, they will do very, very well.Emily: Yeah. You have the employee of a US company who lives in Europe but is on vacation in Africa.Dejan: Exactly. And you go, “Okay, does GDPR win? Does the US win? Does—” how do we do this?Emily: Tell me just a little bit more about how Kubernetes and how the cloud-native architecture has made it possible to scale in the past month or so?Dejan: Well, by removing all manual steps, and by having very clean containers that can be deployed anywhere, we were able to scale roughly 100x. And then, to make it even better, we were able to move from Amazon to Oracle Cloud. We did a very interesting collaboration with Oracle, where we agreed that we want to provide the highest possible security to our video product. We felt we can get that security and enterprise presence through Oracle Cloud, and not only did we scale up video product in the last I would say two months by 100x, we also moved very large parts over to Oracle in order to manage cost and get increased security. 100 percent possible through modern, latest and greatest architecture and automation. Otherwise, I don't know how we wouldn't have done it.Emily: Has there been anything that surprised you over the course of this past two months, as you've seen the scaling. Any problems you encountered, for example, that you hadn't anticipated?Dejan: It was interesting to see some of the monitoring tools have issues, as well as seeing the cost of all of these things really explode. It was also interesting to see the response of various cloud providers. As soon as we saw the cost explode, we reached out to everybody. It was really interesting to see how some are much more open to collaborating, others are less open to collaborating. It's been a very interesting two months. But purely on the technical side, it's the monitoring that actually showed how important it is to be able to know what is really going on everywhere, and monitoring couple of thousand instances around the world, while possible is not necessarily easy.Emily: In terms of things that went wrong, or things that went right, any surprises or any notable examples?Dejan: Nothing—I was amazed that nothing really went wrong. I mean, we really worked our butts off, don't get me wrong. I mean, people have been up 16, 18 hour days for weeks to make sure everything works, and so on. But the fact that there were no serious problems still is amazing to me. I mean, 100x ramp in two months is a very nice ramp. I was amazed with how well the team reacted to this. There was really no complaining. This is what we were all working towards: sort of the best possible technical time mixed with the worst possible personal time because of the virus. So, it's been really interesting to see how people all over the world really stepped up, and kicked butt, and helped us move forward.Emily: Any comments you would have for other companies that are maybe a little behind 8x8 on the cloud transition?Dejan: Yes, figure out your platform. If you don't have a good underlying platform for everything, you will end up paying for it. I'm hoping most people are done forklifting legacy products into Cloud. If you are still in that phase, really start thinking about building cloud-native. And it's not cheap, it's not easy, it's not trivial, but you have to do it if you want to survive. Otherwise, when something like this virus happens to you, you will not be able to scale, and you will fail at the worst possible time. Cloud-native is the way to go. It's the future. Embrace it, but don't think it will all be roses along the way.Emily: Any other thoughts you have about the business of cloud-native, anything else you'd like to add?Dejan: I think the very interesting thing on cloud-native is, cloud-native has many different forms, and thinking across multiple clouds, both public and private, I think is the way to go. I'm sure there are applications out there that are perfectly fine running in a single cloud and scaling in a single cloud. But I would imagine for the vast majority of enterprises, they are in a very similar world to 8x8, where you will be faced with challenges, across the whole world, which are better solved in different cloud providers at a different time. And the sooner you start thinking about your multi-cloud strategy, and how you will shift from one to the other, the better off you will be. And then, if you can throw in the private cloud into all of that, that's where I believe things become truly interesting, and probably the future is, in some form or the other.Emily: All right. Well, I think we can go ahead and wrap it up there. Thank you so much for joining me. This was really great.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

Some of the highlights of the show include The diplomacy that's required between software engineers and management, and why influence is needed to move projects forward to completion. Driving factors behind Ygrene's Kubernetes migration, which included an infrastructure bottleneck, a need to streamline deployment, and a desire to leverage their internal team of cloud experts. Management's request to ship code faster, and why it was important to the organization.  How the company's engineers responded to the request to ship code faster, and overcame disconnects with management. How the team obtained executive buy-in for a Kubernetes migration. Key cultural changes that were required to make the migration to Kubernetes successful. How unexpected challenges forced the team to learn the “depths of Kubernetes,” and how it helped with root cause analysis. Why the transition to Kubernetes was a success, enabling the team to ship code faster, deliver more value, secure more customers, and drive more revenue.  Links: HerdX: https://www.herdx.com/ Ygrene: https://ygrene.com/ Austin Twitter: https://twitter.com/_austbot Austin LinkedIn: https://www.linkedin.com/in/austbot/ Arnold's book on publisher site: https://www.packtpub.com/cloud-networking/the-kubernetes-workshop  Arnold's book on Amazon: https://www.amazon.com/Kubernetes-Workshop-Interactive-Approach-Learning/dp/1838820752/ TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. My name is Emily Omier, and I am here with Austin Adams and Zack Arnold, and we are here to talk about why companies go cloud-native.Austin: So, I'm currently the CTO of a small Agrotech startup called HerdX. And that means I spend my days designing software, designing architecture for how distributed systems talk, and also leading teams of engineers to build proof-of-concepts and then production systems as they take over the projects that I've designed. Emily: And then, what did you do at Ygrene? Austin: I did the exact same thing, except for without the CTO title. And I also had other higher-level engineers working with me at Ygrene. So, we made a lot of technical decisions together. We all migrated to Kubernetes together, and Zack was a chief proponent of that, especially with the culture change. So, I focused on the designing software that teams of implementation engineers could take over and actually build out for the long run. And I think Zack really focused on—oh, I'll let Zack say what he focused on. [laughs].Emily: Go for it, Zach.Zach: Hello. I'm Zack. I also no longer work for Ygrene, although I have a lot of admiration and respect for the people who do. It was a fantastic company. So, Austin called me up a while back and asked me to think about participating in a DevOps engineering role at Ygrene. And he sort of said at the outset, we don't really know what it looks like, and we're pretty sure that we just created a position out of a culture, but would you be willing to embody it? And up until this point, I'd had cloud experience, and I had had software engineering experience, but I didn't really spend a ton of time focused on the actual movement of software from developer's laptops to production with as few hiccups, and as many tests, and as much safety as possible in between. So, I always told people the role felt like it was three parts. It was part IT automation expert, part software engineer, and then part diplomat. And the diplomacy was mostly in between people who are more operations focused. So, support engineers, project managers, and people who were on-call day in and day out, and being a go-between higher levels of management and software engineers themselves because there's this awkward, coordinated motion that has to really happen at a fine-grained level in order to get DevOps to really work at a company. What I mean by that is, essentially, Dev and Ops seem to on the surface have opposing goals, the operation staff, it's job is to maintain stability, and the development side's job is to introduce change, which invariably introduces instability. So, that dichotomy means that being able to simultaneously satisfy both desires is really a goal of DevOps, but it's difficult to achieve at an organizational level without dealing with some pretty critical cultural components. So, what do I spend my day on? The answer to that question is, yes. It really depends on the day. Sometimes it's cloud engineers. Sometimes it's QA folks, sometimes it's management. Sometimes I'm heads-down writing software for integrations in between tools. And every now and again, I get to contribute to open-source. So, a lot of different actual daily tasks take place in my position.Emily: Tell me a little bit more about this diplomacy between software engineers and management.Zach: [laughs]. Well, I'm not sure who's going to be listening in this amazing audience of ours, but I assume, because people are human, that they have capital O-pinions about how things should work, especially as it pertains to either software development lifecycle, the ITIL process of introducing change into a datacenter, into a cloud environment, compliance, security. There's lots of, I'll call them thought frameworks that have a very narrow focus on how we should be doing something with respect to software. So, diplomacy is the—well, I guess in true statecraft, it's being able to work in between countries. But in this particular case, diplomacy is using relational equity or influence, to be able to have every group achieve a common and shared purpose. At the end of the day, in most companies the goal is actually to be able to produce a product that people would want to pay for, and we can do so as quickly and as efficiently as possible. To do that, though, it again requires a lot of people with differing goals to work together towards that shared purpose. So, the diplomacy looks like, aside from just having way too many meetings, it actually looks like being able to communicate other thought frameworks to different stakeholders and being able to synthesize all of the different narrow-focused frameworks into a common shared, overarching process. So, I'll give you a concrete example because it feels like I just spewed a bunch of buzzwords. A concrete example would be, let's say in the common feature that's being delivered for ABC Company, for this feature it requires X number of hours of software development; X number of hours of testing; X number of hours of preparing, either capacity planning, or fleet size recommendations, or some form of operational pre-work; and then the actual deployment, and running, and monitoring. So, in the company that I currently work for, we just described roughly 20 different teams that would have to work together in order to achieve the delivery of this feature as rapidly as possible. So, the process of DevOps and the diplomacy of DevOps, for me looks like—aside from trying to automate as much as humanly possible and to provide what I call interface guarantees, which are basically shared agreements of functionality between two teams. So, the way that the developers will speak to the QA engineers is through Git. They develop new software, and they push it into shared code repositories, the way that the QA engineers will speak to people who are going to be handling the deployments—or at management in this particular case—is going to be through a well-formatted XML test file. So, providing automation around those particular interfaces and then ensuring that everyone's shared goals are met at the particular period of time where they're going to be invoked over the course of the delivery of that feature, is the “subtle art,”—air quotes, you can't see but—to me of DevOps diplomacy. That kind of help?Emily: Yeah, absolutely. Let's take, actually, just a little bit of a step back. Can you talk about what some of the business goals were behind moving to Kubernetes for Ygrene? Who was the champion of this move? Was it business stakeholders saying, “Hey, we really need this to change,” or engineering going to business stakeholders? Who needed a change. I believe that the desire for Kubernetes came from a bottleneck of infrastructure. Not so much around performance, such as the applications weren't performing due to scale. We had projected scale that we were coming to where it would cause a problem potentially, but it was also in the ease of deployment. It had a very operations mindset as Zack was saying, our infrastructure was almost entirely managed—of the core applications set—by outsourcing. And so, we depended on them to innovate, we depended on them to spin up new environments and services. But we also have this internal competing team that always had this cloud background. And so, what we were trying to do was lessen the time between idea to deployment by utilizing platforms that were more scalable, more flexible, and all the things that Docker gives with the Dev/Prod Parity, the ease of packaging your environment together so that small team can ship an entire application. And so, I think our main goal with that was to take that team that already had a lot of cloud experience, and give them more power to drive the innovation and not be bottlenecked just by what the outsourcing team could do. Which, by the way, just for the record, the outsourcing team was an amazing team, but they didn't have the Kubernetes or cloud experience, either. So, in terms of a hero or champion of it, it just started as an idea between me and the new CTO, or CIO that came in, talking about how can we ship code faster? So, one of the things that happened in my career was the desire for a rapid response team which, that sounds like a buzzword or something, but it was this idea that Ygrene was shipping software fairly slow, and we wanted to get faster. So, really the CIO, and one of the development managers, they were the really big champions of, “Hey, let's deliver value to the business faster.” And they had the experience to ask their engineers how to make that happen, and then trust Zack and I through this process of delivering Kubernetes, and Istio, and container security, and all these different things that eventually got implemented.Emily: Why do you think shipping code faster matters?Austin: I think, for this company, why it mattered was the PACE financing industry is relatively new. And while financing has some old established patterns, I feel like there's still always room for innovation. If you hear the early days of the Bridgewater Financial Hedge Fund, they were a source of innovation and they used technology to deliver new types of assets and things like that. And so, our team at Ygrene was excellent because they wanted to try new things. They wanted to try new patterns of PACE financing, or ways of getting in front of the customer, or connections with different analytics so they could understand their customer better. So, it was important to be able to try things, experiment to see what was going to be successful. To get things out into the real world to know, okay, this is actually going to work, or no, this isn't going to work. And then, also, one of the things within financing is—especially newer financing—is there's a lot of speed bumps along the way. Compliance laws can come into effect, as well as working with cities and governments that have specialized rules and specialized things that they need—because everyone's an expert when it comes to legislation, apparently—they decide that they need X, and they give us a time when we have to get it done. And so, we actually have another customer out there, which is the legislative bodies. So, they have to get the software—their features that are needed within the financing system out by certain dates, or we're no longer eligible to operate in those counties. So, one of it was a core business risk, so we needed to be able to deliver faster. The other was how can we grow the business?Emily: Zach, this might be a question for you. Was there anything that was lost in translation as you were explaining what engineering was going to do in order to meet this goal of shipping code faster, of being more agile, when you were talking to C level management? How did they understand, and did anything get lost in translation?Zach: One of the largest disconnects, both on a technical and from a high level speaking to management issue I had was explaining how we were no longer going to be managing application servers as though they were pets. When you come from an on-premise setup, and you've got your VMware ESXi, and you're managing virtual machines, the most important thing that you have is backups because you want to keep those machines exactly as they are, and you install new software on those machines. When Kubernetes says, I'm going to put your pods wherever they fit on the cluster, assuming it conforms with the scheduling pattern, and if a node dies, it's totally fine, I'm going to spin a new one up for you, and move pods around and ensure that the application is exactly as you had stated—as in, it's in its desired state—that kind of thinking from switching from infrastructure as pets to infrastructure as cattle, is difficult to explain to people who have spent their careers in building and maintaining datacenters. And I think a lot—well, it's not guaranteed that this is across the board, but if you want to talk about a generational divide, people that usually occupy the C level office chairs are familiar with—in their heyday of their career—a datacenter-based setup. In a cloud-based consumption model where it really doesn't matter—I can just spin up anything anywhere—when you talk about moving from reasoning about your application as the servers it comprises and instead talking about your application as the workload it comprises, it becomes a place where you have to really, really concretely explain to people exactly how it's going to work that the entire earth will not come crashing down if you lose a server, or if you lose a pod, or if a container hiccups and gets restarted by Kubernetes on that node. I think that was the real key one. And the reason why that actually became incredibly beneficial for us is because once we actually had that executive buy-off when it came to, while I still may not understand, I trust that you know what you're doing and that this infrastructure really is replaceable, it allowed us to get a little bit more aggressive with how we managed our resources. So, now using Horizontal Pod Autoscaling, using the Kubernetes Cluster Autoscaler, and leveraging Amazon EC2 Spot Fleets, we were only ever paying for the exact amount of infrastructure that was required to run our business. And I think that is usually the thing that translates the best to management and non-technical leadership. Because when it comes down to if I'm aware that using this tool, and using a cloud-native approach to running my application, I am only ever going to be paying for the computational resource that I need in that exact minute to run my business, then the budget discussions become a lot easier, because everyone is aware that this is your exact run-rate when it comes to technology. Does that make sense? Emily: Absolutely. How important was having that executive buy-in? My understanding is that a lot of companies, they think that they're going to get all these savings from Kubernetes, and it doesn't always materialize. So, I'm just curious, it sounds like it really did for Ygrene.Zach: There was two things that really worked well for us when this transformation was taking place. The first was, Ygrene was still growing, so if the budget grew alongside of the growth of the company, nobody noticed. So, that was one really incredible thing that happened that, I think, now having had different positions in the industry, I don't know if I appreciated that enough because if you're attempting to make a cost-neutral migration to the Cloud, or to adopt cloud-native management principles, you're going to probably move too little, too late. And when that happens, you run the risk of really doing a poor job of adopting cloud-native, and then scrapping that project, because it never materialized the benefit, as you just described, that some people didn't experience. And the other benefit that we had, I think was the fact that because there were enough incredibly senior technical people—and again, I learned everything from these people—working with us, and because we were all, for the most part, on the same page when it came to this migration, it was easy to have a unified front with our management because every engineer saw the value of this new way of running our infrastructure and running our application. In one non—and this obviously helps with our engineers—one non-monetary benefit that helped really get the buy-in was the fact that, with Kubernetes, our on-call SEV-1 pages went down, I want to say, by over 40 percent which was insane because Kubernetes was automatically intervening in the case where servers went down. JVMs run out of memory, exceptions cause strange things, but a simple restart usually fixes the vast majority of them. Well, now Kubernetes was doing this and we didn't need to wake somebody up in order to keep the machine running.Emily: From when you started this transition to when you, I should say, when you probably left the company, but what were some of the surprises, either surprises for you, or surprises for other people in the organization?Austin: The initial surprise was the yes that we got. So, initially I pitched it and started talking about it, and then the culture started changing to where we realized we really needed to change, and bringing Zack on and then getting the yes from management was the initial surprise. And—Emily: Why was that a surprise?Austin: It was just surprising because, when you work as an engineer—I mean, none of us were C suite, or Dev managers, or anything. We were just highly respected engineers working in the HQ. So, it was just a surprise that what we felt was a semi-crazy idea at the time—because Kubernetes was a little bit earlier. I mean, EKS wasn't even a thing from Amazon. We ran our Kubernetes clusters from the hip, which is using kops, which is—kops is a great tool, but obviously it wasn't managed. It was managed by us, mainly by Zach and his team, to be honest. So, that was a surprise that they would trust a billion-dollar financing engine to run on the proposal of two engineers. And then, the next ones were just how much the single-server, vertical scaling, and depending on running on the same server was into our applications. So, as we started to look at the core applications and moving them into a containerized environment, but also into an environment that can be spun up and spun down, looking at the assumptions the application was making around being on the same server; having specific IP addresses, or hostnames; and things like that, where we had to take those assumptions out and make things more flexible. So, we had to remove some stateful assumptions in the applications, that was a surprise. We also had to enforce more of the idea of idempotency, especially when introducing Istio, and [00:21:44 retryable] connections and retryable logic around circuit breaking and service-to-service communication. So, some of those were the bigger surprises, is the paradigm shift between, “Okay, we've got this service that's always going to run on the same machine, and it's always going to have local access to its files,” to, “Now we're on a pod that's got a volume mounted, and there's 50 of them.” And it's just different. So, that was a big—[laughs], that was a big surprise for us.Emily: Was there anything that you'd call a pleasant surprise? Things that went well that you anticipated to be really difficult?Zach: Oh, my gosh, yes. When you read through Kubernetes for the first time, you tend to have this—especially if somebody else told you, “Hey, we're going to do this,” this sinking feeling of, “Oh my god, I don't even know nothing,” because it's so immense in its complexity. It requires a retooling of how you think, but there have been lots of open-source community efforts to improve the cluster lifecycle management of Kubernetes, and one such project that really helped us get going—do you remember this Austin?—was kops.Austin: Yep. Yep, kops is great.Zach: I want to say Justin Santa Barbara was the original creator of that project, and it's still open source, and I think he still maintains it. But to have a production-ready, and we really mean production-ready: it was private, everything was isolated, the CNI was provisioned correctly, everything was in the right place, to have a fully production-ready Kubernetes cluster ready to go within a few hours of us being able to learn about this tool in AWS was huge because then we could start to focus on what we didn't even understand inside of the cluster. Because there were lots of—Kubernetes is—there's two sides of it, and both of them are confusing. There's the infrastructure that participates in the cluster, and there's the actual components inside of the cluster which get orchestrated to make your application possible. So, not having to initially focus on the infrastructure that made up the cluster, so we could just figure out the difference between our butt and the hole in the ground, when it came to our application inside of Kubernetes was immensely helpful to us. I mean, there are a lot of tools these days that do that now: GKE, EKS, AKS, but we got into Kubernetes right after it went GA, and this was huge to help with that.Emily: Can you tell me also a little bit about the cultural changes that had to happen? And what were these cultural changes, and then how did it go?Zach: As Austin said, the notion of—I think a lot—and I don't want to offer this as a sweeping statement—but I think the vast majority of the engineers that we had in Seattle, in San Jose, and in Petaluma where the company was headquartered, I think, even if they didn't understand what the word idempotent meant, they understood more or less how that was going to work. The larger challenge for us was actually in helping our contractors, who actually made up the vast majority of our labor force towards the end of my tenure there, how a lot of these principles worked in software. So, take a perfect example: part of the application is written in Ruby on Rails, and in Ruby on Rails, there's a concept of one-off tasks called rake tasks. When you are running a single server, and you're sending lots of emails that have attachments, those attachments have to be on the file system. And this is the phrase I always said to people, as we refactor the code together, I repeated the statement, “You have to pretend this request is going to start on one server and finish on a different one, and you don't know what either of them are, ahead of time.” And I think using just that simple nugget really helped, culturally, start to reshape this skill of people because when you can't use or depend on something like the file system, or you can't depend on that I'm still on the same server, you begin to break your task into components, and you begin to store those components in either a central database or a central file system like Amazon S3. And adopting those parts of, I would call, cloud-native engineering were critical to the cultural adoption of this tool. I think the other thing was, obviously, lots of training had to take place. And I think a lot of operational handoff had to take place. I remember for, basically, a fairly long stretch of time, I was on-call along with whoever was also on-call because I had the vast majority of the operational knowledge of Kubernetes for that particular team. So, I think there was a good bit of rescaling and mindset shift from the technical side of being able to adopt a cloud-native approach to software building. Does that make sense?Emily: Absolutely. What do you think actually were some of the biggest challenges or the biggest pain points? Zach: So, challenges of cultural shift, or challenges of specifically Kubernetes adoption?Emily: I was thinking challenges of Kubernetes adoption, but I'm also curious about the cultural shift if that's one of the biggest pain points.Zach: It really was for us. I think—because now it wouldn't—if you wanted to take out Kubernetes and replace it with Nomad there? All of the engineers would know what you're talking about. It wouldn't take but whatever the amount of time it would to migrate your Kubernetes manifests to Nomad HCL files. So, I do think the rescaling and the mindset shift, culturally speaking, was probably the thing that helped solidify it from an engineering level. But Kubernetes adoption—or at least problems in Kubernetes adoption, there was a lot of migration horror stories that we encountered. A lot of cluster instability in earlier versions of Kubernetes prevented any form of smooth upgrades. I had to leave—it was with my brother's—it was his wedding, what was it—oh, rehearsal dinner, that's what it was. I had to leave his rehearsal dinner because the production cluster for Ygrene went down, and we needed to get it back up. So, lots of funny stories like that. Or Nordstrom did a really fantastic talk on this in KubeCon in Austin in 2017. But the [00:28:57 unintelligible] split-brain problem where suddenly the consensus in between all of the Kubernetes master nodes began to fail for one reason or another. And because they were serving incorrect information to the controller managers, then the controller managers were acting on incorrect information and causing the schedulers to do really crazy things, like delete entire deployments, or move pods, or kill nodes, or lots of interesting things. I think we unnecessarily bit off a little bit too much when it came to trying to do tricky stuff when it came to infrastructure. We introduced a good bit of instability when it came to Amazon EC2 Spot that I think, all things considered, I would have revised the decision on that. Because we faced a lot of node instability, which translated into application instability, which would cause really, really interesting edge cases to show up basically only in production.Austin: One of the more notable ones—and I think this is the symptom of one of the larger challenges was during testing, one of our project managers that also helped out in the testing side—technical project managers—which we nicknamed the Edge Case Factory, because she was just, anointed, or somehow had this superpower to find the most interesting edge cases, and things that never went wrong for anyone else always went wrong for her, and it really helped us build more robust software for sure, but there's some people out there with mutant powers to catch bugs, and she was one of them. We had two clusters, we had lower environment clusters, and then we had production cluster. The production cluster hosted two namespaces: the staging namespace, which is supposed to be an exact copy of production; and then the production namespace, so that you can smoke-test legitimate production resources, and blah blah blah. So, one time, we started to get some calls that, all of a sudden, people were getting the staging environment underneath the production URL. Zach: Yeah.Austin: And we were like, “Uh… excuse me?” It comes down to—we eventually figured it out. It was something within the networking layer. But it was this thing, as we rolled along, the deeper understanding of, okay, how does this—to use a term that Zack Arnold coined—this benevolent botnet, how does this thing even work, at the most fundamental and most detailed levels? And so, as problems and issues would occur, pre-production or even in production, we had to really learn the depths of Kubernetes. And I think the reason we had to learn it at that stage was because of how new Kubernetes was, all things considered. But I think now with a lot more of the managed systems, I would say it's not necessary, but it's definitely helpful to really know how Kubernetes works down in the depths. So, that was one of the big challenges was, to put it succinctly, when an issue comes up, knowing really what's going on under the hood, really, really helped us as we discovered and learned things about Kubernetes.Zach: And what you're saying, Austin, was really illuminated by the fact that the telemetry that we had in production was not sufficient, in our minds, at least until very recently, to be able to adequately capture all the data necessary to accurately do root cause analyses on particular issues. In early days, there was far too much root cause analysis by, “It was probably this,” and then we moved on. Now having actually taken the time to instrument tracing, to instrument metrics, to instrument logs with correlation, we used, eventually, Datadog, but working our way through the various telemetry tools to achieve this, we really struggled being able to give accurate information to stakeholders about what was really going wrong in production. And I think Austin was probably the first person in the headquarters side of the company—I'm not entirely certain about some of our satellite dev offices—but to really champion a data-driven way of actually running software. Which, it seems trivial now because obviously that's how a lot of these tools work out of the box. But for us, it was really like, “Oh, I guess we really do need to think about the HTTP error rate.” [laughs].Emily: So, taking another step back here, do you think that Ygrene got everything that it expected, or that it wanted out of moving to Kubernetes?Austin: I think we're obviously playing up some of the challenges that we had because it was our day-to-day, but I do believe that trust in the dev team grew, we were able to deploy code during the day, which we could have done that in the beginning, even with vertically scaled infrastructure, we would have done it with downtime, but it really was that as we started to show that Kubernetes and these cloud-native tools like Fluentd, Prometheus, Istio, and other things like that when you set them up properly, they do take a lot of the risk out. It added trust in the development team. It gave more responsibility to the developers to manage their own code in production, which is the DevOps culture, the DevOps mindset. And I think in the end, we were able to ship code faster, we were able to deliver more value, we were able to go into new jurisdictions and markets quicker, to get more customers, and to ultimately increase the amount of revenue that Ygrene had. So, it built a bridge between the data science side of things, the development side of things, the project management side of things, and the compliance side of things. So, I definitely think they got a lot out of trusting us with this migration. I think that were we to continue, probably Zack and I even to this day, we would have been able to implement more, and more, and more. Obviously, I left the company, Zach left the company to pursue other opportunities, but I do believe we left them in a good spot to take this ecosystem that was put in place and run with it. To continue to innovate and do experiments to get more business.Zach: Emily, I'd characterize it with an anecdote. After our Chief Information Officer left the company, our Chief Operating Officer actually took over the management of the Technology Group, and aside from basically giving dev management carte blanche authority to do as they needed to, I think there was so much trust there that we didn't have at the beginning of our journey with technology and Ygrene. And it was characterized in, we had monthly calls with all of the regional account managers, which are basically our out-of-office sales staff. And generally, the project managers from our group would have to sit in those meetings and hear just about how terrible our technology was relative to the competition, either lacking in features, lacking in stability, lacking in design quality, lacking in user interface design, or way overdoing the amount of compliance we had to have. And towards the end of my tenure, those complaints dropped to zero, which I think was really a testament to the fact that we were running things stably, the amount of on-call pages went down tremendously, the amount of user-impacting production outages was dramatically reduced, and I think the overall quality of software increased with every release. And to be able to say that, as a finance company, we were able to deploy 10 times during the day if we needed to, and not because it was an emergency, but because it was genuinely a value-added feature for customers. I think that that really demonstrated that we reached a level of success adopting Kubernetes and cloud-native, that really helped our business win. And we positioned them, basically, now to make experiments that they thought would work from a business sense we implement the technology behind it, and then we find out whether or not we were right.Emily: Let's go ahead and wrap up. We're nearing the top of the hour, but just two questions for both of you. One is, where could listeners find you or connect with you? And the second one is, do you have a can't-live-without engineering tool?Austin: Yeah, so I'll go first. Listeners can find me on Twitter @_austbot, or on LinkedIn. Those are really the only tools I use. And I can't really live without Prometheus and Grafana. I really love being able to see everything that's happening in my applications. I love instrumentation. I'm very data-driven on what's happening inside. So, obviously Kubernetes is there, but it's almost become that Kubernetes is the Cloud. I don't even think about it anymore. It's these other tools that help us monitor and create active monitoring paradigms in our application so we can deploy fast, and know if we broke something. Zach: And if you want to stay in contact with me, I would recommend not using Twitter, I lost my password and I'm not entirely certain how to get it back. I don't have a blue checkmark, so I can't talk to Twitter about that. I probably am on LinkedIn… you know what, you can find me in my house. I'm currently working. The engineering tool that I really can't live without, I think my IDE. I use IntelliJ by JetBrains, and—Austin: Yeah, it's good stuff.Zach: —I think I wouldn't be able to program without it. I fear for my next coding interview because I'll be pretending that there's type ahead completion in a Google Doc, and it just won't work. So, yeah, I think that would be the tool I'd keep forever.Austin: And if any of Zach's managers are listening, he's not planning on doing any coding interviews anytime soon.Zach: [laughs]. Yes, obviously.Emily: Well, thank you so much. Zach: Emily Omier, thank you so much for your time.Austin: Right, thanks.Austin: And don't forget Zack is an author. He and his team worked very hard on that book.Emily: Zack, do you want to give a plug to your book?Zach: Oh, yeah. Some really intelligent people that, for some reason, dragged me along, worked on a book. Basically it started as an introduction to Kubernetes, and it turned into a Master's Course on Kubernetes. It's from Packt Publishing and yeah, you can find it there, amazon.com or steal it on the internet. If you're looking to get started with Kubernetes I cannot recommend the team that worked on this book enough. It was a real honor to be able to work with people I consider to be heavyweights in the industry. It was really fun.Emily: Thank you so much.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

Some highlights of the show include The challenges of operating digital commerce at scale, including the need for resource pooling and resiliency — and how this caused Ant Financial to re-think their infrastructure.  Ant Financial's former approach to scaling, which was mostly manual, and highly resource-intensive.  How Kubernetes is expediting cloud development for Ant Financial. Haojie's thoughts on the global engineering skills gap, and China's growing cloud computing market including driving factors and barriers.  Why Ant Financial's migration has largely been a success — and why achieving operational security is now a top priority for the company.   How Ant Financial is managing disconnect between its engineers and business leaders.  The company's ongoing mission to migrate its systems and applications away from legacy architectures. Links LinkedIn: https://www.linkedin.com/in/haojiehang/ https://www.investopedia.com/tech/worlds-top-10-fintech-companies-baba/ TranscriptAnnouncer: Welcome to The Business of Cloud Native podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: So, I always start the same way. Can you introduce yourself?Haojie: Hey, my name is Haojie Hang. I'm a product manager in the CTO office at Ant Financial. I work on the product and strategy side for, basically, the CTO and the other executive leaders, as well as leading a small product teams within the org to look at the frontier technology in the cloud and other infrastructure businesses.Emily: And can you tell me a little bit more about what Ant Financial does? And then, also, what do you do on a day to day basis? What do you do when you get into the office?Haojie: Yeah, I'll do a quick introduction about the Ant Financial business. It's not just one business or two business, it's a group of businesses that we innovate and we do, mostly in China, but we're also expanding very rapidly all over the world. So, Ant Financial is basically a group of businesses including credit for both consumers and the enterprise, as well as loan businesses, both consumer and enterprise businesses. We say that the parent organization is basically, we call it Alipay, it's the earliest business we do since 2004 when the business was basically born from Taobao, which is our parent company. So, in short, the Ant Financial Business has a lot of presence in the business of payments business, remittance, credit card, loans, securities, and many other businesses like intelligent technology, blockchain, pretty much everything you can imagine in the FinTech and financial services, we're in there.Emily: Tell me a little bit more about the cloud-native journey for Ant Financial. When did it start? Why did it start? What was some of the motivations behind moving to cloud-native?Haojie: Yeah, it's actually quite interesting. I joined Ant Financial in 2008, but actually, the entire company started to look at cloud-native technology quite early, in 2012. So, back then, people were just looking at these technologies around the world, mostly from the US, they look at this open-source community, look at what other companies are doing, how to use the cloud-native technology to help with their business in the peak time, so during event. There's online promotion event we're doing every year, called Double 11—Shuāng shíyī in Chinese. Every year, so we have a large amount of promotional events happening online, trying to help merchants and the customer is trying to sell and buy stuff in our Tmall and Taobao platform in very, very discounted price. So, for that promotion event online, we have to think about the resilience, the resource pooling, oftentimes the visits has to increase multiple times, sometimes over 100 times the increase compared to the normal time. So in that case, we have to think about how we can be very resilient and efficient infrastructure to support that business needs. So, this is a very large topic. And then, back then, there was a lot of focus and study in our cloud computing department. So, we started looking at this technology called Mesos in 2012. And then, we do a lot of experiments around this technology, but from the business perspective, it's still hard to justify the benefits of moving to Mesos completely. So, we have multiple teams doing a lot of research in Mesos, in Kubernetes, sometimes in our own technology stack, but there's not enough proof or enough confidence for us to move completely over to that technology, until the emergence of Docker container, this Docker technology. Then we started to look at our container infrastructure, really do the investigation around this technology, and understand why this is taking over so quickly over the world, from the business perspective, and from the technology perspective. If you look at the community of Docker, the thing does not really happen until 2015. But we are already in the game for about a year or two. So, we're actually quite happy about our original strategy, but it's just in terms of the research. We're actually a little bit behind in terms of moving to this cloud-native architecture. But as you can see, that I had an interview with CNCF. So, we are very happy about the results that we have right now. Pretty much the entire architecture we run within Ant Financial is, basically, on Kubernetes ecosystem. It's not just using the open-source version of it. We're doing a lot of customization around this open-source framework. Yeah, I can talk more about the details.Emily: Yeah. Well, let's back up just a little bit. I'm curious what you were doing to manage this scaling before? And how did that change? And what about the whole process changed? Like, how stressful is it now, compared to before?Haojie: The process was very manual, I would say. We have extremely large team of engineers, and DevOps, security teams. And oftentimes their responsibility are overlap. So, some engineers are doing security work, some engineers are doing basically operational work. I would say, some people really hated it because they have to be on the computer, look at monitor 24/7, making sure transactions succeeded. When the peak time happens, there's nothing wrong with it. Sometimes they have to keep their phone open 24/7, basically to make sure this thing will not fail, right? And then, just many parts of work has to—so in the previous way, the way we do this operation is quite manual. We don't have a mature system or methodology telling us what we should do first, we should do second, and what's what would you do after this. So, basically the collaboration chain was not there. Therefore, when issue happens, our operation team has to respond very quickly. But then, how can we quickly identify the problem, and make it a problem? That's a problem, right? So, we have to make sure every time we respond, we respond in a very effective manner. That's the problem. In the previous process when something unexpected happen, who had to engage with the entire team from product, engineering, operation, security, everybody has to get up and look at the problem together, which was quite inefficient. So, after we moved to this cloud-native architecture—it's not the standard cloud architect, it's, kind of—we have a lot of innovation on top of this, to make sure that's fitting to our tech community, to our businesses. So, we basically did a lot of innovation in the process to make sure after we had this transition, people are clear about the roles, what they should be responding, and then who should be doing what. That's quite important.Emily: And tell me a little bit more about some of the additional layers or some tools that you've built on top of Kubernetes, and how that's helped you be successful.Haojie: Yeah, so I can give you an example. So, when we look at Kubernetes technology in our intelligent technology, or intelligent cloud business units, we're thinking about how can we use Kubernetes for cloud deployment. Okay, so previously we are using Mesos to do that. But we found this technology that lots of people are familiar with this technology. And then, people are not very sure if Mesos is the right path for container management, resource management or cloud deployment. But when we move to the Kubernetes for cloud deployment, people are actually quite happy. We are seeing a decrease in the amount of—to stand up the cloud. And previously, it took us two to three weeks to build an entire cloud. But after we use the Kubernetes technology, we can do that in a week. Oftentimes, if the scale is smaller, we can do that within three days. That's quite important because people are confident about the community; about this technology. And then, from the users perspective, they are also more willing to invest in Kubernetes. Oftentimes, this is the chicken-egg problem, right? When more companies are hiring more this, these terms appears in the market, in the job descriptions, the more people are willing to learn. So, this is actually what we're seeing a very, very good cycle for me, from both a company perspective and the talent perspective. So, that's actually quite good. But the problem for us is, there's still not enough people, or we say, you know, good talents in the market that we can attract. Basically, we're seeing a shortage of great engineering talent in the market, after the cloud-native transition. So, we're still trying to think about how we can educate the internal audience in the technology community to help them quickly pick up this new technology in the cloud, as well as the practice behind the cloud-native architecture.Emily: You know, I wanted to talk to you a little bit about the overall situation in China and that there's also this, sort of, skills gap. It sounds like it's just as present in China as it is in other parts of the world.Haojie: So, I would say in terms of the cloud computing, cloud-native tech community, it's pretty much—we had community forming as early as the rest of the world. But then, in early days, it was just a marketing term by people saying, oh, this is cloud, we want to learn something. But then really, from the business perspective, there's still not enough customer trying to pay for this technology. Oftentimes, the contract size was not large enough to feed engineers. That's what I say. And then, I think the trend of the serious adoption really happens in the past three to four years when a lot of startups coming out in the cloud businesses. There's a company called [QingCloud]. There's a company called [unlcear]. There's many other unicorns in China in the cloud computing space, and I think two of them just went public in the A-listed share recently. So, from that perspective, I would say, the cloud computing business is really maturing rapidly in the past two years. Because we see some unicorns really coming out of this game, besides Alibaba. I think, from that perspective, I will say, it's getting better and better. It's just in terms of the pay behavior, right? How much customers are willing to pay for this technology, pay for the services, pay for the products? I think it will still take some time to mature.Emily: To what extent do you see Chinese companies using cloud-native services and tools from Europe, from the United States, from elsewhere in the world? To what extent is it a segregated market? Like, the rest of the world doesn't use Chinese tools, Chinese companies don't use tools from the rest of the market.Haojie: It's a good question. To me, I'm coming from an engineering background, I believe open-source community is global. It's a global phenomenon. I think the world is connected. And I would never say that people in China, or Chinese companies, or—they only using technology businesses that created in China, this is not right. And often in many cases, there's not enough options, right? So, I think, even though Chinese companies and startups are trying to innovate very aggressively, but I think the world is still connected, they have to build on top of the innovation that's already happened in the rest of the world. So, in that case, I think we're still seeing a lot of collaboration across the globe. China, United States, for sure, Europe, other parts of the world. It's just how aggressive people are in terms of investing in frontier technology. And are they really seeing the benefits of using the frontier technology? There's question of technology innovation versus business innovation, right? Do you see the business value? Can you really see that in the next five years? I would say in China, most of the non-internet sector, they're quite short-sighted. They're still trying to survive, they're trying to make sure they are doing—they can become the top three, top five in their business. So, technology is oftentimes secondary. But for the leaders in that sector, they have to think about that quite early in order to become the top player in their sector. So, I think, the trend is that people are still collaborating academically and engineering side to make sure the right technology gets applied in the right scenario and trying to improve the technology at home.Emily: It's interesting that you mentioned that some Chinese companies might not focus as much on the technology. Do Chinese companies tend to consider moving to cloud-native, important for their business? Or a strategic move?Haojie: From the strategy perspective, yes. Every leader would definitely know about cloud computing, cloud-native architecture, they would definitely think about moving. It's just that internal execution when they think about moving seriously, they have to evaluate, do we have enough talent? How much business value am I getting out of this? Is it really helping? What is my budget? And all those kind of fears, problems. So, that's what backing them up because oftentimes they don't have enough budget. That's what I say in those non-internet sector. Because I've lived and worked in the US for a while. I think in the US, the non-internet sector are quite advanced in terms of the technology adoption, especially in cloud-native. It's quite easy for them to recruit, then build a large engineering team to work on cloud infrastructure software. But it's not the case in China because people are still trying to, especially the leader who can make the decision, they're still thinking about the ROI, the rate of returns, the rate of investments, for building a strong software teams, making sure they have the robust infrastructure running at the bottom level. So, they are still trying to figure out the budget, make sure they are profitable enough to afford that.Emily: Do you feel like Ant Financial got the business benefits out of the transition that it was looking for?Haojie: Yeah, as I mentioned, the entire organization are quite happy about the move because really, they are, kind of, [unintelligible] move in China. So, basically, even the non-engineering teams started to appreciate this, and talk about this technology, and trying to understand it deeper, because they see the entire organization are quite happy, especially from the business protective. As I mentioned, in the Double 11 event we have—last year in 2019, the GMV we had was 260 billion RMB in total, which is 25 percent growth compared to the last year. So, for that large amount of GMV, we supported the entire infrastructure, are building from our cloud infrastructure. It is quite massive. We don't have a infrastructure for that business, we have the infrastructure for the entire group, including Ant Financial and Alibaba. Basically the entire businesses is running in the cloud. We have very, very few siloed data centers and the infrastructure—you know, uh, data centers—basically, we have the entire thing running in single cloud. That's the largest achievement we had I think since 2019, which was one of the strategic goal we had, we achieved last year. And this year, we're putting a lot of emphasis in the secure operation. It has been one of the primary cloud business goal because when bad things happen, people are literally losing money. Imagine one of the transactions failed. It failed the entire country, right? Like, no one else in China or in the other part of the world can make a purchase from Alipay app. This is quite devastating. So, secure operation has been the only thing we focus on this year, I would say. I remember in some meetings, one of the leader mentioned, “If there's only thing we should do this year, it's secure operation.” We're trying to make sure we operate the entire business safely on top of our new cloud-native architecture, with the minimum amount of incidents and failures.Emily: And what do you think have been some of the challenges? What has been more difficult than you imagined in making this transition?Haojie: Yeah, I think for me, the most obvious point is that we still have a large amount of operating team engineers, and support team, and the product, and the entire organization, basically, to making sure the entire thing working seamlessly because I think it's very hard to quantify it. I think the overall efficiency in running the cloud-native architecture, we're still looking at that. Let me try to find a good example.Emily: Let me ask a question. What's gone unexpectedly well? Was there anything that you thought was going to be really challenging that wasn't?Haojie: Oh, I think after moving to the cloud-native architecture, the engineers are quite happy. They're working much, much harder. They're trying to do things much more quickly than we imagined. Basically, they are very aggressive, and very happy to see the leadership teams really buying this technology, and they're invest—want to invest seriously in this technology. They are building not only the engineering team but also the prod team, the entire organization around to cloud-native technology. So, oftentimes in order to persuade business leaders to do something serious in the technology, they have to spend a lot of time trying to evangelize to the leadership team to making sure they understand, oh, this is the right direction. We have to do this right. It takes oftentimes from six months to a year for them to really doing that. So, for that, I think it's quite successful. We see a very—basically I think the entire engineering culture has changed. People are looking at open-source community more aggressively. They think about how we should contribute back to the community. What community events should I support? What conferences should I go to? There's more and more discussion like that happening within the organization. And, I think, larger Ant Financial has become one of the sponsor in the events. We are one of the most active participants in the community, I think, since 2019, along with Alibaba. So, that's the positive side I'm seeing. People really start to form a culture on their own, especially in open-source community. Trying to be more present, trying to take more active position in the discussion, both within company and outside of company. So, that's actually quite the good. We're happy to see engineers are doing their work, and are doing it more aggressively.Emily: Do you feel like there's any sort of disconnect between the engineering teams and business leaders? Or do you feel like they're mostly on the same page?Haojie: Yeah, I would say there are still some gaps between the business leaders and the engineers. So, oftentimes, I would say the engineers are quite updated with what's going on in this community, in some new plugins, in some new components coming out of this Kubernetes ecosystem, but then the business leaders don't have enough time to to pay attention to this. So, it really depends on how confident they are about this technology. And how much more time do you want to put into this personally. I think the business leader will look at the numbers like KPIs, metrics, the number of accidents, the operating efficiencies, things like that, but that's in the business context, all right. The engineer leaders cares more about what kind of new technology we use, what kind of new technology we created on top of this ecosystem, and how many people are happy about using this technology? And how many more can we do from this transition? So, basically, they are disconnect. So, I think the good part in Ant Financial is that for business leaders, most of the business leaders are coming from engineering background, but they have a strong KPI in their work. And then, most of the engineer leaders has to learn business, because, in order to persuade business leaders to invest in this, they have to think from their perspective. I think, in terms of the communication, they're quite up to date. It's just in terms of the execution and the timelines there are some disconnected happen. Yeah.Emily: What would you say that business leaders are looking for that engineering teams might not be thinking about?Haojie: I think one example that I see is a business leader will think about the team building, the talent building, the culture, and the public image that we had in the public, especially in China. Yeah, let me give you an example. If the technology—if the company is not cool enough, from the technology, from engineering perspective, it's very hard to attract the top talent in engineerings from the business leader. Without strong engineering teams, we cannot execute. We cannot innovate. So, that's something they oftentimes think about when they try to invest in technology. But in terms of the execution, after the engineers gets on board, and work in Alibaba, in Ant Financial, that's something engineers have think about. How do they keep the talent? How do they make sure talents are happy? How do we make sure they are satisfied about what they do? So, I would say these two things have to work at the same time. You cannot have a strong image in technology, in frontier technology. But then, after the talent gets on board, they realize, oh, this is just great from outside, but from inside, we are still working on the legacy technology. It's operate very inefficiently internally, and how can we make sure people are dealing this? And I think that's quite important.Emily: Is there anything that you think is preventing you from moving further along in the cloud-native journey? Anything other than lack of human resources?Haojie: I would say that how can we securely move away from the legacy architecture, whether it's built privately or you built it using other vendor's technology? You know, for that kind of transition we're taking very seriously. We still have a large amount of systems and applications running on Oracle, running on, sometimes in MySQL, sometimes in other siloed stack. And we're not 100 percent. We're in one cloud, we're not 100 percent away from Oracle, MySQL or that type of, we consider legacy, architecture. So, the moving will still take some time. And so, how can we make sure the transition is successful? How can we make sure the transition is less painful? Is something we as the leaders and the business executives will think about because how do you how we can set up the right KPI and the right goal for engineers to feel happy about doing this work? I think that's one of the challenges. Oftentimes people, when they are placed into this kind of work, moving from legacy architecture, to new architecture, just very minimum business value we can see from this transition, right? So, we have to have the right—we have to set the enough goal to motivate them to do the work. That's something we have to really think about that in the long term. Because this is not like we do that for six months, a year. It's going to be an effort for the next three to four years. Imagine, Alipay business started in 2004, and it's been already 16 years. So, the transition was to happen over time. It's just, how we can make sure that the transition that it's less painful?Emily: Tell me just a little bit more about some of the custom capabilities that you built on top of Kubernetes.Haojie: We have our own internal monitoring architecture, which is quite advanced, I would say. And this kind of monitoring infrastructure is built for both developers and operators. I think that is something we invest extremely heavily because we cannot find any other alternatives in the market. I'll give you some background about this monitoring infrastructure. So, the entire tech stack was primarily built on Java stack, the thing starting from 2004. And now a lot of cloud-native technology are leveraging Go technology, right? So, the monitoring of Go is quite different from the monitoring of Java. We have different versions of JDK and JRE that we created—one of them was actually recently open-sourced called Dragon Well. You can check out on online, a lot of posts around that. So, we have to make sure the entire stack, from the application, middleware, in the mesh-level, container host, all the way down to compiler has to be monitored quite efficiently. Once anything happened, from the operation side or from the technology side, we have to quickly respond to identify in what layer the error happened. In order for that mitigation to be efficient, we have to make sure we are monitoring every single thing in the stack. As I mentioned, from the application, middleware, host level, all the way down to hardware level, sometimes a failure in hardware will cost the entire failure in our business. It's quite often. So, we have to make sure we are monitoring our own technology in a very good manner. And also imagine monitoring that amount of infrastructure in that massive scale. It's very challenging. I think before 2014, we had a lot of failure in our monitoring infrastructure. This is quite ridiculous, but this is what happened. So, we spent a lot of time to make sure we have the supporting infrastructure ready for that kind of businesses. That's quite important.Emily: Anything else that you'd like to add about either your own experience moving to cloud-native or some observations about how things are going in China in general?Haojie: I think from the strategy perspective, Chinese company or startups from China are doing quite well. It's just the market is quite different. For companies to survive and thrive in the Chinese market, they have to go with the customers, right? So, even though the innovation happens at the same level, the customers are not at the same level from what I see. But overall, I think the trend is quite positive, I think eventually, be it five years, or seven years, or ten years, Chinese companies, Chinese customers will be at the same level as the rest of the world: in the US, in the UK, in Australia, in the rest of the world. I think people are more and more aggressive, and they would like to allocate more and more budget into technology business. They realize the benefits of it, especially in the current outbreak. When people, they cannot go to work, but they still have to do something. The business has to survive. Like, they have to do something in order for the business to survive. So, from the business perspective, how can they build their strong online presence during the outbreak? Is actually quite important. Before the outbreak, I would say, in the retail business, there still some people think about, “Oh, how can we do this in our traditional manner? How can we open as many stores as possible.” They didn't really care about building a store online. From in Taobao or Tmall, [unintelligible] seriously. But during outbreak, people they have to stay at home. They have nowhere to go. But then the business, they still have to pay their employees. So, how can you do that? The only thing is going online. In order to go online, they have to build online infrastructure for their customers, for their employees, for them to work. So, that's quite—honestly, that's one of the trend I'm seeing: that people are paying more and more attention to work remotely, and use software, SAAS software without on-premise deployments. In that case, people, they are able to work wherever they go. Being at home, office, on the road, people are really interested in the benefits of SAAS, of cloud. I think that's something that I'm seeing. I think after this year, definitely the market of SAAS will become better and better because not only the technology is, but the business leaders will understand the value of using Zoom, using Ding Ding, using WeChat, to make sure their employees, they can work anywhere they want.Emily: Well, thank you so much. A couple finishing up questions. First of all, what is an engineering tool that you couldn't do your job without?Haojie: Do you mean, like, just tools for me to do some engineer work?Emily: Yeah. What's your favorite tool, something you just can't imagine working without?Haojie: We have a lot of tools innovated within the company. I don't think I can mention that in this podcast.Emily: Okay. No problem. And then, how can people connect with you if they want to?Haojie: At work, or outside of work?Emily: like on Twitter or on social media.Haojie: Yeah, I had a lot of invitation from LinkedIn, not so much on Twitter because I'm not active on Twitter. But I think people, they get to know me, oftentimes from word of mouth, they got introduced from other friends of mine, they want to understand about the technology adoption in China, especially in the cloud. Yeah, people oftentimes, which me from LinkedIn, that's the primary source.Emily: Well, thank you so much. I really appreciate you taking the time to chat.Haojie: Thank you, Emily.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

Some of the highlights of the show include How containerization enabled Nav to spread roughly 250 virtual machines across multiple environments, while drastically reducing infrastructure spend Travis's thoughts on buying cloud native software tools versus building them, and what engineers should consider during this process The difficulty of finding security solutions that work inside of a cloud-native ecosystem Why companies should expect to encounter unique challenges when migrating to Kubernetes Why companies need to understand their end goal, and determine an overall objective before beginning a migration Travis's must-have engineering tool, and why he can't live without it  Links LinkedIn: https://www.linkedin.com/in/stmpy/ Twitter: https://twitter.com/stmpy  TranscriptAnnouncer: Welcome to The Business of Cloud Native Podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I'm Emily Omier, your host. And today I'm here with Travis Jeppson. Travis is currently at Kasten, but he's also going to talk about his time as a director of engineering at Nav.Travis: At Nav, my role shifted quite a bit while I was there. I started as a software developer, writing Ruby back end applications for them, and then shifted into—actually within a month of being there, they shifted me over to the operational side because I had previous experience working with containerization, and also in infrastructure. So, they quickly moved me over into that realm and from there, I worked there for about a year until they told me, go spin up a team and get things moving. Help us move to containerization. Help us move to a more modern infrastructure and stuff. And so, about a year after that I became a director of engineering to where I had our ops team that had spun up, and then I also acquired both our QA team and our IT team that was there. And then, about a year after that, I ended up acquiring a little bit more than that. So, I ended up with a fair amount of our front end and some of our backend teams as well, and where they moved me into the senior director position. So, a day in the life, towards the end of when I was at Nav was a lot of working with the teams, helping them to do a lot of architectural perspective, and changes, and outlook to where we were trying to get as far as the company is concerned. We were building a product that we could address both first-party customers where they would log in to the Nav website directly, as well as working with partners so that we could issue out Nav functionality to those partners that they could incorporate to their pages as well. And so, we worked very hard to try to segment those two pieces together so that what we were building could be dispersed between both first-party customers and our third-party customers. And so, towards the end of my time there, it ended up being a lot of working within all of engineering to help facilitate those purposes. Then, just about six months ago, I ended up shifting my role over to a company called Kasten. And, Kasten is strictly working within the Kubernetes ecosystem. So, we do data management for Kubernetes based applications, and I am the site lead in Utah for Kasten, and so my day in and day out, a lot is, it's, kind of, all over the place. Sometimes it's working with engineering to help figure out some things going on there, sometimes it's working with brokers to help find office space for it. And sometimes it's dealing with insurance. It ended up being quite dynamic. But overall, I'd say most of my time is really spent more on the engineering side, just from the perspective of having worked at Nav and having been a consumer of a lot of these technologies, I think that they really appreciate my insights that I'm able to give there. So, I end up working, a lot, with the engineers to help facilitate what we're doing.Emily: Sounds like you end up serving as a bridge from having been an end-user. But do you think that there is common miscommunications that happen, or what do those conversations sound like? Why is that experience valuable?Travis: Yeah, so I don't know if it's as much as a miscommunication as much as what are customers looking for? And what are they trying to achieve? And why are they purchasing different software solutions? And what makes sense for them, more than anything. And I think that, having been a consumer of those products, I was more or less on the front lines there. When I was building our operational team at Nav, that was basically what I was doing is trying to figure out what things are we going to spend time on? And what things are we going to build ourselves, or what things do we need to just go find a solution for and bring them in-house? And the funny thing is when I was doing that for Nav is actually when I was introduced to Kasten and to the CEO here. And so, that ended up changing the way my career went. But overall, I think what Kasten—what those conversations really end up becoming is what are customers trying to do, and where are they trying to go?Emily: Yeah, and in fact, that is exactly what I want to talk about more on this podcast. So, tell me a little bit about what your experience at Nav was. What were you looking for? What did you want to prioritize? What was the company hoping to get out of moving to containers?Travis: So, I would say maybe the piece that really facilitated a lot of the progress in that sense was starting to understand our infrastructure spend. And then, to couple with that was also trying to become more agile. More agile in the sense of being able to push on demand, where previous to that we were pushing—you know, when we push our code, we did it on a bi-weekly basis—well, every other week, and it was always very cumbersome. If we have pictures of us in the early days of Nav, where there would be 10 engineers around someone's desk, and they were the one person that was pushing the code into production, just waiting for the other shoe to fall, or waiting for something to happen. And so, when I started doing operational things for Nav, it started addressing those two things. What can we do to help control our infrastructure, and to understand it a little bit better? And how can we also create more of a dynamic infrastructure? Like, Nav is very much a US-based company. And so, the traffic that we're getting onto our website was regional very, very much. And so, there would be periods where it would be very busy, and then there'd be periods where it wasn't. And the way that our infrastructure was designed, and a lot of times the way that they are designed, especially with virtual machines, is that you're building for capacity. You're building to be able to handle that load, and that has to stay there all the time, regardless of whether that capacity is being used or not. And so, that was one of the biggest questions, and that bill was—we were completely in the clouds. We were completely in AWS, but that bill continued to get more and more expensive every month. To the point of where it warranted the executive team to come down and say, “This needs to be fixed. This is going at an outrageous pace, and we need to be able to figure out how to control this.” And so, that's when they came to me and said, “Okay, get a team spun up, and let's figure out how to control this.” And so, I would say that those were some of the big pieces that really drove us to start looking at cloud-native technologies, containerization, and Kubernetes.Emily: And do you think it was successful?Travis: Yeah. So, I do, for a few reasons. And obviously, we learned some lessons along the way, but what we were able to do is, with the infrastructure that we had growing, we were pushing close to 250 virtual machines across two different environments, that being our production, and a development environment that we had. And when we moved to containerization, we were able to not only spin up more environments, but we were able to still decrease that overall spend as far as the infrastructure was concerned. And so, what used to be, I think we had about 100 VMs for our dev environment and then about 150 for our production environment, and that crossed many different pieces from the front end to the back end to—but that was all it was all compute, right? So, none of that even included the database resources that we were using inside of AWS. And we were able to shrink that down to a nine node Kubernetes cluster where three of those nodes were part of the control plane, and then the other six of those were part of the data plane. And then, we ended up spinning up—we were using HashiCorp Vault, and we ended up moving that outside of the cluster just for sanity purposes. But we were able to drastically decrease the footprint of an environment quite a bit, and on top of that, it also correlated to being able to decrease that spend. And so, once we started turning on everything and turning off all of the older infrastructure, it was something that we really liked. And I almost did this, and I wish I would have just taken a snapshot of those couple months within our Amazon bill and, like, posted it on a wall because it almost cut in half to the spend that we had previous to that.Emily: And then, you mentioned some lessons. What are some top three lessons that you learned along the way?Travis: Oh, man. So, I would say, probably the one that bit us the most was actually the telemetry, observability, being able to see what was happening within our environments, especially during a transitional time like that. Now, we did this a few years ago, and so the tools that are out now weren't necessarily as readily available as they were then. I'm not going to name companies, but the company that we were using at that point in time, we came to them and said, “We don't have this visibility, and this is hurting us. This is, kind of, a deal-breaker, and if we can't get this visibility, then we have to look elsewhere.” And they're like, “Well, it's something we've been talking about, but it's not something that we're doing right now.” And it's like, “Okay.” So, we moved on to a solution that was very much in our hands. So, we went from one to where it's like, “Well, we can't rely on a company, maybe we can just deal with it ourselves.” So, we did that, and then we realized, this is actually a lot of work, and it takes a lot of time and a lot of effort. And so, we actually stayed on that one for about a year, and then we moved off of that one, even. And where we found a middle ground to—we wanted control in certain areas, but we didn't want all of the control. And so, then we found a solution that helped us, kind of, meet a middle ground, to where we got the control we wanted, we got the flexibility using the [unintelligible] tools, primarily Prometheus, and then we were able to hand off a lot of the management of the infrastructure for the metric system and telemetry to a vendor, to where we didn't have to worry about that side of it. But we could pump over anything that we wanted, and we could aggregate the data any way we wanted, and that's exactly what we wanted to get out of that. So, that one, I think, was maybe one of the hardest ones just because we put so much work into multiple different iterations of what that eventually became. So, the one that we finally settled on to where it was, kind of, a happy medium is the one that the company is using to this day. It ended up being a much better solution, but it took us two years to figure it out.I would say maybe the next hardest one after that is, it really comes down to just being flexible. Like, you always go in with a plan, and you always assume that that plan is going to work out, and that everything is going to be perfect. And most of the time, that doesn't end up being true. Most of the time you get to the point where you hit something, you hit a snag, or you hit some issue to where you realize that your plan is basically thrown out the window. And there was a point in time to where we, kind of, just stuck to it. We're like, “Okay, just get it to work, just get it to work, just get it to work.” And we kept trying to slam that effort moving forward until we realized that doesn't work. We're burning time. There's no way we're going to get to the point where we need to be, and we're not getting the results that we want. And so, one day I grabbed my team and we sat down and I just said, “Okay, we have this solution in place, but here's the problems with it. Of those problems, how many of them do we absolutely know how to solve right now?” And so, then we looked at the list and we talked about the ones that we knew we could solve, and it's like, okay, of the list that we don't know how to solve, there was a fair amount still leftover and looking at that list, it's like, is it going to be worthwhile to continue addressing this unknown? Or should we adapt our plan to remove that unknown piece of it, so that we can actually get back on track to what makes sense for us and for our end goals. And so, we decided it may be best to scrap that idea and go back to the drawing board. And so, we took two days to where we took an offsite. So, Nav has a corporate apartment, and we just went there and hung out there for two days, and then we whiteboarded and put post-it notes, the giant post-it pad notes, all over the walls, and then we went back to the drawing board. And this was actually around our Kubernetes management layer, what to use to help us manage Kubernetes. And so, the solution we had in place before just wasn't cutting it. And so, we went back and we literally tried everything that was available. We did a Google search, we went to any site that said, “Here's a Kubernetes management layer.” Either just a CLI to help you get the infrastructure spun up, or if it was a GUI, and it had the full management system baked in, or whatever it ended up being. So, we sat down and took that entire list, and then we took a list of the specific outcomes that we needed; we wanted to be able to do X, Y, and Z, and if any of those cannot be done with one of those solutions, then that solution is cut. And so, we seriously just took hour chunks, two-hour chunks of time, and we would divide that list up of different offerings and we started figuring out which ones would work, which ones wouldn't until we got to the point where we literally had one left and that one ended up being the solution that we used moving forward. But being able to take that stop, and being able to readdress our plan and say, “We still want a particular outcome, but the way that we're approaching it is not working. Can we actually readdress this and change our plans in order to still get us the desired outcome?” And after we did that the one time and we got back on track a lot further than we thought, or a lot quicker than we thought we'd be able to, but after we did that the one time, then we started doing that a lot more with a lot of other issues that would arise, we would come back to it. And after we did that one time, that's when we went back to our metrics and said, “Okay, maybe we need to do the same thing with our metrics.” And that's when we shifted that the final time as well. But I think that the second one, then there really was, you have to understand that the important things out of creating a plan are the results of that plan, not necessarily how you get there. And if you're okay with changing the way that you get there, then you can actually achieve the goals of that plan much quicker.Emily: Was there a third lesson that you learned that, sort of, stuck out?Travis: Yeah. I'd say a third lesson is really understanding why you would want to shift over to a cloud-native infrastructure. Because at first, a lot of the reason that it started was we need to do this for cost savings. We want to be able to wrangle in our infrastructure and do all of that stuff. And it's like, okay, that's an okay reason. But at the end of the day, after I hired a team, and even after we did all the work to push everything out, were we in a net positive as far as the cost was concerned? Because there's a lot to incorporate there. And there's a lot of tools, as well, that you have to also consider, a lot of things that we ended up picking up later on that we weren't necessarily using beforehand. And so, while we were able to wrangle in and control the cost of our cloud spend, I don't know that it actually ended up being more cost-effective overall for the company. Now, that's like apples to apples, right? Let's look at our team size, and let's look at our infrastructure costs before and after. If you combine those two things together, were they less? And I don't know if that's true. But what I do think is true, is after going through all of this, we were able to move drastically faster in our pace inside of engineering. After we were done, all of the teams had their own services set up, they were able to deploy on-demand, things became very, very simplified for them. And on top of that, we even, for quite a while had a development environment that was using containerization, and it was very simple to be able to hire someone in, and just run a command, and you would have your development environment up and running. And we even had quite a few people, just the first day that they were on the job, be able to create a commit and contribute, which was a great thing. And so, if you're comparing apples to apples as far as like, what's the cost, then I don't think that that's a good reason to start addressing cloud-native infrastructure. But if you're looking at the overall cost that we had burned in engineering time trying to get development environments set up, or burned in infrastructure, trying to release a new service, or burned in many, many other ways, then we were absolutely net positive in a situation. So, releasing a service before we moved into containerization took about two weeks, and you had about four or five different people involved in order to get that service released. And it was very time consuming, and very costly. Afterwards, after we moved into containerization, it took a matter of minutes. Like, as soon as a developer wanted to release a new service, they just built out the profile in GitLab, and then they would push the code up, and it would go deploy, and everything would be up, and available, and ready to go. And so, our operating costs, I think is really what I'm coming to, is that those drastically changed. And so, at first when I was reporting about our progress and how things were going, and they kept saying, “Well, where's our cost? Where's our cost? Where's our cost?” And so, I kept showing them, “Okay, well, this is what our infrastructure cost before and this is what it cost after.” And while there was some movement there, the thing that I started learning and started reporting back up toward the exec team later on was, “Okay, let me show you what the scenario is now as opposed to what the scenario was then.” And as soon as I was able to start painting a picture as to how much easier and faster we were able to move, they actually quit asking me. They're like, “Okay. We're good. We're sold on the fact this was successful.” And so, I think the third lesson that we learned is that it is important to understand why. And hopefully, you can figure that out before you start everything, but we didn't quite figure it out at that point in time. But we did figure it out soon enough to where we were able to make choices and adapt to that reason why to make it more beneficial for the company in the long run.Emily: And did you feel like there was anything that was lost in translation when you were talking with the executive team and, sort of, giving updates?Travis: Um, no, not really. I've had a few conversations on that and there's a lot of different things that they care about. Usually from an executive team, you want to make sure that, with what's being produced, it's not only going to be able to facilitate product movement, being able to adapt to the changes of our customers, but that we're not doing it at a pace that is, unmaintainable, which is, kind of, where we had been. And so, my conversations with them went from like, “Let's stop looking at this one particular metric that you keep asking me about, to looking at the bigger picture. How much quicker are we able to push code? How much quicker are the product owners able to adapt? How much quicker are they able to take feedback, and apply that, and put that into our product, and be able to version on top of that, and create iterations?” And on top of that, also saying, “Well yes, of course, we still have this one metric that does still matter. But that aside, I look at the overall operations that are happening now as opposed to the way that they were.” And so, for the most part, sometimes it would take a little bit of explanation, and I'm not going to lie, there are a couple times where I had to make powerpoints, and I had to, kind of, lay things out in a different way, but I think that it ended up being so well received that there was even one point where I had to present to the entire company and tell them about our migration, and what happened, and the impact that it had on our development time, and on our infrastructure costs, and everything else. Because through a migration, there are going to be pains with that migration. And after it was all said and done, the executive team wanted the entire company to understand and know why we had to go through those pains, and why it was necessary to move forward. And so, yeah, I ended up talking to the entire company and illustrating to everyone why this was such a monumental move for us. So, I don't think there was a lot lost in translation. I think it actually was very well received.Emily: Tell me a little bit more—you were talking about how you basically were in charge of prioritizing which tools you were going to buy, what you were going to build internally. Tell me a little bit more about both what you were looking for, how you were making that decision, what some of the choices were that you made?Travis: Yeah, for sure. So, let's kind of like, simplify that a little bit. It comes down to—and I was able to give a talk at a couple conferences about this specific thing, but building versus buying. Why would you want to build versus why would you want to just buy? And the result that I came down to, and with a lot of help from reading a lot of information from the internet and also from some mentors, is that the most expensive resource that you have are the people that are on your team, that are working with you. Anything else above and beyond them is actually second. And so, the thing that you want to put your most expensive resource towards are the things that are going to end up evolving your company and progressing your company the fastest. And so, if there are solutions out there that you could use—or that you could build, you could like, eh, I could save a few thousand dollars and we could build this ourselves and blah, blah, blah. It's like, okay, you're looking at the purchase price of that software. But are you looking at the development time and hours that said company has put behind it, and the time and effort that you're going to end up putting behind it? Because I don't know about most companies, but everyone on my team wasn't free. There was a price behind them at the end of the day, and what they were spending that time and effort on, for me, needed to be absolutely necessary for the progression of the company. So, when I started looking at solutions, I started just deciding if we build this ourselves or if we take the time to do an open-source solution and we have to manage this ourselves, what is going to happen into the management of my team? What's going to happen to the overhead of my team? Because I can't just go hire more developers because I want to use a new open-source solution, and I need someone to maintain it. And I don't necessarily have anything against open source, but a lot of times, that's what it ended up coming down to. I think it's very valuable, and I think there are situations where it is the right way to go. Anyway, with those decisions, it really came down to if we implement this and we have to manage it, then that is time that my team is not going to be able to spend on these other projects, which those other projects are more important to me. So, then we would go and look for a vendor or a solution that could help step in and fill in the gap that we were missing. And so, given containerization, cloud-native, Kubernetes, and even Prometheus, all of those are all open source tools. But a lot of times, what we would do is use something that had that open source side to it, so that we could create a standardization, and use that standardization internally, and one that would be monitored and controlled by the community, which helped a ton. But then we have a solution on top of that, that would help bridge the gap between we don't want to manage it ourselves, or we want help managing it, or we want a solution that can step in, use this standardization, but still provide the functionality that we're looking for. And so, that is that's really where—when we were evaluating what we needed to do, then we, kind of, went through that process of can we find a way to standardize around a toolset using open source? And if so, that was great. Then we would take that and say, “Okay, now can we get help with it?” And then, that's typically the route that we would end up going.Emily: Was there anything that you wanted to buy but couldn't? Like, there wasn't something available?Travis: Some of the really hard ones were actually more niche. So, I would say one of the ones that we really struggled with was on the security side. Finding solutions that worked inside of a cloud-native ecosystem as opposed to a virtual machine ecosystem, from security perspectives, were not advancing nearly as fast as some of the infrastructure tools. And so, that side of things was actually very, very complicated and hard to work with. We found some startups that were starting to address this, and we were working with them and we did purchase a solution from one of them, but we kept running into they only cover this piece, they don't cover all these other pieces, because you have intrusion detection and prevention, you also have network monitoring, and you need to have forensics running against your logs, and you also need runtime protection when your environment is up and going, and then you had the virus protection, too. And so, there wasn't anywhere that we could go to just say we need a full and complete security solution, and we want it to start now; go. So, like, being able to facilitate that part of our infrastructure was actually very complicated, and we ended up having to poke around, and use some antiquated services, and we tried to update to facilitate our needs, and some of them—I hate to say this, but some of them were even just to check a box, because within the containerized world as opposed to VM world, you're not going to get the same kind of coverage. A big one, really, is virus protection. If you look at—even if you go to Docker's website and you read about virus protection, the only way to scan a Docker environment for viruses is to shutdown Docker, which doesn't work. You can't ever shut down Docker, because that's your entire ecosystem, so you just can't do it. But you can use immutability. You can use the fact that you created your images yourself. You can sign your images to verify that they came from a trusted source and stuff like that. And so, we ended up having to piecemeal a fair amount of that together. So, of anything, I would say that's the one thing that you can't just go out and buy right now.Emily: I realized that we've talked a lot about pain points, but I also wanted to ask about pleasant surprises. Was there anything along the journey that went much better, was much easier than you expected?Travis: One big one was actually the overall outcome, because we went in with one perspective of like, let's save money on infrastructure, but then realizing, through the journey, how much simpler a lot of the process became, especially for developers was a very, very pleasant surprise. And on top of that, even the developer adoption of it. I know that sometimes—and I hear a lot that it doesn't go very well for some companies, and developers don't want to learn a new technology, or whatever else, but we put a lot of time and effort upfront to educate our developers. And the adoption actually went really well for us, and that was also a very pleasant surprise. I had my defenses up, I was ready to go to war and be like, “This is happening, regardless of whether you want it or not.” And I didn't ever have to do that. As soon as we sat down and we showed them the differences in the workflow and how much quicker it was to be able to adapt and make changes to their services, as well as push new services. They were just like, “Sign me up. I'm ready to go. This is way better than anything we're doing right now.” And so, that, for me, was also another very pleasant surprise.Emily: Can you tell me a little bit more about how this experience informs your role now at Kasten?Travis: Yeah, so I would say there's a few things. I'd say probably the primary one is having gone through this with a company, and watching the migration, and watching all of the different struggles and the different problems you have to solve to adapt a containerized workflow has definitely influenced how I approach customers working with Kasten, but also engineering, and also the executive team as well here. And working with them, and helping them understand the things that matter to the things that didn't matter. And the things that are going to affect customers more than, maybe, they would think as well, just from my own experience and having to deal with it.Emily: Give me some examples. What are some things that do matter versus don't matter? And where do you think there's sometimes a disconnect?Travis: Yeah. So, you know, I'll be frank here. Kasten is definitely a Kubernetes based vendor, right? And I remember there were there a couple times—and I don't know if I want my CEO to hear this, but if he does, it's okay. There were times where I remember going to KubeCon conferences or different container-based conferences, and looking at the vendors, and just thinking, I don't know if I would ever want to do that. That never makes sense to me. But when you go up and you go talk to a vendor, you go discuss the product that they're building or whatever else, they like to show you all the flashy things, the things that really make them stand out that they're like, “Hey, we can take this process and make it crazy simpler,” or, “We can do this thing for you. We can add in this service mesh, and you're going to get all of this telemetry out of your system,” and all this craziness, or, “We can build an underlying data volume so that you can have stateful applications inside of Kubernetes. And we'll do all of this,” and it's like, every single time—not every time—but most of the time when I would talk to them, and they would give me their flashy approach and tell me, “Hey, this is all the craziness you can do.” Like, I'd go back and I talked to my team and say, “Does this make sense for us? Yeah, this is cool, but the amount of work we're going to have to put in in order to adapt that or to even use that and leverage it, what is it going to buy us? What advantage is it going to give us over what we're doing right now?” And a lot of times, it didn't end up giving a lot of advantage. It didn't make a huge difference. Now, being at one of those vendors, one of the big differences, and this was, kind of, a long-running thing with me and Niraj, our CEO, we ended up having a ton of conversation around this, but the big difference that I see with Kasten, and one thing that I continue to push here, and I told him time and time again, this is why I joined this team, is because Kasten, while they have their—we do data management, we can do backups, we can do recoveries, we have data mobility, right? The thing about Kasten is it actually lets you attack a problem the way that you want to attack it, and that's stateful applications. And a lot of times, you're going to go look in how to run stateful applications and you're going to get this big long—oh, you need a data layer. You need to be able to have your data be—to migrate across availability zones, or across regions to be able to do this. And that adds so much complexity, where at the end of the day, how often does the data infrastructure actually go bad? We have these cloud providers now, and they have spent a lot of time on making sure that their data infrastructure is pretty robust. Why aren't we just using those? Why aren't we just using those and then accounting for disasters or issues coming up around that? And that's actually the way that Kasten has approached it is, you can use your data, you can use whatever you want, and we're just here as a tool to help you facilitate that process. And so, kind of, getting back to your question of, like, what I really feel like makes a difference in this space is you have to understand what that customer is trying to do. And you have to understand how to facilitate their end goals and what they want. It's not about coming in and saying, “I can help you do all of this stuff.” And it's like, “Okay, but what does all that stuff get for me? Because really, the problem that I'm dealing with right now is, is x, y, and z.” And as a vendor, and as talking to customers, it's more about helping them. It's more about solving their problems, allowing them to focus on the tasks that are going to be more monumental for their company, instead of focusing on tasks that aren't. Not everyone is going to be a data management company, and rightfully so. You have other things and important things to be paying attention to. So, let me come in and help you address that need without causing you a lot of pain, and a lot of hardship, to be able to just come in and use a solution and move on, but using a solution in your environment in your way to where I'm helping solve a problem, instead of helping create another problem, for what benefit?Emily: Do you think that most companies that are on this journey are essentially trying to solve similar problems?Travis: And which side? On the vendor side or on the consumption side?Emily: Oh so, like Nav, the end-users. Do you think essentially any company that's moving to containers, that's moving to Kubernetes, are they going to run into essentially the same set of problems?Travis: You know, no, I don't think so. I think that each journey is going to be a little bit different, and it's going to cause different problems. Because if you take a company like Nav to where we had to be PCI compliant. We had different regulations that we had to abide by. And that caused the solution set that worked for us to be drastically different from the company that may not have those issues. A Kasten, for example. We're still very much, even though we have a product in the Kubernetes space, we're also still a consumer of those technologies as well. But our problems, and the things that we're addressing are monumentally different than the ones that we're addressing in Nav. And then, you also get into a lot of questions around what are the things that are important? Because sometimes your SLA is the most important thing and that will cause your solution to differ. Sometimes your SLA can waver a little bit, but you absolutely have to provide a different need for your customers. And so, while all of these tools kind of look the same—like if you're looking out in the morning, and you look at the freeway and you see all of these people that are in vehicles, and they're all traveling somewhere. Sometimes these people are moving large products. Sometimes these people are only moving themselves. But sometimes when only moving yourself, sometimes you're going to work, but sometimes you're going to play. The reason we all acquired a vehicle is because it helps facilitate that process though our need for that vehicle is drastically different. And I think that in cloud-native and Kubernetes it's the same thing. The needs are so varying and so different, but yet you can use similar tools to help facilitate them in different ways.Emily: Do you have one or two examples of how Nav and Kasten have different needs?Travis: Yeah, absolutely. So, I would say that one of the foremost concern that Nav is absolutely security. With the PCI regulation and everything else, protecting the identity of our customers, protecting the data for the company, it is a must. There is no if, and, or but about it, it has to happen. And the way that we ended up using Kubernetes had to facilitate that as well. So, like I had mentioned earlier that we had six nodes that we were using for the compute side of things. The reason we had six is because we had to create a logical segregation within those nodes to protect the services running on them. So, we would only allow back end services that had access to confidential information on a subset of nodes. And we wouldn't allow anything else to run there. So, you could run your front end service and a PCI compliant service on the same node, ever. But if you look at what we do at Kasten, we are running quite a few environments, and being in the Kubernetes ecosystem, and being a vendor there, we end up having to work with every single cloud vendor out there. We're getting certified with all of them—I'm working with a few right now. But we have certifications within AWS, and Google, and Azure, and we also are working with VMware Pivotal. So, it's across the board, and that's something that's been crazy important for Kasten is being able to have that multi-cloud experience. Being able to take data and move it from one environment to another, whether on-premise or off-premise. And so, that being one of our primary needs at Kasten. And so, we build around need, whereas Nav builds around security.Emily: Excellent. Anything else that you'd like to add that I maybe didn't think to ask, didn't know to ask?Travis: Oh, that's an open-ended question. I would say one thing, if nothing else, I am very much in agreement with the fact that almost every company out there is someday going to end up hearing the words Kubernetes, just the same as they ended up learning VMware associated with virtual machines and stuff. It is. And there's a reason behind that, but the reason behind, I don't think is as important as understanding when and why it makes sense for you to start adapting and adopting those technologies because for every company, just as we've been talking about, it ends up being different, drastically different. And I think that it is very important to understand your end goal, and getting into it. Look at the overall outcome of what you're trying to achieve and use that to help drive the movement forward. Because if you look at a lot of—and the reason I say this is because if you look a lot of—I don't know if you want to call them fads or movements within technology—you look at Agile, you look at microservices, you look at a lot of other—even cloud-native. A lot of times people look at that and they're like, “Hey, look at all these good things that come out of this. And they don't typically look at what the trade-off is. Because in a microservice infrastructure, if you've got two developers, then why do you need seven different microservices? It might actually be anti—or working against what your workflow is like. And I think that even containerization is that same way. There are situations to adapt your workflow to start using those technologies, and I think sometimes there aren't. And I think that it is something that—you need to go into it and understand what the outcome is. And if you understand that outcome, then when you engage and start using those technologies, then every decision you make will be to help drive towards that outcome. And I think that that'll help you get through it a lot quicker and a lot easier, and it'll also help you just get rid of a lot of the other noise that's out there. And it'll help you, kind of, get specifically to the point of things that make sense to you and to your company so that you're able to get to that outcome and continue to drive forward and continue to help your company become successful.Emily: All right, one last question. Actually two last questions. What is a can't-live-without engineering tool for you?Travis: Oh, man. There's probably so many. But for me, probably one of—let me think. Is this, like, a tool that I use on my computer, or is this maybe something I use in the process? Or any of the above?Emily: Any of the above. I mean, you could tell me Slack or something, anything that you can't imagine doing your work without.Travis: Yeah, Slack, I think, actually helps deter me from getting work done. [laughs] But I would say for me, the one I cannot live without is a pipelining system. And for me, a lot of times has come down to GitLab. I really love the workflow in GitLab, but any pipelining system, really, is the must-have because if you can get into your process of getting code from a developer's laptop to automating how that gets into an environment, that process saves so much time and so many resources, that I don't even care which system you end up using. But just having that process, having that CI/CD system, I think is an absolute must-have.Emily: Excellent. And then, how can people connect with you?Travis: Yeah, I'm on some social media, I don't do it all, but I'm definitely on LinkedIn. You can just search for me by name on there. I'm also on Twitter. My callsign there is @stmpy. It's kind of a long story, but my friends make fun of me because my legs are short, and so they used to call me Stumpy. So, it's Stumpy without the U, so just S-T-M-P-Y on Twitter. And I think that those are probably the two best ways to get a hold of me.Emily: Excellent. Well, thank you so much for chatting.Travis: Yeah, thank you. I really appreciate it.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

Some of the highlights of the show include Why Adform decided to move to a cloud native architecture and Kubernetes specifically  Who was the driving force behind the move to Kubernetes? Was the switch purely an engineering decision or did it involve people outside of engineering? Positive and less positive surprises that come with switching to cloud native  Organizational and technical problems Edgaras has faced What's next for Adform on their cloud journey Links LinkedIn: https://www.linkedin.com/in/apsega/ Twitter: https://twitter.com/Apsega TranscriptAnnouncer: Welcome to The Business of Cloud Native Podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.Emily: Welcome to The Business of Cloud Native. I'm Emily Omier, your host. And I'm here today with Edgaras Apsega, lead IT systems engineer at AdForm. Edgaras, what I'd like to do is just start out with you introducing yourself.Edgaras: I'm Edgaras. I'm working in the Adform. For anyone that doesn't know, Adform is one of the leading advertising technology companies in the world, and provides the software used by buyers and sellers to automate digital advertising. And, probably one of the most interesting parts of our solution stack is demand-side platform that has real-time bidding. And, what it means is that when that page is loading for some kind of internet users, behind the curtain, there's actually a bidding process that takes place for the placeholders to show ads. So, basically, you're doing low latency stuff. And, in Adform, I'm a lead systems engineer for the cloud services team. Our team consists of eight people, and we are providing private cloud storage, load balancing, CDN, service discovery and Kubernetes platforms for our developers that are in [00:01:36 unintelligible] production services.  So, to better understand the scale that our team is working on, first of all, you can see that we are not using public cloud and we have our own private cloud that has six regions, more than 1500 physical servers, and there are more than 4000 [00:01:55 unintelligible]. And, for Kubernetes, we have seven clusters, more than 50 physical machines and around 300 constantly running [00:02:05 pods].  So, we can say that we prefer bigger clusters with bigger resources sharing pools. And you asked, how do I spend my daily work, right?Emily: Yeah. So, when you get into the office or—right now you're not going into the office—get into your table or your [laughs] home office, what are the first couple things that you do, or…Edgaras: Yeah, so, when I arrive at work, or, like, at these times, just get off the showers straight into work desk, [laughs] actually, I'm most productive in the mornings and evenings. So, in the mornings, when I go to my work desk, I try to do as much as I can. My sprint plan tasks, and then I scroll through the Slacks, emails, and the tickets assigned to me because we have a development team in another region. So, instantly in the mornings, we have some kinds of support tasks that we need to do.Emily: Let's go ahead and talk about what this is all about, the business of cloud native, and tell me a little bit about why Adform decided to move to a cloud native architecture. Why did you decide to use Kubernetes, for example?Edgaras: I'd say, actually, there were two parts. At first, we moved from traditional and, let's say, old-fashioned monitoring solutions to Prometheus, and its integration with service discovery solved lots of operational time for constantly managing and configuring monitoring and alerting for our, quite often, changing infrastructure. And the second part is the adoption of Kubernetes and all of the together coming parts like continuous integration and delivery. So, why we moved to this kind of architecture? It was because the biggest pain points for developers were to maintain actually their virtual machines. And rolling out new software releases in an old-fashioned way, took just lots of time for new software releases to reach production. So, we were looking at the new solutions that were available in the market, and Kubernetes was actually one of them. So, after successful proof of concept, we have selected it as our main application scheduler and orchestration tool.Emily: What would you say was, like, the business value that you were hoping to get out of Kubernetes, out have the ability to release software faster, for example?Edgaras: Yeah.  So, actually, we wanted to remove the operational time from our developers so that they could spend more time coding without taking care of all of the infrastructure surrounding parts, like the application operating system management, [00:04:58 unintelligible] monitoring, alerting, logging, and so on. So, basically what, I'm saying is that the business value was for the developers to be able to ship features faster, and have a more stable platform that scales application [00:05:15 unintelligible] as well. So, in addition to that, we have a big research department, and the research department always wanted us to have a dynamic environment where they could just launch an applications around some research models, and then shut it down.  So, I believe that was the business value.Emily: Who in the organization do you think was motivating, or driving the move to Kubernetes?Edgaras: I'd say, actually, it was more like the operation engineers, because the developers ended taking care of their environment virtual machines. They don't know much about it, but they still have to look after it, and constantly asking us for help. And we wanted to have this operational stuff only in our hands and for the developers to run only the code.  So, I believe, yeah.Emily: To what extent was the move to Kubernetes, or to cloud native in general, just purely an engineering decision? Or did it involve other people outside of engineering?Edgaras: Well, it wasn't only the engineering decision, because we had to take it to the upper levels, just to show this new cloud native, the modern way of developing and running applications. So, the upper management level had to invest time for us to move to microservices oriented architecture and so on. So, basically, we had to show that with a little bit of time investment we can gain lots of benefits, like faster code deploys. So, we are taking the operational work from developers, and developers, when they're releasing their applications, they have full stack monitoring, logging, and they don't need to do any of the operational tasks.Emily: How difficult was it to have this conversation? Do you feel like the upper management, did they understand the value?Edgaras: Yeah, it was kind of hard, because nobody wants to invest time to write the code. And, as we are a software company, we always need to write new features. But, once we showed a good example, when investing not so much time, we have those kinds of benefits, then it was quite easy to change the mindset of upper management.Emily: And, how important do you think this was for Adform?Edgaras: I think it was very important because now what we see, we have, basically, until now we had only dozens of deployments per day. Now with Kubernetes, we have more than 500 deployments per day, which is a big number for us, and this means that we are making releases more faster.Emily: Tell me a little bit about any surprises that you had as you were moving to Kubernetes, as you were moving to microservices. Surprises, and I'm interested in hearing both about surprises that were positive and surprises that were less positive.Edgaras: Probably the biggest surprise for us, for our thing was just how amazing the communities. When we faced any kind of issue, most of the time there's simply a GitHub issue that's described fixes or workarounds. You can always get an answer for questions in Slack. I remember when we had actually an issue with Kubernetes and persistent storage, and in Kubernetes Slack channel, one engineer from a company that provides storage solutions, he just provided me lots of information and several ways of tackling the problem that we were facing, actually, and that really stood me out. And, actually, we just recently started a cloud native [00:09:28 unintelligible] meetup group, with which we gather lots of folks for knowledge sharing presentations, and discussions afterwards, and it feels like the community is really strong and is eager to share their knowledge freely. So, that really amazes me about this journey.Emily: What about some less positive surprises?Edgaras: Yeah.  So, moving to Kubernetes from virtual machines world, first of all, to change the developers mindsets about the resources utilization, I'd say. Because coming to Kubernetes world, developers need to set containers, resource limits, and often they're setting amounts similar to what they had on virtual machines with other services like monitoring, log shippers, and so on. And we see on Kubernetes, that for some applications, the resource usage is very low, but the requests of CPU is quite high, so we're still monitoring resource utilizations, and communicating with teams to lower them. Because one good example would be that while general CPU usage in whole environment is around 30 percent, we're constantly reaching fully CPU requested Kubernetes nodes, and other teams are facing deployment issues. The nodes are full. And, probably I should share that we had one interesting example that when we have migrated a service from virtual machines to Kubernetes, that service was using nine virtual machines with 16 CPUs each. And then they migrated to Kubernetes with all of the built in monitoring tools and so on. They have noticed that for the current workloads, they only needed six CPUs.  So, instead of nine virtual machines with 16 CPUs, they only needed six CPUs, and so they returned just a lot of resources to the shared pool.Emily: Wow.Edgaras: Yeah, that's amazing. And another big pain point is always the security.  So, we're struggling a lot with the security part at the moment. And, as you may know, often security is focused on the IP address based identity, and in Kubernetes, those IPs are always changing and you can't rely on the fact that a specific IP address is tied to a particular service. So, yeah, so all the cloud native mindset needs to be changed, not only for the developers and operational engineers, but for the security engineers as well.Emily: Where would you say you are in the cloud native transition? Are you there, have you done everything that you can, or are you somewhere on the journey?Edgaras: I'd say, we are more than halfway through because simply [00:12:31 unintelligible] have some legacy applications that need to be rewritten so those can run in a containerized workloads. And for our critical and user-facing applications, we're still have lots of discussions with our security team about how the infrastructure and all of those access control things should look like.  So, yeah, at first, load services owners were looking at Kubernetes from a distance, and after a few successful migrations, more and more high load services are scheduled to do migrations. But in terms of the legacy applications, business still doesn't invest money, because it's not a critical application.  So, I think they're going to stick for a while on that kind of phase.Emily: Do you think that that's okay? Would you rather invest the money in—is there any disadvantages to keeping these legacy applications around?Edgaras: I think the one point or another, they'll be completely rewritten or terminated for good. So, actually, it depends. I think, if it's not business critical, then probably it's okay. But if it is business critical, then I'd say migrate it to Kubernetes to have the self-healing infrastructure that scales just beautifully.Emily: When you think about some of your pain points, do you think of them as technical issues? Do you think of them as, sort of, organizational issues? What are some examples of both organizational and technical problems that you've had?Edgaras: Yeah.  So, regarding the resources of the organization on how the developers are setting the resources, I think that it's kind of organizational issue. We did some Kubernetes trainings internally, and developers are always asked us to [00:14:25 unintelligible] one more time, those trainings because they're interactive. And it's a [00:14:29 hike] now. But there's always new developers coming and you still have to share your knowledge about how the resources should be implemented, how they should set the requests, or how they should set their limits and so on. Regarding the security around the Kubernetes, I think that this field is quite new, and I remember the last KubeCon in Barcelona, there were lots of buzz about the Kubernetes security and just shows that this, kind of, new field, and everybody needs information about it.Emily: I think that you're right. It seems like both of those things are really almost skill gap issues. Do you think that there any real technical problems? So, things that the technology isn't quite there, or it's not really a problem with the way that your team members are thinking about something, or that they don't have the skills.Edgaras: Yeah, so actually, about Kubernetes. As I mentioned, we're running Kubernetes on bare metal. And the technical stuff with Kubernetes is that it's actually first class citizen for public clouds, but when you're trying to run it on bare metal, there's some issues that you cannot expose services with, let's say, Type LoadBalancer, you cannot have quite easily service mesh that talks not only within Kubernetes cluster, but also outside Kubernetes cluster with your virtual machines because you need to have BGP mesh, and that's your current network equipment. And there's a lot of technical issues, actually, around running Kubernetes on bare metal.Emily: I think that that's really interesting. What are you doing to make it easier to run Kubernetes on bare metal? Or are you? Is that something that you're investing time and money into making easier?Edgaras: Yeah.  So, for running Kubernetes on bare metal, actually, we're not using any of the automation that's provided publicly. So, we took parts of Kubernetes and automated those parts by ourselves. And we have those three data centers close to each other, connected via that fiber and we have one logical Kubernetes cluster across three data centers. And for the services to be exposed as a, let's say, Type LoadBalancer, they do have some workarounds that will put in a custom load balancers in front of the Kubernetes cluster.Emily: Is that something that you would hope that the community would do more of, or do you feel like you've got a pretty good handle on it at the moment?Edgaras: Probably, I would like to see this addressed by the community more because everything that's being built for Kubernetes, it seems that it's being built for the public clouds, but not for the bare metal.Emily: This actually, sort of, leads me to a future-oriented question. Where do you see, sort of, your next steps on the cloud journey as being?Edgaras: Yeah, so service mesh. [laughs] everyone's talking about the service mesh. Probably, you'll have—actually we have plans to look at it and to do a proof of concept but, as I mentioned before, there are some technical issues if you want to make services talk between Kubernetes and virtual machines services between.  So, looks like a journey.Emily: What do you hope to get out of completing the journey?Edgaras: So, service mesh, I believe, would provide this circuit-breaking, and service discovery would take us to another level. And so, I believe that when we end this journey, the scalability of our platform should improve as much as platform stability, and for the developers it would remove the operational tasks completely.Emily: And, what do you think that that would mean in terms of the business?Edgaras: You know, business is always looking for two things: to have stable platform for our customers, and to run infrastructure at the lowest possible costs. So, I think that the Kubernetes with container orchestration and auto-scaling solves the first problem, while the nature of shared resources in Kubernetes helps teams to achieve lower infrastructure costs.Emily: And are you pretty happy with where you are now, with, sort of, the results that you've gotten at this stage? Would you do it over again?Edgaras: Definitely. As I mentioned before, before Kubernetes, we had like, only tens or twenties deployments per day, now we have 500 deployments per days. And the developers are even happy with more features that they're getting. They're getting feature branch deployments, and green/blue deployments, and so on. So, for us operational engineers, there's less work to maintain everything because everything comes standardized. And for the developers, it's less operational work, and the just develop new service or feature and just push it.Emily: Anything else that you want to add? And then I have a couple, sort of, closing questions to ask as well, but before then, is there anything else that you want to add?Edgaras: What I'd like to add is that with Kubernetes, probably the biggest issues is with security because Kubernetes is kind of new thing. And seems like security stuff around the Kubernetes is one step behind.  So, what I'd like to see is more solutions from the security perspective around Kubernetes.Emily: So, just sort of in closing, I have a couple of fun questions. The first one is, what do you think, for you personally, and possibly organizationally, what's your can't live without engineering tool?Edgaras: Prometheus because if you don't have monitoring, then it's like, flying the plane without any of dashboards, so you'll crash soon.Emily: Excellent. And then how can people connect with you?Edgaras: LinkedIn is always open.Emily: Are you on Twitter?Edgaras: Yes, I am.Emily: Fabulous. I think we can go ahead and wrap it up there, and thank you so much for chatting.Edgaras: Cool. Thanks for having me.Announcer: Thank you for listening to The Business of Cloud Native podcast. Keep up with the latest on the podcast at thebusinessofcloudnative.com and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.This has been HumblePod production. Stay humble.

We are joined by Ellen Körbes for this episode, where we focus on Kubernetes and its tooling. Ellen has a position at Tilt where they work in developer relations. Before Tilt, they were doing closely related kinds of work at Garden, a similar company! Both companies are directly related to working with Kubernetes and Ellen is here to talk to us about why Kubernetes does not have to be the difficult thing that it is made out to be. According to her, this mostly comes down to tooling. Ellen believes that with the right set of tools at your disposal it is not actually necessary to completely understand all of Kubernetes or even be familiar with a lot of its functions. You do not have to start from the bottom every time you start a new project and developers who are new to Kubernetes need not becomes experts in it in order to take advantage of its benefits.The major goal for Ellen and Tilt is to get developers code up, running and live in as quick a time as possible. When the system is standing in the way this process can take much longer, whereas, with Tilt, Ellen believes the process should be around two seconds! Ellen comments on who should be using Kubernetes and who it would most benefit. We also discuss where Kubernetes should be run, either locally or externally, for best results and Tilt's part in the process of unit testing and feedback. We finish off peering into the future of Kubernetes, so make sure to join us for this highly informative and empowering chat! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://www.notion.so/thepodlets/The-Podlets-Guest-Central-9cec18726e924863b559ef278cf695c9 Guest: Ellen Körbes https://twitter.com/ellenkorbes Hosts: Carlisia Campos Bryan Liles Olive Power Key Points From This Episode: Ellen's work at Tilt and the jumping-off point for today's discussion. The projects and companies that Ellen and Tilt work with, that they are allowed to mention! Who Ellen is referring to when they say 'developers' in this context. Tilt's goal of getting all developers' code up and running in the two seconds range. Who should be using Kubernetes? Is it necessary in development if it is used in production? Operating and deploying Kubernetes — who is it that does this? Where developers seem to be running Kubernetes; considerations around space and speed. Possible security concerns using Tilt; avoiding damage through Kubernetes options. Allowing greater possibilities for developers through useful shortcuts. VS Code extensions and IDE integrations that are possible with Kubernetes at present. Where to start with Kubernetes and getting a handle on the tooling like Tilt. Using unit testing for feedback and Tilt's part in this process. The future of Kubernetes tooling and looking across possible developments in the space. Quotes: “You're not meant to edit Kubernetes YAML by hand.” — @ellenkorbes [0:07:43] “I think from the point of view of a developer, you should try and stay away from Kubernetes for as long as you can.” — @ellenkorbes [0:11:50] “I've heard from many companies that the main reason they decided to use Kubernetes in development is that they wanted to mimic production as closely as possible.” — @ellenkorbes [0:13:21] Links Mentioned in Today’s Episode: Ellen Körbes — http://ellenkorbes.com/ Ellen Körbes on Twitter — https://twitter.com/ellenkorbes?lang=en Tilt — https://tilt.dev/ Garden — https://garden.io/ Cluster API — https://cluster-api.sigs.k8s.io/ Lyft — https://www.lyft.com/ KubeCon — https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019/ Unu Motors — https://unumotors.com/en Mindspace — https://www.mindspace.me/ Docker — https://www.docker.com/ Netflix — https://www.netflix.com/ GCP — https://cloud.google.com/ Azure — https://azure.microsoft.com/en-us/ AWS — https://aws.amazon.com/ ksonnet — https://ksonnet.io/ Ruby on Rails — https://rubyonrails.org/ Lambda – https://aws.amazon.com/lambda/ DynamoDB — https://aws.amazon.com/dynamodb/ Telepresence — https://www.telepresence.io/ Skaffold Google — https://cloud.google.com/blog/products/application-development/kubernetes-development-simplified-skaffold-is-now-ga Python — https://www.python.org/ REPL — https://repl.it/ Spring — https://spring.io/community Go — https://golang.org/ Helm — https://helm.sh/ Pulumi — https://www.pulumi.com/ Starlark — https://github.com/bazelbuild/starlark Transcript: EPISODE 22 [ANNOUNCER] Welcome to The Podlets Podcast, a weekly show that explores cloud native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision-maker, this podcast is for you. [EPISODE] [0:00:41.8] CC: Hi, everybody. This is The Podlets. We are back this week with a special guest, Ellen Körbes. Ellen will introduce themselves in a little bit. Also on the show, it’s myself, Carlisia Campos, Michael Gasch and Duffie Cooley. [0:00:57.9] DC: Hey, everybody. [0:00:59.2] CC: Today’s topic is Kubernetes Sucks for Developers, right? No. Ellen is going to introduce themselves now and tell us all about what that even means. [0:01:11.7] EK: Hi. I’m L. I do developer relations at Tilt. Tilt is a company whose main focus is development experience when it comes to Kubernetes and multi-service development. Before Tilt, I used to work at Garden. They basically do the same thing, it's just a different approach. That is basically the topic that we're going to discuss, the fact that Kubernetes does not have to suck for developers. You just need to – you need some hacks and fixes and tools and then things get better. [0:01:46.4] DC: Yeah, I’m looking forward to this one. I've actually seen Tilt being used in some pretty high-profile open source projects. I've seen it being used in Cluster API and some of the work we've seen there and some of the other ones. What are some of the larger projects that you are aware of that are using it today? [0:02:02.6] EK: Oh, boy. That's complicated, because every company has a different policy as to whether I can name them publicly or not. Let's go around that question a little bit. You might notice that Lyft has a talk at KubeCon, where they're going to talk about Tilt. I can't tell you right now that they use Tilt, but there's that. Hopefully, I found a legal loophole here. I think they're the biggest name that you can find right now. Cluster API is of course huge and Cluster API is fun, because the way they're doing things is very different. We're used to seeing mostly companies that do apps in some way or another, like websites, phone apps, etc. Then Cluster API is completely insane. It's something else totally. There's tons of other companies. I'm not sure which ones that are large I can name specifically. There are smaller companies. Unu Motors, they do electric motorcycles. It's a company here in Berlin. They have 25 developers. They’re using Tilt. We have very tiny companies, like Mindspace, their studio in Tucson, Arizona. They also use Tilt and it's a three-person team. We have the whole spectrum, from very, very tiny companies that are using Docker for Mac and pretty happy with it, all the way up to huge companies with their own fleet of development clusters and all of that and they're using Tilt as well. [0:03:38.2] DC: That field is awesome. [0:03:39.3] MG: Quick question, Ellen. The title says ‘developers’. Developers is a pretty broad name. I have people saying that okay, Kubernetes is too raw. It's more like a Linux kernel that we want this past experience. Our business developers, our application developers are developing in there. How would you do describe developer interfacing with Kubernetes using the tools that you just mentioned? Is it the traditional enterprise developer, or more Kubernetes developers developing on Kubernetes? [0:04:10.4] EK: No. I specifically mean not Kubernetes developers. You have people work in Kubernetes. For example, the Cluster API folks, they're doing stuff that is Kubernetes specific. That is not my focus. The focus is you’re a back-end developer, you’re a front-end developer, you're the person configuring, I don't know the databases or whatever. Basically, you work at a company, you have your own business logic, you have your own product, your own app, your own internal stuff, all of that, but you're not a Kubernetes developer.It just so happens that if the stuff you are working on is going to be pointing at Kubernetes, it's going to target Kubernetes, then one, you're the target developer for me, for my work. Two, usually you're going to have a hard time doing your job. We can talk a bit about why. One issue is development clusters. If you're using Kubernetes in prod, rule of thumb, you should be using Kubernetes in dev, because you don't want completely separate environments where things work in your environment as a developer and then you push them and they break. You don't want that. You need some development cluster. The type of cluster that that's going to be is going to vary according to the level of complexity that you want and that you can deal with. Like I said, some people are pretty happy with Docker for Mac. I hear all the time these complaints that, “Oh, you're running Kubernetes on your machine. It's going to catch fire.” Okay, there's some truth to that, but also it depends on what you're doing. No one tries to run Netflix, let's say the whole Netflix on their laptop, because we all know that's not reasonable. People try to do similar things on their mini-Kube, or Docker for Mac. Then it doesn't work and they say, “Oh, Kubernetes on the laptop doesn't work.” No. Yeah, it does. Just not for you. That's a complaint I particularly dislike, because it comes from a – it's a blanket statement that has no – let's say, no facts behind it. Yeah, if you're a small company, Docker for Mac is going to work fine for you. Let's say you have a beefy laptop with 30 gigs of ram, you can put a lot of software in 30 gigs. You can put a lot of microservices in 30 gigs. That's going to work up to a point and then it's going to break. When it breaks, you're going to need to move to a cloud, you're going to need to do remote development and then you're going to Go to GCP, or Azure, or Amazon. You're going to set up a cluster there. Some people use the managed Kubernetes options. Some people just spin up a bunch of machines and wire up Kubernetes by themselves. That's going to depend on basically how much you have in terms of resources and in terms of needs. Usually, keeping up a remote cluster that works is going to demand more infrastructure work. You're going to need people who know how to do that, to keep an eye on that. There's all the billing aspect, which is you can run Docker for Mac all day and you're not going to pay extra. If you leave a bunch of stuff running on Google, you're going to have a bill at the end of the month that you need to pay attention to. That is one thing for people to think about. Another aspect that I see very often that people don't know what to do with is config files. You scroll Twitter, you scroll Kubernetes Twitter for five minutes and there's a joke about YAML. We all hate editing YAML. Again, the same way people make jokes about using about Kubernetes setting your laptop on fire, I would argue that you're not meant to edit Kubernetes YAML by hand. The tooling for that is arguably not as mature as the tooling when it comes to Kubernetes clusters to run on your laptop. You have stuff like YAML templates, you have ksonnet. I think there's one called customize, but I haven't used it myself. What I see in every company from the two-person team to the 600 person team is no one writes Kubernetes YAML by hand. Everyone uses a template solution, a templating solution of some sort. That is the first thing that I always tell people when they start making jokes about YAML, is if you’re editing YAML by hand, you're doing it wrong. You shouldn't do that in the first place. It's something that you set up once at some point and you look at it whenever you need to. On your day-to-day when you're writing your code, you should not touch those files, not by hand. [0:08:40.6] CC: We're five minutes in and you threw so much at us. We need to start breaking some of this stuff down. [0:08:45.9] EK: Okay. Let me throw you one last thing then, because that is what I do personally. One more thing that we can discuss is the development feedback loop. You're writing your code, you're working on your application, you make a change to your code. How much work is it for you to see that new line of code that you just wrote live and running? For most people, it's a very long journey. I asked that on Twitter, a lot of people said it was over half an hour. A very tiny amount of people said it was between five minutes and half an hour and only a very tiny fraction of people said it was two seconds or less. The goal of my job, of my work, the goal of Tilt, the tool, which is made by the company I work for, also called Tilt, is to get everyone in that two seconds range. I've done that on stage and talks, where we take an application and we go from, “Okay, every time you make a change, you need to build a new Docker image. You need to push it to a registry. You need to update your cluster, blah, blah, blah, and that's going to take minutes, whole minutes.” We take that from all that long and we dial it down to a couple seconds. You make a change, or save your file, snap your fingers and poof, it's up and running, the new version of your app. It's basically a real-time, perceptually real-time, just like back and when everyone was doing Ruby on Rails and you would just save your file and see if it worked basically instantly. That is the part of this discussion that personally I focus more on. [0:10:20.7] CC: I'm going to love to jump to the how in a little bit. I want to circle back to the beginning. I love the question that Michael asked at the beginning, what is considered developer, because that really makes a difference, first to understand who we are talking about. I think this conversation can go in circles and not that I'm saying we are going circles, but this conversation out in the wild can go in circles. Until we have an understanding of the difference between can you as a developer use Kubernetes in a somewhat not painful way, but should you? I'm very interested to get your take and Michael and Duffie’s take as well as far as should we be doing this and should all of the developers will be using Kubernetes through the development process? Then we also have to consider people who are not using Kubernetes, because a lot of people out there are not using communities. For developers and special, they hear Kubernetes is painful and definitely enough for developers. Obviously, that is not going to qualify Kubernetes as a tool that they’re going to look into. It's just not motivating. If there is anything that that would make people motivated to look into Kubernetes that would be beneficial for them not just for using Kubernetes for Kubernetes sake, but would it be useful? Basically why? Why would it be useful? [0:11:50.7] EK: I think from the point of view of a developer, you should try and stay away from Kubernetes for as long as you can. Kubernetes comes in when you start having issues of scale. It's a production matter, it's not a development matter. I don't know, like a DevOps issue, operations issue. Ideally, you put off moving your application to Kubernetes as long as possible. This is an opinion. We can argue about this forever. Just because it introduces a lot of complexity and if you don't need that complexity, you should probably stay away from it. To get to the other half of the question, which is if you're using Kubernetes in production, should you use Kubernetes in development? Now here, I'm going to say yes a 100% of the time. Blanket statement of course, we can argue about minutiae, but I think so. Because if you don't, you end up having separate environments. Let's say you're using Docker Compose, because you don't like Kubernetes. You’re using Kubernetes in production, so in development you are going to need containers of some sort. Let's say you're using Docker Compose. Now you're maintaining two different environments. You update something here, you have to update it there. One day, it's going to be Friday, you're going to be tired, you're going to update something here, you're going to forget to update something there, or you're going to update something there and it's going to be slightly different. Or maybe you're doing something that has no equivalent between what you're using locally and what you're using in production. Then basically, you're in trouble. I've heard from many companies that the main reason they decided to use Kubernetes in development is that they wanted to mimic production as closely as possible. One argument we can have here is that – oh, but if you're using Kubernetes in development, that's going to add a lot of overhead and you're not going to be able to do your job right. I agree that that was true for a while, but right now we have enough tooling that you can basically make Kubernetes disappear and you just focus on being a developer, writing your code, doing all of that stuff. Kubernetes is sitting there in the background. You don't have to think about it and you can just go on about your business with the advantage that now, your development environment and your production environment are going to very closely mimic each other, so you're not going to have issues with those potential disparities. [0:14:10.0] CC: All right. Another thing too is that I think we're making an assumption that the developers we are talking about are the developers that are also responsible for deployment. Sometimes that's the case, sometimes that's not the case and I'm going to shut up now. It would be interesting to talk about that too between all of us, is that what we see? Is that the case that now developers are responsible? It's like, developers DevOps is just so ubiquitous that we don't even consider differentiating between developers and ops people? All right? [0:14:45.2] DC: I think I have a different spin on that. I think that it's not necessarily that developers are the ones operating the infrastructure. The problem is that if your infrastructure is operated by a platform that may require some integration at the application layer to really hit its stride, then the question becomes how do you as a developer become more familiar? What is the user experience as of, or what I should say, what's the developer experience around that integration? What can you do to improve that, so that the developer can understand better, or play with how service discovery works, or understand better, or play with how the different services in their application will be able to interact without having to redefine that in different environments? Which is I think what Ellen point was. [0:15:33.0] EK: Yeah. At the most basic level, you have issues as such as you made a change to a service here, let's say on your local Docker Compose. Now you need to update your Kubernetes manifest on your cluster for things to make sense. Let's say, I don't know, you change the name of a service, something as simple as that. Even those kinds of things that sounds silly to even describe, when you're doing that every day, one day you're going to forget it, things are going to explode, you're not going to know why, you're going to lose hours trying to figure out where things went wrong. [0:16:08.7] MG: Also the same with [inaudible] maybe. Even if you use Kubernetes locally, you might run a later version of Kubernetes, maybe use kind for local development, but then your cluster, your remote cluster is on three or four versions behind. Shouldn't be because of the versions of product policy, but it might happen, right? Then APIs might be deprecated, or you're using different API. I totally agree with you, Ellen, that your development environment should reflect production as close as possible. Even there, you have to make sure that prod, like your APIs matches, API types matches and all the stuff right, because they could also break. [0:16:42.4] EK: You were definitely right that bugs are not going away anytime soon. [0:16:47.1] MG: Yeah. I think this discussion also remembers me of the discussion that the folks in the cloud will have with AWS Lambda for example, because there's similar, even though there are tools to simulate, or mimic these platforms, like serverless platforms locally, the general recommendation there is to embrace the cloud and develop in the cloud natively in the cloud, because that's something you cannot resemble. You cannot run DynamoDB locally. You could mimic it. You could mimic lambda runtimes locally. Essentially, it's going to be different. That's also a common complaint in the world of AWS and cloud development, which is it's really not that easy to develop locally, where you're supposed to develop on the platform that the code is being shipped and run on to, because you cannot run the cloud locally. It sounds crazy, but it is. I think the same is with Kubernetes, even though we have the tools. I don't think that every developer runs Kubernetes locally. Most of them maybe doesn't even have Docker locally, so they use some spring tools and then they have some pipeline and eventually it gets shipped as a container part in Kubernetes. That's what I wanted to throw in here as more like a question experience, especially for you Ellen with these customers that you work with, what are the different profiles that you see from the maturity perspective and these customers large enterprises might be different and the smaller ones that you mentioned. How do you see them having different requirements, as also Carlisia said, do they do ops, or DevOps, or is it strictly separated there, especially in large enterprises? [0:18:21.9] EK: What I see the most, let's get the last part first. [0:18:24.6] MG: Yeah, it was a lot of questions. Sorry for that. [0:18:27.7] EK: Yeah. When it comes to who operates Kubernetes, who deploys Kubernetes, definitely most developers push their code to Kubernetes themselves. Of course, this involves CI and testing and PRs and all of that, so it's not you can just go crazy and break everything. When it comes to operating the production cluster, then that's separate. Usually, you have someone writing code and someone else operating clusters and infrastructure. Sometimes it's the same person, but they're clearly separate roles, even if it's the same person doing it. Usually, you go from your IDE to PR and that goes straight into production once the whole process is done. Now we were talking about workflows and Lambda and all of that. I don't see a good solution for lambda, a good development experience for Lambda just yet. It feels a bit like it needs some refinement still. When it comes to Kubernetes, you asked do most developers run Kubernetes locally? Do they not? I don't know about the numbers, the absolute numbers. Is it most doing this, or most doing that? I'm not sure. I only know the companies I'm in touch with. Definitely not all developers run Kubernetes on their laptops, because it's a problem of scale. Right now, we are basically stuck with 30 gigs of RAM on our laptops. If your app is bigger than that, tough luck, you're not going to run it on the laptop. What most developers do is they still maintain a local development environment, where they can do development without going to CI. I think that is the main question. They maintain agility in their development process. What we usually see when you don't have Kubernetes on your laptop and you're using remote Kubernetes, so a remote development cluster in some cloud provider. What most people do and this is not the companies I talk to. This is basically everyone else. What most people will do is they make their development environment be the same, or work the same way as their production environment. You make a change to your code, you have to push a PR that has to get tested by CI. It has to get approved. Then it ends up in the Kubernetes cluster. Your feedback loop as a developer is insanely slow, because there's so much red tape between you changing a line of code and you getting a new process running in your cluster. Now when you use tools, I call the category MDX. I basically coined that category name myself. MDX is a multi-service development experience tooling. When you use MDX tools, and that's not just Tilt; it’s Tilt, it’s Garden where I used to work, people use telepresence like that. There is Scaffold from Google and so on. There's a bunch of tools. When you use a tool like that, you can have your feedback loop down to a second like I said before. I think that is the major improvement developers can do if they're using Kubernetes remotely and even if they’re using Kubernetes locally. I would guess most people do not run Kubernetes locally. They use remotely. We have clients who have clients — we have users who don't even have Docker on their local machines, because if you have the right tooling, you can change the files on your machine. You have tooling running that detects those five changes. It syncs those five changes to your cluster. The cluster then rebuilds images, or restarts containers, or syncs live code that's already running. Then you can see those changes reflected in your development cluster right, away even though you don't even have Docker in your machine. There's all of those possibilities. [0:22:28.4] MG: Do you see security issues with that approach with not knowing the architecture of Tilt? Even though it's just the development clusters, there might be stuff that could break, or you could break by bypassing the red tape as you said? [0:22:42.3] EK: Usually, we assign one user per namespace. Usually, every developer has a namespace. Kubernetes itself has enough options that if that's a concern to you, you can make it secure. Most people don't worry about it that much, because it's development clusters. They're not accessible to the public. Usually, there's – you can only access it through a VPN or something of that sort. We haven't heard about security issues so far. I'm sure they’re going to pop out at some point. I'm not sure how severe it’s going to be, or how hard it's going to be to fix. I am assuming, because none of this stuff is meant to be accessible to the wider Internet that it's not going to be a hard problem to tackle. [0:23:26.7] DC: I would like to back up for a second, because I feel we're pretty far down the road on what the value of this particular pattern is without really explaining what it is. I want to back this up for just a minute and talk about some of the things that a tooling like this is trying to solve in a use case model, right? Back in the day when I was learning Python, I remember really struggling with the idea of being able to debug Python live. I came across iPython, which is a REPL and that was – which was hugely eye-opening, because it gave me the ability to interact with my code live and also open me up to the idea that it was an improve over things like having to commit a new log line against a particular function and then push that new function up to a place where it would actually get some use and then be able to go look at that log line and see what's coming out of it, or do I actually have enough logs to even put together what went wrong. That whole set of use case is I think is somewhat addressed by tooling like this. I do think we should talk about how did we get here and how does that actually and how does like this address some of those things, and which use cases specifically is it looking to address. I guess where I'm going with this is to your point, so a tooling like Tilt, for example, is the idea that you can, as far as I understand it, inject into a running application, a new instance that would be able to – that you would have a local development control over. If you make a change to that code, then the instance running inside of your target environment would be represented by that new code change very quickly, right? Basically, solving the problem of making sure that you have some very quick feedback loop. I mean, functionally, that's the killer feature here. I think it’s really interesting to see tooling like that start to develop, right? Another example of that tooling would be the REPL thing, wherein instead of writing your code and compiling your code and seeing the output, you could do a thing where you're actually inside, running as a thread inside of the code and you can dump a data structure and you can modify that data structure and you can see if your function actually does the right thing, without having to go back and write that code while imagining all those data structures in your head. Basic tooling like this, I think is pretty killer. [0:25:56.8] EK: Yeah. I think one area where that is still partially untapped right now where this tooling could go, and I'm pushing it, but it's a process. It's not something we can do overnight, is to have very high-level patterns, the let's say codified. For example, everyone's copying Docker files and Kubernetes manifests and Terraform can take files, which I forgot what they're called. Everyone's copying that stuff from the Internet from other websites. That's cool. Oh, you need a container that does such-and-such and sets up this environment and provides these tools. Just download this image and everything is all set up for you. One area where I see things going is for us to have that same portability, but for development environments. For example, I did this whole talk about how to take your Go code, your Go application from I don't know, a 30-seconds feedback loop where you're rebuilding an image every time you make a code change and all of that, down to 1 second. There's a lot of hacks in there that span all kinds of stuff, like should you use Go vendor, or should you keep your dependencies cached inside a Docker layer? Those kinds of things. Then I went down a bunch of those things and eventually came up with a workflow that was basically the best I could find in terms of development experience. What is the snappiest workflow? Or for example, you could have what is a workflow that makes it really easy to debug my Go app? You would use applications like Squash and that's a debugger that you can connect to a process running in a large container. Those kinds of things. If we can prepackage those and offer those to users and not just for Go and not just for debugging, but for all kinds of development workflows, I think that would be really great. We can offer those types of experiences to people who don't necessarily have the inclination to develop those workflows themselves. [0:28:06.8] DC: Yeah, I agree. I mean, it is interesting. I've had a few conversations lately about the fact that the abstraction layer of coding in the way that we think about it really hasn't changed over time, right? It's the same thing. That's actually a really relevant point. It's also interesting to think about with these sorts of frameworks and this tooling, it might be interesting to think of what else we can – what else we can enable the developer to have a feedback loop on more quickly, right? To your point, right? We talked about how these different environments, your development environment and your production environment, the general consensus is they should be as close as you can get them reasonably, so that the behavior in one should somewhat mimic the behavior in the other. At least that's the story we tell ourselves. Given that, it would also be interesting if the developer was getting feedback from effectively how the security posture of that particular cluster might affect the work that they're doing. You do actually have to define network policy. Maybe you don't necessarily have to think about it if we can provide tooling that can abstract that away, but at least you should be aware that it's happening so that you understand if it's not working correctly, this is where you might be able to see the sharp edges pop up, you know what I mean? That sort of thing. [0:29:26.0] EK: Yeah. At the last KubeCon, where was it? In San Diego. There was this running joke. I was running around with the security crowd and there was this joke about KubeCon applies security.yaml. It was in a mocking tone. I'm not disparaging their joke. It was a good joke. Then I was thinking, “What if we can make this real?” I mean, maybe it is real. I don't know. I don't do security myself. What if we can apply a comprehensive enough set of security measures, security monitoring, security scanning, all of that stuff, we prepackage it, we allow users to access all of that with one command, or even less than that, maybe you pre-configure it as a team lead and then everyone else in your team can just use it without even knowing that it's there. Then it just lets you know like, “Oh, hey. This thing you just did, this is a potential security issue that you should know about.” Yeah, I think coming up with these developer shortcuts, it's my hobby. [0:30:38.4] MG: That's cool. What you just mentioned Ellen and Duffie remembers me on – reminds me on the Spring community, the Spring framework, where a lot of the boilerplate, or beat security stuff, or connections, integrations, etc., is being abstracted away and you just annotate your code a bit and then some framework and Spring obviously, it's a spring framework. In your case Ellen, what you were hinting to is maybe this build environment that gives me these integration hooks where I just annotate. Or even those annotations could be enforced. Standards could be enforced if I don't annotate at all, right? I could maybe override them. Then this build environment would just pick it up, because it scans the code, right? It has the source code access, so I could just scan it and hook into it and then apply security policies, lock it down, see ports being used, maybe just open them up to the application, the other ones will automatically get blocked, etc., etc. It just came to my mind. I have not done any research there, or whether there's already some place or activity. [0:31:42.2] EK: Yeah. Because I won't shut up about this stuff, because I just love it, we are doing a – it's in a very early stage right now. We are doing a thing at Tile, we're calling extensions. Very creative name, I suppose. It's basically Go in parts, but for those were closed. It's still at a very early stage. We still have some road ahead of us. For example, we have – let's say this one user and they did some very special integration of Helm and Tilt. You don't have to use Helm by hand anymore. You can just make all of your Helm stuff happen automatically when you're using Tilt. Usually, you would have to copy I don't know, a 100 lines of code from your Tilt config file and copy that around for other developers to be able to use it. Now we have this thing that it's basically going parts where you can just say load extension and give it a name, it fetches it from a repository and it's running. I think that is basically an early stage of what you just described with Spring, but more geared towards let's say an infra-Kubernetes, like how do you tie infra-Kubernetes, that stuff with a higher level functionality that you might want to use? [0:33:07.5] MG: Cool. I have another one. Speaking of which, is there any other integrations for IDEs with Tilt? Because I know that VS code for example, has Kubernetes integrations, does the fabric aid and may even plugin, which handles some stuff under the covers. [0:33:24.3] EK: Usually, Tilt watches your code files and it doesn't care which IDEs you use. It has its own dashboard, which is just a page that you open on your browser. I have just heard this week. Someone mentioned on Slack that they wrote an extension for Tilt. I'm not sure if it was for VS code or the other VS code-like .NET editors. I don't remember what it’s called, but they have a family of those. I just heard that someone wrote one of those and they shared the repo. We have someone looking into that. I haven't seen it myself. The idea has come up when I was working at Garden, which is in the same area as Tilt. I think it's pertinent. We also had the idea of a VS code extension. I think the question is what do you put in the extension? What do you make the VS code extension do? Because both Tilt and Garden. They have their own web dashboards that show users what should be shown and in the manner that we think should be shown. If you're going to create a VS code extension, you either replicate that completely and you basically take this stuff that was in the browser and put it in the IDE. I don't particularly see much benefit in that. If enough people ask, maybe we'll do it, but it's not something that I find particularly useful. Either you do that and you replicate the functionality, or you come up with new functionality. In both cases, I just don't see a very strong point as to what different and IDE-specific functionality should you want. [0:35:09.0] MG: Yes. The reason why I was asking is that we see all these Pulumi, CDKs, AWS CDKs coming up, where you basically use a programming language to write your application/application infrastructure code and your IDE and then all the templating, that YAML stuff, etc., gets generated under covers. Just this week, AWS announced the CDKs, like the CDK basically for Kubernetes. I was thinking, with this happening where some of these providers abstract the scaffolding as well, including the build. You don't even have to build, because it's abstracted away under the covers. I was seeing this trend. Then obviously, we still have Helm and the templating and the customize and then you still have the manual as I mentioned in the beginning. I do like the IDE integration, because that's where I spend most of my time. Whenever I have to leave the IDE, it's a context switch that I have to go through. Even if it's just for opening another file also that I need to edit somewhere. That's why I think having IDE integration is useful for developers, because that's where they most spend up their time. As you said, there might be reasons to not do it in an IDE, because it's just replicating functionality that might not be useful there. [0:36:29.8] EK: Yeah. In the case of Tilt, all the config is written in Starlark, which is a language and it's basically Python. If your IDE can syntax highlight Python, it can syntax highlight the Tilt config files. About Pulumi and that stuff, I'm not that familiar. It's stuff that I know how it works, but I haven't used it myself. I'm not familiar with the browse and the IDE integration side of it. The thing about tools like Tilt is that usually, if you set it up right, you can just write your code all day and you don't have to look at the tool. You just switch from your IDE to let's say, your browser where your app is running, so you get feedback and that kind of thing. Once you configure it, you don't really spend much time looking at it. You're going to look at it when there are errors. You try to refresh your application and it fails. You need to find that error. By the time that happened, you already lost focus from your code anyway. Whether you're going to look for your error on a terminal, or on the Tilt dashboard, that's not much an issue. [0:37:37.7] MG: That's right. That’s right. I agree. [0:37:39.8] CC: All this talk about tooling and IDEs is making me think to ask you Ellen. If I'm a developer and let's say, my company decides that we’re going to use Kubernetes. What we are advocating here with this episode is to think about well, if you're going to be the point to Kubernetes in production, you should consider running Kubernetes as a local development environment. Now for those developers who don't even – haven't even worked with Kubernetes, where do you suggest they jump in? Should they get a handle on – because it's too many things. I mean, Kubernetes already is so big and there are so many toolings around to how to operate Kubernetes itself. For a developer who is, “Okay, I like this idea of having my own local Kubernetes environment, or a development environment somehow may also be in the cloud,” should they start with a tooling like Tilt, or something similar? Would that make it easier for them to wrap their head around Kubernetes and what Kubernetes does? Or should they first get a handle on Kubernetes and then look at a tool like this? [0:38:56.2] EK: Okay. There are a few sides to this question. If you have a very large team, ideally you should get one or a few people to actually really learn Kubernetes and then make it so that everyone else doesn't have to. Something we have seen is very large company, they are going to do Kubernetes in development. They set up a developer experience team and then for example, they have their own wrapper around Kubectl and then basically, they automate a bunch of stuff so that everyone in the team doesn't have to take a certified Kubernetes application development certificate. Because for people who don't know that certificate, it's basically how much Kubectl can you do off top of your head? That is basically what that certificate is about, because Kubectl is an insanely huge and powerful tool. On the one hand, you should do that. If you have a big team, take a few people, learn all that you can about Kubernetes, write some wrappers so that people don't have to do Kubectl or something, something by hand. Just make very easy functions, like Kubectl, let’s say you know a name of your wrapper, context and the name and then that's going to switch you to a namespace let's say, where some version of your app is running, so that thing. Now about the tooling. Once you have your development environment set up and you're going to need someone who has some experience with Kubernetes to set that up in the first place, but once that is set up, if you have the right tooling, you don't really have to know everything that Kubernetes does. You should have at least a conceptual overview. I can tell you for sure, that there's hundreds of developers out there writing code that is going to be deployed to Kubernetes, writing codes that whenever they make a change to their code, it goes to a Kubernetes development cluster and they don't have the first – well, I’m not going to say the first clue, but they are not experienced Kubernetes users. That's because of all the tooling that you can put around. [0:41:10.5] CC: Yeah, that makes sense. [0:41:12.2] EK: Yeah. You can abstract a bunch of stuff with basically good sense, so that you know the common operations that need to be done for your team and then you just abstract them away, so that people don't have to become Kubectl experts. On the other side, you can also abstract a bunch of stuff away with tooling. Basically, as long as your developer has the basic grasp of containers and basics of Kubernetes, that stuff, they don't need to know how to operate it, with any depth. [0:41:44.0] MG: Hey Ellen, in the beginning you said that it's all about this feedback loop and iterating fast. Part of a feedback loop for a developer is unit testing, integration testing, or all sorts of testing. How do you see that changing, or benefiting from tools like Tilt, especially when it comes to integration testing? Unit tests usually locally, but the integration testing. [0:42:05.8] EK: One thing that people can do when they're using Tilt is once you have Tilt running, you basically have all of your application running. You can just set up one-off tasks with Tilt. You could basically set up a script that there's a bunch of stuff, which would basically be what your test does. If it returns zero, it succeeded. If it doesn’t, it failed. You can set something up like that. It's not something that we have right now in a prepackaged farm that you can use right away. You would basically just say, “Hey Tilt, run this thing for me,” and then you would see if it worked or not. I have to make a plug to the competition right now. Garden has more of that part of it, that part of things set up. They have tests as a separate primitive right next to building and deploying, which is what you usually see. They also have testing. It does basically what I just said about Tilt, but they have a special little framework around it. With Garden, you would say, “Oh, here's a test. Here's how you run the test. Here's what the test depends on, etc.” Then it runs it and it tells you if it failed or not. With Tilt, it would be a more generic approach where you would just say, “Hey Tilt, run this and tell me if it fails or not,” but without the little wrapping around it that's specific for testing. When it comes to how things work, like when you're trying to push the production, let's say you did a bunch of stuff locally, you're happy with it, now it's time to push the production. Then there's all that headache with CI and waiting for tests to run and flaky tests and all of that, that I don't know. That is a big open question that everyone's unhappy about and no one really knows which way to run to. [0:43:57.5] DC: That’s awesome. Where do you see this space going in the future? I mean, as you look at the tooling that’s out there, maybe not specifically to the Tilt particular service or capability, but where do you see some other people exploring that space? We were talking about AWS dropping and CDK and there are different people trying to solve the YAML problem, but more from the developer user experience tooling way, where do you see that space going? [0:44:23.9] EK: For me, it's all about higher level abstractions and well-defined best practices. Right now, everyone is fumbling around in the dark not knowing what to do, trying to figure out what works and what doesn't. The main thing that I see changing is that given enough time, best practices are going to emerge and it's going to be clear for everyone. If you're doing this thing, you should use this workflow. If you're doing that thing, you should use that workflow. Basically, what happened when IDEs emerged and became a thing, that is the best practice aside. [0:44:57.1] DC: That's a great example. [0:44:58.4] EK: Yeah. What I see in terms of things being offered for me tomorrow of — in terms of prepackaged higher level abstractions. I don't think developers should, everyone know how to deal with Kubernetes at a deeper level, the same way as I don't know how to build the Linux kernel even though I use Linux every day. I think things should be wrapped up in a way that developers can focus on what matters to them, which is right now basically writing code. Developers should be able to get to the office in the morning, open up their computer, start writing code, or doing whatever else they want to do and not worry about Kubernetes, not worry about lambda, not worry about how is this getting built and how is this getting deployed and how is this getting tested, what's the underlying mechanism. I'd love for higher-level patterns of those to emerge and be very easy to use for everyone. [0:45:53.3] CC: Yeah, it's going to be very interesting. I think best practices is such an interesting thing to think about, because somebody could sit down and write, “Oh, these are the best practices we should be following in the space.” I think, my opinion it's really going to come out of what worked historically when we have enough data to look at over the years. I think it's going to be as far as tooling goes, like a survival of the fittest. Whatever tool has been used the most, that's what's going to be the best practice way to do things. Yeah, we know today there are so many tools, but I think probably we're going to get to a point where we know what to use for what in the future. With that, we have to wrap-up, because we are at the top of the hour. It was so great to have Ellen, or L, how they I think prefer to be called and to have you on the show, Elle. Thank you so much. I mean, L. See, I can't even follow my own. You're very active on Twitter. We're going to have all the information for how to reach you on the show notes. We're going to have a transcript. As always people, subscribe, follow us on Twitter, so you can be up to date with what we are doing and suggest episodes too on our github repo. With that, thank you everybody. Thank you L. [0:47:23.1] DC: Thank you, everybody. [0:47:23.3] CC: Thank you, Michael and – [0:47:24.3] MG: Thank you. [0:47:24.8] CC: - thank you, Duffie. [0:47:26.2] EK: Thank you. It was great. [0:47:26.8] MG: Until next time. [0:47:27.0] CC: Until next week. [0:47:27.7] MG: Bye-bye. [0:47:28.5] EK: Bye. [0:47:28.6] CC: It really was. [END OF EPISODE] [0:47:31.0] ANNOUNCER: Thank you for listening to the Podlets Cloud Native Podcast. Find us on Twitter @thepodlets and on thepodlets.io website. That is ThePodlets, altogether, where you will find transcripts and show notes. We’ll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

In this episode, Kanina Blanchard discusses: Her one big message for leaders during Covid-19 (01:31) Why it’s important to create a game plan for surviving a crisis (02:56) Why you should think about how your actions will be judged in the long term (03:22) The three C’s to keep in mind in a crisis (04:16) What great crisis planning looks like (05:38) What professional athletes teach us about preparing for a crisis (06:16) The importance of including diverse perspectives in crisis planning (07:12) Why it’s never too late to address a crisis (09:07) What leaders can be doing right now to deal with he impact of Covid-19 (09:53) The one aspect of culture she’d like to see leaders focus on right now (15:05) How to build collaboration and connection virtually (16:33) Why you shouldn’t try to be a superhero (18:50) Positive lessons from Covid-19 (21:04) What she does to recharge (23:51) Kanina’s advice for leaders: Align your game plan with your values (03:06) Listen and show empathy (03:44) Prepare for worst-case scenario (06:55) Let yourself be uncomfortable (08:26) Think about who you (and your company) want to be when the crisis is over (09:30) Communicate in a way that creates trust (11:27) Make sure your actions and words match – and are consistent with your values (12:15) Don’t expect to have all the answers (13:05) Keep your employees’ needs in mind (17:57) Practice self-care (19:38)   More about Kanina Blanchard:  Kanina Blanchard has led teams through crisis on four different continents, and is recognized for her ability to adapt, lead teams and projects, and navigate complexities across various sectors. She has extensive experience working in international business, the public service, non-profit and consulting in areas that include organizational and communication challenges as well as issues, crisis and change management. Blanchard has coached thousands of CEOs, C-Suite executives and emerging leaders seeking to grow and develop their character, competencies and commitment over the last 30 years.  She is committed to providing targeted and customized strategic solutions to challenges that impact organizational and leadership brand, reputation and bottom line.  She is a lecturer in management communications and general management at the Ivey Business School, the recipient of the Margaret Haughey Master’s Award for Best Master’s Thesis, and is currently working toward her PhD. More about TILTCO: TILTCO is a boutique consulting company that helps leaders define and execute their strategies in order to achieve extraordinary business and personal results. Founded by Tineke Keesmaat who has over 20 years of leadership consulting experience with McKinsey & Company, Accenture and now TILTCO Inc.  More about The Ivey Academy: The Ivey Academy at Ivey Business School is the home for executive Learning and Development (L&D) in Canada. It is Canada’s only full-service L&D house, blending Financial Times top-ranked university-based executive education with talent assessment, instructional design and strategy, and behaviour change sustainment Links to additional resources: https://www.ivey.uwo.ca/academy/about/faculty/kanina-blanchard/ https://www.youtube.com/watch?v=1o0YpWlEugY https://iveybusinessjournal.com/publication/to-a-better-understanding-the-leadership-odyssey-explored/ Memorable quote: “Especially in a time of crisis, leaders need to ensure that what’s being done is the right thing and that it’s being done the right way.” Full Transcript: TINEKE KEESMAAT: LeaderLab is focused on having inspiring leaders share their stories and practical leadership tips to help others be more effective. Today is April 19, 2020 and leaders across the globe are in the uncharted world of dealing with the global health pandemic of coronavirus. There's no perfect playbook for leaders as they tackle the enormous social, emotional, and economic challenges brought on by COVID-19. Our next few episodes will be focused on how leaders are managing through these times. And our hope is that by sharing these leadership stories, we can find ways to help each other navigate through the uncertainty of COVID-19. ANNOUNCER: Welcome to LeaderLab, where we talk to experts about how leaders can excel in a modern world. Helping leaders for over 20 years, your host, Tineke Keesmaat. TINEKE KEESMAAT: Today I'm excited to chat with Kanina Blanchard. Kanina is a recognized public affairs and policy leader who has led teams internationally through crisis, challenge, and change. From bomb threats to environmental and human health disasters, she has helped teams and leaders in both the private and public sector navigate their toughest scenarios. Kanina is a lecturer at the Ivey Business School, where she focuses her research in the areas of women in leadership and responsible leadership. Kanina wants to help leaders navigate successfully through COVID-19 by being purposeful, empowering, and passionate so they cannot just survive, but find a way to thrive into the future. Kanina, thank you so much for joining me on today's LeaderLab. I'd like to start by asking you, what's the one big message you want leaders to take away from our conversation today? KANINA BLANCHARD: I think what we all need to do is start by stopping. And what I mean is to stop. Stop and take a step back from the firefight we find ourselves in. Because in times of crisis, whatever that crisis may be-- if it's a family issue or an illness, or some of my lived experience includes bomb threats and fires and explosions-- that we need to take a step back because we fall into this loop. And this loop is one where we have a sense of fear. We have a sense of powerlessness, a loss of control. And there is this deep need to do something. And that itself creates a problem, because we just do. And we keep doing. And we're driven by this sort of primal instinct, because actually doing something in the middle of a crisis feels good. But that's not good enough, not for a leader. And especially in a time of crisis, leaders need to ensure that what's being done is the right thing and that it's being done the right way. TINEKE KEESMAAT: I love that. So really, although we want to just act, act, act, what I'm hearing you say is pause and make sure that you are being purposeful. From a practical lens, what do you think leaders need to be thinking about as they're creating their crisis game plan? KANINA BLANCHARD: Our game plan for surviving the crisis, but hopefully thriving when this is over, is an important place to start. And not just on what you need to get done, but the why you're doing it-- align those to your values and think about who you want to be remembered as when this is over. So not just you-- your team, your organization. And sometimes when we think about these items, we may make some different decisions. Perhaps we still have to do what we have to do. For example, we're having to let people go. But how do we let people go, that matters. That matters for the kind of relationship you will have, the reputation you will have on the back end. So listen more. Demonstrate through your words and actions that you care about we, not just me. So don't sit back and make decisions in a crisis and articulate them. Involve others. Help other people be purposeful. Find a reason yourself to be passionate, and remember that your attitude as a leader is truly contagious. And this may be a bad play on words, but truly think about, what do you want to be spreading right now as a leader? TINEKE KEESMAAT: I know you talk about the 3 C's-- so crisis management, communication, and culture. And you've encouraged leaders in other talks to think about those three areas as they navigate through. So I'd love to spend a bit of time unpacking those and sharing your thoughts and tips. So if we can start with crisis management, can you talk to me about what that means? KANINA BLANCHARD: In a crisis, we need to lead and we need to manage. But it's not business as usual. So when we talk about leadership in times of crisis, there needs to be a plan. We need to prepare to manage and lead through crisis. And we need to, hopefully, have practiced and been prepared to deal with the absolutely-not-business-as-usual challenges that come up. I think about how many clients and people I speak to right now who say, you know, we've been trying to get a work from home policy in place for years and the company said no way. And now there are so many monitors being delivered to people's homes. So how do we manage? How do we adapt? How do we take on a mindset that we can manage through crisis? TINEKE KEESMAAT: Can you talk to me a bit about planning? What does great planning look like in this context? KANINA BLANCHARD: If we're going to actually lead through crisis, we have to accept that a crisis is different and that there are different emotions and different challenges. And it's uncomfortable to do this. So in a lot of large organizations and in my own professional career, we've done a lot of crisis management planning where we get down into the deep, the dirty, and sometimes the ugly and uncomfortable stories about what keeps us up at night. What is it that we're afraid of, the worst-case scenarios? And we live in a bit of a culture where we don't want to talk about those things. And this is where we can take guidance from professional athletes and professionals in fields like astronauts, who a huge part of their life is to plan for the unexpected and to plan for the worst-case scenario. I love this story that Tessa Virtue and Scott Moir, the very well-known Canadian figure skating champions, talk about-- that when they were practicing for the Olympics, they basically had a protocol where their coach would do the completely unexpected-- where the music would die, where the lights would go off-- and they could continue their performance regardless. So they planned for that. So plan for continuation of your operations and those scenarios that you are most concerned about. TINEKE KEESMAAT: And preparing, what does that look like? KANINA BLANCHARD: Preparing looks like bringing the right people to the table to do the work. And this seems kind of obvious, but one of the things that happens is we bring people around the table who are like us, who think like us, who focus on the things we focus on. But in a crisis, what we need to do is think about our audiences, our stakeholders from a very broad perspective and bring to the table those people who truly represent the voice of our communities, of our suppliers, of government, of other stakeholders. And so to prepare in a way that is mindful of not only kind of our worst-case scenarios or situations that we would normally not want to deal with, but to do it with people and get the insights of the people that matter the most, which is our audience in these situations. TINEKE KEESMAAT: Fantastic. And finally, practicing-- what do leaders need to think about there? KANINA BLANCHARD: Absolutely. So you know, it's funny. When I work with a lot of my clients, when I work with students, we do things like role plays or situations or scenarios. And some people love that, and some people are absolutely petrified. And I tell you, if you're petrified and I'm working with you, I am putting you on point, because we need to get past that. We literally need to practice stepping into the roles, making decisions, taking risk, and learning that there will be failure. There will be failure. And this sort of leads to this idea of communications and culture, and why it's so important to be able to manage through a crisis. TINEKE KEESMAAT: So Kanina, this sounds amazing. But I'm sure there's some people listening to you saying, gosh, I wish I had heard Kanina a year ago, because I would have done the plan, prepare, and practice. But now they're living through COVID. Is it too late for them? How do you talk to leaders about what they can do in the moment if they haven't had the opportunity to do your three P's before? KANINA BLANCHARD: It's not too late. It's never too late. In fact, we know in life there are some crises and issues that we can prepare for and some things that just hit us out of the blue. So we can absolutely bring our best self forward when things start to unravel. And for a lot of us, things are unraveling right now. So what do we do? Let's think about, have we thought through who do we want to be when this crisis is over? Have we planned for who we want to be at the end of this? Have we brought the right people together to talk about and prepare for not only doing what we're doing today better, but preparing for this crisis is not over. And if we're just living in the moment and focusing on the short-term, we're missing both other risks as well as other opportunities. So use this time to prepare. Then the last thing is we can still practice. We can practice, because when we realize that the most important thing we can be doing as leaders right now is not only stepping up into managing and leading through the crisis, but being the best communicators we can be and ensuring that our culture is one that's going to support us through this initiative and this challenge. We can be focused on what's coming. And that's going to help us, hopefully, thrive. TINEKE KEESMAAT: You bring us to the last two parts of your three C's. Communications was one that you mentioned up top. Can you talk to me a bit about what leaders should be doing or how they should be communicating during these uncertain times? KANINA BLANCHARD: Everything that you've been taught from a communications perspective, from a leadership communications perspective in normal business operations, applies today. But it is magnified and it is amplified. So do you communicate with your audiences? Are you authentic in your communications? Are you transparent? Are you consistent? Now, when I say things like "transparent" and "authentic," let's be clear, this isn't about saying whatever is on your mind. This is about putting your audience first. It's about purpose. As a leader, when you communicate you are purposeful in trying to motivate, inspire, compel, and influence others. When you communicate in a crisis, that's even more important. But it needs to be done in a way that people believe and they trust. I've had clients, as well as people that I've been talking with, who've said, look, my company is saying all the right things, but no one believes it. They're not making us feel like they're doing the right things. And that's key about communicating as a leader. TINEKE KEESMAAT: I know some of the people that I've been talking to have raised this point around consistency of message during these times. Their leaders are saying one thing but then doing something else. Can you talk to me about that, how that comes to play and why it matters? KANINA BLANCHARD: So when we talk about our values and all of these issues, what we say has to reflect who we are as an organization, who we are as people. And then we need to be very mindful to be consistent with that. So if we say to our employees in today's world that your health and welfare matters the most but then we back it up with actions and tell people to just go do things and take a risk, there isn't that consistency between what's being said and then what's being asked. So that is really important, because that is your legacy through this crisis. Will people, including your own employees as well as your customers in the community, be able to trust you based on what you did? Don't expect that you are going to have all the answers, that things are going to work at the same pace, that things are going to happen exactly the way that everybody would have expected them in normal times. But as a leader, set expectations. But work with people to bring their best selves forward. And you do that by connecting with people at the emotional level. And we could say, well, you know, how do you do that when everyone is leading at a distance? So there's lots of literature on this topic. But at a very simple tricks and tips level that I would love to see people truly take in, if you're talking to somebody, you're trying to build connection, stop looking at your screen and look at where your camera is. Because the reality is that most of us are speaking on cameras most of the day while we're looking down at someone on the screen. Or we might be looking at ourselves on the screen. What other people are seeing is you looking down at them. Lift your head, and know that that little aperture of your camera is your audience. And care enough to look at them and talk to them clearly, authentically, with purpose. Be values-based, and connect with them as human beings right now. That's what's going to motivate them to bring their best selves forward. TINEKE KEESMAAT: The last of your C's was around culture. I'm curious, Kanina, what do you think are the most important elements of culture that leaders need to get right during COVID-19? KANINA BLANCHARD: Well, there's so much about culture, right? It's hard to change a culture in the middle of a crisis in some ways, because we bring everything that we have been to that moment. However, crises create an incredible opportunity and a raison d'etre, or creates a bias for action. And so if there was one part of culture that I would love to see leaders focus on right now is collaboration. If we lose touch and if we lose connection with people at this time, the good parts of our culture will start to erode. And in the vacuum of collaboration and connection, we will drive movement toward people feeling more isolated, less purposeful, perhaps not knowing where their place is-- which will create more fear. I know this adds a burden to a lot of leaders. It's spending more time trying to connect. But creating opportunities for people to bring their best selves forward to collaborate on aspects of your business continuity plan and your future plan right now can do more to keep your employees, your customers engaged and motivated than anything. TINEKE KEESMAAT: Any practical tips or insights about how to build collaboration when everyone is working at home and has slightly variant schedules and potentially is distracted by their own personal issues or complexities? Any suggestions for folks on how to build virtual connection? KANINA BLANCHARD: Absolutely. There are so many, and we only have so much time. But just a few ideas. One, be patient. Everyone is in a different place. And what we know about communications as a field is that different people resonate with different kinds of communication. So some people read. Some people are auditory. Some people are visual. So keep in mind that if your goal through your communication is to ensure understanding or build awareness or to create collaboration, you need to think about that all your employees or your stakeholders are also different. And so just doing things one way blanket in one email doesn't mean you've communicated. All that means is you've sent out information. So think about the different ways you can communicate. You can write it. You can record a video. You could have open town halls. You could create social spaces like coffee or cooler discussions, where people who need that social interaction can self-select to come in. Offer to do it at different times. If you're working in a global environment-- it's one of the things I've found working internationally, is that my colleagues in Asia were always the ones-- and when I worked in Asia, I learned this myself-- are always the ones starting the earliest and ending the latest. So if you're going to have a session with your employees or you want to meet one-on-one, think about what time zone they're in and do it at a time, it might be inconvenient to you. But boy, that's going to send a signal of collaboration and care to someone sitting in Hong Kong 12 hours away that very few other things can do. TINEKE KEESMAAT: That's fantastic. And again, what I'm hearing from you, again, as a leader is really just thinking about your employees, stakeholders-- where they're at, and kind of bridging the gap by meeting them at that starting point versus just what's convenient for you. So again, some care and compassion in these times. KANINA BLANCHARD: You know, someone might be out there listening going, how much more can I possibly do? It just seems like we load more and more on managers and leaders. And considering these times of opportunity to be the best you can be is really important. But I would say, don't go down the superhero path. We've seen through the last many decades that superheroes tend to not only burn out and harm themselves and their families and their organizations, but that's not a way to win. We are in a marathon. COVID-19 is a marathon. It is not a sprint. So these are all best practices. And I hope the leaders listening are motivated by the difference they can make. But it also starts with self-care. TINEKE KEESMAAT: And what does that mean, self-care? That's a hot term right now. But from your perspective, how can leaders practice that, or what do they need to be mindful of? KANINA BLANCHARD: I think everybody is different, and so there is no one checkbox or list that's going to help. But I think that one thing I've seen developing over the last 30, 35 years that I've been in business is this idea of the importance of reflection, this importance of taking time-- whether it's even 10 minutes-- to stop and to think, and sometimes simply clear our minds. So there are people who are practicing meditation or mindfulness. It could be taking a walk. It's the small things that we do to take care of ourselves and take care of others. TINEKE KEESMAAT: And it's interesting, as you've talked-- and a thread that I'm taking away from this call-- is a bit of this authenticity, right? We're all in it together. Saying I'm tired, my neck is hurting, and just putting that out there allows people to connect at a very human level-- which I think is really important during these times, because people are in that place of fear and uncertainty. So I love this message that I've been hearing anyway around just being you and being authentic, and sharing that with others. So Kanina, you have seen the before and after of many crises through your career. I'm curious, from your vantage point as you think about COVID-19, do you foresee any positive impacts or lessons or practices resulting from this scenario? KANINA BLANCHARD: Absolutely. I think about the things that I'm seeing in the news today-- we have young people who are starting businesses that aren't there to make money. But they're volunteering their skills to develop-- for example, in Portugal I heard about a young student who started a web app where people who cannot rent their homes right now are being matched with health care providers who can't go home in the fear of spreading disease. I've heard about people who have developed apps around COVID where they've been offered money for their apps and instead they've said, no, this is a public service. We're seeing governments do things that are truly unprecedented. We have governments that are stepping up early and fast and trying. And of course, there's always going to be criticism and critique, and there's always going to be things that we can do better. But look at the packages that are being put together to help ensure that people, even in these hard times, we can have food on our table. We're trying to help marginalized people make sure that they have food on their tables. We're trying to help students. There is an effort under way today that we will learn from. And in reality, when the next crisis comes-- and it will-- I think we will have learned so much, a lot of good and some bad. And if we embrace the failures and improve on the successes, we will together succeed moving forward and grow. TINEKE KEESMAAT: I love that, a feeling of hope and a recognition that even in these tough times, people are generally good and everyone is trying their best to kind of navigate through. Kanina, this has been fantastic. And I know I've taken a number of notes and lessons that I will use in my own world and share with friends and colleagues. So I really do appreciate your insights and your enthusiasm that you shared with us today. ANNOUNCER: And now, let's get to know our guest a little better with some rapid-fire questions. TINEKE KEESMAAT: As we wrap up, one of the practices that we would like to do on LeaderLab is to help our listeners get to know you even more as a person with some rapid-fire questions. So if you don't mind, I have five questions for you here around the themes of the time that we're spending at home. So don't think too hard, but your first response. Your go-to comfort food? KANINA BLANCHARD: Gluten-free pasta. TINEKE KEESMAAT: Do you like to go for a workout or curl up with a book to regenerate yourself? KANINA BLANCHARD: Oh gosh, neither. I'm working on my PhD, so I read when I have to. I love my audiobooks, and I love walking with my family and my ridiculous basset hound who howls and sings and always brings a smile to our face. TINEKE KEESMAAT: Nice. One item you could never live without. KANINA BLANCHARD: Well, I think my answer needs to be my husband right now. And I don't think I should call him an "item." But I think one of the things that we're learning-- we have eight people living under our roof right now. And what helps us survive, I think, is each other and that sense that you're not going through it alone. TINEKE KEESMAAT: That's wonderful. Your favorite cartoon character? KANINA BLANCHARD: I think it sort of maybe depends on the day. For some reason, just what popped into my mind right now is the Tasmanian Devil. I have no idea why. Maybe that reflects the way the world is feeling right now. That's what I've got for you. TINEKE KEESMAAT: And your work from home attire-- PJs, fully dressed, or a little bit of both? KANINA BLANCHARD: Absolutely a little bit of both. You'll never catch me in pantyhose. But at least from the waist up, I feel that I am more productive and I am engaging with the people who I'm speaking with in a respectful and honest and a professional manner when I've gone the business casual route, for sure. TINEKE KEESMAAT: I love it. I love it. Amazing. Kanina, thank you again for the time today. ANNOUNCER: Thank you for joining us today on LeaderLab. LeaderLab is powered by Tiltco, helping exceptional leaders achieve extraordinary results, and the Ivey Academy at Ivey Business School, Canada's home for learning and development. You can learn more about Tiltco and LeaderLab at tiltco.ca. And to find out more about The Ivey Academy, go to iveyacademy.com

In this episode, Angela Barkan discusses: Her role at S-Curve Records (02:01) How the music industry has been impacted by COVID-19 (02:51) How technology is helping her team stay connected (05:03) The importance of in-person interactions in creative fields and how her team is recreating that social atmosphere in isolation (05:58) What her team is doing to keep on top of pop culture and entertainment trends (07:37) How people are coming together to make promotions and entertainment work during social isolation (11:10) How they’re trying to learn from the risks they have to take (13:12) Changes she hopes will her team/industry will keep once the crisis passes (17:44) How the crisis has levelled the playing field for everyone in the organization when it comes to ideas and contributions (19:54) How she’s handling the crisis as a leader (21:29) Ways to check in on the mental/emotional health of employees and keep things light (24:28) Her advice to leaders (26:51) The one item she couldn’t live without during the crisis (29:54) Angela’s advice for leaders: Be flexible and find ways to adapt (09:01) Don’t be afraid to take risks – especially when there’s no playbook (09:22) Don’t focus on perfection; try new things and learn from them (13:48) Empower all employees to feel like they have a voice (18:36) Be open to different perspectives from within the organization (19:41) Be kind to yourself (21:35) Show empathy for your team and each member’s situation during the crisis (23:53) Connect with your colleagues on a human level (25:29) Embrace small successes (27:01) More about Angela Barkan: Angela Barkan is vice president of marketing at S-Curve Records/BMG. She has more than 15 years’ experience in the entertainment industry, and has spearheaded integrated campaigns for global celebrities as well as developed new and rising talent. She has worked closely with nationally recognized celebrities including Mariah Carey, Dave Matthews Band, Placido Domingo and Dolly Parton, as well as upcoming talent such as International YouTube stars The Piano Guys. At S-Curve Records, she works with a wide roster of artists including Andy Grammer, Netta and The O’Jays. Barkan has been repeatedly recognized for top performance and selected to work with top company leaders. She’s a frequent speaker on industry panels for events with CMJ, MTV's Youth Marketing Forum and NYU/Stern Alumni. She’s also a member of the Arts Committee for Central Park Summerstage / City Parks Foundations, which brings free concerts and events to Central Park and 17 neighborhood parks in New York City. Barkan holds an MBA from NYU and a BA from Cornell University. Links to additional resources: https://s-curverecords.com https://cityparksfoundation.org/summerstage Produced by: Kara Kennedy from TITLCO Consulting  More about TILTCO: TILTCO is a boutique consulting company that helps leaders define and execute their strategies in order to achieve extraordinary business and personal results. Founded by Tineke Keesmaat who has over 20 years of leadership consulting experience with McKinsey & Company, Accenture and now TILTCO Inc. More about The Ivey Academy at the Ivey Business School:  The Ivey Academy at Ivey Business School is the home for executive Learning and Development (L&D) in Canada. It is Canada’s only full-service L&D house, blending Financial Times top-ranked university-based executive education with talent assessment, instructional design and strategy, and behaviour change sustainment Full transcript:  TINEKE KEESMAAT: Hi, it's Tineke here. Welcome to today's LeaderLab. As you know, LeaderLab is focused on having inspiring leaders share their stories and practical tips in order to help other leaders be even more effective. Today is March 31st, 2020, and leaders across the world are in the uncharted territory of dealing with a global health pandemic of coronavirus. There's no playbook for leaders on how to tackle the enormous economic, social and emotional challenges brought on by this epidemic. Our next few episodes will be focused on how leaders are personally managing through these times, what they're doing and what they're learning. Our hope is that by sharing these lessons and stories, we can help each other as we navigate through these uncertain times. ANNOUNCER: Welcome to LeaderLab where we talk to experts about how leaders can excel in a modern world, helping leaders for over 20 years. Your host, Tineke Keesmaat. TINEKE KEESMAAT: Today, I am thrilled to be talking with Angela Barkan. Angi is a results-oriented, pop culture savvy, media executive. In her role, she spearheads campaigns to bring the work of talented musicians into our homes. She's worked with global celebrities and develops new and rising talent. You'll be familiar with many of the artists she's worked with in her over 20 years in the business: Yo-Yo Ma, Mariah Carey, Dave Matthews Band, Christina Aguilera and The Piano Guys, to name a few. In her current role at S-Curve Records, part of BMG, Angi serves as the vice president of marketing. There she works with a wide roster of artists including Andy Grammer, AJR, Leslie Odom Jr., Netta, The O'Jays, Duran Duran and many others. Angi holds an MBA from NYU and a BA from Cornell University. She's passionate about arts and the youth and is a proud member of the City Parks Foundations Art Committee, which provides free concerts to New York City parks. Angi also speaks at many industry events. Angi, thank you so much for joining us on today's LeaderLab. So Angi, the music industry is fascinating and I'm curious if you can share a perspective on what you do within that industry. ANGELA BARKAN: I'm the vice president of marketing at a record label called S-Curve Records, which is part of BMG. And basically, my role, to dumb it down, is once the music comes into me, it's my job to make sure that the world hears it in any way possible. So that's either by live concerts, on the radio, on TV, on digital platforms-- like Spotify, iTunes-- with brand partnerships and, then, the next level is, of course, to maximize all the revenue streams that come from that. TINEKE KEESMAAT: So Angi, as we talked about, we're focusing our LeaderLab series around how leaders are navigating the uncharted territory of coronavirus. And I was curious if you could share with us how your industry and your role has been impacted. ANGELA BARKAN: Everything has been thrown on its head and changed to really what everyone is calling the new normal. The most obvious impact in our industry would be live touring. Spring and summer festivals are postponed-- postponed or canceled. And that's impacting, of course, not only the artists, but everyone who's involved in that and some roles that you might not think about like the person who takes the tickets at the venues or the person who is setting up the bar, lighting, bus drive-- tour bus drivers. It's really-- the trickle down effect has been pretty devastating for everyone. I think the second area that the virus has really impacted, obviously, is physical sales across all industries, right, because no one can go to the stores. Also on the production level, a lot of the plants and distribution centers are closing or operating at a much lower capacity, so it's just harder to get things out. TINEKE KEESMAAT: And I know that your leadership team at S-Curve, basically, decided that the show needed to go on in this environment. Can you talk to me a little bit about the conversation that your team had to make that decision and then to guide the actions that you guys have been taking. ANGELA BARKAN: I wish it was-- I wish I could say it was a cautious, well-planned out decision, but really it was more kind of triage, right? Like every industry, like every team, I think, has had to do the show must go on, literally and figuratively. And we just had to hunker down and figure out how to adapt to what is our new reality, hopefully, for another month or so but who knows. TINEKE KEESMAAT: Can you give me some examples of how you've had to adapt or initiatives that your teams have put in place in this, hopefully, not forever new normal times? ANGELA BARKAN: Yeah, absolutely. I mean, I think, on a very granular level, when I'm not in the office, I'm in Toronto, some of our colleagues are in Florida and New York and California. We're all spread out. So technology has really been saving grace. Every day we've been doing one or two Zoom calls with the entire team, and that's really helped. We've been communicating on Microsoft Teams, just really upped the communication as much as possible. There are certain things, I think, that it's very easy to take for granted when you can just walk over to someone's office or have a casual business talk over lunch, and you're, actually, really getting things done. And those types of interactions aren't there anymore so you have to figure out a way to replace that. TINEKE KEESMAAT: And has-- do you have some examples of how you've personally been replacing those? ANGELA BARKAN: You know, the entertainment industry thrives on creativity. And so it's a little difficult sometimes to be creative when your little box is on a screen talking to each other. TINEKE KEESMAAT: No doubt. ANGELA BARKAN: And so some of the interactions, like, just at the coffee machine or when we go see one of our artists at a show, like, those types of moments where ideas come through just sort of natural conversation being in a creative environment, aren't there anymore. So we have done some fun things to try to recreate them. For example, we've been sending each other playlists on Spotify. We've been doing virtual happy hours, just to kind of chat, just to talk about what's going on in pop culture and keep our minds in a creative space where we know what's going on beyond the four walls of our apartments or houses where we're-- or, in my case, Airbnb-- where we're stuck right not. TINEKE KEESMAAT: I can imagine that some of those playlists are pretty incredible given the industry that you guys are in. So, hopefully, you're learning some new music and share-- getting to know each other better through that process as well. ANGELA BARKAN: Being in a creative industry, and in the entertainment industry, our job is to entertain, that's our responsibility, and to help our artists continue to do that even with these unusual circumstances. So one of the things that we've been doing to keep the communication going is really simple. We've just been all contributing ideas onto a Google Document of different things that we've seen, different-- what different artists have been doing on social media, how our competitors have been responding to this, how, even brands and outlets beyond our industry, have been responding to try to keep their consumers engaged. And so we've been doing it on a micro level just within our record label, but we've also been doing it on a larger level, globally with BMG. And it's really great because we can see what other countries are doing. This virus has, unfortunately, become an international situation. And so it's really interesting to just see how different cultures, different countries are responding to this and remembering that music is global, right? So we're not just talking about people in our country, we're really trying to reach the world with what we're doing. TINEKE KEESMAAT: I love that collaboration and creativity in this moment, right? Just no ideas, there's no perfect answer. People haven't done this before. So just any idea that people are seeing or feeling, just putting that out there for other people to learn from. I just love how dynamic that feels as you're describing it. ANGELA BARKAN: Absolutely. I mean, adapt is the main word, right? Every single person from a grocery store to the ride share industry to travel industry to retail to music and entertainment has had to adapt and adapt really quickly. And I think there's a lot of trial and error and risks that are being taken that wouldn't have been taken before, and I think that's kind of a blessing. We're all in uncharted territory. And, I think, there's something really freeing about being able to take some creative risks and just try some things out, knowing that it might not work but it might work, and it might be really cool, and that's kind of an exciting thing. TINEKE KEESMAAT: You've done some pretty cool things with your artist and that probably would not fly in the old world, but you've pushed the artist and the technology to support the artists to try different things. Can you give us some examples of what that has meant in the last week or two? ANGELA BARKAN: One of our artists, Leslie Odom Jr.-- you might know him from Hamilton on Broadway, he recently costarred in Harriet and he's also an amazing singer, shameless plug. He has a new album out called Mr. He was-- he's really been-- this whole campaign has really been impacted by this. We were in the middle of a promotion cycle. His tour has been postponed until the fall and, part and parcel of that, his TV appearances have been impacted. So yesterday, he was supposed to do the TV show, Live with Ryan and Kelly on ABC. That show is based in New York, he's based in LA and his band is based all over the place. So we had to adapt. We really wanted the show to go on from an entertainment perspective, certainly, from a commerce perspective. We're in the middle of a campaign for a new single. To his credit, and to the show's credit, we all really got together and took a really big risk. What we did was we filmed each of the musicians doing their parts, performing their-- performing their parts in their living room, in their basement, in their bedroom. And we edited all those parts together and, then, Leslie performed the song from his living room with a video of all the musicians behind him. And-- and it was amazing. Ryan Seacrest was in his kitchen, broadcasting. Kelly Ripa was in her living room. And it was just this-- it was just this really amazing moment of everyone-- everyone from the show, from the producers, from the audio people, from the network, the musicians and, of course, the label, and, of course, Leslie really taking a risk and being vulnerable in this time. TINEKE KEESMAAT: Yeah, I was going to say, these artists-- so that's like a fantastic example and I know a few of your other artists have livecast concerts from their homes. And it does strike me that these artists are vulnerable at the best of times, but often they have a ton of support around them. They've got to have hair and makeup. They've had people soundcheck, test things and they're having to be authentic in these moments and put themselves out there in their homes, probably, with a little less support around them than normal. And I'm just curious how that has felt for them or what you've experienced or observed from these artists that are exposing themselves in a new way. How did it go? ANGELA BARKAN: It actually went great. It was-- it really-- it was really cool to see and it was successful. The song-- we saw the song immediately jump up to number 15 on the iTunes chart, the record jumped up to number 16. There was a ton of social media noise about it. So I think, everyone really appreciated the fact that it wasn't a perfect scenario, but the show must go on. And it's a cliche, I think, for a reason because there's a lot of situations where the show must go on and this was-- this was a perfect example of it. And I think, was it perfect? No. And we did do a post mortem right after. We all got on Zoom, we talked about it. We talked about ways that we might change the audio, change some of the angles, do some things differently for the next time. But the important message that, I think, we can all internalize and then-- and, then, take, just moving forward when things go back to normal, is sometimes it's important to take risks even though the output might not be perfect. There's something to be learned and that can push everyone forward. TINEKE KEESMAAT: I love that. And I think that there's also-- what I'm hearing is also this vulnerability, right? So putting yourself out there in that risk. You're-- you know that you're showing up in a way that you may not normally have or your confidence may not be fully there, but from the story that you told of yesterday or other artists that I've seen live streaming from their home, I don't know, for me, personally, I'm connecting more to them in these moments where their hair is maybe not quite as coiffed as normal or you see their children running in behind them. I don't know, there's something about that authenticity that's happening right now and the vulnerability that it's making it easier to connect, than maybe when I see them perfectly on the screen or in a video. So there is something for me, personally, about that vulnerability and authenticity that is showing up in this moment. ANGELA BARKAN: Totally. I mean, yeah, I love that and I-- and I agree with you. You know, there's really-- there's a connection that, I think, artists are able to make with their fans. And the message is, really, we're all in this together, just like you're, maybe, working in your pajamas. I'm trying to write a song with my kids running around as I'm also trying to make them lunch. These are real-- these are real situations. I also think-- it's interesting, we have another band called AJR, which is three brothers. And for them, now they're playing sold-out shows at Radio City and the Greek in LA. But they started as three brothers busking in Washington Square Park, just outside. And what's interesting with them is, in some ways, this has required them to go back to those roots. The other day, they did an Instagram live from their living room. And they had a keyboard there and Adam, the bass player, had his base, but-- but the lead singer, Jack, obviously, a lot of the instrumentation was missing. And so what did he do? He grabbed a bottle of vitamins and used them as a shaker. TINEKE KEESMAAT: I love that. ANGELA BARKAN: And it was, like, a perfect example of taking a risk and adapting and trying something. And it might not be perfect, but it was still something, and it was great. It was really, really cool. TINEKE KEESMAAT: I'm sure it's probably liberating, too, in some ways, right? Because people are able-- are being a bit more forgiving and recognizing that everyone's just trying their best right now, that if I'm an artist, I love sound and how can I add this vitamin D mix to the-- to the set and see how it feels. There might be some creativity and liberation that is happening for artists during this time-- in this, again, these strange, strange times that are right now. ANGELA BARKAN: Absolutely. And who knows? If there's anyone listening who works for a big vitamin company, call me. You know, there could be some really interesting partnerships that come out of this. Who knows? TINEKE KEESMAAT: I love that, I love that. So Angi, these are incredible examples of creativity and risk-taking and pushing technology and, I think, that they've been very meaningful in order to help people continue to connect to artists and to music in these periods of isolation and uncertainty. It sounds like you've had some fun experimenting. And I'm curious if there are things that you've been doing that you hope, actually, continue when the world goes "back to normal", quote, unquote. ANGELA BARKAN: Absolutely. And I think the thing that I hope that we continue is really experimenting with technology and pushing ourselves and our partners to find new ways of doing things, to find new ways of entertaining, bringing music into the home, making it more accessible, making it more interactive. You know, some of the things Instagram is doing and TikTok are just incredible in terms of bringing-- bringing fans and the artist together. So we do think those things will continue. Maybe some artists that weren't as keen on it originally, now have been forced to use it. TINEKE KEESMAAT: Right. ANGELA BARKAN: And-- and I'm hoping it will-- and I'm hoping it'll stay because it's really- it's really exciting. Something else I've noticed, just from a managerial perspective, is everyone at the table, let's call it at the Zoom table in their own little box, has been kind of empowered to come up with ideas. At S-Curve we're really not hierarchical at all, everyone does have a voice. But I'm noticing it even more now, anyone can come up with a good idea. And, I think, everyone sort of feels empowered in what is the Wild West right now to come up with an idea because it's just sort of this free space of, like, with an attitude of, may as well try it. And so because of that, it's kind of fun and it's giving-- I hope it's giving some people that maybe aren't normally as vocal, at different levels in the company, a chance to really shine and have their ideas noticed more. TINEKE KEESMAAT: Hopefully, it allows people to feel like they can vocalize. But I'm also hearing an openness of leaders to, actually, hear the ideas differently than they may have in the past. Is that fair? ANGELA BARKAN: Absolutely. I mean, I've been-- so I've been in this business for 20 years and I've never experienced anything like this before, right, and none of my colleagues have. And so years of experience is certainly helpful in navigating. It's also very obvious that this is the first time for all of us in this situation. And so anyone can really come up with a great idea. And there's nothing to compare it to based on experience because this is our first time for everyone. It's everyone's first time in this unique situation. So in some ways, that's kind of very-- it really levels the playing field, but in a way that I feel in a creative industry, especially, is very exciting. TINEKE KEESMAAT: Angi, I love just this idea of leaders being open to new ideas because ideas can come from anywhere. And because there's nothing to compare them to right now, in this context, it does make listening to these ideas all the more important. One of the things that I'm really conscious of is that we are asking a lot of our leaders, in this context-- so you've got people that are having to push bounds within their business organizations. You're having them lead teams where their teams are all going through different psychosocial economic challenges, and then you're putting these leaders in places where they have to work from home, where they've got children or aging parents or issues with things, just day-to-day tasks that are, what, how do I get groceries? And I think, the stress that leaders are under is incredible at this time. And I'm curious, for yourself, how are you personally managing through the challenges of this context? ANGELA BARKAN: Yeah, I mean, it's a great-- it's a great question. It's a lot of trial and error. And-- and, I think, I'm just trying to be kind to myself and put a little less pressure on myself. In general, I'm a pretty regimented person in the sense that I like a schedule, I like a checklist, I like to complete my checklist by the end of the day, things like that. And the reality is it's just not happening now. And so, I think, being forgiving with myself, that, just like everyone else is learning how to navigate this new work situation, so am I. And, I mean, I have a 7-year-old so, certainly, homeschooling has played a role in this now. And also just making sure that we're taking care of ourselves. I mean, one good thing that's happened from this is I've started yoga and I'm forcing myself. The Nike app is now free and I've been forcing myself to do yoga every day. And I'm terrible at it, but I think it's helping a little. TINEKE KEESMAAT: For sure. I've been forcing my husband and I, also, to make sure that every day we get some sort of physical activity. So just for the sanity, right, to keep-- to focus on myself for a moment, but also just to stay active physically. It really does help, I think. ANGELA BARKAN: And I think, also, we can't underestimate that there's a lot of pressure right now. I mean, businesses, overall, are at a very vulnerable place, and it is a little scary. And so, I think, we have to really keep in mind that the job has to happen. We have to try to push and be aggressive and get as much done as we possibly can. But we can't forget the emotional element of all of this, which you mentioned. On our team, alone, someone-- someone's family friend just passed away from the virus. Two people on our team, actually, have the virus. Luckily, it's not serious, but they're dealing with it. And so, I think, everyone just-- just needs to be flexible, but also empathetic and realize that people deal with trauma, which is really what this is. It's a traumatic situation, differently, and just to try to kind of be in tune with people's emotions, especially now. TINEKE KEESMAAT: And just tactically, how are you finding it to pick up on people's emotions through Skype or Zoom or Team? Are you able to detect or is there something that you're doing independent of that to keep a pulse of your individual employees? ANGELA BARKAN: I think, we're been pretty good, even on our Zoom calls, in just checking in with people. Everyone's pretty intuitive, I think, in that way. We've added some fun elements to our Zoom calls. For example, one of the guys on our team has the most amazing vinyl collection, probably, I don't know, he probably has like 10,000, 15,000 records and he likes to sit his chair right in front of the vinyl collection. So every day, we have Steve's record of the day and he picks a very eclectic record out of his-- out of his wall and tells us some interesting obscure facts about the band. And we all laugh about it because no one's ever heard it because it's no obscure. And-- and it's become-- it's just a little thing, but it sort of lightens the conversation a little bit and it just adds an element of, like, OK, we all still work in music. We can all still laugh. We can also have a little bit of fun. Another one of my colleagues, we joke that she's dived into the world of TikTok, which is becoming increasingly important in the music industry. So we all talk about her different TikTok videos every day and we're just trying to connect on different levels. I had a call with one of my colleagues, yesterday, about how we're trying to homeschool our kids. So, I think, it's just trying to find a way to add a human, non-business element to the group calls and then also individual-- just individual check-ins to be, like, hey, what's up? How are you feeling? What's going on? What-- how did your workout go other day? Just, how's your relationship going? Just, like, basic-- basic human things-- basic human interactions, I think, are more important than ever right now. TINEKE KEESMAAT: I love that, just connecting as individuals. And this conversations, not only feel light and fun, but they also show that you care about each other. And, I think, that care is so important at this moment. Do you have any advice for leaders in this moment? So one practical thing that you would advise leaders to do as we navigate through this uncharted time. ANGELA BARKAN: I think, one thing that is really important for everyone to remember, not only-- not only leaders, which is everyone going through this, is, like, this too shall pass. This is a terrible time, but there are some good things that will come out of this. And, I think, we need to embrace the small moments, the small successes, where we're working as a team and things are going well, and just accept the things that aren't-- that aren't going so well and I know that they'll get better. Look, 20 years ago-- 19 years ago, actually, I lived through September 11th, living in New York. Granted, these are very different situations, but that feeling of feeling bewildered, feeling confused, not knowing what tomorrow will bring, all of those feelings, I think, are similar. And we all got through it, and we'll all get through this. And, I think, reminding people that this is just another chapter will be helpful and especially taking care of some of the younger-- younger people on the team, especially in music. I work with a lot of people in their early 20s who are living away from home, whose parents might be in other countries. And I think, just taking a page [INAUDIBLE] out of parenting, I think, and being a good friend and being there for emotional support, is one of the most important things leaders can do right now because that's how you'll inspire people and get the best work out of them [INAUDIBLE] as well. If people are feeling good, then they're motivated to produce. TINEKE KEESMAAT: Angi, this has been incredible. I think, in this conversation, I've just loved the messages about being creative, taking risks, being authentic, connecting as people. It's been truly inspiring to hear. So thank you, first of all, for taking the time, but secondly, for also having the show go on because I know that so many people are connecting to artists and to your work during these moments. So from the bottom my heart, thank you so much. ANGELA BARKAN: Thanks so much for having me. This has been great. TINEKE KEESMAAT: No problem. We do want to finish up with one thing that we do on LeaderLab is we ask our guests a bunch of fun questions, and we do want to wrap up with those. Again, don't think too hard, they are just intended to be fun. ANNOUNCER: And now, let's get to know our guest a little better with some rapid fire questions. TINEKE KEESMAAT: We're going to go with the theme of comforts at home in these times. ANGELA BARKAN: OK. TINEKE KEESMAAT: First question, your go to comfort food? ANGELA BARKAN: Mac and cheese. TINEKE KEESMAAT: Your go to, do you go for a workout or curl up with a book? ANGELA BARKAN: Oh, gosh, curl up with a book, but I'm really trying, really trying to make it a workout. TINEKE KEESMAAT: That's [INAUDIBLE]. One item that you could never live without? ANGELA BARKAN: Netflix, I'm addicted to Tiger King like the rest of the world. TINEKE KEESMAAT: Oh, you're the third person that has told me that in the last two days. That's so funny. ANGELA BARKAN: I mean, it's so crazy. TINEKE KEESMAAT: Your favorite cartoon character? ANGELA BARKAN: My favorite cartoon character is Snoopy. TINEKE KEESMAAT: And your work from home attire: PJ's, fully dressed or a little bit of both? ANGELA BARKAN: Well, I'd love to tell you that I'm wearing a Blazer right now, with a very cool rock and roll t-shirt underneath, but, in fact, I am wearing sweatpants from the Gap. TINEKE KEESMAAT: Lovely. I'm in workout gear so, hopefully, that helps you feel better. Angi, again, this has been truly amazing and I really appreciate you taking the time. ANGELA BARKAN: Thank you, guys, so much. This was really fun. ANNOUNCER: Thank you for joining us today on LeaderLab. LeaderLab is powered by Tiltco, helping exceptional leaders achieve extraordinary results, and the Ivey Academy at Ivey Business School, Canada's home for learning and development. You can learn more about Tiltco and LeaderLab @tiltco.ca. And to find out more about the Ivey Academy, go to iveyacademy.com.            

Running Kubernetes on conventional operating systems is time-consuming and labor-intensive. Today’s guests Andrew Rynhard and Timothy Gerla have engineered a product that attempts to provide a solution to this problem. They call it Talos, and it is a modern OS designed specifically to host Kubernetes clusters, managed by a flexible and powerful API. Talos is completely stripped down to the bare components required to run Kubernetes and get information from the system. It stays updated by keeping time with Kubernetes, but also provides the user with a large degree of control in the event that they might need to update a flag. In this episode, Andrew and Timothy get into some of the mechanics and thought processes behind Talos, telling us why they went with a read-only API, how they handle security concerns on the OS, and how a system like theirs might get adopted by the Kubernetes community and layperson more broadly. They get into the advantages provided by a stripped-down solution for systematizing the use of Kubernetes across communities and running new components through clusters rather than on the OS itself. In a space where most participants are largely operating in the dark, it is a pleasure to see innovations like this display such lasting power so make sure you check out this episode. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Guests: Andrew Rynhard https://twitter.com/andrewrynhard Tim Gerla https://twitter.com/tybstar Hosts: Carlisia Campos Bryan Liles Olive Power Key Points From This Episode: What a Kubernetes OS is: a stripped-down OS that integrates with Kubernetes. The difficulties of managing and getting Kubernetes installed on regular OSs. Why a Kubernetes OS? Less attack surface and OS compatibility issues. What Talos does: quickly makes nodes part of a Kubernetes cluster by being stripped down. How replacing SSH with an API alleviates some users’ security concerns. A command-line interface called OSCTL that allows users to explore the API. What does ‘stripped-down’ mean? Talos runs kubelets and gets information from the OS. The ability to run new components through clusters rather than from the OS. How the Kubernetes OS evolves with Kubernetes but gets separately controlled too. Better integrating into Kubernetes by abstracting OS features into Kubernetes as operators. Security precautions: kernel hardening, SSH and Bash removal, and a read-only OS. Usability of Talos for the average Joe, and its consistency across base platforms. Possibilities for interacting with deeper levels of an OS through an API managed OS. How Talos might become appealing to laypeople: decreasing costs for porting to it. Value gained from switching to a purpose-built OS as something which could outweigh costs. Tendencies to hang onto tried and trusted tech even if its predecessors are superior. Quotes: “To me, it’s just about abstracting away the operating system and not even having to worry about it anymore, and looking at Kubernetes and the entire cluster as an operating system.” — Andrew Rynhard [0:05:00] “As rapid as the technology is changing, you need an operating system that is going to evolve with it or at least the operations intelligence to evolve with Kubernetes right alongside it.” — Andrew Rynhard [0:13:08] “The challenge I think for us and for anybody changing the way that operating systems work is is it better enough than what I have today or what I had before?” — @tybstar [0:26:50] “There’s a lot of companies out there who got us at this point in tech that don’t exist anymore, but if they didn’t do what they did, we would not be here right now.” — @bryanl [0:33:41] Links Mentioned in Today’s Episode: Talos — https://www.talos-systems.com/ Timothy Gerla — http://www.gerla.net/ Timothy Gerla on Twitter — https://twitter.com/tybstar Andrew Ryndhard on LinkedIn —https://www.linkedin.com/in/andrewrynhard/ Andrew Ryndhard on GitHub — https://github.com/andrewrynhard Jed Salazar on LinkedIn — https://www.linkedin.com/in/jedsalazar/ Bryan Liles on LinkedIn — https://www.linkedin.com/in/bryanliles/ Carlisia Campos on LinkedIn — https://www.linkedin.com/in/carlisia/ Red Hat — https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux Arch — https://www.archlinux.org/Debian — https://www.debian.org/ Linux — https://www.linux.org/ Bell Labs — http://www.bell-labs.com/ AT&T — https://www.att.com/ Transcript: EPISODE 20 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your Cloud Native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [INTERVIEW] [00:00:41] CC: Hi, everybody. Welcome back to the Podlets. Today we have a special episode, and on the show, we have a special guest, Andrew Ryndhard. Say hi, Andrew. [00:00:53] AR: Hello, how are you? [00:00:55] CC: We also have Timothy Gerla. Say hi, Tim. [00:00:58] TG: Hi. Thanks for having me. [00:01:00] CC: Yeah. Andrew and Timothy are from Talos. Andrew dropped an issue on our GitHub repo and here we are. It was a great suggestion. What we’re going to talk about today is what they are working on, which is a Kubernetes operating system. We have tons of questions for them for sure. We also have a special participant on the episode today as a co-host, Jed Salazar. Hi, Jed. [00:01:28] JS: Hey, everyone. Jed Salazar here from the CRE team here at VMware. [00:01:31] CC: And Bryan Liles. [00:01:32] BL: Hi. [00:01:33] CC: Hi. And me, Carlisia Campos. Who’d like to get the party started and kick this off? [00:01:41] BL: Oh, I’m here. Let’s throw the gauntlet down. We’re talking about Kubernetes operating systems today. I have an operating system, a Mac, or I have Linux. I can run Kubernetes. What is a Kubernetes operating system and why should I even be thinking about this? [00:01:58] AR: Sure. I’d like to think about Kubernetes operating system as an operating system that has stripped down the absolute bare minimum to run Kubernetes. Everything that is required to run the kubelet, and essentially that’s it, at least in my opinion. It should be super minimal to start with. Second of all, I also think that it should integrate with Kubernetes as well. The combination of just being able to strip down Linux as we know as small as possible and then actually integrating with Kubernetes itself using APIs to figure out things about itself, whatever. I think that that, in my opinion, is what I would call a Kubernetes OS. [00:02:42] BL: Interesting. Okay. Now that we know a little bit about Kubernetes operating systems, and like I said, I’m starting in early today as the devil’s advocate. Now, like I said before, I have a Mac and I have Linux and I have Windows on my desktop. There’re been lots of efforts from lots of people trying to get Kubernetes running up on Ubuntu or Fedora, and it’s cool that you’re trying to slim this down, but really why would I look at a Kubernetes operating system over my Linux that I’m familiar with? I like Ubuntu with Debian. [00:03:17] AR: Sure. That’s a great question. It’s one we get a lot. I like to think that you actually just get less operational overhead when you actually have a Kubernetes-specific operating system. I think that Kubernetes itself is a job, managing it, getting it installed, unfortunately. It’s getting better, but it’s still a job at the end of the day. Having to manage Kubernetes and the operating system, everything that you need to pass compliance on the operating system, get all the packages installed, these are all things that we kind of know that Kubernetes needs already and yet we’re still having to go in and app install whatever we might need to get Kubernetes up and running. The idea with a Kubernetes operating system in my mind is that we should stop worrying about the individual node, the underlying operating system and start looking at Kubernetes as a whole as a giant machine and we just add machines, nodes to this giant machine that give us extra resources. The less that we have to care about the machine or the underlying operating system, the better, in my mind. We get to focus on Kubernetes. Not only that, but because it’s minimal, you get a smaller attack surface. There’re just not things there that you would otherwise have to worry about. I’ve done Kubernetes for three years now and having to go in and worry about updating packages that are just completely unrelated, it’s something that I think we shouldn’t have to do anymore. If you’re dedicated to running your apps and your stack in Kubernetes, then why are we going in and managing the nodes on an individual basis. For that matter, managing things that don’t really have any relevance for running Kubernetes. To me, it’s just about abstracting away the operating system and not even having to worry about it anymore and looking at Kubernetes and the entire whole cluster as an operating system. We can’t really get there if we’re having to worry about the two jobs of managing both at the same time. [00:05:17] JS: Andrew, can I ask a follow up question? [00:05:18] AR: Sure. [00:05:20] JS: I fully agree with all of those statements. I think a general purpose operating system might not be the best job for a specific role, like being a Kubernetes node. As you mentioned, you have to deal with kind of all the various packages that might be beneficial to you if you’re running it for some general purpose. It’s really supposed to be running a workload as a Kubernetes node so you can kind of scope that down. I’m just wondering when you kind of make this pitch or kind of let these folks know, how do you get folks to kind of relinquish their desire to have full control over their operating system from being able to install their own security management processes on it or being a little bit shy about not being able to SSH or kind of use their common patterns of operating system management? [00:06:09] AR: Oh, that’s a great question. I think the biggest thing that I always answer back is – I can take this in two parts. Let me first of all talk about what – People, they do want to run things on the host. My answer always back is can you run it in Kubernetes? Kubernetes is sort of your package manager, if you will. They sit back usually and they’re like, “Hmm. Yeah, I probably could.” If you need to run something on every host, Kubernetes has something for that. It’s a daemon set. Run it on Kubernetes and call it a day. This isn’t something that’s going to work for absolutely everything I imagine. Nothing in the world is like that. But I think for the majority of the use cases out there and for the things that people want to run on the host, you could actually just run it in Kubernetes itself. As far as SSH and for those that don’t really know what we’ve done in Talos, in Talos we’ve actually stripped down just the kernel and a small Go lang binary that’s our – That, basically, its whole goal is to create a Kubernetes cluster or make a node part of a Kubernetes cluster as fast as possible, and that’s really it. We’ve gone so far as ripping out Bash and SSH and we’ve actually replaced that with an API. My answer always back to the SSH question is what is it that you really trying to get out of SSH? 9 times out of 10, it’s, “I want to get information about what’s wrong. I want to do troubleshooting.” If our answer back to them is, “Oh! We have an API for that,” you still at the end of the day – it’s really the information that you’re after. It’s not necessarily that you need SSH to do that. You need a way to get this information and not necessarily have to sit there and wait for a Prometheus metric, see it pulled it every minute. You want something right on the spot. You want to ask a question and you want to get an immediate answer. I feel like we can answer that with an API. That tends to satisfy the desire for wanting SSH most of the time. I mean, as you said, people are still going to want to hold on to it, but I think over time we’re going to have to educate people that this is a better way. It’s a read-only API that gives operations engineers a way to get that information that they would otherwise get by SSH-ing and asking via Unix utilities what you want to know. [00:08:27] CC: When you say an API, are you also giving them a command line tool or like in the case of Talos, or only an actual API? [00:08:37] TG: Yeah, we do provide a command line interface to the API. It’s called OSCTL and it basically wraps our API, and our intention is that that will be used for exploration of the system, automation through scripting languages, etc. Then as you get more sophisticated with your environment, you might begin to build your own tools that interact directly with that API. [00:08:56] CC: Cool. Yeah, this is a really cool subject. I wasn’t even aware that Kubernetes operating system was a thing until really recently, and I don’t remember how I came across it. One question I have is, Andrew, you were saying, “Well, we strip down Kubernetes to the bare minimum.” How opinionated is it in your case in specific? When you say you – it’s a stripped down version to the bare minimum, this statement of bare minimum, would there be a consensus in the community that, yes, this set of functionality is the bare minimum? Is it your opinion of what the bare minimum should be? [00:09:38] AR: Sure. I think at the absolute bare minimum, we need to run the kubelet. In my mind, that’s really all we need, but you still have this practical issue of, like you said, you need to get information off that machine. You need to be able to kind of manage Kubernetes without having to need Kubernetes as a chicken and egg’s problem. That’s where the API was actually born. When I started Talos, I actually just built a very minimal strip down route-fs that all that did was run the kubelet. But figuring out why the kubelet wasn’t running successfully obviously was not very easy. I figured, “You know what? Let’s put an API in front of this. I want to keep this as minimal as possible. I want to keep this read-only.” I threw an API in front of it. I think you need two things, really. You need to have what’s required by the kubelet. You need a CNI. You need all the utilities that the kubelet will run and you also need a way to query the system. If that is – If in the case of other operating systems that are minimal operating systems, they have decided to do SSH and all the classic utilities that we all know and love, we went another route with an API. But I don’t think the operating system, the route-fs should have any more than what’s required by the kubelet. That would be the pie in the sky dream right there. [00:11:01] CC: The two questions that come to my mind are if I wanted to add Kubernetes components to that, would it be possible? If I wanted to add anything to the operating system, would it possible? I think the second question you already answered, which is, well, if you need to run – Correct me if I’m wrong. If you need to run something on the operating system that’s not there, you can run it in the actual cluster. [00:11:27] AR: Yeah, that’s the idea, is that Kubernetes gives us the APIs to do – We could schedule to specific nodes. We can schedule to a class of nodes. We can schedule to every single node. I think that you can actually handle a lot of the use cases out there for any kind of application with Kubernetes itself. I think that that’s really strong because you get one single consistent API in managing your infrastructure. I want to deploy applications for this team or this team. At the end of the day, everything is just declarative and Kubernetes will make it happen. You don’t have to worry about the scheduling and all of these different things. The only thing that the operating system is concerned about is making that machine available to the Kubernetes cluster. [00:12:10] BL: This idea of slimmed down operating systems, it’s not a new one. CoreOS was doing this years ago. One issue that CoreOS ran into was like, “Well, what’s current?” Well, it depends on what stream you’re on. How do you manage keeping everything up-to-date? [00:12:28] AR: Our goal is to keep pace with Kubernetes essentially. I know that, traditionally, there’s long-term support and there’s all these different ways of releasing different versions of an operating system, but Kubernetes isn’t really there yet. There is no notion that I know of of LTS in Kubernetes yet. There’s just, I believe, it’s N-2 or something like that where they actually offer official support. I think that the operating system is bound to that. I think that it needs to follow Kubernetes as close as possible. There’re constantly different feature gates being opened up. There’re things being graduated to GA. I think especially at this time right now, as rapid as the technology is changing, you need an operating system that is going to evolve with it or at least the operations intelligence to evolve with Kubernetes right alongside it. [00:13:20] BL: So that brings up an interesting point. I mean, there are two things here. There’s the operating system itself and there’s Kubernetes. Do they upgrade in lockstep or are they upgraded separately? [00:13:29] AR: I could only speak for ourselves. There are people that I think they actually have upgrades kind of be one and the same, where the operating system and a Kubernetes upgrade both happen. We’ve decided started to go the other route where we actually want to evolve our APIs sort of independently, but then give you a way to still manage Kubernetes on its own. We’ve actually done self-hosted Kubernetes. In Talos, we’ll actually bootstrap a lightweight control plane, small control plane and then we’ll spin up another control plane using the Kubernetes API. Then now, Kubernetes upgrades simply look like a kubectl edit. I’m going to update my daemon step for my API server. Then from there, you will have to basically update the kub. We use hyperkub for the kubelet. You have to tell Talos, “Use this kubelet image next time you boot.” We’ve separated the two I think for good reason. I think that the two should be able to evolve independently to give a little bit more power back to the user. If you combine them, if you couple them really closely, it becomes really, really opinionated. I think we should at least support what Kubernetes supports, and that’s the N-2 and leave it up to the user to kind of configure Kubernetes, but we still have same best practices out of the box. [00:14:54] BL: Yeah, that makes sense, because yesterday, what did we get? We got a Kubernetes 1.15.10, and I don’t know 16, but we got 1.17.3 yesterday too. You might not want to move, because you might not – 1.17 introduced a whole bunch of deprecations and for custom resource definition. You’re not ready to move yet. We’re on beta 1 for a while for CRDs. I totally see why you had moved that direction. [00:15:20] AR: Yeah, that’s exactly it. We can’t impose too much opinion, but I think that we should drive – The opinion at least up until like, “Hey, don’t worry about what’s on this machine. I’m going to make it a Kubernetes node for you. Just tell me which version you want.” I think that’s where we should draw the boundary and then we should still give the controls back to the user as far as what flags do I want to specify. What kind of feature gates? All these various things that you don’t get out of a lot of the different managed products out there. Hopefully we’ll be tittering right on the line of having that convenience of managed but still giving you that power and flexibility to update a flag if you need to. [00:16:04] CC: This episode is so in the style of an interrogation. It’s hilarious. [00:16:08] BL: That’s me. I’m digging in. [00:16:09] CC: I feel like – No. We are all digging in. It’s just because – At least speaking from myself. I’m super curious. I wanted to ask you, Andrew, at the beginning you were saying that a Kubernetes operating system needs to integrate with Kubernetes and I was sitting here thinking, “Operate? It’s supposed to be Kubernetes.” What did you have in mind when you said that? Did you mean to be able to interface with another Kubernetes cluster? Was that what you meant? [00:16:36] AR: Not quite. What I meant by that is there’s this really powerful thing that Kubernetes gives us in CRDs and this idea of operators or controllers. If you can actually have a way to use an operator controller, say, for upgrading your operating system, which we have in Talos, it’s just an upgrade operator lives in Kubernetes and knows how to talk to Kubernetes and it knows how to talk to our API and sort of orchestrate upgrades across the board. Part of that is, for example, when you receive the upgrade API on a Talos node, it actually is aware, “Hey, I’m running Kubernetes. I’m going to cordon myself, because I know I’ve gotten this and I know that I’m not going to be able to schedule workload on me.” I think that that’s just one example, but we could probably take that a lot farther one day. But I would like to see everything that we know and love about our operating systems today essentially be abstracted and pushed up into Kubernetes as operators. There’s a lot of power in that where you can actually orchestrate things, like I said, like upgrades. I think that that’s one example of how we can integrate better with Kubernetes as how an operating system should, at least. [00:17:45] CC: Got you. [00:17:46] JS: I was wondering if we can kind of maybe just pivot a little bit, like maybe to satisfy my own curiosities, but I was kind of hoping we could talk a little bit about like some of the selling features. Imagine if I’m a hardened sys admin or security team and basically someone comes up and says, “Hey, I want to run this Kubernetes operating system.” Knowing what I know about the state of security today and operating systems, there’s a lot of efforts to basically kind of contain things. No pun intended, but we have user space operates out of some type of sandbox. We have seccomp to limit sys calls. How does Talos approach security maybe like philosophically or maybe even down to the implementation details? What is security in Talos look like? [00:18:33] AR: Yeah. Again, our goal is to basically – We want people to forget about the operating system. But to forget about the operating system, you have to know it’s secure. You have to go to great lengths to secure that because you can’t forget about it for that reason. We actually go down to the kernel, we actually apply what’s called the kernel self-protection project. We basically try to harden the kernel, and at boot time, we do a bunch of checks to make sure that your kernel is running at least most of those configurations. I think we have a little bit of work to do as far as enforcing all of them. But we do some checks to ensure that your kernel is compatible with KSPP, for example. That alone has a ton of benefits to it. It’s a statically compiled kernel so you it can’t do any kernel module loading and stuff like that. That’s completely prohibited. That alone just kind of cuts off a lot of security issues in itself. Then going up the stack further, we’ve actually stripped out SSH. We stripped out Bash. So you have nothing that you can really log on to anymore. Again, that’s just flat out removes a lot of – A whole category of potential attacks potentially. Going even further than that, we’ve actually have Talos running completely out of RAM and it’s a squash-fs. So it’s a read-only file system. The only thing that actually uses a disk is the kubelet. The idea is that we want to make the operating system, again, just have it go away. Having it read-only I think is a really strong thing, and squash-fs in particular, because you can’t remount it, rewrite if you’re a user or something like that. Then up in Kubernetes we actually – Out of the box, we try to deploy it with all of the security best practices, the CIS benchmarks and all of that. We go to all the way from the kernel, to our user LAN and even to Kubernetes itself. We try to bring out security best practices out of the box. I think that’s something I’d love to see for Kubernetes itself upstream, but for now that’s what we’re doing. [00:20:33] BL: Can we go back to the interrogation? No. Let’s not go back to an interrogation. Thinking of – If we take the concept of a Kubernetes operating system, that can be updated in a different cadence, then the Kubernetes running on it – Who is Talos for? Who does it make – Could Joe as a neophyte or someone who doesn’t really know the space, will this make their life any easier or is there a special set of expertise that we would need to be fruitful with this? [00:21:06] TG: I think from our perspective, we hope that everybody who uses Kubernetes would find something useful in Talos, or a system like Talos. Number one, I think Talos would be a great way to get started on your laptop or workstation. I got some basic features to standup a small Kubernetes cluster there. That’s one place to start. As you move further into production, I think that a Kubernetes OS-based platform would be particularly useful in an environment where you might have multiple clusters spread across different geographical locations, spread across different teams. Maybe spread across different hosting environments. We’ve talked to a number of folks who have been running Kubernetess in production for a couple of years now, and these clusters kind of come up organically within a larger organization in different areas, doing different things for the business, managed by different teams. Now that a little bit of time has passed, these organizations are realizing that, “Hey, we’ve got kind of a Kubernetes sprawl problem. We have this team over here on Amazon managing and running Kubernetes one way. We have a separate team managing and running Kubernetes a different way over here on a different kind of platform.” I think anything that – anywhere where we can drive some consistency across the tooling, consistency across the base platforms would be useful. We also think that the minimal aspect of our system and some of the design decisions we’ve made around security and make it particularly useful in maybe a regulated environment. I think that that claim would hold true for any sort of special purpose operating system or minimal operating system designed for a specific task. [00:22:35] BL: Interesting. Just thinking about a concept of a Kubernetes operating system, what’s next? I’m not asking what’s next from Talos, but given all the opportunity all the time and all the knowledge. What should we be doing that we’re not doing right now? [00:22:49] AR: Specifically around operating systems or Kubernetes? [00:22:52] BL: Well, you know what? You can start with operating systems. I mean, you can go to Kubernetes and then we’ll see if our lists match. [00:22:57] AR: That’s a good question. Right off the bat, I’m going to say I don’t really know. I think this is new space. I think that we have a big task in front of us already in getting people to use these kinds of operating systems, hopefully not too big of a task. I’m hoping to see – Because you find these big companies, “Oh! We can’t do this. We can’t do that,” because getting a new OS is hard. I think we first of all need to win people over on just these even more minimal operating systems beyond what CoreOS has done. Personally, I don’t know if I could answer that question honestly without just owing something. [00:23:33] TG: I’ve got a thought here. One of the things that I’m really interested in beyond just Kubernetes and beyond just the operating system – what is computing going to look like in 5, 10, 15 years? I don’t know if Kubernetes is going to be around. I’m kind of a tech-cynic, right? I’ve seen a lot of fads in my career and things that pop up and are very popular for a couple of years and then sort of disappear. I don’t think Kubernetes is one of those. I think Kubernetes and the concepts and the layers of abstraction that Kubernetes has provided, all of that will remain useful and powerful in distant future whether or not it’s called Kubernetes or if it’s called something else, some new paradigm. But what I’m really interested in is seeing what can we do with this idea of an API-managed OS? If you look at the general purpose operating systems out there, some aspects of the system might expose an API. But for the most part, you’re still interacting and interfacing with this system like you were 30 years ago, 35, 40 years ago even. That’s fine. What works works, but everything else today has an API. Kubernetes has a powerful and extensible API and I think that your operating system should have something similar, something comparable, something that you can interact with using the same tools and the same processes and the same ideas that you can at the top of the stack and move some of those concepts down to the host OS level where we’re talking about today. [00:24:51] CC: This brings up a point that I’m so curious about, not only the idea of having a Kubernetes operating system, but any idea that is new that you were just talking about, Tim, is – So what works works. For example, every year or every couple of years, I am evaluating a new code editor or I am evaluating a new note taking app, or do-to-list app, those three things. I’m continuously finding something to reevaluate because what I have has never worked for me just the way I think. Actually, recently I found a couple of things that are really good. In any case, the thing is they just never worked for years. They’re very limited. They don’t match my thinking. But operating system, I would never – Well, I’m not an administrator also, but just like from having my own laptops forever, I’m not going to go out there. That’s not true either, but I was going to say I’m not going to go out there and try a new operating system to see if it’s offering that I already have, then it might be better for me. But that’s not true, because I have done that many times too. So never mind. But I think the idea of my question is stance, is how are you communicating to people out there that, “Hey, there is this new thing that maybe it’s working for you – Maybe you think it’s working for you, but you just don’t know that there is a new different way of doing.” When you do try to do that, how are people responding? I mean, of course, there are those cases where people just know they get it and they immediately resonate with them. But I’m talking about the people who like might benefit from this but they don’t quite grasp. How do you break through that barrier? [00:26:38] TG: Sure. Maybe the lay majority. [00:26:40] CC: Yeah, and how are people responding? [00:26:42] TG: Yeah. The great thing about Talos is that people understand pretty immediately what it is, how it works and why we’ve done it. The challenge I think for us and for anybody changing the way that operating systems work. Is it better enough than what I have today or what I had before? Is it worth the switch in costs? I think that switching cost is something that’s pretty well understood in the industry. People have gone through this process and they’ve moved from virtualization to containers, from Docker to Kubernetes, etc. They understand that process and they understand there’s a technical cost. There’s a people cost, etc. We have to show that value. I think that progress in our industry is incremental. Our industry is young. We’re not building bridges. We’re not at the level of like the internal combustion engine where the engineering is understood and we know how to build it and we know how to make it so that it doesn’t fall over and explode. Clearly, we’re not quite there yet in the broader world of computing. I think anywhere where we can show a little bit of incremental improvement where we can tackle one narrow slice of a problem and make it a little bit better and get to a point where computing is just a little bit safer and a little bit easier and a little bit faster. I think that’ll be a pretty compelling argument and there’s a lot of details involved and we have to talk about how do you get your applications from one operating system to the next? 15 years ago, it may have been a very big ask to ask someone to port their enterprise application from one operating system to another. They’re so inextricably linked. There’re a lot of connections between the OS and the applications, but today, we have these levels of abstraction. We have containers. We have the Kubernetes orchestration mechanisms and I think that switching cost is going down every release of Kubernetes and every step along the way as people change the way that applications are deployed that switching cost gets a little bit cheaper. It will be easier for us to prove that the value you gain by moving to a purpose-built operating system is greater than the switching cost. [00:28:41] CC: Very good points. [00:28:42] JS: I feel like there’s a lot of emphasis and focus on the move over. The first steps toward migrating to something new. There’s a lot of emphasis on bootstrapping a cluster. There’s a lot of emphasis on how do I get started. I’m part of a team called customer reliability engineering and we see operators running Kubernetes environments that are durable and have been in the field for many years. I think that there’s kind of a hidden cost in these day two operations where, like today, to effectively be a Kubernetes operator, you need to also have a great deal of understanding of Linux internal operating systems or Linux operating systems internals. These are abstractions on top, but sometimes those abstractions are leaky. So you need to be able to parse IP tables rules. You need to be able to understand how traffic gets routed, all of these aspects of it. I’m just wondering how do we kind of get folks shifted from this mindset of I’m going to start with something that’s general purpose and then I’m going to basically make it do what I want it to do by making all of these configuration changes and installing things on top of it to kind of make that not general purpose, but kind of specific focus on it and kind of get people to move back more fundamentally and think, “Well, what if we just started with something that is strictly for running workloads?” We don’t have to worry about installing a security suite on top of this or making this configuration change or hardening requirements or what have you. We’re fundamentally in a better place because we’ve started with something that’s arguably more secure. [00:30:21] BL: You know that – I mean, I’m old. I’m old now. I’m realizing this. When I started – Back in my day when we started with Linux, we went through this whole thing of Linux installers and there’re many iterations of Linux installers and it depended on, “Well, did you like what Red Hat was doing? Did you like what Debian project was doing? Oh! Did you like what Arch was doing? Oh! Did you want to do it yourself? Do you want to merge the world with gen 2?” Really, we come to this point now, no one ever talks about Linux installers anymore. You just put it on there. I think what I’m getting at is that we don’t actually know what we want. I mean we say that we want it to be simpler. We say we want it to be more secure, but we don’t know. Only time will tell, and I think it’s going to be a lot of chipping away at problems. Then people who are wanting to have the bold ideas are saying, “I’m going to out there and create a Kubernetes operating system.” In reality, it may work. We hope it works, or it may not work, but at least we gained just a little tiny bit more knowledge on how we want to run this thing. I think – And I’ll just say one more last thing, is that if you look at like Bell Labs, Bell Labs created the vacuum tube, and then like 20 years later, 20 or 30 years later, they created the transistor, twice actually. It took a long time to get the vacuum tube out because it kind of just worked and they just said, “We can’t throw it back. We just can’t throw that away.” Maybe we’re seeing a lot of that in Kubernetes. We’re holding on to some good things even though some greater things are going to come, but it might not be here this year or next year. It might be 18 months. It might be 24 months. We just got to really pay attention to that. [00:32:02] AR: Brian, when you said you were old, I was going to shake my head internally and then you brought up the vacuum tube and I’m like, “Okay.” [00:32:07] BL: I mean, I’m not that old. [00:32:09] JS: Yeah, I think that’s a good point, Brian. The thing I like to point out is the allegory of the cave. People have been living a certain way for so long they think that these shadows are real and they just know that way of life until some crazy comes along and says, “Hey, there’s a whole world out there,” and no one believes him. I think we just need to do – Like you said, we just need to do it. When you just create and make it happen and hopefully educate people in the process and just keep chipping away at it. Do the good work. [00:32:38] BL: That’s the important piece and that was the power of Bell Labs. You probably can tell. I just read a book about Bell Labs. I’m an expert now. But that was the power of Bell Labs. They didn’t just focus on making product for AT&T. They focused on changing the world, like literally. Who creates a transistor if you knew what one was? You just don’t create that. That’s like some really crazy stuff. I try to bring the parallel back to what we’re doing here. We can’t just create this perfect Kubernetes thing, because really, we don’t know what it is. I mean, we can be smart and say, “Well, it needs to be secure. It needs to be networking,” and all these stuff. But you know what? We don’t even have cgroups v2 support yet. We don’t even know where we are. Let’s figure out – Let’s just keep going down the path, but we will suss out these better patterns. [00:33:23] CC: Yeah, I like that. [00:33:24] BL: That’s it. It is incremental. Here’s the crazy part, and this is the real tough part. You know what? It is incremental, and reality says that not everybody can win. Don’t take your failures as a loss. Take them as, “well, maybe we shouldn’t have done that,” and keep on moving forward because there’s a lot of companies out there who got us at this point in tech that don’t exist anymore, but if they didn’t do what they did, we would not be here right now. It’s not [inaudible 00:33:52]. [00:33:53] CC: Why are we talking about failures? [00:33:55] BL: I’m sorry. It’s the ultimate success. [00:34:01] CC: Oh gosh! Let’s not end the show in such a downer. [00:34:04] BL: No. That’s a happy point though. Let me put the bow on the happy point and then I will stop talking. The thing is, is it’s not the glass is half empty. It is glass is half full. The path to success is littered with failure and it’s not a bad thing. It’s a good thing, because it’s good that we can continue making those failures because we know they lead to successes. That is actually a happy thing. [00:34:29] CC: I wonder if Andrew and Tim want to do a little bit of interrogating of us. I think that would be fair. [00:34:36] AR: I wouldn’t know what to interrogate you guys about. [00:34:40] CC: Well, we are coming up at the top of the hour. So it’s time to say goodbye. It was great having you, Andrew, and you, Tim, on the call. Jed, thank you for participating as well. I think it was very informative. With that, I will say, until next week. Bye everybody. [00:34:59] TG: Bye. Thanks for having us. [00:35:00] CC: My pleasure. [00:35:00] AR: Bye-bye. Thank you. [00:35:02] JS: Thank you. Bye. [END OF INTERVIEW] [0:35:05.3] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

Today on the show we are very lucky to be joined by Chris Umbel and Shaun Anderson from Pivotal to talk about app transformation and modernization! Our guests help companies to update their systems and move into more up-to-date setups through the Swift methodology and our conversation focusses on this journey from legacy code to a more manageable solution. We lay the groundwork for the conversation, defining a few of the key terms and concerns that arise for typical clients and then Shaun and Chris share a bit about their approach to moving things forward. From there, we move into the Swift methodology and how it plays out on a project before considering the benefits of further modernization that can occur after the initial project. Chris and Shaun share their thoughts on measuring success, advantages of their system and how to avoid roll back towards legacy code. For all this and more, join us on The Podlets Podcast, today! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Josh Rosso Duffie Cooley Olive Power Key Points From This Episode: A quick introduction to our two guests and their roles at Pivotal. Differentiating between application organization and application transformation. Defining legacy and the important characteristics of technical debt and pain. The two-pronged approach at Pivotal; focusing on apps and the platform. The process of helping companies through app transformation and what it looks like. Overlap between the Java and .NET worlds; lessons to be applied to both. Breaking down the Swift methodology and how it is being used in app transformation. Incremental releases and slow modernization to avoid roll back to legacy systems. The advantages that the Swift methodology offers a new team. Possibilities of further modernization and transformation after a successful implementation. Measuring success in modernization projects in an organization using initial objectives. Quotes: “App transformation, to me, is the bucket of things that you need to do to move your product down the line.” — Shaun Anderson [0:04:54] “The pioneering teams set a lot of the guidelines for how the following teams can be doing their modernization work and it just keeps rolling down the track that way.” — Shaun Anderson [0:17:26] “Swift is a series of exercises that we use to go from a business problem into what we call a notional architecture for an application.” — Chris Umbel [0:24:16] “I think what's interesting about a lot of large organizations is that they've been so used to doing big bang releases in general. This goes from software to even process changes in their organizations.” — Chris Umbel [0:30:58] Links Mentioned in Today’s Episode: Chris Umbel — https://github.com/chrisumbel Shaun Anderson — https://www.crunchbase.com/person/shaun-anderson Pivotal — https://pivotal.io/ VMware — https://www.vmware.com/ Michael Feathers — https://michaelfeathers.silvrback.com/ Steeltoe — https://steeltoe.io/ Alberto Brandolini — https://leanpub.com/u/ziobrando Swiftbird — https://www.swiftbird.us/ EventStorming — https://www.eventstorming.com/book/ Stephen Hawking — http://www.hawking.org.uk/ Istio — https://istio.io/ Stateful and Stateless Workload Episode — https://thepodlets.io/episodes/009-stateful-and-stateless/ Pivotal Presentation on Application Transformation: https://content.pivotal.io/slides/application-transformation-workshop Transcript: EPISODE 19 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.0] CC: Hi, everybody. Welcome back to The Podlets. Today, we have an exciting show. It's myself on, Carlisia Campos. We have our usual guest hosts, Duffie Cooley, Olive Power and Josh Rosso. We also have two special guests, Chris Umbel. Did I say that right, Chris? [0:01:03.3] CU: Close enough. [0:01:03.9] CC: I should have checked before. [0:01:05.7] CU: Umbel is good. [0:01:07.1] CC: Umbel. Yeah. I'm not even the native English speaker, so you have to bear with me. Shaun Anderson. Hi. [0:01:15.6] SA: You said my name perfectly. Thank you. [0:01:18.5] CC: Yours is more standard American. Let's see, the topic of today is application modernization. Oh, I just found out word I cannot pronounce. That's my non-pronounceable words list. Also known as application transformation, I think those two terms correctly used alternatively? The experts in the house should say something. [0:01:43.8] CU: Yeah. I don't know that I would necessarily say that they're interchangeable. They're used interchangeably, I think by the general population though. [0:01:53.0] CC: Okay. We're going to definitely dig into that, how it does it not make sense to use them interchangeably, because just by the meaning, I would think so, but I'm also not in that world day-to-day and that Shaun and Chris are. By the way, please give us a brief introduction the two of you. Why don't you go first, Chris? [0:02:14.1] CU: Sure. I am Chris Umbel. I believe it was probably actually pronounced Umbel in Germany, but I go with Umbel. My title this week is the – I think .NET App Transformation Journey Lead. Even though I focus on .NET modernization, it doesn't end there. Touch a little bit of everything with Pivotal. [0:02:34.2] SA: I'm Shaun Anderson and I share the same title of the week as Chris, except for where you say .NET, I would say Java. In general, we play the same role and have slightly different focuses, but there's a lot of overlap. [0:02:48.5] CU: We get along, despite the .NET and Java thing. [0:02:50.9] SA: Usually. [0:02:51.8] CC: You both are coming from Pivotal, yeah? As most people should know, but I'm sure now everybody knows, Pivotal was just recently as of these date, which is what we are? End of January. This episode is going to be a while to release, but Pivotal was just acquired by VMware. Here we are. [0:03:10.2] SA: It's good to be here. [0:03:11.4] CC: All right. Somebody, one of you, may be let's say Chris, because you've brought this up, how this application organization differs from application transformation? Because I think we need to lay the ground and lay the definitions before we can go off and talk about things and sound experts and make sure that everybody can follow us. [0:03:33.9] CU: Sure. I think you might even get different definitions, even from within our own practice. I'll at least lay it out as I see it. I think it's probably consistent with how Shaun's going to see it as well, but it's what we tell customers anyway. At the end of the day, there are – app transformation is the larger [inaudible] bucket. That's going to include, say just the re-hosting of applications, taking applications from point A to some new point B, without necessarily improving the state of the application itself. We'd say that that's not necessarily an exercise in paying down technical debt, it's just making some change to an application or its environment. Then on the modernization side, that's when things start to get potentially a little more architectural. That's when the focus becomes paying down technical debt and really improving the application itself, usually from an architectural point of view and things start to look maybe a little bit more like rewrites at that point. [0:04:31.8] DC: Would you say that the transformation is more in-line with re-platforming, those of you that might think about it? [0:04:36.8] CU: We'd say that app transformation might include re-platforming and also the modernization. What do you think of that, Shaun? [0:04:43.0] SA: I would say transformation is not just the re-platforming, re-hosting and modernization, but also the practice to figure out which should happen as well. There's a little bit more meta in there. Typically, app transformation to me is the bucket of things that you need to do to move your product down the line. [0:05:04.2] CC: Very cool. I have two questions before we start really digging to the show, is still to lay the ground for everyone. My next question will be are we talking about modernizing and transforming apps, so they go to the clouds? Or is there a certain cut-off that we start thinking, “Oh, we need to – things get done differently for them to be called native.” Is there a differentiation, or is this one is the same as the other, like the process will be the same either way? [0:05:38.6] CU: Yeah, there's definitely a distinction. The re-platforming bucket, that re-hosting bucket of things is where your target state, at least for us coming out of Pivotal, we had definitely a product focus, where we're probably only going to be doing work if it intersects with our product, right? We're going to be doing both re-platforming targeted, say typically at a cloud environment, usually Cloud Foundry or something to that effect. Then modernization, while we're usually doing that with customers who have been running our platform, there's nothing to say that you necessarily need a cloud, or any cloud to do modernization. We tend to based on who we work for, but you could say that those disciplines and practices really are agnostic to where things run. [0:06:26.7] CC: Sorry, I was muted. I wanted to ask Shaun if you wanted to add to that. Do you have the same view? [0:06:33.1] SA: Yeah. I have the same view. I think part of what makes our process unique that way is we're not necessarily trying to target a platform for deployment, when we're going through the modernization part anyway. We're really looking at how can we design this application to be the best application it can be. It just so happens that that tends to be more 12-factor compliant that is very cloud compatible, but it's not necessarily the way that we start trying to aim for a particular platform. [0:07:02.8] CC: All right. If everybody allows me, after this next question, I'll let other hosts speak too. Sorry for monopolizing, but I'm so excited about this topic. Again, in the spirit of understanding what we're talking about, what do you define as legacy? Because that's what we're talking about, right? We’re definitely talking about a move up, move forwards. We're not talking about regression and we're not talking about scaling down. We're talking about moving up to a modern technology stack. That means, that implies we're talking about something that's legacy. What is legacy? Is it contextual? Do we have a hard definition? Is there a best practice to follow? Is there something public people can look at? Okay, if my app, or system fits this recipe then it’s considered legacy, like a diagnosis that has a consensus. [0:07:58.0] CU: I can certainly tell you how you can't necessarily define legacy. One of the ways is by the year that it was written. You can certainly say that there are certainly shops who are writing legacy code today. They're still writing legacy code. As soon as they're done with a project, it's instantly legacy. There's people that are trying to define, like another Michael Feathers definition, which is I think any application that doesn't have tests, I don't know that that fits what – our practice necessarily sees legacy as. Basically, anything that's occurred a significant amount of technical debt, regardless of when the application was written or conceived fits into that legacy bucket. Really, our work isn't necessarily as concerned about whether something's legacy or not as much as is there pain that we can solve with our practice? Like I said, we've modernized things that were in for all intents and purposes, quite modern in terms of the year they were written. [0:08:53.3] SA: Yeah. I would double down on the pain. Legacy to us often is something that was written as a prototype a year ago. Now it's ready to prove itself. It's going to be scaled up, but it wasn't built with scale in mind, or something like that. Even though it may be the latest technology, it just wasn't built for the load, for example. Sometimes legacy can be – the pain is we have applications on a mainframe and we can't find Cobol developers and we're leasing a giant mainframe and it's costing a lot of money, right? There's different flavors of pain. It also could be something as simple as a data center move. Something like that, where we've got all of our applications running on Iron and we need to go to a virtual data center somewhere, whether it's cloud or on-prem. Each one of those to us is legacy. It's all about the pain. [0:09:47.4] CU: I think is miserable as that might sound, that's really where it starts and is listening to that pain and hearing directly from customers what that pain is. Sounds terrible when you think about it that you're always in search of pain, but that isn't indeed what we do and try to alleviate that in some way. That pain is what dictates the solution that you come up with, because there are certain kinds of pain that aren't going to be solved with say, modernization approach, a more a platformed approach even. You have to listen and make sure that you're applying the right medicine to the right pain. [0:10:24.7] OP: Seems like an interesting thing bringing what you said, Chris, and then what you said earlier, Shaun. Shaun you had mentioned the target platform doesn't necessarily matter, at least upfront. Then Chris, you had implied bringing the right thing in to solve the pain, or to help remedy the pain to some degree. I think what's interesting may be about the perspectives for those on this call and you too is a lot of times our entry points are a lot more focused with infrastructure and platform teams, where they have these objectives to solve, like cost and ability to scale and so on and so forth. It seems like your entry point, at least historically is maybe a little bit more focused on finding pain points on more of the app side of the house. I'm wondering if that's a fair assessment, or if you could speak to how you find opportunities and what you're really targeting. [0:11:10.6] SA: I would say that's a fair assessment from the perspective of our services team. We're mainly app-focused, but it's almost there's a two-pronged approach, where there's platform pain and application pain. What we've seen is often solving one without the other is not a great solution, right? I think that's where it's challenging, because there's so much to know, right? It's hard to find one team or one person who can point out the pain on both sides. It just depends on often, how the customer approaches us. If they are saying something like, “We’re a credit card company and we're getting our butts kicked by this other company, because they can do biometrics and we can't yet, because of the limitations of our application.” Then we would approach it from the app-first perspective. If it's another pain point, where our operations, day two operations is really suffering, we can't scale, where we have issues that the platform is really good at solving, then we may start there. It always tends to merge together in the end. [0:12:16.4] CU: You might be surprised how much variety there is in terms of the drivers for people coming to us. There are a lot of cases where the work came to us by way of the platform work that we've done. It started with our sister team who focuses on the platform side of things. They solve the infrastructure problems ahead of us and then we close things out on the application side. We if our account teams and our organization is really listening to each individual customer that you'll find that there – that the pain is drastically different, right? There are some cases where the driver is cost and that's an easy one to understand. There are also drivers that are usually like a date, such as this data center goes dark on this date and I have to do something about it. If I'm not out of that data center, then my apps no longer run. The solution to that is very different than the solution you would have to, "Look, my application is difficult for me to maintain. It takes me forever to ship features. Help me with that." There's two very different solutions to those problems, but each of which are things that come our way. It's just that former probably comes in by way of our platform team. [0:13:31.1] DC: Yeah, that’s an interesting space to operate in in the application transformation and stuff. I've seen entities within some of the larger companies that represent this field as well. Sometimes that's called production engineering or there are a few other examples of this that I'm aware of. I'm curious how you see that happening within larger companies. Do you find that there is a particular size entity that is actually striving to do this work with the tools that they have internally, or do you find that typically, most companies are just need something like an application transformation so you can come in and help them figure out this part of it out? [0:14:09.9] SA: We've seen a wide variety, I think. One of them is maybe a company really has a commitment to get to the cloud and they get a platform and then they start putting some simple apps up, just to learn how to do it. Then they get stuck with, “Okay. Now how do we with trust get some workloads that are running our business on it?” They will often bring us in at that point, because they haven't done it before. Experimenting with something that valuable to them is — usually means that they slow down. There's other times where we've come in to modernize applications, whether it's a particular business unit for example, that may have been trying to get off the mainframe for the last two years. They’re smart people, but they get stuck again, because they haven't figured out how to do it. What often happens and Chris can talk about some examples of this is once we help them figure out how to modernize, or the recipes to follow to start getting their systems systematically on to the platform and modernize, that they tend to like forming a competency area around it, where they'll start to staff it with the people who are really interested and they take over where we started from. [0:15:27.9] CU: There might be a little bit of bias to that response, in that typically, in order to even get in the door with us, you're probably a Fortune 100, or at least a 500, or government, or something to that effect. We're going to be seeing people that one, have a mainframe to begin with. Two, would have say, capacity to fund say a dedicated transformation team, or to build a unit around that. You could say that the smaller an organization gets, maybe the easier it is to just have the entire organization just write software the modern way to begin with. At least at the large side, we do tend to see people try to build a – they'll use different names for it. Try to have a dedicated center of excellence or practice around modernization. Our hope is to help them build that and hopefully, put them in a position that that can eventually disappear, because eventually, you should no longer need that as a separate discipline. [0:16:26.0] JR: I think that's an interesting point. For me, I argue that you do need it going forward, because of the cognitive overhead between understanding how your application is going to thrive on today's complex infrastructure models and understanding how to write code that works. I think that one person that has all of that in their head all the time is a little too much, a little too far to go sometimes. [0:16:52.0] CU: That's probably true. When you consider the size the portfolios and the size of the backlog for modernization that people have, I mean, people are going to be busy on that for a very long time anyway. It's either — even if it is finite, it still has a very long life span at a minimum. [0:17:10.7] SA: At a certain point, it becomes like painting the Golden Gate Bridge. As soon as you finish, you have to start again, because of technology changes, or business needs and that thing. It's probably a very dynamic organization, but there's a lot of overlap. The pioneering teams set a lot of the guidelines for how the following teams can be doing their modernization work and it just keeps rolling down the track that way. It may be that people are busy modernizing applications off of WebLogic, or WebSphere, and it takes a two years or more to get that completed for this enterprise. It was 20, 50 different projects. To them, it was brand-new each time, which is cool actually to come into that. [0:17:56.3] JR: I'm curious, I definitely love hear it from Olive. I have one more question before I pass it out and I think we’d love to hear your thoughts on all of this. The question I have is when you're going through your day-to-day working on .NET and Java applications and helping people figure out how to go about modernizing them, what we've talked about so far is that represents some of the deeper architectural issues and stuff. You've already mentioned 12 factor after and being able to move, or thinking about the way that you frame the application as far as inputs of those things that it takes to configure, or to think with the lifecycle of those things. Are there some other common patterns that you see across the two practices, Java and .NET, that you think are just concrete examples of stuff that people should take away maybe from this episode, that they could look at their app – and they’re trying to get ahead of the game a little bit? [0:18:46.3] SA: I would say a big part of the commonality that Chris and I both work on a lot is we have a methodology called the SWIFT methodology that we use to help discover how the applications really want to behave, define a notional architecture that is again, agnostic of the implementation details. We’ll often come in with a the same process and I don't need to be a .NET expert and a .NET shop to figure out how the system really wants to be designed, how you want to break things into microservices and then the implementation becomes where those details are. Chris and I both collaborate on a lot of that work. It makes you feel a little bit better about the output when you know that the technology isn't as important. You get to actually pick which technology fits the solution best, as opposed to starting with the technology and letting a solution form around it, if that makes sense. [0:19:42.4] CU: Yeah. I'd say that interesting thing is just how difficult it is while we're going through the SWIFT process with customers, to get them to not get terribly attached to the nouns of the technology and the solution. They've usually gone in where it's not just a matter of the language, but they have something picked in their head already for data storage, for messaging, etc., and they're deeply attached to some of these decisions, deeply and emotionally attached to them. Fundamentally, when we're designing a notional architecture as we call it, really you should be making decisions on what nouns you're going to pick based on that architecture to use the tools that fit that. That's generally a bit of a process the customers have to go through. It's difficult for them to do that, because the more technical their stakeholders tend to be, often the more attached they are to the individual technology choices and breaking that is the principal role for us. [0:20:37.4] OP: Is there any help, or any investment, or any coordination with those vendors, or the purveyors of the technologies that perhaps legacy applications are, or indeed the platforms they're running on, is there any help on that side from those vendors to help with application transformation, or making those applications better? Or do organizations have to rely on a completely independent, so the team like you guys to come in and help them with that? Do you understand my point? Is there any internal – like you mentioned WebLogic, WebSphere, do the purveyors of those platforms try and drive the transformation from within there? Or is it organizations who are running those apps have to rely on independent companies like you, or like us to help them with that? [0:21:26.2] SA: I think some of it depends on what the goal of the modernization is. If it's something like, we no longer want to pay Oracle licensing fees, then of course, obviously they – WebLogic teams aren't going to be happy to help. That's not always the case. Sometimes it's a case where we may have a lot of WebLogic. It's working fine, but we just don't like where it's deployed and we'd like to containerize it, move it to Kubernetes or something like that. In that case, they're more willing to help. At least in my experience, I've found that the technology vendors are rightfully focused just on upgrading things from their perspective and they want to own the world, right? WebLogic will say, “Hey, we can do everything. We have clustering. We have messaging. We've got good access to data stores.” It's hard to find a technology vendor that has that broader vision, or the discipline to not try to fit their solutions into the problem, when maybe they're not the best fit. [0:22:30.8] CU: I think it's a broad generalization, but specifically on the Java side it seems that at least with app server vendors, the status quo is usually serving them quite well. Quite often, we’re adversary – a bit of an adversarial relationship with them on occasion. I could certainly say that within the .NET space, we've worked a relatively collaboratively with Microsoft on things like Steeltoe, which is a I wouldn't say it's a springboot analog, but at least a microservice library that helps people achieve 12-factor cloud nativeness. That's something where I guess Microsoft represents both the legacy side, but also the future side and were part of a solution together there. [0:23:19.4] SA: Actually, that's a good point because the other way that we're seeing vendors be involved is in creating operators on Kubernetes side, or Cloud Foundry tiles, something that makes it easy for their system to still be used in the new world. That's definitely helpful as well. [0:23:38.1] CC: Yeah, that's interesting. [0:23:39.7] JR: Recently, myself me people on my team went through a training from both Shaun and Chris, interestingly enough in Colorado about this thing called the SWIFT methodology. I know it's a really important methodology to how you approach some of the application transformation-like engagements. Could you two give us a high-level overview of what that methodology is? [0:24:02.3] SA: I want to hear Chris go through it, since I always answer that question first. [0:24:09.0] CU: Sure. I figured since you were the inventor, you might want to go with it Shaun, but I'll give it a stab anyway. Swift is a series of exercises that we use to go from a business problem into what we call a notional architecture for an application. The one thing that you'll hear Shaun say all the time that I think is pretty apt, which is we're trying to understand how the application wants to behave. This is a very analog process, especially at the beginning. It's one where we get people who can speak about the business problem behind an application and the business processes behind an application. We get them into a room, a relatively large room typically with a bunch of wall space and we go through a series of exercises with them, where we tease that business process apart. We start with a relatively lightweight version of Alberto Brandolini’s event storing method, where we map out with the subject matter experts, what all of the business events that occur in a system are. That is a non-technical exercise, a completely non-technical exercise. As a matter of fact, all of this uses sticky notes and arts and crafts. After we've gone through that process, we transition into Boris diagram, which is an exercise of Shaun's design that we take the domains that we've, or at least service candidates that we've extrapolated from that event storming and start to draw out a notional architecture. Like an 80% idea of what we think the architecture is going to look like. We're going to do this for slices of – thin slices of that business problem. At that point, it starts to become something that a software developer might be interested in. We have an exercise called Snappy that generally occurs concurrently, which translates that message flow, Boris diagram thing into something that's at least a little bit closer to what a developer could act upon. Again, these are sticky note and analog exercises that generally go on for about a week or so, things that we do interactively with customers to try to get a purely non-technical way, at least at first, so that we can understand that problem and tell you what an architecture is that you can then act on. We try to position this as a customer. You already have all of the answers here. What we're going to do as facilitators of these is try to pull those out of your head. You just don't know how to get to the truth, but you already know that truth and we're going to design this architecture together. How did I do, Shaun? [0:26:44.7] SA: I couldn't have said it better myself. I would say one of the interest things about this process is the reason why it was developed the way it was is because in the world of technology and especially engineers, I definitely seen that you have two modes of thought when you come from the business world to the to the technical world. Often, engineers will approach a problem in a very different way and a very focused, blindered way than business folks. Ultimately, what we try to think of is that the purpose for the software is to enable the business to run well. In order to do that, you really need to understand at least at a high-level, what the heck is the business doing? Surprisingly and almost consistently, the engineering team doing the work is separated from the business team enough that it's like playing the telephone game, right? Where the business folks say, “Well, I told them to do this.” The technical team is like, “Oh, awesome. Well then, we're going to use all this amazing technology and build something that really doesn't support you.” This process really brings everybody together to discover how the system really wants to behave. Also as a side effect, you get everybody agreeing that yes, that is the way it's supposed to be. It's exciting to see teams come together that actually never even work together. You see the light bulbs go on and say, “Oh, that's why you do that.” The end result is in a week, we can go from nobody really knows each other, or quite understands the system as a whole, to we have a backlog of work that we can prioritize based on the learnings that we have, and feel pretty comfortable that the end result is going to be pretty close to how we want to get there. Then the biggest challenge is defining how do we get from point A to point B. That's part of that layering of the Swift method is knowing when to ask those questions. [0:28:43.0] JR: A micro follow-up and then I'll keep my mouth shut for a little bit. Is there a place that people could go online to read about this methodology, or just get some ideas of what you just described? [0:28:52.7] SA: Yeah. You can go to swiftbird.us. That has a high-level overview of more the public facing of how the methodology works. Then there's also internal resources that are constantly being developed as well. That's where I would start. [0:29:10.9] CC: That sounds really neat. As always, we are going to have links on the show notes for all of this. I checked out the website for the EventStorming book. There is a resources page there and has a list of a bunch of presentations. Sounds very interesting. I wanted to ask Chris and Shaun, have you ever seen, or heard of a case where a company went through the transformation, or modernization process and then they roll back to their legacy system for any reason? [0:29:49.2] SA: That's actually a really good question. It implies that often, the way people think about modernization would be more of a big bang approach, right? Where at a certain point in time, we switch to the new system. If it doesn't work, then we roll back. Part of what we try to do is have incremental releases, where we're actually putting small slices into production where you're not rolling back a whole – from modern back to legacy. It's more of you have a week's worth of work that's going into production that's for one of the thin slices, like Chris mentioned. If that doesn't work where there's something that is unexpected about it, then you're rolling back just a small chunk. You're not really jumping off a cliff for modernization. You're really taking baby steps. If it's a two step forward and one step back, you're still making a lot of really good progress. You're also gaining confidence as you go that in the end in two years, you're going to have a completely shiny new modern system and you're comfortable with it, because you're getting there an inch of the time, as opposed to taking a big leap. [0:30:58.8] CU: I think what's interesting about a lot of large organizations is that they've been so used to doing big bang releases in general. This goes from software to even process changes in their organizations. They’ve become so used to that that it often doesn't even cross their mind that it's possible to do something incrementally. We really do often times have to get spend time getting buy-in from them on that approach. You'd be surprised that even in industries that you’d think would be fantastic with managing risk, when you look at how they actually deal with deployment of software and the rolling out of software, they’re oftentimes taking approaches that maximize their risk. There's no way to make something riskier by doing a big bang. Yeah, as Shaun mentioned, the specifics of the swift are to find a way, so that you can understand where and get a roadmap for how to carve out incremental slices, so that you can strangle a large monolithic system slowly over time. That's something that's pretty powerful. Once someone gets bought in on that, they absolutely see the value, because they're minimizing risk. They're making small changes. They're easy to roll back one at a time. You might see people who would stop somewhere along the way, and we wouldn't necessarily say that that's a problem, right? Just like not every app needs to be modernized, maybe there's portions of systems that could stay where they are. Is that a bad thing? I wouldn't necessarily say that it is. Maybe that's the way that – the best way for that organization. [0:32:35.9] DC: We've bumped into this idea now a couple of different times and I think that both Chris and Shaun have brought this up. It's a little prelude to a show that we are planning on doing. One of the operable quotes from that show is the greatest enemy of knowledge is not the ignorance, it is the illusion of knowledge. It's a quote by Stephen Hawking. It speaks exactly to that, right? When you come to a problem with a solution in your mind that is frequently difficult to understand the problem on its merit, right? It’s really interesting seeing that crop up again in this show. [0:33:08.6] CU: I think even oftentimes, the advantage of a very discovery-oriented method, such as Swift is that it allows you to start from scratch with a problem set with people maybe that you aren't familiar with and don't have some of that baggage and can ask the dumb questions to get to some of the real answers. It's another phrase that I know Shaun likes to use is that our roles is facilitator to this method are to ask dumb questions. I mean, you just can't put enough value on that, right? The only way that you're going to break that established thinking is by asking questions at the root. [0:33:43.7] OP: One question, actually there was something recently that happened in the Kubernetes community, which I thought was pretty interesting and I'd like to get your thoughts on it, which is that Istio, which is a project that operates as a service mesh, I’m sure you all are familiar with it, has recently decided to unmodernize itself in a way. It was originally developed as a set of microservices. They have had no end of difficulty in getting in optimizing the different interactions between those services and the nodes. Then recently, they decided this might be a good example of when to monolith, versus when to microservice. I'm curious what your thoughts are on that, or if you have familiarity with it. [0:34:23.0] CU: What's actually quite – I'm not going to necessarily speak too much to this. Time will tell as to if the monolithing that they're doing at the moment is appropriate or not. Quite often, the starting point for us isn't necessarily a monolith. What it is is a proposed architecture coming from a customer that they're proud of, that this is my microservice design. You'll see a simple system with maybe hundreds of nano-services. The surprise that they have is that the recommendation from us coming out of our Swift sessions is that actually, you're overthinking this. We're going to take that idea that you have any way and maybe shrink that down and to save tens of services, or just a handful of services. I think one of the mistakes that people make within enterprises, or on microservices at the moment is to say, “Well, that's not a microcservice. It’s too big.” Well, how big or how small dictates a microservice, right? Oftentimes, we at least conceptually are taking and combining services based on the customers architecture very common. [0:35:28.3] SA: Monoliths aren't necessarily bad. I mean, people use them almost as a pejorative, “Oh, you have a monolith.” In our world it's like, well monoliths are bad when they're bad. If they're not bad, then that's great. The corollary to that is micro-servicing for the sake of micro-servicing isn't necessarily a good thing either. When we go through the Boris exercise, really what we're doing is we're showing how domain-based, or capabilities relate to each other. That happens to map really well in our opinion to first, cut microservices, right? You may have an order service, or a customer service that manages some of that. Just because we map capabilities and how they relate to each other, it doesn't mean the implementation can't even be as a single monolith, but componentized inside it, right? That's part of what we try really hard to do is avoid the religion of monolith versus microservices, or even having to spend a lot of time trying to define what a microservice is to you. It's really more of well, a system wants to behave this way. Now, surprise, you just did domain-driven design and mapped out some good 12-factor compliant microservices should you choose to build it that way, but there's other constraints that always apply at that point. [0:36:47.1] OP: Is there more traction in organizations implementing this methodology on a net new business, rather than current running businesses or applications? Is there are situations more so that you have seen where a new project, or a new functionality within a business starts to drive and implement this methodology and then it creeps through the other lines of business within the organization, because that first one was successful? [0:37:14.8] CU: I'd say that based on the nature of who our customers are as an app transformation practice, based on who those customers are and what their problems are, we're generally used to having a starting point of a process, or software that exists already. There's nothing at all to mandate that it has to be that way. As a matter of fact, with folks from our labs organization, we've used these methods in what you could probably call greener fields. At the end of the day when you have a process, or even a candidate process, something that doesn't exist yet, as long as you can get those ideas onto sticky notes and onto a wall, this is a very valid way of getting – turning ideas into an architecture and an architecture into software. [0:37:59.4] SA: We've seen that happen in practice a couple times, where maybe a piece of the methodology was used, like EventStorming just to get a feel for how the business wants to behave. Then to rapidly try something out in maybe more of a evolutionary architecture approach, MVP approach to let's just build something from a user perspective just to solve this problem and then try it out. If it starts to catch hold, then iterate back and now drill into it a little bit more and say, “All right. Now we know this is going to work.” We're modernizing something that may be two weeks old just because hooray, we proved it's valuable. We didn't necessarily have to spend as much upfront time on designing that as we would in this system that's already proven itself to be of business value. [0:38:49.2] OP: This might be a bit of a broad question, but what defines success of projects like this? I mean, we mentioned earlier about cost and maybe some of the drivers are to move off certain mainframes and things like that. If you're undergoing an application transformation, it seems to me like it's an ongoing thing. How do enterprises try to evaluate that return on investment? How does it relate to success criteria? I mean, faster release times, etc., potentially might be one, but how was that typically evaluated and somebody internally saying, “Look, we are running a successful project.” [0:39:24.4] SA: I think part of what we tried to do upfront is identify what the objectives are for a particular engagement. Often, those objectives start out with one thing, right? It's too costly to keep paying IBM or Oracle for WebLogic, or WebSphere. As we go through and talk through what types of things that we can solve, those objectives get added to, right? It may be the first thing, our primary objective is we need to start moving workloads off of the mainframe, or workloads off of WebLogic, or WebSphere, or something like that. There's other objectives that are part of this too, which can include things as interesting as developer happiness, right? They have a large team of a 150 developers that are really just getting sick of doing the same old thing and having new technology. That's actually a success criteria maybe down the road a little bit, but it's more of a nice to have. In a long-winded answer of saying, when we start these and when we incept these projects, we usually start out with let's talk through what our objectives are and how we measure success, those key results for those objectives. As we're iterating through, we keep measuring ourselves against those. Sometimes the objectives change over time, which is fine because you learn more as you're going through it. Part of that incremental iterative process is measuring yourself along the way, as opposed to waiting until the end. [0:40:52.0] CC: Yeah, makes sense. I guess these projects are as you say, are continuous and constantly self-adjusting and self-analyzing to re-evaluate success criteria to go along. Yeah, so that's interesting. [0:41:05.1] SA: One other interesting note though that personally we like to measure ourselves when we see one project is moving along and if the customers start to form other projects that are similar, then we know, “Okay, great. It's taking hold.” Now other teams are starting to do the same thing. We've become the cool kids and people want to be like us. The only reason it happens for that is when you're able to show success, right? Then other teams want to be able to replicate that. [0:41:32.9] CU: The customers OKRs, oftentimes they can be a little bit easier to understand. Sometimes they're not. Typically, they involve time or money, where I'm trying to take release times from X to Y, or decrease my spend on X to Y. The way that we I think measure ourselves as a team is around how clean do we leave the campsite when we're done. We want the customers to be able to run with this and to continue to do this work and to be experts. As much as we'd love to take money from someone forever, we have a lot of people to help, right? Our goal is to help to build that practice and center of excellence and expertise within an organization, so that as their goals or ideas change, they have a team to help them with that, so we can ride off into the sunset and go help other customers. [0:42:21.1] CC: We are coming up to the end of the episode, unfortunately, because this has been such a great conversation. It turned out to be a more of an interview style, which was great. It was great getting the chance to pick your brains, Chris and Shaun. Going along with the interview format, I like to ask you, is there any question that wasn't asked, but you wish was asked? The intent here is to illuminates what this process for us and for people who are listening, especially people who they might be in pain, but they might be thinking this is just normal. [0:42:58.4] CU: That's an interesting one. I guess to some degree, that pain is unfortunately normal. That's just unfortunate. Our role is to help solve that. I think the complacency is the absolute worst thing in an organization. If there is pain, rather than saying that the solution won't work here, let’s start to talk about solutions to that. We've seen customers of all shapes and sizes. No matter how large, or cumbersome they might be, we've seen a lot of big organizations make great progress. If your organization's in pain, you can use them as an example. There is light at the end of the tunnel. [0:43:34.3] SA: It's usually not a train. [0:43:35.8] CU: Right. Usually not. [0:43:39.2] SA: Other than that, I think you asked all the questions that we always try to convey to customers of how we do things, what is modernization. There's probably a little bit about re-platforming, doing the bare minimum to get something onto to the cloud. We didn't talk a lot about that, but it's a little bit less meta, anyway. It's more technical and more recipe-driven as you discover what the workload looks like. It's more about, is it something we can easily do a CF push, or just create a container and move it up to the cloud with minimal changes? There's not conceptually not a lot of complexity. Implementation-wise, there's still a lot of challenges there too. They're not as fun to talk about for me anyway. [0:44:27.7] CC: Maybe that's a good excuse to have some of our colleagues back on here with you. [0:44:30.7] SA: Absolutely. [0:44:32.0] DC: Yeah, in a previous episode we talked about persistence and state of those sorts of things and how they relate to your applications and how when you're thinking about re-platforming and even just where you're planning on putting those applications. For us, that question comes up quite a lot. That's almost zero trying to figure out the state model and those sort of things. [0:44:48.3] CC: That episode was named States in Stateless Apps, I think. We are at the end, unfortunately. It was so great having you both here. Thank you Duffie, Shaun, Chris and I'm going by the order I'm seeing people on my video. Josh and Olive. Until next time. Make sure please to let us know your feedback. Subscribe. Give us a thumbs up. Give us a like. You know the drill. Thank you so much. Glad to be here. Bye, everybody. [0:45:16.0] JR: Bye all. [0:45:16.5] CU: Bye. [END OF EPISODE] [0:45:17.8] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

John Lounds reflects on how he achieved so much success. Good mentorship, building trust and building culture are all a part of the equation.  Listen and Subscribe:  Apple Podcasts | Google Podcasts | Stitcher | Spotify  More about John Lounds: John Joined the Nature Conservancy of Canada in 1997. Under John’s Leadership, NCC has grown from an annual budget of $8 million to approximately $80 million in 2017. NCC and its partners have helped to protect over 14 million hectares of ecologically significant land since 1962. John was previously a governor of the University of Waterloo, as well as a member of the Dean’s Advisory Committee at the Faculty of Environment at the University of Waterloo. He has served as a director of the George Cedric Metcalf Charitable Foundation, the Smart Prosperity Initiative, the International Land Conservation Network and on the Canadian councils of the North American Bird Conservation Initiative and the North American Wetlands Conservation Act. Links to Additional Resources: natureconservancy.ca Get involved with the Nature Conservancy of Canada   Memorable Quote: “Nobody can do everything. If you can build partnerships, if you can build a team with those skills and abilities, the team can make it happen – but no one should think that they are the only person able to do all of this.”   Full Episode Transcript: TINEKE KEESMAAT: This leader has helped protect over 35 million acres of ecologically-significant land across Canada. ANNOUNCER: Welcome to Leader Lab, where we talk to experts about how leaders can excel in a modern world. Helping leaders for over 20 years. Your host, Tineke Keesmaat. TINEKE KEESMAAT: John Lounds is the president and CEO of Nature Conservancy of Canada. He is passionate about nature, conservation, and leadership. Under his guidance, the Nature Conservancy has grown exponentially over the past two years, from a budget of $8 million to $80 million. On today's Leader Lab, he'll share some of the leadership lessons he learned along the way. John, welcome to the Leader Lab. JOHN LOUNDS: Thank you. TINEKE KEESMAAT: It's great to have you here, and I'm very curious if you could start by telling us a little bit about the path that brought you to the Nature Conservancy. JOHN LOUNDS: Well, I never thought I'd start off working in a nonprofit charity. Many years ago in high school, I was-- studying computer science was one of the big areas I was going to go into, but I had a geography teacher who really inspired me and wanted me to think about how the world could be changed as a result of how you think about organizing on the landscape. TINEKE KEESMAAT: And what's kept you in nonprofit for so long, and specifically, in the environmental space? JOHN LOUNDS: Well, the environmental space is my passion. I think a big part of working in a nonprofit charity is that you need to have the passion for the work. If you don't feel it, believe it, think it, dream it, live it, the people that you're talking to about the work you're doing will not hear you, they won't understand why it's important or what the impact can be. This field is my field, that's where I want to be. I also wanted to work in an organization that worked right across the country. I'm a proud Canadian and believe that we have one of the best countries in the world, and I just want to make sure that that's what I'm doing as well. So coupling the nonprofit work with my interest in the environmental world has been great for me. TINEKE KEESMAAT: That's amazing. And 20 years ago, environmental issues were not as talked about as they are now. So what has been the big shift in leading an organization? May not have been on the first page every day to now where it's everywhere you look. JOHN LOUNDS: I think that's true, and they weren't-- these issues weren't being discussed so much many years ago, but there were some big problems that came to the surface while I was growing up, and folks may not remember the Cuyahoga River in Cleveland catching on fire in Lake Erie, other events such as that, those-- a river catching on fire? Like, what is going on here? Those kind of events really affected my thinking then. I would say today with the awareness that people have and concern about-- whether it be climate change, biodiversity conservation, et cetera, we're seeing way more interest in the work we're doing, and I think that's somewhat contributed to the growth of the work of the Nature Conservancy of Canada, because there's more people who are understanding the importance of this work, and we've been welcoming them to the fold. TINEKE KEESMAAT: That's amazing. And over the last two decades that you've been leading the conservancy, you have had a dramatic impact. So you raised the budget from $8 million to $80 million, you've protected over 35 million acres of Canadian land. How did you create such an ambitious vision? JOHN LOUNDS: Well, it wasn't just me. I'd say one of my first lessons was the importance of finding and surrounding myself with really great people who would always challenge me and the team to think bigger. If you can find them, if you can listen to what they have to say, and know in your heart that you can actually accomplish more than you think you can, that combination can lead to some incredible things. So I remember one of my past board members who spoke about what is the conservation equivalent of a nation-building exercise? Like really thinking beyond we're not just going to solve this property problem or we're just going to solve this little issue. What is a big way of thinking about it? I also had some mentors. We had a session where we were talking about how much money we could possibly raise for one of our campaigns, and this was-- we had thought we would set a goal of $300 million thinking that was a very big number over several years. And this gentleman came to the front of the room to speak and he said, $300 million? That's not nearly enough. It needs to be $500 million. And the $500 million was actually what we then went away to do as a result of just that person pushing the boundaries of my thinking. I would never have thought of that. They push, they ask the tough questions, that's how the people that you surround yourself with can help you. TINEKE KEESMAAT: So John, that's amazing to have different perspectives pushing and challenging you and helping you to imagine what is possible. I can imagine that time that that might create some tension if your board is saying $500 million and your team is saying $200 million. How do you manage that tension? JOHN LOUNDS: Well I actually find that tension to be important. It's that space between the staff who are obviously implementing the work that needs to be done and the board's role pushing and asking tough questions to come to a place where we can all agree on what the right-- or the best way forward would be. And I'm a firm believer that if you have the staff being stronger than the board or the board being stronger than the staff, that you end up in a space that isn't as productive, doesn't create as much energy, and doesn't challenge-- whether challenging staff or challenging board members-- to get to the right answer going forward. So I look at this as a very important aspect of board-staff relationships, and it's a really important role for the CEO and the chair of the board to handle. TINEKE KEESMAAT: And I imagine that requires creating a lot of transparency and trust, right? So the board has to have some depth into what the organization's capabilities are, what they can actually do, and conversely, the staff needs to understand the role of the board. JOHN LOUNDS: Transparency, making sure that you're prepared, you've informed the board, you haven't hidden anything-- good or bad, the information, that's all important and needs to be shone a light on and discussed. I sometimes find-- I've seen in other organizations where the CEO-- because on the role, you know a lot about what's going on in all aspects of the business, but sometimes these CEOs are impatient and want the board to decide quickly and will jump in and say, no, no, what about this, what about that? But that's probably the worst thing you can do. It's better to just sit back, let the board have its full discussion, gain understanding, and come to conclusions that they wish to take, because by doing that, you're going to end up heading in the right direction. TINEKE KEESMAAT: That's great. So when we started this conversation, you talked about how nature is your passion, and I've been reading more and more about the importance of purpose in organizations-- so really helping people connect their passion to the work that they are doing. And I imagine that in your organization, you have lots of passion-driven individuals. JOHN LOUNDS: 340 of them. TINEKE KEESMAAT: That's awesome. Not everybody can say that. JOHN LOUNDS: No. TINEKE KEESMAAT: And I'm just curious, could you talk to me a little bit about the benefits of leading a passion-driven organization and maybe what some of the unintended challenges might be? JOHN LOUNDS: The benefit of leading a passion-driven organization is that you really don't have to motivate people to get up in the morning and come to work and do the work they do. That is not the issue. They are ready to run and ready to do what they can because they so fully believe in the mission of the organization. The key, then, is how do you direct that energy and enthusiasm? How do you keep that enthusiasm going, but how do you direct that energy and enthusiasm? So the 340 people kind of working in the same direction, that's the challenge. TINEKE KEESMAAT: And what have you found to be helpful in channeling that energy? JOHN LOUNDS: Well, one of the ways that nonprofits and charities proceed is they organize campaigns. And often people will see in the news that there's a campaign for x hundred million dollars or whatever the case might be. And the number is important, because you do need funds to run the business. But more importantly than the money is actually the alignment that a campaign provides for everybody working in the organization. By setting a common goal, describing the impact of that-- what's the vision? Not the big vision over time, but over the next five years. And by organizing people toward campaigns, it's a really great way of ensuring that everybody's energy is channeled together. TINEKE KEESMAAT: Because I mentioned, the campaign is-- the fundraiser, clearly they're the target for them, but also it connects to the programmers because they need to think about how they'll use that money and have a clear message on it, and then your communications folks, the stories that they are telling. So all of a sudden this big goal that you've set out in the campaign, every individual that knows what they need to do to make it happen. JOHN LOUNDS: Absolutely. And it starts with what conservation work are you're going to get done, right? And what is the impact of that conservation work and can you describe it well to people? Because you can't raise money for just raising money. What is going to be the outcome? If I invest in the Nature Conservancy of Canada, what will happen over the next five years? All donors and funders are looking to know what that is. TINEKE KEESMAAT: I'm curious again on this notion of passionate-driven team members. I'm curious if you've ever had instances where somebody is hired, they're super passionate about the cause, they have great enthusiasm, and then they walk in the door and that enthusiasm doesn't necessarily translate to impact at an individual level. I'm curious if that happens and then how you handle it. JOHN LOUNDS: I'd say folks that have come into the organization that don't have that passion, we've made a hiring mistake there, or they've made a hiring-- they've made a choice to come. We've had some people that come from private sector organizations that think, oh, I'll kind of retire on my way into the work here. That's never the case. And then they are suddenly surprised that they're working more than they were before. I think you want to make sure you're getting the right people in the right seats on the bus, which is common parlance, but in nonprofits-- I'm a big fan of Peter Drucker in this regard. Basically that you need to look for that person's contribution. If they aren't working out in the role, it's best to think about can you re-pot these people into another role where they will be able to live their passion? And sometimes those require pretty tough conversations to get there, but I've found that that's not only for the person involved, but for the organization as a whole a better way to go. TINEKE KEESMAAT: So really thinking about what are their strengths, how can they have a contribution or make an impact here, and then thinking about where that actually fits with what the organization-- JOHN LOUNDS: Right. If they've truly come for the mission, if they're passionate about it, just leaving them by the wayside isn't going to actually help the overall cause as I was just describing. So you have to figure out how to use-- now sometimes the fit isn't quite right and those decisions sometimes are mutual, and perhaps other organizations that are working on environmental causes are a better fit in terms of their particular interests. So we have lots of alumni from the Nature Conservancy of Canada and lots of other places for all sorts of good reasons. TINEKE KEESMAAT: That's great. I love that. It's just this idea of really keeping the passion of the organization and the purpose, and then thinking about the individual-- what do they need? What are their strengths? Where is that going to be a fit? And sometimes it's here and sometimes it's elsewhere, but it really is thinking about what's going to make that individual thrive. JOHN LOUNDS: In terms of choosing to work from home or wherever, that particular time is where we understand that, especially team members with young families. And in terms of the organization itself, we try and walk the talk that we are interested in the communities where we work, and where we believe that nature conservation is an important thing for Canada. One of the things we actually instituted-- we did it as a special a couple of years ago, but one of the things we instituted this past year permanently was to provide staff with two nature days during the summer months so that they can go and appreciate and reflect on the work they do. TINEKE KEESMAAT: That's amazing. I want to nature conservative day. [LAUGHS] JOHN LOUNDS: Well, we'll set up a program and try and get many companies to do this. That'd be fantastic. TINEKE KEESMAAT: I think it'd be super fantastic. That's great. John, I just want to continue this conversation on nonprofits, and I'm wondering from your perspective, what you think some of the unique characteristics are of these organizations and how, as a leader, you may have to adapt our style to manage them. JOHN LOUNDS: Well, I think one of the important aspects, obviously, is reputation and trust. We're not selling a good or service, really, so unless our reputation is beyond reproach and people trust us with the funds that they're giving us, the rest doesn't really happen. As I said, we have to remember every day that every dollar is a gift and people have voluntarily provided this to us. So I think the reputation, being transparent, integrity, all the good things that should be part of any business are even heightened further in a non-profit charity. And I think part of it is just knowing yourself. That's a common phrase, but knowing who you are and who you're not, and then nobody can do everything, but if you can build partnerships, if you can build a team and make sure the team around you has all those skills and has all those abilities, the team can make it happen, but no one should think that they are the only person able to do all of this. TINEKE KEESMAAT: That amazing. I've been thinking more and more about the importance of leader self-awareness in being able to drive impact, because I believe that-- exactly that. If you-- nobody can do it all with themselves, and so by being aware, you can know where you need to augment your team or what you need to keep your energy up through the highs and lows of driving or leading an organization. What have you learned about yourself over the years that you've had to kind of not deal with, but that you've had to incorporate into your leadership style? And how have you done that? JOHN LOUNDS: I think I'm in with a good group of people, because I would say that largely, the team here at the Nature Conservancy of Canada are likely skewed to the introvert side of the scale rather than the extrovert side of the scale. I'm one of those, and I've had to learn and train myself to push through my inclination to not want to talk about what we're doing, not want to get out there and yell in the bright lights about the work that's being done. We're plant and animal people. We would like to talk to the plants and animals, we don't actually know people, a lot about them. But since our business is a relationship with people business, frankly, that, I think, I've had to strengthen, I think I have a very good understanding of how to individually relate to people. The challenge has been to speak more broadly and speak to larger groups, and I've been able to get there. TINEKE KEESMAAT: And what have you done to help yourself in those moments? JOHN LOUNDS: Practice, practice, practice. It's about the only way to get over it. And then when I'm giving those talks, that I've checked in with the people around me to make sure I've asked, how did I do? You can always improve, and you can always do better. It's important that you get others to-- who will tell you the truth, to reflect on what you did that could be improved and what you did that maybe you should leave behind next time. TINEKE KEESMAAT: So John, when we were talking before the podcast, you mentioned how important you feel it is for leaders to the culture of their organizations. Can you tell me why this is important to you and examples of how you've made this happen? So the nature days would be one of those examples, but what else have you done to really make the culture come to life? JOHN LOUNDS: What I've tried to do is instill a culture where people should listen to each other. You can learn a lot from not assuming that when somebody has said something, that that's actually what they're thinking, and get underneath that and listen to what they're really saying. The culture as I see it is you listen hard, you work hard, you play some, and again, you need to know that you can actually accomplish a lot more than you think you can, especially if you're working with your team. And I try and walk that talk. Like I said, trying to be a flexible, caring place to work, having people get out into nature so that they understand-- I mean, we've got a lot of people that work in the field, but we also have a lot of people that work in the office doing finance and other things that it's important for them to actually get to see the work. And if you're here for 15 years, which sounds like a long time, we give you a week and some funds to go and travel anywhere in Canada to go and understand what that part of the world is like and get outside. So we try and really live that as much as we can. TINEKE KEESMAAT: That's amazing. John, I really enjoyed the conversation, and lots of amazing insights for myself and for the people who listen to the Leader Lab. I'm curious, just as we round out our conversation, if you had one practical piece of advice for leaders, something they can take away from this conversation and go do tomorrow that would really impact their effectiveness, what would that piece of advice be? JOHN LOUNDS: Well, I always start with my Stephen Leacock quote, which is, "I'm a great believer in luck, and the harder I work, the more I have of it." That's always watchwords for me. And one of the ways I try and do that is by not get caught up in the day-to-day and remember what the important things are. And I've had to do that. I do try and set aside three to four hours at least once a week to work on something important, because once you set aside that much time, you actually can't do your job, which is to think several years out, not just worry about what happened this month, last month. And I even take that to another place where I actually will go out to a place that's likely within a forest or nearby, and I'll take two to three days and actually just sequester myself and go and do that, because I find unless you actually step back from the day-to-day, you forget your perspective on what the important things are and what needs to be done in order to take you out for the next several years. ANNOUNCER: And now, let's get to know our guest a little better with some rapid fire questions. TINEKE KEESMAAT: As we wrap up the podcast, we have my favorite part. JOHN LOUNDS: Uh oh. TINEKE KEESMAAT: The random questions. Don't worry, they're not hard. [LAUGHTER] And just your first responses. JOHN LOUNDS: OK. TINEKE KEESMAAT: First, the craziest place in the world that you've been. JOHN LOUNDS: Oh no. [LAUGHTER] The craziest place in the world that I've been? That's supposed to be my first response? TINEKE KEESMAAT: Yeah. There's no right answer. JOHN LOUNDS: I know there's no right answer, but I could do a lot of places. TINEKE KEESMAAT: That's awesome. Or most surprising place. JOHN LOUNDS: Manila. Oh, a surprising place? Labrador. TINEKE KEESMAAT: Are you an early bird or a night owl? JOHN LOUNDS: Early bird. TINEKE KEESMAAT: I'm not surprised. You as a teenager in three words. JOHN LOUNDS: Lost, driven, and a bit unsure of myself. TINEKE KEESMAAT: So a typical teenager. JOHN LOUNDS: Yeah. TINEKE KEESMAAT: And your favorite emoji? JOHN LOUNDS: I hate emojis. TINEKE KEESMAAT: [LAUGHS] Fair enough. And the all-important final question-- how do you feel about Brussels sprouts? JOHN LOUNDS: I'm not a fan. TINEKE KEESMAAT: [LAUGHS] Awesome. ANNOUNCER: Thank you for joining us today on Leader Lab. Leader Lab is powered by Tiltco, helping exceptional leaders achieve extraordinary results. And the Ivey Acedmy at Ivey Business School, Canada's home for learning and development. You can learn more about Tiltco and Leader Lab a tiltco.ca. And to find out more about the Ivey Academy, go to iveyacademy.com.  

The question of diving into Kubernetes is something that faces us all in different ways. Whether you are already on the platform, are considering transitioning, or are thinking about what is best for your team moving forward, the possibilities and the learning-curve make it a somewhat difficult question to answer. In this episode, we discuss the topic and ultimately believe that an individual is the only one who can answer that question well. That being said, the capabilities of Kubernetes can be quite persuasive and if you are tempted then it is most definitely worth considering very seriously, at least. In our discussion, we cover some of the problems that Kubernetes solves, as well as some of the issues that might arise when moving into the Kubernetes space. The panel shares their thoughts on learning a new platform and compare it with other tricky installations and adoption periods. From there, we look at platforms and how Kubernetes fits and does not fit into a traditional definition of what a platform constitutes. The last part of this episode is spent considering the future of Kubernetes and how fast that future just might arrive. So for all this and a bunch more, join us on The Podlets Podcast, today! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Josh Rosso Duffie Cooley Bryan Liles Key Points From This Episode: The main problems that Kubernetes solves and poses. Why you do not need to understand distributed systems in order to use Kubernetes. How to get around some of the concerns about installing and learning a new platform. The work that goes into readying a Kubernetes production cluster. What constitutes a platform and can we consider Kubernetes to be one? The two ways to approach the apparent value of employing Kubernetes. Making the leap to Kubernetes is a personal question that only you can answer. Looking to the future of Kubernetes and its possible trajectories. The possibility of more visual tools in the UI of Kubernetes. Understanding the concept of conditions in Kubernetes and its objects. Considering appropriate times to introduce a team to Kubernetes. Quotes: “I can use different tools and it might look different and they will have different commands but what I’m actually doing, it doesn’t change and my understanding of what I’m doing doesn’t change.” — @carlisia [0:04:31] “Kubernetes is a distributed system, we need people with expertise across that field, across that whole grouping of technologies.” — @mauilion [0:10:09] “Kubernetes is not just a platform. Kubernetes is a platform for building platforms.” — @bryanl [0:18:12] Links Mentioned in Today’s Episode: Weave — https://www.weave.works/docs/net/latest/overview/ AWS — https://aws.amazon.com/ DigitalOcean — https://www.digitalocean.com/ Heroku — https://www.heroku.com/ Red Hat — https://www.redhat.com/en Debian — https://www.debian.org/ Canonical — https://canonical.com/ Kelsey Hightower — https://github.com/kelseyhightower Joe Beda — https://www.vmware.com/latam/company/leadership/joe-beda.html Azure — https://azure.microsoft.com/en-us/ CloudFoundry — https://www.cloudfoundry.org/ JAY Z — https://lifeandtimes.com/ OpenStack — https://www.openstack.org/ OpenShift — https://www.openshift.com/ KubeVirt — https://kubevirt.io/ VMware — https://www.vmware.com/ Chef and Puppet — https://www.chef.io/puppet/ tgik.io — https://www.youtube.com/playlist?list=PL7bmigfV0EqQzxcNpmcdTJ9eFRPBe-iZa Matthias Endler: Maybe You Don't Need Kubernetes - https://endler.dev/2019/maybe-you-dont-need-kubernetes Martin Tournoij: You (probably) don’t need Kubernetes - https://www.arp242.net/dont-need-k8s.html Scalar Software: Why most companies don't need Kubernetes - https://scalarsoftware.com/blog/why-most-companies-dont-need-kubernetes GitHub: Kubernetes at GitHub - https://github.blog/2017-08-16-kubernetes-at-github Debugging network stalls on Kubernetes - https://github.blog/2019-11-21-debugging-network-stalls-on-kubernetes/ One year using Kubernetes in production: Lessons learned - https://techbeacon.com/devops/one-year-using-kubernetes-production-lessons-learned Kelsey Hightower Tweet: Kubernetes is a platform for building platforms. It's a better place to start; not the endgame - https://twitter.com/kelseyhightower/status/935252923721793536?s=2 Transcript: EPISODE 18 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.9] JR: Hello everyone and welcome to The Podlets Podcast where we are going to be talking about should I Kubernetes? My name is Josh Rosso and I am very pleased to be joined by, Carlisia Campos. [0:00:55.3] CC: Hi everybody. [0:00:56.3] JR: Duffy Cooley. [0:00:57.6] DC: Hey folks. [0:00:58.5] JR: And Brian Lyles. [0:01:00.2] BL: Hi. [0:01:03.1] JR: All right everyone. I’m really excited about this episode because I feel like as Kubernetes has been gaining popularity over time, it’s been getting its fair share of promoters and detractors. That’s fair for any piece of software, right? I’ve pulled up some articles and we put them in the show notes about some of the different perspectives on both success and perhaps failures with Kub. But before we dissect some of those, I was thinking we could open it up more generically and think about based on our experience with Kubernetes, what are some of the most important things that we think Kubernetes solves for? [0:01:44.4] DC: All right, my list is very short and what Kubernetes solves for my point of view is that it allows or it actually presents an interface that knows how to run software and the best part about it is that it doesn’t – the standard interface. I can target Kubernetes rather than targeting the underlying hardware. I know certain things are going to be there, I know certain networking’s going to be there. I know how to control memory and actually, that’s the only reason that I really would give, say for Kubernetes, we need that standardization and you don’t want to set up VM’s, I mean, assuming you already have a cluster. This simplifies so much. [0:02:29.7] BL: For my part, I think it’s life cycle stuff that’s really the biggest driver for my use of it and for my particular fascination with it. I’ve been in roles in the past where I was responsible for ensuring that some magical mold of application on a thousand machines would magically work and I would have all the dependencies necessary and they would all agree on what those dependencies were and it would actually just work and that was really hard. I mean, getting to like a known state in that situation, it’s very difficult. Having something where either both the abstractions of containers and the abstraction of container orchestration, the ability to deploy those applications and all those dependencies together and the ability to change that application and its dependencies, using an API. That’s the killer part for me. [0:03:17.9] CC: For me, from a perspective of a developer is very much what Duffy just said but more so the uniformity that comes with all those bells and whistles that we get by having that API and all of the features of Kubernetes. We get such a uniformity across such a really large surface and so if I’m going to deploy apps, if I’m going to allow containers, what I have to do for one application is the same for another application. If I go work for another company, that uses Kubernetes, it is the same and if that Kubernetes is a hosted Kubernetes or if it’s a self-managed, it will be the same. I love that consistency and that uniformity that even so I can – there are many tools that help, they are customized, there’s help if you installing and composing specific things for your needs. But the understanding of what you were doing is it’s the same, right? I can use different tools and it might look different and they will have different commands but what I’m actually doing, it doesn’t change and my understanding of what I’m doing doesn’t change. I love that. Being able to do my work in the same way, I wish, you know, if that alone for me makes it worthwhile. [0:04:56.0] JR: Yeah, I think like my perspective is pretty much the same as what you all said and I think the one way that I kind of look at it too is Kubernetes does a better job of solving the concerns you just listed, then I would probably be able to build myself or my team would be able to solve for ourselves in a lot of cases. I’m not trying to say that specialization around your business case or your teams isn’t appropriate at times, it’s just at least for me, to your point Carlisia, I love that abstraction that’s consistent across environments. It handles a lot of the things, like Brian was saying, about CPU, memory, resources and thinking through all those different pieces. I wanted to take what we just said and maybe turn it a bit at some of the common things that people run in to with Kubernetes and just to maybe hit on a piece of low hanging fruit that I think is oftentimes a really fair perspective is Kubernetes is really hard to operate. Sure, it gives you all the benefits we just talked about but managing a Kubernetes cluster? That is not a trivial task. And I just wanted to kind of open that perspective up to all of us, you know? What are your thoughts on that? [0:06:01.8] DC: Well, the first thought is it doesn’t have to be that way. I think that’s a fallacy that a lot of people fall into, it’s hard. Guess what? That’s fine, we’re in the sixth year of Kubernetes, we’re not in the sixth year of stability of a stable release. It’s hard to get started with Kubernetes and what happens is we use that as an excuse to say well, you know what? It’s hard to get started with so it’s a failure. You know something else that was hard to get started with? Whenever I started with it in the 90s? Linux. You download it and downloading it on 30 floppy disks. There was the download corruption, real things, Z modem, X modem, Y modem. This is real, a lot of people don’t know about this. And then, you had to find 30 working flopping disk and you had to transfer 30, you know, one and a half megabyte — and it still took a long time to floppy disk and then you had to run the installer. And then most likely, you had to build a kernel. Downloading, transferring, installing, building a kernel, there was four places where just before you didn’t have windows, this was just to get you to a log in prompt, that could fail. With Kubernetes, we had this issue. People were installing Kubernetes, there’s cloud vendors who are installing it and then there’s people who were installing it on who knows what hardware. Guess what? That’s hard and it’s not even now, it’s not even they physical servers that’s networking. Well, how are you going to create a network that works across all your servers, well you’re going to need an overlay, which one are you going to use, Calico? Use Weave? You’re going to need something else that you created or something else if it works. Yeah, just we’re still figuring out where we need to be but these problems are getting solved. This will go away. [0:07:43.7] BL: I’m living that life right now, I just got a new laptop and I’m a Linux desktop kind of guy and so I’m doing it right now. What does it take to actually get a recent enough kernel that the hardware that is shipped with this laptop is supported, you know? It’s like, those problems continue, even though Linux has been around and considered stable and it’s the underpinning of much of what we do on the internet today, we still run into these things, it’s still a very much a thing. [0:08:08.1] CC: I think also, there’s a factor of experience, for example. This is not the first time you have to deal with this problem, right Duffy? Been using Linux on a desktop so this is not the first hardware that you had to setup Linux on. So you know where to go to find that information. Yeah, it’s sort of a pain but it’s manageable. I think a lot of us are suffering from gosh, I’ve never seen Kubernetes before, where do I even start and – or, I learned Kubernetes but it’s quite burdensome to keep up with everything as opposed to let’s say, if 10 years from now, we are still doing Kubernetes. You’ll be like yeah, okay, whatever. This is no big deal. So because we have done these things for a few years that we were not possibly say that it’s hard. I don’t’ think we would describe it that way. [0:09:05.7] DC: I think there will still be some difficulty to it but to your point, it’s interesting, if I look back like, five years ago, I was telling all of my friends. Look, if you’re a system’s administrator, go learn how to do other things, go learn how to be, go learn an API centric model, go play with AWS, go play with tools like this, right? If you’re a network administrator, learn to be a system’s administrator but you got to branch out. You got to figure out how to ensure that you’re relevant in the coming time. With all the things that are changing, right? This is true, I was telling my friend this five years ago, 10 years ago, continues, I continue to tell my friends that today. If I look at the Kubernetes platform, the complexity that represents in operating it is almost tailor made to those people though did do that, that decided to actually branch out and to understand why API’s are interesting and to understand, you know, can they have enough of an understanding in a generalist way to become a reasonable systems administrator and a network administrator and you know, start actually understanding the paradigms around distributed systems because those people are what we need to operate this stuff right now, we’re building – I mean, Kubernetes is a distributed system, we need people with expertise across that field, across that whole grouping of technologies. [0:10:17.0] BL: Or, don’t. Don’t do any of that. [0:10:19.8] CC: Brian, let me follow up on that because I think it’s great that you pointed that out Duffy. I was thinking precisely in terms of being a generalist and understanding how Kubernetes works and being able to do most of it but it is so true that some parts of it will always be very complex and it will require expertise. For example, security. Dealing with certificates and making sure that that’s working, if you want to – if you have particular needs for networking, but, understanding the whole idea of this systems, as it sits on top of Kubernetes, grasping that I think is going to – have years of experience under their belt. Become relatively simple, sorry Brian that I cut you off. [0:11:10.3] BL: That’s fine but now you gave me something else to say in addition to what I was going to say before. Here’s the killer. You don’t need to know distributed systems to use Kubernetes. Not at all. You can use a deployment, you can use a [inaudible] set, you can run a job, you can get workloads up on Kubernetes without having to understand that. But, Kubernetes also gives you some good constructs either in the Kubernetes API's itself or in its client libraries where you could build distributed systems in easier way but what I was going to say before that though is I can’t build a cluster. Well don’t. You know what you should do? Use a cloud vendor, use AWS, use Google, use Microsoft or no, I mean, did I say Microsoft? Google and Microsoft. Use Digital Ocean. There’s other people out there that do it as well, they can take care of all the hard things for you and three, four minutes or 10 minutes if you’re on certain clouds, you can have Kubernetes up and running and you don’t even have to think about a lot of these networking concerns to get started. I think that’s a little bit of the thud that we hear, "It’s hard to install." Well, don’t install it, you install it whenever you have to manage your own data centers. Guess what? When you have to manage your own data centers and you’re managing networking and storage, there’s a set of expertise that you already have on staff and maybe they don’t want to learn a new thing, that’s a personal problem, that’s not really a Kubernetes problem. Let’s separate those concerns and not use our lack or not wanting to, to stop us from actually moving forward. [0:12:39.2] DC: Yeah. Maybe even taking that example step forward. I think where this problem compounds or this perspective sometimes compounds about Kubernetes being hard to operate is coming from of some shops who have the perspective of are operational concerns today, aren’t that complex. Why are we introducing this overhead, this thing that we maybe don’t need and you know, to your point Brian, I wonder if we’d all entertain the idea, I’m sure we would that maybe even, speaking to the cloud vendors, maybe even just a Heroku or something. Something that doesn’t even concern itself with Kube but can get your workload up and running and successful as quickly as possible. Especially if you’re like, maybe a small startup type persona, even that’s adequate, right? It could have been not a failure of Kubernetes but more so choosing the wrong tool for the job, does that resonate with you all as well, does that make sense? [0:13:32.9 DC: Yeah, you know, you can’t build a house with a screwdriver. I mean, you probably could, it would hurt and it would take a long time. That’s what we’re running into. What you’re really feeling is that operationally, you cannot bridge the gap between running your application and running your application in Kubernetes and I think that’s fair, that’s actually a great thing, we prove that the foundations are stable enough that now, we can actually do research and figure out the best ways to run things because guess what? RPM’s from Red Hat and then you have devs from the Debian project, different ways of getting things, you have Snap from Canonical, it works and sometimes it doesn’t, we need to actually figure out those constructs in Kubernetes, they’re not free. These things did not exist because someone says, "Hey, I think we should do this." Many years. I was using RPM in the 90s and we need to remember that. [0:14:25.8] JR: On that front, I want to maybe point a question to you Duffy, if you don’t mind. Another big concern that I know you deal with a lot is that Kubernetes is great. Maybe I can get it up no problem. But to make it a viable deployment target at my organization, there’s a lot of work that goes into it to make a Kubernetes cluster production ready, right? That could be involving how you integrate storage and networking and security and on and on. I feel like we end up at this tradeoff of it’s so great that Kubernetes is super extensible and customizable but there is a certain amount of work that that kind of comes with, right? I’m curious Duff, what’s your perspective on that? [0:15:07.3] DC: I want to make a point that bring back to something Brian mentioned earlier, real quick, before I go on to that one. The point is that, I completely agree that yo do not have to actually be a distributed systems person to understand how to use Kubernetes and if that were a bar, we would have set that bar and incredibly, the inappropriate place. But from the operational perspective, that’s what we were referring to. I completely also agree that especially when we think about productionalizing clusters, if you’re just getting into this Kubernetes thing, it may be that you want to actually farm that out to another entity to create and productionalize those clusters, right? You have a choice to make just like you had a choice to make what when AWS came along. Just like you had a choice to make — we’re thinking of virtual machines, right? You have a choice and you continue to have a choice about how far down that rabbit hole as an engineering team of an engineering effort your company wants to go, right? Do you want farm everything out to the cloud and not have to deal with the operations, the day to day operations of those virtual machines and take the constraints that have been defined by that platformer, or do you want to operate that stuff locally, are you required by the law to operate locally? What does production really mean to you and like, what are the constraints that you actually have to satisfy, right? I think that given that choice, when we think about how to production Alize Kubernetes, it comes down to exactly that same set of things, right? Frequently, productionalizing – I’ve seen a number of different takes on this and it’s interesting because I think it’s actually going to move on to our next topic in line here. Frequently I see that productionizing or productionalizing Kubernetes means to provide some set of constraints around the consumption of the platform such that your developers or the focus that are consuming that platform have to operate within those rails, right? They could only define deployments and they can only define deployments that look like this. We’re going to ask them a varied subset of questions and then fill out all the rest of it for them on top of Kubernetes. The entry point might be CICD, it might be a repository, it might be code repository, very similar to a Heroku, right? The entry point could be anywhere along that thing and I’ve seen a number of different enterprises explore different ways to implement that. [0:17:17.8] JR: Cool. Another concept that I wanted to maybe have us define and think about, because I’ve heard the term platform quite a bit, right? I was thinking a little bit about you know, what the term platform means exactly? Then eventually, whether Kubernetes itself should be considered a platform. Backing u, maybe we could just start with a simple question, for all of us, what makes something a platform exactly? [0:17:46.8] BL: Well, a platform is something that provides something. That is a Brian Lyles exclusive. But really, what it is, what is a platform, a platform provides some kind of service that can be used to accomplish some task and Kubernetes is a platform and that thing, it provides constructs through its API to allow you to perform tasks. But, Kubernetes is not just a platform. Kubernetes is a platform for building platforms. The things that Kubernetes provides, the workload API, the networking API, the configuration and storage API’s. What they provide is a facility for you to build higher level constructs that control how you want to run the code and then how you want to connect the applications. Yeah, Kubernetes is actually a platform for platforms. [0:18:42.4] CC: Wait, just to make sure, Brian. You’re saying, because Kelsey Hightower for example is someone who says Kubernetes is a platform of platforms. Now, is Kubernetes both a platform of platforms, at the same time that it’s also a platform to run apps on? [0:18:59.4] BL: It’s both. Kelsey tweeted that there is some controversy on who said that first, it could have been Joe Beda, it could have been Kelsey. I think it was one of those two so I want to give a shout out to both of those for thinking in the same line and really thinking about this problem. But to go back to what you said, Carlisia, is it a platform for providing platforms and a platform? Yes, I will explain how. If you have Kubernetes running and what you can do is you can actually talk to the API, create a deployment. That is platform for running a workload. But, also what you can do is you can create through Kubernetes API mechanisms, ie. CRD’s, custom resource definitions. You can create custom resources that I want to have something called an application. You can basically extend the Kubernetes API. Not only is Kubernetes allowing you to run your workloads, it’s allowing you to specify, extend the API, which then in turn can be run with another controller that’s running on your platform that then gives you this thing when you cleared an application. Now, it creates deployment which creates a replica set, which creates a pod, which creates containers, which downloads images from a container registry. It actually is both. [0:20:17.8] DC: Yeah, I agree with that. Another quote that I remember being fascinated by which I think kind of also helps define what a platform is Kelsey put on out quote that said, Everybody wants platform at a service with the only requirement being that they’ve built it themselves." Which I think is awesome and it also kind of speaks, in my opinion to what I think the definition of a platform is, right? It’s an interface through which we can define services or applications and that interface typically will have some set of constraints or some set of workflows or some defined user experience on top of it. To Brian's point, I think that Kubernetes is a platform because it provides you a bunch of primitive s on the back end that you can use to express what that user experience might be. As we were talking earlier about what does it take to actually – you might move the entry point into this platform from the API, the Kubernetes API server, back down into CICD, right? Perhaps you're not actually defining us and called it a deployment, you’re just saying, I want so many instances off this, I don’t want it to be able to communicate with this other thing, right? It becomes – so my opinion, the definition about of a platform it is that user experience interface. It’s the constraints that we know things that you're going to put on top of that platform. [0:21:33.9] BL: I like that. I want to throw out a disclaimer right here because we’re here, because we’re talking about platforms. Kubernetes is not a platform, it’s as surface. That is actually, that’s different, a platform as a service is – from the way that we look at it, is basically a platform that can run your code, can actually make your code available to external users, can scale it up, can scale it down and manages all the nuances required for that operation to happen. Kubernetes does not do that out of the box but you can build a platform as a surface on Kubernetes. That’s actually, I think, where we’ll be going next is actually people, stepping out of the onesy-twosy, I can deploy a workload, but let’s actually work on thinking about this level. And I’ll tell you what. DEUS who got bought by Azure a few years ago, they actually did that, they built a pass that looks like Heroku. Microsoft and Azure thought that was a good idea so they purchased them and they’re still over there, thinking about great ideas but I think as we move forward, we will definitely see different types of paths on Kubernetes. The best thing is that I don’t think we’ll see them in the conventional sense of what we think now. We have a Heroku, which is like the git-push Heroku master, we share code through git. And then we have CloudFoundry idea of a paths which is, you can run CFPush and that actually is more of an extension of our old school Java applications, where we could just push [inaudible] here but I think at least I am hoping and this is something that I am actually working on not to toot my own horn too much but actually thinking about how do we actually – can we build a platform as a service toolkit? Can I actually just build something that’s tailing to my operation? And that is something that I think we’ll see a lot more in the next 18 months. At least you will see it from me and people that I am influencing. [0:23:24.4] CC: One thing I wanted to mention before we move onto anything else, in answering “Is Kubernetes right for me?” We are so biased. We need to play devil’s advocate at some point. But in answering that question that is the same as in when we need to answer, “Is technology x right for me?” and I think there is at a higher level there are two camps. One camp is very much of the thinking that, "I need to deliver value. I need to allow my software and if the tools I have are solving my problem I don’t need to use something else. I don’t need to use the fancy, shiny thing that’s the hype and the new thing." And that is so right. You definitely shouldn't be doing that. I am divided on this way of thinking because at the same time at that is so right. You do have to be conscious of how much money you’re spending on things and anyway, you have to be efficient with your resources. But at the same time, I think that a lot of people who don’t fully understand what Kubernetes really can do and if you are listening to this, if you maybe could rewind and listen to what Brian and Duffy were just saying in terms of workflows and the Kubernetes primitives. Because those things they are so powerful. They allow you to be so creative with what you can do, right? With your development process, with your roll out process and maybe you don’t need it now. Because you are not using those things but once you understand what it is, what it can do for your used case, you might start having ideas like, “Wow, that could actually make X, Y and Z better or I could create something else that could use these things and therefore add value to my enterprise and I didn’t even think about this before.” So you know two ways of looking at things. [0:25:40.0] BL: Actually, so the topic of this session was, “Should I Kubernetes” and my answer to that is I don’t know. That is something for you to figure out. If you have to ask somebody else I would probably say no. But on the other side, if you are looking for great networking across a lot of servers. If you are looking for service discovery, if you are looking for a system that can restart workloads when they fail, well now you should probably start thinking about Kubernetes. Because Kubernetes provides all of these things out of the box and are they easy to get started with though? Some of these things are harder. Service discovery is really easy but some of these things are a little bit harder but what Kubernetes does is here comes my hip-hop quote, Jay Z said this, basically he’s talking about difficult things and he basically wants difficult things to take a little bit of time and impossible things or things we thought that were impossible to take a week. So basically making difficult things easy and making things that you could not even imagine doing, attainable. And I think that is what Kubernetes brings to the table then I’ll go back and say this one more time. Should you use Kubernetes? I don’t know that is a personal problem that is something you need to answer but if you’re looking for what Kubernetes provides, yes definitely you should use it. [0:26:58.0] DC: Yeah, I agree with that I think it is a good summary there. But I also think you know coming back to whether you should Kubernetes part, from my perspective the reason that I Kubernetes, if you will, I love that as a verb is that when I look around at the different projects in the infrastructure space, as an operations person, one of the first things I look for is that API that pattern around consumption, what's actually out there and what’s developing that API. Is it a the business that is interested in selling me a new thing or is it an API that’s being developed by people who are actually trying to solve real problems, is there a reasonable way to go about this. I mean when I look at open stack, OpenStack was exactly the same sort of model, right? OpenStack existed as an API to help you consume infrastructure and I look at Kubernetes and I realize, “Wow, okay well now we are developing an API that allows us to think about the life cycle and management of applications." Which moves us up the stack, right? So for my part, the reason I am in this community, the reason I am interested in this product, the reason I am totally Kubernetes-ing is because of that. I realized that fundamentally infrastructure has to change to be able to support the kind of load that we are seeing. So whether you should Kubernetes, is the API valuable to you? Do you see the value in that or is there more value in continuing whatever paradigm you’re in currently, right? And judging that equally I think is important. [0:28:21.2] JR: Two schools of thoughts that I run into a lot on the API side of thing is whether overtime Kubernetes will become this implementation detail, where 99% of users aren’t even aware of the API to any extent. And then another one that kind of talks about the API is consistent abstraction with tons of flexibility and I think companies are going in both directions like OpenShift from Red Hat is perhaps a good example. Maybe that is one of those layer two platforms more so Brian that you were talking about, right? Where Kubernetes is the platform that was used to build it but the average person that interacts with it might not actually be aware of some of the Kubernetes primitives and things like that. So if we could all get out of our crystal balls for a second here, what do you all think in the future? Do you see the Kubernetes API becoming just a more prevalent industry standard or do you see it fading away in favor of some other abstraction that makes it easier? [0:29:18.3] BL: Oh wow, well I already see it as I don’t have to look too far in the future, right? I can see the Kubernetes API being used in ways that we could not imagine. The idea that I will think of is like KubeVirt. KubeVirt allows you to boot basically pods on whatever implements that it looks like a Kubelet. So it looks like something that could run pods. But the neat thing is that you can use something like KubeVirt with a virtual Kubelet and now you can boot them on other things. So ideas in that space, I don’t know VMware is actually going on that, “Wow, what if we can make virtual machines look like pods inside of Kubernetes? Pretty neat." Azure has definitely led work on this as now, we can just bring up either bring up containers, we can bring up VM’s and you don’t actually need a Kube server anymore. Now but the crazy part is that you can still use a workloads API’s, storage API’s with Kubernetes and it does not matter what backs it. And I’ll throw out one more suggestion. So there is also projects like AWS operators in [inaudible] point and what they allow you to do is to use the Kubernetes API or actually in cluster API, I'll use all three. But I use the Kubernetes API to boot things that aren’t even in the cluster and this will be AWS services or this could be databases across multiple clouds or guess what? More Kubernetes services. Yeah, so we are on that path but I just can’t wait to see what people are going to do with that. The power of Kubernetes is this API, it is just so amazing. [0:30:50.8] DC: For my part, I think is that I agree that the API itself is being extended in all kinds of amazing ways but I think that as I look around in the crystal ball, I think that the API will continue to be foundational to what is happening. If I look at the level two or level three platforms that are coming, I think those will continue to be a thing for enterprises because they will continue to innovate in that space and then they will continue to consume the underlying API structure and that portability Kubernetes exposes to define what that platform might look like for their own purpose, right? Giving them the ability to effectively have a platform as a service that they define themselves but using and under – you know, using a foundational layer that it’s like consistent and extensible and extensive I think that that’s where things are headed. [0:31:38.2] CC: And also more visual tools, I think is in our future. Better, actual visual UI's that people can use I think that’s definitely going to be in our future. [0:31:54.0] BL: So can I talk about that for a second? [0:31:55.9] CC: Please, Brian. [0:31:56.8] BL: I am wearing my octant hoodie today, which is a visual tool for Kubernetes and I will talk now as someone who has gone down this path to actually figure this problem out. As a prediction for the future, I think we’ll start creating better API’s in Kubernetes to allow for more visual things and the reason that I say that this is going to happen and it can’t really happen now is because for inside of an octant and whenever creating new eye views, pretty much happened now what that optic is. But what is going to happen and I see the rumblings from the community, I see the rumblings from K-native community as well is that we are going to start standardizing on conditions and using conditions as a way that we can actually say what’s going on. So let me back it up for a second so I can explain to people what conditions are. So Kubernetes, we think of Kubernetes as YAML and in a typical object in Kubernetes, you are going to have your type meta data. What is this, you are going to have your object meta data, what’s name this and then you are going to have a spec, how is this thing configured and then you are going to have a status and the status generally will say, “Well what is the status of this object? Is it deployment? How many references out? If it is a pod, am I ready to go?" But there is also this concept and status called conditions, which are a list of things that say how your thing, how your object is working. And right now, Kubernetes uses them in two ways, they use them in the negative way and the positive way. I think we are actually going to figure out which one we want to use and we are going to see more API’s just say conditions. And now from a UI developer, from my point of view, now I can just say, “I don’t really care what your optic is. You are going to give me conditions in a format that I know and I can just basically report on those in the status and I can tell you if the thing is working or not.” That is going to come too. And that will be neat because that means that we get basically, we can start building UI’s for free because we just have to learn the pattern. [0:33:52.2] CC: Can you talk a little bit more about conditions? Because this is not something I hear frequently and that I might know but then not know what you are talking about by this name. [0:34:01.1] BL: Oh yeah, I will give you the most popular one. So everything in Kubernetes is an object and that even means that the nodes that your workloads run on, are objects. If you run KubeControl, KubeCuddle, Kube whatever, git nodes, it will show you all the nodes in your cluster if you have permission to see that and if you do KubeCTL, gitnode, node name and then you actually have the YAML output what you will see in the bottom is an object called 'conditions'. And inside of there it will be something like is there sufficient memory, is the node – I actually don’t remember all of them but really what it is, they’re line items that say how this particular object is working. So do I have enough memory? Do I have enough storage? Am I out of actual pods that can be launched on me and what conditions are? It is basically saying, “Hey Brian, what is the weather outside?” I could say it's nice. Or I could be like, “Well, it’s 75 degrees, the wind is light but variable. It is not humid and these are what the conditions are.” They allow the object to specify things about itself that might be useful to someone who is consuming it. [0:35:11.1] CC: All right that was useful. I am actually trying to bring one up here. I never paid attention to that. [0:35:18.6] BL: Yeah and you will see it. So the two ones that are most common right now, there is some competition going on in Kubernetes architecture, trying to figure out how they are going to standardize on this but with pods and nodes you will see conditions on there and those are just telling you what is going on but the problem is that a condition is a type, a message, a status and something else but the problem is that the status can be true of false — oh and a reason, the status can be true or false but sometimes the type is a negative type where it would be like “node not ready”. And then it will say false because it is. And now whenever you’re inspecting that with automated code, you really want the positive condition to be true and the negative condition to be false and this is something that the K-native community is really working on now. They have the whole facility of this thing called duck typing. Which they can actually now pattern-match inside of optics to find all of these neat things. It is actually pretty intriguing. [0:36:19.5] CC: All right, it is interesting because I very much status is everything for objects and that is very much a part of my work flow. But I never noticed that there was some of the objects had conditions. I never noticed that and just a plug, we are very much going to have the K-native folks here to talk about duck typing. I am really excited about that. [0:36:39.9] BL: Yeah, they’re on my team. They’ll be happy to come. [0:36:42.2] CC: Oh yes, they are awesome. [0:36:44.5] JR: So I was thinking maybe we could wrap this conversation up and I think we have acknowledged that “Should I Kubernetes?” is a ridiculously hard question for us to answer for you and we should clearly not be the ones answering it for you but I was wondering if we could give some thoughts around — for the Podlet listener who is sitting at their desk right now thinking like, “Is now the right time for my organization to bring this in?” And I will start with some thought and then open it all up to you. So one common thing I think that I run into a lot is you know your current state and you know your desired state to steal a Kubernetes concept for a moment. And the desired state might be more decoupled services that are more scalable and so on and I think oftentimes at orgs we get a little bit too obsessed with the desired state that we forget about how far the gap is between the current state and the desired state. So as an example, you know maybe your shop’s biggest issue is the primary revenue generating application is a massive dot-net framework monolith, which isn’t exactly that easy to just port over into Kubernetes, right? So if a lot of your friction right now is teams collaborating on this tool, updating this tool, scaling this tool, maybe before even thinking about Kubernetes, being honest with the fact that a lot of value can be derived right now from some amount of application architecture changes. Or even sorry to use a buzzword but some amount of modernization of aspects of that application before you even get to the part of introducing Kubernetes. So that is one common one that I run into with orgs. What are some other kind of suggestion you have for people who are thinking about, “Is it the right time to introduce Kube?” [0:38:28.0] BL: So here is my thought, if you work for a small startup and you’re working on shipping value and you have no Kubernetes experience and staff and you don’t want to use for some reason you don’t want to use the cloud, you know go figure out your other problems then come back. But if you are an enterprise and especially if you work in a central enterprise group and you are thinking about “modernization”, I actually do suggest that you look at Kubernetes and here is the reason why. My guess is that if you’re a business of a certain size, you run VMware in your data center. I am just guessing that because I haven’t been to a company that doesn’t. Because we learned a long time ago that using virtual machines in many cases is way more efficient than just running hardware because what happens is we can’t use our compute capacity. So if you are working for a big company or even like a medium sized company, I don’t think – I am not telling you to run for it but I am telling you to at least have someone go look at it and investigate if this could ultimately be something that could make your stack easier to run. [0:39:31.7] DC: I think I am going to take the kind of the operations perspective. I think if you are in the business of coming up with a way to deploy applications on the servers and you are looking at trying to handle the lifecycle of that and you’re pretty fed up with the tooling that is out there and things like Puppet and Chef and tooling like that and you are looking to try and understand is there something in Kubernetes for me? Is there some model that could help me improve the way that I actually handle a lifecycle of those applications, be they databases or monoliths or compostable services? Any which way you want to look at it like are there tools there that can be expressed. Is the API expressive enough to help me solve some of those problems? In my opinion the answer is yes. I look at things like DaemonSet and the things like scheduling [inaudible] that are exposed by Kubernetes. And there is actually quite a lot of power there, quite a lot of capability in just the traditional model of how do I get this set of applications onto that set of servers or some subset they’re in. So I think it is worth evaluating if that is the place you’re in as an organization and if you are looking at fleets of equipment and trying to handle that magical recipe of multiple applications and dependencies and stuff. See what is the water is like on this side, it is not so bad. [0:40:43.1] CC: Yes, I don’t think there is a way to answer this question. It is Kubernetes for me without actually trying it, giving it a try yourself like really running something of maybe low risk. We can read blogposts to the end of the world but until you actually do it and explore the boundaries is what I would say, try to learn what else can you do that maybe you don’t even need but maybe might become useful once you know you can use. Yeah and another thing is maybe if you are a shop that has one or two apps and you don’t need full blown, everything that Kubernetes has to offer and there is a much more scaled down tool that will help you deploy and run your apps, that’s fine. But if you have more, a certain number, I don’t know what that number would be but multiple apps and multiple services just think about having that uniformity across everything. Because for example, I’ve worked in shops where the QA machines were taking care by a group of dev ops people and the production machines, oh my god they were taken care by other groups and now the different group of people and the two sides of these groups used were different and I as a developer, I had to know everything, you know? How to deploy here, how to deploy there and I had to have my little notes and recipes because whenever I did it – First of all I wasn’t doing that multiple times a day. I had to read through the notes to know what to do. I mean just imagine if it was one platform that I was deploying to with the CLI comments there, it is very easy to use like Kubernetes has, gives us with Kubes ETL. You know you have to think outside of the box. Think about these other operations that you have that people in your company are going to have to do. How is this going to be taught in the future? Having someone who knows your stack because your stack is the same that people in your industry are also using. I think about all of these things not just – I think people have to take it across the entire set of problems. [0:43:01.3] BL: I wanted to mention one more thing and this is we are producing lots of content here with The Podlets and with our coworkers. So I want to actually give a shout out to the TGIK. We want to know what you can do in Kubernetes and you want to have your imagination expanded a little bit. Every Friday we make a new video and actually funny enough, three fourths of the people on this call have actually done this. Where, on Friday, we pick a topic and we go in and it might be something that would be interesting to you or it might not and we are all over the place. We are not just doing applications but we are applications low level, mapping applications on Kubernetes, new things that just came out. We have been doing this for a 101 episodes now. Wow. So you can go look at that if you need some examples of what things you could do on Kubernetes. [0:43:51.4] CC: I am so glad to tgik.io maybe somebody, an English speaker should repeat that because of my accent but let me just say I am so glad you mentioned that Brian because I was sitting here as we are talking and thinking there should be a catalog of used cases of what Kubernetes can do not just like the rice and beans but a lot of different used cases, maybe things that are unique that people don’t think about to use because they haven’t run into that need yet. But they could use it as a pause, okay that would enable me to do these thing that I didn’t even think about. That is such a great catalog of used cases. It is probably the best resource. Somebody say the website again? Duffy what is it? [0:44:38.0] DC: tgik.io and it is every Friday at 1 PM Pacific. [0:44:43.2] CC: And it is live. It’s live and it’s recorded, so it is uploaded to the VMware Cloud Native YouTube and everything is going to be on the show notes too. [0:44:52.4] DC: It’s neat, you can come ask us questions there is a live chat inside of that and you can use that live chat. You can ask us questions. You can give us ideas, all kinds of crazy things just like you can with The Podlets. If you have an idea for an episode or something that you want us to cover or if you have something that you are interested in, you can go to thepodlets.io that will link you to our GitHub pages where you can actually open an issue about things you’d love to hear more about. [0:45:15.0] JR: Awesome and then maybe on that note, Podlets, is there anything else you all would like to add on “Should I Kubernetes?” or do you think we’ve – [0:45:22.3] BL: As best as our bias will allow it I would say. [0:45:27.5] JR: As best as we can. [0:45:27.9] CC: We could go another hour. [0:45:29.9] JR: It’s true. [0:45:30.8] CC: Maybe we’ll have “Should I Kubernetes?” Part 2. [0:45:34.9] JR: All right everyone, well that wraps it up for at least Part 1 of “Should I Kubernetes?” and we appreciate you listening. Thanks so much. Be sure to check out the show notes as Duffy mentioned for some of the articles we read preparing for this episode and TGIK links and all that good stuff. So again, I am Josh Russo signing out, with us also Carlisia Campos. [0:45:55.8] CC: Bye everybody, it was great to be here. [0:45:57.7] JR: Duffy Coolie. [0:45:58.5] DC: Thanks you all. [0:45:59.5] JR: And Brian Lyles. [0:46:00.6] BL: Until next time. [0:46:02.1] JR: Bye. [END OF EPISODE] [0:46:03.5] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

If you work in Kubernetes, cloud native, or any other fast-moving ecosystem, you might have found that keeping up to date with new developments can be incredibly challenging. We think this as well, and so we decided to make today’s episode a tribute to that challenge, as well as a space for sharing the best resources and practices we can think of to help manage it. Of course, there are audiences in this space who require information at various levels of depth, and fortunately the resources to suit each one exist. We get into the many different places we go in order to receive information at each part of the spectrum, such as SIG meetings on YouTube, our favorite Twitter authorities, the KubeWeekly blog, and the most helpful books out there. Another big talking point is the idea of habits or practices that can be helpful in consuming all this information, whether it be waiting for the release notes of a new version, tapping into different TLDR summaries of a topic, streaming videos, or actively writing posts as a way of clarifying and integrating newly learned concepts. In the end, there is no easy way, and passionate as you may be about staying in tune, burnout is a real possibility. So whether you’re just scratching the cloud native surface or up to your eyeballs in base code, join us for today’s conversation because you’re bound to find some use in the resources we share. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Josh Rosso Duffie Cooley Olive Power Michael Gasch Key Points From This Episode: Audiences and different levels of depth that our guests/hosts follow Kubernetes at. What ‘keeping up’ means: merely following news, or actually grasping every new concept? The impossibility of truly keeping up with Kubernetes as it becomes ever more complex. Patterns used to keep up with new developments: the TWKD website, release notes, etc. Twitter’s helpful provision of information, from opinions to tech content, all in one place. How helpful Cindy Sridharan is on Twitter in her orientation toward distributed systems. The active side of keeping up such as writing posts and helping newcomers. More helpful Twitter accounts such as InfoSec. How books provide one source of deep information as opposed to the noise on Twitter. Books: Programming Kubernetes; Managing Kubernetes; Kubernetes Best Practices. Another great resource for seeing Kubernetes in action: the KubeWeeky blog. A call to watch the SIG playlists on the Kubernetes YouTube channel. Tooling: tab management and Michael’s self-built Twitter searcher. Live streaming and CTF live code demonstrations as another resource. How to keep a team updated using platforms like Slack and Zoom. The importance of organizing shared content on Slack. Challenges around not knowing the most important thing to focus on. Cognitive divergence and the temptation of escaping the isolation of coding by socializing. The idea that not seeing keeping up to date as being a personal sacrifice is dangerous. Using multiple different TLDR summaries to cement a concept in one’s brain. Incentives for users rather than developers of projects to share their experiences. The importance of showing appreciation for free resources in keeping motivation up. Quotes: “An audience I haven’t mentioned is the audience that basically just throws up their hands and walks away because there’s just too much to keep track of, right?” — @mauilion [0:05:15] “Maybe it’s because I’m lazy, I don’t know? But I wait until 1.17 drops, then I go to the release notes and really kind of ingest it because I’ve just struggled so much to kind of keep up with the day to day, ‘We merged this, we didn’t merge this,’ and so on.” — @joshrosso [0:10:18] “If you find value in being up to date with these things, just figure out – there are so many resources out there that address these different audiences and figure out what the right measure for you is. You don’t have to go deep on the code on everything.” — @mauilion [0:27:57] “Actually putting the right content in the right channel, at least from a higher level, helps me decide whether I want to like look at that channel today, and stuff that should be in the channel is not kind of in a conversation channel.” — @opowero [0:32:21] “When I see something that is going to give me the fundamentals, like I have other priorities now, I sort of always want to consume that to learn the fundamentals, because I think in the long term phase of, but then I neglect physically what I need to know to do in the moment.” — @carlisia [0:33:39] “Just do nothing, because our brain needs that. We need to not be listening, not be reading, just nothing. Just sit and look at the ceiling. Our brain needs that. Ideally, look at nature, like look outside, look at the air, go for a walk. We need that, because that recharges the brain.” — @carlisia [0:42:38] “Just consuming and keeping up, that doesn’t necessarily mean you don’t give back.” — @embano1 [0:49:32] Links Mentioned in Today’s Episode: Chris Short — https://chrisshort.net/ Last Week in Kubernetes Development — http://lwkd.info/ 1.17 Release Notes — https://kubernetes.io/docs/setup/release/notes/ Release Notes Filter Page — https://relnotes.k8s.io/ Cindy Sridharan on Twitter — https://twitter.com/copyconstruct InfoSec on Twitter — https://twitter.com/infosec?lang=en Programming Kubernetes on Amazon —https://www.amazon.com/Programming-Kubernetes-Developing-Cloud-Native-Applications/dp/1492047104 Managing Kubernetes on Amazon — https://www.amazon.com/Managing-Kubernetes-Operating-Clusters-World/dp/149203391X Brendan Burns on Twitter — https://twitter.com/brendandburns Kubernetes Best Practices on Amazon — https://www.amazon.com/Kubernetes-Best-Practices-Blueprints-Applications-ebook/dp/B081J62KLW/ KubeWeekly — https://kubeweekly.io/ Kubernetes SIG playlists on YouTube — https://www.youtube.com/channel/UCZ2bu0qutTOM0tHYa_jkIwg/playlists Twitch — https://www.twitch.tv/ Honeycomb — https://www.honeycomb.io/ KubeKon EU 2019 — https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019/ Aaron Crickenberger on LinkedIn — https://www.linkedin.com/in/spiffxp/ Stephen Augustus on LinkedIn — https://www.linkedin.com/in/stephenaugustus Office Hours — https://github.com/kubernetes/community/blob/master/events/office-hours.md Transcript: EPISODE 17[INTRODUCTION][0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you.[EPISODE][0:00:41.5] DC: Good afternoon everybody and welcome to The Podlets. In this episode, we’re going to talk about, you know, one of the more challenging things that we all have to do, just kind of keep up with cloud native and how we each approach that and what we do. Today, I have a number of cohosts with me, I have Olive Power.[0:00:56.6] OP: Hi.[0:00:57.4] DC: Carlisia Campos.[0:00:58.6] CC: Hi everybody.[0:00:59.9] DC: Josh Rosso.[0:01:01.3] JR: Hey all.[0:01:02.8] DC: And Michael.[0:01:01.1] MICHAEL: Hey, hello.[0:01:04.8] DC: This episode, we’re going to do something a little different than we normally do. In most of our episodes, we try to remain somewhat objective around the problem and the potential solutions for it, rather than prescribing a particular solution. In this episode, however, since we’re talking about how we keep up with all of the crazy things that happen in such a fast ecosystem, we’re going to probably provide quite a number of examples or resources that you yourself could use to drive and to try and keep up to date with what’s happening out there.Be sure to check out the notes after the episode is over at thepodlets.io and you will find a link to the episodes up at the top part, click down to this episode, and check out the notes. There will be tons of resources. Let’s get started.One of the things I think about that’s interesting about keeping up with something like, you know, a Kubernetes or a fast-moving project, regardless of what that project is, whether it’s Kubernetes or, you know, for a while, it was the Mesos that I was following or OpenStack or a number have been big infrastructure projects that have been very fast moving over time and I think what’s interesting is I find that there’s multiple audiences that we kind of address when we think about what it means to ‘keep up,’ right?Keeping up with something like a project is interesting because I feel like there’s an audience that it’s actually very interested in what’s happening with the design goals or the code base of the project, and there’s an audience that is very specific to wanting to understand at a high level – like, “Give me the State of the World report like every month or so just so I can understand generally what’s happening with the project, like is it thriving? Is it starting to kind of wane? Are there big projects that it’s taking on?”And then there’s like, then I feel like there’s an audience somewhere in the middle there where they really want to see people using the project and understand, and know how to learn from those people who are using it so that they can elevate their own use of that project. They’re not particularly interested in the codebase per se but they do want to understand, are they exploring this project at a depth that makes sense for themselves? What do you all think about that?[0:03:02.0] CC: I think one thing that I want to mention is that this episode, it’s not so much about on-boarding people onto Kubernetes and the Kubernetes ecosystem. We are going to have an episode soon to talk specifically about that. How you get going, like get started. I think Duffy mentioned this so we’re going to be talking about how we all keep up with things. Definitely, there are different audiences, even when we’re talking about keeping up.[0:03:32.6] JR: Yeah, I think what’s funny about your audience descriptions, Duffy, is I feel like I’ve actually slid between those audiences a bit, right? It’s funny, back in the day, Kubernetes like one-four, one-five days, I feel like I was much more like, “What’s going on in the code?” Like trying to keep track of like how things are progressing.Now my role is a lot more focused with working with customers and standing up cube and like making a production ready. I feel like I’m a lot more, kind of reactive and more interested to see like, what features have become stable and impact me, you know what I mean? I’m far less in the weeds than I used to be. It’s a super interesting thing.[0:04:08.3] OP: Yeah, I tend to – for my role, I tend to definitely fall into the number three first which is the kind of general keeping an eye on things. Like when you see like interesting articles pop up that maybe have been linked internally because somebody said, “Oh, check out this article. It’s really interesting.”Then you find that you kind of click through five or six articles similar but then you can kind of flip to that kind of like, “Oh, I’m kind of learning lots of good stuff generally about things that folks are doing.” To actually kind of having to figure out some particular solution for one of my customers and so having to go quite deep into that particular feature.You kind of go – I kind of found myself going right in and then back out, right in, going back out depending on kind of where I am on a particular day of the week. It’s kind of a bit tricky. My brain sometimes doesn’t kind of deal with that sort of deep concentration into one particular topic and then back out again. It’s not easy.I find it quite tough actually some of the time.[0:05:05.0] DC: Yeah, I think we can all agree on that. Keeping track of everything is – it’s why the episode, right? How do we even approach it? It seems – I feel like, an audience I haven’t mentioned is the audience that basically just throws up their hands and walks away because there’s just too much to keep track of, right? I feel like we are all that at some point, you know?I get that.[0:05:26.4] OP: That’s why we have Christmas holidays, right? To kind of refresh the brain.[0:05:31.4] CC: Yeah, I maybe purposefully or maybe not even – not trying to keep up because it is too much, it is a lot, and what I’m trying to do is, go deeper on the things that I already, like sort of know. And things that I am working with on a day to day basis. I only really need to know, I feel like, I really only need to know – because I’m not working directly with customers.My scope is very well defined and I feel that I really only need to know whenever there’s a new Kubernetes release. I need to know what the release is. We usually – every once in a while, we update our project to the – we bump up the Kubernetes release that we are working against and in general, yeah, it’s like if things come my way, if it’s interesting, I’ll take a look, but mostly, I feel like I work in a spiral.If I’m doing codes related to controllers and there’s a conference talk about controllers then okay, let me take a look at this to maybe learn how to design this thing better, implement in a better way if I know more about it. If I’m doing, looking at CRDs, same thing. I really like conference talks for education but that’s not so much keeping up with what’s new. Are we talking about educating ourselves with things that we don’t know about?Things that we don’t know about. Or are we talking about just news?[0:07:15.6] JR: I think it’s everything. That’s a great question. One of my other questions when we were starting to talk about this was like, what is keeping up even mean, right? I mean, does it mean, where do you find resources that are interesting that keep you interested in the project or are you looking for resources that just kind of keep you up to date with what’s changing? It’s a great question.[0:07:36.2] MICHAEL: Actually, there was some problem that I faced when I edit the links that I wanted to share in the show. I started writing the links and then I realized, “Well, most of the stuff is not keeping up with news, it’s actually understanding the technology,” because I cannot keep up.What does help me in understanding specific areas, when I need to dig into them and I think back five or four years into early days of Kubernetes, it was easy to catch up by the time because it was just about Kubernetes. Later right, it became this platform. We realized that it actually this platform thing. Then we extended Kubernetes and then we realized there are CICD-related stuff and operations and monitoring and so the whole ecosystem grew. The landscape grew so much that today, it’s impossible to keep up, right?I think I’m interested in all those patterns that you have developed over the years that help you to manage this, let’s say complexity or stream of information.[0:08:33.9] DC: Yeah, I agree. This year, I was thinking about putting up a talk with Chris Short, it was actually last year. That was about kind of on the same topic of keeping up with it. In that, I kind of did a little research into how that happens and I feel like some of the interesting stuff that came out of that was that there are certain patterns that a project might take on that make it easier or more approachable to, you know, stay in contact with what’s happening.If we take Kubernetes as an example, there are a number of websites I think that pretty much everybody here kind of follows to some degree, that helps, sort of, kind of, address those different audiences that we were talking about.One of the ones that I’ve actually been really impressed with is LWKD which stands for Last Week in Kubernetes Development, and as you can imagine, this is really kind of focused on, kind of – I wouldn’t say it’s like super deep on the development but it is watching for things that are changing, that are interesting to the people who are curating that particular blog post, right?They’ll have things in there like, you know, code freezes coming up on this date, IPV6, IPV4, duel stack is merging, they’ll have like some of the big mile markers that are happening in a particular release and where they are in time as it relates to that release. I think if that’s a great pattern and I think that – it’s a very narrow audience, right? It would really only be interesting to people who are interested in, or who are caught up in the code base, or just trying to understand like, maybe I want a preview of what the release notes might look like, so I might just like look for like a weekly kind of thing.[0:10:03.4] JR: Yeah, speaking of the release notes, right? It’s funny. I do get to look at Last Week in Kubernetes development every now and then. It’s an awesome resource but I’ve gotten to the point where the release notes are probably my most important thing for staying up to date.Maybe it’s because I’m lazy, I don’t know, but I wait till 1.17 drops, then I go to the release notes and really kind of ingest it because I’ve just struggled so much to kind of keep up with the day to day, “We merged this, we didn’t merge this,” and so on. That has been a huge help for me, you know, day to day, week to week, month to month.[0:10:37.0] MICHAEL: Well, what was also helpful just on the release notes that the new filter webpage that they put out in 1.15, starting 1.15. Have you all seen that?[0:10:44.4] JR: I’ve never heard of it.[0:10:45.4] DC: Rel dot, whatever it is. Rel dot –[0:10:47.7] MICHAEL: Yeah, if you can share it Duffy, that’s super useful. Especially like if you want to compare releases and features added and –[0:10:55.2] DC: I’ll have to dig it up as well. I don’t remember exactly what –[0:10:56.7] CC: I’m sorry, say? Which one is that again?[0:10:59.1] MICHAEL: The real notes. I’ll put it in the hackMD.[0:11:02.8] DC: Yeah relnotes.k8s.io which is an interesting one because it’s sort of like a comparison engine that allows you to kind of compare what it would have featured like how to feature relates to different versions of stuff.[0:11:14.4] CC: That’s great. I cannot encourage enough for the listeners to look at the show notes because we have a little document here that we – can I? The resources are amazing. There are so many things that I have never even heard about and sound great – is – I want to go to this whole entire list. Definitely check it out. We might not have time to mention every single thing. I don’t want people to miss on all the goodness that’s been put together.[0:11:48.7] DC: Agreed, and again, if you’re looking for those notes, you just go to the podlets.io. Click on ‘episodes’ at the right? And then look for this episode and you’ll find that it’s there.[0:11:58.0] CC: I can see that a lot of the content in those notes are like Twitter feeds. Speaking personally, I’m not sure I’m at the stage yet where I learn a lot about Twitter feeds in terms of technical content. Do you guys find that it’s more around people’s thoughts around certain things so thought-provoking things around Kubernetes and the ecosystem rather than actual technical content. I mean, that’s my experience so far.But looking at those Twitter feeds, maybe I guess I might need to follow some of those feeds. What do you all think?[0:12:30.0] MICHAEL: Do you mean the tweets are from those like learn [inaudible 0:12:32] or the person to be tweets?[0:12:35.3] OP: You’ve listed some of there, Michael, and some sort of.[0:12:37.6] MICHAEL: I just wanted to get some clarity. The reason I listed so many Twitter accounts there is because Twitter is my only kind of newsfeed if you will. I used Feedly and RSS and others before and emails and threads. But then I just got overwhelmed and I had this feeling of missing out on all of those times.That’s why I said, “Okay, let’s just use Twitter.” To your question, most of these accounts are people who have been in the Kubernetes space for very long, either running Kubernetes, developing on Kubernetes, having opinions about Kubernetes.Opinions in general on topics related to cloud native because we didn’t want to make the search just about Kubernetes. Most of these people, I really appreciate their thoughts and some of them also just a retweet things that they see which I missed somewhere else and not necessarily just opinions. I think It’s a good mix of these accounts, providing options, some guidance, and also just news that I miss out on because not being on the other channels.[0:13:35.6] OP: Yeah, I agree because sometimes you can kind of read – I tend to require a lot of sort of blog posts and sort of web posts which, you know, without realizing it can be kind of opinionated and then, you know, it’s nice to then see some Twitter feeds that kind of actually just kind of give like a couple of words, a kind of a different view which sometimes makes me think “Okay, I understand that topic from a certain article that I’ve read, it’s just really nice to hear a kind of a different take on it through Twitter.”[0:14:03.0] CC: I think some of the accounts, like fewer of the accounts – and there are a bunch of things that – there are listed accounts here that I didn’t know before so I’ll check them out. I think fewer of the accounts are providing technical content, for example, Cindy Sridharan, not pronouncing it correctly but Cindy is great, she puts out a lot of technical content and a lot of technical opinion and observations that is really good to consume. I wish I had time to just read her blog posts and Twitter alone.She’s very oriented towards distributed systems in general, so she’s not even specific just Kubernetes. Most of the accounts are very opinionated and the benefit for me is that sometimes I catch people talking about something that I didn’t even know was a thing. It’s like, “Oh, this is a thing I should know about for the work that I do,” and like Michael was saying, you know, sometimes I catch retweets that I didn’t catch before and I just – I’m not checking out places, I’m not checking – hash tagging Reddit.I rely on Twitter and the people who I follow to – if there is a blog post that sounds important, I just trust that somebody would, that I’m going to see it multiple times until like, “Okay, this is content that is related to something and I’m working on, that I want to get better at.” Then I’ll go and look at it. My sources are mainly Twitter and YouTube and it’s funny because I love blog posts but it’s like I haven’t been reading them because it takes a long time to read a blogpost.I give preference to video because I can just listen while I’m doing stuff. I sort of stopped reading blog post which is sad. I also want to start writing posts because it’s so helpful for me to engrain the things that I’m learning and hopefully it will be helpful to other people too. But in any case, go Duffy.[0:16:02.8] DC: A number of people that I follow – I have been cultivating my feed pretty carefully, trying to get a broad perspective of technical stuff that’s happening. But also I’ve been trying to develop my persona on Twitter a bit more, right? I’m actually trying to build my audience there. What’s interesting there is I’ve been trying to – to that end, what I’ve been doing is like trying to amplify voices that I think aren’t heard enough out there, right?If I see an article by somebody who is just coming into Kubernetes. or just coming into distributed systems and they’ve taken an effort to really lay out something that they found really interesting about pretty much anything, right? I’m like, “Okay, that’s pretty awesome,” and I’ll try to amplify that, right? Sometimes I even get involved or I’ll, not directly in public on Twitter but I’ll offer to help edit or help provide whatever our guidance I can provide around that sort of stuff.If I see people like having a difficult time with a particular project or something like that, I’ll reach out privately and say, “Hey, can I help you with it so you can go out there and do a great job,” you know? That is something I love to do. I think your point about like not necessarily going at Twitter for the deep knowledge stuff but more just like making sure that you have a broad enough awareness of what’s happening in different ecosystems that you’re not surprised by the things when the things change, right?A couple of other people that I follow are Akira Asuta, I can’t say enough about that person. They are amazing, they have been doing like, incredibly deep security stuff as it relates to containerization and stuff like that for quite a while. I’m always like, learning brand new things to me when following folks like that. I’ve been kind of getting more interested in InfoSec Twitter lately, learning how people kind of approach that problem.Also some of the bias arounds that which has been pretty interesting. Both the bias against people who are in InfoSec which seems weird to me. Also, how InfoSec approaches a problem, like do they put it like a learning experience or they approach it like an attack experience.It’s been kind of fascinating to get in there.[0:18:08.1] OP: You know, I kind of use Twitter as well for some of this stuff but you know, books are kind of a resource as well but in my head, kind of like at the opposite scale. You know, I obviously don’t read as many books as I read twitter feeds, right? It’s just kind of like, with Twitter, you can kind of digest the whole of the stuff and with books, it’s kind of like – I tend to be trying – because I know, I’m only going to read – like I’m only going to read maybe one/two books a year.I’ve kind of like – as I said before, blog posts seem to take up my reading time and books kind of tend to be for like on airplanes and stuff. So if – they’re just kind of two opposite resources for me but I find actually, the content of books are probably stuff that I digest a bit more because you know, it’s kind of like, I don’t know, back to the old days. It’s kind of a physical thing on hand and I can kind of read it and digest it a bit more than the kind of throwaway stuff that kind of keeps on Twitter.Because to be honest, I don’t know what’s on Twitter. Who is kind of a person to listen to or who is not or who is – I just try and form my own opinions and then, again, it kind of gets a bit overwhelming, because it’s a lot of content just streaming through continuously, whereas a book, it’s kind of like just one source of information that is kind of like a bit more personal that I can digest a bit more.[0:19:18.1] JR: Any particular book recommendation in 2019, Olive, that you found particularly interesting?[0:19:23.5] OP: I’m still reading, and it’s on the list for the episode notes actually, Programming Kubernetes. I just want to kind of get into that sort of CRD sort of mindset a bit. I think that’s kind of an area that’s interesting and an area that a lot of people will want to use in their organizations, right, because it’s going to do some of the extensibility to Kubernetes that’s just not there out of the box and everybody wants something that’s not out of the box or always in my experience.[0:19:47.4] MICHAEL: I found the Managing Kubernetes, I think was it, by – from Brendan Burns and some other folks which was just released I think in the end of last year. Super deep and that is kind of the opposite to the Programming Kubernetes, because I like that as well. That is more geared towards understanding architecture and operations.Operational concepts –[0:20:05.0] OP: They’re probably the two books I’ve read.[0:20:08.4] MICHAEL: Okay.[0:20:08.9] OP: One a year, remember?[0:20:11.4] MICHAEL: Yeah.[0:20:14.6] OP: Prolific reading.[0:20:19.6] CC: I think if you know what you need to learn about cloud native or Kubernetes, there’s amazing books out there, and if you are still exploring Kubernetes and trying to learn, I cannot recommend this book enough. If you are watching this on YouTube, you’ll see the cover. It’s called Kubernetes Best Practices because it’s about Kubernetes best practices but what they did simultaneously and maybe they didn’t even realize is just they gave a map for the entire thing.You go, “Oh, these are all the elements in Kubernetes.” Of course, it’s saying, “Okay, this is the best way to go about setting the stuff up,” and this is relatively thin but I just think that going through this book, you get really fast overview of the elements in Kubernetes. Then you can go to other books like Managing Kubernetes to go deep and understand all of the knobs and switches.[0:21:24.6] DC: I want to bring it back to the patterns that we see successful projects. Projects that you think are approachable but, you know, projects that are out there that make it easy for you to kind of stay – or easier at least to stay up to date with them, what some of those patterns are that you think are useful for projects.We’re talking about like having a couple of different entry points from kind of a weekly report mechanism, we’ve talked about the one that LWKD is, I don’t think we got to talk about KubeWeekly which is actually a weekly blog that is actually curated by a lot of the CNCF ambassadors. KubeWeekly is also broken up in different sections, so like sometimes they’ll just talk about – but they’re actually going out actively and trying to find articles of people using Kubernetes and then trying to post those.If you’re interested in understanding how people are actually out there using it, then that’s a great place to go find articles that are kind of related to that. What are some other patterns that we see that are out there that are useful for books?[0:22:27.6] DC: One that I really like. Kubernetes, for everyone listening has this notion of special interest groups, SIGs oftentimes. They’re focused on certain areas of the project. There’s some for networking and storage and life cycles of clusters and what’s amazing, I try to watch them somewhat weekly, I don’t always succeed.They’re all on YouTube and if you go to the Kubernetes project YouTube, there’s playlists for every SIG. A lot of times I’m doing work relating to life cycles of clusters. I’ll open up the cluster life cycle playlist and I’ll just watch the weekly meetings. While it doesn’t always pertain to completely to me, it lets me understand kind of where the developers and contributor’s heads are at and where they’re kind of headed with a lot of different things.There’s a link to that as well if anyone wants to check it out.[0:23:15.9] MICHAEL: Exactly, to add to that. If you don’t have the time to watch the videos, the meeting notes that these gentlemen and women put together are amazing. Usually, I just scroll through and if it’s something to triggers, I go into the episode and watch it.[0:23:28.7] OP: I almost feel like we should talk about tooling to handle all of this stuff, for example, right now, I think I have 200 tabs opened. I just started learning about some chrome extensions to manage tabs. I haven’t started really using them but I need. I don’t have a good system. My system is open a video that I’m pretty sure I want to watch and just get to that tab eventually until something happens in my chrome goes bust and I lose everything.I wanted to mention that when we say watch YouTube, some things you don’t need to sit there and actually watch, you can just listen to it and if you pay for the five bucks for YouTube premium – I don’t get a commission you people, but I’m just saying, for me, it’s so helpful. I can just turn off you know, put my phone on my pocket and keep listening to it without having to have the phone open and on the whole time. It’s very handy.It’s just like listening to a podcast. I also listen to podcasts lots of days.[0:24:35.1] MICHAEL: For tooling, since I’m just mostly on Twitter and by the time I was using or starting to use Twitter, they didn’t have this bookmark function, so I was basically abusing likes or favorites at the time, I think, to bookmark. What I realized later, my bookmarks grew, well, my likes grew.I wanted to go back and find something but that through the Twitter search was just impossible. I blew the tiny little go tool, kind of my first exercise there to just parse my likes and then use JQ because it’s all JSON to query and manipulate the stuff. I almost use it every day because I was like, that was a talk or blog post about scheduling and just correct for scheduling and the likes.I’m sure there’s a better tool or way of doing that but for me, that’s mine too. Because that’s my workflow.[0:25:27.6] DC: Both of the two blogs that you mentioned both KubeWeekly and LWKD, they both have the ability to take – you can submit stories to them. If you come across things that are interesting and you’d like to put that up on an aggregator somewhere, this is one of the ways to kind of solve that problem because at least if it gets cleared up on an aggregator, you know that you go back to the aggregator to see it, so that helps.Some other ones I’ve seen out there, I’ve seen people, I’ve seen a number of interesting startups now, starting to kind of like put out a podcast or – and I have started to see a number of people like you know, engaging with Twitch and also doing things like what we do with TJK.io which is like have sort of some kind of a weekly thing where you are just hacking on stuff live and just exploring it whether that is related to – if you think of about TJK is we’re going to do without being related necessarily to anything that we are doing at VMware just anything to do with the community but obviously if you are working for one of the small companies like Honeycomb or some other company.A smaller kind of startup, you can really just get people more aware of that because for some reason people love to watch others code. They love to understand how people go through that, what are their thought process is and I find it awesome as well. I think it is amazing to me how big a draw that is, you know?[0:26:41.1] OP: And is there lots of them out there Duffy? Is that kind of an easy searchable thing or is it like how do you know those things are going on?[0:26:48.4] DC: Oddly enough Twitter, most of the time, yeah. I mean, most of the time I see that kind of stuff happening on Twitter, like somebody will like – I will scope with this or a number of other people will say, “Hey, I am going to do a live stream during this period of time on this,” and I have actually seen a number of people doing live streams on CTFs, which are capture the flags. That one’s really been fascinating to me because it has been how do people think about approaching the security of an application.Like where do they look for weak spots and how do you determine, how do you approach that kind of a problem, which is fascinating. So yeah, I think it is important to remember that like you know, you are not the only one trying to keep up to date with all of this stuff, right? The one thing we all have said pretty consistently here is that it is a lot, and it is not just Kubernetes, right? Like any fast moving project. It could be your favorite Ruby module that has 200 contributors, right?It doesn’t matter what it is, it is a lot to keep a track of, and it represents some of that cognitive overheads that you have to think about. That is a lot to take on. Even if it is overwhelming, if you find value in being up to date with these things, just figure out – there are so many resources out there that address these different audiences and figure out what the right measure for you is. You don’t have to go deep on the code on everything.Sometimes it might be better to just try and find a source of information that gives you a high enough of a view. Maybe you are looking at the blog posts that come out on Kubernetes.io every release and you are just looking at the release notes and if you just read the release notes every release, that is already miles ahead of what I have seen a lot of folks out there when they are starting to ask me questions about how do you keep up to date.[0:28:35.9] JR: I’m curious, we have been talking a lot about keeping up as an individual. Do you all have strategies for how you help, let’s say your overall team, keep up with all the things that are going on? To give an example, Duffy, Olive and myself, at least at one point, were on the same team and we’d go out to disparate customers and see all of these different new things that they are trying to do or new projects that they are using.So we’d have to think about how do we get together and share that internally to make sure we are bringing the whole team along with what is going on in the ecosystem especially from a customer perspective. I know one of the ways that we do that is having demos and things of that nature that we share weekly. Are there other strategies that you all use with your teams to kind of share interesting information and news?[0:29:25.5] M: So what we do is mostly the way we share in our team, and we are a small team. We use Slack. We pre-filter in terms of like if there is stuff that I think is valuable for me and probably not for the whole team – obviously we are not going to share, but I think if it is related to something that the team has or to come grant and then I will share on Slack but we don’t have any formal way. I know people use some reports, weekly reports, or other platforms to distribute but we just use Slack.[0:29:53.0] DC: I think one of the things – one of the patters that we had at [inaudible 0:29:54] that I thought was actually super helpful was that we would engage a conversation. “I learned a cool new thing about whatever today,” and so we would say, “I am going to – ” and then we would start a Zoom call around that and then people could join if they wanted to, to be a part of the live discussion or not, and if they didn’t, they would still be able to see a recorded Zoom pop up in the channel later on.So even if your time zones don’t line up, like I know it is 2 AM or 3 AM or something like that for Olive right now, you can still go back to those recorded sessions and you’ll just see it on your daily Slack stuff. You would be able to see, “Oh there was a conversation about whether you should deploy Kubernetes crossed availibility zones or not. I would like to go see that,” and see what the inputs were, and so that can be helpful.[0:30:42.5] JR: Yeah, that is a super interesting observation. It is almost like remote-first teams that are used to these processes of recording everything and putting it in a Google doc. They are more equipped for that information sharing perhaps than like the water cooler conversations you’d have in the office.[0:30:58.5] OP: And on the Slack or any of the communication tool, we have different channels because we are all in lots of channels and to have channels dedicated to a particular subject is absolutely the way to go because otherwise in my previous company that seem to be kind of one main channel that all the architect used to discussed everything on and you know sometimes you join and you’re like, “What is everybody talking about?”There would be literally about a hundred messages on some sort of theme that I have never heard of. So you come away from that thinking that, “That is the main channel. Where is the bit – is there messages in the middle that I missed that were just normal discussions as opposed to in around the technical stuff,” and so it made me a bit sad, right? I would be like, “I haven’t understood something and there is a whole load of stuff on this channel that I don’t understand.”But it is the kind of central channel for everyone. So I think you end up then start looking up things that they are discussing and then realizing actually that is not really anything related to what I need to know about today or next week. It might be something for the future but I’ve got other stuff to focus on. So my point is that those communication channels for me sometimes can make me feel a little bit behind the curve or very much sort of reactive in trying to jump on things that are actually not really anything to do with me for me now and wasting my time slightly and kind of messing with my head a little bit in that like, “I really need to try and focus out stuff,” and actually putting the right content in the right channel, at least from a higher level, helps me decide whether I want to like look at that channel today, and stuff that should be in the channel is not kind of in a conversation channel. So organization of where that content is, is important to me.[0:32:37.6] CC: I am so in the same page with you Olive. That is the way my brain works as well. I want to have multiple channels, like if we are talking about Slack or any chat tool, but some people have such aversion to multiple channels. They really have a hard time dealing with too many – like testing their threshold of what they think is too many channels. So I am always mindful too, like it has to work for everybody but if it was up to me, there will be one channel per topic. So I know where to focus on.But you said something that is so interesting. How do we even just – like you were saying in the context of channel, multiple channels, and I go, if I need to pay attention to this this week as oppose to like, I don’t need to look at this until some time in the future. How do we even decide what we focus on that is useful for us in the moment versus it would be good for me to know but I don’t need to know right now.I am super bad at this. When I see something that is going to give me the fundamentals, like I have other priorities now, I sort of always want to consume that to learn the fundamentals because I think in the long term phase of, but then I neglect physically what I need to know to do in the moment and I am trying to sort of fish there and get focused on in the moment things. Anybody else have a hard time?[0:34:04.5] DC: You are not alone on that, yeah.[0:34:06.7] CC: It is terrible.[0:34:08.3] MICHAEL: Something that I wish I would do more often as like being a good citizen is like when you read a lot, probably 90% of my time is not writing but reading, maybe even more and then I share and then on Twitter, the tweet for them the most successful ones in terms of retweets or likes are the ones where I do like TLDR’s or some screen captures like too long to read. Where people don’t have the time, they might want to read the article but they don’t have the time.But if you put in like a TLDR like either a tweet or a thread on it, a lot of people would jump onto it because they can just easily capture it and they can still read the full article if they want but that is something that I learned and it is pretty – what is the right word? Helpful to my followers and the community but I just don’t do it that often unfortunately. If I am writing, summarizing, writing, I kind of remember. That is how the brain works. It is a nice side effect.[0:35:04.9] DC: I was saying, this is definitely one of those things where you can be the change you want to see if you, you know?[0:35:08.6] M: Yeah, I know.[0:35:10.0] DC: This is awesome. I would also say that what you just raised Carlisia is like a super valid point. I mean like not everybody’s brain works the same way, right? There are people who are neuro-divergent. There are people who think very linearly and they are very comfortable with that and there are people who don’t. So it is a struggle I think regardless of how your brain is wired to understand to how to prioritize the attention you will give any given subject.In some cases, your brain is not wired – your brain is almost wired against that whole idea, like you are just not set up for success when it comes to figuring out how to prioritize your attention.[0:35:49.0] CC: You hit the nail on the head. We are so set up for failure in that department because there are so many interesting conversations and you want to hop in and you want to be a part of the conversation and part of the group and socialize. Our work is so isolating to really put our heads down and just work, it can be so isolating. So it is great to participate in conversations out there even if it is for only via Twitter. I mean, obviously we are very biased towards Twitter here in this group.But I am not even this on Twitter so just keep that in mind that we are cognizant of that but in any case, I don’t know what the answer is but what I am trying always to cut down on that, those social activities that seem so appealing. I don’t know how to do that from working out.[0:36:43.9] JR: I am in the same boat. 2020, I am hoping to let more of that go and to your point, it is not that there is no value in it. It is just, I don’t know, I am not deriving the same amount of quality out of it because I am so just multiplexed all over the place, right? So we’ll see how it goes.[0:36:59.9] CC: Oh if any listener has opinions and obviously it seems that all of us are helpless in that department. Share with us, please.[0:37:12.5] DC: It is a tricky one. I think it is also interesting because I find that when we talk about things like work-life balance, we think of the idea of maybe work-life balance is that when you come at the end of the day and you go home and you don’t think about work, right? Sometimes we think that work-life balance means that you have a certain amount of time off that you can actually spend with your family and your friends or your community, what have you, and not be engaging on multiple fronts.Just be that – have that be your focus, but when it comes to things like keeping up, when it comes to things like learning or elevating your education and stuff, it seems like, for the most part, and this is just my own assumption, I am curious how you all feel about this, that we don’t – that that doesn’t enter into it, right? Your personal time is totally on the table when it comes to how do you keep up with these things. We don’t even think about it that way, right?I know I personally don’t. I definitely have to do more and cut back on the amount of time that I spend reading. I am right there with Michael on 90% of my time when my eyes are open, they are either reading or staring up on the sky while I try to think about what I am going to write next. You know one way or the other it is like that is what I am doing.[0:38:24.0] CC: Yeah.[0:38:25.1] MICHAEL: I noticed last year on my Twitter feed, more people than the years before will complain about like personal burn out. I saw a pattern, like reading those people’s tweets, I saw a pattern there. It wasn’t really like a spiral and then they realized and they shot down like deleted Twitter from their phones or any messaging and other stuff, and I think I am at the point where I also need to do that when it comes to vacation PDO, or whatever.Because I am just like, as you said Duffy, my free time is on the table when it comes to Twitter and catching up and keeping up because work-life balance in my mind is not work but what is not work for like – Kubernetes is exciting, adding in all the space, like what is not work there? I need to really get better at that because I think I might end in the same spiral of just soaking in more until I just –[0:39:17.7] CC: Yeah and like Josh said, it is not that there isn’t a value. Obviously we derive a huge value, that is why we’re on it, but you have to weigh things and what are your goals and is that the best way to your goals from where you are right now, and maybe you know, Twitter you use for a while, ramp up your knowledge, ramp up the connections because it is great for making connections, and then you step back and focus on something else, then to go on a cycle.This is how I am thinking now. It is just like what Olive was saying, you know, books are great, blog posts are great, and I absolutely agree with that. It is just that I don’t have even the time and when I have the time, I would be reading code and I would be reading things all day long, it is just really tiring for me at the end of the day to sit down and read more. I want to invest in learning how to speed read to solve that problem because I read a lot of books and blog posts. So something on my list.[0:40:22.8] DC: One of the biggest tips on speed reading I ever learned is that frequently when you read you think of saying the word and if you can get out of that habit, if you get out of the habit of saying the word even with your mouth or you just get out of that habit that will already increase the quickness of what you read.[0:40:39.5] CC: That is so interesting.[0:40:41.4] DC: Yeah, that is a trippy one.[0:40:43.1] CC: Because I think being bilingual, I totally like – that really helps me understand things, by saying the words.[0:40:52.9] DC: I think the point that we are all working around here is, there is a great panel that came out at KubeCon EU in 2019 was put on by Aaron Crickenberger, Esther McNaMara, Steven Augustus, these folks are all very high output people. I mean, they do a lot of stuff especially with regard to community and so they put on a panel that was talking about burn out and self-care and I think that it is definitely worth checking that one out.And actually also thinking about what keeping up means to you and making sure that you are measuring that against your ability to sustain, is incredibly important, right? I feel like keeping up is one of those subjects where we end up – it is almost insidious in its way to – it is a thing that we can just do all the time. We can just spend all of our time, any free moment that you have, you are sitting on the bus, you are trying to keep up with things.And because that happens so much, I feel like that is sort of one of the ways that we can feel burnt out as you are seeing today. We can feel like we did a lot of things but there was no real result to it and keep in mind that that’s part of it, right? Like when you are thinking about how we are keeping up with it, make sure that the value to your time is still something that you have some cognizance about, that you have some thought about, like is it worth it to me to just spend this six hours reading everything, right?Or would it be better for me to spend some amount of time just not reading, you know? Like doing something else, you know? Like bake a cake for crying out loud, you know?[0:42:29.5] CC: Something that a lot of times we don’t allow ourselves to do and I decided to speak for everybody I am sorry, I just do nothing, because our brain needs that. We need to not be listening, not be reading, just nothing. Just sit and look at the ceiling, our brain needs that. Ideally, look at nature, like look outside, look at the air, go for a walk. We need that, because that recharges the brain. Anyway, one thing also that I want to bring up, maybe we can mention real quick because we are coming up at the top of the hour.How do people, projects, how do we really help the users of those projects to be up to date with what they are doing?[0:43:18.4] DC: Well yeah I mean this is the different patterns that we are talking about. So I think the blog posts help. I like the idea of having blogs that are targeted towards different audiences. I like the idea of having an aggregate here for putting up a big project. I mean obviously Kubernetes is such a huge ecosystem that if you have things like KubeWeekly and I know that there are actually quite a number of things out there that try and do this.But if we can kind of agree on one like KubeWeekly I think is a pretty good one because it is actually run by the CNCF. So it kind of falls within that sort of governance as a model but having an aggregator where you can actually produce content or curate content as it relates to your project that’s helpful, and then office-hours I think is also helpful to Josh’s point. I mean office-hours and SIG hours are very similar things. I mean like office-hours there like how to developers think about what’s happening with the space.This is an opportunity for you as an end user to show up and ask questions, those sorts of patterns I think all are incredibly helpful as a project to figure out there to those things.[0:44:17.8] OP: Yeah, I know summary articles or the sort of TLDRs that Michael mentioned earlier, I think I need more of those things in my life because I do a lot of reading, because I think my brain is a bit weird in that I need to read something about five or six different times from five or six different articles for it to sort of frame in my head.So what I am trying to – like for 2020, I have almost tried to do this, is like if I think somebody knows all about this and it would save me reading those five, six, seven articles and if that person has the time, I try and sort of reach out to them and say, “Listen, have you got 20 minutes or so to explain this topic to me? Can I ask you questions about it?” It just saves me, saves my eyes reading the screen, and it just saves me time. I just need a TLDR summary of a project or a feature or something just so I can know what it is all about in my head and talk fairly sort of confidently about it.If I need to get in front and down under the weeds then there is more reading to kind of do for me maybe the coding on the technical side, but sometimes I can’t figure out what this feature sort of means and what is its use case in the real world and I have to read through lots of articles and sometimes kind of vendor specific ones and they’ve got a different slant than maybe an independent one and trying to marry those bits up my head is a bit hard for me and there is sort of wealth of information.So if you are interested in a topic and there is hundreds of articles and you start reading four or five and they are all slightly different, eventually you figure out that – you are confident and I understand what that product is about but it has taken a long time to get there and it is taken a lot of reading time. So TLDRs is like really work and I think as Josh mentioned before, we have this thing internally where we do bench demos.And that is like a TLDR and a show and tell really quickly, like, “This is what this does and this is why we need to know about it and this is why our customers needs to know about it, the end,” you know? And that’s really, really useful because that just saves a whole bunch of people a whole bunch of time figuring out A, whether they need to know about it and B, actually now understanding that product or feature at the end of the five, 10 minutes which is what they typically are. So they are very useful short snippets of information. Maybe we are back to Twitter.[0:46:37.8] JR: Similar to the idea of giving a demo Olive, you made me think of something and that is that I think one of the ways that I keep up with the space is actually through writing along with reading and I think the notion of like – and this admittedly takes up time and the whole quality of life conversation comes in but using writing to help develop your thoughts and kind of aggregate all of these crazy inputs and try to be somewhat concise, which I know I struggle with, around something I’ve learned.It’s helped me a ton and then that asset kind of becomes reusable to share with other people the thing that you wrote. So for people listening to this I guess maybe a call to action for 2020 if that is your style as well, consider starting to write yourself and becoming a resource, right? Because even if you are new to this space, you’d be amazed at just how writing from your perspective can help other people.[0:47:26.3] DC: I think another one that I actually have been impressed with lately is that a number of consumer companies like people out there like Lyft and companies like that have actually started to surface engineering blogs around how they are using technology and how they are using technology to solve things, which I think, as a service provider, as somebody who is involved in the community of Kubernetes, I find those to be incredibly valuable because I get to actually see how those things are doing.I mean at the same time, I see things like – we talked about KubeCon, which is a convention that they have every year. Obviously the project is large enough to support it but there is actually an incentive if you are a consumer of that project to go and talk about how you are using it, right? It is incentivized in that it is more likely your talk will be accepted if you are a consumer of the product than somebody building it, right? We hear from people building it all the time.I love that idea of incentivizing people who are using this thing get out there and talk about it or share their ideas about it or how they are using it, what problems did it solve for them. That is critical I think.[0:48:31.0] CC: Can I also make a suggestion – is to not so much following on the thread that we are talking about just now but kind of on the general thread of this episode. If you have resources that you do use to keep up with things, stop this recording right now and go and give them a like, give them a follow, give them a thumbs up, show somehow appreciation because what Duffy said just now, he was saying, “Oh it is so helpful when I read a blog post.”But people who are writing, they want to know that. So give them some indication, it counts a lot. It takes a lot of effort to sit down and write something or produce a podcast and if you take any, derive any benefit from it, show appreciation. It motivates people to keep doing it.[0:49:26.4] DC: Yeah, agreed.[0:49:27.9] M: I think that is a great bind maybe to close off this episode because it reiterates that just consuming and keeping up that doesn’t necessarily mean you don’t give back, right? So this is a way of giving back, which is really important to keep that flow and creativeness.[0:49:41.8] CC: I go through a lot of YouTube videos and sometimes I just play one after the other but sometimes, you know, I have been making a point of going back and liking it. Liking the ones that I like – obviously I don’t like everything. I mean things that I don’t like I don’t listen in but you know what I mean? It takes no effort but just so people know, “OK, you did a good job here.” By the way, go to iTunes and rate us. So we will know that you liked it and it will help people find our show, our podcast, and if you are watching us on YouTube, give us a like.[0:50:16.1] DC: All right, well unless anybody has any final thoughts, that is what we wanted to cover this session. So thank you all very, very much and I look forward to seeing you next week.[0:50:25.3] M: Bye-bye.[0:50:26.3] CC: Thank you so much.[0:50:27.4] OP: Bye.[0:50:28.1] JR: Bye.[END OF EPISODE][0:50:28.7] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing.[END]See omnystudio.com/listener for privacy information.

Do you know what cloud native apps are? Well, we don’t really either, but today we’re on a mission to find out! This episode is an exciting one, where we bring all of our different understandings of what cloud native apps are to the table. The topic is so interesting and diverse and can be interpreted in a myriad of ways. The term ‘cloud native app’ is not very concrete, which allows for this open interpretation. We begin by discussing what we understand cloud native apps to be. We see that while we all have similar definitions, there are still many differences in how we interpret this term. These different interpretations unlock some other important questions that we also delve into. Tied into cloud native apps is another topic we cover today – monoliths. This is a term that is used frequently but not very well understood and defined. We unpack some of the pros and cons of monoliths as well as the differences between monoliths and microservices. Finally, we discuss some principles of cloud native apps and how having these umbrella terms can be useful in defining whether an app is a cloud native one or not. These are complex ideas and we are only at the tip of the iceberg. We hope you join us on this journey as we dive into cloud native apps! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Bryan Liles Josh Rosso Nicholas Lane Key Points From This Episode: What cloud native applications mean to Carlisia, Bryan, Josh, and Nicholas. Portability is a big factor of cloud native apps. Cloud native applications can modify their infrastructure needs through API calls. Cloud native applications can work well with continuous delivery/deployment systems. A component of cloud native applications is that they can modify the cloud. An application should be thought of as multiple processes that interact and link together. It is possible resources will begin to be requested on-demand in cloud native apps. An explanation of the commonly used term ‘monolith.’ Even as recently as five years ago, monoliths were still commonly used. The differences between a microservice approach and a monolith approach. The microservice approach requires thinking about the interface at the start, making it harder. Some of the instances when using a monolith is the logical choice for an app. A major problem with monoliths is that as functionality grows, so too does complexity. Some other benefits and disadvantages of monolith apps. In the long run, separating apps into microservices gives a greater range of flexibility. A monolith can be a cloud native application as well. Clarification on why Brian uses the term ‘microservices’ rather than cloud native. ‘Cloud native’ is an umbrella term and a set of principles rather than a strict definition. If it can run confidently on someone else’s computer, it is likely a cloud native application. Applying cloud native principles when building an app from scratch makes it simpler. It is difficult to adapt a monolith app into one which uses cloud native principles. The applications which could never be adapted to use cloud native principles. A checklist of the key attributes of cloud native applications. Cloud native principles are flexible and can be adapted to the context. It is the responsibility of thought leaders to bring cloud native thinking into the mainstream. Kubernetes has the potential to allow us to see our data centers differently. Quotes: “An application could be made up of multiple processes.” — @joshrosso [0:14:43] “A monolith is simply an application or a single process that is running both the UI, the front-end code and the code that fetches the state from a data store, whether that be disk or database.” — @joshrosso [0:16:36] “Separating your app is actually smarter than the long run because what it gives you is the flexibility to mix and match.” — @bryanl [0:22:10] “A cloud native application isn’t a thing. It is a set of principles that you can use to guide yourself to running apps in cloud environments.” — @bryanl [0:26:13] “All of these things that we are talking about sound daunting. But it is better that we can have these conversations and talk about things that don’t work rather than not knowing what to talk about in general.” — @bryanl [0:39:30] Links Mentioned in Today’s Episode: Red Hat — https://www.redhat.com/en IBM — https://www.ibm.com/ VWware — https://www.vmware.com/ The New Stack — https://thenewstack.io/ 10 Key Attributes of Cloud-Native Applications — https://thenewstack.io/10-key-attributes-of-cloud- native-applications/ Kubernetes — https://kubernetes.io/ Linux — https://www.linux.org/ Transcript: EPISODE 16 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.4] NL: Hello and welcome back, my name is Nicholas Lane. This time, we’ll be diving into what it’s all about. Cloud native applications. Joining me this week are Brian Liles. [0:00:53.2] BL: Hi. [0:00:54.3] NL: Carlisia Campos. [0:00:55.6] CC: Hi everybody, glad to be here. [0:00:57.6] NL: And Josh Rosso. [0:00:58.6] JR: Hey everyone. [0:01:00.0] NL: How’s it going everyone? [0:01:01.3] JR: It’s been a great week so far. I’m just happy that I have a good job and able to do things that make me feel whole. [0:01:08.8] NL: That’s awesome, wow. [0:01:10.0] BL: Yeah, I’ve been having a good week as well in doing a bit of some fun stuff after work. Like my soon to be in-laws are in town so I’ve been visiting with them and that’s been really fun. Cloud native applications, what does that mean to you all? Because I think that’s an interesting topic. [0:01:25.0] CC: Definitely not a monolith. I think if you have a monolith running on the clouds, even if you start it out that way, I wouldn’t say it’s a cloud native app, I always think of containerized applications and if you’re using the container system then it’s usually because you want to have a smaller systems in more of them, that sort of thing. Also, when I think of cloud native applications, I think that they were developed the whole strategy of the development in the whole strategy of deploying and shipping has been designed from scratch to put on the cloud system. [0:02:05.6] JR: I think of it as applications that were designed to run in container. And I also think about things like services, like micro services or macro services to know what you want to call them that we have multiple applications that are made to talk not just with themselves but with other apps and they deliver a bigger functionality through their coordination. Then what I also want to go cloud native apps, I think of apps that we are moving to the cloud, that’s a big topic in itself but applications that we run in the cloud. All of our new fancy services and our SaaS offerings, a lot of these are cloud native apps. But then on the other side, I think about applications, they are cloud native are tolerant to failure and on the other side, can actually talk about sells of their health and who they’re talking to. [0:02:54.8] CC: Gets very complicated. [0:02:56.6] BL: Yeah. That was the side of that I haven’t thought about. [0:03:00.7] JR: Actually, it’s for me that always come to mind are obviously portability, right? Wherever you're running this application, it can run somewhat consistently, be it on different clouds or even a lot of people, you know, are running their own cloud which is basically their on-prem cloud, right? That application being able to move across any of those places and often times, containerization is one of the mechanisms we use to do that, right? Which is what we all stated. Then I guess the other thing too is like, this whole cloud ecosystem, be it a cloud provider or your own personal – are often times very API driven, right? So, the applications, maybe being able to take advantage of some of those API’s should they need to. Be it for scaling purposes otherwise. It’s really interesting model. [0:03:43.2] NL: It’s interesting, for me like this question because so far, everyone is getting similar but also different answers. And for me, I’m going to give a silent answer to me, a cloud native application is a lot of things we said like portable. I think of micro services when II] think of a cloud native application. But it’s also an application that can modify the infrastructure it needs via API calls, right? If your application needs a service or needs a networking connection, it can – the application itself can manifest that via cloud offering, right? That’s what I always thought of as a cloud native application, right? If you need like a database, the application can reach out to like AWS RDS and spin up the database and that was an aspect of I always found very fascinating with cloud native applications, it isn’t necessarily the definition but for me, that’s the part that I was really focused on I think is quite interesting. [0:04:32.9] BL: Also, CI/CD cloud native apps are made to work well with our CI, our seamless integration and our continuous delivery/deployment systems as well, that’s like another very important aspect of cloud native applications. We should be able to deploy them to production without typing anything in. should be some kind of automated process. [0:04:56.4] NL: Yeah, that is for sure. Carlisia, you mentioned something that I think it’s good for us to talk about a little bit which is terminology. I keeping coming back to that. You mentioned monolithic apps, what are monoliths then? [0:05:09.0] CC: I am so hung up on what you just said, can we table that for a few minutes? You said cloud native applications for you is an application that can interact with the infrastructure and maybe for example, is the database. I wonder if you have an example or if you could expand on that, I want to – if everybody agrees with that, I’m not clear on what that even is. Because as a developer which is always my point of view is what I know. It’s a lot of responsibility for the application to have. And for example, when I would think cloud native and I’m thinking now, maybe I’m going off on a tangent here. But we have Kubernetes, isn’t that what Kubernetes is supposed to do to glue it all together? So, the application only needs to know what it needs to do. But spinning up an all tight system is not one of the things it would need to do? [0:05:57.3] BL: Sure, actually, I was going to use Kubernetes as my example for cloud native application. Because Kubernetes is what it is, an app, right? It can modify the cloud that it’s running. And so, if you have Kubernetes running in AWS, you can create ELB’s, elastic load balancers. It can create new nodes. It can create new databases if you need, as I mentioned. Kubernetes itself is my example like a cloud native application. I should say that that’s a good callout. My example of what a cloud native application isn’t necessarily like that’s a rule. All cloud native applications have to modify the cloud in which they exist in. It’s more that they can modify. That is a component of a cloud native application. Kubernetes is being an example there. I don’t know, I guess things like operators inside of Kubernetes like the rook operator will create storage for you when you spin up like root create a Ceph cluster, it will also spin up like the ELB’s behind it or at least I believe it does. Or that kind of functionality. [0:06:57.2] CC: I can see what you're saying because for example, if I choose to use the storage inside something like Kubernetes, then you will be required of my app to call an SDK and connect so that their storage somehow. So, in that sense I guess, you are using your app. Someone correct me if I’m wrong but that’s how the connection is created, right? You just request – but you’re not necessarily saying I want this thing specific, you just say I want this sort of thing like which has their storage and then you define that elsewhere. So, your applications don’t need to know details bit definitely needs to say, I need this. I’m talking about again, when your data storage is running on top of Kubernetes and not outside of it. [0:07:46.4] BL: Yeah. [0:07:47.3] NL: That brings up an interesting part of this whole term cloud native app. Because it’s like everything else in the space, our terms are not super concrete and an interesting piece about this is that an application – does an application half the map one to one with the running process? What is an application? [0:08:06.1] NL: That is interesting because it could say that a serverless app or a serverless rule, whatever serverless really is, I guess we can get into that in another episode. Are those cloud native applications? They’re not just running anywhere. [0:08:19.8] JR: I will punt on that because I know my boundaries are and that definitely not in my boundaries. But the reason I bring this up is because a little while ago, it’s probably year ago in a Kubernetes [inaudible 0:08:32] apps, we actually have a conversation about what an application was. And the consensus from the community and from the group members was that actually, an application could be made up of multiple processes. So, let’s say you were building this large SaaS service and because you’re selling dog food online, your application could be your dog food application. But you have inventory management. You have a front end, maybe you haven’t had service, you have a shipping manager and things like that. Sales tax calculator. Are those all applications? Or is it one application? The piece about cloud application are cloud native applications because what we found in Kubernetes is that the way we’re thinking about applications is, an application is multiple processes, that can be linked together and we can tell the whole story of how all those interact and working. Just something else, another way to think about this. [0:09:23.5] NL: Yeah, that is interesting, I never really considered that before but that makes a lot of sense. Particularly with the rise of things like GRPC and the ability to send dedicated messages to are like well codified messages too different processes. That gives rise to things like this multi-tenant process as an application. [0:09:41.8] BL: Right. But going back to your idea around cloud native applications being able to commandeer the resources that they’re needing. That’s something that we do see. We see it within Kubernetes right now. I’ll give you above and beyond the example that you gave is that whenever you create a staple set. And Kubernetes, the operator behind staple set that actually goes and provisions of PPC for you, you requested a resource and whatever you change the number of instances from one to like five, guess what? you get four more PPC’s. Just think about it, that is actually something that is happening, it’s a little transparent with people. but I can see to the point of we’re just requesting a new resource and if we are using cloud services to watch our other things, or our cloud native services to watch our applications, I could see us asking for this on demand or even a service like a database or some other type of queuing thing on demand. [0:10:39.2] CC: When I hear things like this, I think, “ Wow, it sounds very complicated. "But then I start to think about it and I think it’s really neat because it is complicated but the alternative would have been way more complicated. I mean, we can talk about, this is sort of how it’s done now. I mean, it’s really hard to go into details on a one-hour episode. We can’t cover the how it’s done or make conceptually, we are sort of throwing a lot of words out there sort of conceptualize it but we can also try to talk about it in a conceptual way how it is done in a non-cloud native world. [0:11:15.3] NL: Yeah, I kind of want to get back to the question I posed before, what is a monolithic app, what is a none cloud native app? And not all none cloud native apps are monoliths but this is actually something that I’ve heard a lot and I’ll be honest. I have an idea of what a monolithic app is but I think I don’t have a very good grasp of it. We kind of talked a bit about like what a cloud native app is, what is a none cloud native or what came before a cloud native applications. What is a monolith? [0:11:39.8] CC: I’m personally not a big fan of monoliths. Of course, I worked with them but once micro services started becoming common and started developing in that mode. I am much more of a fan of breaking things down for so many different reasons. It is a controversial topic for sure. But to go back to your question, the monolith is basically, you have an app, sort of goes to what Brian was saying, it’s like, what is an app? If you think of an app and like one thing, Amazon is an app, right? It’s an app that we use to buy things as consumers. And you know, the other part is the cloud. But let’s look at it like it’s an app that we use to buy things as consumers, we know it’s broken down to so many different services. There is the checkout service, there is the cart service. I mean, I’m imagining, these I can imagine thought, the small services that compose that one Amazon app. If it was a monolith, those services that you know – those things are different systems that are talking together. The whole thing would be on one code base. It would reside in same code base or it will be deployed together. It will be shipped together. If you make a change in one place and you needed to deploy that, you have to deploy the whole thing together. You might have teams that are working on separate aspects but they’re working against the same code base. And maybe because of that, that will lend itself to teams not really specializing on separate aspects because everything is together so you might make one change of the impacts another place and then you have to know that part as well. So, there is a lot less specialization and separation of teams as well. [0:13:32.3] BL: Maybe to give an example of my experience and I think it aligns with a lot of the details Carlisia just went over. Even taking five years back, my experience at least was, we’d write up a ticket and we’d ask somebody to make a server space for us, maybe run [inaudible 0:13:44] on it, right? We’d write all this Java code and we’d package it into these things that run on a JDL somewhere, right? We would deploy this whole big application you know?Let’s call it that dog food app, right? It would have maybe even like a state layer and have the web server layer, maybe have all these different pieces all running together, this big code base as Carlisia put it. And we’d deploy it, you know, that process took a lot of time and was very consuming especially when we needed to change stuff, we didn’t have all these modern API’s and this kind of decoupled applications, right? But then, over time, you know, we started learning more and more about the notion of isolating each of these pieces or layers. So that we could have the web server, isolated in its how, put some site container or a unit and then the state layer and the other layers even isolated, you know, the micro service approach more or less. And then we were able to scale independently and that was really awesome. so we saw a lot of the gains in that respect. We basically moved our complexity to other areas, we took our complexity that you need to all happen in the same memory space and we moved a lot of it into the network with this new protocols of that different services talk to one another. It’s been an interesting thing kind of seeing the monolith approach and the micro service approach and how a lot of these micro service apps are in my opinion a lot more like cloud native aligned, if that makes sense? Just seeing how the complexity shows around in that regard. [0:15:05.8] CC: Let me just say one more thing because it’s actually the biggest aspect of micro services that I like the most in comparison, you know, the aspect of monolith that I hate the most and that I don’t hate it, I appreciate the least, let’s put it that way. Is that, when you have a monolith, it is so easy because design is hard so it’s so easy to couple different parts of your app with other parts of your app and have couples cold and coupled functionality. When you break this into micro services, that is impossible. Because it was working with separate code bases. If you force to think what is your interface, you’re always thinking about the interface and what people need to consume from you, your interface is the only way into your app, into your system. I really like the aspect that it forces you to think about your API. And people will argue, “Well you can’t put the same amount of effort into that if you have a monolith.” Absolutely, but in reality, I don’t see it. And like Josh was saying, it is not a walk on the park, but I’d much rather deal with those issues, those complexities that Microsoft has create then the challenges of running a big – I’m talking about big monoliths, right? Not something trivial. [0:16:29.8] JR: I will come to distil this about how I look at monoliths and how it fits into this conversation. A monolith is simply an application that is or a single process in this case that is running both the UI, the front-end code and the code that fetches the state from a data store, whether that be disk or database. That is what a monolith is. The reasons people use monoliths are many but I can actually think of some very good reasons. If you have code reuse and let’s say you have a website and you were trying to – you have farms and you want to be able to use those form libraries or you have data access and you want to be able to reuse that data access code, a monolith is great. The problem with monoliths is as functionality becomes larger, complexity becomes larger and not at the same rate. I’m not going to say that it is not linear but it’s not quite exponential. Maybe it logs into or something like that. But the problem is that at a certain point, you’re going to have so much functionality, you’re not going to be able to put it inside of one process, see Rails. Rails is actually a great example of this where we run into the issues where we put so much application into a rail source directory and we try to run it and we basically run up with these huge processes. And we split them up. But what we found is that we could actually split out the front-end code to one process. We could spit out the middle ware, see multiple process in the middle, the data access layer to another process and we could use those, we could actually take advantage of multiple CPU cores or multiple computers. The problem with this is that with splitting this out, it’s complexity. So, what if you have a [inaudible 0:18:15] is, what I’m trying to say here in a very long way is that monoliths have their places. As a matter of fact, the encourage, at least I still encourage people to start with the monolith. Put everything in one place. Whenever it gets too big, you spit it out. But in a cloud native world, because we’re trying to take advantage of containers, we’re trying to take advantage of cords on CPUs, we’re trying to take advantage of multiple computers to do that in the most efficient way, you want to split your application up into smaller pieces so that your front end versus your middle layer, versus your data access layer versus your data layer itself can run on as many computers and as many cores as possible. Therefore, spreading thee risk and spreading the usage because everything should be faster. [0:19:00.1] NL: Awesome. That is some great insight into monolithic apps and also the benefit and pros and cons of them. Like something I didn’t have before. Because I’ve only ever heard of a praise monolithic apps and then it’s like said in hushed tones or what the swear word directly after it. And so, it’s interesting to hear the concept of it being that each way you deploy your application is complex but there are different tradeoffs, right? It’s the idea that I was like, “Why don’t you want to turn your monolithic into micro services? Well, there’s so much more overhead, so much more yak shaving you have to do to get there to take advantage of micro services. That was awesome, thank you so much for that insight. [0:19:39.2] CC: I wanted to reiterate a couple aspects of what Brian said and Josh said in regards to that. One huge advantage, I mean, your application needs to be substantial enough that you feel like you need to do that, you’re going to get some advantage from it. when you had that point, and you do that, you’re clearing to services like Josh was saying and Brian was saying, you have the ability to increase your capabilities, your process capabilities based on one aspect of the system that needs it. So, you have something that requires very low processing, you run that service with certain level of capabilities. And something that like your orders process or your orders micro service. You increase the processing power for that much more than some other part. When it comes to running this in the cloud native world, I think this is more an infrastructure aspect. But my understanding is that you can automate all of that, you can determine, “Okay, I have analyzed my requirements based on history and what I need is stacks. So, I’m going to say tell the cloud native infrastructure, this is what I need in the automation will take care of bringing the system up to that if anything happens.” We are always going to be healing your system in an automated way and this is something that I don’t think gets talked about enough like we say, we talk about, “Oh things split up this way and they’re run this way but in an automated mode that these makes all of the difference. [0:21:15.4] NL: Yeah that makes a lot of sense actually. So, basically analytic apps don’t give us the benefit of automation or automated deployment versus like micro services kind of give us and cloud native applications give us the rise. [0:21:28.2] BL: Yes, and think about this, whenever you have five micro services delivering your applications functionality and you need to upgrade the front-end code for the HTML, whatever generates the HTML. You can actually replace that piece or replace that piece and that not bring your whole application down. And even better yet, you can replace that piece one at a time or two at a time, still have the majority of your applications still running and maybe your users won’t even know at all. So, let’s say you have a monolith and you are running multiple versions of this monoliths. When you take that whole application down, you literally take the whole application down not only do you lose front-end capacity, you also lose back-end capacity as well. So, separating your app is actually smarter than the long run because what it gives you is the flexibility to mix and match and you could actually scale the front end at a different level than you did at the backend. And that is actually super important in [inaudible 0:22:22] land and actually Python land and .NET land if you’re writing monoliths. You have to scale at the level of your monolith and if you can scale that then you are having wasted resources. So smaller micro services, smaller cloud native apps makes the run of containers, actually will use less resources. [0:22:41.4] JR: I have an interesting question for us all. So obviously a lot of cloud native applications usually maybe look like these micro services we’re describing, can a monolith be a cloud native application as well? [0:22:54.4] BL: Yes, it can. [0:22:55.1] JR: Cool. [0:22:55.6] NL: Yeah, I think so. As long as the – basically monolith can be deployed in the mechanism that we described like CSAD or can take advantage of the cloud. I believe the monolith can be a cloud native application, sure. [0:23:08.8] CC: There are monolith – because I am glad you brought that up because I was going to bring that up because I hear Brian using the micro services in cloud native apps interchangeably and it makes it really hard for me to follow, “Okay, so what is not cloud native application or what is not a cloud native service and what is not a cloud native monolith?” So, to start this thread with the question that Josh just asked, which also became my question: if I have a monolith app running a cloud provider is that a cloud native app? If it is not what piece of puzzle needs to exists for that to be considered a cloud native app? And then the follow up question I am going to throw it out there already is why do we care? What is the big deal if it is or if it isn’t? [0:23:55.1] BL: Wow, okay. Well let’s see. Let’s unpack this. I have been using micro service and cloud native interchangeably probably not to the best effect. But let me clear up something here about cloud native versus micro services. Cloud native is a big term and it goes further than an application itself. It is not only the application. It is also the environment of the application can run in. It is the process that we use to get the application to production. So, monoliths can be cloud native apps. We can run them through CI/CD. They can run in containers. They can take advantage of their environment. We can scale them independently. but we use micro services instead this becomes easier because our surface area is smaller. So, what I want to do is not use that term like this. Cloud native applications is an umbrella term but I will never actually say cloud native application. I always say a micro service and the reason why I will say the micro service is because it is much more accurate description of that process that is running. Cloud native applications is more of the umbrella. [0:25:02.0] JR: It is really interesting because a lot of the times that we are working with customers when they go out and introduce them to Kubernetes, we are often times asked, “How do I make my application cloud native?” To what you are talking about Brian and to your question Carlisia, I feel like a lot of times people are a little bit confused about it because sometimes they are actually asking us, “How do I break this legacy app into smaller micro services,” right? But sometimes they are actually asking like, “How do I make it more cloud native?” And usually our guidance or the things that we are working with them on is exactly that, right? It is like getting that application container so we can get it portable whether it is a monolith or a micro service, right? We are containerizing it. We are making it more portable. We are maybe helping them out with health checks that the infrastructure environment that they are running in can tap into it and know the health of that application whether it’s to restart it with Kubernetes as an example. We are going through and helping them understand those principles that I think fall more into the umbrella of cloud native like you are saying Brian if I am following you correctly and helping them kind of enhance their application. But it doesn’t necessarily mean splitting it apart, right? It doesn’t mean running it in smaller services. It just means following these more cloud native principles. It is hard talk up so that was continuing to say cloud native right? [0:26:10.5] BL: So that is actually a good way of putting it. A cloud native application isn’t a thing. It is a set of principles that you can use to guide yourself to running apps in cloud environments. And it is interesting. When I say cloud environments I am not even really particularly talking about Kubernetes or any type of scheduler. I am just talking about we are running apps on other people’s computers in the cloud this is what we should think about and it goes through those principles. Where we use CI/CD, storage maybe most likely will be ephemeral. Actually, you know what? That whole process, that whole virtual machine that we are running on that is ephemeral too, everything will go away. So, cloud native applications is basically a theory that allows us to be strategic about running applications with other people’s computers and storage and networking and compute may go away. So, we do this at this way, this is how to get our 5-9’s or 4-9’s above time because we can actually do this. [0:27:07.0] NL: That is actually a great point. The cloud native application is one that can confidently run on somebody else’s computer. That is a good stake in the ground. [0:27:15.9] BL: I stand behind that and I like the way that you put it. I am going to steal that and say I made it up. [0:27:20.2] NL: Yeah, go ahead. We have been talking about monoliths and cloud native applications. I am curious, since you all are developers, what is your experience writing cloud native applications? [0:27:31.2] JR: I guess for green field projects where we are starting from scratch and we are kind of building this thing, it is a really pleasant experience because a lot of things are sort of done for us. We just need to know how to interact with the API or the contract to get the things we need. So that is kind of my blanket statement. I am not trying to say it is easy, I am just saying like it has become quite convenient in a lot of respects when adopting these cloud native principles. Like the idea that I have a docker file and I build this container and now I am running this app that I am writing code for all over the place, it’s become such a more pleasant experience and at least in my experience years and years ago with like dropping things into the tomcat instances running all over the place, right? But I guess what’s also been interesting is it’s been a bit hard to convert older applications into the cloud native space, right? Because I think the point Carlisia had started with around the idea of all the code being in one place, you know it is a massive undertaking to understand how some of these older applications work. Again, not saying that all older applications are only monoliths. But my experience has been that they generally are. Their bigger code base is hard to understand how they work and where to modify things without breaking other things, right? When you go and you say, “All right, let’s adopt some cloud native principles on this app that has been running on the mainframe for decades” right? That is a pretty hard thing to do but again, green field projects I found it to be pretty convenient. [0:28:51.6] CC: It is actually easy, Josh. You just rewrite it. [0:28:54.0] JR: Totally yes. That is always a piece of cake. ,[0:28:56.9] BL: You usually write it in Go and then it is cloud native. That is actually the secret to cloud native apps. You write it in Go, you install it, you deploy in Kubernetes, mission accomplish, cloud native to you. [0:29:07.8] CC: Anything written in Go is cloud native. We are declaring that here, you heard that here first. [0:29:13.4] JR: That is a great question, it’s like how do we get there? That is a hard question and not one that I would basically just wave a magic set of words over and say that we are there. But what I would say is that as we start thinking of moving applications to cloud native first, we need to identify applications that cannot be called updated and I could actually give you some. Your Windows 2003 applications and yes, I do know some of you are running 2003 still. Those are not cloud native and they never will be and the problem is that you won’t be able to run them in a containerized environment. Microsoft says stop using 2003, you should stop using it. Other applications that won’t be cloud native are applications that require a certain level of machine or server access. We have been able to attract GPU’s. But if you’re working on the IO level like you are actually looking at IO or if you are looking at hardware interrupts. Or you are looking at anything like that, that application will never be cloud native. Because there is no way that we can in a shared environment, which most likely your application will be running in, in the cloud. There is no way that first of all that the hypervisor that is actually running your virtual machine wants to give you that process or give you that access or that is not being shared from one to 200 other processes on that server. So, applications that want low level access or have real time, you don’t want to run those in the cloud. They cannot be cloud native. That still means a lot of applications can be. [0:30:44.7] CC: So, I keep thinking of if I own a tech stack and I every once in a while stop and evaluate, if I am squeezing as most tech as I can out of my system? Meaning am I using the best technology out there to the extent that fits my needs? If I am that kind of person and I don’t know – it’s like when I say I am a decision maker and even if I was a tech person like I am also a tech person, I still would not have – unless I am one of the architects. And sometimes even the architects don’t have an entire vision. I mean they have to talk to other architects who have a greater vision of the whole system because systems that can be so big. But at any rate, if I am an architect or I own the tech stack one way or another, my question is, is my system a cloud native system? Is my app a cloud native app? I am not even sure that we clarified enough for people to answer that. I mean it is so complicated, maybe we did hopefully we helped a little bit. So basically, this will be my question, how do I know if I am there or not? Because my next step would be well if I am not there then what am I missing? Let me look into it and see if the cost benefit is worth it. But if I don’t know what is missing, what do I look at? How do I evaluate? How do I evaluate if I am there and if I am not, what do I need to do? So, we talked about this a little bit on episode one, which we talked about cloud native like what is cloud native in general and now we are talking about apps. And so, you know, there should be a checklist of things that cloud native should at least have these sets of things. Like the 12-factor app, what do you need to have to be considered 12 factor app. We should have a checklist, 12 factor app I think having that checklist is being part of micro-service in the cloud native app. But I think there needs to be more. I just wish we would have that not that we need to come up with that list now but something to think about. Someone should do it, you know? [0:32:57.5] JR: Yeah, it would be cool. [0:32:58.0] CC: Is it reasonable or now to want to have that checklist? [0:33:00.6] BL: So, there is, there is that checklist that exist I know that Red Hat has one. I know that IBM has one. I would guess VMware has one on one of our web pages. Now the problem is they’re all different. What I do and this is me trying to be fair here. The New Stack basically they talk about things that are happening in the cloud and tech. If you search for The New Stack in cloud native application, there is a 10-bullet list. That is what I send to people now. The reason I send that one rather than any vendor is because a vendor is trying to sell you something. They are trying to sell you their vision of cloud native where they excel and they will give you products that help you with that part like CI/CD, “oh we have a product for that.” I like The New Stack list and actually, I Googled it while you were talking Carlisia because I wanted it to bring it up. So, I will just go through the titles of this list and we’ll make sure that we make this link available. So, there is 10 Key Attributes of Cloud-Native Applications. Package as light weight to containers. Developed with best-of-breed languages and frameworks, you know that doesn’t mean much but that is how nebulous this is. Designed as loosely coupled microservices. Centered around API’s for interaction and collaboration. Architected with clean separation of stateless and stateful services. Isolated from server and operating system dependencies. Deployed on self-service elastic cloud infrastructure. Managed through agile DevOps processes. Automated capabilities. And the last one, Defined policy-driven resource allocation. And as you see, those are all very much up for interpretation or implementations. So, a cloud native app from my point of view tries to target most of these items and has an opinion on most of these items. So, a cloud native app isn’t just one thing. It is a mindset that I am running. Like I said before, I am running my software on other people’s computers, how can I best do this.? [0:34:58.1] CC: I added the link to our shownotes. When I look at this list, I don’t see observability. That word is not there. Does it fall under one of those points because observability is another new-ish term that seems to be in parcel of cloud native? Correct me here, people. [0:35:19.1] JR: I am. Actually, the eighth item, ‘Manage through agile DevOps processes,’ is actually – they don’t talk about monitoring observability. But for an application for a person who is not developing application, so whether you have a dev ops team or you have an SRE practice, you are going to have to be able to communicate the status and the application whether it be through metrics logs or metrics logs or whatever the other one is. I am thinking – traces. So that is actually I think is baked in it is just not called out. So, to get the proper DevOps you would need some observability that is how you get that status when you have a problem. [0:35:57.9] CC: So, this is how obscure these things can be. I just want to point this out. It is so frustrating, so literally we have item eight, which Brian has been, as the main developer so he is super knowledgeable. He can look at that and know what it means. But I look at that and the words log metrics, observability none of these words are there and yet Brian knew that that is what it means that that is what he meant. And I don’t disagree with him. I can see it now but why does it have to be so obscure? [0:36:29.7] JR: I think a big thing to consider too is like it very much lands on spectrum, right? Like something you would ask Carlisia is how do I qualify if my app is cloud native or what do I need to do? And you know a lot of people in my experience are just adopting parts of this list and that’s totally fine. You know worrying about whether you fully qualify as a cloud native app since we have talked about it as more of a set of principles is something – I don’t know if there is too too much value in worrying about whether you can block that label onto your app as much as it is, “Oh I can see our organization our applications having these problems.” Like lacking portability when we move them across providers or going back to observability, not being able to know what is going on inside of the application and where the network packets are headed and they switched to being asked we’re late to see these happening. And as those problems come on, really looking at and adopting these principles where it is appropriate. Sometimes it might not be with the engineering efforts without them, one of the more cloud native principles. You know you just have to pick and choose what is most valuable to you. [0:37:26.7] BL: Yes, and actually this is what we should be doing as experts, as thought-leaders, as industry movers and shakers. Our job is to make this easier for people coming behind us. At one time, it was hard to even start an application or start your operating system. Remember when we had to load AN1, you know? Remember we had to do that back in the day on our basic, on our Comado64’s or Apple or Apple2. Now you turn your computer on and it comes with instantly. We click on application and it works. We need to actually bring this whole cloud movement to that point where things like if you include these libraries and you code with these API’s you get automatic observability. And I am saying that with air quotes but you get the ability to have this thing to monitor it in some fashion. If you use this practice and you have this stack, CI/CD should be super simple for you and we are just not quite there yet. And that is why the industry is definitely rotating around this and that is why there has been a lot of buzz around cloud native and Kubernetes is because people are looking at this to actually solve a lot of these problems that we’ve had. Because they just haven’t been solvable because everybody stacks are too different. But this one though, the reason Linux is I think ultimately successful is because it allowed us to do things and all of these Linux things we liked and it worked on all sorts of computers. And it got that mindset behind it behind companies. Kubernetes could also do this. It allows us to think about our data centers as potentially one big computer or fewer computers that allows us to make sure things are running. And once we have this, now we can develop new tools that will help us with our observability, with our getting software into production and upgraded and where we need it. [0:39:17.1] NL: Awesome. So, on that, we are going to have to wrap up for this week. Let’s go ahead and do a round of closing thoughts. [0:39:22.7] JR: I don’t know if I have any closing thoughts. But it was a pleasure talking about cloud native applications with you all. Thanks. [0:39:28.1] BL: Yeah, I have one thought is that all of these things that we are talking about it sounds kind of daunting. But it is better that we can have these conversations and talk about things that don’t work rather than not knowing what to talk about in general. So this is a journey for us and I hope you come for more of our journey. [0:39:46.3] CC: First I was going to follow up on Josh and say I am thoughtless. But now I want to fill up on Brian’s and say, no I have no opinions. It is very much what Brian said for me, the bridging of what we can do using cloud native infrastructure in what we read about it and what we hear about it like for people who are not actually doing it is so hard to connect one with the other. I hope by being here and asking questions and answering questions and hopefully people will also be very interactive with us. And ask us to talk about things they want to know that we all try to connect it too little by little. I am not saying it is rocket science and nobody can understand it. I am just saying for some people who don’t have multi background experience, they might have big gaps. [0:40:38.7] NL: And that is for sure. This was a very useful episode for me. I am glad to know that everybody else is just as confused at what cloud native applications actually mean. So that was awesome. It was a very informative episode for me and I had a lot of fun doing it. So, thank you all for having me. Thank you for joining us on this week of the Kublets Podcast. And I just want to wish our friend Brian a very happy birthday. Bye you all. [0:41:03.2] CC: Happy birthday Brian. [0:41:04.7] BL: Ahhhh. [0:41:05.9] NL: All right, bye everyone. [END OF EPISODE] [0:41:07.5] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

There are two words that get the blame more often than not when a problem cannot be rooted: the network! Today, along with special guest, Scott Lowe, we try to dig into what the network actually means. We discover, through our discussion that the network is, in fact, a distributed system. This means that each component of the network has a degree of independence and the complexity of them makes it difficult to understand the true state of the network. We also look at some of the fascinating parallels between networks and other systems, such as the configuration patterns for distributed systems. A large portion of the show deals with infrastructure and networks, but we also look at how developers understand networks. In a changing space, despite self-service becoming more common, there is still generally a poor understanding of networks from the developers’ vantage point. We also cover other network-related topics, such as the future of the network engineer’s role, transferability of their skills and other similarities between network problem-solving and development problem-solving. Tune in today! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Duffie Cooley Nicholas Lane Josh Rosso Key Points From This Episode: • The network is often confused with the server or other elements when there is a problem.• People forget that the network is a distributed system, which has independent routers.• The distributed pieces that make up a network could be standalone computers.• The parallels between routing protocols and configuration patterns for distributed systems.• There is not a model for eventually achieving consistent networks, particularly if they are old.• Most routing patterns have a time-sensitive mechanism where traffic can be re-dispersed.• Understanding a network is a distributed system gives insights into other ones, like Kubernetes.• Even from a developers’ perspective, there is a limited understanding of the network.• There are many overlaps between developers and infrastructural thinking about systems.• How can network engineers apply their skills across different systems?• As the future changes, understanding the systems and theories is crucial for network engineers.• There is a chasm between networking and development.• The same ‘primitive’ tools are still being used for software application layers.• An explanation of CSMACD, collisions and their applicability. • Examples of cloud native applications where the network does not work at all.• How Spanning Tree works and the problems that it solves.• The relationship between software-defined networking and the adoption of cloud native technologies.• Software-defined networking increases the ability to self-service.• With self-service on-prem solutions, there is still not a great deal of self-service. Quotes: “In reality, what we have are 10 or hundreds of devices with the state of the network as a system, distributed in little bitty pieces across all of these devices.” — @scott_lowe [0:03:11] “If you understand how a network is a distributed system and how these theories apply to a network, then you can extrapolate those concepts and apply them to something like Kubernetes or other distributed systems.” — @scott_lowe [0:14:05] “A lot of these software defined networking concepts are still seeing use in the modern clouds these days” — @scott_lowe [0:44:38] “The problems that we are trying to solve in networking are not different than the problems that you are trying to solve in applications.” — @mauilion [0:51:55] Links Mentioned in Today’s Episode: Scott Lowe on LinkedIn — https://www.linkedin.com/in/scottslowe/ Scott Lowe’s blog — https://blog.scottlowe.org/ Kafka — https://kafka.apache.org/ Redis — https://redis.io/ Raft — https://raft.github.io/ Packet Pushers — https://packetpushers.net/ AWS — https://aws.amazon.com/ Azure — https://azure.microsoft.com/en-us/ Martin Casado — http://yuba.stanford.edu/~casado/ Transcript: EPISODE 15 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.4] DC: Good afternoon everybody. In this episode, we’re going to talk about the network. My name is Duffie Cooley and I’ll be the lead of this episode and with me, I have Nick. [0:00:49.0] NL: Hey, what’s up everyone. [0:00:51.5] DC: And Josh. [0:00:52.5] JS: Hi. [0:00:53.6] DC: And Mr. Scott Lowe joining us as a guest speaker. [0:00:56.2] SL: Hey everyone. [0:00:57.6] DC: Welcome, Scott. [0:00:58.6] SL: Thank you. [0:01:00.5] DC: In this discussion, we’re going to try and stay away, like we do always, we’re going to try and stay away from particular products or solutions that are related to the problem. The goal of it is to really kind of dig in to like what the network means when we refer to it as it relates to like cloud native applications or just application design in general. One of the things that I’ve noticed over time and I’m curious, what you all think but like, one of the things I’ve done over time is that people are kind of the mind that if it can’t root cause a particular issue that they run into, they’re like, “That was the network.” Have you all seen that kind of stuff out there? [0:01:31.4] NL: Yes, absolutely. In my previous life, before being a Kubernetes architect, I actually used my networking and engineering degree to be a network administrator for the Boeing Company, under the Boeing Corporation. Time and time again, someone would come to me and say, “This isn’t working. The network is down.” And I’m like, “Is the network down or is the server down?” Because those are different things. Turns out it was usually the server. [0:01:58.5] SL: I used to tell my kids that they would come to me and they would say, the Internet is down and I would say, “Well, you know. I don’t think the entire Internet is down, I think it’s just our connection to the Internet.” [0:02:10.1] DC: Exactly. [0:02:11.7] JS: Dad, the entire global economy is just taking a total hit. [0:02:15.8] SL: Exactly, right. [0:02:17.2] DC: I frequently tell people that my first distributed system that I ever had a real understanding of was the network, you know? It’s interesting because it kind of like, relies on the premises that I think a good distributed system should in that there is some autonomy to each of the systems, right? They are dependent on each other or even are inter communicate with each other but fundamentally, like when you look at routers and things like that, they are autonomous in their own way. There’s work that they do exclusive to the work that others do and exclusive to their dependencies which I think is very interesting. [0:02:50.6] SL: I think the fact that the network is a distributed system and I’m glad you said that Duffie, I think the fact the network is a distributed system is what most people overlook when they start sort of blaming the network, right? Let’s face it, in the diagrams, right, the network’s always just this blob, right? Here’s the network, right? It’s this thing, this one singular thing. When in reality, what we have are like 10 or hundreds of devices with the state of the network as a system, distributed in little bitty pieces across all of these devices. And no way, aside from logging in to each one of these devices are we able to assemble what the overall state is, right? Even routing protocols mean, their entire purpose is to assemble some sort of common understanding of what the state of the network is. Melding together, not just IP addresses which are these abstract concept but physical addresses and physical connections. And trying to reason to make decisions about them, how we center across and it’s far more complex and a lot of people understand, I think that’s why it’s just like the network is down, right? When reality, it’s probably something else entirely. [0:03:58.1] DC: Yeah, absolutely. Another good point to bring up is that each of these distributed pieces of this distributed system are in themselves like basically like just a computer. A lot of times, I’ve talked to people and they were like, “Well, the router is something special.” And I’m like, “Not really. Technically, a Linux box could just be a router if you have enough ports that you plug into it. Or it could be a switch if you needed to, just plug in ports.” [0:04:24.4] NL: Another good interesting parallel there is like when we talk about like routing protocols which are a way of – a way that allow configuration changes to particular components within that distributed system to be known about by other components within that distributed system. I think there’s an interesting parallel here between the way that works and the way that configuration patterns that we have for distributed systems work, right? If you wanted to make a configuration only change to a set of applications that make up some distributed system, you might go about like leveraging Ansible or one of the many other configuration models for this. I think it’s interesting because it represents sort of an evolution of that same idea in that you’re making it so that each of the components is responsible for informing the other components of the change, rather than taking the outside approach of my job is to actually push a change that should be known about by all of these concepts, down to them. Really, it’s an interesting parallel. What do you all think of that? [0:05:22.2] SL: I don’t know, I’m not sure. I’d have to process that for a bit. But I mean, are you saying like the interesting thought here is that in contrast to typical systems management where we push configuration out to something, using a tool like an Ansible, whatever, these things are talking amongst themselves to determine state? [0:05:41.4] DC: Yeah, it’s like, there are patterns for this like inside of distributed systems today, things like Kafka and you know, Kafka and Gossip protocol, stuff like this actually allows all of the components of a particular distributed system to understand the common state or things that would be shared across them and if you think about them, they’re not all that different from a routing protocol, right? Like the goal being that you give the systems the ability to inform the other systems in some distributed system of the changes that they may have to react to. Another good example of this one, which I think is interesting is like, what they call – when you have a feature behind a flag, right? You might have some distributed configuration model, like a Redis cache or database somewhere that you’ve actually – that you’ve held the running configuration of this distributed system. And when you want to turn on this particular feature flag, you want all of the components that are associated with that feature flag to enable that new capability. Some of the patterns for that are pretty darn close to the way that routing protocol models work. [0:06:44.6] SL: Yeah, I see what you're saying. Actually, that’ makes a lot of sense. I mean, if we think about things like Gossip protocols or even consensus protocols like Raft, right? They are similar to routing protocols in that they are responsible for distributing state and then coming to an agreement on what that state is across the entire system. And we even apply terms like convergence to both environments like we talk about how long it takes routing protocol to converge. And we might also talk about how long it takes for and ETCD cluster to converge after changing the number of members in the cluster of that nature. The point at which everybody in that distributed system, whether it be the network ETCD or some other system comes to the same understanding of what that shared state is. [0:07:33.1] DC: Yeah, I think that’s a perfect breakdown, honestly. Pretty much every routing technology that’s out there. You know, if you’re taking that – the computer of the network, you know, it takes a while but eventually, everyone will reconcile the fact that, “Yeah, that node is gone now.” [0:07:47.5] NL: I think one thing that’s interesting and I don’t know how much of a parallel there is in this one but like as we consider these systems like with modern systems that we’re building at scale, frequently we can make use of things like eventual consistency in which it’s not required per se for a transaction to be persisted across all of the components that it would affect immediately. Just that they eventually converge, right? Whereas with the network, not so much, right? The network needs to be right now and every time and there’s not really a model for eventually consistent networks, right? [0:08:19.9] SL: I don’t know. I would contend that there is a model for eventually consistent networks, right? Certainly not on you know, most organizations, relatively simple, local area networks, right? But even if we were to take it and look at something like a Clos fabric, right, where we have top of rack switches and this is getting too deep for none networking blokes that we know, right? Where you take top of rack switches that are talking layer to the servers below them or the end point below them. And they’re talking layer three across a multi-link piece up to the top, right? To the spine switches, so you have leaf switches, talking up spine switches, they’re going to have multiple uplinks. If one of those uplinks goes down, it doesn’t really matter if the rest off that fabric knows that that link is down because we have the SQL cost multi pathing going across that one, right? In a situation like that, that fabric is eventually consistent in that it’s okay if you know, knee dropping link number one of leaf A up to spine A is down and the rest of the system doesn’t know about that yet. But, on the other hand, if you are looking at network designs where convergence is being handled on active standby links or something of that nature or there aren’t enough paths to get from point A to point B until convergence happens then yes, you’re right. I think it kind of comes down to network design and the underlying architecture and there are so many factors that affect that and so many designs over the years that it’s hard to – I would agree and from the perspective of like if you have an older network and it’s been around for some period of time, right? You probably have one that is not going to be tolerant, a link being down like it will cause problems. [0:09:58.4] NL: Adds another really great parallel in software development, I think. Another great example of that, right? If we consider for a minute like the circuit breaking pattern or even like you know, most load balancer patterns, right? In which you have some way of understanding a list of healthy end points behind the load balancer and were able to react when certain end points are no longer available. I don’t consider that a pattern that I would relate to specifically if they consent to eventual consistency. I feel like that still has to be immediate, right? We have to be able to not send the new transaction to the dead thing. That has to stop immediately, right? It does in most routing patterns that are described by multi path, there is a very time sensitive mechanism that allows for the re-dispersal of that traffic across known paths that are still good. And the work, the amazing amount of work that protocol architects and network engineers go through to understand just exactly how the behavior of those systems will work. Such that we don’t see traffic. Black hole in the network for a period of time, right? If we don’t send traffic to the trash when we know or we have for a period of time, while things converge is really has a lot going for it. [0:11:07.0] SL: Yeah, I would agree. I think the interesting thing about discussing eventual consistency with regards to the networking is that even if we take a relatively simple model like the DOD model where we only have four layers to contend with, right? We don’t have to go all the way to this seven-layer OSI model. But even if we take a simple layer like the DOD four-layer model, we could be talking about the rapid response of a device connected at layer two but the less than rapid response of something operating at layer three or layer four, right? In the case of a network where we have these discreet layers that are intentionally loosely coupled which is another topic, we could talk about from a distribution perspective, right? We have these layers that are intentionally loosely coupled, we might even see consistency and the application of the cap theorem, behave differently at different layers of their model. [0:12:04.4] DC: That’s right. I think it’s fascinating like how much parallel there is here. As you get into like you know, deep architectures around software, you’re thinking of these things as it relates to like these distributed systems, especially as you’re moving toward more cloud native systems in which you start employing things like control theory and thinking about the behaviours of those systems both in aggregate like you know, some component of my application, can I scale this particular component horizontally or can I not, how am I handling state. So many of those things have parallels to the network that I feel like it kind of highlights I’m sure what everybody has heard a million times, you know, that there’s nothing new under the sun. There’s million things that we could learn from things that we’ve done in the past. [0:12:47.0] NL: Yeah, totally agree. I recently have been getting more and more development practice and something that I do sometimes is like draw out like how all of my functions and my methods, and take that in rack with each other across a consisting code base and lo and behold when I draw everything out, it sure does look a lot like a network diagram. All these things have to flow together in a very specific way and you expect the kind of returns that you’re looking for. It looks exactly the same, it’s kind of the – you know, how an atom kind of looks like a galaxy from our diagram? All these things are extrapolated across like – [0:13:23.4] SL: Yeah, totally. [0:13:24.3] NL: Different models. Or an atom looks like a solar system which looks like a galaxy. [0:13:28.8] SL: Nicholas, you said your network administrator at Boeing? [0:13:30.9] NL: I was, I was a network engineer at Boeing. [0:13:34.0] SL: You know, as you were sitting there talking, Duffie, so, I thought back to you Nick, I think all the times, I have a personal passion for helping people continue to grow and evolve in their career and not being stuck. I talk to a lot of networking folks, probably dating because of my involvement, back in the NSX team, right? But folks being like, “I’m just a network engineer, there’s so much for me to learn if I have to go learn Kubernetes, I wouldn’t even know where to start.” This discussion to me underscores the fact that if you understand how a network is a distributed system and how these theories apply to a network, then you can extrapolate those concepts and apply them to something like Kubernetes or other distributed systems, right? Immediately begin to understand, okay. Well, you know, this is how these pieces talk to each other, this is how they come, the consensus, this is where the state is stored, this is how they understand and exchange date, I got this. [0:14:33.9] NL: if you want to go down that that path, the controlled plane of your cluster is just like your central routing back bone and then the kublets themselves are just your edge switches going to each of your individual smaller network and then the pods themselves have been nodes inside of the network, right? You can easily – look at that, holy crap, it looks exactly the same. [0:14:54.5] SL: Yeah, that’s a good point. [0:14:55.1] DC: I mean, another interesting part, when you think about how we characterize systems, like where we learn that, where that skillset comes from. You raise a very good point. I think it’s an easier – maybe slightly easier thing to learn inside of networking, how to characterize that particular distributed system because of the way the components themselves are laid out and in such a common way. Where when we start looking at different applications, we find a myriad of different patterns with particular components that may behave slightly differently depending, right? Like there are different patterns within software like almost on per application bases whereas like with networks, they’re pretty consistently applied, right? Every once in a while, they’ll be kind of like a new pattern that emerges, that it just changes the behavior a little bit, right? Or changes the behavior like a lot but at the same time, consistently across all of those things that we call data center networks or what have you. To learn to troubleshoot though, I think the key part of this is to be able to spend the time and the effort to actually understand that system and you know, whether you light that fire with networking or whether you light that fire with like just understanding how to operationalize applications or even just developing and architecting them, all of those things come into play I think. [0:16:08.2] NL: I agree. I’m actually kind of curious, the three of us have been talking quite a bit about networking from the perspective that we have which is more infrastructure focused. But Josh, you have more of a developer focused background, what’s your interaction and understanding of the network and how it plays? [0:16:24.1] JS: Yeah, I’ve always been a consumer of the network. It’s something that is sat behind an API and some library, right? I call out to something that makes a TCP connection or an http interaction and then things just happen. I think what’s really interesting hearing talk and especially the point about network engineers getting into thee distributed system space is that I really think that as we started to put infrastructure behind API’s and made it more and more accessible to people like myself, app developers and programmers, we started – by we, you know, I’m obviously generalizing here. But we started owning more and more of the infrastructure. When I go into teams that are doing big Kubernetes deployments, it’s pretty rare, that’s the conventional infrastructure and networking teams that are standing up distributed systems, Kubernetes or not, right? It's a lot of times, a bunch of app developers who have maybe what we call dev-ops, whatever that means but they have an application development background, they understand how they interact with API’s, how to write code that respects or interacts with their infrastructure and they’re standing up these systems and I think one of the gaps of that really creates is a lot of people including myself just hearing you all talk, we don’t understand networking at that level. When stuff falls over and it’s either truly the network or it’s getting blamed on the network, it’s often times, just because we truly don’t understand a lot of these things, right? Encapsulation, meshes, whatever it might be, we just don’t understand these concepts at a deep level and I think if we had a lot more people with network engineering backgrounds, shifting into the distributed system space. It would alleviate a bit of that, right? Bringing more understanding into the space that we work in nowadays. [0:18:05.4] DC: I wonder if maybe it also would be a benefit to have like more cross discussions like this one between developers and infrastructure kind of focused people, because we’re starting to see like as we’re crossing boundaries, we see that the same things that we’re doing on the infrastructure side, you’re also doing in the developer side. Like cap theorem as Scott mention which is the idea that you can have two out of three of consistency, availability and partitioning. That also applies to networking in a lot of ways. You can only have a network that is either like consistent or available but it can’t handle partitioning. It can be a consistent to handle partitioning but it’s not always going to be available, that sort of thing. These things that apply in from the software perspective also apply to us but we think about them as being so completely different. [0:18:52.5] JS: Yeah, I totally agree. I really think like on the app side, a couple of years ago, you know, I really just didn’t care anything outside of the JVM like my stuff on the JVM and if it got out to the network layer of the host like just didn’t care, know, need to know about that at all. But ever since cloud computing and distributed systems and everything became more prevalent, the overlap has become extremely obvious, right? In all these different concepts and it’s been really interesting to try to ramp up on that. [0:19:19.6]:19.3] NNL: Yeah, I think you know Scott and I both do this. I think as I imagine, actually, this is true of all four of us to be honest. But I think that it’s really interesting when you are out there talking to people who do feel like they’re stuck in some particular role like they’re specialists in some particular area and we end up having the same discussion with them over and over again. You know, like, “Look, that may pay the bills right now but it’s not going to pay the bills in the future.” And so you know, the question becomes, how can you, as a network engineer take your skills forward and not feel as though you’re just going to have to like learn everything all over again. I think that one of the things that network engineers are pretty decent at is characterizing those systems and being able to troubleshoot them and being able to do it right now and being able to like firefight those capabilities and those skills are incredibly valuable in the software development and in operationalizing applications and in SRE models. I mean, all of those skills transfer, you know? If you’re out there and you’re listening and you feel like I will always be a network engineer, consider that you could actually take those skills forward into some other role if you chose to. [0:20:25.1] JS: Yeah, totally agree. I mean, look at me, the lofty career that I’ve been come to. [0:20:31.4] SL: You know, I would also say that the fascinating thing to me and one of the reasons I launched, I don’t say this to like try and plug it but just as a way of talking about the reason I launched my own podcast which is now part of packet pushers, was exploring this very space and that is like we’ve got folks like Josh who comes from the application development spacing is now being, you know, in a way, forced to own and understand more infrastructure and we’ve got the infrastructure folks who now in a way, whether it be through the rise of cloud computing and abstractions away from visible items are being forced kind of up the stack and so they’re coming together and this idea of what does the future of the folks that are kind of like in our space, what does that look like? How much longer does a network engineer really need to be deeply versed in all the different layers? Because everything’s been abstracted away by some other type of thing whether it’s VPC’s or Azure V Nets or whatever the case is, right? I mean, you’ve got companies bringing the VPC model to on premises networks, right? As API’s become more prevalent, as everything gets sort of abstracted away, what does the future look like, what are the most important skills and it seems to me that it’s these concepts that we’re talking about, right? This idea of distributed systems and how distributed systems behave and how the components react to one another and understanding things like the cap theorem that are going to be most applicable rather than the details of trouble shooting VGP or understanding AWS VPC’s or whatever the case may be. [0:22:08.5] NL: I think there is always going to be a place for the people who know how things are running under the hood from like a physical layer perspective, that sort of thing, there’s always going to be the need for the grave beards, right? Even in software development, we still have the people who are slinging kernel code in C. And you know, they’re the best, we salute you but that is not something that I’m interested in it for sure. We always need someone there to pick up the pieces as it were. I think that yeah, having just being like, I’m a Cisco guy, I’m a Juniper guy, you know? I know how to pawn that or RSH into the switch and execute these commands and suddenly I’ve got this port is now you know, trunk to this V neck crap, I was like, Nick, remember your training, you know? How to issue those commands, I wonder, I think that that isn’t necessarily going away but it will be less in demand in the future. [0:22:08.5] SL: I’m curious to hear Josh’s perspective as like having to own more and more of the infrastructure underneath like what seems to be the right path forward for those folks? [0:23:08.7] JS: Yeah, I mean, unfortunately, I feel like a lot of times, it just ends up being trial by fire and it probably shouldn’t be that. But the amount of times that I have seen a deployment of some technology fall over because we overlapped the site range or something like that is crazy. Because we just didn’t think about it or really understand it that well. You know, like using one protocol, you just described BGP. I never ever dreamt of what BGP was until I started using attributed systems, right? Started using BGP as a way to communicate routes and the amount off times that I’ve messed up that connection because I don’t have a background in how to set that up appropriately, it’s been rough. I guess my perspective is that the technology has gotten better overall and I’m mostly obviously in the Kubernetes space, speaking to the technologies around a lot of the container networking solutions but I’m sure this is true overall. It seems like a lot of the sharp edges have been buffed out quite a bit and I have less of an opportunity to do things terribly wrong. I’ve also noticed for what it’s worth, a lot of folks that have my kind of background or going out to like the AWS is the Azure’s of the world. They’re using all these like, abstracted networking technologies that allow t hem to do really cool stuff without really having to understand how it works and they’re often times going back to their networking team on prem when they have on prem requirements and being like it should be this easy or XY and Z and they’re almost like pushing the networking team to modernize that and make things simpler. Based on experiences they’re having with these cloud providers. [0:24:44.2] DC: Yeah, what do you mean I can’t create a load balancer that crosses between these two disparate data centers as it easily is. Just issuing a single command. Doesn’t this just exist from a networking standpoint? Even just the idea that you can issue an API command and get a load balancer, just that idea alone, the thousands of times I have heard that request in my career. [0:25:08.8] JS: And like the actual work under the hood to get that to work properly is it’s a lot, there’s a lot of stuff going on. [0:25:16.5] SL: Absolutely, yeah, [0:25:17.5] DC: Especially when you’re into plumbing, you know? If you’re going to create a load balancer with API, well then, what API does the load balancer use to understand where to send that traffic when it’s being balanced. How do you handle discovery, how do you hit like – obviously, yeah, there’s no shortage on the amount of work there. [0:25:36.0] JS: Yeah. [0:25:36.3] DC: That’s a really good point, I mean, I think sometimes it’s easy for me to think about some of these API driven networking models and the cost that come with them, the hidden cost that come with them. An example of this is, if you’re in AWS and you have a connectivity between wo availability, actually could be any cloud, it doesn’t have to be an AWS, right? If you have connectivity between two different availability zones and you’re relying on that to be reliable and consistent and definitely not to experience, what tools do you have at your disposal, what guarantees do you have that that network has even operating in a way that is responsive, right? And in a way, this is kind of taking us towards the observability conversation that I think we’ve talked a little bit about the past. Because I think it highlights the same set of problems again, right? You have to understand, you have to be able to provide the consumers of any service, whether that service is plumbing, whether it’s networking, whether it’s your application that you’ve developed that represents a set of micro service. You have to provide everybody a way or you know, have to provide the people who are going to answer the phone at two in the morning. Or even the robots that are going to answer the phone at two in the morning. I have to provide them some mechanism by which to observe those systems as they are in use. [0:26:51.7] JS: I’m not convinced that very many of the cloud providers do that terribly well today, you know? I feel like I’ve been burned in the past without actually having an understanding of the state that we’re in and so it is interesting maybe the software development team can actually start pushing that down toward the networking vendors out there out in the world. [0:27:09.9] NL: Yeah that would be great. I mean I have been recently using a managed Kubernetes service. I have been kicking the tires on it a little bit. And yeah there has been a couple of times where I had just been got by networking issues. I am not going to get into what I have seen in a container network interface or any of the technologies around that. We are going to talk about that another time. But the CNI that I am using in this managed service was just so wonky and weird. And it was failing from a network standpoint. The actual network was failing in a sense because the IP addresses for the nodes themselves or the pods wasn’t being released properly and because of our bag. And so, the rules associated with my account could not remove IP addresses from a node in the network because it wasn’t allowed to and so from a network, I ran out of IP addresses in my very small site there. [0:28:02.1] SL: And this could happen in database, right? This could happen in a cache of information, this could happen in pretty much the same pattern that you are describing is absolutely relevant in both of these fields, right? And that is a fascinating thing about this is that you know we talk about the network generally in these nebulous terms and that it is like a black box and I don’t want them to know anything about it. I want to learn about it, I don’t want to understand it. I just want to be able to consume it via an API and I want to have the expectation that everything will work the way it is supposed to. I think it is fascinating that on the other side of that API are people maybe just like you who are doing their level best to provide, to chase the cap theorum into it’s happy end and figure out how to actually give you what you need out of that service, you know? So, empathy I think is important. [0:28:50.4] NL: Absolutely, to bring that to an interesting thought that I just had where on both sides of this chasm or whatever it is between networking and develop, the same principles exists like we have been saying but just to elicited on it a little bit more, it’s like on one side you have like I need to make sure that these ETCD nodes communicate with each other and that the data is consistent across the other ones. So, we use a protocol called RAFT, right? And so that’s eventually existent tool then that information is sent onto a network, which is probably using OSPF, which is “open shortest path first” routing protocol to become eventually consistent on the data getting from one point to the other by opening the shortest path possible. And so these two things are very similar. They are both these communication protocols, which is I mean that is what protocol means, right? The center for communication but they’re just so many different layers. Obviously of the OSI model but people don’t put them together but they really are and we keep coming back to that where it is all the same thing but we think about it so differently. And I am actually really appreciating this conversation because now I am having a galaxy brain moment like boo. [0:30:01.1] SL: Another really interesting one like another galaxy moment, I think that is interesting is if you think about – so let us break them down like TCP and UTP. These are interesting patterns that actually do totally relate again just in software patterns, right? In TCP the guarantee is that every data gram, if you didn’t get the entire data gram you will understand that you are missing data and you will request a new version of that same packet. And so, you can provide consistency in the form of retries or repeats if things don’t work, right? Not dissimilar from the ability to understand like that whether you chuck some in data across the network or like in a particular data base, if you make a query for a bunch of information you have to have some way of understanding that you got the most recent version of it, right? Or ETCD supports us by using the revision by understanding what revision you received last or whether that is the most recent one. And other software patterns kind of follow the same model and I think that is also kind of interesting. Like we are still using the same primitive tools to solve the same problems whether we are doing it at a software application layer or whether we are doing it down in the plumbing at the network there, these tools are still very similar. Another example is like UTP where it is basically there are no repeats. You either got the packet or you didn’t, which sounds a lot like an event stream to me in some ways, right? Like it is very interesting, you just figured out like I put in on the line, you didn’t get it? It is okay, I will put another line here in a minute you can react to that one, right? It is an interesting overlap. [0:31:30.6] NL: Yeah, totally. [0:31:32.9] JS: Yeah, the comparison to event streams or message queues, right? There is an interesting one that I hadn’t considered before but yeah, there are certainly parallels between saying, “Okay I am going to put this on the message queue,” and wait for the acknowledgement that somebody has taken it and taken ownership of it as oppose to an event stream where it is like this happened. I admit this event. If you get it and you do something with it, great. If you don’t get it then you don’t do something with it, great because another event is going to come along soon. So, there you go. [0:32:02.1] DC: Yep, I am going to go down a weird topic associated with what we are just talking about. But I am going to get a little bit more into the weeds of networking and this is actually directed into us in a way. So, talking about the kind of parallels between networking and development, in networking at least with TCP and networking, there is something called CSMACD, which is “carry your sense multi,” oh I can’t remember what the A stands for and the CD. [0:32:29.2] SL: Access. [0:32:29.8] DC: Multi access and then CD is collision detection and so basically what that means is whenever you sent out a packet on the network, the network device itself is listening on the network for any collisions and if it detects a collision it will refuse to send a packet until a certain period of time and they will do a retry to make sure that these packets are getting sent as efficiently as possible. There is an alternative to that called CMSCA, which was used by Mac before they switched over to using a Linux based operating system. And then putting a fancy UI in front of it, which collision avoidance would listen and try and – I can’t remember exactly, it would time it differently so that it would totally just avoid any chance that there could be collision. It would make sure that no packets were being sent right then and then send it back up. And so I was wondering if something like that exists in the realm between the communication path between applications. [0:33:22.5] JS: Is it collision two of the same packets being sent or what exactly is that? [0:33:26.9] DC: With the packets so basically any data going back and forth. [0:33:29.7] JS: What makes it a collision? [0:33:32.0] SL: It is the idea that you can only transmit one message at a time because if they both populate the same media it is trash, both of them are trash. [0:33:39.2] JS: And how do you qualify that. Do you receive an ac from the system or? [0:33:42.8] NL: No there is just nothing returned essentially so it is like literally like the electrical signals going down the wire. They physically collide with each other and then the signal breaks. [0:33:56.9] JS: Oh, I see, yeah, I am not sure. I think there is some parallels to that maybe with like queuing technologies and things like that but can’t think of anything on like direct app dev side. [0:34:08.6] DC: Okay, anyway sorry for that tangent. I just wanted to go down that little rabbit-hole a little bit. It was like while we are talking about networking, I was like, “Oh yeah, I wanted to see how deep down we can make this parallel going?” so that was the direction I went. [0:34:20.5] SL: Like where is that that CSMACD, a piece is like seriously old school, right? Because it only applied to half duplex Ethernet and as soon as we went to full duplex Ethernet it didn’t matter anymore. [0:34:33.7] DC: That is true. I totally forgot about that. [0:34:33.8] JS: It applied the satellite with all of these as well. [0:34:35.9] DC: Yeah, I totally forgot about that. Yeah and with full duplex, we totally just space on that. This is – damn Scott, way to make me feel old. [0:34:45.9] SL: Well I mean satellite stuff, too, right? I mean it is actually any shared media upon which you have to – where if this stuff goes and overlap there, you are not going to be able to make it work right? And so, I mean it is interesting. It is actually an interesting PNL. I am struggling to think of an example of this as well. I mean my brain is going towards circuit breaking but I don’t think that that is quite the same thing. It is sort the same thing that in a circuit breaking pattern, the application that is making the request has the ability obviously because it is the thing making the request to understand that the target it is trying to connect to is not working correctly. And so, it is able to make an almost instantaneous decision or at least a very shortly, a very timely decision about what to do when it detects that state. And so that’s a little similar and that you can and from the requester side you can do things if you see things going awry. And really and in reality, in the circuit breaking pattern we are making the assumption that only the application making the request will ever get that information fast enough to react to it. [0:35:51.8] JS: Yeah where my head was kind of going with it but I think it is pretty off is like on a low level piece of code like it is maybe something you write in C where you implement your own queue in that area and then multiple threads are firing off the same time and there is no block system or mechanism if two threads contend to put something in the same memory space that that queue represents. That is really going down the rabbit hole. I can’t even speak to what degree that is possible in modern programming but that is where my head was. [0:36:20.3] NL: Yeah that is a good point. [0:36:21.4] SL: Yeah, I think that is actually a pretty good analogy because the key commonality here is some sort of shared access, right? Multiple threads accessing the same stack or memory buffer. The other thing that came to mind to me was like some sort of session multiplexing, right? Where you are running multiple application layer sessions inside a single sort of network connection and those network sessions getting comingled in some fashion. Whether through identifiers or sequence number or something else of that nature and therefore, you know garbling the ultimate communication that is trying to be sent. [0:36:59.2] DC: Yeah, locks are exactly the right direction, I think. [0:37:03.6] NL: That is a very good point. [0:37:05.2] DC: Yeah, I think that makes perfect sense. Good, all right. Yes, we nailed it. [0:37:09.7] SL: Good job. [0:37:10.8] DC: Can anybody here think of a software pattern that maybe doesn’t come across that way? When you are thinking about some of the patterns that you see today in cloud native applications, is there a counter example, something that the network does not do at all? [0:37:24.1] NL: That is interesting. I am trying to think where event streams. No, that is just straight up packets. [0:37:30.7] JS: I feel like we should open up one of those old school Java books of like 9,000 design patterns you need to know and we should go one by one and be like, “What about this” you know? There is probably something I can’t think of it off the top of my head. [0:37:43.6] DC: Yeah me neither. I was trying to think of it. I mean like I can think of a myriad of things that do cross over even the idea of only locally relevant state, right? That is like a cam table on a switch that is only locally relevant because once you get outside of that switching domain it doesn’t matter anymore and it is like there is a ton of those things that totally do relate, you know? But I am really struggling to come up with one that doesn’t – One thing that is actually interesting is I was going to bring up – we mentioned the cap theorem and it is an interesting one that you can only pick like two and three of consistency availability and partition tolerance. And I think you know, when I think about the way that networks solve or try to address this problem, they do it in some pretty interesting way. It’s like if you were to consider like Spanning Tree, right? The idea that there can really only be one path through a series of broadcast domains. Because we have multiple paths then obviously we are going to get duplicity and the things are going to get bad because they are going to have packets that are addressed the same things across and you are going to have all kinds of bad behaviors, switching loops and broadcast storms and all kinds of stuff like that and so Spanning Tree came along and Spanning Tree was invented by an amazing woman engineer who created it to basically ensure that there was only one path through a set of broadcast domains. And in a way, this solved that camp through them because you are getting to the point where you said like since I understand that for availability purpose, I only need one path through the whole thing and so to ensure consistency, I am going to turn off the other paths and to allow for partition tolerance, I am going to enable the system to learn when one of those paths is no longer viable so that it can re-enable one of the other paths. Now the challenge of course is there is a transition period in which we lose traffic because we haven’t been able to open one of those other paths fast enough, right? And so, it is interesting to think about how the network is trying to solve with the part that same set of problems that is described by the cap theorem that we see people trying to solve with software routine. [0:39:44.9] SL: No man I totally agree. In a case like Spanning Tree, you are sacrificing availability essentially for consistency and partition tolerance when the network achieves consistency then availability will be restored and there is other ways to doing that. So as we move into systems like I mentioned clos fabrics earlier, you know a cost fabric is a different way of establishing a solution to that and that is saying I’d later too. I will have multiple connections. I will wait those connections using the higher-level protocol and I will sacrifice consistency in terms of how the routes are exchanged to get across that fabric in exchange for availability and partition columns. So, it is a different way of solving the same problem and using a different set of tools to do that, right? [0:40:34.7] DC: I personally find it funny that in the cap theorem there is at no point do we mention complexity, right? We are just trying to get all three and we don’t care if it’s complex. But at the same time, as a consumer of all of these systems, you care a lot about the complexity. I hear it all the time. Whether that complexity is in a way that the API itself works or whether even in this episode we are talking about like I maybe don’t want to learn how to make the network work. I am busy trying to figure out how to make my application work, right? Like cognitive load is a thing. I can only really focus on so many things at a time where am I going to spend my time? Am I going to spend it learning how to do plumbing or am I going to spend it actually trying the right application that solves my business problem, right? It is an interesting thing. [0:41:17.7] NL: So, with the rise of software defined networking, how did that play into the adoption of cloud native technologies? [0:41:27.9] DC: I think it is actually one of the more interesting overlaps in the space because I think to Josh’s point again. his is where we were taking I mean I work for a company called [inaudible 0:41:37], in which we were virtualizing the network and this is fascinating because effectively we are looking at this as a software service that we had to bring up and build and build reliably and scalable. Reliably and consistently and scalable. We want to create this all while we are solving problems. But we need it to do within an API. It is like we couldn’t make the assumption with the way that networks were being defined today like going to each component and configuring them or using protocols was actually going to work in this new model of software confined networking. And so, we had an incredible amount of engineers who were really focused from a computer science perspective on how to effectively reinvent network as a software solution. And I do think that there is a huge amount of cross over here like this is actually where I think the waters meet between the way the developers think about the problems and the way that network engineers think about the problem but it has been a rough road I will say. I will say that STN I think is actually has definitely thrown a lot of network engineers under their heels because they’re like, “Wait, wait but that is not a network,” you know? Because I can’t actually look at it and characterize it in the way that I am accustomed to looking at characterizing the other networks that I play with. And then from the software side, you’re like, “Well maybe that is okay” right? Maybe that is enough, it is really interesting. [0:42:57.5] SL: You know I don’t know enough about the details of how AWS or Azure or Google are actually doing their networking like and I don’t even know and maybe you guys all do know – but I don’t even know that aside from a few tidbits here and there that AWS is going to even divulge the details of how things work under the covers for VPC’s right? But I can’t imagine that any modern cloud networking solution whether it would be VBPC’s or VNET’s or whatever doesn’t have a significant software to find aspect to it. You know, we don’t need to get into the definitions of what STN is or isn’t. That was a big discussion Duffie and I had six years ago, right? But there has to be some part of it that is taking and using the concepts that are common in STN right? And applying that. Just as the same way as the cloud vendors are using the concepts from compute virtualization to enable what they are doing. I mean like the reality is that you know the work that was done by the Cambridge folks on Zen was a massive enabler trade for AWS, right? The word done on KVM also a massive enabler for lots of people. I think GCP is KBM based and V Sphere where VM Ware data as well. I mean all of this stuff was a massive enablers for what we do with compute virtualization in the cloud. I have to think that whether it is – even if it wasn’t necessarily directly stemming out of Martin Casado’s open flow work at Stanford, right? That a lot of these software define networking concepts are still seeing use in the modern clouds these days and that is what enables us to do things like issue an API call and have an isolated network space with its own address space and its own routing and satiated in some way and managed. [0:44:56.4] JS: Yeah and on that latter point, you know as a consumer of this new software defined nature of networking, it is amazing the amount of I don’t know, I started using like a blanket marketing term here but agility that it is added, right? Because it has turned all of these constructs that I used to file a ticket and follow up with people into self-service things that when I need to poke holes in the network, hopefully the rights are locked down, so I just can’t open it all up. Assuming I know what I am doing and the rights are correct it is totally self-service for me. I go into AWS, I change the security group roll and boom, the ports have changed and it never looked like that prior to this full takeover of what I believe is STN almost end to end in the case of AWS and so on. So, it is really just not only has it made people like myself have to understand more about networking but it has allowed us to self-service a lot of the things. That I would imagine most network engineers were probably tired of doing anyways, right? How many times do you want to go to that firewall and open up that port? Are you really that excited about that? I would imagine not so. [0:45:57.1] NL: Well I can only speak from experience and I think a lot of network engineers kind of get into that field because it really love control. And so, they want to know what these ports are that are opening and it is scary to be like this person has opened up these ports, “Wait what?” Like without them even totally knowing. I mean I was generalizing, I was more so speaking to myself as being self-deprecating. It doesn’t apply to you listener. [0:46:22.9] JS: I mean it is a really interesting point though. I mean do you think it makes the networking people or network engineers maybe a little bit more into the realm of observability and like knowing when to trigger when something has gone wrong? Does it make them more reactive in their role I guess. Or maybe self-service is not as common as I think it is. It is just from my point of view, it seems like with STN’s the ability to modify the network more power has been put into the developers’ hands is how I look at it, you know? [0:46:50.7] DC: I definitely agree with that. It is interesting like if we go back a few years there was a time when all of us in the room here I think are employed by VMware. So, there was a time where VMware’s thing was like the real value or one of the key values that VMware brought to the table was the idea that a developer come and say “Give me 10 servers.” And you could just call an API or make it or you could quickly provision those 10 servers on behalf of that developer and hand them right back. You wouldn’t have to go out and get 10 new machines and put them into a rack, power them and provision them and go through that whole process that you could actually just stamp those things out, right? And that is absolutely parallel to the network piece as well. I mean if there is nothing else that SPN did bring to the fore is that, right? That you can get that same capability of just stamping up virtual machines but with networks that the API is important in almost everything we do. Whether it is a service that you were developing, whether it is a network itself, whether it is the firewall that we need to do these things programmatically. [0:47:53.7] SL: I agree with you Duffie. Although I would contend that the one area that and I will call it on premises STN shall we say right? Which is the people putting on STN solutions. I’d say the one area at least in my observation that they haven’t done well is that self-service model. Like in the cloud, self-service is paramount to Josh’s point. They can go out there, they can create their own BPC’s, create their own sub nets, create their own NAT gateways, Internet gateways to run security groups. Load balancers, blah-blah, all of that right? But it still seems to me that even though we are probably 90, 95% of the way there, maybe farther in terms of on premise STN solutions right that you still typically don’t see self-service being pushed out in the same way you would in the public cloud, right? That is almost the final piece that is needed to bring that cloud experience to the on-premises environment. [0:48:52.6] DC: That is an interesting point. I think from an infrastructure as a service perspective, it falls into that realm. It is a problem to solve in that space, right? So when you look at things like OpenStack and things like AWS and things like JKE or not JKE but GCE and areas like that, it is a requirement that if you are going to provide infrastructure as a service that you provide some capability around networking but at the same time, if we look at some of the platforms that are used for things like cloud native applications. Things like Kubernetes, what is fascinating about that is that we have agreed on a least come – we agreed on abstraction of networking that is maybe I don’t know, maybe a little more precooked you know what I mean? In the assumption within like most of the platforms as a service that I have seen, the assumption is that when I deploy a container or I deploy a pod or I deploy some function as a service or any of these things that the networking is going to be handled for me. I shouldn’t have to think about whether it is being routed to the Internet or not or routed back and forth between these domains. I should if anything only have to actually give you intent, be able to describe to you the intent of what could be connected to this and what ports I am actually going to be exposing and that the platform actually hides all of the complexity of that network away from me, which is an interesting round to strike. [0:50:16.3] SL: So, this is one of my favorite things, one of my favorite distinctions to make, right? And that is this is the two worlds that we have been talking about, applications and infrastructure and the perfect example of these different perspectives and you even said it or you talked there Duffie like from an IS perspective it is considered a given that you have to be able to say I want a network, right? But when you come at this from the application perspective, you don’t care about a network. You just want network connectivity, right? And so, when you look at the abstractions that IS vendors and solutions or products have created then they are IS centric but when you look at the abstractions that have been created in the cloud data space like within Kubernetes, they are application centric, right? And so, we are talking about infrastructure artifacts versus application artifacts and they end up meeting but they are coming at this from two different very different perspectives. [0:51:18.5] DC: Yeah. [0:51:19.4] NL: Yeah, I agree. [0:51:21.2] DC: All right, well that was a great discussion. I imagine that we are probably get into – at least I have a couple of different networking discussions that I wanted to dig into and this conversation I hope that we’ve helped draw some parallels back and forth between the way – I mean there is both some empathy to spend here, right? I mean the people who are providing the service of networking to you in your cloud environments and your data centers are solving almost exactly the same sorts of availability problems and capabilities that you are trying to solve with your own software. And I think in itself is a really interesting takeaway. Another one is that again there is nothing new under the sun. The problems that we are trying to solve in networking are not different than the problems that you are trying to solve in applications. We have far fewer tools and we generally network engineers are focused on specific changes that happen in the industry rather than looking at a breathe of industries like I mean as Josh pointed out, you could break open a Java book. And see 8,000 patterns for how to do Java and this is true, every programming language that I am aware of I mean if you look at Go and see a bunch of different patterns there and we have talked about different patterns for just developing cloud native aware applications as well, right? I mean there is so many options in the software versus what we can do and what are available to us within networks. And so I think I am rambling a little bit but I think that is the takeaway from this session. Is that there is a lot of overlap and there is a lot of really great stuff out there. So, this is Duffie, thank you for tuning in and I look forward to the next episode. [0:52:49.9] NL: Yep and I think we can all agree that Token Ring should have won. [0:52:53.4] DC: Thank you Josh and thank you Scott. [0:52:55.8] JS: Thanks. [0:52:57.0] SL: Thanks guys, this was a blast. [END OF EPISODE] [0:52:59.4] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

Our topic in today's great episode is how we think jobs in software engineering have changed since the advent of cloud native computing. We begin by giving our listeners an idea of our jobs and speak more to what a job in cloud native would look like as well as how Kubernetes fits into the whole picture. Next up we cover some old challenges and how advances in the field have made those go away while simultaneously opening the gateway to even more abstract problems. We talk about some of the specific new developments and how they have changed certain jobs. For example, QA has not disappeared but rather evolved toward becoming ever more automated, and language evolution has left more space for actual development instead of debugging. Our conversation shifts toward some tips for what to know to get into cloud native and where to find this information. We wrap up our conversation with some thoughts on the future of this exciting space, predicting how it might change but also how it should change. Software engineering is still in a place where it is continuously breaking new ground, so tune in to hear why you should be learning as much as you can about development right now. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Bryan Liles Nicholas Lane Key Points From This Episode: • The work descriptions of our hosts who merge development, sysadmin, and consulting.• What a cloud native related job looks like.• Conceptualizing cloud native in relation to development, sysadmin, and DevOps.• A cloud native job is anything related to building software for other people’s computers.• Kubernetes is just one way of helping software run easily on a cloud.• Differences between cloud native today and 10 years ago: added ease through more support.• How cloud native developing is the new full stack due to the wide skillset required.• An argument that old challenges are gone but have introduced more abstract ones.• Advances making transitioning from testing to production more problem-free.• How QA has turned into SDE, meaning engineers now write software that tests.• Why jobs have matured after the invention of cloud native.• Whether the changes in jobs have been one of titles or function.• How languages like Rust, Go, and Swift have changed developer jobs by being less buggy.• What good support equates to, beyond names like CRE and company size.• The many things people who want to get into cloud native should know.• Prospective cloud native workers should understand OSs, networking, and more.• Different training programs for learning Kubernetes such as CKA and CKAD.• Resources for learning such as books, YouTube videos, and podcasts.• Predictions and recommendations for the future of cloud native. • Tips for recruiters such as knowing the software they are hiring for. Quotes: “What is the cloud? The cloud is other people’s computers. It's LPC, and what is Kubernetes? Well, basically, it’s a way that we can run our software on other people’s computers, AKA the cloud.” — @bryanl [0:07:35] “What we have now is we know what we can do with distributed computing and now we have a great set of software for multiple vendors who allow us to do what we want to do.” — @bryanl [0:10:03] “There are certain challenges now in cloud native that are gone, so the things that were hard before like spinning up a server or getting the database are gone and that frees us to worry about more complicated or more abstract ideas.” — @apinick [0:12:58] “The biggest problem with what we are doing is that we are trailblazing. So a lot of the things that are happening, like the way that Kubernetes advances every few months is new, new, new, new.” — @bryanl [0:36:11] “Now is the literal best time to get into writing software and specifically for cloud native applications.” — @bryanl [0:42:22] Links Mentioned in Today’s Episode: Azure — https://azure.microsoft.com/en-us/ Google Cloud Platform — https://cloud.google.com/ AWS — https://aws.amazon.com/ Amazon RDS — https://aws.amazon.com/rds/ Mesosphere — https://d2iq.com/ Aurora — https://stackshare.io/stackups/aurora-vs-mesos-vs-mesosphere Marathon — https://mesosphere.github.io/marathon/ Rails Rumble — http://blog.railsrumble.com/ Terraform — https://www.terraform.io/intro/index.html Swift — https://developer.apple.com/swift/ Go — https://golang.org/ Rust — https://www.rust-lang.org/ DigitalOcean — https://www.digitalocean.com/ Docker — https://www.docker.com/ Swarm — https://www.santafe.edu/research/results/working-papers/the-swarm-simulation-system-a-toolkit-for-building HashiCorp — https://www.hashicorp.com/ Programming Kubernetes on Amazon — https://www.amazon.com/Programming-Kubernetes-Developing-Native-Applications/dp/1492047104 The Kubernetes Cookbook on Amazon — https://www.amazon.com/Kubernetes-Cookbook-Building-Native-Applications/dp/1491979682 Kubernetes Patterns on Amazon — https://www.amazon.com/Kubernetes-Patterns-Designing-Cloud-Native-Applications/dp/1492050288 Cloud Native DevOps with Kubernetes on Amazon — https://www.amazon.com/Cloud-Native-DevOps-Kubernetes-Applications/dp/1492040762 Kubernetes in Action on Amazon — https://www.amazon.com/Kubernetes-Action-Marko-Luksa/dp/1617293725 Managing Kubernetes on Amazon — https://www.amazon.com/Managing-Kubernetes-Operating-Clusters-World/dp/149203391X Transcript: EPISODE 14 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.3] NL: Hello and welcome back. This week, we’ll be discussing the thing that’s brought us all together, our jobs. But not just our jobs. I think we’re going to be talking about the difference kind of jobs you can find in cloud native land. This time, I’m your host, Nicolas Lane and with me are Brian Liles. [0:00:57.1] BL: Howdy. [0:00:58.0] NL: And Carlisia Campos. [0:00:59.6] CC: Hi everybody, glad to be here. [0:01:02.6] NL: How’s it going you all? [0:01:03.7] CC: Very good. [0:01:05.4] NL: Cool. To get us started, let’s talk about our jobs and like what it means to have a job and like the cloud native land from our current perspective. Brian, you want to go ahead and kick us off? [0:01:17.8] BL: Wow, cloud native jobs. What is my job? My job is – I look at productivity of developers and people who are using Kubernetes. My job is to understand cloud native apps but also understand that the systems that they are running on are complex and whether they’d be Windows or Linux or Mac based, being able to understand those too. Really, my job is the combination of a senior developer, composed with a senior level admin. Whether it be Windows or Linux. Maybe I am the actual epitome of DevOps. [0:01:58.5] NL: Yeah you seem to be kind of fusion of the two. Carlisia? [0:02:03.3] CC: My job is – so I’m mainly a developer but to do the job that I need to do, I need to be a bit of a DevOps person as well because as I’ve talked many times here on the show, I work on an open source too called Valero that does backup and recovery for Kubernetes clusters. I need to be able to boot up our cluster at least with three main providers. Azure, Google Cloud Platform, and AWS. I need to know how to do that, how to tweak things, how to troubleshoot things and I don’t think when we think of just a straight up developer, that usually is not part of the daily activity. In that sense, I think, I’m not sure how we would define the cloud native job but I think my job, if there is such a thing, my job definitely is a cloud native job because I have to interact with these cloud native technologies, even beyond what I – the actual app that I’m developing which runs inside a Kubernetes cluster so it all ties in. You Nick? [0:03:16.0] NL: My job is I’m a cloud native architect or a Kubernetes architect, I’m not sure what we’re calling ourselves these days honestly. What that means is we work with customers to help them along their cloud native journey. Either that means helping them set up like a Kubernetes cluster and then getting them like running with certain tools that are going to make there life easier or helping them develop tools in their cloud environments to help make the running of their jobs easier. We kind of run the gamut of developers and sys. admins a bit and consultants. We kind of touch a little bit of everything. Let’s take a step back now and talk about what we think a cloud native job looks like? Because for me, that’s kind of hard to describe. A cloud native job seems to be any job that has to do with some cloud native technology but that’s kind of broad, right? You could have things from sysadmins, people who are running their cloud infrastructure for the company who are like managing things like, you know, rights access, accounting, that sort of thing, to people who are doing development like yourselves, like Brian and Carlisia, you guys are doing this type of work. Is there anything that you think is like unique to a cloud native job? [0:04:35.2] CC: Yeah, it’s very interesting to talk about I think because especially in relation to if you don’t have a cloud native job, what do you have and how is it different? I wonder if the new cloud native job title is the new full stack developer for developers because, I think it’s easier to conceptualize what a cloud native job is for a systems admin or dev ops person. But for developer, I think it’s a little more tricky, right? Is it the new full stack? Is it now that the developer even if you’re not doing – for example, my application runs inside Kubernetes, it’s an extension of Kubernetes but some applications just run on Kubernetes as a platform. Now, are we talking about developers with a cloud native title like ‘cloud native software engineer’ and for those developers, does it mean that they now have to design, code and deploy consistently? You know, in my old days, when I – before doing this type of work, I would deploy apps but it was not all the time. There was a system, every single job I had, the system was different. The one thing that I love about Kubernetes is that if I was just a regular app developer, again as supposed to like extending Kubernetes, right? If I was building apps that would run on Kubernetes as supposed to extending Kubernetes, and if I had to deploy them at Kubernetes, if I move jobs and they were working with Kubernetes, this process would be exactly the same and that’s one really cool thing about. I wouldn’t mind – in other words, I wouldn’t mind so much if I had to do deployment in the deployment, the process was the same everywhere. Because it’s really painful to do like a one off deployment here and there, each place was different, I had to write a ton of notes to make sure, you know – it was like, 200 stacks and if anyone of them, you had to troubleshoot and I’m not a systems admin so it will be a struggle. [0:06:44.6] BL: Yeah. [0:06:45.8] CC: Because each system – it’s not that I couldn’t learn but each system would be different and I make – anyway, I think I went off on a tangent. [0:06:53.1] NL: No worries. [0:06:54.1] CC: But I also wanted to mention that I searched on LinkedIn for cloud native in the jobs section and there are a ton of job titles, job postings with cloud native in the title like a lot of it is architect but there is also product manager, there is also software engineer, I found the one that was senior Kubernetes engineer. It’s definitely a thing. [0:07:21.0] BL: All right. What is the question here? [0:07:25.6] NL: It was what do we think a cloud native job looks like essentially? [0:07:29.6] BL: All right. I’m going to blow your mind here. Basically, what is the cloud? The cloud is other people’s computers. It's LPC and what is Kubernetes? Well, basically, it’s a way that we can run our software on other people’s computers, AKA the cloud. Kubernetes makes running software in the cloud easier. What that really breaks down to is if you are writing software on other people’s computers or if you were designing software that runs well on clouds, well, you’re a cloud native person. Actually, the term is basically been co opted for marketing purposes by who knows who.Basically, everyone. But what I think is, as long as you are working on software that runs on modern infrastructure which means that nodes may go away, you might not own all of your services, you might use a least database server, you know, something like RDS from Amazon. Everyone working in that realm, working with software that may go away with things that aren’t theirs, was doing cloud native work. We just happen to be doing on Kubernetes because that’s the most popular option right now. It isn’t the only option and it probably won’t be the final option. [0:08:48.7] CC: Do you see any difference between what is required for a job like that today? Versus maybe 10 years ago or five years ago? Brian? [0:08:58.0] BL: Yeah, actually, I do see some differences. One of the biggest differences is that there’s a lot more services out there that are provided to help you do what you need to do and so 10 years ago, having a database provider would be hard because one, the network wouldn’t be good enough and you’re hosting company probably didn’t have that unless you were at AWS and even they didn’t have that. Now, what we get to take advantage off is things are just easier, it’s easier to fire up databases, it’s easier to add nodes to our production. It’s easier to have multiple productions, it’s easier to keep that all in order. It’s easier to put automated configuration around that than it was 10 years ago. Now, five years ago, back in 2014, I would actually say that the way that we progressed since then is that we became more mature. I remember when Kubernetes came out and I thought it was going to win but Mesosphere was, mesosphere with Aurora, or marathon was actually better than Kubernetes, just it worked out of the box, for what we thought we could do with it but now, what we have now is we know what we can do with distributed computing and now we have a great set of software for multiple vendors who allow us to do what we want to do. That’s the best part about now versus five years ago. [0:10:17.7] CC: Yeah, I have to agree with that, it’s definitely easier. As a developer, I’m not going to tell you it’s easy but it’s easier. As an example. I remember when that was Rails Rumble maybe 10 years ago, I don’t know. [0:10:31.3] BL: Yeah, I remember. [0:10:34.0] CC: You did a video showing step by step how to boot up a Linux server to run apps on that server. I don’t remember why we needed to boot up from scratch. Remember that Brian? [0:10:46.9] BL: I do remember that. That was 2007 or eight? It was a long time ago. [0:10:53.0] CC: That was one of the place that made me very impressed about you because I followed all the steps and at the end it worked. You just was – you were right on with – as far as the instructions went. I think doing that, I think it took me about two hours, I remember it took a long time and because this again, these are things that I do once in a while, I don’t do these things all the time. Now, we can use a Terraform script and have something running in a matter of 15 minutes if you have. [0:11:26.9] BL: Side bar. Quick side bar. Yeah, we can use Terraform. I use Terraform for even all my personal infrastructure so things that are running in my house use Terraform. All my work stuff uses Terraform. But still, it’s sometimes easier to just write a script or type in the commands on the command line or click something. We’re still not to the point where using things like Terraform actually makes us not want to do it manually. That’s how I know that we’re not to our ultimate level maturity yet. But, if you want to, the options are there and they’re pretty good. [0:12:01.8] CC: Yeah. [0:12:03.6] NL: Carlisia, you said something that kind of reminded me and maybe kind of get down this path. While we’re talking about like there are certain challenges that we aren’t faced anymore in a cloud native land like things are easier, there are certain things that are easier, not to say that our jobs are easy, like you’re saying Carlisia. But it was something along the lines of like a developer now needs to be – like a cloud native job is now the full stack kind of job or full stack developer. That was the name of the game back in the day, now, it’s a cloud native job. I actually kind of agree with that in a sense where a cloud native developer or anyone in the cloud native realm has to exist not just in their own silo anymore. You need to understand more of the infrastructure that you’re using to write your code on someone else’s computer better. I actually kind of like that. [0:12:56.6] CC: Exactly. [0:12:58.0] NL: Yeah, there are certain challenges now in cloud native that are gone so the things that were hard before like spinning up a server, you know, getting the database, these things are gone and that now that frees us to worry about more complicated or more abstract ideas like how do we have everyone agree on the API to use and thus rises Kubernetes. [0:13:19.1] CC: Yeah, I see that as a very positive thing. It might sound like – it’s a huge burden to ask developers to now have to now this but again, if we stick to the same stack, the burden diminishes really quickly because you learn it once and then that’s it. That’s been a huge advantage. If it works out this way, I mean, I’m all for like you know, the best technology should win. But there is that advantage. If we remain using the same container orchestrator, you know, we use containers, we can run our code as if we were running any machine. One advantage that I see is that I’ve had cases where you know, these was working on my computer (™) and it will be deployed and one little stupid thing wouldn’t work because the way the URL was redirected, didn’t work, broke things, I got yelled at. I’m like, “Okay, you want me to do this right? Give me a server.” Back then, good luck, “I’m going to give you a server, no way.” It was just so expensive, developers will be lucky to get a tasking of our staging environments. And, even when you get there, you had to coordinate with QA and there was a process. Now, because I have access to my own servers here, right? I can just imagine if I were a developer, building apps to run Kubernetes, admin could just say, “Okay, you have these resources, go for it.” I’ll have my own name space and I could run my code as if it was running and a production environment and I’ll have just more assurance that my code works basically. Which is to me so satisfying. It’s one less thing to worry about if I deploy something to production, I have already tested it. [0:15:17.3] NL: Yeah, that’s great. That’s something I really do cherish about the current landscape. We can actually test these things out locally and have confidence that they’ll work at least fairly well in production, right? [0:15:30.2] CC: It’s not just running things locally, you can actually get access to like a little slice of let’s say an AWS server and just shift your things there and test it there. But because these system admins people, they can just carve out that little one slice for your team of even in the per person basis, maybe that’s too much but it’s relatively uncomplicated to do that and not very costly. [0:15:56.9] NL: Yeah. You mentioned a team and the name of a team that I haven’t heard of in quite some time which is QA. How do we think the rise of cloud native have affected jobs and also kind of tangential to that, what were jobs like prior to cloud native because I haven’t heard of a QA team in many of the organizations that I’ve touched. Now, I’m not touching their like production dev team that they actually make this, I just haven’t heard of that name in a while and I’m wondering if like jobs like that have kind of gone away with the rise of cloud native. [0:16:27.7] BL: No, I’m going to end that rumor right here, that is a whole untruth. [0:16:33.8] NL: That was not a rumor, I’ve just conjecture I my part, literally unfounded. [0:16:38.7] BL: We got to think what does QA do? QA is supposed to be responsible for the quality of our applications and when they first started, there wasn’t a lot of good tooling so a lot of our QA people were manual testers. They started the app, they clicked on everything, they put in all the inputs until it came back and they were professional app breakers. I’d say, over a decade ago, we got more automated tools and moving into now, you can automate your web browser, you can actually write software to do all the actions that a human would do. What we found is that QA as profession has actually matured and you can see that because Google, I don’t think they even have QA, they have, what do they call them? Software engineers under test or SDE’s. What they do are – they’re developers in their own right but they write software that makes it easier for developers to write code that works well and a code that can be tested. I think that the roll has matured and has taken another angle in a lot of cases but even where we work. There are QA engineers in our group and we still need them because you’ve seen the meme where you talk about unit testing and it would be like a door that had all the right parts but it didn’t fit in it’s casing or too hot handles on a sink. The pieces work right? They both put out hot water but together they didn’t work. We still have that, it just looks a little bit different now. Also, a lot of software is not written in a huge monolithic cycle where we would take six months to release anew version or a year or a year and a half. Now, people are trying to turn around a lot of software quicker so QA has had to optimize how they work to work in those processes. They’re still there though. [0:18:37.8] CC: I would hope so. I mean, I can’t answer the question if the question is do we have as much QA efforts out there as before. I don’t know, but I hope so because if you don’t have a QA, if you’re not QAing your apps, then you’re users are. That’s not good. For my team for example, we do our own QA but we do QA. We don’t have separate people doing it, we do it ourselves. It might be just because it’s pretty special, I mean, we are a small team to begin with and what we do is very specialized. It will be difficult to bring someone in and teach them and if they’re just running QA, I don’t know, maybe – I don’t think it will be that difficult, we can just have constructions, you know. “Run this command, this is what the output should be” – I don’t’ think it would be that difficult, take that back, but still, we do it ourselves. [0:19:31.7] NL: The question was more – less in line with like, “What happened to QA?” It was more like, how do we think that cloud native has affected jobs and the job market and it sounds like, that jobs have changed because of cloud native, they’ve matured as we were just discussing with QA where people aren’t doing the same kind of drudgery or the same kind of toil that they were doing before. Now, we’re using more tooling to do our jobs and kind of lifting up each position to be more cloud native-y, right? More development and infrastructure focus at the same time. At least, that’s what I was getting from it. [0:20:09.8] BL: Yeah, I think that is true but I think all types of development jobs, especially jobs that are in the cloud native space have changed. One good example would be, with organizations moving to cloud native apps, we’re starting to see, and this is all anecdotal, I have no evidence to back this up, that there are more developers who are on call for the software they write because one, they know it better than anyone else and they’re closer to it. And two, because having an ops group that just supports app isn’t conducive to being productive because there’s no way that one group can understand all apps. What we’re finding is that in this new cloud native era that jobs are maturing and they’re getting new functionality, they’re losing some functionality, some jobs are combining but it’s still at the end of the day, it’s the same thing we were doing 20 years ago but it all just has new titles and we use new software to do it. Which is good. Because on some of these ideas that we came up with, 20, 30 years ago are still good ones today. [0:21:15.5] NL: Yeah, that’s actually an interesting question. Do you think that it’s just the titles that are changing or are the functions changing, right? It’s like sys admins used to be sys admins, now they’re CREs, well then there are dev ops for a while and now they’re CRE’s, more SRA’s I should say. Our support team are now CREs, Customer Reliability Engineering. Is that just a title change or are there functional differences? I’m inclined to believe that they’re functional differences. [0:21:43.9] BL: I think it’s both, I think it’s the same reason why all engineers after two years in the field are somehow senior engineers. People feel like they have progress when they get new titles even though you’re the most junior engineer on this team, how can you be a senior engineer? And then also the same thing with CRE, shout out to Google for making that term popular but really, what it comes down to is maybe the focus changed but maybe it didn’t. Maybe we were already doing that, maybe we were already doing resilience engineering with our customers and maybe we already had great customer support or customer success team. But I do think that there has been some changes in jobs because what we’re finding is that with modern languages that people are using so teams are moving away from C++ to things like Swift and Go and Rust. We’re finding that because our software is easier to write, we actually don’t have to think about some of the things that we did before. With Go, technically, you don‘t have to worry about memory access. With Rust, 100%, you don’t have to worry about null pointer exceptions, they don’t exist. Now that we freed our developers to do more development rather than more debugging, then what we can find is that the jobs will actually change over time. But at the end of the day, and even where we work right now and then all over the place, people are devs, they do ops stuff, they do security stuff or they’re pointing here at Boston. I challenge anyone listening to this to find something where I am not telling the truth, we might do both or more than one thing but at the end of the day, we can still break it down to what people do. [0:23:24.8] NL: Yeah, Carlisia, any thoughts on that? [0:23:26.1] CC: No, I think that was nice with me, sounds right. [0:23:29.8] NL: Yeah, I agree. I think that there are some functional changes. I think that support versus CRE isn’t just like getting tickets received and then going to a ticket queue and filing those things. I think there are some changes with like, I know from our CRE team they are actively going out and saying like, “Here’s our opinion based on these technologies and this is like why we validated these things,” that are reevaluating their support model constantly and just making sure that they’re like abreast of everything that’s going on so they can more resiliently engineer their customer support. [0:24:04.5] BL: But hold on, one second though. That’s what I’m talking about with the marketing because guess what? It is supported, a good support team would be doing all those things whether it’s called customer reliability engineering or whatever, it’s support, it’s customer success, it’s getting in front of our customer’s problems and having the answers before they even ask the question, that’s good support. Whenever we label things like CRE, that’s somebody and some corporate marketing center who thought that that was a good idea. But it doesn’t mean because you don’t call that CRE, it’s not good support because I will tell you in the past, DigitalOcean, we did that and the term CRE didn’t even exist yet but we were out there in front of problems whenever we could be and we thought that was good for our customers. What we’re finding is that people have the capabilities now with the progress of whatever technologies we have that we can actually give our customers good support, and you don’t have to be a Google sized company to do that anymore, that’s the plus. [0:25:02.9] NL: Yeah, I agree with that. [0:25:05.5] CC: I want us to talk a little bit about for people who are not working in a cloud native space but they see it coming or they want to move towards doing something more in that area, what should they be looking at? What should that be brushing up on their learning or incorporating into their what they are currently doing and of course different roles and so it will be different for each different role. We have developers we have DevOps or SRE or admins or operators, managers, recruiters. It changes a little bit for everybody. [0:25:47.5] BL: Well I will hop in here first and say it is all code at the end of the day. When it comes down to what we are doing in cloud native for ops, it doesn’t really matter. You could take a lot of the same principles and do them on prem or wherever else you happen to be. I mean I am not trying to diminish the role of anyone that we work with or anyone in our industry whenever I say this though but when it comes down to it, what I see is people understand the operating system, mostly Linux. People understand public key encryption so they understand PKI, you know we deal with a lot of certs. They understand networking, they can tell you how many IPs are in a 23 and if I am giving you side or numbers out there. These are things that people know. I don’t think there is anything specific for cloud native other than learning Kubernetes itself or Mesosphere or Docker, Swarm or whatever or the tool from HashiCorp that always escapes me whenever I have to say it out loud. But it is all the same thing. What you need to know and to be good at any job where you are doing ops, you need to understand the theory of operating computers. You need to understand operating systems, networking and how that all works and then all things around and some security. For developers now, it is a little bit interesting because a lot of the apps that we are writing these days are more stateless. So for a developer you need to think more about my app may crash. So anything that I am holding a memory that is important can go away at any given time. So either, one, I need to store it on more than one thing, I need to have it in a restrictive fashion or, two, I need to store it in the database instantly And I would once again challenge anyone to say that if you are a developer who can actually understand those topics, you would be able to apply for a cloud native job in this space because frankly a lot of developers, a lot of cloud native developers writing apps working cloud native, two years ago they were doing something else. [0:27:50.1] CC: Yeah that sounds right. I think for developers where you said I think focusing on authentication, how do you handle secret keys and the question of row authentication and row authorization and if you can even be well-versed in developing clients and servers and handling certs for that interaction and I guess it comes down to being well-versed in distributed systems development is what this whole cloud native is all about and on top of that I think being well-versed on how to push your apps into containers. You know create images, creating containers, pushing them into repository, pulling them from the repository and manipulating and creating containers in different ways and then on top of that maybe you want to learn Kubernetes and we can talk about that too but I wanted to give Nick a chance to talk about his aspects. [0:28:59.0] NL: I agree with pretty much everything you guys have said. I think the only thing I would add is like really understanding how to use and work with an API and an API driven environment because that is what a lot of cloud native is, right? It is using someone else’s computer so how do you do that? It is via an API like we’re talking about containers and orchestration, those are all done hopefully within API. Luckily, if you are using Kubernetes, which likely you are. It is all API driven. And so using an API, I think, and getting familiar with that. Most developers I think at some point are familiar with that but just that would be the main thing I would think too, outside of what you and Brian have already said are what is needed to do like a cloud native job. [0:29:40.2] CC: Yeah. Now if someone wanted to learn Kubernetes, well there is the Kubernetes Academy. [0:29:47.5] NL: There is a Kubernetes Academy. [0:29:49.4] CC: That is pretty awesome but do you think going through the certification would help? [0:29:55.3] NL: I think that is a good place to start. So the current certification that exists is the CKA, the Certified Kubernetes Administrator and I think that is a good starting place, especially for someone who is not really touched Kubernetes before. If they’re like, “How do I know the basics of Kubernetes?” going through that certification process I think will be a huge step forward because that really covers most of what you are going to touch on day to day basis for Kubernetes. [0:30:21.6] CC: And there is the CKAD as well, which is for developer. The CKAD is Certified Kubernetes – [0:30:29.7] BL: Application Developer. [0:30:31.7] CC: Application Developer and the other one is Certified Kubernetes Admin. [0:30:34.9] NL: Yeah I was like, “administrative developer?” like. [0:30:38.2] CC: If you are brand new I think it is worth while doing the developer first because it is mostly the commands. You go through the commands just so you have a knowledge of how to interact with Kubernetes and the admin is more like how you manage and how do you troubleshoot a cluster and how do you manage cluster. So it is more involved I think. You need to know more but in any case, I agree with you that it would help because it serves as a syllabus for what to learn. It is like, “Okay, these other things that if you know these things, it would help you a lot if you had to do anything with Kubernetes.” [0:31:14.6] NL: Yeah, I don’t think that you need to have a certification to do a job. I really don’t think so unless it is like required by law like you have to. [0:31:23.2] CC: No, yeah not at all what I am saying but if you don’t know anything at all and you’re like, “Where do I start?” I would recommend that. That is not a bad place to start or if you know some things but you feel like you don’t know others and you want to fill in the gaps and you don’t know what your gaps are, also same idea. What do you think Brian? Do you think having this certification would be useful? [0:31:46.5] BL: I don’t know, some people need it but I am also barely graduated from high school and I don’t have a college degree. So I have always leaned on myself for learning things on my own schedule, in my own pace on my own terms but some people do need the structure provided to them by certifications and I’ve only heard good things of people taking those tests. So I think for some people it is actually really good but for others, it might be a waste of time because what will actually happen if you get that certification? If you work at some large companies, I do know this for a fact by getting your AWS certificates actually had a money thing behind it but in a lot of places I don’t know but it couldn’t hurt. That is the most important piece. It can’t hurt. [0:32:36.3] NL: Yeah, I totally agree. You learn at least something even when I am taking a certification exam for something that I was already pretty aware of, I always learned at least one thing by taking like an examination. The last good question that you likely have never even thought off but I also agree with Brian where it is like I don’t have my CKA and I think I am a pretty damn good expert of Kubernetes. So I don’t think anything would change for me to take the exam. [0:33:00.2] CC: Oh yeah. I work with so many people who have none of those sort of certifications and they are absolutely experts. I was talking about like it would help me. I want to take those two certifications because it helped me fill in the gaps and I know there is a lot that I am going to learn especially with the admin one. So it is using the curriculum as a guide for what I need to learn and then testing, did I really learn and also it made me feel good but other than that, I don’t think it has any – I don’t know, I don’t think it is bad either. [0:33:33.4] BL: And that is the most important piece, what you just said, it made you feel good because you take certifications to test your knowledge against yourself in a lot of cases. So I think it is good. I just realized you can – I mean people cannot see behind me. I don’t think I have as many books as Carlisia’s up there but I have read all mine except for like four of them. [0:33:52.6] CC: Yeah, I did not read all of these books. I mean a lot of these books are school related books that I kept because they are really good and books that I have acquired and I have read ome but not all the entire book. Some things I use for reference but definitely have not read. Don’t be impressed, I have not read all of these books. Hopefully one day when I retire maybe. Anyway – [0:34:17.7] BL: I think that one interesting thing would be the amount of study that you need to do to gain a certification when you are not working in the space actually gives you that little bit of push that you need to make sure that you understand that you know what you need to know. So if you organically came to cloud native as I did, as I’ll explain in my story, you know I am not really interested in that certification. But if I wanted to change, and maybe I wanted to change my focus to doing more graphic stuff and there was a certification for this, maybe I would think about it just to make sure that I knew I was eligible for these jobs that I am trying for, so. [0:34:57.8] CC: Yeah. [0:34:58.8] NL: Yeah, that makes sense. Also, my books are over there and I have read most of the way through many of my books but not all the way through because a lot of them are boring. [0:35:09.5] BL: But I will say and since we are talking about books and talking about getting yourself into Kubernetes land, right now is actually the best time to buy books because there is lots of them and I am not actually saying that these books are super awesome but some of them are. Notably this Programming Kubernetes book is pretty awesome and the reason it is so awesome is because my quote is on the back of it. [0:35:33.3] NL: I was going to say. [0:35:34.4] BL: Yeah, my name is on the back of it and then another book that I just picked up lately is called The Kubernetes Cookbook and it is for building cloud native applications from O’Reilly and the reason that I like it is because I have always, since I mean 20 years ago, I love creating O’Reilly cookbooks because small problem, answer with an exclamation, and then there is another one called Kubernetes Patterns, which I just started and I think it is pretty good too. And just to say that these are not endorsements but this is what I am reading right now. It is like a thousand pages here. The things that I am trying to get through right now to keep up to date with what we are trying to do because actually the biggest problem with what we are doing is that we are trail blazing. So a lot of the things that are happening, like the way that Kubernetes advances every few months is new, new, new, new. So there is not a lot of higher art in what we are doing that is public. So what you need to do is turn yourself into someone who actually understands the theory of what we are doing rather than the practical application of it. Understand that piece too but you got to understand the theory, which is why I said I’ve literally doing the same thing for the last 25 years because I learned how to program and I learned a Unix and then I learned Linux and then I learned networking. Take all of those lessons and I can apply them all the time. So that is actually the most important part about any of this. [0:36:56.9] CC: Yeah, I agree with you like going through the fundamentals helps so much more than going through the specifics and in fact, trying to learn specifics without having fundamentals it can be very painful but then you try to learn the fundamentals and then you go, “Oh yeah, it totally makes sense. I have been trying to listen to YouTube lectures on the server systems and I have a lot of moments of, “Ah that is why Kubernetes works this way to address this problem.” And I have that programming book, which is not in my office. I have to find it but yes that is a very good book, I have this. [0:37:37.9] BL: Oh Cloud Native DevOps with Kubernetes. That is another good book. [0:37:43.5] CC: Yes. [0:37:44.3] BL: I have it too. [0:37:46.3] CC: I have like that as one? [0:37:47.6] BL: Yes. [0:37:48.2] CC: Good book and this, I haven’t gotten through it yet. [0:37:52.7] BL: It is called Kubernetes in Action. [0:37:54.3] CC: Yes, thank you for saying the name because if you are not in the video you wouldn’t know. [0:37:58.5] BL: So really what we are saying now – [0:37:59.7] CC: People say great things about the Kubernetes in Action, this one. [0:38:02.9] BL: So I actually want to bring up another thing and say, I read a lot. I like to read. I read a lot of blog posts and here is another crazy thing, the YouTube videos from like KubeCon every year or every few months, we publish a 180 talks for free and there is some good lessons in those. So the good thing about getting into cloud native is that you can get into it for cheap because all of this information out here Kubernetes source is free. Go read it. I mean 5,000 developers have worked on it. I am sure you will get a lot out of that, go do that but like YouTube talks, blog post, just following your favorite SIGK’s, Special Interest Group for Kubernetes, their community meetings. You can learn so much about how this space works and really how to write software in it without spending a dime other than have a computer and Internet. [0:38:55.5] CC: Yeah and I am going to give a tip for people that I actually caught on not too long ago. I subscribed to YouTube premium, which I think is $5 a month. It is the best $5 I have ever spent because really I don’t have time to sit in front of a video unless it is very special and just watch something and reading is also very – after I spend a whole day reading codes, my mind doesn’t want to read anything else. So I love podcasts and I listen to a lot of podcast. And now the YouTube videos are even have been more educational to me because the premium version of YouTube is if your phone locks it will still play. [0:39:40.2] BL: And you can download the videos. [0:39:42.0] CC: You can download the videos too. Yeah if you go on a camping trip or airplane you have them so it’s been fantastic. I just put my headset, my little Bluetooth headset and as I am doing laundry or as I’m cooking or anything, I am always listening to something. There goes the tip. [0:40:01.9] NL: Yeah, I totally agree. I love YouTube premium. No ads as Brian said is the best. I am going to throw out a book recommendation, one written by my colleague and a good friend, Craig Tracy, or co-written called Managing Kubernetes and it is actually like I was saying that these tech books are kind of boring, this one is actually a lot of fun to read. It is written well in a way that I found I kept turning at the page. So I really liked it. [0:40:26.3] BL: Yeah, it is only 150 pages too. [0:40:29.3] NL: Yeah that is pretty short. [0:40:30.5] BL: And the software that Carlisia writes is the last chapter of it, the next to last chapter so. [0:40:37.6] NL: Oh shoot, all right throw it out then. [0:40:40.4] BL: Well no, I am just saying it is another good book and I like the way you bring this up because this information is out there but I know were coming close to the end and I had one thing that I want to talk about today. [0:40:50.3] NL: I was just about to bring that up, please take us away. [0:40:52.3] BL: All right, so we talked about where we come from and we talked about things in the space about the jobs, how we keep up to date but really, the most important piece is what happens in the future. You know Kubernetes is only five years old so theoretically cloud native jobs are only a few years old. So how does cloud native move in the future and I do have some thoughts on this one. So what we are going to see is what we have seen over the last two decades is that our stacks will get more complex, we will run more apps, we will have more CPU’s and more networking and it is not even Morris Law stuff. We’ll just have more stuff. So what I find is that in the future, what we need to think about are things like automation. We need to think about better resilience. Apps that can actually take care of themselves. So your app goes down, what happens? Well nothing because it brought itself back up. So I see that the jobs that we have now are just going to evolve into better versions of what we have right now. So developers will still be developing. The more interesting piece is that we are going to have more developers because more people are taking these boot camp courses, more people are going into computer science in school. So we are actually going to have more developers out there. So all that means is that we are just going to have more problems to solve at least for the next few years. The generation from now, I couldn’t tell you what is going to happen. Maybe we will all be out of work. I will be retired so I probably won’t care but just think about this. Now is the literal best time to get into writing software and specifically for cloud native applications whether you are in operations or you are writing applications that run on clouds or anything like this. This is the best time because it is still beginning and there is more work to do than we have people and if you look through jobs postings you’ll realize that wow, everyone is looking for this. [0:42:48.3] CC: Yeah and at the same time, there is a sufficient amount of resources out there for you to learn even if you don’t want to – if you want to or you can’t pay. We now are so much at the beginning that there is nothing so it is a very good time. [0:43:04.6] NL: Yeah, the wealth of knowledge is out there that is for free is unheard of. It is unprecedented and yeah, I totally agree that this is the best time. Brian, if we go by your thesis throughout this entire episode, basically we are going to be doing the same thing in 20 years as we are doing now. It is the same thing we did 20 years ago. So it is probably going to be like you said, developers are going to develop-ate, sys admins are going to sys administrate. [0:43:28.6] CC: I love that. [0:43:30.1] BL: And security people are going to complain about everything. [0:43:33.4] NL: That is how we are going to change. So we are just going to be running on like quantum applications in 20 years but they are still going to be if/else statements. [0:43:41.1] CC: My prediction is that we are going to have greater server access, like easier server access, and especially developers and there will be more buttons to press and more visual tools so you don’t have to be necessarily logging into a server to command lines that we have more tools abstracting all of that detailed work that develops. [0:44:07.0] BL: So more abstractions on top of abstractions. [0:44:10.1] CC: Yeah that is my prediction. Why not? [0:44:13.3] BL: Well you know what? I mean if that is true because that is what we have been doing forever now so we are going to continue on doing this thing. [0:44:20.0] CC: Because it is what people want. [0:44:22.0] BL: Because it works. [0:44:23.0] CC: Yeah, it makes life easier for some people. I don’t see why we wouldn’t move in that direction but before we wrap up, unless you guys want to make predictions too, I really wanted to touch base on the hiring side of things. The recruiters and hiring managers before interviewing, I can’t imagine there is a whole bunch of people out there who need to recruit people to do these cloud native jobs and how can we help them? Like can we give them some tips? How can they attract people? What should they be looking for? [0:45:03.4] NL: Well, I guess my thought is that I really feel like recruiters need to start learning the technology that they are hiring for. I don’t think that they can hide behind the idea that they’re recruiters and they don’t need to know. If you want to hire good people, if you want to weed out the bad people or whatever it is that you are trying to do, you need to actually learn the technology that you are hiring for and I think like we are saying, there is now a wealth of knowledge that is free for you to access, please look. [0:45:32.9] CC: I am not going to disagree with that. [0:45:34.3] BL: And the interesting thing is when he says learn it, he doesn’t mean that you have to be able to produce it but you should understand how it works at the minimum. [0:45:42.8] NL: Yeah and also know when someone’s BS-ing you in the text screen. [0:45:48.2] CC: But it is not easy because you might be going in the direction with the intention of learning and you might misunderstand things and you know how deep do you have to go to not misunderstand the technology? [0:46:06.1] BL: You know what? I don’t think there is an answer for that. I think it is just you don’t know and there is something in between being an expert. You need to be something in between where if you’re hiring for cloud native in Kubernetes, you can’t offer a job that wants 10 years of Kubernetes experience. First of all, Kubernetes is huge and no one has all Kubernetes experience throughout the whole stack and second of all, Kubernetes is only five years old. So please don’t do that to yourself as well. So you should know how old it is and at least know the parts and what your team is going to be working on but for managers, wow, actually I don’t have a good answer for that. So I am just going to I’ll plan on that one. [0:46:45.1] CC: Well, how would it be different? Actually it is going to sound like I asked a loaded question but I just now had this realization. I don’t think it would be different from what we were saying in regards to giving tips for people to prepare themselves, to make a move into this space if they are not working with any of this stuff. It will be the same, like try to find people who know distributed systems, they can debug well. I am not even to go into working well with people. That is such a given. Let’s just keep it to the text stack and all of those things that we recommended for people to learn, I don’t know. [0:47:26.0] BL: Yeah, it sounds good to me. [0:47:28.1] NL: All right, well I think that just about wraps it up for this week of the podcast, the Kubelets Podcast. I thought this was a really interesting discussion. It was cool to talk about where we were and where we are going and you know, and what brought us all together as I said. [0:47:44.2] CC: Nick, do you want to share with us what your tagline for this episode was? [0:47:48.1] NL: Yeah, the tagline for this episode is CREAM: Cash Rules Everything Around Me. [0:47:53.3] BL: Dollar-dollar bills you all. [0:47:55.8] CC: Ka-ching, ka-ching, ka-ching [0:47:58.8] NL: All right, thank you so much. Thank you Brian, thanks for joining us. [0:48:03.9] BL: Thank you for having me. [0:48:05.3] NL: Yeah and thank you Carlisia. [0:48:07.6] CC: This was really good, thank you. [0:48:09.7] NL: Yeah, I had a lot of fun. Bye, y’all. [0:48:13.5] BL: Bye. [0:48:14.1] CC: Bye. [END OF EPISODE] [0:48:14.8] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

Today on The Podlets Podcast, we are joined by VMware's Vice President of Research and Development, Craig McLuckie! Craig is also a founder of Heptio, who were acquired by VMware and during his time at Google he was part of bringing Kubernetes into being. Craig has loads of expertise and shareable experience in the cloud native space and we have a fascinating chat with him, asking about his work, Heptio and of course, Kubernetes! Craig shares some insider perspective on the space, the rise of Kubernetes and how the increase in Kubernetes' popularity can be managed. We talk a lot about who can use Kubernetes and the prerequisites for implementation; Craig insists it is not a one-size-fits-all scenario. We also get into the lack of significantly qualified minds and how this is impacting competition in the hiring pool. Craig comments on taking part in the open source community and the buy-in that is required to meaningfully contribute as well as sharing his thoughts on the need to ship new products and services regularly. We finish off the episode with some of Craig's perspectives on the future of Kubernetes, dangers it poses to code if neglected and the next phase of its lifespan. For this amazing chat with a true expert in his field, make sure to join us on for this episode! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Special guest: Craig McLuckie Hosts: Carlisia Campos Duffie Cooley Josh Rosso Key Points From This Episode: • A brief introduction to Craig's history and his work in the cloud native space. • The questions that Craig believes more people should be asking about Kubernetes. • Weighing the explosion of the Kubernetes space; fragmentation versus progress. • The three pieces of enterprise software and aiming to enlarge the 'crystalline core'.• Craig's thoughts on specialized Kubernetes operating systems and their tradeoffs. • Quantifying the readiness of an organization to implement Kubernetes. • Craig's reflections on Heptio and the lessons he feels he learned in the process.• The skills shortage for Kubernetes and how companies are approaching this issue. • Balancing the needs and level of the community and shipping products regularly.• Involvement in the open source community and the leap of faith that is inherent in the process. • The question of microliths; making monoliths more complex and harder to manage. • Masking problems with Kubernetes and how detrimental this can be to your code. • Craig's thoughts on the future of the Kubernetes space and possible changes.• The two duty cycles of any technology; the readiness phase that follows the hype. Quotes: “I think Kubernetes has opened it up, not just in terms of the world of applications that can run Kubernetes, but also this burgeoning ecosystem of supporting technologies that can create value.” — @cmcluck [0:06:20] “You're not a cool mainstream enterprise software provider if you don’t have a Kubernetes story today. I think we’ll start to see continued focus and consolidation around a set of the larger organizations that are operating in this space.” — @cmcluck [0:06:39] “We are so much better served as a software company if we can preserve consistency from environment to environment.” — @cmcluck [0:09:12] “I’m a fan of rendered down, container-optimized operating system distributions. There’s a lot of utility there, but I think we also need to be practical and recognize that enterprises have gotten comfortable with the OS landscape that they have.” — @cmcluck [0:14:54] Links Mentioned in Today’s Episode: Craig McLuckie on LinkedIn Craig McLuckie on Twitter The Podlets on Twitter Kubernetes VMware Brendan Burns Cloud Native Computing Foundation Heptio Mesos Valero vSphere Red Hat IBM Microsoft Amazon KubeCon Transcript: EPISODE 13 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically-minded decision maker, this podcast is for you. [INTERVIEW] [00:00:41] CC: Hi, everybody. Welcome back to The Podlets podcast, and today we have a special guest, Craig McLuckie. Craig, I have the hardest time pronouncing your last name. You will correct me, but let me just quickly say, well, I’m Carlisia Campos and today we also have Duffy Colley and Josh Rosso on the show. Say that three times fast, Craig McLuckie. Please help us say your last name and give us a brief introduction. You are super well-known in the Kubernetes community and inside VMware, but I’m sure there are not enough people that should know about you that didn’t know about you. [00:01:20] CM: All right. I’ll do a very quick intro. Hi, I’m Craig McLuckie. I’m a Vice President of Research and Development here at VMware. Prior of VMware, I spent a fair amount of time at Google where my friend Joe and I were responsible for building and shipping Google Compute Engine, which was an interesting exercise in bringing traditional enterprise virtualized workloads into the very sophisticated Google data center. We then went ahead and as our next project with Brendan Burns, started Kubernetes, and that obviously worked out okay, and I was also responsible for the ideation and formation of the Cloud Native Computing Foundation. I then wanted to work with Joe again. So we started Heptio, a little startup in the Kubernetes ecosystem. Almost precisely a year ago, we were acquired by VMware. So I’m now part of the VMware company and I’m working on our broader strategy around cloud native apps under the brand [inaudible 00:02:10]. [00:02:11] CC: Let me start off with a question. I think it is going to be my go-to first question for every guest that we have in the show. Some people are really well-versed in the cloud native technologies and Kubernetes and some people are completely not. Some people are asking really good questions out there, and I try to too as I’m one of those people who are still learning. So my question for you is what do you think people are asking that they are not asking the right frame, that you wish they would be asking that question in a different way. [00:02:45] CM: It’s a very interesting question. I don’t think there’s any bad questions in the world, but one question I encountered a fair bit is, “Hey, I’ve heard about this Kubernetes thing and I want one.” I’m not sure it’s actually the right question, right? Kubernetes is a powerful technology. I definitely think we’re in this sort of peak hype phase of the project. There are a set of opportunities that Kubernetes really brings a much more robust ability to manage, it abstracts a way infrastructure — there are some very powerful things. But to be able to be really successful with Kubernetes project, there’re a number of additional ingredients that really need to be thought through. The questions that ought to be asked are, "I understand the utility of Kubernetes and I believe that it would bring value to my organization, but do I have the skills and capabilities necessary to stand up and run a successful Kubernetes program?" That’s something to really think about. It’s not just about the nature of the technology, but it really brings in a lot of new concepts that challenge organizations. If we think about applications that exist in Kubernetes, there’s challenges with observability. When you think the mechanics of delivering into a containerized sort of environment, there are a lot of dos and don’ts that make a ton of sense there. A lot of organizations I’ve worked with are excited about the technology, but they don’t necessarily have the depth of understanding of where it's best used and then how to operate it. The second addendum to that is, “Okay, I’m able to deploy Kubernetes, but what happens the next day? What happens if I need to update it? When I need to maintain it? What happens when I discover that I need not one Kubernetes cluster or even 10 Kubernetes clusters, but a hundred or a thousand or 10,000.” Which is what we are starting to see out there in the industry. “Have I taken the right first step on that journey to set me up for success in the long-term?” I do think there’s just a tremendous amount of opportunity and excitement around the technology, but also think it’s something that organizations really need to look at as not just about deploying a platform technology, but introducing the necessary skills that are necessary to operate and maintain it and the supporting technologies that are necessary to get the workloads on to it in a sustainable way. [00:04:42] JR: You’ve raised a number of assumptions around how people think about it I think, which are interesting. Even just starting with the idea of the packaging problem that represents containerization is a reasonable start. So infrequently, do we describe like the context of the problems that — all of the problems that Kubernetes solve that frequently I think people just get way ahead of themselves. It’s a pretty good description. [00:05:04] DC: So maybe in a similar vein, Craig, we had mentioned all the pieces that go into running Kubernetes successfully. You have to bolt some things on maybe for security or do some things to ensure observability as adequate, and it seems like the ecosystem has taken notice of all those needs and has built a million projects and products around that space. I’m curious of your thoughts on that because it’s like in one way it’s great because it shows it’s really healthy and thriving. In another way, it causes a lot of fragmentation and confusion for people who are thinking whether they can or cannot run Ku, because there are so many options out there to accomplish those kinds of things. So I was just curious of your general thoughts on that and where it’s headed. [00:05:43] CM: It’s fascinating to see the sort of burgeoning ecosystem around Kubernetes, and I think it’s heartening, because if you think at the very highest level, the world is going to go one of two ways with the introduction of the hyper-scale public cloud. It’s either going to lead us into a world which feels like mainframe era again, where no one ever got [inaudible 00:06:01] Amazon in this case, or by Microsoft, whatever the case. Whoever sort of merges over time as the dominant force. But it also represents some challenges where you have these vertically integrated closed systems, innovation becomes prohibitively difficult. It’s hard to innovate in a closed system, because you’re innovating only for organizations that have already taken that dependancy. I think Kubernetes has opened it up, not just in terms of the world of applications that can run Kubernetes, but also this burgeoning ecosystem of supporting technologies that can create value. There’s a reason why startups are building around Kubernetes. There’s a reason they’re looking to solve these problems. I do think we’ll see a continued period of consolidation. You're not a cool mainstream enterprise software provider if you don’t have a Kubernetes story today. I think we’ll start to see continued focus and consolidation around a set of the larger organizations that are operating in this space. It’s not accidental that Heptio is a part of VMware at this point. When I looked at the ecosystem, it was pretty clear we need to take a boat to fully materialize the value of Kubernetes and I am pleased to be part of this organization. So I do think you’ll start to see a variety of different vendors emerging with a pretty clear, well-defined opinions and relatively turnkey solutions that address the gamut of capabilities. One organization needs to get into Kubernetes. One of the things that delights me about Kubernetes is that if you are a sophisticated organization that is self-identifying as a software company, and this is sort of manifest in the internet space if you’re running a sort of hyper-scale internet service, you are kind of by definition a software company. You probably have the skills on hand to make great choices around what projects, follow the communities, identify when things are reaching point of critical mass. You’re running in a space where your system is relatively homogenous. You don’t have just the sort of massive gamut of workloads, a lot of dimension enterprise organizations have. There’s going to be different approaches to the ecosystem depending on which organization is looking at the problem space. I do think this is prohibitively challenging for a lot of organizations that are not resourced at the level of a hyper-scale internet company from a technology perspective, where their day job isn’t running a production service for millions or billions of users. I do think situations like that, it makes a tremendous amount of sense to identify and work with someone you trust in the ecosystem, that can help you just navigate the wild map that is the Kubernetes landscape, that can participate in a number of these emerging communities that has the ability to put their thumb on the scale where necessary to make sure that things converge. I think it’s situational. I think the lovely thing about Kubernetes is that it does give organizations a chance to cut their teeth without having to dig into like a deep procurement cyclewith a major vendor. We see a lot of self-service Kubernetes projects getting initiated. But at some point, almost inevitably, people need a little bit more help, and that’s the role of a lot of these vendors. I think that I truly hope that I’m personally committed to, is that as we start to see the convergence of this ecosystem, as we start to see the pieces falling into place, that we retain an emphasis on the value of community that we also sort of avoid the balkanization and fragmentation, which sometimes comes out of these types of systems. We are so much better served as a software company if we can preserve consistency from environment to environment. The reality is as we start looking at large organizations, enterprises that are consuming Kubernetes, it’s almost inevitable that they’re going to be consuming Kubernetes from a number of different sources. Whether the sources are cloud provider delivering Kubernetes services or whether they handle Kubernetes clusters that are dedicated centralized IT team is delivering or whether it’s vendor provided Kubernetes. There’s going to be a lot of different flavors and variants on it. I think working within the community not as king makers, but as concerned citizens that are looking to make sure that there are very high-levels of consistency from offering to offering, means that our customers are going to be better served. We’re right now in a time where this technology is burgeoning. It’s highly scrutinized, but it’s not necessarily very widely deployed. So I think it’s important to just keep an eye on that sort of community centricity. Stay as true to our stream as possible. Avoid balkanization, and I think everyone will benefit from that. [00:10:16] DC: Makes sense. One of the things I took away from my year, I was just looking kind of back at my year and learning, consolidating my thoughts on what had happened. One of the big takeaways for me in my customer engagements this year was that a number of customers outright came out explicitly and said, “Our success as a company is not going to be measured by our ability to operate Kubernetes, which is true and obvious.” But at the same time, I think that that’s a really interesting moment of awareness for a lot of the people that I work with out there in the field, where they realized, you know what, Kubernetes may be the next best thing. It may be an incredible technology, but fundamentally, it’s not going to be the measure by which we are graded success. It’s going to be what we do on top of that that is more interesting. So I think that your point about that ecosystem is large enough that people will be consuming Kubernetes for multiple searches is sort of amplified by that, because people are going to look for that easy button as inroad. They’re going to look for some way to get the Kubernetes thing so that they can actually start exploring what will happen on top of it as their primary goal rather than how to get Kubernetes from an operational perspective or even understand the care and feeding of it because they don’t see that as the primary measure of success. [00:11:33] CM: That is entirely true. When I think about enterprise software, there’s sort of these three pieces of it. The first piece is the sort of crystaline core of enterprise software. That’s consistent from enterprise to enterprise to enterprise. It’s purchased from primary vendors or it’s built by open source communities. It represents a significant basis for everything. There’s the sort of peripheral, the sort of sea of applications that exist around that enterprises built that are entirely unique to their environment, and they’re relatively fluid. Then there’s this weird sort of interstitial layer, which is the integration glue that exists between their crystalline core and those applications and operating practices that enterprises create. So I think from my side, we benefit if that crystalline core is as large as possible so that enterprises don’t have to rely on bespoke integration practices as much possible. We also need to make allowances for the idea that that interstitial layer between the sort of core of a technology like Kubernetes and the applications may be modular or sort of extended by a variety of different vendors. If you’re operating in this space, like the telco space, your problems are going to be unique to telco, but they’re going to be shared by every other telco provider. One of the beautiful things about Kubernetes is it is sufficiently modular, it is a pretty well-thought resistant. So I think we will start to see a lot of specialization in terms of those integration pieces. A lot of specialization in terms of how Kubernetes is fit to a specific area, and I think that represents an awful opportunity for the community to continue to evolve. But I also think it means that we as contributors to the project need to make allowances for that. We can’t hold opinion to the point where it precludes massive significant value for organizations as they look at modularized and extending the platform. [00:13:19] CC: What is your opinion on people making specialized Kubernetes operating systems? For example, we’re talking about telcos. I think there’s a Kubernetes OSS specifically for telcos that strip away things that kind of industry doesn’t need. What are the tradeoffs that you see? [00:13:39] CM: It’s almost inevitable that you’re going to start to see specialized operating system distributions that are tailored to container-based workloads. I think as we start looking at like the telco space with network function virtualization, Kubernetes promises to be something that we never really saw before. At the end of the day, telco is very broadly deployed open stack as this primary substrate for network function virtualization. But at the end of the day, they ended up not just deploying one rendition of open stack. But in many cases, three, four, five, depending on what functions they wanted to run, and there wasn’t a sufficient commonality in terms of the implementations. It became very sort of vendor-centric and balkanized in many ways. I think there’s an opportunity here to work hard as a community to drive convergence around a lot of those Kubernetes constructs so that, sure, the operating system is going to be different. If you’re running an NFV data plane implementation, doing a lot of bit slinging, it’s going to look fundamentally different to anything else in the industry, right? But that shouldn’t necessarily mean that you can’t use the same tools to organize, manage and reason about the workloads. A lot of the innovations that happen above that shouldn’t necessarily be tied to that. I think there’s promise there and it’s going to be an amazing test for Kubernetes itself to see how well it scales into those environments. By and large, I’m a fan of rendered down, container-optimized operating system distributions. There’s a lot of utility there, but I think we also need to be practical and recognize that enterprises have gotten comfortable with the OS landscape that they have. So we have to make allowances that as part of containerizing and distributing your application, maybe you don’t necessarily need to and hopefully re-qualify the underlying OS and challenge a lot of the assumptions. So I think we just need to pragmatic about it. [00:15:19] DC: I know that’s a dear topic to Josh and I. We’ve fought that battle in the past as well. I do think it’s another one of those things where it’s a set of assumptions. It’s fascinating to me how many different ecosystems are sort of collapsing, maybe not ecosystems. How many different audiences are brought together by a technology like container orchestration. That you are having that conversation with, “You know what? Let’s just change the paradigm for operating systems.” That you are having that conversation with, “Let’s change the paradigm for observability and lifecycle stuff. Let’s change the paradigm for packaging. We’ll call it containers.” You know what I mean? It’s so many big changes in one idea. It’s crazy. [00:15:54] CM: It’s a little daunting if you think about it, right? I always say, change is easiest across one dimension, right? If I’m going to change everything all at once across all the dimensions, life gets really hard. I think, again, it’s one of these things where Kubernetes represents a lot of value. I walk into a lot of customer accounts and I spend a lot of time with customers. I think based on their experiences, they sort of make one of two assumptions. There’s a set of vendors that will come into an environment and say, “Hey, just run this tool against your virtual machine images – and Kubernetes, right?” Then they have another set of vendors that will come in and say, “Yeah. Hey, you just need to go like turn this thing into 12 factor cloud native service mesh-linked applications driven through CICD, and your life is magic.” There are some cases where it makes sense, but there’re some cases where it just doesn’t. Hey, what uses a 24 gigabyte container? Is that really solving the problems that you have in some systematic way? At the other end of the spectrum, like there’s no world in which an enterprise organization is rewriting 3,000, 5,000 applications to be cloud native from the ground up. It just is not going to happen, right? So just understanding the return investment associated with the migration into Kubernetes. I’m not saying where it make sense and where it doesn’t. It’s such an important part of this story. [00:17:03] JR: On that front, and this is something Duffy and I talk to our customers about all the time. Say you’re sitting with someone and you’re talking about potentially using Kubernetes or they’re thinking about it, are there like some key indicators that you see, Craig, as like, “Okay. Maybe Kubernetes does have that return on investment pretty soon to justify it." Or maybe even in the reverse, like some things where you think, “Okay, these people are just going to implement Kubernetes and it’s going to become shelf weary.” How do you qualify as an org, “I might be ready to bring on something like Kubernetes.” [00:17:32] CM: It’s interesting. For me, it’s almost inevitably – as much about the human skills as anything else. I mean, the technology itself isn’t rocket science. I think the sort of critical success criteria, when I start looking at engagement, is there a cultural understanding of what Kubernetes represents? Kubernetes is not easy to use. That initial [inaudible 00:17:52] to the face is kind of painful for people that are used to different experiences. Making sure that the basic skills and expectations are met is really important. I think there’s definitely some sort of acid test around workloads fit as you start looking at Kubernetes. It’s an evolving ecosystem and it’s maturing pretty rapidly, but there are still areas that need a little bit more heavy lifting, right? So if you think about like, “Hey, I want to run a vertically-scaled OLTP database in Kubernetes today.” I don’t know. Maybe not the best choice. If the customer knows that, if they have enough familiarity or they’re willing to engage, I think it makes a tremendous amount of sense. By and large, the biggest challenge I see is not so much in the Kubernetes space. It’s easy enough to get to a basic cluster. There’re sort of two dimensions to this, there is day two operations. I see a lot of organizations that have worked to create scale up programs of platform technologies. Before Kubernetes there was Mesos and there’s obviously PCF that we’ll be coming more increasingly involved in. Organizations that have chewed on creating and deploying a standardized platform often have the operational skills, but you also need to look at like why did that previous technology really meet sort of criteria, and do you have the skills to operate it on a day two basis? Often there’s not – They’ve worked out the day two operational issues, but they still haven’t figured out like what it means to create a modern software supply chain that can deliver into the Kubernetes space. They haven’t figured out necessarily how to create the right incentive structures and experiences for the developers that are looking to build, package and deliver into that environment. That’s probably the biggest point of frustration I see with enterprises, is, “Okay. I got to Kubernetes. Now what?” That question just hasn’t been answered. They haven’t really thought through, “These are the CICD processes. This is how you engage your cyber team to qualify the platform for these classes of workloads. This is how you set up a container repo and run scans against it. This is how you assign TTL on images, so you don’t just get massive repo.” There’s so much in the application domain that just needs to exist that I think people often trivialize and it’s really taking the time and picking a couple of projects being measured in the investments. Making sure you have the right kind of cultural profile of teams that are engaged. Create that sort of celebratory moment of success. Make sure that the team is sort of metricking and communicating the productivity improvements, etc. That really drives the option and engagement with the whole customer base. [00:20:11] CC: It sounds to me like you have a book in the making. [00:20:13] CM: Oh! I will never write a book. It just seems like a lot of work. Brendan and a buch of my friends write books. Yeah, that seems like a whole lot of work. [00:20:22] DC: You had mentioned that you decided you wanted to work with Joe again. You formed Heptio. I was actually there for a year. I think I was around for a bit longer than that obviously. I’m curious what your thoughts about that were as an experiment win. If you just think about it as that part of the journey, do you think that was a success and what did you learn from that whole experiment that you wished everybody knew, just from a business perspective? It might have been business or it might have been running a company, any of that stuff. [00:20:45] CM: So I’m very happy with the way that Heptio went. There were a few things that sort of stood out for me as things that folks should think about if they’re going to start a startup or they want to join a startup. The first and foremost I would say is design the culture to the problem at hand. Culture isn’t accidental. I think that Heptio had a pretty distinct and nice culture, and I don’t want to sound self-congratulatory. I mean, as with anything, a certain amount of this is work, but a lot of it is luck as well. Making sure that the cultural identity of the company is well-suited to the problem at-hand. This is critical, right? When I think about what Heptio embodied, it was really tailored to the specific journey that we were setting ourselves up for. We were looking to be passionate advocates for Kubernetes. We were looking to walk the journey with our customers in an authentic way. We were looking to create a company that was built around sustainability. I think the culture is good and I encourage folks either the thing you’re starting is a startup or looking to join one, to think hard about that culture and how it’s going to map to the problems they’re trying to solve. The other thing that I think really motivated me to do Heptio, and I think this is something that I’m really excited to continue on with VMware, was the opportunity to walk the journey with customers. So many startups have this massive reticence to really engage deeply in professional services. In many ways, Google is fun. I had a blast there. It’s a great company to work for. We were able to build out some really cool tech and do good things. But I grew kind of tired of writing letters from the future. I was, “Okay, we are flying cars." When you're interacting with the customer. I can’t start my car and get to work. It’s great that you have flying cars, but right now I just need to get in my car, drive down the block and get out and get to work. So walking the journey with customers is probably the most important learning from Heptio and it’s one of the things I’m kind of most proud of. That opportunity to share the pain. Get involved from day one. Look at that as your most valuable apparatus to not just build your business, but also to learn what you need to build. Having a really smart set of people that are comfortable working directly with customers or invested in the success of those customers is so powerful. So if you’re in the business or in the startup game, investors may be leery of building out a significant professional service as a function, because that’s just how Silicon Valley works. But it is absolutely imperative in terms of your ability to engage with customers, particularly around nascent technologies, filled with gaps where the product doesn’t exist. Learn from those experiences and bring that back into the core product. It’s just a huge part of what we did. If I was ever in a situation where I had to advice a startup in the sort of open source space, I’d say lean into the professional service. Lean into field engineering. It’s a critical way to build your business. Learn what customers need. Walk the journey with them and just develop a deep empathy. [00:23:31] CC: With new technology, that was a concern about having enough professionals in the market who are knowledgeable in that new technology. There is always a gap for people to catch up with that. So I’m curious to know what customers or companies, prospective customers, how they are thinking in terms of finding professionals to help them? Are they’re concerned that there’s enough professionals in the market? Are they finding that the current people who are admins and operators are having an easy time because their skills are transferable, if they’re going to embark on the Kubernetes journey? What are they telling you? [00:24:13] CM: I mean, there’s a huge skills shortage. This is one of the kind of primary threats to the short term adoption of Kubernetes. I think Kubernetes will ultimately permeate enterprise organizations. I think it will become a standard for distributed systems development. Effectively emerging as an operating system for distributed systems, is people build more natively around Kubernetes. But right now it’s like the early days of Linux, where you deploy Linux, you’d have to kind of build it from scratch type of thing. It is definitely a challenge. For enterprise organizations, it’s interesting, because there’s a war for talent. There’s just this incredible appetite for Kubernetes talent. There’s always that old joke around the job description for like 10 years of Kubernetes experience on a five-year project. That certainly is something we see a lot. I’d take it from two sides. One is recognizing that as an enterprise organization, you are not going to be able to hire this talent. Just accept that sad truth. You can hire a seed crystal for it, but you really need to look at that as something that you’re going to build out as an enablement function for your own consumption. As you start assessing individuals that you’re going to bring on in that role, don’t just assess for Kubernetes talent. Assess for the ability to teach. Look for people that can come in and not just do, but teach and enable others to do it, right? Because at the end of the day, if you need like 50 Kubernauts at a certain level, so does your competitor and all of your other competitors. So does every other function out there. There’s just massive shortage of skills. So emphasizing your own – taking on the responsibility of building your own expertise. Educating your own organization. Finding ways to identify people that are motivated by this type of technology and creating space for them and recognizing and rewarding their work as they build this out. Because it’s far more practical to hire into existing skillset and then create space so that the people that have the appetite and capability to really absorb these types of disruptive technologies can do so within the parameters of your organization. Create the structures to support them and then make it their job to help permeate that knowledge and information into the organization. It’s just not something you can just bring in. The skills just don’t exist in the broader world. Then for professionals that are interested in Kubernetes, this is definitely a field that I think we’ll see a lot of job security for a very long-time. Taking on that effort, it’s just well worth the journey. Then I’d say the other piece of this is for vendors like VMware, our job can’t be just delivering skills and delivering technology. We need to think about our role as an enablers in the ecosystem as folks that are helping not just build up our own expertise of Kubernetes that we can represent to customers, but we’re well-served by our customers developing their own expertise. It’s not a threat to us. It actually enables them to consume the technologies that we provide. So focusing on that enablement through us as integration partners and [inaudible] community, focusing on enablement for our customers and education programs and the things that they need to start building out their capacity internally, is going to serve us all well. [00:27:22] JR: Something going back to maybe the Heptio conversation, I’m super interested in this. Being a very open source-oriented company, at VMware this is of course this true as well. We have to engage with large groups of humans from all different kinds of companies and we have to do that while building and shipping product to some degree. So where I’m going with this is like – I remember back in the Heptio days, there was something with dynamic audit logging that we were struggling with, and we needed it for some project we were working on. But we needed to get consensus in a designed approve at like a bigger community level. I do know to some degree that did limit our ability to ship quickly. So you probably know where I’m going with this. When you’re working on projects or products, how do you balance, making sure the whole community is coming along with you, but also making sure that you can actually ship something? [00:28:08] DM: That harkens back to that sort of catch phrase that Tim Sinclair always uses. If you want to go fast, go alone. If you want to go far, go together. I think as with almost everything in the world, these things are situational, right? There are situations where it is so critical that you bring the community along with you that you don’t find yourself carrying the load for something by yourself that you just have to accept and absorb that it’s going to be pushing string. Working with an engaged community necessitates consensus, necessitates buy-in not just from you, but from potentially your competitors. The people that you’re working with and recognizing that they’ll be doing their own sort of mental calculus around whether this advantages them or not and whatnot. But hopefully, I think certainly in the Kubernetes community, this is general recognition that making the underlying technology accessible. Making it ubiquitous, making it intrinsically supportable profits everyone. I think there’re a couple of things that I look at. Make the decision pretty early on as to whether this is something you want to kind of spark off and sort of stride off on your own an innovate around, whether it’s something that’s critical to bring the community along with you around. I’ll give you two examples of this, right? One example was the work we did around technologies like Valero, which is a backup restore product. It was an urgent and critical need to provide a sustainable way to back up and recover Kubernetes. So we didn’t have the time to do this through Kubernetes. But also it didn’t necessarily matter, because everything we’re doing was build this addendum to Kubernetes. That project created a lot of value and we’ve donated to open source project. Anyone can use it. But we took on the commitment to drive the development ourselves. It’s not just we need it to. Because we had to push very quickly in that space. Whereas if you look at the work that we’re doing around things like cluster API and the sort of broader provisioning of Kubernetes, it’s so important that the ecosystem avoids the tragedy of the commons around things like lifecycle management. It’s so important that we as a community converge on a consistent way to reason about the deployment upgrade and scaling of Kubernetes clusters. For any single vendor to try to do that by themselves, they’re going to take on the responsibility of dealing with not just one or two environments if you’re a hyperscale cloud provider [inaudible 00:30:27] many can do that. But we think about doing that for, in our case, “Hey, we only deploy into vSphere. Not just what’s coming next, but also earlier versions of vSphere. We need to be able to deploy into all of the hyper-scalers. We need to deploy into some of the emerging cloud providers. We need to start reasoning about edge. We need to start thinking about all of these. We’re a big company and we have a lot of engineers. But you’re going to get stretched very thin, very quickly if you try to chew that off by yourself. So I think a lot of it is situational. I think there are situations where it does pay for organizations to kind of innovate, charge off in a new direction. Run an experiment. See if it sticks. Over time, open that up to the community as it makes sense. The thing that I think is most important is that you just wear your heart on your sleeve. The worst thing you can do is to present a charter that, “Hey, we’re doing this as a community-centric, open project with open design, open community, open source,” and then change your mind later, because that just creates dramas. I think it’s situational. Pick the path that makes sense to the problem at-hand. Figure out how long your customer can wait for something. Sometimes you can bring things back to communities that are very open and accepting community. You can look at it as an experiment, and if it makes sense in that experiment perform factor, present it back to the Kubernetes communities and see if you can kind of get it back in. But in some case it just makes sense to work within the structure and constraints of the community and just accept that great things from a community angle take a lot of time. [00:31:51] CC: I think too, one additional thing that I don’t think was mentioned is that if a project grows too big, you can always break it off. I mean, Kubernetes is such a great example of that. Break it off into separate components. Break it off into separate governance groups, and then parts can move with different speeds. [00:32:09] CM: Yeah, and there’s all kinds of options. So the heart of it is no one rule, right? It’s entirely situational. What are you trying to accomplish on what arise and acknowledge and accept that the evolution of the core of Kubernetes is slowing as it should. That’s a signal that the project is maturing. You cannot deliver value at a longer timeline that your business or your customers can absorb then maybe it makes sense to do something on the outside. Just wear your heart on your sleeve and make sure your customers and your partners know what you’re doing. [00:32:36] DC: One of your earlier points about how do companies – I think Josh's question and was around how do companies attract talent. You’re basically pointing, and I think that there are some relation to this particular topic because, frequently, I’ve seen companies find some success by making room for open source or upstream engineers to focus on the Kubernetes piece and to help drive that adoption internally. So if you’re going to adopt something like a Kubernetes strategy as part of a larger company goal, if you can actually make room within your organization to bring people who are – or to support people who want to focus on that up stream, I think that you get a lot of ancillary benefits from that, including it makes it easier to adopt that technology and understand it and actually have some more skin in the game around where the open source project itself is going. [00:33:25] CM: Yeah, absolutely. I think one of the lovely things about the Kubernetes community is this idea of your position is earned, not granted, right? The way that you earn influence and leadership and basically the good will of everyone else in that community is by chopping wood, carrying water. Doing the things that are good for the community. Over time, any organization, any human being can become influential and lead based on their merits of their contributions. It’s important that vendors think about that. But at the same time, I have a hard time taking exception with practically any use of open source. At the end of the day, open source by its nature is a leap of faith. You’re making that technology accessible. If someone else can take it, operationalize it well and deliver value for organizations, that’s part of your contract. That’s what you absorb as a vendor when you start the thing. So people shouldn’t feel like they have to. But if you want to influence and lead, you do need to. Participate in these communities in an open way. [00:34:22] DC: When you were helping form the CNCF and some of those projects, did you foresee it being like a driving goal for people, not just vendors, but also like consumers of the technologies associated with those foundations? [00:34:34] CM: Yeah, it was interesting. Starting the CNCF, I can speak from the position of where I was inside Google. I was highly motivated by the success of Kubernetes. Not just personally motivated, because it was a project that I was working on. I was motivated to see it emerge as a standard for distributed systems development that attracts the way the infrastructure provider. I’m not ashamed of it. It was entirely self-serving. If you looked at Google’s market position at that time, if you looked at where we were as a hyper-scale cloud provider. Instituting something that enabled the intrinsic mobility of workloads and could shuffle around the cards on the deck so to speak [inaudible 00:35:09]. I also felt very privileged that that was our position, because we didn’t necessarily have to create artificial structures or constraints around the controls of the system, because that process of getting something to become ubiquitous, there’s a natural path if you approach it as a single provider. I’m not saying who couldn’t have succeeded with Kubernetes as a single provider. But if Red Hat and IBM and Microsoft and Amazon had all piled on to something else, it’s less obvious, right? It’s less obvious that Kubernetes would have gone as far as it did. So I was setting up CNCF, I was highly motivated by preserving the neutrality. Creating structures that separated the various sort of forms of governance. I always joke that at the time of creating CNCF, I was motivated by the way the U.S. Constitution is structured. Where you have these sort of different checks and balances. So I wanted to have something that would separate vendor interests from things that are maintaining taste on the discreet project. The sort of architecture integrity, and maintain separation from customer segments, so that you’d create the sort of natural self-balancing system. It was definitely in my thinking, and I think it worked out pretty well. Certainly not perfect, but it did lead down a path which I think has supported the success of the project a fair bit. [00:36:26] DC: So we talked a lot about Kubernetes. I’m curious, do you have some thoughts, Carlisia? [00:36:31] CC: Actually, I know you have a question about microliths. I was very interested in exploring that. [00:36:37] CM: There’s an interesting pattern that I see out there in the industry and this manifests in a lot of different ways, right? When you think about the process of bringing applications and workloads into Kubernetes, there’s this sort of pre-dispositional bias towards, “Hey, I’ve got this monolithic application. It’s vertically scaled. I’m having a hard time with the sort of team structure. So I’m going to start tuning it up into a set of microservices that I can then manage discretely and ideally evolve on a separate cadence. This is an example of a real customer situation where someone said, “Hey, I’ve just broken this monolith down into 27 microservices.” So I was sort of asking a couple of questions. The first one was when you have to update those 27 – if you want to update one of those, how many do you have to touch? The answer was 27. I was like, “Ha! You just created a microlith.” It’s like a monolith, except it’s just harder to live with. You’re taking a packaging problem and turn it into a massively complicated orchestration problem. I always use that jokingly, but there’s something real there, which is there’s a lot of secondary things you need to think through as you start progressing on this cloud native journey. In the case of microservice development, it’s one thing to have API separated microservices. That’s easy enough to institute. But instituting the organization controls around an API versioning strategy such you can start to establish stable API with consistent schema and being able to sort of manage the dependencies to consuming teams requires a level of sophistication that a lot of organizations haven’t necessarily thought through. So it’s very easy to just sort of get caught up in the hype without necessarily thinking through what happens downstream. It’s funny. I see the same thing in functions, right? I interact with organizations and they’re like, “Wow! We took this thing that was running in a container and we turned it into 15 different functions.” I’m like, “Ha! Okay.” You start asking questions like, “Well, do you have any challenges with state coherency?” They’re like, “Yeah! It’s funny you say that. Because these things are a little bit less transactionally coherent, we have to write state watches. So we try and sort of watermark state and watch this thing." I’m like, “You’re building a distributed transaction coordinator on your free time. Is this really the best use of your resources?" Right? So it really gets back to that idea that there’s a different tool for a different job. Sometimes the tool is a virtual machine. Sometimes it’s not. Sometimes the tool is a bare metal deployment. If you’re building a quantitative trading application that’s microsecond latency sensitive, you probably don’t want to hypervisor there. Sometimes a VM is the natural destination and there’s no reason to move from a VM. Sometimes it’s a container. Sometimes you want to start looking at that container and just modularizing it so you can run a set of things next to each other in the same process space. Sometimes you’re going to want to put APIs between those things and separate them out into separate containers. There’s an ROI. There’s a cause and there’s a benefit associated with each of those transitions. More importantly, there are a set of skills that you have to have as you start looking at their continuum and making sure that you’re making good choices and being wise about it. [00:39:36] CC: That is a very good observation. Design is such an important part of software development. I wonder if Kubernetes helps mask these design problems. For example, the ones you are mentioning, or does Kubernetes sort of surfaces them even more? [00:39:53] CM: It’s an interesting philosophical question. Kubernetes certainly masks some problems. I ran into an early – this is like years ago. I ran into an early customer, who confided in me, "I think we’re writing worse code now." I was like, ”What do you mean?” He was like, “Well, it used to be when we went out of memory on something, we get paged. Now we’ve set out that we go and it just restarts the container and everything continuous.” There’s no real incentive for the engineers to actually go back and deal with the underlying issues and recourse it, because the system is just more intrinsically robust and self-healing by nature. I think there's definitely some problems that Kubernetes will compound. If you’re very sloppy with your dependencies, if you create a really large, vertically scaled monolith that’s running at VM today, putting it in a container is probably strictly going to make your life worse. Just be respectful of that. But at the same time, I do think that the discipline associated with transition to Kubernetes, if you walk it a little bit further along. If you start thinking about the fact that you’re not running a lot of imperative processes during a production in a push, where deployment container is effectively a bin copy with some minimal post-deployment configuration changes that happen. It sort of leads you on to a much happier path naturally. I think it can mask some issues, but by and large, the types of systems you end up building are going to be more intrinsically operationally stable and scalable. But it is also worth recognizing that it’s — you are going to encounter corner cases. I’ve run into a lot of customers that will push the envelope in a direction that was unanticipated by the community or they accidentally find themselves on new ground that’s just unstable, because the technology is relatively nascent. So just recognizing that if you’re going to walk down a new path, I’m not saying don’t, just recognize that you’re probably going to encounter some stuff that’s going to take over to working through. [00:41:41] DC: We get an earlier episode about API contracts, which I think highlights some of these stuff as well, because it sort of gets into some of those sharp edges of like why some of those things are super important when you start thinking about microservices and stuff. We’re coming to the end of our time, but one of the last questions I want to ask you, we’ve talked a lot about Kubernetes in this episode, I’m curious what the future holds. We see a lot of really interesting things happening in the ecosystem around moving more towards serverless. There are a lot of people who are like — thinking that perhaps a better line would be to move away from like infrastructure offering and just basically allow cloud providers in this stuff to manage your nodes for you. We have a few shots on goal for that ourselves. It’s been really an interesting evolution over the last year in that space. I’m curious, what sort of lifetime would you ascribe to it today? What do you think that this is going to be the thing in 10 years? Do you think it will be a thing in 5 years? What do you see coming that might change it? [00:42:32] CM: It’s interesting. Well, first of all, I think 2018 was the largest year ever for mainframe sales. So we have these technologies, once they’re in enterprise, it tends to be pretty durable. The duty cycle of enterprise software technology is pretty long-lived. The real question is we’ve seen a lot of technologies in this space emerge, ascend, reach a point of critical mass and then fade and they’re disrupted by the technologies. Is Kubernetes going to be a Linux or is Kubernetes going to be a Mesos, right? I mean, I don’t claim to know the answer. My belief, and I think this is probably true, is that it’s more like a Linux. When you think about the heart of what Kubernetes is doing, is it’s just providing a better way to build and organized distributed systems. I’m sure that the code will evolve rapidly and I’m sure there will be a lot of continued innovation enhancement. But when you start thinking about the fact that what Kubernetes has really done is brought controller reconciler based management to distributed systems developed everywhere. When you think about the fact that pretty much every system these days is distributed by nature, it really needs something that supports that model. So I think we will see Kubernetes sticking. We’ll see it become richer. We’ll start to see it becoming more applicable for a lot of things that we’re starting to just running in VMs. It may well continue to run in VMs and just be managed by Kubernetes. I don’t have an opinion about how to reason about the underlying OS and virtualization structure. The thing I do have opinion about is it makes a ton of sense to be able to use a declarative framework. Use a set of well-structured controllers and reconcilers to drive your world into a non-desired state. I think that pattern will be – it’s been quite successful. It can be quite durable. I think we’ll start to see organizations embrace a lot of these technologies over time. It is possible that something brighter, shinier, newer, comes along. Anyone will tell you that we made enough mistakes during the journey and there is stuff that I think everyone regret some of the Kubernetes train. I do think it’s likely to be pretty durable. I don’t think it’s a silver bullet. Nothing is, right? It’s like any of these technologies, there’s always the cost and there’s a benefit associated with it. The benefits are relatively well-understood. But there’s going to be different tools to do different jobs. There’s going to be new patterns that emerge that simplify things. Is Kubernetes the best framework for running functions? I don’t know. Maybe. Kind of like what the [inaudible] people are doing. But are there more intrinsically optimal ways to do this, maybe. I don’t know. [00:45:02] JR: It has been interesting watching Kubernetes itself evolve in that moving target. Some of the other technologies I’ve seen kind of stagnate on their one solution and don’t grow further. But that’s definitely not what I see within this community. It’s like always coming up with something new. Anyway, thank you very much for your time. That was an incredible session. [00:45:22] CM: Yeah. Thank you. It’s always fun to chat. [00:45:24] CC: Yeah. We’ll definitely have you back, Craig. Yes, we are coming up at the end, but I do want to ask if you have any thoughts that you haven’t brought up or we haven’t brought up that you’d like to share with the audience of this podcast. [00:45:39] CM: I guess the one thing that was going through my head earlier I didn’t say which is as you look at these technologies, there’s sort of these two duty cycles. There’s the hype duty cycle, where technology ascends in awareness and everyone looks at it as an answer to all the everythings. Then there’s the readiness duty cycle, which is sometimes offset. I do think we’re certainly peak hype right now in Kubernetes if you attended KubeCon. I do think there’s perhaps a gap between the promise and the reality for a lot of organizations. It's always just council caution and just be judicious about how you approach this. It’s a very powerful technology and I see a very bright future for it. Thanks for your time. [00:46:17] CC: Really, thank you so much. It’s so refreshing to hear from you. You have great thoughts. With that, thank you very much. We will see you next week. [00:46:28] JR: Thanks, everybody. See you. [00:46:29] DC: Cheers, folks. [END OF INTERVIEW] [00:46:31] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

In this episode of The Podlets Podcast, we welcome Michael Gasch from VMware to join our discussion on the necessity (or not) of formal education in working in the realm of distributed systems. There is a common belief that studying computer science is a must if you want to enter this field, but today we talk about the various ways in which individuals can teach themselves everything they need to know. What we establish, however, is that you need a good dose of curiosity and craziness to find your feet in this world, and we discuss the many different pathways you can take to fully equip yourself. Long gone are the days when you needed a degree from a prestigious school: we give you our hit-list of top resources that will go a long way in helping you succeed in this industry. Whether you are someone who prefers learning by reading, attending Meetups or listening to podcasts, this episode will provide you with lots of new perspectives on learning about distributed systems. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Duffie Cooley Michael Gasch Key Points From This Episode: • Introducing our new host, Michael Gasch, and a brief overview of his role at VMware. • Duffie and Carlisia’s educational backgrounds and the value of hands-on work experience. • How they first got introduced to distributed systems and the confusion around what it involves. • Why distributed systems are about more than simply streamlining communication and making things work. • The importance and benefit of educating oneself on the fundamentals of this topic. • Our top recommended resources for learning about distributed systems and their concepts. • The practical downside of not having a formal education in software development. • The different ways in which people learn, index and approach problem-solving. • Ensuring that you balance reading with implementation and practical experience. • Why it’s important to expose yourself to discussions on the topic you want to learn about. • The value of getting different perspectives around ideas that you think you understand. • How systems thinking is applicable to things outside of computer science.• The various factors that influence how we build systems. Quotes: “When people are interacting with distributed systems today, or if I were to ask like 50 people what a distributed system is, I would probably get 50 different answers.” — @mauilion [0:14:43] “Try to expose yourself to the words, because our brains are amazing. Once you get exposure, it’s like your brain works in the background. All of a sudden, you go, ‘Oh, yeah! I know this word.’” — @carlisia [0:14:43] “If you’re just curious a little bit and maybe a little bit crazy, you can totally get down the rabbit hole in distributed systems and get totally excited about it. There’s no need for having formal education and the degree to enter this world.” — @embano1 [0:44:08] Learning resources suggested by the hosts: Book, Designing Data-Intensive Applications, M. Kleppmann Book, Distributed Systems, M. van Steen and A.S. Tanenbaum (free with registration) Book, Thesis on Raft, D. Ongaro. - Consensus - Bridging Theory and Practice (free PDF) Book, Enterprise Integration Patterns, B.Woolf, G. Hohpe Book, Designing Distributed Systems, B. Burns (free with registration) Video, Distributed Systems Video, Architecting Distributed Cloud Applications Video, Distributed Algorithms Video, Operating System - IIT Lectures Video, Intro to Database Systems (Fall 2018) Video, Advanced Database Systems (Spring 2018) Paper, Time, Clocks, and the Ordering of Events in a Distributed System Post, Notes on Distributed Systems for Young Bloods Post, Distributed Systems for Fun and Profit Post, On Time Post, Distributed Systems @The Morning Paper Post, Distributed Systems @Brave New Geek Post, Aphyr’s Class materials for a distributed systems lecture series Post, The Log - What every software engineer should know about real-time data’s unifying abstraction Post, Github - awesome-distributed-systems Post, Your Coffee Shop Doesn’t Use Two-Phase Commit Podcast, Distributed Systems Engineering with Apache Kafka ft. Jason Gustafson Podcast, The Systems Bible - The Beginner’s Guide to Systems Large and Small - John Gall Podcast, Systems Programming - Designing and Developing Distributed Applications - Richard Anthony Podcast, Distributed Systems - Design Concepts - Sunil Kumar Links Mentioned in Today’s Episode: The Podlets on Twitter — https://twitter.com/thepodlets Michael Gasch on LinkedIn — https://de.linkedin.com/in/michael-gasch-10603298 Michael Gasch on Twitter — https://twitter.com/embano1 Carlisia Campos on LinkedIn — https://www.linkedin.com/in/carlisia Duffie Cooley on LinkedIn — https://www.linkedin.com/in/mauilion VMware — https://www.vmware.com/ Kubernetes — https://kubernetes.io/ Linux — https://www.linux.org Brian Grant on LinkedIn — https://www.linkedin.com/in/bgrant0607 Kafka — https://kafka.apache.org/ Lamport Article — https://lamport.azurewebsites.net/pubs/time-clocks.pdf Designing Date-Intensive Applications — https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable-ebook/dp/B06XPJML5D Designing Distributed Systems — https://www.amazon.com/Designing-Distributed-Systems-Patterns-Paradigms/dp/1491983647 Papers We Love Meetup — https://www.meetup.com/papers-we-love/ The Systems Bible — https://www.amazon.com/Systems-Bible-Beginners-Guide-Large/dp/0961825170 Enterprise Integration Patterns — https://www.amazon.com/Enterprise-Integration-Patterns-Designing-Deploying/dp/0321200683 Transcript: EPISODE 12 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [00:00:41] CC: Hi, everybody. Welcome back. This is Episode 12, and we are going to talk about distributed systems without a degree or even with a degree, because who knows how much we learn in university. I am Carlisia Campos, one of your hosts. Today, I also have Duffie Cooley. Say hi, Duffie. [00:01:02] DC: Hey, everybody. [00:01:03] CC: And a new host for you, and this is such a treat. Michael Gasch, please tell us a little bit of your background. [00:01:11] MG: Hey! Hey, everyone! Thanks, Carlisia. Yes. So I’m new to the show. I just want to keep it brief because I think over the show we’ll discuss our backgrounds a little bit further. So right now, I’m with VMware. So I’ve been with VMware almost for five years. Currently, I'm in the office of the CTO. I’m a platform architect in the office of the CTO and I mainly use Kubernetes on a daily basis from an engineering perspective. So we build a lot of prototypes based on customer input or ideas that we have, and we work with different engineering teams. Kurbernetes has become kind of my bread and butter but lately more from a consumer perspective like developing with Kurbenetes or against Kubernetes, instead of the formal ware of mostly being around implementing and architecting Kubernetes. [00:01:55] CC: Nice. Very impressive. Duffie? [00:01:58] MG: Thank you. [00:01:59] DC: Yeah. [00:02:00] CC: Let’s give the audience a little bit of your backgrounds. We’ve done this before but just to frame the episodes, so people will know how we come in as distributed systems. [00:02:13] DC: Sure. In my experience, I spent – I don’t have a formal education history. I spent most of my time kind of like in a high school time. Then from there, basically worked into different systems administration, network administration, network architect, and up into virtualization and now containerization. So I’ve got a pretty hands-on kind of bootstrap experience around managing infrastructure, both at small-scale, inside of offices, and the way up to very large scale, working for some of the larger companies here in the Silicon Valley. [00:02:46] CC: All right. My turn I guess. So I do have a computer science degree but I don’t feel that I really went deep at all in distributed systems. My degree is also from a long time ago. So mainly, what I do know now is almost entirely from hands-on work experience. Even so, I think I'm very much lacking and I’m very interested in this episode, because we are going to go through some great resources that I am also going to check out later. So let’s get this party started. [00:03:22] DC: Awesome. So you want to just talk about kind of the general ideas behind distributed systems and like how you became introduced to them or like where you started in that journey? [00:03:32] CC: Yeah. Let’s do that. [00:03:35] DC: My first experience with the idea of distributed systems was in using them before I knew that they were distributed systems, right? One of the very first distributed systems as I look back on it that I ever actually spent any real time with was DNS, which I consider to be something of a distributed system. If you think about it, they have name servers, they have a bunch of caching servers. They solve many of the same sorts of problems. In a previous episode, we talked about how networking, just the general idea of networking and handling large-scale architecting networks. It’s also in a way very – has a lot of analogues into distributed systems. For me, I think working with and helping solve the problems that are associated with them over time gave me a good foundational understanding for when we were doing distributed systems as a thing later on in my career. [00:04:25] CC: You said something that caught my interest, and it’s very interesting, because obviously for people who have been writing algorithms, writing papers about distributed systems, they’re going to go yawning right now, because I’m going to say the obvious. As you start your journey programming, you read job requirements. You read or you must – should know distributed systems. Then I go, “What is distributed system? What do they really mean?” Because, yes, we understand apps stuck to apps and then there is API, but there’s always for me at least a question at the back of my head. Is that all there is to it? It sounds like it should be a lot more involved and complex and complicated than just having an app stuck on another app. In fact, it is because there are so many concepts and problems involved in distributed systems, right? From timing, clock, and sequence, and networking, and failures, how do you recover. There is a whole world in how do you log this properly, how do you monitor. There’s a whole world that revolves around this concept of systems residing in different places and [inaudible 00:05:34] each other. [00:05:37] DC: I think you made a very good point. I think this is sort of like there’s an analog to this in containers, oddly enough. When people say, “I want a container within and then the orchestration systems,” they think that that's just a thing that you can ask for. That you get a container and inside of that is going to be your file system and it’s going to do all those things. In a way, I feel like that same confusion is definitely related to distributed systems. When people are interacting with distributed systems today or if I were to ask like 50 people what a distributed system is, I would probably get 50 different answers. I think that you got a pretty concise definition there in that it is a set of systems that intercommunicate to perform some function. It’s like found at its base line. I feel like that's a pretty reasonable definition of what distributed systems are, and then we can figure out from there like what functions are they trying to achieve and what are some of the problems that we’re trying to solve with them. [00:06:29] CC: Yeah. That’s what it’s all about in my head is solving the problems because at the beginning, I was thinking, “Well, it must be just about communicating and making things work.” It’s the opposite of that. It’s like that’s a given. When a job says you need to understand about distributed systems, what they are really saying is you need to know how to deal with failures, not just to make it work. Make it work is sort of the easy part, but the whole world of where the failures can happen, how do you handle it, and that, to me is what needing to know distributed system comes in handy. In a couple different things, like at the top layer or 5% is knowing how to make things work, and 95% is knowing how to handle things when they don’t work, because it’s inevitable. [00:07:19] DC: Yeah, I agree. What do you think, Michael? How would you describe the context around distributed systems? What was the first one that you worked with? [00:07:27] MG: Exactly. It’s kind of similar to your background, Duffie, which is no formal degree or education on computer science right after high school and jumping into kind of my first job, working with computers, computer administration. I must say that from the age of I think seven or so, I was interested in computers and all that stuff but more from a hardware perspective, less from a software development perspective. So my take always was on disassembling the pieces and building my own computers than writing programs. In the early days, that just was me. So I completely almost missed the whole education and principles and fundamentals of how you would write a program for a single computer and then obviously also for how to write programs that run across a network of computers. So over time, as I progress on my career, especially kind of in the first job, which was like seven years of different Linux systems, Linux administrations, I kind of – Like you, Duffie, I dealt with distributed systems without necessarily knowing that I'm dealing with distributed systems. I knew that it was mostly storage systems, Linux file servers, but distributed file servers. Samba, if some of you recall that project. So I knew that things could fail. I know it could fail, for example, or I know it could not be writable, and so a client must be stuck but not necessarily I think directly related to fundamentals of how distributed systems work or don’t work. Over time, and this is really why I appreciate the Kubernetes project in community, I got more questions, especially when this whole container movement came up. I got so many questions around how does that thing work. How does scheduling work? Because scheduling kind of was close to my interest in the hardware design and low-level details. But I was looking at Kubernetes like, “Okay. There is the scheduler.” In the beginning, the documentation was pretty scarce around the implementation and all the control as for what’s going on. So I had to – I listen to a lot of podcasts and Brian Grant’s great talks and different shows that he gave from the Kubernetes space and other people there as well. In the end, I had more questions than answers. So I had to dig deeper. Eventually, that led me to a path of wanting to understand more formal theory behind distributed systems by reading the papers, reading books, taking some online classes just to get a basic understanding of those issues. So I got interested in results scheduling in distributed systems and consensus. So those were two areas that kind of caught my eyes like, “What is it? How do machines agree in a distributed system if so many things can go wrong?” Maybe we can explore this later on. So I’m going to park this for a bit. But back to your question, which was kind of a long-winded answer or a road to answering your question, Duffie. For me, a distributed system is like this kind of coherent network of computer machines that from the outside to an end-user or to another client looks like one gigantic big machine that is [inaudible 00:10:31] to run as fast. That is performing also efficient. It constitutes a lot of characteristics and properties that we want from our systems that a single machine usually can’t handle. But it looks like it's a big single machine to a client. [00:10:46] DC: I think that – I mean, it is interesting like, I don’t want to get into – I guess this is probably not just a distributed systems talk. But obviously, one of the questions that falls out for me when I hear that answer is then what is the difference between a micro service architecture and distributed systems, because I think it's – I mean, to your point, the way that a lot of people work with the app to learn to develop software, it’s like we’re going to develop a monolithic application just by nature. We’re going to solve a software problem using code. Then later on, when we decide to actually scale this thing or understand how to better operate it under a significant load, then we started thinking about, “Okay. Well, how do we have to architect this differently in such a way that it can support that load?” That’s where I feel like the beams cut across, right? We’re suddenly in a world where you’re not only just talking about microservices. You’re also talking about distributed systems because you’re going to start thinking about how to understand transactionality throughout that system, how to understand all of those consensus things that you're referring to. How do they affect it when I add mister network in there? That’s cool. [00:11:55] MG: Just one comment on this, Duffie, which took me a very long time to realize, which is coming – From my definition of what a distributed system is like this group of machines that they perform work in a certain sense or maybe even more abstracted like at a bunch of computers network together. What I kind of missed most of the time, and this goes back to the DNS example that you gave in the beginning, was the client or the clients are also part of this distributed system, because they might have caches, especially in DNS. So you always deal with this kind of state that is distributed everywhere. Maybe you don't even know where it kind of is distributed, and the client kind of works with a local stale data. So that is also part of a distributed system, and something I want to give credit to the Kafka community and some of the engineers on Kafka, because there was a great talk lately that I heard. It’s like, “Right. The client is also part of your distributed system, even though usually we think it's just the server. That those many server machines, all those microservices.” At least I missed that a long time. [00:12:58] DC: You should put a link to that talk in our [inaudible 00:13:00]. That would be awesome. It sounds great. So what do you think, Carlisia? [00:13:08] CC: Well, one thing that I wanted to mention is that Michael was saying how he’s been self-teaching distributed systems, and I think if we want to be competent in the area, we have to do that. I’m saying this to myself even. It’s very refreshing when you read a book or you read a paper and you really understand the fundamentals of an aspect of distributed system. A lot of things fall into place in your hands. I’m saying this because even prioritizing reading about and learning about the fundamentals is really hard for me, because you have your life. You have things to do. You have the minutiae in things to get done. But so many times, I struggle. In the rare occasions where I go, “Okay. Let me just learn this stuff trial and error,” it makes such a difference. Then once you learn, it stays with you forever. So it’s really good. It’s so refreshing to read a paper and understand things at a different level, and that is what this episode is. I don’t know if this is the time to jump in into, “So there are our recommendations.” I don't know how deep, Michael, you’re going to go. You have a ton of things listed. Everything we mention on the show is going to be on our website, on the show notes. So nobody needs to be necessarily taking notes. Anything thing I wanted to say is it would be lovely if people would get back to us once you listened to this. Let us know if you want to add anything to this list. It would be awesome. We can even add it to this list later and give a shout out to you. So it’d be great. [00:14:53] MG: Right. I don’t want to cover this whole list. I just wanted to be as complete as possible about a stuff that I kind of read or watched. So I just put it in and I just picked some highlights there if you want. [00:15:05] CC: Yeah. Go for it. [00:15:06] MG: Yeah. Okay. Perfect. Honestly, even though not the first in the list, but the first thing that I read, so maybe from kind of my history of how I approach things, was searching for how do computers work and what are some of the issues and how do computers and machines agree. Obviously, the classic paper that I read was the Lamport paper on “Time, Clocks, and the Ordering of Events in a Distributed System”. I want to be honest. First time I read it, I didn’t really get the full essence of the paper, because it doesn't prove in there. The mathematic proof for me didn't click immediately, and there were so many things and concepts and physics and time that were thrown at me where I was looking for answers and I had more questions than answers. But this is not to Leslie. This is more like by the time I just wasn't prepared for how deep the rabbit hole goes. So I thought, if someone asked me for – I only have time to read one book out of this huge list that I have there and all the other resources. Which one would it be? Which one would I recommend? I would recommend Designing Data-Intensive Apps by Martin Kleppmann, which I’ve been following his blog posts and some partial releases that he's done before fully releasing that book, which took him more than four years to release that book. It’s kind of almost the Bible, state-of-the-art Bible when it comes to all concepts in distributed systems. Obviously, consensus, network failures, and all that stuff but then also leading into modern data streaming, data platform architectures inspired by, for example, LinkedIn and other communities. So that would be the book that I would recommend to someone if – Who does have time to read one book. [00:16:52] DC: That’s a neat approach. I like the idea of like if you had one thing, if you have one way to help somebody ramp on distributed systems and stuff, what would it be? For me, it’s actually I don't think I would recommend a book, oddly enough. I feel like I would actually – I’d probably drive them toward the kind of project, like the kind [inaudible 00:17:09] project and say, “This is a distributed system all but itself.” Start tearing it apart to pieces and seeing how they work and breaking them and then exploring and kind of just playing with the parts. You can do a lot of really interesting things. This is actually another book in your list that was written by Brendan Burns about Designing Distributed Systems I think it’s called. That book, I think he actually uses Kubernetes as a model for how to go about achieving these things, which I think is incredibly valuable, because it really gets into some of the more stable distributed systems patterns that are around. I feel like that's a great entry point. So if I had one thing, if I had to pick one way to help somebody or to push somebody in the direction of trying to learn distributed systems, I would say identify those distributed systems that maybe you’re already aware of and really explore how they work and what the problems with them are and how they went about solving those problems. Really dig into the idea of it. It’s something you could put your hands on and play with. I mean, Kubernetes is a great example of this, and this is actually why I referred to it. [00:18:19] CC: The way that works for me when I’m learning something like that is to really think about where the boundaries are, where the limitations are, where the tradeoffs are. If you can take a smaller system, maybe something like The Kind Project and identify what those things are. If you can’t, then ask around. Ask someone. Google it. I don’t know. Maybe it will be a good episode topic for us to do that. This part is doing this to map things out. So maybe we can understand better and help people understand things better. So mainly like yeah. They try to do the distributed system thesis are. But for people who don’t even know what they could be, it’s harder to identify it. I don’t know what a good strategy for that would be, because you can read about distributed systems and then you can go and look at a project. How do you map the concept to learning to what you’re seeing in the code base? For me, that’s the hardest thing. [00:19:26] MG: Exactly. Something that kind of I had related experience was like when I went into software development, without having formal education on algorithms and data structures, sometimes in your head, you have the problem statement and you're like, “Okay. I would do it like that.” But you don't know the word that describes, for example, a heap structure or queue because you’ve never – Someone told you that is heap, that is a queue, and/or that is a stick. So, for me, reading the book was a bit easier. Even though I have done distributed systems, if you will, administration for many years, many years ago, I didn't realize that it was a distributed system because I never had this definition or I never had those failure scenarios in mind and it never had a word for consensus. So how would I search for something like how do machines agree? I mean, if you put that on Google, then likely they will come – Have a lot of stuff. But if you put it in consensus algorithm, likely you get a good hit on what the answer should be. [00:20:29] CC: It is really problematic when we don't know the names of things because – What you said is so right, because we are probably doing a lot of distributed systems without even knowing that that’s what it is. Then we go in the job interview, and people are, “Oh! Have you done a distributed system?” No. You have but you just don’t know how to name things. But that’s one – [00:20:51] DC: Yeah, exactly. [00:20:52] CC: Yeah. Right? That’s one issue. Another issue, which is a bigger issue though is at least that’s how it is for me. I don’t want to speak for anybody else but for me definitely. If I can’t name things and I face a problem and I solve it, every time I face that problem it’s a one-off thing because I can’t map to a higher concept. So every time I face that problem, it’s like, “Oh!” It’s not like, “Oh, yeah!” If this is this kind of problem, I have a pattern. I’m going to use that to this problem. So that’s what I’m saying. Once you learn the concept, you need to be able to name it. Then you can map that concept to problems you have. All of a sudden, if you have like three things [inaudible 00:21:35] use to solve this problem, because as you work with computers, coding, it’s like you see the same thing over and over again. But when you don’t understand the fundamentals, things are just like – It’s a bunch of different one-offs. It’s like when you have an argument with your spouse or girlfriend or boyfriend. Sometimes, it’s like you’re arguing 10 times in a month and you thought, “Oh! I had 10 arguments.” But if you’d stop and think about it, no. We had one argument 10 times. It’s very different than having 10 problems versus having 1 problem 10 times, if that makes sense. [00:22:12] MG: It does. [00:22:11] DC: I think it does, right? [00:22:12] MG: I just want to agree. [00:22:16] DC: I think it does make sense. I think it’s interesting. You’ve highlighted kind of an interesting pattern around the way that people learn, which I think is really interesting. That is like some people are able to read about patterns or software patterns or algorithms or architectures and have that suddenly be an index of their heads. They can actually then later on correlate what they've read with the experience that they’re having around the things they're working on. For some, it needs to be hands-on. They need to actually be able to explore that idea and understand and manipulate it and be able to describe how it works or functions in person, in reality. They need to have that hands-on like, “I need to touch it to understand it,” kind of experience. Those people also, as they go through those experiences, start building this index of patterns or algorithms in their head. They have this thing that they can correlate to, right, like, “Oh! This is a time problem,” or, “This is a consensus problem,” or what have you, right? [00:23:19] CC: Exactly. [00:23:19] DC: You may not know the word for that saying but you're still going to develop a pattern in your mind like the ability to correlate this particular problem with some pattern that you’ve seen before. What's interesting is I feel like people have taken different approaches to building that index, right? For me, it’s been troubleshooting. Somebody gives me a hard problem, and I dig into it and I figure out what the problem is, regardless of whether it's to do with distributed systems or cooking. It could be anything, but I always want to get right in there and figure out what that problem and start building a map in my mind of all of the players that are involved. For others, I feel like with an educational background, if you have an education background, I think that sometimes you end up coming to this with a set of patterns already instilled that you understand and you're just trying to apply those patterns to the experience you’re having instead. It’s just very – It’s like horse before the cart or cart before the horse. It’s very interesting when you think about it. [00:24:21] CC: Yes. [00:24:22] MG: The recommendation that I just want to give to people that are like me who like reading is that I went overboard a bit in the beginnings because I was so fascinated by all the stuff, and it went down the rabbit hole deeper, deeper, deeper, deeper. Reading and reading and reading. At some point, even coming to weird YouTube channels that talk about like, “Is time real and where does time emerge from?” It became philosophical even like the past where I went to. Now, the thing is, and this is why I like Duffie’s approach with like breaking things and then undergo like trying to break things and understanding how they work and how they can fail is that immediately you practice. You’re hands-on. So that would be my advice to people who are more like me who are fascinated by reading and all the theory that your brain and your mind is not really capable of kind of absorbing all the stuff and then remembering without practicing. Practicing can be breaking things or installing things or administrating things or even writing software. But for me, that was also a late realization that I should have maybe started doing things earlier than the time I spent reading. [00:25:32] CC: By doing, you mean, hands-on? [00:25:35] MG: Yeah. [00:25:35] CC: Anything specific that you would have started with? [00:25:38] MG: Yes. On Kubernetes – So going back those 15 years to my early days of Linux and Samba, which is a project. By the time, I think it was written in C or C++. But the problem was I wasn’t able to read the code. So the only thing that I had by then was some mailing lists and asking questions and not even knowing which questions to ask because of lack of words of understanding. Now, fast-forward into Kubernetes’ time, which got me deeper in distributed systems, I still couldn't read the code because I didn't know [inaudible 00:26:10]. But I forced myself to read the code, which helped a little bit for myself to understand what was going on because the documentation by then was lacking. These days, it’s easier, because you can just install [inaudible 00:26:20] way easier today. The hands-on piece, I mean. [00:26:23] CC: You said something interesting, Michael, and I have given this advice before because I use this practice all the time. It's so important to have a vocabulary. Like you just said, I didn't know what to ask because I didn’t know the words. I practice this all the time. To people who are in this position of distributed systems or whatever it is or something more specific that you are trying to learn, try to expose yourself to the words, because our brains are amazing. Once you get exposure, it’s like your brain works in the background. All of a sudden, you go, “Oh, yeah! I know this word.” So podcasts are great for me. If I don't know something, I will look for a podcast on the subject and I start listening to it. As the words get repeated, just contextually. I don’t have to go and get a degree or anything. Just by listening to the words being spoken in context, absorb the meaning of it. So podcasting is great or YouTube or anything that you can listen. Just in reading too, of course. The best thing is talking to people. But, again, it’s really – Sometimes, it’s not trivial to put yourself in positions where people are discussing these things. [00:27:38] DC: There are actually a number of Meetups here in the Bay Area, and there’s a number of Meetups – That whole Meetup thing is sort of nationwide across the entire US and around the world it seems like now lately. Those Meetups I feel like there are a number of Meetups in different subject areas. There’s one here in the Bay Area called Papers We Love, where they actually do explore interesting technical papers, which are obviously a great place to learn the words for things, right? This is actually where those words are being defined, right? When you get into the consensus stuff, they really get into – One even is Raft. There are many papers on Raft and many papers on multiple things that get into consensus. So definitely, whether you explore a meetup on a distributed system or in a particular application or in a particular theme like Kubernetes, those things are great places just to kind of get more exposure to what people are thinking about in these problems. [00:28:31] CC: That is such a great tip. [00:28:34] MG: Yeah. The podcast is twice as good as well, because for people, non-natives – English speaker, I mean. Oh, people. Not speakers. People. The thing is that the word you’re looking for might be totally different than the English word. For example, consensus in Germany has this totally different meaning. So if I would look that up in German, likely I would find nothing or not really related at all. So you have to go through translation and then finding the stuff. So what you said, Duffie, with PWL, Papers We Love, or podcasts, those words, often they are in English, those podcasts and they are natural consensus or charting or partitioning. Those are the words that you can at least look up like what does it mean. That’s what I did as well thus far. [00:29:16] CC: Yes. I also wanted to do a plus one for Papers We Love. It’s – They are everywhere and they also have an online. They have an online version of the Papers We Love Meetup, and a lot of the local ones film their meetups. So you can go through the history and see if they talked about any paper that you are interested in. Probably, I’m sure multiple locations talk about the same paper, so you can get different takes too. It’s really, really cool. Sometimes, it’s completely obscure like, “I didn’t get a word of what they were saying. Not one. What am I doing here?” But sometimes, they talk about things. You at least know what the thing is and you get like 10% of it. But some paper you don’t. People who deal with papers day in and day out, it’s very much – I don’t know. [00:30:07] DC: It’s super easy when going through a paper like that to have the imposter syndrome wash over you, right, because you’re like – [00:30:13] CC: Yes. Thank you. That’s what I wanted to say. [00:30:15] DC: I feel like I’ve been in this for 20 years. I probably know a few things, right. But in talking about reading this consensus paper going, “Can I buy a vowel? What is happening?” [00:30:24] CC: Yeah. Can I buy a vowel? That’s awesome, Duffie. [00:30:28] DC: But the other piece I want to call out to your point, which I think is important is that some people don't want to go out and be there in person. They don’t feel comfortable or safe exploring those things in person. So there are tons of resources like you have just pointed out like the online version of Papers We Love. You can also sign into Slack and just interact with people via text messaging, right? There’s a lot of really great resources out there for people of all types, including the amount of time that you have. [00:30:53] CC: For Papers We Love, it’s like going to language class. If you go and take a class in Italian, your first day, even though that is going to be super basic, you’re going to be like, “What?” You’ll go back in your third week. You start, “Oh! I’m getting this.” Then a month, three months, “Oh! I’m starting to be competent.” So you go once. You’re going to feel lost and experience imposter syndrome. But you keep going, because that is a format. First, you start absorbing what the format is, and that helps you understand the content. So once your mind absorbs the format, you’re like, “Okay. Now, I have – I know how to navigate this. I know what’s coming next.” So you don’t have to focus on that. You start focusing in the content. Then little but little, you become more proficient in understanding. Very soon, you’re going to be willing to write a paper. I’m not there yet. [00:31:51] DC: That’s awesome. [00:31:52] CC: At least that’s how I think it goes. I don’t know. [00:31:54] MG: I agree. [00:31:55] DC: It’s also changed over time. It’s fascinating. If you read papers from like 20 years ago and you read papers that are written more recently, it's interesting. The papers have changed their language when considering competition. When you're introducing a new idea with a paper, frequently that you are introducing it into a market full of competition. You're being very careful about the language, almost in a way to complicate the idea rather than to make it clear, which is challenging. There are definitely some papers that I’ve read where I was like, “Why are you using so many words to describe this simple idea?” It makes no sense, but yeah. [00:32:37] CC: I don’t want to make this episode all about Papers We Love. It was so good that you mentioned that, Duffie. It’s really good to be in a room where we’ll be watching something online where you see people asking questions and people go, “Oh! Why is this thing like this? Why is X like this,” or, “Why is Y doing like this?” Then you go, “Oh! I didn’t even think that X was important. I didn’t even know that Y was important.” So you stop picking up what the important things are, and that’s what makes it click is now you’ve – Hooking into the important concepts because people who know more than you are pointing out and asking questions. So you start paying attention to learning what the main things it should be paying attention to, which is different from reading the paper by yourself. It’s just a ton of content that you need to sort through. [00:33:34] DC: Yeah. I frequently self-describe it as a perspective junkie, because I feel like for any of us really to learn more about a subject that we feel we understand, we need the perspective of others to really engage, to expand our understanding of that thing. I feel like and I know how to make a peanut butter and jelly sandwich. I’ve done it a million times. It’s a solid thing. But then I watch my kid do it and I’m like, “I hadn’t thought of that problem.” [inaudible 00:33:59], right? This is a great example of that. Those communities like Papers We Love are great opportunity to understand the perspective of others around these hard ideas. When we’re trying to understand complex things like distributed systems, this is where it’s at. This is actually how we go about achieving this. There is a lot that you can do on your own but there is always going to be more that you can do together, right? You can always do more. You can always understand this idea faster. You can understand the complexity of a system and how to break it down into these things by exploiting it with other people. That's I feel like – [00:34:40] CC: That is so well said, so well said, and it’s the reason for this show to exist, right? We come on a show and we give our perspectives, and people get to learn from people with different backgrounds, what their takes are on distributed systems, cloud native. So this was such a major plug for the show. Keep coming back. You’re going to learn a ton. Also, it was funny that you – It was the second time you mentioned cooking, made a cooking reference, Duffie, which brings me to something I want to make sure I say on this episode. I added a few things for reference, three books. But the one that I definitely would recommend starting with is The Systems Bible by John Gall. This book is so cool, because it helps you see everything through systems. Everything is a system. A conversation can be a system. An interaction between two people can be a system. I’m not saying this book says that. It’s just like my translation and that you can look – Cooking is a system. There is a process. There is a sequence. It’s really, really cool and it really helps to have things framed in this way and then go out and read the other books on systems. I think it helps a lot. This is definitely what I am starting with and what I would recommend people start with, The Systems Bible. Did you two know this book? [00:36:15] MG: I did not. I don’t. [00:36:17] DC: I’m not aware of it either but I really appreciate the idea. I do think that that's true. If you develop a skill for understanding systems as they are, you’ll basically develop – Frequently, what you’re developing is the ability to recognize patterns, right? [00:36:32] CC: Exactly. [00:36:32] DC: You could recognize those patterns on anything. [00:36:37] MG: Yeah. That's a good segue for just something that came to my mind. Recently, I gave a talk on event-driven architectures. For someone who's not a software developer or architect, it can be really hard to grab all those concepts on asynchrony and eventual consistency and idempotency. There are so many words of like, “What is this all – It sounds weird, way too complex.” But I was reading a book some years ago by Gregor Hohpe. He’s the guy behind Enterprise Integration Patterns. That’s also a book that I have on my list here. He said, “Your barista doesn't use two-phase commit.” So he was basically making this analogy of he was in a coffee shop and he was just looking at the process of how the barista makes the coffee. You pay for it and all the things that can go wrong while your coffee is brewed and served to you. So he was making this relation between the real world and the life and human society to computer systems. There it clicked to me where I was like, “So many problems we solve every day, for example, agreeing on a time where we should meet for dinner or cooking, is a consensus problem, and we solve it.” We even solve it in the case of failure. I might not be able to call Duffie, because he is not available right now. So somehow, we figure out. I always thought that those problems just exist in computer science and distributed systems. But I realized actually that's just a subset of the real world as is. Looking at those problems through the lens of your daily life and you get up and all the stuff, there are so many things that are related to computer systems. [00:38:13] CC: Michael, I missed it. Was it an article you read? [00:38:16] MG: Yes. I need to put that in there as well. Yeah. It’s a plug. [00:38:19] CC: Please put that in there. Absolutely. So far from being any kind of expert in distributed systems, but I have noticed. I have caught myself using systems thinking for even complicated conversations. Even in my personal life, I started approaching things in the systems oriented and just the – just a high-level example. When I am working with systems, I can approach from the beginning, the end. It’s like a puzzle, putting the puzzle together, right? Sometimes, it starts from the middle. Sometimes, it starts from the edges. When I‘m having conversations that I need to be very strategic like I have one shot. Let’s say maybe I’m in a school meeting and I have to reach a consensus or have a solution or have a plan of action. I have to ask the right questions. My private self would do things linearly. Historically like, “Let’s go from the beginning and work out through the end.” Now, I don’t do that anymore. Not necessarily. Sometimes, I like, “Let me maybe ask the last question I would ask and see where it leads and just approach things from a different way.” I don’t know if this is making sense. [00:39:31] MG: It does. It does. [00:39:32] CC: But my thinking has changed. The way I see the possibilities is not a linear thing anymore. I see how you can truly switch things. I use this in programming a lot and also writing. Sometimes, when you’re a beginner writer, you start at the top and you go down to the conclusion. Sometimes, I start I the middle and go up, right? So you can start anywhere. It’s beautiful or it just gives you so many more options. Or maybe I’m just crazy. Don’t listen to me. [00:40:03] DC: I don’t think you’re crazy. I was going to say, one of the funny things about Michael’s point and your point both, it’s like in a way that they have kind of referred to Conway's law, the idea that people will build systems in the way that they communicate. So this is actually – It totally brings it back to that same point of thing, right? We by nature will build systems that we can understand, because that is the constraint in which we have to work, right? So it’s very interesting. [00:40:29] CC: Yeah. But it’s an interesting thing, because we are [inaudible 00:40:32] by the way we are forced to work. For example, I work with constraints and what I'm saying is that that has been influencing my way of thinking. So, yes, I built systems in the way I think but also because of the constraints that I’m dealing with that I have to be – the tradeoffs I need to make, that also turns around and influences the way I think, the way I see the world and the rest of the systems and all the rest of the world. Of course, as I change my thinking, possibly you can theorize that you go back and apply that. Apply things that you learn outside of your work back to your work. It’s a beautiful back-and-forth I think. [00:41:17] MG: I had the same experience with some – When I had to design kind of my first API and think of, “Okay. What would the consumer contract be and what would a consumer expect me to deliver in response and so on?” I was forcing myself and being explicit in communicating and not throwing everything at the client back to confusing but being very explicit and precise. Also on communication every day when you talk to people, being explicit and precise really helps to avoid a lot of problems and trouble. Be it partnership or amongst friends or at work. This is what I took from computer science actually back into my real world in order to taking all those perceptions, perceiving things from a different perspective, and being more precise and explicit in how I respond or communicated. [00:42:07] CC: My take on what you just said, Michael, is we design systems thinking how is this going to fail. We know this is going to fail. We’re going to design for that. We’re going to implement for that. In real life, for example, if I need to get an agreement from someone, I try to understand the person's thinking and just go, “I just had this huge thing this week. This is in my mind.” I’m not constantly thinking about this, I’m not crazy like that. Just a little bit crazy. It’s like, “How does this person think? What do they need to know? How far can I push?” Right? We need to make a decision quickly, so the approach is everything, and sometimes you only get one shot, so yeah. I mean, correct me if I’m wrong. That's how I heard or I interpreted what you just said. [00:42:52] MG: Yeah, absolutely. Spot on. Spot on. So I’m not crazy as well. [00:42:55] CC: Basically, I think we ended up turning this episode into a little bit of like, “Here are great references,” and also a huge endorsement for really going deep into distributed systems, because it’s going to be good for your jobs. It’s going to be good for your life. It’s going to be good for your health. We are crazy. [00:43:17] DC: I’m definitely crazy. You guys might be. I’m not. All right. So we started this episode with the idea of coming to learning distributed systems perhaps without a degree or without a formal education in it. We talked about a ride of different ideas on that subject. Like different approaches that each of us took, how each of us see the problem. Is there any important point that either of you want to throw back into the mix here or bring up in relation to that? [00:43:48] MG: Well, what I take from this episode, being my first episode and getting to know your background, Duffie and Carlisia, is that whoever is going to listen to this episode, whatever background you have, even though you might not be in computer systems or industry at all, I think we three all had approved that whatever background you have, if you’re just curious a little bit and maybe a little bit crazy, you can totally get down the rabbit hole in distributed systems and get totally excited about it. There’s no need for having formal education and the degree to enter this world. It might help but it’s kind of not a high bar that I was perceiving it to be 10 years ago, for example. [00:44:28] CC: Yeah. That’s a good point. My takeaway is it always puzzled me how some people are so good and experienced and such experts in distributed systems. I always look at myself. It’s like, “How am I lacking?” It’s like, “What memo did I miss? What class did I miss? What project did I not work on to get the experience?” What I’m seeing is you just need to put yourself in that place. You need to do the work. But the good news is achieving competency in distributed systems is doable. [00:45:02] DC: My takeaway is as we discussed before, I think that there is no one thing that comprises a distributed system. It is a number of things, right, and basically a number of behaviors or patterns that we see that comprise what a distributed system is. So when I hear people say, “I’m not an expert in distributed systems,” I think, “Well, perhaps you are and maybe you don’t know it already.” Maybe there's some particular set of patterns with which you are incredibly familiar. Like you understand DNS better than the other 20 people in the room. That exposes you to a set of patterns that certainly give you the capability of saying that you are an expert in that particular set of patterns. So I think that to both of your points, it’s like you can enter this stage where you want to learn about distributed systems from pretty much any direction. You can learn it from a CIS background. You can come it with no computer experience whatsoever, and it will obviously take a bit more work. But this is really just about developing and understanding around how these things communicate and the patterns with which they accomplish that communication. I think that’s the important part. [00:46:19] CC: All right, everybody. Thank you, Michael Gasch, for being with us now. I hope to – [00:46:25] MG: Thank you. [00:46:25] CC: To see you in more episodes [inaudible 00:46:27]. Thank you, Duffie. [00:46:30] DC: My pleasure. [00:46:31] CC: Again, I’m Carlisia Campos. With us was Duffie Cooley and Michael Gesh. This was episode 12, and I hope to see you next time. Bye. [00:46:41] DC: Bye. [00:46:41] MG: Goodbye. [END OF EPISODE] [00:46:43] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

A warm welcome to John Harris who will be joining us for his first time on the show today to discuss our exciting topic, CI and CD in cloud native! CI and CD are two terms that usually get spoken about together but are actually two different things entirely if you think about them. We begin by getting into exactly what these differences are, highlighting the regulatory aspects of CD in contrast to the future-focussed nature of CI. We then move on to a deep exploration of their benefits in optimizing processes in cloud native space through automation and surveillance from development to production environments. You’ll hear about the benefits of automatic building in container orchestration, the value of make files and local test commands, and the evolution of CI from its ‘rubber chicken’ days with Martin Fowler and Jez Humble. We take a deep dive into the many ways that containers differ from regular binary as far as deployment methods, build speed, automation, run targets, realtime reflections of changes, and regulation. Moreover, we talk to the challenges of transitioning between testing and production environments, getting past human error through automation, and using sealed secrets to manage clusters. We also discuss the benefits and drawbacks of different CI tools such as Kubebuilder, Argo, Jenkins X, and Tekton. Our conversation gets wrapped up by looking at some of the exciting developments on the horizon of CI and CD, so make sure to tune in! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Bryan Liles Nicholas Lane Key Points From This Episode: • The difference between CI and CD.• Understanding the meaning of CD: ‘continuous delivery’ and ‘continuous deployment’.• Building an artifact that can be deployed in the future is termed ‘continuous integration’.• The benefits of continuous integration for container orchestration: automatic building.• What to do before starting a project regarding make files and local test commands.• Kubebuilder is a tool that scaffolds out the creation of controllers and web hooks.• Where CI has got to as far as location since its ‘rubber chicken’ co-located days.• The prescience of Martin Fowler and Jez Humble regarding continuous integration.• The value of running tests in a CI process for quality maintenance purposes.• What makes containers great as far as architecture, output, deployment, and speed.• The benefits of CD regarding deployment automation, reflection, and regulation.• Transitioning between testing and production environments using targets, clusters, pipelines.• Getting past human error through automation via continuous deployment.• What containers mean for the traditional idea of environments.• How labeling factors into the simplicity of transitioning from development to production.• What GitOps means for keeping track of changes in environments using tags.• How sealed secrets stop the need to change an app when managing clusters.• The tools around CD and what a good CD system should look like.• Using Argo and Spinnaker to take better advantage of hardware.• How JenkinsX helps mediate YAML when installing into clusters.• Why the customizable nature of CI tools can be seen as negative.• The benefits of using cloud native-built tools like Tekton.• Perspectives on what is missing in the cloud native space.• A definition of blue-green deployments and how they operate in service meshes.• The business abstraction elements of CI tools that are lacking.• Testing and data storage-related aspects of CI/CD that need to be developed. Quotes: “With the advent of containers, now it’s as simple as identifying the images you want and basically running that image in that environment.” — @bryanl [0:18:32] “The whole goal whenever you’re thinking about continuous delivery or continuous deployment is that any human intervention on the actual moving of code is a liability and is going to break.” — @bryanl [0:21:27] “Any time you’re in developer tooling, everyone wants to do something slightly differently. All of these tools are so tweak-able that they become so general.” — @johnharris85 [0:34:23] Links Mentioned in Today’s Episode: John Harris — https://www.linkedin.com/in/johnharris85/Jenkins — https://jenkins.io/CircleCI — https://circleci.com/Drone — https://drone.io/Travis — https://travis-ci.org/GitLab — https://about.gitlab.com/Docker — https://www.docker.com/Go — https://golang.org/Rust — https://www.rust-lang.org/Kubebuilder — https://github.com/kubernetes-sigs/kubebuilderMartin Fowler — https://martinfowler.com/Jez Humble — https://continuousdelivery.com/about/David Farley — https://dfarley.com/index.htmlAMD — https://www.amd.com/enIntel — https://www.intel.com/content/www/us/en/homepage.htmlWindows — https://www.microsoft.com/en-za/windowsLinux — https://www.linux.org/Intel 386 — http://www.computinghistory.org.uk/det/6192/Introduction-of-Intel-386/386SX — https://www.computerworld.com/article/2475341/flashback--remembering-the-386sx.html386DX — https://en.wikipedia.org/wiki/Intel_80386Pentium — https://www.intel.com/content/www/us/en/products/processors/pentium.htmlAMD64 — https://www.webopedia.com/TERM/A/AMD64.htmlARM — https://en.wikipedia.org/wiki/ARM_architectureTomcat — http://tomcat.apache.org/Netflix — https://www.netflix.com/za/GitOps — https://www.weave.works/technologies/gitops/Weave — https://www.weave.works/Argo — https://www.intuit.com/blog/technology/introducing-argo-flux/Spinnaker — https://www.spinnaker.io/Google X — https://x.company/Jenkins X — https://jenkins.io/projects/jenkins-x/YAML — https://yaml.org/Tekton — https://github.com/tektonCouncourse CI — https://concourse-ci.org/ Transcript: EPISODE 11 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically-minded decision maker, this podcast is for you. [EPISODE] [00:00:41] BL: Back to the Kubelets Podcast, episode 11. I’m Bryan Liles, and today we have Nicholas Lane. [00:00:50] NL: Hello! [00:00:51] BL: And joining us for the first time, we have John Harris. [00:00:55] JH: Hey everyone. How is it going? [00:00:56] BL: All right! So today we’re going to talk about CI and CD in cloud native. I want to start this off with this whole term CI and CD. We talk about them together, that are two different things almost entirely if you think about them. But CI stands for continuous integration, and then we have CD. What does CD stand for? [00:01:19] NL: Compact disk. [00:01:20] BL: Right. True, and actually I’ve used that term before. I actually do agree. But what else does CD stand for? [00:01:28] NL: It’s continuous deployment right? [00:01:30] BL: Yeah, and? [00:01:31] JH: Continuous delivery. [00:01:32] NL: Oh! I forgot about that one. [00:01:35] BL: Yeah, that’s the interesting thing, is that as we talk about tech and we give things acronyms, CD is just a great one. Change in directories, compact disk, continuous delivery and continuous deployment. Here’s the bonus question, does anyone here know the difference between continuous delivery and continuous deployment? [00:01:58] NL: Now that’s interesting. [00:01:59] JH: I would go ahead and say continuous delivery is the ability to move changes through the pipeline, but you still have the ability to do human intervention at any stage, and usually deployments production and continuous delivery would be a business decision, whereas continuous deployment is no gating and everything just go straight to product. [00:02:18] BL: Oh, John! Gold start for you, because that is one of the common ones. I just like to bring that up because we always talk about CI and CD as they are just one thing, but they’re actually way bigger topics and we’ve already introduced three things here. Let’s start at the beginning and let’s talk about continuous integration, a.k.a CI. I’ll start off. We have CI, and what is the goal of CI? I think that we always get boggled down with tech terms and all these technology and all these packages from all these companies. But I’d like to boil CI down to one simple thing. The process of continuous integration is to build an artifact that can be deployed somewhere at some future date at some future time by some future person, process. Everything else is a detail of the system you choose to use. Whether you use Jenkins, or CircleCI, or Drone, or you built your own thing, or you’re using Travis, or any of the other online CI tools. At the end of the day, you’re building either – If you’re doing web development. Maybe you’re building out Docker files, because we’re in cloud native. I mean docker images, because we’re in cloud native. But if you’re not, maybe you’re just building JARs, WARs, or EARs, or a ZIP file, or a binary, or something. I’d just like to start off, start this off with there. Any more thoughts on continuous integration? [00:03:48] NL: Yeah. I think the only times that I’ve ever used something that’s like continuous integration is when I’ve been doing like more container orchestration, like development, things on top of like things like Kubernetes, for instance. The thing I really like about it is like the concept of being able to like, from my computer, save and do an automatic save and push to a local repo and have all of the pieces get built for me automatically somewhere else, and I just love that so much because it saves so much brain thinky juice to run every command to make the binary you need. [00:04:28] BL: So did you actually create those scripts yourself? [00:04:30] NL: Some of them. When I’ve used things like GitLab, I use the pipeline that exists there and just fiddled around with like a little bit of code, like some bash there, but like not too much because GitLab has a pretty robust pipeline. Travis — I don’t think I needed to actually. Travis had a pretty good just go make Docker build, scripts already templated out for you. [00:04:53] JH: Yeah. I’d like to tell people whenever you start any project, whether it’s big or small, especially if it’s on – Not on Windows. I’ll tell you something different if it’s on Windows. But if you’re developing on a Mac or developing on Linux, the first thing you should do in your project is create a make file or your programming language equivalent of a make file, and then in that make file what you should do is write a command that will build your software that runs its tests locally, and also builds – whatever the process is. I mean, if you’re running in Go, you do a Go build. If you’re using Rust, build with Rust, or C++, or whatever before you even write any code. The reason why is because the hardest part is making your code build, and if you leave that to the end, you’re actually making it harder on yourself. If your code build works from the beginning, all you have to do is change it to fit what you’re doing rather than thinking about it when it’s crunch time. [00:05:57] NL: I actually ran into that exact scenario recently, because I’ve been building some tooling around some Kubernetes stuff, and the first one I did, I built it all manually by hand. Then at the end I was like – I gave it to the person who wanted it and they’re like, “So, where’s the make file?” I’m like, “Where’s the what?” So I had go in and like fill in the make file, and that was a huge pain in the butt. Then recently the other thing I’ve been using is Kubebuilder. John, you and I have been talking about Kubebuilder quite a bit, but using Kubebuilder, and one of the things it does for you is it scaffolds out and a make file for you, and that was like going from me doing it by myself to having it already exist for you or just having it at the beginning was so much better. I totally agree with you, Brian. [00:06:42] BL: So quick point of order here. For those of us who don’t know what Kubebuilder is. What is Kubebuilder? [00:06:48] NL: Kubebuilder is a tool that was created by members of the Kubernetes Community to scaffold out the creation of controllers and web hooks. What a controller is in Kubernetes is a piece of software that waits, sort of watches a specific object or many specific objects and reconciles them. If they noticed that something has changed and you want to make an action based on that change, the controller does that for you. [00:07:17] JH: Okay. So it actually makes the action of working with CRDs and Kubernetes much easier than creating it all yourself. [00:07:26] NL: Correct. Yeah. So, for instance, the one that I made for myself was a tool that watched, updated and watched a specific CRD, but it wasn’t necessarily a controller. It was just like flagging on whether or not a change occurred, and I used the dynamic client, and that was a huge headache on of itself. Kubebuilder has like the ability to watch not just CRDs, but any object in Kubernetes and then reconcile them based on changes. [00:07:53] NL: It’s pretty great. [00:07:54] BL: All right. So back to CI. John, do you have any opinions on CI or anecdotes or anything like that? [00:07:59] JH: Yeah. I think one of the interesting things about the original kind of philosophy of CI outside of tooling was like trunk-based development that every develop changes get integrated into trunk as soon as possible. You don’t get into integration hell and rebasing. I guess it’s kind of interesting when you apply that to a cloud native landscape where like when that stuff came out with like Martin Fowler or Jez Humble probably 10, 15 years ago almost now, a lot of dev teams were co-located. You could do CI. I think there was a rubber chicken method where you didn’t use a tool. It was just whoever had the chicken that’s responsible for the build. Just to pull everyone else’s changes. But now it seems like everything is branch-based. When you look at a project like Kubernetes, there’s a huge number of contributors all geographically displaced, different time zones, lots of different branches and features going on at the same time. It’s interesting how these original principles of continuous integration from the beginning now apply to these huge projects in the cloud native landscape. [00:08:56] BL: Yeah, that’s actually a great point of how prescient Martin Fowler has been for many, many years, and even with Jez Humble being able to see these problems 10, 15 years ago and be able to describe them. I believe Jez Humble wrote the CD book, the continuous delivery book. [00:09:15] JH: Yeah, with David Farley, I think. [00:09:18] NL: Yeah. Yeah, he did. So, John, you brought up some good things about CI. I try to simplify everything. I think the mark of someone who really knows what they’re talking about is being able to explain everything in the simplest words possible, and then you can work backwards when people understand. I started off by saying that CI produces an artifact. I didn’t talk about branches or anything like that, or even the integration piece. But now let’s go into that a little bit. There are a lot of misconceptions about CI in general, but one of the things that we talk about is that you have to run test. No, you don’t have to run test, but should you? Yes, 100% of the time. Your CI process, your integration process should actually build your software and run the test, because running the test on this dedicated service or hardware wherever it is ensures that the quality of your software is there at least as much as your developers have insured the quality in the test. It’s very important those run, and a lot of bugs of course can be spotted by running a CI. I mean, we are all sorts of developers here, and I tell you what, sometimes I forget to run the test locally and CI catches me before a commit makes it into master and it has a huge typo or a whole bunch of print lines in there. Moving on here, thinking about CI and cloud native. Whenever you’re creating a cloud native app, have you ever thought about the differences between let’s say creating just a regular binary that maybe runs on a server, but not in a container on somebody’s cloud native stack, i.e. Kubernetes? Have you ever thought about the differences of things to think about? [00:11:04] BL: Yeah. So part of it is – I would imagine or I believe it’s like things like resource, like what resources you need or what architecture you’re deploying into. You need the binary to make like run in this – With containerization, it’s easy because you’re like, “I know that the container is going to be this architecture,” but you can’t necessarily guarantee that outside of a containerized world. I mean, I suppose you can being like with the right tooling setup you can be like, “I only want to run on this.” But that isn’t necessarily guaranteed, because any computer that runs on could be just whatever architecture that happens to land on, right? Also, something to – I think of is like how do you start processes on disparate computers in a controlled fashion? Something like, again, with containers, you can trust that the container runtime will run it for you. But without that, it seems like a much harder task. [00:12:01] NL: Yeah, I would agree. Then I said that containers in general just help us out, because most of our workloads go on some AMD or Intel 64 bit and it’s Linux. We know what our output is going to be. So it’s not like in the old days where you had to actually figure out what your run target was. I mean, that’s even on Intel stacks. I mean, I’m updating myself here where you had like – When the 386 was out and then you had the 386SX and the 386DX, there were different things there, and you actually compile your code different. Then when the 46 came out and then when we had introduction of Pentium chips, things were different. But now we can pretty much all target AMD64, and in some cases, I mean, there are some chip things like the bigger encryption things that are in the newer chips. But for the most part, we know what our deployed target is going to be. But the cool thing is also that we don’t have to have Intel or AMD64. It could be ARM32 or ARM64, and with the addition to a lot of the work that has been going on in Windows land lately, we can have Windows images. I don’t know so many people were doing that yet. I’m not out and part of the field, but I like that the opportunity is there. [00:13:25] JH: Oh! I think one of the interesting things is the deployment method as well. Now with containers, everything is kind of an immutable rip and replace. Like if we develop an application, we know that the old container is going to stop when I deploy a new one. I think Netflix were doing a little bit of this before containers and some other folks with like baking AMIs and using that immutable method. But I think before that it was if we had a WAR file, we had to throw it back into Tomcat, let Tomcat pick it up or whatever. Everything was a little bit more flaky in terms of deployment. We had to do a lot of checks around deployment rather than just bring something out, bring something back in blue/green, whatever. [00:13:59] BL: Well, I actually like that you brought that up, because that’s actually one of the greatest parts of this whole cloud native thing, is that when we’re using containers and we’re deploying with containers, we know what our file system is going to look like, because we created it. There would not be some rogue file or another configuration there that will trip up our deployment, because at build time, we’ve created the environment. It’s much better than that facility that Netflix was doing with baking AMIs. In a previous life, I actually ran the facility for baking AMIs at a large company where we had thousands of developers on more than a thousand dev teams, and we had a lot of spyware. Whenever you had to build an image, it was fine in one account, but if you had let’s say a thousand accounts with the way that AWS works and encrypted images, you actually had to copy all the images to all the accounts. It couldn’t actually boot it from your account. That process would literally take all night to get it done across all of our accounts. If you made a mistake, guess what? You get to do it again. So I am glad that we actually have this thing called a container and all these things based on CRI, the container runtime, that we are able to quickly build containers. I don’t want to just limit this conversation to continuous integration. Let’s get into the other parts too with deployment and delivery. What is so novel about CD and the cloud native world? [00:15:35] NL: I think to me it’s the ability to have your code or your artifact or whatever it is, whatever you’re working on. When you make a change, you can see the change reflected in reality, whatever your reality looks like, without your intervention. I mean, you might have had to set up all the pipelines and all that jargon, but when you press save in VS code and it creates a branch and runs all your tests and then deploys it for you or delivers it for you into what you’d define as reality, that’s just so nice, because it really kind of sucks having to do the like, “Okay, I’ve got a new deployment. Destroy the old deployment. Put in the new one or like rev the new image tag or whatever in the deployment you’re doing.” All these manual steps, again, thinky-brain juice, it takes pieces of your attention away, and having these pieces like added for you is just so nice. [00:16:30] BL: Yeah, what do you think, John? [00:16:32] JH: Yeah. I think just something in the state of DevOps we’ve bought one of the best predictors for a company’s success is like cycle time of feature from ideation to production. I think like the faster we can get that cycle – It kind of gets me interested. How long does an application take to build? If it takes two hours, how good are you at getting features out there quickly? Maybe one of the drivers with microservices, smaller pieces independently deployed, we can get features out to production quicker, because I think the name of the game is just about enabling developers to put the decision in the hands of the business to decide when the customer should see that feature. I think the tighter we can make that cycle, the better for everyone. [00:17:14] BL: Oh, no! I agree. I love and hate web services, but what I do like is the idea of making these abstractions smaller, and if the abstractions are smaller, it’s less code. A lot of the languages we use now are faster compiling, let’s say, a large C++ project. That could take literally two hours to compile. But now when we have languages like Go, and Rust is not as fast, but it’s not slow as well. Then we have all of our interpret languages, whether it’d be Python, or JavaScript, or TypeScript, where we can actually go from an idea, run the test in a few minutes and build this image that we can actually run and see it almost in real-time. Now with the complexity of the tools, I mean, the features that are built in the tools, we can now easily manage multiple deployment environments, because think about before, you would have a dev environment, and that would be the Wild West. That would be literally where it would be awful. You might have to rebuild it every couple of months. Then you would have staging, and then maybe you would have some kind of pre-prod environment just as like your final smoke test, and then you would have your production. Maintaining all the software on all those was extremely hard. But now with the advent of containers, now it’s as simple as identifying the images you want and basically running that image in that environment. I like where we’ve ended up. But with all power comes new problems, and just because we can deploy quicker means we just run into a lot of different problems we didn’t run into before. The first one that I’ll bring up is the complexity. Auto conversion between environments, so moving code between test staging and production. How do we do that? Any ideas before I throw some out there? [00:19:11] NL: I guess you would have different, or maybe the same pipeline but different targets for like if say you’re using something like Kubernetes. You could have one part of your pipeline deploy initially to this Kubernetes context, which points to like one cluster. It’s building up clusters by environment type and then deploying into those, running your tests, see if it runs properly and then switch over to the next context to apply that image tag and that information and then just go down the chain until you go to production. [00:19:44] BL: Well, that’s interesting. One thing I’d like to throw out there, and I’m not advocating any particular product. But the idea of having pipelines for continuous integration and your CD process is great, where you can now have gates and you can basically automate the whole thing. Code goes into CI and we built an artifact, and a message can go out automatically to an approver or not, and that message could say, “Hey! This code is going to be integrated into our trunk or our master branch.” They can either do it themselves manually as a lot of people do or they can actually maybe click on a link or check a checkbox and this gets integrated in. Then what automatically could happen at this point is, and I’ve seen a lot of companies doing this, is now we take that software and we spin up a new whole environment and we just install that software. For that one particular feature that you worked on, you can actually get an automatic environment for that. Then what we can do is we can take that environment itself and we can now merge this maybe into a staging branch or tag it with a staging label, and that automatically gets moved to staging. Depending on how complicated you are, how advanced you are, now you can actually have it go out to your product people or people who make decisions, maybe your executives, and they can view the software in whatever context it happens to be in. Then they can say, “Okay.” Now that’s when we’re talking about now we can hit okay and the software just keeps on moving to the pipeline and it gets into production. The whole goal here, and this is actually where your goal should be just in general whenever you’re thinking about continuous delivery or continuous deployment is that any human intervention on the actual moving of code is a liability and is going to break, and it’s going to break because on Friday afternoon at 5:25 PM, someone’s thinking about the weekend and they’re not thinking about code, and they’re going to break your build. Our goal is to build these delivery systems that are Friday afternoon proof. We can push code anytime. It doesn’t matter. We trust our process. [00:22:03] JH: I think it’s a great point about environments. I think back in the day, an environment used to be a set of machines, and then test used to be – staging was where there were kind of more stable versions of APIs and folks were more coordinated pushing things into them. What really is an environment? Like you said, when we push micro services or whatever service, we can spin up an entire Kubernetes cluster just for that service. We can set it up. We can run whatever tests we want. We could tear it down. With the advent of Elastic compute, and now containers, they really enabled this world where like the traditional idea of an environment and what constitutes an environment is starting to get a bit kind of sloppy and blend into each other. [00:22:42] BL: I like it though. I think it’s progress. [00:22:45] NL: I totally agree. The one that scares me but I also find like really interesting, is the idea of having all of your environments in one set of machines. So clusters. Having a multi-tenanted set of machines for like dev staging and production, they’re all running in the same place and they’re all just separated by like what configuration of like connectivity from different networking and things like that set up. When a user hits your website, bryanliles.com, they should go to the production images, but those are binaries, and those binaries should be running in the same space essentially as the development ones. It’s scary, but it’s also like allows for like some really fast testing and integration. I find it to be very fascinating. [00:23:33] BL: I mean that’s where we want to be. I find more often than not that people have separate clusters for dev and staging and production. But using the Kubernetes API, you don’t have to do that, because what we can do is we can force deployment or workload to a set of machines based on their label. That’s actually one of the very strong positives for Kubernetes. Forget all the complexity. One of the things that makes it easy is to say that I want this particular deployment to only live on my development machines. Well, which development machine? I don’t care. What if we increase our development pool size? We just re-label nodes. It doesn’t matter. Now we can just control that. When it comes down to controlling cost and complexity, this is actually one idea that Kubernetes is leading and just making it easier to actually use more of your hardware. [00:24:31] NL: Yeah. Absolutely. That’s so great because if you think about it from a CI/CD standpoint, at that point all you have to do is just change the label to where you’re applying this piece of code. So you’re like, “Node selector, label equals dev. Okay, now it’s staging. Okay, now it’s prod.” [00:24:47] BL: So this brings me into the next part of what I want to talk about or introduce to you all today. We’re on a journey as you probably can tell. Now whenever we have our CI process and we’re building and we’re deploying, where do we store our configurations? [00:25:04] NL: [inaudible 00:25:04]. [00:25:06] BL: Ever thought about that? [00:25:08] NL: Okay. I mean, in a Kubernetes perspective, you might be using something like etcd to sort of – But like everything else, what if you’re using Travis? [inaudible 00:25:16] store everything. Everything should be versioned, right? Everything should be – [00:25:20] BL: Yeah, 100%. [00:25:24] NL: I would store everything these as much as possible. Now, do I do that all the time? God, no! Absolutely not. I’m a human being after all. [00:25:32] BL: I mean, that’s what I actually want to bring up, is this concept of GitOps. GitOps was a coined term by my friend, Alexis, who works at Weave. I think Weave created this. Really what it’s about is instead of having – basically, Kubernetes is declarative, and our configurations can be declarative too, because what we can do is make sure is we can have tech space configurations, and for one reason it’s because tech space means it can be versioned. It can be diffs. We take those text versions and we put them in our same repository we put our code in. How do we know what’s in production at any given time or any given time in the past? We just look at the tags of what we did. We had a push at 5:15 on August 13th. Of course, this is 5:15, you could see time, because any other time doesn’t exist in the computer land. So what we could do is we could just basically tag that particular version as like 2019-08-13. If I said 5-17-55, and we call 01 just so we could have 100 deploys in a day. If we started doing that, now not only can we control what we have, but we can also know what was on in any given environment at any given time. Because with Git and with Mercurial and any other of these – Well, only the popular ones, with Git and Mercurial, you can definitely do this. Any given commit can have multiple tags. You could actually have a tag that hit dev and then a tag that, let’s say, hits staging, and then a tag that hit production, the exact same code but three different tags. So you know at any given time what happened. [00:27:18] JH: Yeah, the config thing is so important. I think that was another Jez Humble quote where it was like, “Give me three hours access to your code and I’ll break it. But give me 5 minutes with your configurations and I’ll break it.” Almost like every big bug is, right, someone was accidentally pointing the prod server to the staging database like, “Oops! Their API was pointing to the wrong port, and everything came down,” or we changed the wrong versions or whatever. I think that’s one of the intersections of developers and operations folks. We kind of talked about like Dev Ops and things like that. I really love the idea of everything being kept in Git and using GitOps, but then we’ve got things like secrets and configuration that shouldn’t be seen or being able to be edited by developers, but need to be for ops folks. But we still want to keep the single point of truth. Things like sealed secrets have really enabled us to move along in this area where we can keep everything in text-based version. [00:28:08] BL: All right. Quick point of order here. Sealed secrets is a controller/CRD created by Bitnami. What it allows you do is, John – [00:28:23] JH: It allows you – It creates a CRD, which is sealed secret, which is a special resource type in your cluster and also creates a key, which is only available to that operator running in your cluster. You can submit a sealed secret in plain text or you can submit a secret in plain text and it will throw it back out as an encrypted secret with that key and then you can check that into version control. Then when you go to deploy your software, you can deploy that encrypted secret into the cluster. The operator will pick it up, decrypt it using only the key that it has access to and then put it back in the cluster as a regular secret. Your application just interacts with regular Kubernetes secrets. You don’t need to change your app. They deal with all the encryption outside of the user intervention. [00:29:03] BL: I think the most important part of what you said is that this allows us to have no excuses about what we can store in our repositories for our configuration, because someone is going to make the argument, “No, we can’t store secrets, because someone’s going to be able to see them.” Well, guess what? We never even stored an unencrypted secret in our repository. They’re all encrypted, and it’s still secrets. It’s [inaudible 00:29:25]. I don’t know if anyone’s cracked yet. I’m sure maybe a state level actor has thought of it. But for us regular people, even our companies, like even at VMware, or even at Google, they have not done it yet. So it’s still pretty safe. Thinking even further now, and really what I’m trying to paint the picture of is not just how do you do CD, but really what CD could look like and how it can actually make you happy rather than sad. The next item I wanted to think about was tools around CD and creating tools and what does a good continuous delivery system look like. I kind of hinted about this earlier whenever I was talking about pipelines. The ability to take advantage of your hardware, so we’re deploying to let’s say 100 servers. We’re pulling 5 or 6 services to 100 node cluster. We can do those all at once, and what we can do is you want to have a system that can actually run like this. I could think of a couple. From Intuit, there is Argo, and they have Argo CD. There is the tool created by Google and maybe Netflix. I want to have to look that one up. It’s funny, because they quoted – [00:30:40] JH: Spinnaker? [00:30:42] BL: Spinnaker. They quoted me in their book, and I don’t remember their name. I’m sorry anyone from Spinnaker product listening. Once again, not advocating any products, but they have the concept of doing pipelines. Then you also have other things for your projects, like if you’re using open source, Drone. Another X Google – I think it was X-Googler that made this. Basically, they have ways you can do more than one thing at a time. The most important piece about this is not only can you do more than one thing at a time, is that you have a programmatic check that it’ll make sure that you can verify that whatever you did was successful. We deployed to staging or we deployed to our smoke test servers for our smoke test, and that requires our testing people and an executive signoff. They can actually just wait until they get their signoff or maybe if it goes over a day or so, they can actually – It just fails, and now the build is done. But that part is pretty neat. Any other topics over here before I start throwing out more? [00:31:45] NL: I think I just have thoughts on some of the tools that we’ve used. Everyone Jenkins. Jenkins can do anything that you want it to do, but you really have to tighten the screws on it. It is super powerful. It’s kind of like Bash, like Bash scripting. It’s super powerful, but you have to know precisely what you’re doing, otherwise it can really hurt you. Actually, I have used Spinnaker in the past, and I’ve really liked it. It has a good UI, very good pipelines. Easy blue/green or canary deployment mechanism, I thought that was great. I’ve looked at Drone, believe it or not, but Drone is actually pretty cool. Check out Drone. I really liked it. [00:32:25] BL: Well, since we’re throwing out products, Jenkins, does have JenkinsX. I have not given it the full rundown yet. But what I do like about it, and I think everyone should pay attention to this if you’re doing a product in this space, is that when you install JenkinsX, you install it locally to your machine. You basically get this binary called JX, and you then tell JX to install it into your cluster. Instead of just doing kubectl apply-f a whole bunch of YAML, it actually ask you questions and it sets up GitHub repositories or wherever you need these repositories. It sets up [inaudible 00:33:01] spaces for you. There’s no just [inaudible 00:33:05] kubectl apply-f HTTPS: I just owned your system, because that’s actually a problem. Then it solves the YAML sprawl, because YAML and Kubernetes is something that is complained about a lot, but it’s how it’s configured. But it’s also just a detail what we’re supposed to be doing, and we actually work with Joe Beda and I could talk about this all the time, is that the YAML is the implementation, but it’s not the idea. The idea is that we build tools on top of that that create YAML so users have to see less YAML. I think that’s a problem with Jenkins, is that it’s so powerful and they’re like, “Well, we want powerful people or smart people to be able to do smart things. So here you go.” The problem with that is that where do I start? It’s a little daunting. So I do think that they definitely came with the much stronger game with this JX command. Just as a little sidebar, we do it as well with our Valero project, and I think that just speaks, should be like the bar for anything. If you’re installing something into a cluster, you should come up with a command line tool that helps you manage the lifecycle of whatever you’re installing to the operator, YAML, whatever. [00:34:18] JH: I think what’s interesting about the options, this is definitely one area where there’s so much nuance. Any time you’re in developer tooling, everyone wants to do something slightly differently. All of these tools are so tweak-able that they become so general. I think it’s probably one of the criticisms that could be leveraged against Jenkins is that you can do everything, and that’s actually a negative as well as a positive. Sometimes it’s too overwhelming. There are too many ways of doing things. I’m a fan of some of the more kind opinionated tools in that space. [00:34:45] BL: Yeah. I like opinionated tools as well, but the problem that we’re having in this cloud native space is that, yeah, Kubernetes is five-years-old now. We are just getting to the point where we actually understand what a good decision is, because there was a lot of guesses before and we’ve done a lot of things, and some of these have been good ideas, but in some cases they have not been great ideas. Even I ran the project case on it. Great idea on paper, but implementation, it required people to know too many things. We’d learned a lot of lessons from that. That’s what I think we’re going to find out in this space is that we’re going to learn little lessons. I say this project from my last project that I was going to bring up is something that I think has learned some of the lessons. Google sponsors a project called Tekton, and if you go to – It’s like I believe, and they have some continuous delivery stuff in there and they implement pipelines. But the neat part is, and this is actually the best part, it’s actually a cloud native built service. So every step of your delivery process, from creating images, to actually putting them on clusters, is backed by a Docker image or a container, and I think that part is pretty neat. So now you can define your steps. What is your step? Well, you can use one of their pre-baked, run this command, or if you have something special, like the example before I was giving out where you would say that you need an approval, maybe it’s a Slack approval. You send something with Slack and it has a checkbox, check yes if you like me. What we can do now is we can actually control that and it’s easy to write something a little Docker image that can actually make that call and then get the request and then it can move it on. If you’re looking at more of a toolkit full of good ideas, I do think that Tekton has definitely has some lots of industry. People are looking at it and it’s probably the best example of getting it right in the cloud native way. Because a lot of the products we have now are not cloud native. We’re talking about Jenkins. We’re talking about Spinnaker and we talk about Drone and Travis, which is totally a SaaS product. They’re not cloud native. Actually, the neat part about Tekton is that it actually comes with its own controllers and its own CRDs. So you can actually build these things up using your familiar Kubernetes tooling, which means in theory we could actually use the tooling that we are deploying. We can actually control it in the same way as our applications, because it’s just yet another object that goes in our cluster. [00:37:21] NL: That does sound pretty cool. One other that I meant to bring up was Concourse. Have you check out Concourse yet? [00:37:27] BL: CouncourseCI. I have not. I have used it, but never in a way where I would have a big opinion on it. [00:37:34] NL: I’m kind of in the same place. I think it’s a good idea. It seems really neat, but I need to kick the tires a little more. I will say that I really like the UI. The structure of the UI is really nice. Everything makes sense, and anything you can click on like drills into something a bit deeper. I think that’s pretty cool, but it is one of the shout that I went out to as well as like another tool that I’m aware of. [00:37:52] BL: Yeah, that’s pretty interesting. So we’ve gone about 40 minutes now. Let’s actually start winding this down, and the way that I’m going to suggest that we wind this down is thinking about where we are now. What’s missing in this space and what else could we actually be doing in the cloud native space to make this work out better? [00:38:12] NL: I think I’d like to see better structured or better examples of blue-green or canary deployments with tests associated, and that might just be like me not looking hard enough at this problem. But anytime I began looking at blue-green, I get the idea of what someone’s done, but I would love to see some implementation details, or any of these opinionated tools having opinions around blue-green and what they specifically do to test it. I feel like I’m just not seeing that. [00:38:41] BL: With blue-green, blue-green is hard to do in Kubernetes without an external tool, because for everyone, a blue-green deployment is, I have a software deployment and we’ll give it a color. We’ll call it blue, and I have the next version, and we’ll call it green. Really what I can do is I basically have two versions of my application deployed and I can use my load balancer, or in this case, my service to just change the label or the selector in my service and now I can point at at my green from my blue. Then I want to deploy again, I can just deploy another blue and then change my label selector again. The problem with this is that you can do it in Kubernetes, just fine. But out of the box with Kubernetes, you will drop traffic, because guess what? What happens to a connection that was initiated or a session that was initiated on the blue cluster when you went to green? Actually, this is a whole conversation in itself about service meshes and this is actually one of the reasons service mesh is a big topic, because you can do this blue-green, or another example would be Netflix and Redblack, or you get the creative people who are like rainbow deployments, because just having two is not good enough for them. So they want to have any number of deployments going at one time. I agree with that 100%. [00:39:57] JH: I think, yeah, integrating tools like launch. [inaudible 00:40:01] and I think there are more which enable – I think we’re missing the business abstractions on this stuff so far. Like you said, it’s kind of hard to do if you need to go into the gritty of it right now, but I think the business abstractions of if we deploy a different version to a certain subset of customers, can we get all of those metrics? Can we get those traces back in? Will you automate it, roll it out? Can we increase the percentage of customers that are seeing those things? Have that all controlled in a Kubernetes native way, but having roll it up to a business and more of an abstraction. I think that stuff is currently missing. I think the underpinning kind of technologies are coming up, stuff like service mesh, but I think it’s the abstraction that’s really going to make it useful, which doesn’t exist today. [00:40:39] BL: Yeah. Actually, that’s pretty close to what I was going to say. We built all these tooling that helps us basically as technologists, but really what it comes down to is the business. A lot of the things we’re talking about where we’re talking about CD is important to the business, but when we’re talking about metrics or trace collection, that’s not important to the business, because they only care about the SLA. This is on the SLO side. What we really need to do is mature our processes enough that we can actually marry our outputs to something that other people can understand that has no jargon and it’s sales going up, sales going down. Everything else is just a detail. So, anything else? [00:41:20] NL: Something I think I’d like to see is in our testing, if there was a good way to accurately show the effect of something at load in a CI/CD component. Because one of the things that I’ve run into is like I’ve got this great idea for how this code should work and when I deploy it, it works great. The like a thousand people touch it all at once and it doesn’t work right anymore. I’d love to have some tool along the way that can test things out of load and like show me something that I could fix before all those people touch it. [00:41:57] BL: Yes, that would be a good tool to have. So John, anything else for you? [00:42:02] JH: I’ll open a can of worms right at the end and say the biggest problem here is probably going to be data when we have a lot of systems we need to talk to each other and we need the data to align between those systems and we have now proliferation of environments and clusters. Like how do we get that data reliably into the place that it needs to be to make up testing robust enough to get things out there? It’s probably an episode on some – [00:42:23] BL: Yeah, that’s a big conversation that if we could answer it, we wouldn’t working at VMware. We would have our own companies doing all these great things. But we can definitely iterate on it. So with that, I think we’re going to wrap it up. Thanks for listening to the Kubelets. I’m Bryan Liles, and with me today was Nicholas Lane and John – Yeah, and John Harris. [00:42:47] JH: Thanks everyone. [00:42:47] BL: All right, we’ll see you next time. [END OF EPISODE] [00:42:50] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

Security is inherently dichotomous because it involves hardening an application to protect it from external threats, while at the same time ensuring agility and the ability to iterate as fast as possible. This in-built tension is the major focal point of today’s show, where we talk about all things security. From our discussion, we discover that there are several reasons for this tension. The overarching problem with security is that the starting point is often rules and parameters, rather than understanding what the system is used for. This results in security being heavily constraining. For this to change, a culture shift is necessary, where security people and developers come around the same table and define what optimizing to each of them means. This, however, is much easier said than done as security is usually only brought in at the later stages of development. We also discuss why the problem of security needs to be reframed, the importance of defining what normal functionality is and issues around response and detection, along with many other security insights. The intersection of cloud native and security is an interesting one, so tune in today! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Duffie Cooley Bryan Liles Nicholas Lane Key Points From This Episode: Often application and program security constrain optimum functionality. Generally, when security is talked about, it relates to the symptoms, not the root problem. Developers have not adapted internal interfaces to security. Look at what a framework or tool might be used for and then make constraints from there. The three frameworks people point to when talking about security: FISMA, NIST, and CIS. Trying to abide by all of the parameters is impossible. It is important to define what normal access is to understand what constraints look like. Why it is useful to use auditing logs in pre-production. There needs to be a discussion between developers and security people. How security with Kubernetes and other cloud native programs work. There has been some growth in securing secrets in Kubernetes over the past year. Blast radius – why understanding the extent of security malfunction effect is important. Chaos engineering is a useful framework for understanding vulnerability. Reaching across the table – why open conversations are the best solution to the dichotomy. Security and developers need to have the same goals and jargon from the outset. The current model only brings security in at the end stages of development. There needs to be a place to learn what normal functionality looks like outside of production. How Google manages to run everything in production. It is difficult to come up with security solutions for differing contexts. Why people want service meshes. Quotes: “You’re not able to actually make use of the platform as it was designed to be made use of, when those constraints are too tight.” — @mauilion [0:02:21] “The reason that people are scared of security is because security is opaque and security is opaque because a lot of people like to keep it opaque but it doesn’t have to be that way.” — @bryanl [0:04:15] “Defining what that normal access looks like is critical to us to our ability to constrain it.” — @mauilion [0:08:21] “Understanding all the avenues that you could be impacted is a daunting task.” — @apinick [0:18:44] “There has to be a place where you can go play and learn what normal is and then you can move into a world in which you can actually enforce what that normal looks like with reasonable constraints.” — @mauilion [0:33:04] “You don’t learn to ride a motorcycle on the street. You’d learn to ride a motorcycle on the dirt.” — @apinick [0:33:57] Links Mentioned in Today’s Episode: AWS — https://aws.amazon.com/Kubernetes https://kubernetes.io/IAM https://aws.amazon.com/iam/Securing a Cluster — https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/TGI Kubernetes 065 — https://www.youtube.com/watch?v=0uy2V2kYl4U&list=PL7bmigfV0EqQzxcNpmcdTJ9eFRPBe-iZa&index=33&t=0sTGI Kubernetes 066 —https://www.youtube.com/watch?v=C-vRlW7VYio&list=PL7bmigfV0EqQzxcNpmcdTJ9eFRPBe-iZa&index=32&t=0sBitnami — https://bitnami.com/Target — https://www.target.com/Netflix — https://www.netflix.com/HashiCorp — https://www.hashicorp.com/Aqua Sec — https://www.aquasec.com/CyberArk — https://www.cyberark.com/Jeff Bezos — https://www.forbes.com/profile/jeff-bezos/#4c3104291b23Istio — https://istio.io/Linkerd — https://linkerd.io/ Transcript: EPISODE 10 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores cloud native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.2] NL: Hello and welcome back to The Kubelets Podcast. My name is Nicholas Lane and this time, we’re going to be talking about the dichotomy of security. And to talk about such an interesting topic, joining me are Duffie Coolie. [0:00:54.3] DC: Hey, everybody. [0:00:55.6] NL: Bryan Liles. [0:00:57.0] BM: Hello [0:00:57.5] NL: And Carlisia Campos. [0:00:59.4] CC: Glad to be here. [0:01:00.8] NL: So, how’s it going everybody? [0:01:01.8] DC: Great. [0:01:03.2] NL: Yeah, this I think is an interesting topic. Duffie, you introduced us to this topic. And basically, what I understand, what you wanted to talk about, we’re calling it the dichotomy of security because it’s the relationship between security, like hardening your application to protect it from attack and influence from outside actors and agility to be able to create something that’s useful, the ability to iterate as fast as possible. [0:01:30.2] DC: Exactly. I mean, the idea from this came from putting together a talks for the security conference coming up here in a couple of weeks. And I was noticing that obviously, if you look at the job of somebody who is trying to provide some security for applications on their particular platform, whether that be AWS or GCE or OpenStack or Kubernetes or anything of these things. It’s frequently in their domain to kind of define constraints for all of the applications that would be deployed there, right? Such that you can provide rational defaults for things, right? Maybe you want to make sure that things can’t do a particular action because you don’t want to allow that for any application within your platform or you want to provide some constraint around quota or all of these things. And some of those constraints make total sense and some of them I think actually do impact your ability to design the systems or to consume that platform directly, right? You’re not able to actually make use of the platform as it was designed to be made use of, when those constraints are too tight. [0:02:27.1] DC: Yeah. I totally agree. There’s kind of a joke that we have in certain tech fields which is the primary responsibility of security is to halt productivity. It isn’t actually true, right? But there are tradeoffs, right? If security is too tight, you can’t move forward, right? Example of this that kind of mind are like, if you’re too tight on your firewall rules where you can’t actually use anything of value. That’s a quick example of like security gone haywire. That’s too controlling, I think. [0:02:58.2] BM: Actually. This is an interesting topic just in general but I think that before we fall prey to what everyone does when they talk about security, let’s take a step back and understand why things are the way they are. Because all we’re talking about are the symptoms of what’s going on and I’ll give you one quick example of why I say this. Things are the way they are because we haven’t made them any better. In developer land, whenever we consume external resources, what we were supposed to do and what we should be doing but what we don’t do is we should create our internal interfaces. Only program to those interfaces and then let that interface of that adapt or talk to the external service and in security world, we should be doing the same thing and we don’t do this. My canonical example for this is IAM on AWS. It’s hard to create a secure IM configuration and it’s even harder to keep it over time and it’s even harder to do it whenever you have 150, 100, 5,000 people dealing with this. What companies do is they actually create interfaces where they could describe the part of IAM they want to use and then they translate that over. The reason I bring this up is because the reason that people are scared of security is because security is opaque and security is opaque because a lot of people like to keep it opaque. But it doesn’t have to be that way. [0:04:24.3] NL: That’s a good point, that’s a reasonable design and wherever I see that devoted actually is very helpful, right? Because you highlight a critical point in that these constraints have to be understood by the people who are constrained by them, right? It will just continue to kind of like drive that wedge between the people who are responsible for them top finding t hem and the people who are being affected by them, right? That transparency, I think it’s definitely key. [0:04:48.0] BM: Right, this is our cloud native discussion, any idea of where we should start thinking about this in cloud native land? [0:04:56.0] DC: For my part, I think it’s important to understand if you can like what the consumer of a particular framework or tool might need, right? And then, just take it from there and figure out what rational constraints are. Rather than the opposite which is frequently where people go and evaluate a set of rules as defined by some particular, some third-part company. Like you look at CIS packs and you look at like a lot of these other tooling. I feel like a lot of people look at those as like, these are the hard rules, we must comply to all of these things. Legally, in some cases, that’s the case. But frequently, I think they’re just kind of like casting about for some semblance of a way to start defining constraint and they go too far, they’re no longer taking into account what the consumers of that particular platform might meet, right? Kubernetes is a great example of this. If you look at the CIS spec for Kubernetes or if you look at a lot of the talks that I’ve seen kind of around how to secure Kubernetes, we defined like best practices for security and a lot of them are incredibly restrictive, right? I think of the problem there is that restriction comes at a cost of agility. You’re no longer able to use Kubernetes as a platform for developing microservices because you provided so much constraints that it breaks the model, you know? [0:06:12.4] NL: Okay. Let’s break this down again. I can think of a top of my head, three types of things people point to when I’m thinking about security. And spoiler alert, I am going to do some acronyms but don’t worry about the acronyms are, just understand they are security things. The first one I’ll bring up is FISMA and then I’ll think about NIST and the next one is CIS like you brought up. Really, the reason they’re so prevalent is because depending on where you are, whether you’re in a highly regulated place like a bank or you’re working for the government or you have some kind of automate concern to say a PIPA or something like that. These are the words that the auditors will use with you. There is good in those because people don’t like the CIS benchmarks because sometimes, we don’t understand why they’re there. But, from someone who is starting from nothing, those are actually great, there’s at least a great set of suggestions. But the problem is you have to understand that they’re only suggestions and they are trying to get you to a better place than you might need. But, the other side of this is that, we should never start with NIST or CIS or FISMA. What we really should do is our CISO or our Chief Security Officer or the person in charge of security. Or even just our – people who are in charge, making sure our stack, they should be defining, they should be taking what they know, whether it’s the standards and they should be building up this security posture in this security document and these rules that are built to protect whatever we’re trying to do. And then, the developers of whoever else can operate within that rather than everything literally. [0:07:46.4] DC: Yeah, agreed. Another thing I’ve spent some time talking to people about like when they start rationalizing how to implement these things or even just think about the secure surface or develop a threat model or any of those things, right? One of the things that I think it’s important is the ability to define kind of like what normal looks like, right? What normal access between applications or normal access of resources looks like. I think that your point earlier, maybe provides some abstraction in front of a secure resource such that you can actually just share that same fraction across all the things that might try to consume that external resource is a great example of the thing. Defining what that normal access looks like is critical to us to our ability to constrain it, right? I think that frequently people don’t start there, they start with the other side, they’re saying, here are all the constraints, you need to tell me which ones are too tight. You need to tell me which ones to loosen up so that you can do your job. You need to tell me which application needs access to whichever application so that I can open the firewall for you. I’m like, we need to turn that on its head. We need the environments that are perhaps less secure so that we can actually define what normal looks like and then take that definition and move it into a more secured state, perhaps by defining these across different environments, right? [0:08:58.1] BM: A good example of that would be in larger organizations, at every part of the organization does this but there is environments running your application where there are really no rules applied. What we do with that is we turn on auditing in those environments so you have two applications or a single application that talks to something and you let that application run and then after the application run, you go take a look at the audit logs and then you determine at that point what a good profile of this application is. Whenever it’s in production, you set up the security parameters, whether it be identity access or network, based on what you saw in auditing in your preproduction environment. That’s all you could run because we tested it fully in our preproduction environment, it should not do any more than that. And that’s actually something – I’ve seen tools that will do it for AWS IM. I’m sure you can do for anything else that creates auditing law. That’s a good way to get started. [0:09:54.5] NL: It sounds like what we’re coming to is that the breakdown of security or the way that security has impacted agility is when people don’t take a rational look at their own use case. instead, rely too much on the guidance of other people essentially. Instead of using things like the CIS benchmarking or NIST or FISMA, that’s one that I knew the other two and I’m like, I don’t know this other one. If they follow them less as guidelines and more as like hard set rules, that’s when we get impacts or agility. Instead of like, “Hey. This is what my application needs like you’re saying, let’s go from there.” What does this one look like? Duffie is for saying. I’m kind of curious, let’s flip that on its head a little bit, are there examples of times when agility impacts security? [0:10:39.7] BM: You want to move fast and moving fast is counter to being secure? [0:10:44.5] NL: Yes. [0:10:46.0] DC: Yeah, literally every single time we run software. When it comes down to is developers are going to want to develop and then security people are going to want to secure. And generally, I’m looking at it from a developer who has written security software that a lot of people have used, you guys had know that. Really, there needs to be a conversation, it’s the same thing as we had this dev ops conversation for a year – and then over the last couple of years, this whole dev set ops conversation has been happening. We need to have this conversation because from a security person’s point of view, you know, no access is great access. No data, you can’t get owned if you don’t have any data going across the wire. You know what? Can’t get into that server if there’s no ports opened. But practically, that doesn’t work and we find is that there is actually a failing on both sides to understand what the other person was optimizing for. [0:11:41.2] BM: That’s actually where a lot of this comes from. I will offer up that the only default secure posture is no access to anything and you should be working from that direction to where you want to be rather than working from, what should we close down? You should close down everything and then you work with allowing this list for other than block list. [0:12:00.9] NL: Yeah, I agree with that model but I think that there’s an important step that has to happen before that and that’s you know, the tooling or thee wireless phone to define what the application looks like when it’s in a normal state or the running state and if we can accomplish that, then I feel like we’re in a better position to find what that LOI list looks like and I think that one of the other challenges there of course, let’s backup for a second. I have actually worked on a platform that supported many services, hundreds of services, right? Clearly, if I needed to define what normal looked like for a hundred services or a thousand services or 2,000 services, that’s going to be difficult in a way that people approach the problem, right? How do you define for each individual service? I need to have some decoration of intent. I need the developer to engage here and tell me, what they’re expecting, to set some assumptions about the application like what it’s going to connect to, those dependences are – That sort of stuff. And I also need tooling to verify that. I need to be able to kind of like build up the whole thing so that I have some way of automatically, you know, maybe with oversight, defining what that security context looks like for this particular service on this particular platform. Trying to do it holistically is actually I think where we get into trouble, right? Obviously, we can’t scale the number of people that it takes to actually understand all of these individual services. We need to actually scale this stuff as software problem instead. [0:13:22.4] CC: With the cloud native architecture and infrastructure, I wonder if it makes it more restrictive because let’s say, these are running on Kubernetes, everything is running at Kubernetes. Things are more connected because it’s a Kubernetes, right? It’s this one huge thing that you’re running on and Kubernetes makes it easier to have access to different notes and when the nodes took those apart, of course, you have to find this connection. Still, it’s supposed to make it easy. I wonder if security from a perspective of somebody, needing to put a restriction and add miff or example, makes it harder or if it makes it easier to just delegate, you have this entire area here for you and because your app is constrained to this space or name space or this part, this node, then you can have as much access as you need, is there any difference? Do you know what I mean? Does it make sense what I said? [0:14:23.9] BM: There was actually, it’s exactly the same thing as we had before. We need to make sure that applications have access to what they need and don’t have access to what they don’t need. Now, Kubernetes does make it easier because you can have network policies and you can apply those and they’re easier to manage than who knows what networking management is holding you have. Kubernetes also has pod security policies which again, actually confederates this knowledge around my pod should be able to do this or should not be able to run its root, it shouldn’t be able to do this and be able to do that. It’s still the same practice Carlisia, but the way that we can control it is now with a standard set off tools. We still have not cracked the whole nut because the whole thing of turning auditing on to understand and then having great tool that can read audit locks from Kubernetes, just still aren’t there. Just to add one more last thing that before we add VMWare and we were Heptio, we had a coworker who wrote basically dynamic audit and that was probably one of the first steps that we would need to be able to employ this at scale. We are early, early, super early in our journey and getting this right, we just don’t have all the necessary tools yet. That’s why it’s hard and that’s why people don’t do it. [0:15:39.6] NL: I do think it is nice to have t hose and primitives are available to people who are making use of that platform though, right? Because again, kind of opens up that conversation, right? Around transparency. The goal being, if you understood the tools that we’re defining that constraint, perhaps you’d have access to view what the constraints are and understand if they’re actually rational or not with your applications. When you’re trying to resolve like I have deployed my application in dev and it’s the wild west, there’s no constraints anywhere. I can do anything within dev, right? When I’m trying to actually promote my application to staging, it gives you some platform around which you can actually sa, “If you want to get to staging, I do have to enforce these things and I have a way and again, all still part of that same API, I still have that same user experience that I had when just deploying or designing the application to getting them deployed.” I could still look at again and understand what the constraints are being applied and make sure that they’re reasonable for my application. Does my application run, does it have access to the network resources that it needs to? If not, can I see where the gaps are, you know? [0:16:38.6] DC: For anyone listening to this. Kubernetes doesn’t have all the documentation we need and no one has actually written this book yet. But on Kubernetes.io, there are a couple of documents about security and if we have shownotes, I will make sure those get included in our shownotes because I think there are things that you should at least understand what’s in a pod security policy. You should at least understand what’s in a network security policy. You should at least understand how roles and role bindings work. You should understand what you’re going to do for certificate management. How do you manage this certificate authority in Kubernetes? How do you actually work these things out? This is where you should start before you do anything else really fancy. At least, understand your landscape. [0:17:22.7] CC: Jeffrey did a TGI K talk on secrets. I think was that a series? There were a couple of them, Duffie? [0:17:29.7] DC: Yeah, there were. I need to get back and do a little more but yeah. [0:17:33.4] BM: We should then add those to our shownotes too. Hopefully they actually exist or I’m willing to see to it because in assistance. [0:17:40.3] CC: We are going to have shownotes, yes. [0:17:44.0] NL: That is interesting point, bringing up secrets and secret management and also, like secured Inexhibit. There are some tools that exist that we can use now in a cloud native world, at least in the container world. Things like vault exist, things like well, now, KBDM you can roll certificate which is really nice. We are getting to a place where we have more tooling available and I’m really happy about it. Because I remember using Kubernetes a year ago and everyone’s like, “Well. How do you secure a secret in Kubernetes?” And I’m like, “Well, it sure is basics for you to encode it. That’s on an all secure.” [0:18:15.5] BM: I would do credit Bitnami has been doing sealed secrets, that’s been out for quite a while but the problem is that how do you suppose to know about that and how are you supposed to know if it’s a good standard? And then also, how are you supposed to benchmark against that? How do you know if your secrets are okay? We haven’t talked about the other side which is response or detection of issues. We’re just talking about starting out, what do you do? [0:18:42.3] DC: That’s right. [0:18:42.6] NL: It is tricky. We’re just saying like, understanding all the avenues that you could be impacted is kind of a daunting task. Let’s talk about like the Target breach that occurred a few years ago? If anybody doesn’t remember this, basically, Target had a huge credit card breach from their database and basically, what happened is that t heir – If I recalled properly, their OIDC token had a – not expired but the audience for it was so broad that someone had hacked into one computer essentially like a register or something and they were able to get the OIDC token form the local machine. The authentication audience for that whole token was so broad that they were able to access the database that had all of the credit card information into it. These are one of these things that you don’t think about when you’re setting up security, when you’re just maybe getting started or something like that. What are the avenues of attack, right? You’d say like, “OIDC is just pure authentication mechanism, why would we need to concern ourselves with this?” And then but not understanding kind of what we were talking about last because the networking and the broadcasting, what is the blast radius of something like this and so, I feel like this is a good example of sometimes security can be really hard and getting started can be really daunting. [0:19:54.6] DC: Yeah, I agree. To Bryan’s point, it’s like, how do you test against this? How do you know that what you’ve defined is enough, right? We can define all of these constraints and we can even think that they’re pretty reasonable or rational and the application may come up and operate but how do you know? How can you verify that? What you’ve done is enough? And then also, remember. With OIDC has its own foundations and loft. You realize that it’s a very strong door but it’s only a strong door, it also do things that you can’t walk around a wall and that it’s protecting or climb over the wall that it’s protecting. There’s a bit of trust and when you get into things like the target breach, you really have to understand blast radius for anything that you’re going to do. A good example would be if you’re using shared key kind of things or like public share key. You have certificate authorities and you’re generating certificates. You should probably have multiple certificate authorities and you can have a basically, a hierarchy of these so you could have basically the root one controlled by just a few people in security. And then, each department has their own certificate authority and then you should also have things like revocation, you should be able to say that, “Hey, all this is bad and it should all go away and it probably should have every revocation list,” which a lot of us don’t have believe it or not, internally. Where if I actually kill our own certificate, a certificate was generated and I put it in my revocation list, it should not be served and in our clients that are accepting that our service is to see that, if we’re using client side certificates, we should reject these instantly. Really, what we need to do is stop looking at security as this one big thing and we need to figure out what are our blast radius. Firecracker, blowing up in my hand, it’s going to hurt me. But Nick, it’s not going to hurt you, you know? If someone drops in a huge nuclear bomb on the United States or the west coast United States, I’m talking to myself right now. You got to think about it like that. What’s the worst that can happen if this thing gets busted or get shared or someone finds that this should not happen? Every piece off data that you have that you consider secure or sensitive, you should be able to figure out what that means and that is how whenever you are defining a security posture that’s butchered to me. Because that is why you’ll notice that a lot of companies some of them do run open within a contained zone. So, within this contained zone you could talk to whomever you want. We don’t actually have to be secure here because if we lose one, we lost them all so who cares? So, we need to think about that and how do we do that in Kubernetes? Well, we use things like name spaces first of all and then we use things like this network policies and then we use things like pod security policies. We can lock some access down to just name spaces if need be. You can only talk to pods and your name space. And I am not telling you how to do this but you need to figure out talking with your developer, talking to the security people. But if you are in security you need to talk to your product management staff and your software engineering staff to figure out really how does this need to work? So, you realize that security is fun and we have all sorts of neat tools depending on what side you’re on. You know if you are on red team, you’re half knee in, you’re blue team you are saving things. We need to figure out these conversations and tooling comes from these conversations but we need to have these conversation first. [0:23:11.0] DC: I feel like a little bit of a broken record on this one but I am going to go back to chaos engineering again because I feel like it is critical to stuff like this because it enables a culture in which you can explore both the behavior of applications itself but why not also use this model to explore different ways of accessing that information? Or coming up with theories about the way the system might be vulnerable based on a particular attack or a type of attack, right? I think that this is actually one of the movements within our space that I think provides because then most hope in this particular scenario because a reasonable chaos engineering practice within an organization enables that ability to explore all of the things. You don’t have to be red team or blue team. You can just be somebody who understands this application well and the question for the day is, “How can we attack this application?” Let’s come up with theories about the way that perhaps this application could be attacked. Think about the problem differently instead of thinking about it as an access problem, think about it as the way that you extend trust to the other components within your particular distributed system like do they have access that they don’t need. Come up with a theory around being able to use some proxy component of another system to attack yet a third system. You know start playing with those ideas and prove them out within your application. A culture that embraces that I think is going to be by far a more secure culture because it lets developers and engineers explore these systems in ways that we don’t generally explore them. [0:24:36.0] BM: Right. But also, if I could operate on myself I would never need a doctor. And the reason I bring that up is because we use terms like chaos engineering and this is no disrespect to you Duffie, so don’t take it as this is panacea or this idea that we make things better and true. That is fine, it will make us better but the little secret behind chaos engineering is that it is hard. It is hard to build these experiments first of all, it is hard to collect results from these experiments. And then it is hard to extrapolate what you got out of the experiments to apply to whatever you are working on to repeat and what I would like to see is what people in our space is talking about how we can apply such techniques. But whether it is giving us more words or giving us more software that we can employ because I hate to say it, it is pretty chaotic in chaos engineering right now for Kubernetes. Because if you look at all the people out there who have done it well. And so, you look at what Netflix has done with pioneering this and then you listen to what, a company such us like Gremlin is talking about it is all fine and dandy. You need to realize that it is another piece of complexity that you have to own and just like any other things in the security world, you need to rationalize how much time you are going to spend on it first is the bottom line because if I have a “Hello, World!” app, I don’t really care about network access to that. Unless it is a “Hello, World!” app running on the same subnet as some doing some PCI data then you know it is a different conversation. [0:26:05.5] DC: Yeah. I agree and I am certainly not trying to version as a panacea but what I am trying to describe is that I feel like I am having a culture that embraces that sort of thinking is going to enable us to be in a better position to secure these applications or to handle a breach or to deal with very hard to understand or resolve problems at scale, you know? Whether that is a number of connections per second or whether that is a number of applications that we have horizontally scaled. You know like being able to embrace that sort of a culture where we asked why where we say “well, what if…” or if we actually come up you know embracing the idea of that curiosity that got you into this field, you know what I mean like the thing that is so frequently our cultures are opposite of that, right? It becomes a race to the finish and in that race to the finish, lots of pieces fall off that we are not even aware of, you know? That is what I am highlighting here when I talk about it. [0:26:56.5] NL: And so, it seems maybe the best solution to the dichotomy between security and agility is really just open conversation, in a way. People actually reaching across the aisle to talk to each other. So, if you are embracing this culture as you are saying Duffie the security team should be having constant communication with the application team instead of just like the team doing something wrong and the security team coming down and smacking their hand. And being like, “Oh you can’t do it this way because of our draconian rules” right? These people are working together and almost playing together a little bit inside of their own environment to create also a better environment. And I am sorry.I didn’t mean to cut you off there, Bryan. [0:27:34.9] BM: Oh man, I thought it was fleeting like all my thoughts. But more about what you are saying is, is that you know it is not just more conversations because we can still have conversations and I am talking about sider and subnets and attack vectors and buffer overflows and things like that. But my developer isn’t talking, “Well, I just need to be able to serve this data so accounting can do this.” And that’s what happens a lot in security conversations. You have two groups of individuals who have wholly different goals and part of that conversation needs to be aligning or jargon and then aligning on those goals but what happens with pretty much everything in the development world, we always bring our networking, our security and our operations people in right at the end, right when we are ready to ship, “Hey make this thing work.” And really it is where a lot of our problems come out. Now security either could or wanted to be involved at the beginning of a software project what we actually are talking about what we are trying to do. We are trying to open up this service to talk to this, share this kind of data. Security can be in there early saying, “Oh no you know, we are using this resource in our cloud provider. It doesn’t really matter what cloud provider and we need to protect this. This data is sitting here at rest.” If we get those conversations earlier, it would be easier to engineer solutions that to be hopefully reused so we don’t have to have that conversation in the future. [0:29:02.5] CC: But then it goes back to the issue of agility, right? Like Duffie was saying, wow you can develop, I guess a development cluster which has much less restrictive restrictions and they move to a production environment where the proper restrictions are then – then you find out or maybe station environment let’s say. And then you find out, “Oh whoops. There are a bunch of restrictions I didn’t deal with but I didn’t move a lot faster because I didn’t have them but now, I have to deal with them.” [0:29:29.5] DC: Yeah, do you think it is important to have a promotion model in which you are able to move toward a more secure deployment right? Because I guess a parallel to this is like I have heard it said that you should develop your monolith first and then when you actually have the working prototype of what you’re trying to create then consider carefully whether it is time to break this thing up into a set of distinct services, right? And consider carefully also what the value of that might be? And I think that the reason that that’s said is because it is easier. It is going to be a lower cognitive load with everything all right there in the same codebase. You understand how all of these pieces interconnect and you can quickly develop or prototype what you are working on. Whereas if you are trying to develop these things into individual micro services first, it is harder to figure out where the line is. Like where to divide all of the business logic. I think this is also important when you are thinking about the security aspects of this right? Being able to do a thing when which you are not constrained, define all of these services and your application in the model for how they communicate without constraint is important. And once you have that when you actually understand what normal looks like from that set of applications then enforce them, right? If you are able to declare that intent you are going to say like these are the ports on the list on for these things, these are the things that they are going to access, this is the way that they are going to go about accessing them. You know if you can declare that intent then that is actually that is a reasonable body of knowledge for which the security people can come along and say, “Okay well, you have told us. You informed us. You have worked with us to tell us like what your intent is. We are going to enforce that intent and see what falls out and we can iterate there.” [0:31:01.9] CC: Yeah everything you said makes sense to me. Starting with build the monolith first. I mean when you start out why which ones will have abstract things that you don’t really – I mean you might think you know but you’re only really knowing practice what you are going to need to abstract. So, don’t abstract things too early. I am a big fan of that idea. So yeah, start with the monolith and then you figure out how to break it down based on what you need. With security I would imagine the same idea resonates with me. Don’t secure things that you don’t need you don’t know just yet that needs securing except the deal breaker things. Like there is some things we know like we don’t want production that are being accessed some types of production that are some things we know we need to secure so from the beginning. [0:31:51.9] BM: Right. But I will still iterate that it is always denied by default, just remember that. It is security is actually the opposite way. We want to make sure that we have the least amount and even if it is harder for us you always want to start with un-allowed TCP communication on port 443 or UDP as well. That is what I would allow rather than saying shut everything else off. But this, I would rather have the way that we only allow that and that also goes in with our declarative nature in cloud native things we like anyways. We just say what we want and everything else doesn’t exists. [0:32:27.6] DC: I do want to clarify though because I think what you and I, we are the representative of the dichotomy right at this moment, right? I feel like what you are saying is the constraint should be the normal, being able to drop all traffic, do not allow anything is normal and then you have to declare intent to open anything up and what I am saying is frequently developers don’t know what normal looks like yet. They need to be able to explore what normal looks like by developing these patterns and then enforce them, right, which is turning the model on its head. And this is actually I think the kernel that I am trying to get to in this conversation is that there has to be a place where you can go play and learn what normal is and then you can move into a world in which you can actually enforce what that normal looks like with reasonable constraint. But until you know what that is, until you have that opportunity to learn it, all we are doing here is restricting your ability to learn. We are adding friction to the process. [0:33:25.1] BM: Right, well I think what I am trying to say here layer on top of this is that yes, I agree but then I understand what a breach can do and what bad security can do. So I will say, “Yeah, go learn. Go play all you want but not on software that will ever make it to production. Go learn these practices but you are going to have to do it outside of” – you are going to have a sandbox and that sandbox is going to be unconnected from the world I mean from our obelisk and you are going to have to learn but you are not going to practice here. This is not where you learn how to do this. [0:33:56.8] NL: Exactly right, yeah. You don’t learn to ride a motorcycle on the street you know? You’d learn to ride a motorcycle on the dirt and then you could take those skills later you know? But yeah I think we are in agreement like production is a place where we do have to enforce all of those things and having some promotion level in which you can come from a place where you learned it to a place where you are beginning to enforce it to a place where it is enforced I think is also important. And I frequently describe this as like development, staging and production, right? Staging is where you are going to hit the edges from because this is where you’re actually defining that constraint and it has to be right before it can be promoted to production, right? And I feel like the middle ground is also important. [0:34:33.6] BM: And remember that production is any environment production can reach. Any environment that can reach production is production and that is including that we do data backup dumps and we clean them up from production and we use it as data in our staging environment. If production can directly reach staging or vice versa, it is all production. That is your attack vector. That is also what is going to get in and steal your production data. [0:34:59.1] NL: That is absolutely right. Google actually makes an interesting not of caveat to that but like side point to that where like if I understand the way that Google runs, they run everything in production, right? Like dev, staging and production are all the same environment. I am more positing this is a question because I don’t know if anybody of us have the answer but I wonder how they secure their infrastructure, their environment well enough to allow people to play to learn these things? And also, to deploy production level code all in the same area? That seems really interesting to be and then if I understood that I probably would be making a lot more money. [0:35:32.6] BM: Well it is simple really. There were huge people process at Google that access gatekeeper for a lot of these stuff. So, I have never worked in Google. I have no intrinsic knowledge of Google or have talked to anyone who has given me this insight, this is all speculation disclaimer over. But you can actually run a big cluster that if you can actually prove that you have network and memory and CPU isolation between containers, which they can in certain cases and certain things that can do this. What you can do is you can use your people process and your approvals to make sure that software gets to where it needs to be. So, you can still play on the same clusters but we have great handles on network that you can’t talk to these networks or you can’t use this much network data. We have great things on CPU that this CPU would be a PCI data. We will not allow it unless it’s tied to CPU or it is PCI. Once you have that in place, you do have a lot more flexibility. But to do that, you will have to have some pretty complex approval structures and then software to back that up. So, the burden on it is not on the normal developer and that is actually what Google has done. They have so many tools and they have so many processes where if you use this tool it actually does the process for you. You don’t have to think about it. And that is what we want our developers to be. We want them to be able to use either our networking libraries or whenever they are building their containers or their Kubernetes manifest, use our tools and we will make sure based on either inspection or just explicit settings that we will build something that is as secure as we can given the inputs. And what I am saying is hard and it is capital H hard and I am actually just pitting where we want to be and where a lot of us are not. You know most people are not there. [0:37:21.9] NL: Yeah, it would be nice if we had like we said earlier like more tooling around security and the processes and all of these things. One thing I think that people seem to balk on or at least I feel is developing it for their own use case, right? It seems like people want an overarching tool to solve all the use cases in the world. And I think with the rise of cloud native applications and things like container orchestration, I would like to see people more developing for themselves around their own processes, around Kubernetes and things like that. I want to see more perspective into how people are solving their security problems, instead of just like relying on let’s say like HashiCorp or like Aqua Sec to provide all the answers like I want to see more answers of what people are doing. [0:38:06.5] BM: Oh, it is because tools like Vault are hard to write and hard to maintain and hard to keep correct because you think about other large competitors to vault and they are out there like tools like CyberArk. I have a secret and I want to make sure only certain will keep it. That is a very difficult tool but the HashiCorp advantage here is that they have made tools to speak to people who write software or people who understand ops not just as a checkbox. It is not hard to get. If you are using vault it is not hard to get a secret out if you have the right credentials. Other tools is super hard to get the secret out if you even have the right credential because they have a weird API or they just make it very hard for you or they expect you to go click on some gooey somewhere. And that is what we need to do. We need to have better programming interfaces and better operator interfaces, which extends to better security people are basis for you to use these tools. You know I don’t know how well this works in practice. But the Jeff Bezos, how teams at AWS or Amazon or forums, you know teams communicate on API and I am not saying that you shouldn’t talk, but we should definitely make sure that our API’s between teams and team who owns security stuff and teams who are writing developer stuff that we can talk on the same level of fidelity that we can having an in person conversation, we should be able to do that through our software as well. Whether that be for asking for ports or asking for our resources or just talking about the problem that we have that is my thought-leadering answer to this. This is “Bryan wants to be a VP of something one day” and that is the answer I am giving. I’m going to be the CIO that is my CIO answer. [0:39:43.8] DC: I like it. So cool. [0:39:45.5] BM: Is there anything else on this subject that we wanted to hit? [0:39:48.5] NL: No, I think we have actually touched on pretty much everything. We got a lot out of this and I am always impressed with the direction that we go and I did not expect us to go down this route and I was very pleased with the discussion we have had so far. [0:39:59.6] DC: Me too. I think if we are going to explore anything else that we talked about like you know, get it more into that state where we are talking about like that we need more feedback loops. We need people developers to talk to security people. We need security people talk to developers. We need to have some way of actually pushing that feedback loop much like some of the other cultural changes that we have seen in our industry are trying to allow for better feedback loops and other spaces. And you’ve brought up dev spec ops which is another move to try and open up that feedback loop but the problem I think is still going to be that even if we improved that feedback loop, we are at an age where – especially if you ended up in some of the larger organizations, there are too many applications to solve this problem for and I don’t know yet how to address this problem in that context, right? If you are in a state where you are a 20-person, 30-person security team and your responsibility is to secure a platform that is running a number of Kubernetes clusters, a number of Vsphere clusters, a number of cloud provider implementations whether that would be AWS or GC, I mean that is a set of problems that is very difficult. It is like I am not sure that improving the feedback loop really solves it. I know that I helps but I definitely you know, I have empathy for those folks for sure. [0:41:13.0] CC: Security is not my forte at all because whenever I am developing, I have a narrow need. You know I have to access a cluster.I have to access a machine or I have to be able to access the database. And it is usually a no brainer but I get a lot of the issues that were brought up. But as a builder of software, I have empathy for people who use software, consume software, mine and others and how can’t they have any visibility as far as security goes? For example, in the world of cloud native let’s say you are using Kubernetes, I sort of start thinking, “Well, shouldn’t there be a scanner that just lets me declare?” I think I am starting an episode right now –should there be a scanner that lets me declare for example this node can only access this set of nodes like a graph. But you just declare and then you run it periodically and you make sure of course this goes down to part of an app can only access part of the database. It can get very granular but maybe at a very high level I mean how hard can this be? For example, this pod can only access that pods but this pod cannot access this name space and just keep checking what if the name spaces changes, the permission changes. Or for example would allow only these answers can do a backup because they are the same users who will have access to the restore so they have access to all the data, you know what I mean? Just keep checking that is in place and it only changes when you want to. [0:42:48.9] BM: So, I mean I know we are at the end of this call and I want to start a whole new conversation but this is actually is why there are applications out there like Istio and Linkerd. This is why people want service meshes because they can turn off all network access and then just use the service mesh to do the communication and then they can use, they can make sure that it is encrypted on both sides and that is a honey cave on all both sides. That is why this is operated. [0:43:15.1] CC: We’ll definitely going to have an episode or multiple on service mesh but we are on the top of the hour. Nick, do your thing. [0:43:23.8] NL: All right, well, thank you so much for joining us on another interesting discussion at The Kubelets Podcast. I am Nicholas Lane, Duffie any final thoughts? [0:43:32.9] DC: There is a whole lot to discuss, I really enjoyed our conversations today. Thank you everybody. [0:43:36.5] NL: And Bryan? [0:43:37.4] BM: Oh it was good being here. Now it is lunch time. [0:43:41.1] NL: And Carlisia. [0:43:42.9] CC: I love learning from you all, thank you. Glad to be here. [0:43:46.2] NL: Totally agree. Thank you again for joining us and we’ll see you next time. Bye. [0:43:51.0] CC: Bye. [0:43:52.1] DC: Bye. [0:43:52.6] BM: Bye. [END OF EPISODE] [0:43:54.7] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

This week on The Podlets Cloud Native Podcast we have Josh, Carlisia, Duffie, and Nick on the show, and are also happy to be joined by a newcomer, Brian Liles, who is a senior staff engineer at VMWare! The purpose of today’s show is coming to a deeper understanding of the meaning of ‘stateful’ versus ‘stateless’ apps, and how they relate to the cloud native environment. We cover some definitions of ‘state’ initially and then move to consider how ideas of data persistence and co-ordination across apps complicate or elucidate understandings of ‘stateful’ and ‘stateless’. We then think about the challenging practice of running databases within Kubernetes clusters, which effectively results in an ephemeral system becoming stateful. You’ll then hear some clarifications of the meaning of operators and controllers, the role they play in mediating and regulating states, and also how important they are in a rapidly evolving but skills-scarce environment. Another important theme in this conversation is the CAP theorem or the impossibility of consistency, availability and partition tolerance all at once, but the way different databases allow for different combinations of two out of the three. We then move on to chat about the fundamental connection between workloads and state and then end off with a quick consideration about how ideas of stateful and stateless play out in the context of networks. Today’s show is a real deep dive offering perspectives from some the most knowledgeable in the cloud native space so make sure to tune in! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Duffie Cooley Bryan Liles Josh Rosso Nicholas Lane Key Points From This Episode: • What ‘stateful’ means in comparison to ‘stateless’.• Understanding ‘state’ as a term referring to data which must persist.• Examples of stateful apps such as databases or apps that revolve around databases.• The idea that ‘persistence’ is debatable, which then problematizes the definition of ‘state’. • Considerations of the push for cloud native to run stateless apps.• How inter-app coordination relates to definitions of stateful and stateless applications.• Considering stateful data as data outside of a stateless cloud native environment.• Why it is challenging to run databases in Kubernetes clusters.• The role of operators in running stateful databases in clusters.• Understanding CRDs and controllers, and how they relate to operators.• Controllers mediate between actual and desired states.• Operators are codified system administrators.• The importance of operators as app number grows in a skill-scarce environment.• Mechanisms around stateful apps are important because they ensure data integrity.• The CAP theorem: the impossibility of consistency, availability, and tolerance.• Why different databases allow for different iterations of the CAP theorem.• When partition tolerance can and can’t get sacrificed.• Recommendations on when to run stateful or stateless apps through Kubernetes.• The importance of considering models when thinking about how to run a stateful app.• Varying definitions of workloads.• Pods can run multiple workloads• Workloads create states, so you can’t have one without the other.• The term ‘workloads’ can refer to multiple processes running at once.• Why the ephemerality of Kubernetes systems makes it hard to run stateful applications. • Ideas of stateful and stateless concerning networks.• The shift from server to browser in hosting stateful sessions. Quotes: “When I started envisioning this world of stateless apps, to me it was like, ‘Why do we even call them apps? Why don’t we just call them a process?’” — @carlisia [0:02:60] “‘State’ really is just that data which must persist.” — @joshrosso [0:04:26] “From the best that I can surmise, the operator pattern is the combination of a CRD plus a controller that will operate on events from the Kubernetes API based on that CRD’s configuration.” — @bryanl [0:17:00] “Once again, don’t let developers name them anything.” — @bryanl [0:17:35] “Data integrity is so important” — @apinick [0:22:31] “You have to really be careful about the different models that you’re evaluating when trying to think about how to manage a stateful application like a database.” — @mauilion [0:31:34] Links Mentioned in Today’s Episode: KubeCon+CloudNativeCon — https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2019/Google Spanner — https://cloud.google.com/spanner/CockroachDB — https://www.cockroachlabs.com/CoreOS — https://coreos.com/Red Hat — https://www.redhat.com/enMetacontroller — https://metacontroller.app/Brandon Philips — https://www.redhat.com/en/blog/authors/brandon-phillipsMySQL — https://www.mysql.com/ Transcript: EPISODE 009 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [INTERVIEW] [00:00:41] JR: All right! Hello, everybody, and welcome to episode 6 of The Cubelets Podcast. Today we are going to be discussing the concept of stateful and stateless and what that means in this crazy cloud native landscape that we all work. I am Josh Rosso. Joined with me today is Carlisia. [00:00:59] CC: Hi, everybody. [00:01:01] JR: We also have Duffie. [00:01:03] D: Hey, everybody. [00:01:04] JR: Nicholas. [00:01:05] NL: Yo! [00:01:07] JR: And a newcomer to the podcast, we also have Brian. Brian, you want to give us a little intro about yourself? [00:01:12] BL: Hi! I’m Brian. I work at VMWare. I do lots of community stuff, including sharing the KubeCon+CloudNativeCon. [00:01:22] JR: Awesome! Cool. All right. We’ve got a pretty good cast this week. So let’s dive right into it. I think one of the first things that we’ve been talking a bit about is the concept of what makes an application stateful? And of course in reverse, what makes an application stateless? Maybe we could try to start by discerning those two. Maybe starting with stateless if that makes? Does someone want to take that on? [00:01:45] CC: Well, I’m going to jump right in. I have always been a developer, as supposed to some of you or all of you have who have system admin backgrounds. The first time that I heard the stateless app, I was like, “What?” That wasn’t recent, okay? It was a long time ago, but that was a knot in my head. Why would you have a stateless app? If you have an app, you’re going to need state. I couldn’t imagine what that was. But of course it makes a lot of sense now. That was also when we were more in the monolithic world. [00:02:18] BM: Actually that’s a good point. Before you go into that, it’s a great point. Whenever we start with apps or we start developing apps, we think of an application. An application does everything. It takes input and it does stuff and it gives output. But now in this new world where we have lots of apps, big apps, small apps, we start finding that there’s apps that only talk and coordinate with other apps. They don’t do anything else. They don’t save any data. They don’t do anything. That’s what – where we get into this thing called stateless apps. Apps don’t have any type of data that they store locally. [00:02:53] CC: Yeah. It’s more like when I envision in my head. You said it brilliantly, Brian. It’s almost like a process. When I started envisioning this world of stateless apps, to me it was like, “Why do we even call them apps? Why don’t we just call them a process?” They’re just shifting back data and forth but they’re not – To me, at the beginning, apps were always stateless. They went together. [00:03:17] D: I think, frequently, people think of applications that have only locally relevant stuff that is actually not going to persist to disc, but maybe held in memory or maybe only relevant to the type of connection that’s coming through that application also as stateless, which is interesting, because there’s still some state there, but the premise is that you could lose that state and not lose the functionality of that code. [00:03:42] NL: Something that we might want to dive into really quickly when talking about stateless and stateful apps. What do we mean by the word state? When I first learned about these things, that was what always screwed me up. I’m like, “What do you mean state? Like Washington? Yeah. We got it over here.” [00:03:57] JR: Oh! State. That’s that word. State is one of those words that we use to sound smarter than we actually are 95% of the time, and that’s a number I just made up. When people are talking about state, they mean databases. Yeah. But there are other types of state as well. If you maintain local cache that needs to be persistent, if you have local files that you’re dealing with, like you’re opening files. That’s still state. State really is just that it’s data that must persist. [00:04:32] D: I agree with that definition. I think that state, whether persisted to memory or persisted to disc or persisted to some external system, that’s still what we refer to as state. [00:04:41] JR: All right. Makes sense and sounds about like what I got from it as well. [00:04:45] CC: All right. So now we have this world where we talk about stateless apps and stateful apps. Are there even stateful apps? Do we call a database an app? If we have a distributed system where we have one stateless app over here, another stateless app over there and then we have the database that’s connected to the two of them, are we calling the database a stateful app or is that whole thing – How do we call this? [00:05:15] NL: Yeah. The database is very much a state as an app with state. I’m very much – [00:05:19] D: That’s a close definition. Yeah. [00:05:21] NL: Yeah. Literally, it’s the epitome of a stateful app. But then you also have these apps that talk to databases as well and they might have local data, like data that – they start a transaction and then complete it or they have a long distributed type transaction. Any apps that revolve around a database, if they store local data, whether it’s within a transaction or something else, they’re still stateful apps. [00:05:46] D: Yup. I think you can modify and input data or modify state that has to be persisted in some way I think is a stateful app, even though I do think it’s confusing because of what – As I said before, I think that there are a bunch of applications that we think of, like not everybody considers Spark jobs to be stateful. Spark jobs, for example, are something that would bring data in, mutate that data in some way, produce some output and go away. The definition there is that Spark would generally push the resulting data into some other external system. It’s interesting, because in that model, Spark is not considered to be a stateful app because the Spark job could fail, go away, get recreated, pick up the pieces where it left off or just redo that work until all of the work is done. In many cases, people consider that to be a stateless application. That’s I think is like the crux – In my opinion, the crux of the confusion around what a stateful and stateless application is, is that people frequently – I think it’s more about where you store – what you mean by persistence and how that actually realizes in your application. If you’re pushing your state to an external database, is your application still stateful? [00:06:58] NL: I think it’s a good question, or if you are gathering data from an external source and mutating it in some way, but you don’t need data to be present when you start up, is that a stateful app or a stateless app? Even though you are taking in data, modifying it and checking it, sending out to some other mechanism or serving it in your own way, does that become like a stateless app? If that app gets killed and it comes back and it’s able to recover, is it stateful or stateless? That’s a bit of a gray area, I think. [00:07:26] JR: Yeah. I feel like a lot of the customers I work with, if the application can get killed even if it has some type of local state, they still refer to it as stateless usually, to me at least, when we talk about it because they think, “I can kind of restart this application and I’m not too worried about losing whatever it may have had.” Let’s say cached for simplicity, right? I think that kind of leads us into an interesting question. We’ve talked a lot on this podcast about cloud native infrastructure and cloud native applications and it seems like since the inception of cloud native, there’s always been this push that a stateless app is the best candidate to run or the easiest candidate to run. I’m just curious if we could dive into that for a moment. Why in the cloud native infrastructure area has there always been this push for running stateless applications? Why is it simpler? Those kinds of things. [00:08:15] BL: Before we dive into that, we have to realize – And this is just a problem of our whole ecosystem, this whole cloud native. We’re very hand-wavy in our descriptions for things. There’re a lot of ambiguous descriptions, and state is one of those. Just keep that in mind, that when we’re talking today, we’re really just talking about these things that store data and when that’s the state. Just keep that in mind as you’re listening to this. But when it comes to distributed systems in general, the easiest system is a system that doesn’t need coordination with any other system. If it happens to die, that’s okay. We can just restart it. People like to start there. It’s the easiest thing to start. [00:08:58] NL: Yeah, that was basically what I was going to say. If your application needs to tie into other applications, it becomes significantly more complicated to implement it, at least for your first time and in your system. These small applications that only – They don’t care about anybody else, they just take in data or not, they just do whatever. Those are super easy to start with because they’re just like, “Here. Start this up. Who cares? Whatever happens, it happens.” [00:09:21] CC: That could be a good boundary to define – I don’t want to jump back too far, but to define where is the stateless app to me is part of a system and just say it depends for it to come back up. Does it depend on something else that has state? [00:09:39] BL: I’ll give you an example. I can give you a good example of a stateless app that we use every day, every single one of us, none of us on this call, but when you search Google. You go to google.com and you go to the bar and you type in a search, what’s happening is there is a service at the beginning that collects that search and it federates the search over many different probably clusters of computers so they can actually do the search currently. That app that actually coordinates all that work is a stateless app most likely. All it does is just splits it up and allows more CPUs to do the work. Probably, that goes away. Probably not a problem. You probably have 10 more of them. That’s what I consider stateless. It doesn’t really own any of the data. It’s the coordinator. [00:10:25] CC: Yeah. If it goes down, it comes back up. It doesn’t need to reset itself to the state where it was before. It can truly be considered a stateless because it can just, “Okay. I reset. I’m starting from the beginning from this clear state.” [00:10:43] BL: Yes. That’s a good summary of that. [00:10:45] CC: Because another way to think about stateless – What makes an app stateful app, does it have to be combined or like deployed and shipped together with the part that maintains the state? That’s a more clear cut definition. Then that app is definitely a stateful app. [00:11:05] D: What we frequently talk about in like the cloud native space is like you know that you have a stateless app if you can just create 20 of them and not have to worry about the coordination of them. They are all workers. They are all going to take input. You could spread the load across those 20 in an identical way and not worry about which one you landed on. That’s stateless application. A stateful application is a very different thing. You have to have some coordination. You have to say how many databases can you have on a backend? Because you’re persisting data there, you have to be really careful about that you only write to the master database or to the writing database and you could read of any other memories of that database cluster, that sort of stuff. [00:11:44] CC: It might seem that we are going so deep into this differentiating between stateful and stateless, but this is so important because clusters are usually designed to be ephemeral. Ephemeral means obviously they die down, they are brought back up, the nodes, and you should worry as least as possible with the state of things. Then going back to what Joshua is saying, when we are in this cloud native world, usually we are talking about stateless apps, stateless workloads and then we’re going to just talk about what workload means. But then if that’s the case, where are the stateful apps? It’s like we have this vision that the stateful apps live outside the cloud native world? How does it work? But it’s supposed to work. [00:12:36] BL: Yup. This is the question that keeps a lot of people employed. Making sure my state is available when I need it. You know what? I’m not going to even use that word state. Making sure my data is available wherever I need it and when I need it. I don’t want to go too deep in right now, but this is actually a huge problem in the Kubernetes community in general, and we see it because there’s been lots of advice given, “Don’t run things like databases in your clusters.” This is why we see people taking the ideas of Google Spanner and like CockroachDB and actually going through a lot of work to make sure that you can run databases in Kubernetes clusters. The interesting piece about this is that we’re actually to the point where we can run these types of workloads in our clusters, but with a caveat, big star at the end, it’s very difficult and you have to know what you’re doing. [00:13:34] JR: Yeah. I want to dovetail on that Brian, because it’s something that we see all the time. I feel like when we first started setting up, let’s call them clusters, but in our case it was Kubernetes, right? We always saw that data level always being delegated to like if you’re in Amazon, some service that they hosted and so on. But now I think more and more of the customers that at least I’m seeing. I’m sure Nicholas and Duffie too, they’re interested in doing exactly what you just described. Cockroach is an example I literally just worked with recently, and it’s just interesting how much more thoughtful they have to be about their cluster operations. Going back to what you said Carlisia, it’s not as easy as just like trashing a cluster and instantiating a new one anymore, like they’re used to. They need to be more thoughtful about keeping that data integrity intact through things like upgrades and disaster recover. [00:14:18] D: Another interesting point kind to your point, Brian, is that like, frequently, people are starting to have conversations and concerns around data gravity, which means that I have a whole bunch of data that I need to work with, like to a Spark job, which I mentioned earlier. I need to basically put my compute where that data is. The way that I store that data inside the cluster and use Kubernetes to manage it or whether I just have to make sure that I have some way of bringing up compute workloads close to that data. It’s actually kind of introducing a whole new layer to this whole thing. [00:14:48] BL: Yeah! Whole new layer of work and a whole new layer of complexity, because that’s actually – The crux of all this is like where we slide the complexity too, but this is interesting, and I don’t want to go too far to this one definitely. This is why we’re seeing more people creating operators around managing data. I’ve seen operators who are bringing databases up inside of Kubernetes. I’ve seen operators that actually can bring up resources outside of Kubernetes using the Kubernetes API. The interesting thing about this is that I looked at both solutions and I said, “I still don’t know what the answer is,” and that’s great. That means that we have a lot to learn about the problem, and at least we have some paths for it. [00:15:29] NL: Actually, that kind of reminds me of the first time I ever heard the word stateful or stateless – I’m an infrastructure guy. Was around the discussion of operators, which there’s only a couple of years ago when operators were first introduced at CoreOS and some people were like, “Oh! Well, this is how you now operate a stateful mechanism inside of Kubernetes. This is the way forward that we want to propose.” I was just like, “Cool! What is that? What’s state? What do you mean stateful and stateless?” I had no idea. Josh, you were there. You’re like, “Your frontend doesn’t care about state and your backend does.” I’m like, “Does it? I don’t know. I’m not a developer.” [00:16:10] JR: Let’s talk about exactly that, because I think these patterns we’re starting to see are coming out of the needs that we’re all talking about, right? We’ve seen at least in the Kubernetes community a lot of push for these different constructs, like something called a stateful [inaudible 00:16:21], which isn’t that important right now, but then also like an operator. Maybe we can start by defining what is an operator? What is that pattern and why does it relate to stateful apps? [00:16:31] CC: I think that would be great. I am not clear what an operator is. I know there’s going to be a controller involved. I know it’s not a CRD. I am not clear on that at all, because I only work with CRDs and we don’t define – like the project I worked on with Velero, we don’t categorize it as an operator. I guess an operator uses specific framework that exists out there. Is it a Kubernetes library? I have no idea. [00:16:56] BL: We did it to ourselves again. We’re all doing these to ourselves. From the best that I can surmise, the operator pattern is the combination of a CRD plus a controller that will operate on events from the Kubernetes API based on that CRD’s configuration. That’s what an operator is. [00:17:17] NL: That’s exactly right. [00:17:18] BL: To conflate this, Red Hat created the operator SDK, and then you have [inaudible 00:17:23] and you have a Metacontroller, which can help you build operators. Then we actually sometimes conflate and call CRDs operators, and that’s pretty confusing for everyone. Once again, don’t let developers name anything. [00:17:41] CC: Wait. So let’s back up a little. Okay. There is an actual library that’s called an operator. [00:17:46] BL: Yes. There’s an operator SDK. [00:17:47] CC: Referred to as an operator. I heard that. Okay. Great. But let me back up a little because – [00:17:49] D: The word operator can [00:17:50] CC: Because if you are developing an app for Kubernetes, if you’re extending Kubernetes, you are – Okay, you might not use CRDs, but if you are using CRDs, you need a controller, right? Because how will you do actions? Then every app that has a CRD – because the alternative to having CRDs is just using the API directly without creating CRDs to reflect to resources. If you’re creating CRDs to reflect to resources, you need controllers. All of those apps, they have CRDs, are operators. [00:18:24] D: Yip [inaudible 00:18:25] is an operator. [00:18:26] CC: [inaudible 00:18:26] not an operator. How can you extend Kubernetes and not be qualified [inaudible 00:18:31] operator? [00:18:32] BL: Well, there’s a way. There is a way. You can actually just create a CRD and use a CRD for data storage, you know, store states, and you can actually query the Kubernetes API for that information. You don’t need a controller, but we couple them with controllers a lot to perform action based on that state we’ve saved to etcd. [00:18:50] CC: Duffie. [00:18:51] D: I want to back up just for a moment and talk about the controller pattern and what it is and then go from there to operators, because I think it makes it easier to get it in your head. A control pattern is effectively a way to understand desired state and real state and provide some logic or business code that will allow you to converge those two states, your actual state and your desired state. This is a pattern that we see used in almost everything within a distributed system. It’s like within Kubernetes, within most of the kind of more interesting systems that are out there. This control pattern describes a pretty good way of actually managing application flow across distributed systems. Now, operators, when they were initially introduced, we were talking about that this is a slightly different thing. Operators, when we introduced the idea, came more from like the operational burden of these stateful applications, things like databases and those sorts of stuff. With the database, etcd for example, you have a whole bunch of operational and runtime concerns around managing the lifecycle of that system. How do I add a new member to the cluster? What do I do when a member dies? How do I take action? Right now, that’s somebody like myself waking up at 2 in the morning and working through a run book to basically make sure that that service remains operational through the night. But the idea of an operator was to take that control pattern that we described earlier and make it wake up at 2 in the morning to fix this stuff. We’re going to actually codify the operational knowledge of managing the burden of these stateful applications so that we don’t have to wake up at 2 in the morning and do it anymore. Nobody wants to do that. [00:20:32] BL: Yeah. That makes sense. Remember back at KubCon years ago, I know it was one in Seattle where Brandon Philips was on stage talking about operators. He basically was saying if we think about SysOp, system operators, it was a way to basically automate or capture the knowledge of our system administrators in scripts or in a process or in code a la operators. [00:20:57] D: The last part that I’ll add to this thing, which I think is actually what really describes the value of this idea to me is that there are only so many people on the planet that do what the people in this blog post do. Maybe you’re one of them that listen to this podcast. People who are operating software or operating infrastructure at scale, there just aren’t that many of us on the planet. So as we add more applications, as more people adopt the cloud native regime or start coming to a place where they can crank out more applications more quickly, we’re going to have to get to a place where we are able to automate the burden of managing those applications, because there just aren’t enough of us to be able to support the load that is coming. There just aren’t enough people on the planet that do this to be able to support that. That’s the thing that excites me most about the operator pattern, is that it gives us a place to start. It gives us a place to actually start thinking about managing that burden over time, because if we don’t start changing the way we think about managing that burden, we’re going to run out of people. We’re not going to be able to do it. [00:22:05] NL: Yeah. It’s interesting. With stateful apps, we keep kind of bringing them – coming back to stateful apps, because stateful apps are hard and stateless apps are easy, and we’ve created all these mechanisms around operating things with state because of how just complicated it is to make sure that your data is ready, accessible and has integrity. That’s the big one that I keep not thinking about as a SysOps person coming into the Dev world. Data integrity is so important and making sure that your data is exactly what it needs to be and was the last time you checked it, is super important. It’s only something I’m really starting to grasp. That’s why I was like these things, like operators and all these mechanisms that we keep creating and recreating and recreating keep coming about, because making sure that your stateful apps have the right data at the right time is so important. [00:22:55] BL: Since you brought this up, and we just talked about why a state is so hard, I want to introduce the new term to this conversation, the whole CAP theorem, where data would typically be – in a distributed system at least, your data will be consistent or your data can be available, or if your distributed systems falls in multiple parts, you can have partition tolerance. This is one of those computer science things where you can actually pick two. You can have it be available and have partition tolerance, but your data won’t be consistent, or you can have consistency and availability, but you won’t have partition tolerance. If your cluster splits into two for some reason, the data will be bad. This is why it’s hard, this is why people have written basically lots of PhD dissertations on this subject, and this is why we are talking about this here today, is because managing state, and particularly managing distributed, is actually a very, very hard problem. But there’s software out there that will help us, and Kubernetes is definitely part of that and stateful sets are definitely part of that as well. [00:24:05] JR: I was just going to say on those three points, consistently, availability and partition tolerance. Obviously, we’d want all three if we could have them. Is there one that we most commonly tradeoff and give up or does it go case-by-case? [00:24:17] BL: Actually, it’s been proven. You can’t have all three. It’s literally impossible. It depends. If you have a MySQL server and you’re using MySQL to actually serve data out of this, you’re going to most likely get consistency and availability. If you have it replicated, you might not have partition tolerance. That’s something to think about, and there are different databases and this is actually one of the reasons why there are different databases. This is why people use things like relational databases and they use key value stores not because we really like the interfaces, but because they have different properties around the data. [00:24:55] NL: That’s an interesting point and something that I had recently just been thinking about, like why are there so many different types of databases. I just didn’t know. It was like in only recently heard of CAP theorem as well just before you mentioned it. I’m like, “Wow! That’s so fascinating.” The whole thing where you only pick two. You can’t get three. Josh, to kind of go back to your question really quickly, I think that partition tolerance is the one that we throw away the most. We’re willing to not be able to segregate our database as much as possible because C and A are just too important, I think. At least that’s what I’m saying, like I am wearing an [inaudible 00:25:26] shirt and [inaudible 00:25:27] is not partition tolerant. It’s bad at it. [00:25:31] BL: This is why Google introduced Spanner, and Spanner in some situations can get free with tradeoffs and a lot of really, really smart stuff, but most people can’t run this scale. But we do need to think about partition tolerance, especially with data whenever – Let’s say you run a store and you have multiple instances across the world and someone buys something from inventory, what is your inventory look like at any particular point? You don’t have to answer my question, of course, but think about that. These are still very important problems if fiber gets cut across the Atlantic and now I’ve sold more things than I have. Carlisia, speaking to you as someone who’s only been a developer, have you moved your thoughts on state any further? [00:26:19] CC: Well, I feel that I’m clear on – Well, I think you need to clarify your question better for me. If you’re asking if I understand what it means, I understand what it means. But I actually was thinking to ask this question to all of you, because I don’t know the answer, if that’s the question you’re asking me. I want to put that to the group. Do you recommend people, as in like now-ish, to run stateful workloads? We need to talk about workloads mean. Run stateful apps or database in sites if they’re running a Kubernetes cluster or if they’re planning for that, do you all as experts recommend that they should already be looking into doing that or they should be running for now their stateful apps or databases outside of the cloud native ecosystem and just connecting the two? Because if that’s what your question was, I don’t know. [00:27:21] BL: Well, I’ll take this first. I think that we should be spending lots of more time than we are right now in coming up community-tested solutions around using stateful sets to their best ability. What that means is let’s say if you’re running a database inside of Kubernetes and you’re using a stateful set to manage this, what we do need to figure out is what happens when my database goes down? The pod just kills? When I bring up a new version, I need to make sure that I have the correct software to verify integrity, rebuilt things, so that when it comes back up, it comes back up correctly. That’s what I think we should be doing. [00:27:59] JR: For me, I think working with customers, at least Kubernetes-oriented folks, when they’re trying to introduce Kubernetes as their orchestration part of their overall platform, I’m usually just trying to kind of meet them where they’re at. If they’re new to Kubernetes and distributed systems as a whole, if we have stateless, let’s call them maybe simpler applications to start with, I generally have them lean into that first, because we already have so much in front of us to learn about. I think it was either Brian or Duffie, you said it introduces a whole bunch more complexity. You have to know what you’re doing. You have to know how to operate these things. If they’re new to Kubernetes, I generally will advise start with stateless still. But that being said, so many of our customers that we work with are very interested in running stateful workloads on Kubernetes. [00:28:42] CC: But just to clarify what you said, Josh, because you spoke like an expert, but I still have beginner’s ears. You said something that sounded to me like you recommend that you go stateless. It sounded to me like that. What you really say is that they take out the stateless part of what they have, which they might already have or they might have to change and put the stateless. You’re not suggesting that, “Oh! You can’t do stateful anymore. You need to just do everything stateless.” What you’re saying is take the stateless part of your system, put that in Kubernetes, because that is really well-tested and keep the stateful outside of that ecosystem. Is that right? [00:29:27] JR: I think that’s a better way to put it. Again, it’s not that Kubernetes can’t do stateful. It’s more of a concept of biting off more than you can chew. We still work with a lot of people who are very new to these distributed systems concepts, and to take on running stateful workloads, if we could just delegate that to some other layer, like outside of the cluster, that could be a better place to start, at least in my experience. Nicholas and Duff might have different – [00:29:51] NL: Josh, you basically nailed it like what I was going to say, where it’s like if the team that I’m working with is interested in taking on the complexity of maintaining their databases, their stateful sets and making sure that they have data integrity and availability, then I’m all for them using Kubernetes for a stateful set. Kubernetes can run stateful applications, but there is all this complexity that we keep talking about and maintaining data and all that. If they’re willing to take on their complexity, great, it’s there for you. If they’re not, if they’re a little bit kind of behind as – Not behind, but if they’re kind of starting out their Kubernetes journey or their distributed systems journey, I would recommend them to move that complexity to somebody else and start with something a little bit easier, like a stateless application. There are a lot of good services that provide data as a service, right? You’ve got dataview as RDS is great for creating stateful application. You can leverage it anytime and you’ve got like dedicated wires too. I would point them to there first if they don’t want to take on like complexity. [00:30:51] D: I completely agree with that. An important thing I would add, which is in response to the stateful set piece here, is that as we’ve already described, managing a stateful application like a database does come with some complexity. So you should really carefully look at just what these different models provide you. Whether that model is making use of a stateful set, which provides you like ordinality, ensuring that things start up in a particular order and some of the other capabilities around that stuff. But it won’t, for example, manage some of the complexity. A stateful set won’t, for example, try and issue a command to the new member to make sure that it’s part of an existing database cluster. It won’t manage that kind of stuff. So you have to really be careful about the different models that you’re evaluating when trying to think about how to manage a stateful application like a database. I think because it’s actually why the topic of an operator came up kind of earlier, which was that like there are a lot of primitives within Kubernetes in general that provide you a lot of capability for managing things like stateful applications, but they may not entirely suit your needs. Because of the complexity with stateful applications, you have to really kind of be really careful about what you adopt and where you jump in. [00:32:04] CC: Yeah. I know just from working with Velero, which is a tool for doing backup and recovery migration of Kubernetes clusters. I know that we backup volumes. So if you have something mounted on a volume, we can back that up. I know for a fact that people are using that to backup stateful workloads. We need to talk about workloads. But at any case, one thing to – I think one of you mentioned is that you definitely also need to look at a backup and recovery strategy, which is ever more important if you’re doing stateful workloads. [00:32:46] NL: That’s the only time it’s important. If you’re doing stateless, who cares? [00:32:49] BL: Have we defined what a workload is? [00:32:50] CC: Yeah. But let me say something. Yeah, I think we should do an episode on that maybe, maybe not. We should do an episode on GitOps type of thing for related things, because even though you – Things are stateless, but I don’t want to get into it. Your cluster will change state. You can recover in stuff from like a fresh version. But as it goes through a lifecycle, it will change state and you might want to keep that state. I don’t know. I’m not the expert in that area, but let’s talk about workloads, Brian. Okay. Let me start talking about workloads. I never heard the term workload until I came into the cloud native world, and that was about a year ago or when they started looking in this space more closely. Maybe a little bit before a year ago. It took me forever to understand what a workload was. Now I understand, especially today, we’re talking about a little bit before we started recording. Let me hear from you all what it means to you. [00:34:00] BL: This is one of those terms, and I’m sure like the last any ex-Googlers about this, they’ll probably agree. This is a Google term that we actually have zero context about why it’s a term. I’m sure we could ask somebody and they would tell us, but workloads to me personally are anything that ultimately creates a pod. Deployments create replica sets, create pods. That whole thing is a workload. That’s how I look at it. [00:34:29] CC: Before there were pods, were there workloads, or is a workload a new thing that came along with pods? [00:34:35] BL: Once again, these words don’t make any sense to us, because they’re Google terms. I think that a pod is a part of a workload, like a deployment is a part of a workload, like a replica set is part of a workload. Workload is the term that encompasses an entire set of objects. [00:34:52] D: I think of a workload as a subset of an application. When I think of an application or a set of microservices, I might think of each of the services that make up that entire application as a workload. I think of it that way because that’s generally how I would divide it up to Brian’s point into different deployment or different stateful sets or different – That sort of stuff. Thinking of them each as their own autonomous piece, and altogether they form an application. That’s my think of it. [00:35:20] CC: To connect to what Brian said, deployment, will always run in the pods, which is super confusing if you’re not looking at these things, just so people understand, because it took me forever to understand that. The connection between a workload, a deployment and a pod. Pods contain – If you have a deployment that you’re going to shift Kubernetes – I don’t know if shift is the right word. You’re going to need to run on Kubernetes. That deployment needs to run somewhere, in some artifact, and that artifact is called a pod. [00:35:56] NL: Yeah. Going back to what Duffie said really quickly. A workload to me was always a process, kind of like not just a pod necessarily, but like whatever it is that if you’re like, “I just need to get this to run,” whatever that is. To me that was always a workload, but I think I’m wrong. I think I’m oversimplifying it. I’m just like whatever your process is. [00:36:16] BL: Yeah. I would give you – The reason why I would not say that is because a pod can run multiple containers at once, which ergo is multiple processes. That’s why I say it that way. [00:36:29] NL: Oh! You changed my mind. [00:36:33] BL: The reason I bring this up, and this is probably a great idea for a future show, is about all the jargon and terminology that we use in this land that we just take as everyone knows it, but we don’t all know it, and should be a great conversation to have around that. But the reason I always bring up the whole workload thing is because when we think about workloads and then you can’t have state without workloads, really. I just wanted to make sure that we tied those two things together. [00:36:58] CC: Why can you not have state without workloads? What does that mean? [00:37:01] BL: Well, the reason you can’t have state without workloads is because something is going to have to create that state, whether that workload is running in or out a cluster. Something is going to have to create it. It just doesn’t come out of nowhere. [00:37:11] CC: That goes back to what Nick was saying, that he thinks a workload is a process. Was that was you said, Nick? [00:37:18] NL: It is, yeah, but I’m renegading on that. [00:37:23] CC: At least I could see why you said that. Sorry, Brian. I cut you off. [00:37:28] BL: What I was saying is a workload ultimately is one or more processes. It’s not just a process. It’s not a single process. It could be 10, it could be 1. [00:37:39] JS: I have one final question, and we can bail on this and edit it out if it’s not a good one to end with. I hope it’s not too big, but I think maybe one thing we overlooked is just why it’s hard to run stateful workloads in these new systems like Kubernetes. We talked about how there’s more complexity and stuff, but there might be some room to talk about – People have been spinning up an EC2 server, a server on the web and running MySQL on it forever. Why in like the Kubernetes world of like pods and things is it a little bit harder to run, say, MySQL just [inaudible 00:38:10]. Is that something worth diving into? [00:38:13] NL: Yeah, I think so. I would say that for things like, say, applications, like databases particularly, they are less resilient to outages. While Kubernetes itself is dedicated to – Or most container orchestrations, but Kubernetes specifically, are dedicated to running your pods continuously as long as they will, that it is still somewhat of a shifting landscape. You do have priority and preemption. If you don’t set those things up properly of if there’s just like a total failure of your system at large, your stateful application can just go down at any time. Then how do you reconcile the outage in data, whatever data that might have gotten lost? Those sorts of things become significantly more complicated in an environment like Kubernetes where you don’t necessarily have access to a command line to run the commands to recover as easy. You may not, but it’s the same. [00:39:01] BL: Yes. You got to understand what databases do. Disk is slow, whether you have spinning disk or you have disk on chip, like SSD. What databases do in a lot of cases is they store things in memory. So if it goes away, didn’t get stored. In other cases, what databases do is they have these huge transactional logs, maybe they write them out in files and then they process the transaction log whenever they have CPU time. If a database dies just suddenly, maybe its state is inconsistent because it had items that were to be processed in a queue that haven’t been processed. Now it doesn’t know what’s going on, which is why – [00:39:39] NL: That’s interesting. I didn’t know that. [00:39:40] BL: If you kill MySQL, like kill MySQL D with a -9, why it might not come back up. [00:39:46] JR: Yeah. Going back to Kubernetes as an example, we are living in this newer world where things can get rescheduled and moved around and killed and their IPs changed and things. It seems like this environment is, should I say, more ephemeral, and those types of considerations becoming to be more complex. [00:40:04] NL: I think that really nails it. Yeah. I didn’t know that there were transactional logs about databases. I should, I feel like, have known that but I just have no idea. [00:40:11] D: There’s one more part to the whole stateful, stateless thing that I think is important to cover, but I don’t know if we’ll be able to cover it entirely in the time that we have left, and that is from the network perspective. If you think about the types of connections coming into an application, we refer to some of those connections as stateful and stateless. I think that’s something we could tackle in our remaining time, or what’s everybody’s thought? [00:40:33] JR: Why don’t you try giving us maybe a quick summary of it, Duffie, and then we can end on that. [00:40:36] CC: Yeah. I think it’s a good idea to talk about network and then address that in the context of network. I’m just thinking an idea for an episode. But give us like a quick rundown. [00:40:45] D: Sure. A lot of the kind of older monolithic applications, the way that you would scale these things is you would have multiple of them and then you would have some intelligence in the way that you’re routing connections down to those applications that would describe the ability to ensure that when Bob accesses a website and he authenticates, he’s going to authenticate to one specific instance of this application and the intelligence up in the frontend is going to handle the routing to make sure that Bob’s connection always comes back to that same instance. This is an older pattern. It’s been around for a very long time and it’s certainly the way that we first kind of learned to scale applications before we’ve decided to break into maker services and kind of handle a lot of this routing in a more resilient way. That was kind of one of the early versions of how we do this, and that is a pretty good example of a stateful session, and that there is actually some – Perhaps Bob has authenticated and he has a cookie that allows him, that when he comes back to that particular application, a lot of the settings, his browser settings, whether he’s using the dark theme or the light theme, that sort of stuff, is persisted on the server side rather than on the client side. That’s kind of what I mean by stateful sessions. Stateless sessions mean it doesn’t really matter that the user is terminating to the same end of point, because we’ve managed to keep the state either with the client. We’re handling state on the browser side of things rather on the server side of things. So you’re not necessarily gaining anything by pushing that connection back to the same specific instance, but just to a service that is more widely available. There are lots of examples of this. I mean, Brian’s example of Google earlier. Obviously, when I come back to Google, there are some things I want it to remember. I want it to remember that I’m logged in as myself. I want it to remember that I’ve used a particular – I want it to remember my history. I want it to remember that kind of stuff so that I could go back and find things that I looked at before. There are a ton of examples of this when we think about it. [00:42:40] JR: Awesome! All right, everyone. Thank you for joining us in episode 6, Stateful and Stateless. Signing off. I’m Josh Rosso, and going across the line, thank you Nicholas Lane. [00:42:54] NL: Thank you so much. This was really informative for me. [00:42:56] JR: Carlisia Campos. [00:42:57] CCC: This was a great conversation. Bye, everybody. [00:42:59] JR: Our new comer, Brian Liles. [00:43:01] BL: Until next time. [00:43:03] JR: And Duffie Cooley. [00:43:05] DCC: Thank you so much, everybody. [00:43:06] JR: Thanks all. [00:43:07] CCC: Bye! [END OF EPISODE] [0:50:00.3] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

Today on the show we have esteemed Kubernetes thought-leader, Kelsey Hightower, with us. We did not prepare a topic as we know that Kelsey presents talks and features on podcasts regularly, so we thought it best to pick his brain and see where the conversation takes us. We end up covering a mixed bag of super interesting Kubernetes related topics. Kelsey begins by telling us what he has been doing and shares with us his passion for learning in public and why he has chosen to follow this path. From there, we then talk about the issue of how difficult many people still think Kubernetes is. We discover that while there is no doubting that it is complicated, at one point, Linux was the most complicated thing out there. Now, we install Linux servers without even batting an eyelid and we think we can reach the same place with Kubernetes in the future if we shift our thinking! We also cover other topics such as APIs and the debates around them, common questions Kelsey gets before finally ending with a brief discussion on KubeCon. From the attendance and excitement, we saw that this burgeoning community is simply growing and growing. Kelsey encourages us all to enjoy this spirited community and what the innovation happening in this space before it simply becomes boring again. Tune in today! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Duffie Cooley Bryan Liles Michael Gasch Key Points From This Episode: Learn more about Kelsey Hightower, his background and why he teaches Kubernetes! The purpose of Kelsey’s course, Kubernetes the Hard Way. Why making the Kubernetes cluster disappear will change the way Kubernetes works. There is a need for more ops-minded thinking for the current Kubernetes problems. Find out why Prometheus is a good example of ops-thinking applied to a system. An overview of the diverse ops skillsets that Kelsey has encountered. Being ops-minded is just an end –you should be thinking about the next big thing! Discover the kinds of questions Kelsey is most often asked and how he responds. Some interesting thinking and developments in the backup space of Kubernetes. Is it better to backup or to have replicas? If the cost of losing data is very high, then backing up cannot be the best solution. Debates around which instances are not the right ones to use Kubernetes in. The Kubernetes API is the part everyone wants to use, but it comes with the cluster. Why the Kubernetes API is only useful when building a platform. Can the Kubernetes control theory be applied to software? Protocols are often forgotten about when thinking about APIs. Some insights into the interesting work Akihiro Suda’s is doing. Learn whether Kubernetes can run on Edge or not. Verizon: how they are changing the Edge game and what the future trajectory is. The interesting dichotomy that Edge presents and what this means. Insights into the way that KubeCon is run and why it’s structured in the way it is. How Spotify can teach us a lesson in learning new skills! Quotes: “The real question to come to mind: there is so much of that work that how are so few of us going to accomplish it unless we radically rethink how it will be done?” — @mauilion [0:06:49] “If ops were to put more skin in the game earlier on, they would definitely be capable of building these systems. And maybe they even end up more mature as more operations people put ops-minded thinking into these problems.” — @kelseyhightower [0:04:37] “If you’re in operations, you should have been trying to abstract away all of this stuff for the last 10 to 15 years.” — @kelseyhightower [0:12:03] “What are you backing up and what do you hope to restore?” — @kelseyhightower [0:20:07] “Istio is a protocol for thinking about service mesh, whereas Kubernetes provides the API for building such a protocol.” — @kelseyhightower [0:41:57] “Go to sessions you know nothing about. Be confused on purpose.” — @kelseyhightower [0:51:58] “Pay attention to the fundamentals. That’s the people stuff. Fundamentally, we’re just some people working on some stuff.” — @kelseyhightower [0:54:49] Links Mentioned in Today’s Episode: The Podlets on Twitter — https://twitter.com/thepodlets Kelsey Hightower — https://twitter.com/kelseyhightower Kelsey Hightower on GitHub — https://github.com/kelseyhightower Interaction Protocols: It's All about Good Manners — https://www.infoq.com/presentations/history-protocols-distributed-systems Akihiro Suda — https://twitter.com/_AkihiroSuda_ Carlisia Campos on LinkedIn — https://www.linkedin.com/in/carlisia/ Kubernetes — https://kubernetes.io/ Duffie Cooley on LinkedIn — https://www.linkedin.com/in/mauilion/ Bryan Liles on LinkedIn — https://www.linkedin.com/in/bryanliles/ KubeCon North America — https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-north-america-2019/ Linux — https://www.linux.org/ Amazon Fargate — https://aws.amazon.com/fargate/ Go — https://golang.org/ Docker — https://www.docker.com/ Vagrant — https://www.vagrantup.com/ Prometheus — https://prometheus.io/ Kafka — https://kafka.apache.org/ OpenStack — https://www.openstack.org/ Verizon — https://www.verizonwireless.com/ Spotify — https://www.spotify.com/ Transcript: EPISODE 7 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [INTERVIEW] [00:00:41] CC: Hi, everybody. Welcome back to The Podlets, and today we have a special guest with us, Kelsey Hightower. A lot of people listening to us today will know Kelsey, but as usual, there are a lot of new comers in this space. So Kelsey, please give us an introduction. [00:01:00] KH: Yeah. So I consider myself a minimalist. So I want to keep this short. I work at Google, on Google Cloud stuff. I’ve been involved with the Kubernetes community for what? 3, 4, 5 years ever since it’s been out, and one main goal, learning in public and helping other people do the same. [00:01:16] CC: There you go. You do have a repo on your GitHub that it’s about learning Kubernetes the hard way. Are you still maintaining that? [00:01:26] KH: Yeah. So every six months or so. So Kubernetes is a hard way for those that don’t know. It’s a guide, a tutorial. You can copy and paste. It takes about three hours, and the whole goal of that guide was to teach people how to stand up a Kubernetes cluster from the ground up. So starting from scratch, 6 VMs, you install etcd, all the components, the nodes, and then you run a few test workloads so you can get a feel for Kubernetes. The history behind that was when I first joined Google, we were all concerned about the adaption of such a complex system that Kubernetes is, right? Docker Swarm is out at the time. A lot of people are using Mesos and we’re wondering like a lot of the feedback at that time was Kubernetes is too complex. So Kubernetes the hard way was built as an idea that if people understand how it worked just like they understand how Linux works, because that’s also complex, that if people just saw how the moving pieces fit together, then they would complain less about the complexity and have a way to kind of grasp it. [00:02:30] DC: I’m back. This is Duffie Colley. I’m back this week, and then we also have Michael and Bryan with us. So looking forward to this session talking through this stuff. [00:02:40] CC: Yeah. Thank you for doing that. I totally forgot to introduce who else is in this show, and me, Carlisia. We didn’t plan what the topic is going to be today. I will take a wild guess, and we are going to touch on Kubernetes. I have so many questions for you, Kelsey. But first and foremost, why don’t you tell us what you would love to talk about? One thing that I love about you is that every time I hear an interview of you, you’re always talking about something different, or you’re talking about the same thing in a different way. I love that about the way you speak. I know you offer to be on a lot of podcast shows, which is how we ended up here and I was thinking, “Oh my gosh! We’re going to talk about what everybody is going to talk about, but I know that’s not going to happen.” So feel free to get a conversation started, and we are VMware engineers here. So come at us with questions, but also what you would like to talk about on our show today. [00:03:37] KH: Yeah. I mean, we’re all just coming straight off the hills of KubeCon, right? So this big, 12,000 people getting together. We’re super excited about Kubernetes and the Mister V event, things are wrapping up there as well. When we start to think about Kubernetes and what’s going to happen, and a lot of people saw Amazon jump in with Fargate for EKS, right? So those unfamiliar with that offering, over the years, all the cloud providers have been providing some hosted Kubernetes offering, the ideas that the cloud provider, just like we do with hypervisors and virtual machines, would provide this base infrastructure so you can focus on using Kubernetes. You’ve seen this even flow down on-prem with VMware, right? VMware saying, “Hey, Kubernetes is going to be a part of this control plane that you can use to Kubernetes’ API to manage virtual machines and containers on-prem.” So at some point now, where do we go from here? There’s a big serverless movement, which is trying to eliminate infrastructure for all kinds of components, whether that’s compute, database as a storage. But even in the Kubernetes world, I think there’s an appetite when we saw this with Fargate, that we need to make the Kubernetes cluster disappear, right? If we can make it disappear, then we can focus on building new platforms that extend the API or, hell, just using Kubernetes as is without thinking about managing nodes, operating systems and autoscalers. I think that’s kind of been the topic that I’m pretty interested in talking about, because that feature means lots of things disappear, right? Programming languages and compilers made assembly disappear for a lot of developers. Assembly is still there. I think people get caught up on nothing goes away. They’re right. Nothing goes away, but the number of people who have to interact with that thing is greatly reduced. [00:05:21] BL: You know what, Kelsey? I’m going to have you get out of my brain, because that was the exact example that I was going to use. I was on a bus today and I was thinking about all the hubbub, about the whole Fargate EKS thing, and then I was thinking, “Well, Go, for example, can generate assembler and then it compiles that down.” No one complains about the length of the assembler that Go generates. Who cares? That’s how we should think about this problem. That’s a whole solvable problem. Let’s think about bigger things. [00:05:51] KH: I think it’s because in operations we tend to identify ourselves as the people responsible for running the nodes. We’re the people responsible for tuning the API server. When someone says it’s going to go away, in ops – And you see this in some parts, right? Ops, some people focus a lot more on observability. They can care less about what machine something runs on. They’re still going to try to observe and tune it. You see this in SRE and some various practices. But a lot of people who came up in a world like I have in a traditional ops background, you were the one that pixie-booted the server. You installed that Linux OS. You configured it with Puppet. When someone tells you, “We’re going to move on from that as if it’s a good thing.” You’re going to be like, “Hold up. That’s my job.” [00:06:36] DC: Definitely. We’ve touched this topic through a couple of different times on this show as well, and it definitely comes back to like understanding that, in my opinion, it’s not about whether there will be a worker for people who are in operations, people who want to focus on that. The real question that come to mind is like there is so much of that work that how are so few of us are going to be able to accomplish it unless we radically re-sync how it will be done. We’re vastly outnumbered. The number of people walking into the internet for the first time every day is mind-boggling. [00:07:08] KH: In early days, we have this goal of abstract or automating ourselves out of a job, and anyone that tried that a number of times knows that you’re always going to have something else to do. I think if we carry that to the infrastructure, I want to see the ops folks. I was very surprised that Docker didn’t come from operations folks. It came from the developer folks. Same thing for Vagrant and the same thing from Kubernetes. These are developer-minded folks that want to tackle infrastructure problems. If I think if ops were to put more skin in the game earlier on, definitely capable of building these systems and maybe they even end up more mature as more operations people put ops-minded thinking to these problems. [00:07:48] BL: Well, that’s exactly what we should do. Like you said, Kelsey, we will always have a job. Whenever we solve one problem, we could think about more interesting problems. We don’t think about Linux on servers anymore. We just put Linux on servers and we run it. We don’t think about the 15 years where it was little rocky. That’s gone now. So think about what we did there and let’s do that again with what we’re doing now. [00:08:12] KH: Yeah. I think the Prometheus community is a good example of operations-minded folks producing a system. When you meet the kind of the originators of Prometheus, they took a lot of their operational knowledge and kind of build this metrics and monitoring standard that we all kind of think about now when we talk about some levels of observability, and I think that’s what happens when you have good operations people that take prior experience, the knowledge, and that can happen over code these days. This is the kind of systems they produce, and it’s a very robust and extensible API that I think you start to see a lot of adaption. [00:08:44] BL: One more thing on Prometheus. Prometheus is six-years-old. Just think about that, and that’s not done yet, and it’s just gotten better and better and better. We go to give up our old thing so we can get better and better and better. That’s just what I want to add. [00:08:58] MG: Kelsey, if you look at the – Basically your own history of coming from ops, as I understood your own history, right? Now being kind of one of the poster childs in the Kubernetes world, you see the world changing to serverless, to higher abstractions, more complex systems on one hand, but then on the other side, we have ops. Looking beyond or outside the world of Silicon Valley into the traditional ops, traditional large enterprise, what do you think is the current majority level of these ops people? I don’t want to discriminate anyone here. I’m just basically throwing this out as a question. Where do you think do they need to go in terms of to keep up with this evolving and higher level abstractions where we don’t really care about nitty-gritty details? [00:09:39] KH: Yes. So this is a good, good question. I spent half of my time. So I probably spent time onsite with at least 100 customers a year globally. I fly on a plane and visit them in their home turf, and you definitely meet people at various skill levels and areas of responsibility. I want to make sure that I’m clear about the areas of responsibility. Sometimes you’re hired in an area of responsibility that’s below your skillset. Some people are hired to manage batch jobs or to translate files from XML to JSON. That really doesn’t say a lot about their skillset. It just kind of talks about the area of responsibility. So shout out to all the people that are dealing with main frames and having to deal with that kind of stuff. But when you look at it, you have the opportunity to rise up to whatever level you want to be in in terms of your education. When we talk about this particular question, some people really do see themselves as operators, and there’s nothing wrong with that. Meaning, they could come in. They get a system and they turn the knobs. You gave me a mainfrastructure me, I will tell you how to turn the knobs on that mainframe. You buy me a microwave, I’ll tell you how to pop popcorn. They’re not very interested in building a microwave. Maybe they have other things that are more important to them, and that is totally okay. Then you have people who are always trying to push the boundaries. Before Kubernetes, if I think back to 10 years ago, maybe 8. When I was working in a traditional enterprise, like kind of the ones you’re talking about or hinting at, the goal has always been to abstract away all of these stuff that it means to deploy an application the right way in a specific environment for that particular company. The way I manage to do it was say, “Hey, look. We have a very complex change in management processes.” I work in finance at that time. So everything had to have a ticket no matter how good the automation was. So I decided to make JIRA the ticketing system their front door to do everything. So you go to JIRA. There’ll be a custom field that says, “Hey, here are all the RPMs that have been QA’d by the QA team. Here are all the available environments.” You put those two fields in. That ticket goes to change in management and approval, and then something below the scenes automated everything, in that case it was Puppet, Red Hat and VMware, right? So I think what most people have been doing if you’re in the world of abstracting this stuff away and making it easier for the company to adapt, you’ve already been pushing these ideas that we call serverless now. I think the cloud providers put these labels on platforms to describe the contract between us and the consumer of the APIs that we present. But if you’re in operations, you should have been trying to abstract away all of these stuff for the last 10 or 15 years. [00:12:14] BL: I 100% agree. Then also, think about other verticals. So 23 years ago, I did [inaudible 00:12:22] work. That was my job. But we learned how to program in C and C++ because we were on old Suns, not even Spark machines. We’re on the old Suns, and we wanted to write things in CVE and we wanted to write our own Window managers. That is what we’re doing right now, and that’s why you see like Mitchell Hashimoto with Vagrant and you’re seeing how we’re pushing this thing. We have barely scratched the surface of what we’re trying to do. For a lot of people who are just ops-minded, understand that being ops-minded is just the end. You have to be able to think outside of your boundaries so you can create the next big thing. [00:12:58] KH: Of you may not care about creating the next big thing. There are parts of my life where I just don’t care. For example, I pay Comcast to get internet access, and my ops involvement was going to BestBuy and buying a modem and screwing it into the wall, and I troubleshoot this thing every once in a while when someone in the household complains the internet is down. But that’s just far as I’m ever going to push the internet boundaries, right? I am not really interested in pushing that forward. I’m assuming others will, and I think that’s one thing in our industry where sometimes we believe that we all need to contribute to pushing things forward. Look, there’s a lot of value in being a great operations person. Just be welcomed to saying that what we operate will change overtime. [00:13:45] DC: Yeah, that’s fair. Very fair. For me, personally, I definitely identify as an operations person. I don’t consider it my life’s goal to create new work necessarily, but to expand on the work that has been identified and to help people understand the value of it. I find I sit in between two roles personally. One is to help figure out all of the different edges and pieces and parts of Kubernetes or some other thing in the ecosystem. Second, to educate others on those things, right? Take what I’ve learned and amplify it. Having the amplifying effect. [00:14:17] CC: One thing that I wanted to ask you, Kelsey is – I work on the Valero project, and that does back and recovery of Kubernetes clusters. Some people ask me, “Okay. So tell me about the people who are doing?” I’m like, “I don’t want to talk about that. That’s boring. I wanted to talk about the people who are not doing backups.” “Okay. Let’s talk about why you should be doing maybe thinking about that.” Well, anyway. I wonder if you get a lot of questions in the area of Kubernetes operations or cloud native in general, infrastructure, etc., that in the back of your mind you go, “That’s the wrong question or questions.” Do you get that? [00:14:54] KH: Yeah. So let’s use your backup example. So I think when I hear questions, at least it lets me know what people are thinking and where they’re at, and if I ask enough questions, I can kind of get a pulse in the trend of where the majority of the people are. Let’s take the backups questions. When I hear people say, “I want to back up my Kubernetes cluster.” I rewind the clock in my mind and say, “Wow! I remember when we used to backup Linux servers,” because we didn’t know what config files were on the disk. We didn’t know where processes are running. So we used to do these PS snapshots and we used to pile up the whole file system and store it somewhere so we can recover it. Remember Norton Ghost? You take a machine and ghost it so you can make it again. Then we said, “You know what? That’s a bad idea.” What we should be doing is having a tool that can make any machine look like the way we want it. Config management is boring. So we don’t back those up anymore. So when I hear that question I say, “Hmm, what is happening in the community that’s keeping people to ask these questions?” Because if I hear a bunch of questions that already have good answers, that means those answers aren’t visible enough and not enough people are sharing these ideas. That should be my next key note. Maybe we need to make sure that other people know that that is no longer a boring thing, even though it’s boring to me, it’s not boring to the industry in general. When I hear these question I kind of use it as a keeps me up-to-date, keeps me grounded. I hear stuff like how many Kubernetes clusters should I have? I don’t think there’s a best practice around that answer. It depends on how your company segregates things, or depends on how you understand Kubernetes. It depends on the way you think about things. But I know why they’re asking that question, is because Kubernetes presents itself as a solution to a much broader problem set than it really is. Kubernetes manages a group of machines typically backed by IS APIs. If you have that, that’s what it does. It doesn’t do everything else. It doesn’t tell you exactly how you should run your business. It doesn’t tell you how you should compartmentalize your product teams. Those decisions you have to make independently, and once you do, you can serialize those into Kubernetes. So that’s the way I think about those questions when I hear them, like, “Wow! Yeah, that is a crazy thing that you’re still asking this question six years later. But now I know why you’re asking that question.” [00:17:08] CC: That is such a great take on this, because, yes, it in the area of backup, people who are doing backup in my mind – Yeah, they should be independent of Kubernetes or not. But let’s talk about the people who are not doing backups. What motivates you to not do backups? Obviously, backups can be done in many different ways. But, yes. [00:17:30] BL: So think about it like this way. Some people don’t exercise, because exercise is tough and it’s hard, and it’s easier to sit on the couch and eat a bag of potato chips than exercise. It’s the same thing with backups. Well, backing up my Kubernetes cluster before Valero was so hard that I’d rather just invest brain cycles in figuring out how to make money. So that’s where people come from when it comes to hard things like backups. [00:17:52] KH: There’s a trust element too, right? Because we don’t know if the effort we’re putting in is worth it. When people do unit testing, a lot of times unit testing can be seen as a proactive activity, where you write unit tests to catch bugs in the future. Some people only write unit test when there’s a problem. Meaning, “Wow! There’s an odd things in a database. Maybe we should write a test to prove that our code is putting odd things. Fix the code, and now the test pass.” I think it’s really about trusting that the investment is worth it. I think when you start to think about backups – I’ve seen people back up a lot of stuff, like every day or every couple of hours, they’re backing up their database, but they’d never restored the database. Then when you read their root cause analysis, they’re like, “Everything was going fine until we tried to restore a 2 terabyte database over 100 meg link. Yeah, we never exercised that part.” [00:18:43] CC: That is very true. [00:18:44] DC: Another really fascinating thing to think about the backup piece is that especially like in the Kubernetes with Valero and stuff, we’re so used to having the conversation around stateless applications and being able to ensure that you can redeploy in the case of a failure. You’re not trying to actually get back to a known state the way that like a backup traditionally would. You’re just trying to get back to a running state. So there’s a bit of a dichotomy there I think for most folks. Maybe they’re not conceptualizing the need for having to deal with some of those stateful applications when they start trying to just think about how Valero fits into the puzzle, because they’ve been told over and over again, “This is about immutable infrastructure. This is about getting back to running. This is not about restoring some complex state.” So it’s kind of interesting. [00:19:30] MG: I think part of this is also that for the stateful services that why we do backups actually, things change a lot lately, right? With those new databases, scale out databases, cloud services. Thinking about backup also has changed in the new world of being cloud native, which for most of the people, that’s also a new learning experiment to understand how should I backup Kafka? It’s replicated, but can I backup it? What about etcd and all those things? Little different things than backing up a SQL database like more traditional system. So backup, I think as you become more complex, stays if needed for [inaudible 00:20:06]. [00:20:06] KH: Yeah. The case is what are you backing up and what do you hope to restore? So replication, global replication, like we do with like cloud storage and S3. The goal is to give some people 11 9s of reliability and replicate that data almost as many geographies as you can. So it’s almost like this active backup. You’re always backing up and restoring as a part of the system design versus it being an explicit action. Some people would say the type of replication we do for object stores is much closer to active restoring and backing up on a continuous basis versus a one-time checkpoint. [00:20:41] BL: Yeah. Just a little bit of a note, you can back up two terabytes over 100 meg link in like 44 hours and a half. So just putting out there, it’s possible. Just like two days. But you’re right. When it comes to backups, especially for like – Let’s say you’re doing MySQL or Postgres. These days, is it better to back it up or is it better to have a replica right next to it and then having like a 10 minute delayed replica right next to that and then replicating to Europe or Asia? Then constantly querying the data that you’re replicating. That’s still a backup. What I’m saying here is that we can change the way that we talk about it. Backup started as conventional as they used to be. There are definitely other ways to protect your data. [00:21:25] KH: Yeah. Also, I think the other part too around the backup thing is what is the price of data loss? When you take a backup, you’re saying, “I’m willing to lose this much data between the last backup and the next.” That cost is too high than backing up cannot be your primary mode of operation, because the cost of losing data is way too high, then replication becomes a complementing factor in the whole discussion of backups versus real-time replication and shorter times to recovery. I have a couple of questions. When should people not use Kubernetes? Do you know what I mean? I visit a lot of customers, I work with a lot of eng teams, and I am in the camp of Kubernetes is not for everything, right? That’s a very obvious thing to say. But some people don’t actually practice it that way. They’re trying to jam more and more into Kubernetes. So I love to get your insights on where do you see Kubernetes being like the wrong direction for some folks or workloads. [00:22:23] MG: I’m going to scratch this one from my question list to Kelsey. [00:22:26] KH: I’ll answer it too then. I’ll answer it after you will answer it. [00:22:29] MG: Okay. Who wants to go first? [00:22:30] BL: All right. I’ll go first. There are cases when I’m writing a piece of software where I don’t care about the service discovery. I don’t care about ingress. It’s just software that needs to run. When I’m running it locally, I don’t need it. If it’s simple enough where I could basically throw it into a VM through a CloudNet script, I think that is actually lower friction than Kubernetes if it’s simple. Now, but I’m also a little bit jaded here, because I work for the dude who created Kubernetes, and I’m paid to create solutions for Kubernetes, but I’m also really pragmatic about it as well. It’s all about effort for me. If I can do it faster in CloudNet, I will. [00:23:13] DC: For my part, I think that there’s – I have a couple of – I got follow on questions to this real quick. But I do think that if you’re not actively trying to develop a distributed systems, something where you’re actually making use of the primitives that Kubernetes provides, then that already would kind of be a red flag for me. If you’re building a monolithic application or if you’re in that place where you’re just rapidly iterating on a SaaS product and you’re just trying to like get as many commits on this thing until it works and like just really rapidly prototype or even create this thing. Maybe Kubernetes isn’t the right thing, because although we’ve come a long way in improving the tools that allow for that iteration, I certainly wouldn’t say that we’re like all the way there yet. [00:23:53] BL: I would debate you that, Duffy. [00:23:55] DC: All right. Then the other part of it is Kubernetes aside, I’m curious about the same question as it relates to containerization. Is it containerization the right thing for everyone, or have we made that pronouncement, for example? [00:24:08] KH: I’m going to jump in and answer on this one, because I definitely think we need a way to transport applications in some way, right? We used to do it on floppy disks. We used to do it on [inaudible 00:24:18]. I think the container to me I treat as a glorified [inaudible 00:24:23]. That’s the way I’ve been seeing it for years. Registry store them. They replace [inaudible 00:24:28]. Great. Now we kind of have a more maybe universal packaging format that can handle simple use cases, scratch containers where it’s just your binary, and the more complex use cases where you have to compose multiple layers to get the output, right? I think RPM spec files used to do something very similar when you start to build those thing in [inaudible 00:24:48], “All right. We got that piece.” Do people really need them? The thing I get weary about is when people believe they have to have Kubernetes on their laptop to build an app that will eventually deploy to Kubernetes, right? If we took that thinking about the cloud, then everyone would be trying to install open stack on their laptop just to build an app. Does that even make sense? Does that make sense in that context? Because you don’t need the entire cloud platform on your laptop to build an app that’s going to take a request and respond. I think Kubernetes people, I guess because it’s easier to put your on laptop, people believe that it needs to be there. So I think Kubernetes is overused, because people just don’t quite understand what it does. I think there’s a case where you don’t use Kubernetes, like I need to read a file from a bucket. Someone uploaded an XML file and my app is going to translate it into JSON. That’s it. In that case, this is where I think functions as a service, something like Cloud Run or even Heroku make a lot more sense to me because the operational complexity is kind of hitting within a provider and is linked almost like an SDK to the overall service, which is the object store, right? The compute part, I don’t want to make a big deal about, because it’s only there to process the file that got uploaded, right? It’s almost like a plug-in to an FTP server, if you will. Those are the cases where I start to see Kubernetes become less of a need, because I need a custom platform to do such an obvious operation. [00:26:16] DC: Those applications that require the primitives that Kubernetes provides, service discovery, the ability to define ingress in a normal way. When you’re actually starting to figure out how you’re going to platform that application with regard to those primitives, I do see the argument for having Kubernetes locally, because you’re going to be using those tools locally and remotely. You have some way of defining what that platforming requirement is. [00:26:40] KH: So let me pull on that thread. If you have an app that depends on another app, typically we used to just have a command line flag that says, “This app is over there.” Local host when it’s on my laptop. Some DNS name when it’s in the cluster, or a config file can satisfy that need. So the need for service discovery usually arises where you don’t know where things are. But if you’re literally on your laptop, you know where the things are. You don’t really have that problem. So when you bring that problem space to your laptop, I think you’re actually making things worse. I’ve seen people depend on Kubernetes service discovery for the app to work. Meaning, they just assume they can call a thing by name and they don’t support IPs, and ports. They don’t support anything, because they say, “Oh! No. No. No. You’ll always be running into Kubernetes.” You know what’s going to happen? In 5 or 10 years, we’re going to be talking like, “Oh my God! Do you remember when you used to use Kubernetes? Man! That legacy thing. I built my whole career porting apps away from Kubernetes to the next thing.” The number one thing we’ll talk about is where people lean too hard on service discovery, or people who built apps that taught to config maps directly. Why are you calling the Kubernetes API from your app? That’s not a good design. I think we got to be careful coupling ourselves too much to the infrastructure. [00:27:58] MG: It’s a fair point too. Two answers from my end, to your question. So one is I just build an appliance, which basically priced to bring an AWS Lambda experience to the Vsphere ecosystem. Because we don’t – Or actually my approach is that I don’t want any ops people who needs to do some one-off things, like connect this guy to another guy. I don’t want him to learn Kubernetes for that. It should be as simple as writing a function. So for that appliance, we had to decide how do we build it? Because it should be scalable. We might have some function as a service component running on there. So we looked around and we decided to put it on Kubernetes. So build the appliance as a traditional VM using Kubernetes on top. For me as a developer, it gave me a lot of capabilities, like self-healing, the self-healing capabilities. But it’s also a fair point that you wrote, Kelsey, about how much do we depend or write our applications being depend on those auxiliary features from Kubernetes? Like self-healing, restarts, for example. [00:28:55] KH: Well, in your case, you’re building a platform. I would hate for you to tell me that you rebuilt a Kubernetes-like thing just for that appliance. In your case, it’s a great use case. I think the problem that we have as platform builders is what happens when things start leaking up to the user? You tell a user all they have to care about is functions. Then they get some error saying, “Oh! There’s some Kubernetes security context that doesn’t work.” I’m like, “What the hell is Kubernetes?” That leakage is the problem, and I think that’s the part where we have to be careful, and it will take time, but we don’t start leaking the underlying platform making the original goal untrue. [00:29:31] MG: The point is where I wanted to throw this question back was now these functions being written as simple scripts, whatever, and the operators put in. They run on Kubernetes. Now, the operators don’t know that it runs in Kubernetes. But going back to your question, when should we not use Kubernetes. Is it me writing in a higher level abstraction like a function? Not using Kubernetes in first sense, because I don’t know actually I’m using it. But on the covers, I’m still using it. So it’s kind of an answer and not an answer to your question because – [00:29:58] KH: I’ve seen these single node appliances. There’s only one node, right? They’re only there to provide like email at a grocery store. You don’t have a distributed system. Now, what people want is the Kubernetes API, the way it deploys things, the way it swaps out a running container for the next one. We want that Kubernetes API. Today, the only way to get it is by essentially bringing up a whole Kubernetes cluster. I think the K3S project is trying to simplify that by re-implementing Kubernetes. No etcd, SQLite instead. A single binary that has everything. So I think when we start to say what is Kubernetes, there’s the implementation, which is a big distributed system. Then there’s the API. I think what’s going to happen is if you want the Kubernetes API, you’re going to have so many more choices on the implementation that makes better sense for the target platform. So if you’re building an appliance, you’re going to look at K3S. If you’re a cloud provider, you’re going to probably look something like what we see on GitHub, right? You’re going to modify and integrate it into your cloud platform. [00:31:00] BL: Of maybe what happened with Kubernetes over the next few years is what happened with the Linux API, or the API. Firecracker and gVisor did this, and WSL did this. We can basically swap out Linux from the backend because we can just get on with the calls. Maybe that will happen with Kubernetes as well. So maybe Kubernetes will become a standard where Kubernetes standard and Kubernetes implementation that we have right now. I don’t even know about that one. [00:31:30] KH: We’re starting to see it, right? When you say here is my pod, and we can just look at Fargate for EKS as an example. When you give them a pod, their implementation is definitely different than what most people are thinking about running these days, right? One pod per VM. Not using Virtual Kube. So they’ve taken that pod spec and tried to uphold its means. But the problem with that, you get leaks. For example, they don’t allow you to bind to a host 4. Well, the pod spec says you can bind to a host 4. Their implementation doesn’t allow you to do it, and we see the same problem with gVisor. It doesn’t implement all the system calls. You couldn’t run the Docker daemon on top of gVisor. It wouldn’t work. So I think as long as we don’t leak, because when we leak, then we start breaking stuff. [00:32:17] BL: So we’re doing the same thing with Project Pacific here at VMware, where this concept of a pod is actually a virtual machines that loops in like a tenth of a second. It’s pretty crazy how they’ve been able to figure that out. If we can get this right, that’s huge for us. That means we can move out of our appliance and we can create better things that actually work. I’m VMware specific. I’m on AWS and I want this name space. I can use Fargate and EKS. That’s actually a great idea. [00:32:45] MG: I remember this presentation, Kelsey, that you gave. I think two or three years ago. It might be three years, where you took the Kubernetes architecture and you removed the boxes and the only thing remaining was the API server. This is where it clicked to me as like, “This is right,” because I was focused on the scheduler. I wanted to understand the scheduler. But then you zoomed out or your stripped off all these pieces and the only thing remaining was the API server. This is where it clicked to me. It’s like [inaudible 00:33:09] or like the syscall interface. It’s basically my API to do some crazy things that I would have write on my own and assembly kind of something before I could even get started. As well the breakthrough moment for me, this specific presentation. [00:33:24] KH: I’m working on an analogy to talk about what’s happening with the Kubernetes API, and I haven’t refined it yet. But when the web came out, we had all of these HTTP verbs, put post git. We have a body. We have headers. You can extract that out of the whole web, the web browser plus the web server. If you have tracked out that one piece, the instead of building web package, we can build APIs and GraphQL, because we can reuse many of those mechanisms, and we just call that RESTful interfaces. Kubernetes is going through the same evolution, right? The first thing we built was this container orchestration tool. But if you look at the CRDs, the way we do RBAC, the way we think about the status field in a custom object, if you extract those components out, then you end up with this Kubernetes style APIs where we start to treat infrastructure not as code, but as data. That will be the restful moment for Kubernetes, right? The web, we extracted it out, then we have REST interfaces. In Kubernetes, once we extracted out, we’ll end up with this declarative way of describing maybe any system. But right now, the fine, or the perfect match is infrastructure. Infrastructure as data and using these CRDs to allow us to manipulate that data. So maybe you start with Helm, and then Helm gets piped into something like Customize. That then gets piped into a mission controller. That’s how Kubernetes actually works, and that data model to API development I think is going to be the unique thing that lasts longer then the Kubernetes container platform does. [00:34:56] CC: But if you’re talking about – Correct me if I misinterpret it, platform as data. Data to me is meant to be consumed, and I actually have been thinking since you said, “Oh, developers should not be developing apps that connect directly to Kubernetes,” or I think you said the Kubernetes API. Then I was thinking, “Wait. I’ve heard so many times people saying that that’s one great benefit of Kubernetes, that the apps have that access.” Now, if you see my confusion, please clarify it. [00:35:28] KH: Yeah. Right. I remember early on when we’re doing config maps, and a big debate about how config maps should be consumed by the average application. So one way could be let’s just make a configs map API and tell every developer that they need to import a Kubernetes library to call the API server, right? Now everybody’s app doesn’t work anymore on your laptop. So we were like, “Of course not.” What we should do is have config maps be injected into the file system. So that’s why you can actually describe a config map as a volume and say, “Take these key values from the config map and write them as normal files and inject them into the container so you can just read them from the file system. The other option also was environment variables. You can take a config map and translate them into an environment variables, and lastly, you can take those environment variables and put them into command line flags. So the whole point of that is all three of the most popular ways of configuring an app, environment variables, command line flags and files. Kubernetes molded itself into that world so that developers would never tightly couple themselves to the Kubernetes API. Now, let’s say you’re building a platform, like you’re building a workflow engine like Argo, or you’re building a network control plane like Istio. Of course, you should use a Kubernetes API. You’re building a platform on top of a platform. I would say that’s kind of the exception to the rule if you’re building a platform. But a general application that’s leveraging the platform, I really think you should stay away from the Kubernetes API directly. You shouldn’t be making sys calls directly [inaudible 00:37:04] of your runtime. The unsafe package in Go. Once you start doing that, Go can’t really help you anymore. You start pining yourself to specific threads. You’re going to be in a bad time. [00:37:15] CC: Right. Okay. I think I get it. But you can still use Kubernetes to decouple your app from the machine by using objects to generate those dependencies. [00:37:25] KH: Exactly. That was the whole benefit of Kub, and Docker even, saying, “You know what? Don’t worry too much more about C groups and namespaces. Don’t even try to do that yourself.” Because remember, there was a period of time where people were actually trying to build C groups and network namespaces into the runtime. There’s a bunch of like Ruby and Python projects that they were trying to containerize themselves within the runtime. Whoa! What are we doing? Having that second layer now with Containerd on C, we don’t have to implement that 10,000 times for every programming language. [00:37:56] DC: One of the things I want to come back to is your point that you’d made about the Kubernetes API being like one of the more attractive parts of the projects, and people needing that to kind of move forward in some of these projects, and I wonder if it’s more abstract than that. I wonder if it’s abstract enough to think about in terms of like a level triggered versus edge triggered stuff. Taking control theory, the control theory that basically makes Kubernetes such a stable project and applying that to software architecture rather than necessarily bringing the entire API with you. Perhaps, what you should take from this is the lessons that we’ve learned in developing Kubernetes and apply that to your software. [00:38:33] KH: Yeah. I have the fortunate time to spend some time with Mark Burgess. He came out with the Promise Theory, and the Promise Theory is the underpinnings of Puppet Chef, Ansible, CF Engine, and this idea that we would make promises about something and eventually convergent to that state. The problem was with Puppet Chef and Ansible, we’re basically doing this with shell scripts and Ruby. We were trying to write all of these if, and, else statements. When those didn’t work, what did you do? You made an exec statement at the bottom and then you’re like, “Oh! Just run some batch, and who knows what’s going to happen?” That early implementations of Promise Theory, we didn’t own the resource that we were making promises about. Anyone could go behind this and remove the user, or the user could have a different user ID on different systems but mean the same thing. In the Kubernetes world, we push a lot of that if, else statements into the controller. Now, we force the API not have any code. That’s the big difference. If you look at the Kubernetes API, you can’t do if statements. Terraform, you can do if statements. So you kind of fall into the imperative trap at the worst moments when you’re doing dry runs or something like that. It does a really good of it. Don’t get me wrong. So the Kubernetes API says, “You know what? We’re going to go all-in on this idea.” You have to change the controller first and then update the API. There is no escape patches in the API. So it forces a set of discipline that I think gets us closer to the promises, because we know that the controller owns everything. There’s no way to escape in the API itself. [00:40:07] DC: Exactly. That’s exactly what I was pushing for. [00:40:09] MG: I have a somewhat related question and I’m just not sure how to frame it correctly. So yesterday I saw a good talk by someone talking about protocols, like they somewhat forgotten power of protocols in the world of APIs. We got Swagger. We got API definitions. But he made the very easy point of if I give you an open, a close and a write and read method, or an API, you’d still don’t know how to call them in sequence and which one to call it off. This is same for [inaudible 00:40:36] library if you look at that. So I always have to force myself, “Should I do anything [inaudible 00:40:40] or I’m not leaking some stuff.” So I look it up. Versus on protocols, if you look at the RFC definitions, they are very, very precise and very plainly outlined of what you should do, how you should behave, how you should communicate between these systems. This is more of a communication and less about the actual implementation of an API. I still have to go through that talk again, and I’m going to put it in the show notes. But this kind of opened my mind again a little bit to think more about communication between systems and contracts and promises, as you said, Carlisia. Because we make so many assumptions in our code, especially as we have to write a lot of stuff very quickly, which I think will make things brittle overtime. [00:41:21] KH: So the gift and the curse of Kubernetes that it tries to do both all the time. For some things like a pod or a deployment, we all feel that. If I give any Kubernetes cluster a deployment object, I’m going to get back out running pod. This is what we all believe. But the thing is it may not necessarily run on the same kernel. It may not run on the same OS version. It may not even run on the same type of infrastructure, right? This is where I think Kubernetes ends up leaking some of those protocol promises. A deployment gets you a set of running pods. But then we dropdown to a point where you can actually do your own API and build your own protocol. I think you’re right. Istio is a protocol for thinking about service mesh, whereas Kubernetes provides the API for building such a protocol. [00:42:03] MG: Yeah, good point. [inaudible 00:42:04]. [00:42:04] DC: On the Fargate stuff, I thought was a really interesting article, or actually, an interesting project by [inaudible 00:42:10], and I want to give him a shout out on this, because I thought that was really interesting. He wrote an admission controller that leverages autoscaler, node affinity and pod affinity to effectively do the same thing so that whenever there is a new pod created, it will spin up a new machine and associate only that pod with that machine. I was like, “What a fascinating project.” But also just seeing this come up from like the whole Fargate ECS stuff. I was like – [00:42:34] KH: I think that’s the thread that virtual kubelet is pulling on, right? This idea that you can simplify autoscalling if you remove that layer, right? Because right now we’re trying to do this musical chairs dance, right? Like in a cloud. Imagine if someone gave you the hypervisor and told you you’re responsible for attaching hypervisor workers and the VMs. It would be a nightmare. We’re going to be talking about autoscalling the way we do in the cloud. I think Kubernetes moving into a world where a one pod per resource envelope. Today we call them VMs, but I think at some point we’re going to drop the VM and we would just call it a resource envelope. VMs, this is the way we think about that, Firecrackers. Like, “Hey, does it really need to be a complete VM?” Firecracker is saying, “No. It doesn’t. It just needs to be a resource envelope that allows you to run their particular workload.” [00:43:20] DC: Yeah. Same thing we’re doing here. It’s just enough VM to get you to the point where you can drop those containers on to it. [00:43:25] CC: Kelsey, question. Edge? Kubernetes on edge. Yes or no? [00:43:29] KH: Again, it’s just like compute on edge has been a topic for discussion forever. Problem is when some people say compute on edge, they mean like go buy some servers from Dell and put it in some building somewhere close to your property as you can. But then you have to go build the APIs to deploy it to that edge. What people want, and I don’t know how far off it is, but Kubernetes has set the bar so high that the Kubernetes API comes with a way to low balance, attach storage, all of these things by just writing a few YAML files. What I hear people saying is I want that close to my data center or store as possible. When you say Kubernetes on the edge, that’s what they’re saying, is like, “But we currently have one at edge. It’s not enough.” We’ve been providing edge for a very longtime. Open stack was – Remember open stack? Oh! We’re going to do open stack on the edge. But now you’re a pseudo cloud provider without the APIs. I think what Kubernetes is bringing to the table is that we have to have a default low balancer. We have to have a default block store. We have to have a default everything and on or for to mean Kubernetes like it does today centralized. [00:44:31] BL: Well, Doors have been doing this forever in some form or another. 20 years ago I worked for a Duty Free place, and literally traveled all over the world replacing point of sale. You might think of point of sales as a cash register. There was a computer in the back and it was RS-232 links from the cash register to the computer in the back. Then there was dial-up, or [inaudible 00:44:53] line to our central thing. We’ve been doing edge for a long time, but now we can do edge. The central facility can actually manage the compute infrastructure. All they care about is basically CPU and memory and network storage now, and it’s a lot more flexible. The surety is long, but I think we’re going to do it. It’s going to happen, and I think we’re almost right – People are definitely experimenting. [00:45:16] KH: You know what, Carlisia? You know what’s interesting now though? I was watching the Reinvent announcement. Verizon is starting to allow these edge components to leverage 5G for the last mile, and that’s something game-changer, because most people are very skeptical about 5G being able to provide the same coverage as 4G because of the wavelength and point-to-point, all of these things. But for edge, this thing is a game-changer. Higher bandwidth, but shorter distance. This is exactly what edge want, right? Now you don’t have to dig up the ground and run fiber from point-to-point. So if you could buy in these Kubernetes APIs, plus concepts like 5G, and get in that closer to people, yeah, I think that’s going to change the way we think about regions and zones. That kind of goes away. We’re going to move closer to CDNs, like Cloudflare has been experimenting with their worker technology. [00:46:09] DC: On the edge stuff, I think that there’s also an interesting dichotomy happening, right? There’s a definition of edge that we referred to, which is storage stuff and one that you’re alluding to, which is that there may be like some way of actually having some edge capability and a point of presence in a 5G tower or some point with that. In some cases, edge means data gravity. You’re actually taking a bunch of data from sensors and you’re trying to store it in a place where you don’t have to pay the cost of moving all of the data form one point to another where you can actually centralize compute. So in those edge cases, you’re actually willing to invest in a high-end compute to allow for the manipulation of that data where that data lake is so that you can afford to move it into some centralized location later. But I think that that whole space is so complex right now, because there are so many different definitions and so many different levels of constraints that you have to solve for under one umbrella term, which is the edge. [00:47:04] KH: I think Bryan was pulling on that with the POS stuff, right? Because instead of you going to go buy your own cash registry and gluing everything together, that whole space got so optimized that you can just buy a square terminal. Plug it on some Wi-Fi and then there you go, right? You now have that thing. So once we start to do this for like ML capabilities, security capabilities, I think you’re going to see that POS-like thing expand and that computer get a little bit more robust to do exactly what you’re saying, right? Keep the data local. Maybe you ship models to that thing so that it can get smarter overtime, and then upload the data from various stores overtime. [00:47:40] DC: Yup. [00:47:40] MG: One last question from my end. Switching gears a bit, if allow it. KubeCon. I left KubeCon with some mixed feelings this years. But my perspective is different, because I’m not the typical, one of the 12,000 people, because most of them were new comers actually. So I looked at them and I asked myself, “If I would be new to this huge big world of CNCF and Kubernetes and all these stuff, what would I take from that?” I would be confused. Confused like how from [inaudible 00:48:10] talks, which make it sound like it’s so complex to run all these things through the keynotes, which seems to be like just a lineup of different projects that I all have to get through and install and run. I was missing some perspective and some clarity from KubeCon this year, especially for new comers. Because I’m afraid, if we don’t retain them, attract them, and maybe make them contributors, because that’s another big problem. I’m afraid that we’ll lose our base that is using Kubernetes. [00:48:39] BL: Before Kelsey says anything, and Kelsey was a Kub contrary before I was, but I was a Kub contrary this time, and I can tell you exactly why everything is like it is. Well, fortunately and unfortunately, this cloud native community is huge now. There’s lots of money. There are lots of people. There are lots of interests. If we went back to KubeCon when it was in San Francisco years ago, or even like the first Seattle one, that was a community event. We could make the event for the community. Now, there’s community. The people who are creating the products. There’s the end users, the people who are consuming the products, and there are these big corporations and companies, people who are actually financing this whole entire thing. We actually have to balance all three of those. As a person who just wants to learn, what are you trying to learn from? Are you learning from the consumption piece? Are you learning to be a vendor? Are you learning to be a contributor? We have to think about that. At a certain point, that’s good for Kubernetes. That means that we’ve been able to do the whole chasm thing. We’ve cross over to chasm. This thing is real. It’s big. It’s going to make a lot of people a lot of money one day. But I do see the issue for the person who’s trying to come in and say, “What do I do now?” Well, unfortunately, it’s like anything else. Where do you start? Well, you got to take it all in. So you need to figure out where you want to be. I’m not going to be the person that’s going to tell you, “Well, go do a sig.” That’s not it. What I want to tell you is like anything else that we’d have to learn is real hard, whether it’s a programming language or a new technique. Figure out where you want to be and you’re going to have to do some research. Then hopefully you can contribute. I’m sure Kelsey has opinions on this as well. [00:50:19] KH: I think Brian is right. I mean, I think it’s just like a pyramid happening. A the very bottom, we’re new. We need to get everybody together in one space and it becomes more of a tradeshow, like an introductory, like a tasting, right? When you’re hungry and you go and just taste everything. Then when you figure out what you want, then that will be your focus, and that’s going to change every year for a lot of people. Some people go from consumer to contributor, and they’re going to want something out of the conference. They’re only going to want to go to the contributor day and maybe some of the deep-dive technical tracks. You’re trying to serve everybody in two or three days. So you’re going to start to have like everything pulling for your attention. I think what you got to do is commit. If you go and you’re a contributor, or you’re someone what’s building on top, you may have to find a separate event to kind of go with it, right? Someone told me, “Hey, when you go to all of these conferences, make sure you don’t forget to invest in the one-on-one time.” Me going to Oslo and spending an evening with Mark Burgess and really talk about Promise Theory outside of competing for attention with the rest of the conference. When I go, I’d like to meet new people. Sit down with them. Out of the 12,000 people, I call it a win if I can meet three new people that I’ve never met before. You know what? I’ll do a follow-up hangout with them to go deeper in some areas. So I think it’s more of a catch all. It’s definitely has a tradeshow feel now, because it’s big and there’s a lot of money and opportunity involved. But at the same time, you got to know that, “Hey, you got to go and seek out.” You go to Spotif

For this special episode, we are joined by Joe Beda who is currently Principal Engineer at VMware. He is also one of the founders of Kubernetes from his days at Google! We use this open table discussion to look at a bunch of exciting topics from Joe's past, present, and future. He shares some of the invaluable lessons he has learned and offers some great tips and concepts from his vast experience building platforms over the years. We also talk about personal things like stress management, avoiding burnout and what is keeping him up at night with excitement and confusion! Large portions of the show are obviously spent discussion different aspects and questions about Kubernetes, including its relationship with etcd and Docker, its reputation as a very complex platform and Joe's thoughts for investing in the space. Joe opens up on some interesting new developments in the tech world and his wide-ranging knowledge is so insightful and measured, you are not going to want to miss this! Join us today, for this great episode! Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Special guest: Joe Beda Hosts: Carlisia Campos Bryan Liles Michael Gasch Key Points From This Episode: A quick history of Joe and his work at Google on Kubernetes. The one thing that Joe thinks sometimes gets lost in translation on these topics. Lessons that Joe has learned in the different companies where he has worked. How Joe manages mental stress and maintains enough energy for all his commitments. Reflections on Kubernetes relationship with and usage of etcd. Is Kubernetes supposed to be complex? Why are people so divided about it? Joe's experience as a platform builder and the most important lessons he has learned. Thoughts for venture capitalists looking to invest in the Kubernetes space. Joe's thoughts on a few different recent developments in the tech world. The relationship and between Kubernetes and Docker and possible ramifications of this. The tech that is most exciting and alien to Joe at the moment! Quotes: “These things are all interrelated. At a certain point, the technology and the business and career and work-life – all those things really impact each other.” — @jbeda [0:03:41] “I think one of the things that I enjoy is actually to be able to look at things from all those various different angles and try and find a good path forward.” — @jbeda [0:04:19] “It turns out that as you bounced around the industry a little bit, there's actually probably more alike than there is different.” — @jbeda [0:06:16] “What are the things that people can do now that they couldn't do pre-Kubernetes? Those are the things where we're going to see the explosion of growth.” — @jbeda [0:32:40] “You can have the most beautiful technology, if you can't tell the human story about it, about what it does for folks, then nobody will care.” — @jbeda [0:33:27] Links Mentioned in Today’s Episode: The Podlets on Twitter — https://twitter.com/thepodlets Kubernetes — https://kubernetes.io/Joe Beda — https://www.linkedin.com/in/jbedaEighty Percent — https://www.eightypercent.net/Heptio — https://heptio.cloud.vmware.com/Craig McLuckie — https://techcrunch.com/2019/09/11/kubernetes-co-founder-craig-mcluckie-is-as-tired-of-talking-about-kubernetes-as-you-are/Brendan Burns — https://thenewstack.io/kubernetes-co-creator-brendan-burns-on-what-comes-next/Microsoft — https://www.microsoft.comKubeCon — https://events19.linuxfoundation.org/events/kubecon-cloudnativecon-europe-2019/re:Invent — https://reinvent.awsevents.com/etcd — https://etcd.io/CosmosDB — https://docs.microsoft.com/en-us/azure/cosmos-db/introductionRancher — https://rancher.com/PostgresSQL — https://www.postgresql.org/Linux — https://www.linux.org/Babel — https://babeljs.io/React — https://reactjs.org/Hacker News — https://news.ycombinator.com/BigTable — https://cloud.google.com/bigtable/Cassandra — http://cassandra.apache.org/MapReduce — https://www.ibm.com/analytics/hadoop/mapreduceHadoop — https://hadoop.apache.org/Borg — https://kubernetes.io/blog/2015/04/borg-predecessor-to-kubernetes/Tesla — https://www.tesla.com/Thomas Edison — https://www.biography.com/inventor/thomas-edisonNetscape — https://isp.netscape.com/Internet Explorer — https://internet-explorer-9-vista-32.en.softonic.com/Microsoft Office — https://www.office.comVB — https://docs.microsoft.com/en-us/visualstudio/get-started/visual-basic/tutorial-console?view=vs-2019Docker — https://www.docker.com/Uber — https://www.uber.comLyft — https://www.lyft.com/Airbnb — https://www.airbnb.com/Chromebook — https://www.google.com/chromebook/Harbour — https://harbour.github.io/Demoscene — https://www.vice.com/en_us/article/j5wgp7/who-killed-the-american-demoscene-synchrony-demoparty Transcript: BONUS EPISODE 001 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.9] CC: Hi, everybody. Welcome back to The Podlets. We have a new name. This is our first episode with a new name. Don’t want to go much into it, other than we had to change from The Kubelets to The Podlets, because the Kubelets conflicts with an existing project and we’ve thought it was just better to change. The show, the concept, the host, everything stays the same. I am super excited today, because we have a special guest, Joe Beda and Bryan Liles, Michael Gasch. Joe, just give us a brief introduction. The other hosts have been on the show before. People should know about them. Everybody should know about you too, but there's always newcomers in the space, so give us a little bit of a background. [0:01:29.4] JB: Yeah, sure. I'm Joe Beda. I was one of the founders of Kubernetes back when I was at Google, along with Craig McLuckie and Brendan Burns, with a bunch of other folks joining on soon after. I'm currently Principal Engineer at VMware, helping to cover all things Kubernetes and Tanzu related across the company. I came into VMware via the acquisition of Heptio, where Bryan's wearing the shirt today. Left Google, did that with Craig for about two years. Then it's almost a full year here at VMware. We're at 11 months officially as of two days ago. Yeah, really excited to be here. [0:02:12.0] CC: Yeah, I am so excited. Your name is Joe Beda. I always say Joe Beda. [0:02:16.8] JB: You know what? It's four letters and it's easy – it's amazing how many different ways there are to pronounce it. I don't get picky about it. [0:02:23.4] CC: Okay, cool. Well, today I learned. I am very excited about this show, because basically, I get to ask you anything I want. [0:02:35.9] JB: I’ll do my best to answer. [0:02:37.9] CC: Yeah. You can always not answer. There are so many interviews of you out there on YouTube, podcasts. We are going to try to do something different. Let me fire the first question I have for you. When people interview you, they ask you yeah, the usual questions, the questions that are very useful for the community. I want to ask you is this, what are people asking you that you think are the wrong questions? [0:03:08.5] JB: I don't think there's any bad questions like this. I think that there's a ton of interest that's when we're talking about technical stuff at different parts of the Kubernetes stack, I think that there's a lot of business context around the container ecosystem and the companies and around to forming Heptio, all that. A lot of times, I'll have discussions around career and what led me to where I'm at now. I think those are all a lot of really interesting things to talk about all around all that. The one thing that I think is doesn't always come across is these things are all interrelated. At a certain point, the technology and the business and career and work-life – all those things really impact each other. I think it's a mistake to try and take these things in isolation. There's a ton of lead over. I think one of the things that we tried to do at Heptio, and I think we did a good job is recognized that for anybody senior enough inside of any organization, they really have to be able to play all roles, right? At a certain point, everybody is as a business person, fundamentally, in terms of actually moving the ball forward for the company, for the business as a whole. Yeah. I think one of the things that I enjoy is actually to be able to look at things from all those various different angles and try and find a good path forward. [0:04:28.7] BL: All right. Taking that, so you've gone from big co to big co, to VC to small co to big co. What does that unique experience taught you and what can you share with us? [0:04:45.5] JB: Bryan, you know my resume better than I do apparently. I started my career at Microsoft and cut my teeth working on Internet Explorer and doing client side stuff there. I then went to Google in the office up here in Seattle. It was actually in Kirkland, this little hole-in-the-wall, temporary office, preemie work type of thing. I’m thinking, “Hey, I want to do some server-side stuff.” Worked on Google Talk, worked on ads, worked on cloud, started Kubernetes, was a little burned out. Took some time off, goofed off. Did this entrepreneur-in-residence thing for VC and then started Heptio and then sold the VMware. [0:05:23.7] BL: When you're in a big company, especially when you're more junior, it's easy to get caught up in playing the game inside of that company. When I say the game, what I mean is that there are measures of success within big companies and there are ways to advance see approval, see rewards that are all very specific to that company. I think the culture of a company is really defined by what are the parameters and what are the successes, the success factors for getting ahead inside of each of those different companies. I think a lot of times, especially when as a Microsoft straight out at college, I did a couple internships at Microsoft and then joining – leaving Microsoft that first time was actually really, really difficult because there is this fear of like, “Oh, my God. Everything's going to be super different.” It turns out that as you bounced around the industry a little bit, there's actually probably more alike than there is different. The biggest difference I think between large company and small company is really, and I'll throw out some science analogies here. I think, oftentimes organizations are a little bit like the ideal gas law. Okay, maybe going past y'all, but this is – PV = nRT. Pressure times volume equals number of molecules times temperature and the R is a constant. The idea here is that this is an equation where as you add more molecules to a constrained space, that will actually change the temperature and the pressure and these things all rise. What happens is inside of a large company, you end up with so many people within a constrained space in terms of the product space. When you add more people to the organization, or when you're looking to get ahead, it feels very zero-sum. It very much feels like, “Hey, for me to advance, somebody else has to lose.” That's not how the real world works, but oftentimes that's how it feels inside of the big company, is that if it feels zero-sum like that. The liberating thing for being at a startup and I think why so many people get addicted to working at startups is that startups are fundamentally not zero-sum. Everybody succeeds and fails together. When a new person shows up, your thought process is naturally like, “Awesome, we got more cylinders in the engine. We’re going to go faster,” which is not always the case inside of a big company. Now, I think as you get senior enough, all of a sudden these things changes, because you're not just operating within the confines of that company. You're actually again, playing a role in the business, you're looking at the ecosystem, you're looking at the community, you're looking at the competitive landscape and that's where you have your eye on the ball and that's what defines success for you, not the internal company metrics, but really the business metrics is what defines success for you. The thing that I'm trying to do, here at VMware now is as we do Tanzu is make sure that we recognize the unbounded possibilities in front of us inside of this world, make sure that we actually focus our energy on serving customers. In doing so, out-compete others in the market. It's not a zero-sum game, it's not something where as we bring more folks on that we feel we're competing with them. That's a little rambling of an answer. I don't know if that links together for you, Bryan. [0:08:41.8] BL: No, no. That was pretty good. [0:08:44.1] JB: Thanks. [0:08:46.6] MG: Joe, that's probably going to be a context switch now. You touched on the time when you went through the burnout phase. Then last week, I think you put out a tweet on there's so much stuff going on, which tweet I'm talking about. Yeah. In the Kubernetes community, you’re a rock star. At VMware, you're already a rock star being on stage at VMware shaking hands with Pat. I mean, there's so many people, so many e-mails, so many slacks, whatever that you get every day, but still I feel you are able to keep the balance, stay grounded and always have a chat, even though sometimes I don't want to approach you, but sometimes I do when I have some crazy questions maybe. Still you’re not pushing people away. How do you manage with mental stress preventing another burnout? What is the secret sauce here? Because I feel I need to work on that. [0:09:37.4] JB: Well, I mean it's hard. The tweet that I put out was last week I was coming back from Barcelona and tired of travel. I'm looking forward to right now, we're recording this just before KubeCon. Then after KubeCon, planning to go to re:Invent in Vegas, which is just a social denial-of-service. It's just overwhelming being with that. I was tired of traveling. I posted something and came across a little stronger than I wanted to. That I just hate people, right? I was at that point where it's just you're traveling and you just don't want to deal with anybody and every little thing is really bugging you and annoying you. I think burnout is an interesting thing. For me and I think there's different causes for different folks. Number one is that it's always fascinating when you start a new job, your calendar is empty, your responsibilities are low. Then as you are successful and you integrate yourself into the organization, all of a sudden you find that you have more work than you have time to do. Then you hit this point where you try and like, “I'm just going to keep doing it. I'm going to power through.” Then you finally hit this point where you're like, “This is just not humanly possible.” Then you go into a triage mode and then you have to decide what's important. I know that there's more to be done than I can do. I have to be very thoughtful about prioritizing what I'm doing. There's a lot of techniques that you can bring to bear there. Being explicit about what your goals are and what your priorities are, writing those things down, whether it's an OKR process, or whether it's just here's the my top three things that I'm focusing on. Making sure that those things are purposefully meaningful to you, right? Understanding the difference between urgent and important, which these are business booky type of things, but it's this idea of there are things that feel they have to get done right now and then there are things that are long-term important. If you're not thoughtful about how you do things, you spend all your time doing the urgent things, but you never get to the stuff that's the actually long-term important. That's a really easy trap to get yourself into. Finding ways to delegate to folks is really, really helpful here, in terms of empowering others, trusting them. It's hard to let go sometimes, but I think being able to set the stage for other people to be successful is really empowering. Then just recognizing it's not all going to get done and that's okay. You can't hold yourself to expect that. Now with respect to burnout, for me, the biggest driver for burnout in my career has been when I felt personal responsibility over something, but I have been had the tools, or the authority, or the ability to impact it.When you feel in your bones ownership over something, but yet you can't actually really own it, that is what causes burnout for me. I think there are studies talking about how the worst job is middle management. I think it's not being the CEO. It's not being new to the organization, being junior. It's actually being stuck in the middle. Because you're given a certain amount of responsibility, but you aren't always given the tools necessary to be able to drive that. Whereas the folks at the top, oftentimes they don't have those constraints, so they actually own stuff and have agency to be able to take care of it. I think when you're starting on more junior in the organization, the scope of ownership that you feel is relatively minor. That being stuck in the middle is the biggest driver for me for burnout. A big part of that is just recognizing that sometimes you have to take a step back and personally divest that feeling of ownership when really it's not yours to own. I'll give you an example, is that I started Google Compute Engine at Google, which is arguably the foundational cloud service for GCP. As it grew, as it became more important to Google, as it got reorged, more or more of the leadership and responsibilities and decision-making, I’m up here in Seattle, move down the mountain view, a lot of that stuff was focused at had been in the cloud market, but then at Google for 10 or 15 years coming in and they're like, “Okay, that's cute. We got it from here,” right? That was a case where it was my thing. I felt a lot of ownership over it. It was clear after a certain amount of time, hey, you know what? I just work here. I'm just doing my job and I do what I do, but really it’s these other folks that are driving the bus. That's a painful transition to actually go from that feeling of ownership to I just work here. That I think is one of the reasons why oftentimes, people leave the companies. I think that was one of the big drivers for why I ended up leaving Google, was that lack of agency to be able to impact things that I cared about quite a bit. [0:13:59.8] CC: I think that's one reason why – well, I think that working in the companies where things are moving fast, because they have a very clear, very worthwhile goal provides you the opportunity to just have so much work that you have to say no to a lot of things like where you were saying, and also take ownership of pieces of that work, because there's more work to go around than people to do it. For example, since Heptio and VM – okay, I’m plugging. This is a big plug for VMware I guess, but it definitely is a place that's moving fast. It's not crazy. It's reasonable, because everybody, pretty much, wherever one of us grown up. There is so much to do and people are glad when you take ownership of things. That really for me is a big source of work satisfaction. [0:14:51.2] JB: Yeah. I think it's that zero-sum versus positive-sum game. I think that when you – there's a lot more room for you to actually feel that ownership, have that agency, have that responsibility when you're in a positive-sum environment, versus a zero-sum environment. [0:15:04.9] BL: All right, so now I want to ask your technical question. [0:15:08.1] JB: All right. [0:15:09.5] BL: Not a really hard one. Just more of how you think about this. Kubernetes is five and almost five and a half years old. One of the key components of Kubernetes is etcd. Now knowing what we know now and 2019 with Kubernetes have you used etcd as its key store? Or would you have gone another direction? [0:15:32.1] JB: I think etcd is a good fit. The truth of the matter is that we didn't give that decision as much thought as we probably should have early on. We saw that it was relatively easy to stand up and get going with. At least on paper, it had the qualities that we were looking for, so we started building with it and then just ran with it. Something like ZooKeeper was also something we could have taken, but the operational overhead at the time of ZooKeeper was very different from etcd. I think we could have gone in the direction of them and this is why [inaudible 0:15:58.5] for a lot of their tools, where they actually build the data store into the tool in a native way. I think that can lead in some ways to a simpler getting started experience, because there's just one thing to boot up, but also it's more monolithic from a backup, maintenance, recovery type of thing. The one thing that I think we probably should have done there in retrospect is to try and create a little bit more of an arm's length relationship in Kubernetes and etcd. In terms of having some cleaner interfaces, some more contractor and stuff, so that we could have actually swapped something else out. There's folks that are doing it, so it's not impossible, but it's definitely not something that's easy to do, or well-supported. I think that that's probably the thing that I wouldn't change in that space. Another thing we might want to change, I think it might have been good to be more explicit about being able to actually shard things out, so that you could have multiple data stores for multiple resources and actually find a way to horizontally scale. Now we do that with events, because we were writing events into etcd and that's just a totally different stream of data, but everything else right now – I think now there's room to do this into the future. I think we've been able to push etcd vertically up until now. There will come a time where we need to find ways to shard that thing up horizontally. [0:17:12.0] CC: Is it possible though to use a different data store than etcd for Kubernetes? [0:17:18.4] JB: The things that I'm aware of here and there may be more and I may not be a 100% up to date, is I do know that the Azure folks created a proxy layer that speaks to the etcd protocol, but that is actually implemented on the backend using CosmoDB. That approach there was to essentially create a translation layer. Then Rancher created this project, which is a little bit if you've – been added a bit of a fork of Kubernetes, where they're I believe using PostgresSQL as the database for Kubernetes. I haven't looked to see exactly how they ended up swapping that in. My guess is that there's some chewing gum and bailing wiring and it's quite a bit of effort for each version upgrade to be able to actually adapt that moving forward. Don't know for sure. I haven't looked deeply. [0:18:06.0] CC: Okay. Now I would love to philosophize a little bit, or maybe a lot about Kubernetes. In the spirit of thinking of different questions to ask, so I had a bunch of questions and then I was thinking, “How could I ask this question in a different way?” Maybe this is not the right “question.” Here is the way I came up with this question. We’re so divided out there. One camp loves Kubernetes, another camp, "So hard, so complicated, it’s so complex. Why even bother with it? I don't understand why people are using this." Basically, there is that sentiment that Kubernetes is complicated. I don't think anybody would refute that. Now is that even the right way to talk about Kubernetes? Is it even not supposed to be complicated? I mean, what kind of a tool is it that we are thinking, it should just work, it should be just be super simple. Is it true that it should be a super simple tool to use? [0:19:09.4] JB: I mean, that's a loaded question [inaudible]. Let me just first say that number one, if people are complaining, I mean, I'm stealing this from Tim [inaudible], who I think this is the way he takes some of these things in stride. If people are complaining, then you're relevant, right? If nobody is complaining, then nobody cares about what you're doing. I think that it's a good thing that folks are taking a critical look at Kubernetes. That means that they're taking a look at it, right? For five years in, Kubernetes is on an upswing. That's not going to necessarily last forever. I think we have work to do to continually earn Kubernetes’s place in the technology stack over time. Now that being said, Kubernetes is a super, super flexible tool. It can do so many things in so many different situations. It's used from everything from in retail stores across the tens of thousands of stores, any type of solutions. People are looking at it for telco, 5G. People are looking at it to even running it inside cars, which scares me, right? Then all the way up to folks like at CERN using it to do data analytics for hiring and physics, right? The technology that I look at that's probably most comparable to that is something like Linux. Linux is actually scalable from everything from a phone, all the way up to an IBM mainframe, but it's not easy, right? I mean, to be able to adapt it across all that things, you have to essentially download the kernel type, make config and then answer 5,000 questions, right, for those who haven't done that. It's not an easy thing to do. I think that a lot of times, people might be looking at Kubernetes at the wrong level to be able to say this should be simple. Nobody looks at the Linux kernel that you get from git cloning, Linux’s fork and compiling it and saying, “Yeah, this is too hard.” Of course it's hard. It's the Linux kernel. You expect that you're going to have a curated experience if you want something easy, right? Whether that be an Android phone or Ubuntu or what have you. I think to some degree, we're still in the early days where people are dealing with it perhaps at to raw level, versus actually dealing with it in a more opinionated way. Now I think the fascinating thing for Kubernetes is that it provides a lot of the extension points and patterns, so that we don't know exactly what those higher-level easier-to-use abstractions are going to look like, but we know, or at least we're pretty confident that we have the right tools and the right environment to be able to experiment our way there. I think we're not there yet, but we're set up for success. That's the first thing. The second thing is that Kubernetes introduces a whole bunch of different concepts and ideas and these things are different and uncomfortable for folks. It's hard to learn new things. It's hard for me to learn new things and it's hard for everybody to learn new things. When you compare Kubernetes to say, getting started with the modern front-end web development stack, with things like Babel and React and how do you deploy this and what are all these different options and it changes on a weekly basis. There's a hell of a lot in common actually between these two ecosystems. They're both really hard, they both introduce all these new concepts and you have to be embedded in it to really get it. Now that being said, if you just wanted take raw JavaScript, or jQuery and have at it, you can do it and you'll see on Hacker News articles every once in a while where people are like, “Hey, I've programmed my site with jQuery and it's just fine. I don't need all this new stuff,” right? Just like you'll see folks saying like, “I just SSH’d in and actually ran some stuff and it works fine. I don't need all this Kubernetes stuff.” If that works for you, that's great. Kubernetes doesn't have to solve every problem for every person. Then the next thing is that I think that there's a lot of people who've been solving these problems again and again and again and again, but they've been solving them in their own way. It's not uncommon when you look at back-end systems, to join a company, look at what they've built and found that it's a complicated, bespoke system of chewing gum and baling wire with maybe a little bit Ansible, maybe a little bit of Puppets and bash. Everybody has built their own, complex, overwrought system to do a lot of the stuff that Kubernetes does. I think one of the values that we see here is that these things are complex, unique complex to do it, but shared complexity is more valuable than personal complexity. If we can agree on some of these concepts, then that's something that can be leveraged widely and it will fade to the background over time, versus having everybody invent their own complex system every time they need to solve these problems. With that all said, we got a ton of work to do. It's not like we're done here and I'm not going to actually sit here and say Kubernetes is easy, or that every complex thing is absolutely necessary and that we can't find ways to simplify it. We clearly can. I just think that when folks say, “Hey, I just want this to be easy." I think they're being a little bit too naïve, because it's a very difficult problem domain. [0:23:51.9] BL: I'd like to add on to that. I think about this a lot as well. Something that Joe said to me few years back, where Kubernetes is the platform for creating platforms, it is very applicable here. Where we are looking at as an industry, we need to stop looking at Kubernetes as some estimation. Your destination is really running your applications that give you pleasure, or make your business money. Kubernetes is a tool to enable us to think about our applications more, rather than the underlying ecosystem. We don't think about servers. We want to think about storage and networking, even things like finding things in your cluster. You don't think about that. Kubernetes gives it to you. If we start thinking about Kubernetes as a way to enable us to do better things, we can go back to what Joe said about Linux. Back whenever I started using Linux in the mid-90s, guess what? We compiled it. Make them big. That stuff was hard and it was slow. Now think about this, in my office I have three different Linux distributions running. You know what? I don't even think about it anymore. I don't think about configuring X. I don't think about anything. One thing that from Kubernetes is going to grow is it's going to – we're going to figure out these problems and it's going to allow us to think of these other crazy things, which is going to push the industry further. Think maybe 20 years from now if we're still running Kubernetes, who cares? It's just going to be there. We're going to think about some other problem and it could be amazing. This is good times. [0:25:18.2] JB: At one point. Sorry, the dog’s going to bark here. I mean, at one point people cared about some of the BIOS that they were running on our computers, right? That was something that you stressed out about. I mean, back in the bad old days when I was doing DOS gaming and you're like, “Oh, well this BIOS is incompatible with the –” IRQ's and all that. It's just background now. [0:25:36.7] CC: Yeah, I think about this too as a developer. I might have mentioned this before in this podcast. I have never gone from one job to another job and had to use the same deployment system. Every single job I've ever had, the deployment system is completely different, completely different set of tooling and completely different process. Just being able to walk out from one job to another job and be able to use the same platform for deployment, it must be amazing. On the flip side, being able to hire people that will join your organization already know how your deployment works, that has value in itself. It's a huge value that I don't think people talk about enough. [0:26:25.5] JB: Well honestly, this was one of the motivations for creating Kubernetes, is that I looked around Google early on and Google is really good at importing open source, circa 2000, right? This is like, “Hey, you want to use libpng, or you want to use this library, or whatever.” That was the type of open source that Google is really, really good at using. Then Google did things, like say release the Big Table paper. Then somebody went through and then created Cassandra out of it. Maybe there's some ideas in Cassandra that actually build on top of big table, or you're looking at MapReduce versus Hadoop. All of a sudden, you found that these things diverge and Google had zero ability to actually import open source, circa 2010, right? It could not back import systems, because the operational characteristics of these things were solely alien when compared to something like Borg. You see this also, like we would acquire companies and it would take those companies way too long to be able to essentially re-platform themselves on top of Borg, because it was just so different. This is one of the reasons, honestly, why we ended up doing something like GCE is to actually have a platform that was actually more familiar from acquisition. It's one of the reasons we did it. Then also introducing Kubernetes, it's not Borg. It's a cousin of Borg inside of Google. For those who don't know, Borg is the container system that’s been in production at Google for probably 15 years now, and the spiritual grandfather to Kubernetes in a lot of ways. A lot of the ideas that you learn from Kubernetes are applicable to Borg. It's not nearly as big a leap for people to actually change between them, as it was before, Kubernetes was out there. [0:27:58.6] MG: Joe, I got a similar question, because it seems to be like you're a platform builder. You've worked on GCE, Kubernetes obviously. If you would be talking to another platform architect or builder, what would be something that you would recommend to them based on your experiences? What is a key ingredient, technically speaking of a platform that you should be building today, or the main thing, or the lesson learned that you had from building those platforms, like technical advice, if you will? [0:28:26.8] JB: I mean, that's a really good question. I think in my mind, the mark of a good platform is when people can use it to do things that you hadn't imagined when you were building it, right? The goal here is that you want a platform to be a force multiplier. You wanted to enable people to do amazing things. You compare, again the Linux kernel, even something as simple as our electrical grid, right? The folks who established those standards, God knows how long ago, right? A 150 years ago or whenever, the whole Tesla versus Thomas Edison, [inaudible]. Nobody had any idea the long-term impact that would have on society over time. I think that's the definition of a successful platform in my mind. You got to keep that in mind, right? I think that for me, a lot of times people design for the first five minutes at the expense of the next five years. I've seen in a lot of times where you design for hey, I'm getting a presentation. I want to be able to fit something amazing on one slot. You do it, but then all of a sudden somebody wants to do something different. They want to go off course, they want to go off the rails, they want to actually experiment and the thing is just brittle. It's like, “Hey, it does this. It doesn't do anything else. Do you want to do something else? Sorry, this isn't the tool for you.” For me, I think that's a trap, right? Because it's easy to get it early users based on that very curated experience. It's hard to keep those users as they actually start using the thing in anger, as they start interfacing with the real world, as they deal with things that you didn't think of as a platform. I'm always thinking about how can every that you put in the platform be used in multiple ways? How can you actually make these things be composable building blocks, because then that gives you the opportunity for folks to actually compose them in ways that you didn't imagine, starting out. I think that's some of it. I started my career at Microsoft working on Internet Explorer. The fascinating thing about Microsoft is that through and through and through and through Microsoft is a platform company. It started with DOS and Windows and Office, but even though Office is viewed as a platform inside of Microsoft. They fundamentally understand in their bones the benefit of actually starting that platform flywheel. It was really interesting to actually be doing this for the first browser wars of IE versus Netscape when I started my own career, to actually see the fact that Microsoft always saw Internet Explorer as a platform, whereas I think Netscape didn't really get it in the same way, right? They didn't understand the potential, I think in the way that Microsoft did it. For me, I mean, just being where you start your career, oftentimes you actually sets your patterns in terms of how you look at things over time. I think a lot of this platform thinking comes from just imprinting when I was a baby developer, I think. I don't know. It takes a lot of time to really internalize that stuff. [0:31:14.1] BL: The lesson here is this a good one, is that when we're building things that are way bigger than us, don't think of your product as the end goal. Think of it as an enabler. When it's an enabler, that's where you get that X multiplier. Then that's where you get all the residuals. Microsoft actually is a great example of it. My gosh. Just think of what Microsoft has been able to do with the power of Office? [0:31:39.1] JB: Yeah. I look at something like VB in the Microsoft world. We still don't have VB for the cloud era. We still haven't created that. I think there's still opportunity there to actually strike. VB back in the day, for those who weren't there, struck this amazing balance of being easy to get started with, but also something that could actually grow with you over time, because it had all these extension mechanisms where you could actually – there's the marketplace controls that you could buy, you could partner with other developers that were writing C or C++. It was an incredible platform. Then they leverage to Office to extend the capabilities of VB. It's an amazing ecosystem. Sorry. I didn't mean to interrupt you, Bryan. [0:32:16.0] BL: Oh, no. That's all good. I get as excited about it as you do whenever I think about it. It's a pretty exciting place to be. [0:32:21.8] JB: Yeah. I'll talk to VC's, because I did a startup and the EIR thing and I'll have them ask me things like, “Hey, where should we invest in the Kubernetes space?” My answer is using the BS analogy like, “You got to go where the puck is going.” Invest in the things that Kubernetes enables. What are the things that people can do now that they couldn't do pre-Kubernetes? Those are the things where we're going to see the explosion of growth. It's not about the Kubernetes. It's really about a larger ecosystem that Kubernetes is the seed crystal for. [0:32:56.2] BL: For those of you listening, if you want to get anything out of here, rewind back about 20 seconds and play that over and over again, what Joe just said. [0:33:04.2] MG: Yeah. This was brilliant. [0:33:05.9] BL: It’s where the puck is going. It's not where we are now. We're building for the future. We're not building for now. [0:33:11.1] MG: I'm looking at this tweetable quotes here, the last 20 seconds, so many tweetable quotes. We have to decide which ones to tweet then. [0:33:18.5] CC: Well, we’ll tweet them all. [0:33:20.0] MG: Oh, yes. [0:33:21.3] JB: Here’s another thing. Here’s another piece of career advice. Successful people are good storytellers. You can have the most beautiful technology, if you can't tell the human story about it, about what it does for folks, then nobody will care. I spend a lot of the time on Twitter and probably too much time, if you ask my family. That medium of being able to actually distill your thoughts down into something that is tweetable, quotable, really potent, that is a skill that's worth developing and it's a skill that's worth valuing. Because there's things that are rolling around in my head and I still haven't found a way to get them into a tweet. At some point, I'll figure it out and it'll be a thing. It takes a lot of time to build that skill to be able to refine like that. [0:34:08.5] CC: I want to say an anecdote of myself. I interview a small – so tiny startup, maybe less than 10 people at the time in Cambridge back when I lived up there. The guy was borderline wanting to hire me and no. I sent him an e-mail to try to influence his decision and it was a long-ass e-mail. They said, “No, thank you.” Then I think we had a good rapport. I said, well, anything you can tell me about your decision then? He said something along the lines like, I was too verbose. That was pre-Twitter. Twitter I think existed, but it was at the very beginning, I wasn't using it. Yeah, people. Be concise. Decision-makers don't have time to read long things. You need to be able to convey your message in short sentences, few sentences. It's crucial. [0:35:07.5] BL: All right, so we're nearing the end. I want to ask another question, because these are random questions for Joe. Joe, it is the week before KubeCon North America 2019 and today is actually an interesting day. A couple of neat things happened today. We had Docker. It was neat. Docker split somewhat and it sold part of it and now they're going to be a tools company. That's neat. We're all still trying decoding what that actually is. Here's the neat piece, Apple released a laptop that can have 64 gigabytes of memory. [0:35:44.4] MG: Has an escape key. [0:35:45.7] BL: It has an escape key. [0:35:47.6] MG: This is brilliant. [0:35:48.6] BL: Yeah. I think the question was what do you think about that? [0:35:52.8] JB: Okay. Well, so first of all, I mean, Docker is fascinating and I think this is – there's a lot of lessons there and I'm not sure I'm the one to tell them. I think it's easy to armchair-quarterback these things. It's hard to live that story. I think that it's fun to play that what-if game. I think it does show that this stuff is hard. You can have everything in your grasp and then just have it all slip away. I think that's not anybody's fault. It's just there's different strategies, different approaches in how this stuff plays out over time. On the laptop thing, I think my current laptop has 16 gigs of RAM. One of the things that we're seeing is that as we move towards a microservices world, I gave a talk about this probably three or four years ago. As we move to a microservices world, I think there's one stage where you create a bunch of microservices, but you still view those things as an app. You say, "This microservice belongs to this app." Within a mature organization, those things start to grow and eventually what you find is that you have services that are actually useful for multiple apps. Your entire production infrastructure becomes this web of services that are calling each other. Apps are just entry points into these things at different points of that web of infrastructure. This is the way that things work at Google. When you see companies that are microservices-based, let's take an Uber, or Lyft or an Airbnb. As they diversify the set of products that they're offering, you know they're not running completely independent stacks. You know that there's places where these things connect to behind the scenes in a microservices world. What does that mean for developers? What it means is that you can no longer fit an entire company's worth of infrastructure on your laptop anymore. Within a certain constraint, you can go through and actually say, “Hey, I can bring up this canonical cut of microservices. I can bring that up on my laptop, but it will have dependencies that I either have to actually call into the prod dependencies, call into specialized staging, or mock those things out, so that I can actually run this thing locally and develop it.” With 64 gig of RAM, I can run more on my laptop, right? There's a little bit of kick in that can down the road in terms of okay, there's this race between more microservicey versus how much I can port on my laptop. The interesting thing is that where is this going to end? Are we going to have the ability to bring more and more with your laptop? Are you going to be able to run in the split brain thing across like there's people who will create network connections between these things? Or are we going to move to a world where you're doing more development on cluster, in the cloud and your laptop gets thinner and thinner, right? Either you absolutely need 64 gig because you're pushing up against the boundaries of what you can do on your laptop, or you've given up and it's all running in the cloud. Yet anyways, you might as well just use a Chromebook. It's fascinating that we're seeing this divergence of scaling up, versus actually moving stuff to the cloud. I can tell you at Google, a lot of folks, even developers can actually be super, super productive with something relatively thin like Chromebook, because there's so many tools there that really are targeted at doing all that stuff remotely, in Google's production data centers and such. That's I think the interesting implication from a developer point of view with 64 gigabytes of RAM. What you going to do Bryan? You're going to get the 64 gig Mac? You’re going to do it? [0:39:11.2] BL: It’s already coming. They'll be here week after next. [0:39:13.2] JB: You already ordered it? You are such an Apple fanboy. Oh, man. [0:39:18.6] BL: Oh, I'm actually so not to go too much into it. I am a fan of lots of memory. You know what? We work in this cloud native world. Any given week, I’ll work on four to five projects. I'm lazy. I don't want to shut any of them down. Now with 64 gigs, I don't have to shut anything down. [0:39:37.2] JB: It was so funny. When I was at Microsoft, everybody actually focused on Microsoft Windows boot time. They’re like, “We got to make it boot faster. We got to make it boot faster.” I'm like, I don't boot that often. I just want the thing to resume from sleep, right? If you can make that reliable on that theme. [0:39:48.7] CC: Yeah. I frequently have to restart my computer, because of memory issues. I don't want to know which app is taking up memory. I have a tool that I can look up, but I just shut it down, flush the memory. I do have a question related to Docker. Kubernetes, I don't know if it's right to say that Kubernetes is so reliant on Docker, because I know it works with other container technologies as well. In the worst case scenario, it's obviously, I have no reason to predict this, but in the worst case scenario where Docker, let's say is discontinued, how would that affect Kubernetes? [0:40:25.3] JB: Early on when we were doing Kubernetes and you're in this relationship with a company like Docker, I looked at what Docker was doing and you're like, “Okay, where is the real value here over time?” In my mind, I thought that the interface with developers that distributed kernel, that API surface area of Kubernetes, that was really the thing and that a lot of the Docker stuff was over time going to fade to the background. I think we've seen that happen, because when we talk about production systems, we definitely have moved past Docker and we have the CRI, we have Container D, which it was essentially built by Docker, donated to the CNCF as it made its way towards graduation. I think it's graduated now. The governance ties to Docker have been severed at this point. In production systems for Kubernetes, we've moved past that. I still think that there's developer experiences oftentimes reliant on Docker and things like Docker files. I think we're moving past that also. I think that if Docker were to disappear off the face of the earth, there would be some adjustment, but I think we have the right toolkits and the right systems to be able to do that. Some of that is open sourced by Docker as part of the Moby project. The whole Docker file evaluation flow is actually in this thing called Build Kit that you can actually use in different contexts outside of the Docker game. I think there's a lot of the building action. The thing that I think is the most influential thing that actually I think will stand the test of time is the Docker container image format. That artifact that you upload, that you download, the registry APIs. Now those things have been codified and are moving forward slowly under the OCI, the open container initiative project, which is a little bit of a sister foundation niche type of thing to the CNCF. I think that's the influence over time. Then related to that, I think the world should be a little bit worried about Docker Hub and what that means for Docker Hub over time, because that is not a cheap service to run. It's done as a public good, similar to github. If the commercial aspects of that are not healthy, then I think it might be disruptive if we see something bad happen with Docker Hub itself. I don't know what exactly the replacement for that would be overnight. That'd be incredibly disruptive. [0:42:35.8] CC: Should be Harbour. [0:42:37.7] JB: I mean, Harbour is a thing, but somebody's got a run it and somebody's got to pay the bandwidth bills, right? Thank you to Docker for paying those bandwidth bills, because it's actually been good for not just Docker, but for our entire ecosystem to be able to do that. I don't know what that looks like moving forward. I think it's going to be – I mean, maybe github with github artifacts and it's going to pick up the slack. We’re going to have to see. [0:42:58.6] MG: Good. I have one last question from my end. Totally different topic, not Docker at all. Or maybe, depends on your answer to it. The question is you're very technical person, what is the technology, or the stuff that your brain is currently spinning on, if you can disclose? Obviously, no secrets. What keeps you awake at night, in your brain? [0:43:20.1] JB: I mean, I think the thing that – a couple of things, is that stuff that's just completely different from our world, I think is interesting. I think we've entered at a place where programming computers, and so stuff is so specialized. That again, I talk about if you made me be a front-end developer, I would flail for several months trying to figure out how to even be productive, right? I think similar when we look at something like machine learning, there's a lot of stuff happening there really fast. I understand the broad strokes, but I can't say that I understand it to any deep degree. I think it's fascinating and exciting the amount of diversity in this world and stuff to learn. Bryan's asked me in the past. It's like, “Hey, if you're going to quit and start a new career and do something different, what would it be?” I think I would probably do something like generative art, right? Essentially, there's folks out there writing these programs to generate art, a little bit of the moral descendant of Demoscene that was I don't know. I wonder was the Demoscene happened, Bryan. When was that? [0:44:19.4] BL: Oh, mid 90s, or early 90s. [0:44:22.4] JB: That’s right. I was never super into that. I don't think I was smart enough. It's crazy stuff. [0:44:27.6] MG: I actually used to write demoscenes. [0:44:28.8] JB: I know you did. I know you did. Okay, so just for those not familiar, the Demoscene was essentially you wrote essentially X86 assembly code to do something cool on screen. It was all generated so that the amount of code was vanishingly small. It was this puzzle/art/technical tour de force type of thing. [0:44:50.8] BL: We wrote trigonometry in a similar – that's literally what we did. [0:44:56.2] JB: I think a lot of that stuff ends up being fun. Stuff that's related to our world, I think about how do we move up the stack and I think a lot of folks are focused on the developer experience, how do we make that easier. I think one of the things through the lens of VMware and Tanzu is looking at how does this stuff start to interface with organizational mechanics? How does the typical enterprise work? How do we actually make sure that we can start delivering a toolset that works with that organization, versus working against the organization? That I think is an interesting area, where it's hard because it involves people. Back-end people like programmers, they love it because they don't have to deal with those pesky people, right? They get to define their interfaces and their interfaces are pure and logical. I think that UI work, UX work, anytime when you deal with people, that's the hardest thing, because you don't get to actually tell them how to think. They tell you how to think and you have to adapt to it, which is actually different from a lot of back-end here in logical type of folks. I think there's an aspect of that that is user experience at the consumer level. There's developer experience and there's a whole class of things, which is maybe organizational experience. How do you interface with the organization, versus just interfacing, whether it's individuals in the developer, or the end-user point of view? I don't know if as an industry, we actually have our heads wrapped around that organizational limits. [0:46:16.6] CC: Well, we have arrived at the end. Makes me so sad, because we could talk for easily two more hours. [0:46:24.8] JB: Yeah, we could definitely keep going. [0:46:26.4] CC: We’re going to bring you back, Joe. Don’t worry. [0:46:28.6] JB: For sure. Anytime. [0:46:29.9] CC: Or do worry. All right, so we are going to release these episodes right after KubeCon. Glad everybody could be here today. Thank you. Make sure to subscribe and follow us on Twitter. Follow us everywhere and suggest episode topics for us. Bye and until next time. [0:46:52.3] JB: Thank you so much. [0:46:52.9] MG: Bye. [0:46:54.1] BL: Bye. Thank you. [END OF EPISODE] [0:46:55.1] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter at https://twitter.com/ThePodlets and on the http://thepodlets.io/ website, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

Welcome to the fourth episode of The Podlets podcast! Today we speak to the topic of observability: what the term means, how it relates to the process of software development, and the importance of investing in a culture of observability. Each of us has a slightly different take on what exactly observability is, but roughly we agree that it is a set of tools that you can use to observe the interactions and behavior of distributed systems. Kris gives some handy analogies to help understand the growing need for observability due to rising scale and complexity. We then look at the three pillars of observability, and what each of these pillars look like in the process of testing and running a program. We also think more about how observability applies to the external problems that might arise in a system. Next up, we cover how implementing observability in teams is a cultural process, and how it is important to have a culture that accepts the necessity of failure and extensive time spend problem-solving in coding. Finally, the conversation shifts to how having a higher culture of observability can do away with the old problem of calling the dinosaur in a team who knows the code backward every time an error crops up. Note: our show changed name to The Podlets. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Kris Nóva Duffie Cooley Key Points from This Episode: • Duffy and Kris’s different interpretations of observability.• Why we should bake observability into applications before a catastrophic failure.• Observability is becoming more necessary due to scale and complexity.• New infrastructures require new security systems.• Observability is a term for new ways of observing code to catch failures as they happen.• The three pillars of observability: events, metrics, and traceability.• How events, metrics, and traceability play out in an example of a WordPress blog.• Why metrics and events are necessary for observing patterns in problems.• Measuring time series data and how it is managed in a similar way to git deltas.• Why the ephemerality of events in cloud-native architectures urges a new way of thinking.• Countering exterior application issues such as a hard drive getting bumped.• The role of tracing in correlating internal and external issues with a system.• Tracing is about understanding all the bits that are being touched in a problem.• Kubernetes can be broken down into three things: compute, network and storage.• How human experience is a major factor in good observability.• The fact that embracing observability and chaos engineering is a cultural practice.• Understanding observability and chaos testing through the laser metaphor.• The more valuable the application, the higher the need for observability.• The necessity for a cultural turn toward seeing the importance of observability.• Seeming bad at debugging vs convincing teams to implement observability.• The value of having empathy for how the difficulty of software engineering.• Developing more intuition by spending time debugging.• The way automated observability tools can possibly help with developing intuition.• How observability and having common tools removes or normalizes the problem of ‘the guy’ Quotes: “Building software is very hard and complex, so if you are not making mistakes, you either are not human or you are not making enough changes.” — @carlisia [0:33:37] “Observability is just a fancy word for all of the tools to help us solve a problem.” — @krisnova [0:23:09] “You’ll be a better engineer for distributed systems if you are in a culture that is blameless, that gives you tools to experiment, and gives you tools to validate those experiments and come up with new ones.” – @mauilion [0:36:08] Links Mentioned in Today’s Episode: Velero — https://www.velero.ioCloud Native Infrastructure — https://www.amazon.com/Cloud-Native-Infrastructure-Applications-Environment/dp/1491984309 Distributed Systems Observability — https://www.goodreads.com/book/show/40182805-distributed-systems-observability Transcript: EPISODE 04 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:40.5] CC: Hi everyone, welcome back to episode four. Today we’re going to talk about observability. I am Carlisia Campos. Today here on the show with me are Duffy Coolie. [0:00:52.7] DC: How are you doing folks? I’m Duffy Coolie, I’m staff field engineer here at VMware and looking forward to this topic. [0:00:58.9] CC: Also with us is Kris Nova. [0:01:03.2] CN: Hey everyone, I’m Kris Nova. I’m a developer advocate. I code a lot. I hang out in Kubernetes. [0:01:09.7] CC: I don’t want to be left out. I’m an engineer in the open source project called Valero that does backup and recovery for your Kubernetes applications. So, observability, why do we care? [0:01:25.6] KN: That’s the million-dollar question right there honestly. [0:01:28.0] DC: It sure is. [0:01:30.3] KN: I don’t know, I have a lot of thoughts on observability. I feel like it’s one of those words that it’s kind of like dev ops. It depends which day of the week you ask a specific person, what observability means that you’ll get a different answer. [0:01:43.3] DC: Yeah, I agree with that. It seems like it’s one of those very hot topics. I mean, it feels like people often conflate the idea of monitoring and logging of an application with the idea of observability and what that means. I’m looking forward to kind of digging into the details of that. [0:01:59.9] KN: What does observability mean to you Duffy? [0:02:04.0] DC: In my take, observability is a set of tools that can be applied to describe the way that data moves through a distributed system. Whether that data is a particular request or a particular transaction, in this way, you can actually understand the way all of these distributed parts of this system that we’re building are actually interacting. As you can imagine, things like monitoring and metrics are a part of it, right? Like being able to actually understand how the code is operating for this particular piece of the system, it’s definitely a key part of understanding how that system is operating but when we think of it as a big distributed system with terrible network demons in between and lots of other kind of stuff in between. I feel like we need kind of a higher level of context for what’s actually happening between all those things and that’s where I feel like the term observability fits. [0:02:55.5] KN: Year, I think I generally agree with that. I’ve got a few nuances that I like to pick out but I have high opinions but yeah, I mean, I hear a lot about it. I have my own ideas on what it means but like, why do we need it? [0:03:06.2] DC: I want to hear your idea to what it is, how would you define it? [0:03:08.8] KN: I mean, we have an hour to listen to me rant about observability. I mean, basically, okay, I’m an infrastructure engineer. I wrote this book Cloud Native Infrastructure, everything to me is some layer of software running on top of it, infrastructure, and observability to me is, it solves this problem of how do I gain visibility into something that I want to learn more about. I think my favorite analogy for observability, have you all ever been to like you know, like a gas station or a convenience store? On the front door, it’s like a height scale chart, it will say like four feet, five feet, six feet, seven feet. I always wondered what that was for and I remember I went home one day and I googled it and it turns out, that’s actually for, if the place ever gets robbed as the person runs out the front door, you get a free height measurement of how tall they are so you can help identify them later. To me, that’s like the perfect description of observability. It’s like cleverly sneaking and things into your system that can help you with a problem later downstream. [0:04:10.0] DC: I like that. [0:04:11.0] KN: Yeah. [0:04:12.5] CC: Observability is sort of a new term because it’s not necessarily something that I, as a developer would jump in and say, “Gee, my project doesn’t do observability. I need it.” I understand metrics and I understand logging, monitoring. Now I hear observability. Of course I read about it to talk about it on the show and I have been running into this word everywhere but why are people talking about observability? That’s my question. [0:04:45.8] KN: Yeah. Well, I think this kind of goes back to the gas station analogy again, right? What do you do when your metaphorical application gets robbed? What happens in the case of a catastrophic problem and how do you go about preparing yourself the best way possible to have an upper hand at solving that problem? [0:05:04.6] DC: Leah. [0:05:05.2] KN: Right? You know, some guy robbed a store and ran out the front door and we realized, “We have no idea how tall he is, he could be four feet tall or he could be six feet tall.” You know, we learn the hard way that maybe we should start putting markers on the door. I feel like observability is the same thing but I feel like people just kind of wake up and say like, “I need observability. I’m going to go and you know, I need all of these bells and whistles because my application of course is going to break,” and I feel like in a weird way that’s almost a cop out. We should be working on application before we work on preparing for catastrophic failure. [0:05:37.6] CC: Why didn’t I hear the word observability 10 years ago or even five years ago? I think it’s about two years ago. [0:05:45.9] DC: I’ll argue that the term observability is coming up more frequently and it’s certainly a hot topic today because of effectively context, still comes back down to context. When you’re in a situation where your application, you built like a cloud native architecture of your application, you got a bunch of different services that are intercommunicating or maybe all communicating to some put together shared resource. And things are misbehaving, you’re going to need to have the context to be able to understand how it’s breaking or at what point it’s breaking or where in the tangled web that we move is the problem actually occurring and can we measure that at that point? Traditionally like in a monolithic architecture, you're not really looking at that, maybe break over the monolith, you set up a couple of set points, you’re looking for the way particular clip pads work or if you’re on top of the game, you might like instrument your code in such a way that will emit events when particular transactions happen or particular things happen. You’re going to be looking at those events and logs and looking at metrics to figure out how this one application is performing or behaving. With observability, we have to solve that problem across many systems. [0:06:54.2] CC: That is why I put on the shownotes that it has to do something with the idea of cattle versus [inaud]. Because, I’m saying this because Duffy was asking me before we started recording why was that on the show notes. Because correct me if I’m wrong, I think you are going the direction of saying you don’t see, you don’t see the relation but the relation that I was thinking about was exactly what you just said. If I have a monolith, I’m looking at one thing, we’re both looking at one log. I can treat that as my little pad as supposed to when I have many micro services interacting. I can’t even treat anything, if I treated them as badly without that, right? Because I can’t. This is too much. The idea of the reason why observability is necessary sounds to me like that is a problem of scale and complexity. [0:07:46.2] KN: Yeah, I think that explains why we’re just now hearing it too, right? I’m trying to think of another metaphor here. I guess today’s going to be a metaphor day for me. Got it, okay. I just got back from London last week. I had gotten of the tube and I remember I came up to the surface and blinding light is in my eyes and all of a sudden it’s all sign for Scotland Yard and I was like “Wow, I remember this from all the detective sleuth stories of my childhood.” It dawned on me that the entire point of this part of London was there to help people recover from disasters. I thought about why we don’t have Scotland Yard type places anymore and it’s because we have security systems and we have like different things in place that we had to kind of learn the hard way we needed and we had to develop technology to help make that easier for us and I feel like we’re just kind of at that cusp of like our first wave of security cameras. Metaphorical security cameras with observability. We’re at that first wave of, we can instrument our code and we can start building our systems out with this idea of, “I want to be able to view it or observe it over time in the case of trying to learn more about it or debugging a problem.” [0:08:52.1] DC: Yeah. [0:08:52.9] CC: How do people handle – I’m asking this question because truly, I have yet to have this problem for my project that I need to do observability in my project. I need to make sure my project is observable. I mean, other than the bread and butter metrics and logging, that’s what we do at Valero. We don’t do anything further than that. But, I don’t know if those are the things are constitute observability but what Nova just said, my question is, when we want to look at this stuff later but we’re also talking about cattle and this things. Supposedly, your servers are ephemeral. They can go away and go back. How do we look at, how do we observe things if they have gone away? [0:09:41.1] KN: Year. That’s where we get into like, this exciting world of like, how long do we persist our data and which data do we track? And there’s, you know, a lot of schools of thought and a lot of different opinions around what the right solution here is but I think it kind of just boils down to every application and set of concerns is going to be unique and you’re just going to have to give it some thought. [0:10:01.9] CC: Should we talk more about that because that sounds very interesting. [0:10:04.4] KN: Yeah. I mean, I guess we should probably just start off with like, given a simple application, concretely, what does it mean to build out ‘observability’ for that application? [0:10:18.3] DC: There’s this idea of in a book called Distributed Systems Observability by Cindy Sridharan. I’m probably slaughtering her name but she wrote that there’s like these three pillars and the three pillars are events, metrics and traceability or tracing. Events, metrics and tracing. These are the three pillars of observability. If we were going to lay out the way that those things might apply to just any old application like a monolith then we might look at how – [0:10:45.3] KN: Can we just use like a WordPress blog, just like for an example. It’s got a data score, it’s got a thin layer of software and an API. [0:10:53.0] DC: Sure, like a WordPress app. The first thing we try to do is actually figure out what events we would like to get from the application and figure out how the instrument our application such that we’re getting useful data back as far as like the event stream. Frequently I think that – or in my experience, the things that you want to instrument in your application or it calls that your application is going to make that might represent a period of time, right? If it’s going to make a call to an external system, that’s something that you would definitely want to emit an event for if you’re trying to understand you know, where the problems are going sideways, like how long it took to actually make a query against the database in the back of a WordPress blog. It’s a great example, right? [0:11:33.5] KN: Question, you said the word instrumentation. My understanding of instrumentation is there’s kind of a bit of an art to it and you’re actually going in and you're adding like lines of code to your application that on line 13, we say ‘starting transaction’, on line 14, we make an https transaction and on the next line, we have, ‘the event is now over’ and we can sort of see that and discover that we made this https transaction and see where it broke, if it broke at all. Am I thinking about that right? [0:12:05.2] DC: I think you are but what’s interesting about that, the reporting on line 14, right? Where you're actually saying the event is over, right? I think that we end up actually measuring this in both in event stream and also in a metric, right? So that we can actually understand, of the last hundred transactions to the database, you know, are we seeing any increase in the amount of time and the process takes, like are we actually, you know, is this something that we can measure with metrics and understand, like, is this value changing over time? And then from the event perspective, that’s where we start tying in things like, contextually, in this transaction, what happened, right? In this particular event, is there some way that we can correlate the event with perhaps a trace and we’ll talk a little bit more about tracing too but like – so that we can understand, “okay, well, we have” – at 2:00 we see that there is like an incredible amount of latency being introduced when my WordPress blog tries to write to the database and it happens, every day at 2:00. I need to figure out what’s happening there. That’s a great – to even get to the point where I understand it’s happening at 2:00, I need things like metrics and need things like the events, specifically to give me that time correlation to understand, it’s at two. [0:13:18.7] KN: This is where we get into what Carlisia just asked about which was how do we solve this problem of what do we do when it goes away? In the case of our 2 PM database latency. For a lack of a better term, let’s just call it the heartbeat, the 2 PM heartbeat. What happens when the server that we’re experiencing, that latency, mysteriously goes away? Where does that data go and then you look at tools like I know Prometheus does this in elastic search, has capability to do this, but you look at how do we start managing time series data and how do we start tracking that and recording it. It’s a fascinating problem because you don’t actually record 2 PM. To this second and this degree of a second, this thing happened. You record how long it’s been since the previous event. You’re just constantly measuring deltas. It’s the same way that git works. Every time you do a get commit, you don’t’ actually write all 1,000 lines of software, you just write the one line that changed. [0:14:15.6] DC: Yeah. I think you highlight a really – I mean, both of the two of you highlighted a really good point around like this little cattle versus pets thing. This actually is something that I spent a little time with in a previous life and the challenge is that especially in systems like Kubernetes and other systems where you have – perhaps your application is running or being scaled out dynamically or scaled down dynamically based on load. You have all of these ephemeral events. You have all of this events that are from pods or from particular instances of your application that are ephemeral, they’re not going to be long lived. This highlights a kind of a new problem that we have to solve, I think, when we start thinking about cloud native architectures in that we have to be able to correlate that particular application with information that gives us the context to understand, like perhaps, this was this version of this application and these events are related to that particular version of the app. When we made a change, we saw a great reduction in the amount of time it takes to make that database call and we can correlate those new metrics based on the new version of the app and because we don’t have this like, as a long term entity that we can measure, like this isn’t like a single IP and a single piece of software that is not changing. This is any number of instances of our application deployed – like it makes you have to think about this problem fundamentally differently and how you store that data. This is where that cardinality problem that you’re highlighting comes in. [0:15:45.2] KN: Yeah. Okay, I have a question. Open question for the group. What is the scope here? I guess, to like kind of like build on our WordPress analogy. Let’s say that every day at 2 PM we notice there’s this latency and we’ve spent the last two weeks just endlessly digging through our logs and trying to come up with some sort of hypothesis of what’s going on here and we just can’t find anything. Everything we’ve talked about so far has been at the application layer of the stack. Instrumenting our application, debugging our application, making it https request. What should we do, or does observability even care if one of our hard drives is failing every day at 2 PM when the cleaning service comes by and accidentally bumps into it or something? How are we going to start learning about these deeper problems that might exist outside of our application layer which in my experience, those are the problems that really stick with you and really cause a lot of trouble. [0:16:40.0] DC: Yeah, agreed. Or somebody has like scheduled a backup of your database every day at two is what locks the database for a period of time of the backup and you're like “Wait, when did that happen? Why did that happen?” [0:16:51.4] KN: Somebody like commented out a line in the chron tab and then the server got reset and there’s like some magical bash grip somewhere on the server that goes and rewrites the chron tab. Who knows? [0:17:00.9] DC: Yeah, these are the needles in the haystack that we’ve all stumbled upon one way or another. [0:17:05.1] KN: Yeah, does observability, like, are we responsible for instrumenting like the operating system layer, the hardware layer? [0:17:14.0] CC: Isn’t that what monitoring is, like, some sort of testing from the outside, like an external testing that – of course, you only – it gives us the information after the fact, right? The server already died. My application’s already not available so now I know. [0:17:31.8] KN: Yeah. [0:17:32.4] CC: But isn’t monitoring what would address a problem like that? [0:17:36.1] DC: I think it definitely helps. I think what you're digging at Kris is correlation. Being able to actually identify at a particular period of time, what’s happening across our infrastructure, not just to our application. Being able to – and the important part is like how you even got to that time of day? Like, how do you know that this is happening, like, when you're looking for those patterns, how did you get to the point where you knew that it was happening at 2:00. If you know that it’s happening at 2:00 because of the event stream per se, right? That actually gives you a time correlation. Now you can look at, “Okay, well now I have a time and now I need to like, scoot back to like a macro level and see” – [0:18:10.7] KN: Crank it up at 2 PM. [0:18:12.9] DC: Yeah. Globally at 2:00, what’s going on in my world, right? Is there, you know – I know that these are the two entities that are responsible. I know that I have a bunch of pods that are running on this cluster, I know that I have a database that may be external to my cluster or maybe on the cluster. I need to really understand what’s happening in the world around those two entities as it correlates to that period of time to give me enough context to even troubleshoot. [0:18:39.7] CC: How do you do it though? Because I’m still going to go back to the monitoring, I mean, if I’m using external service to ping my service and my service is down, yeah, I’m going to get the timing – right, I can go back and look at the information, the log stream. Would I know that it was because of the server? No. But should I be pinging the server too? Should I ping every layer of the infrastructure? How do people do that? [0:19:05.4] KN: Yeah, that’s kind of what I was eluding to is like, where does observability at the application level stop and systems observability across the entire stack start and what tools do we have and where are the boundaries there? [0:19:20.3] DC: I think this is actually where we start talking about the third pillar that we were referring to earlier which was tracing and the ability to understand from the perspective of a particular transaction across the system. What entities that particular transaction will touch and where it spends its time across that entire transaction so my query – what I was trying to do was actually like, you know, submit a comment on a WordPress blog. If I had a way of implementing tracing through that WordPress blog, I might be able to leave myself little breadcrumbs throughout the entire set of systems and understand, “Well, at what point did I – I mean, where in this particular web transaction am I spending time?” I might see that you know, from the load balancer, I begin my trace ID and that load balancer terminates to this pod and inside of that pod, I can see where I’m spending my time. A little bit of time to kind of load the assets and stuff, a little bit of time for pushing my comments to the database and identifying what that database is, is the important part of that trace. If I understand – I need to know where that traffic is going to go next and how much time I spent in that transaction. You know, again, this is like down to that code layer. We should have some way of actually like leaving us – producing an event that may be related to a particular trace ID so that we can correlate the entire life cycle of that transaction. That unique trace ID across the entire process. [0:20:45.2] KN: Interesting. [0:20:45.7] DC: It helps us narrow the field to understand what all the bits are that are actually being touched that are part of the problem. Otherwise, we’re looking at the whole world and like obviously, that’s a much bigger haystack, right? [0:21:00.4] KN: One of the things that I’ve kind of learned about Kubernetes is as I’ve been like working with Kubernetes and explaining it to people and going down the road and talking and doing public speaking. I found that it’s very easy for users to understand Kubernetes if you break it down into three things. Compute, network and storage. What I’m kind of getting that here is like the application layer is probably going to be more relevant to the compute layer. Storage is going to be where – that’s observability. Storage is going to be more monitoring. That’s going to be what is my system doing, where am I storing my data, and then network is kind of related to tracing, what you’re looking at here, and these aren’t like necessarily one to one but it just kind of have – distribution of concerns here. Am I thinking about that? Kind of the same way you are Duffy? [0:21:42.8] DC: I think you are. I think what I’m trying to get to is like, I’m trying to identify the tools that I need to be able to understand what’s happening at 2:00 and all of the players involved in that, right? For that, I’m actually relying on tools that are pretty normal like the ability to actually monitor all the systems and understand and have like real time stamps and stuff that describes you know, that nodule server or what have you that says that you know, my backup for my SQL database started at 2:00 and ends at 2:30. I’m relying on things like an event stream to say you know, get to give me some context of time like when my problem is happening and I’m relying on things like tracing perhaps just should narrow the field so that I can actually understand what’s happening with this particular transaction and what are the systems that I should be looking at, whether that is – there’s a bunch of time being spent on the network, so what’s going on with the network at 2:00. There’s a bunch of times being spent on persisting data to a database, what’s going on with the database? You know, like, this kinda gives me I think enough context to actually get into troubleshooting mode, right? [0:22:50.2] KN: Yeah and I don’t want to take away from this lovely definition you just dropped on us but I want to take a stab at trying to summarize this. So observability, expands the whole stack. So I mean it is like if you look at the OSI reference model it is going to cover every one of those layers and all it really is just a fancy word for all of the tools to help us solve a problem. Yeah, sorry I am not trying to take away from your definition, right? I want to just simplify it so that like I can grapple it a little bit better. [0:23:22.2] CC: How about people? Does culture factor into it or it is just tools? [0:23:26.8] KN: I think culture is a huge part of it. Pesky humans. [0:23:28.0] DC: Yeah it is. [0:23:29.8] CC: Would this culture be tremendously different from what we get now usually at least with modern companies that doing modern software? [0:23:40.5] KN: I mean I definitely think like – [0:23:41.7] CC: Would it look different? [0:23:42.8] KN: Yeah, I definitely think there’s like – you can always tell. Like somebody once asked, “What is the difference between an SRE and a senior SRE?” And they were like, “patience,” And it is like you can always tell folks who have been burned because they take this stuff extremely seriously and I think that culture, like there is commodity there, like people are willing to pay for it if you can actually do a good job at going from chaotic problem, “I have no idea what is going on.” And making sense of that noise and coming up with a concrete tangible output that humans can take action on, I mean that is huge. [0:24:16.8] DC: Yeah it is. I was recently discussing the ability – in another medium. We were having a conversation around doing chaos testing and I think that this relates. And the interesting thing that came out of that for me was the idea that you know – I spent a pretty good portion of my career teaching people to troubleshoot, which is kind of weird. You know like teaching somebody to have an intuition about the way that a system works and giving them a place to even begin to troubleshoot a particularly complex problem, especially as we start building more and more complex systems, is really a weird thing to try and do. And I think that culturally, when you have embraced technologies like observability and embrace technologies like chaos engineering, I think that culturally you are actually not only enabling your developers, your operators, your SRE’s to experiment and understand how the system breaks at any point, but you are also enabling them to better understand how to troubleshoot and characterize these distributed systems that they are building. So I think that – and if that is a part, if that is a cultural norm within your company, I mean think about how many miles ahead you are of like the other people in your industry, right? You have made it through adopting these technologies. You have enabled your engineering teams, whether they’d be the people who are writing the code, whether they’d be the people who are operating the code, or the people who are just trying to keep the whole system up or provide you feedback to experiment and to develop hypothesis around how the system might break at a particular scale and to test that, right? And giving them the tools with which to actually observe this is critical, you know? Like it is amazing. [0:26:03.5] KN: Yeah, in my mind, again on my metaphor kick again, I think of the bank robber movies where they take dust and blow it into the air then all of a sudden you can see the lasers. Yeah, I’m feeling like that is what is happening here, is we’re kind of purpose – like chaos testing would just be the practice of intentionally breaking the lasers to make sure the security system works and observability is the practice of actually doing something to make those lasers visible so we can see what is going on. [0:26:31.0] CC: So because the two of you spend time with customers, or maybe Duffy more so than Nova, but definitely, I spent zero time. I spent zero. I am curious to know if someone, let’s say an SRE, wants to implement a set of practices that comprise what we are talking about and saying it is observability but they need to get a buy out from other people. How do you suggest they go about doing that? Because they might know how to do it or be willing to learn but they might need to get approval or they need to get a buy out – I am sorry, a buy in from their managers, from their colleagues, you know, there is a benefit and there is a cost. How will somebody present that? I mean we just talked about – I am sorry Nova, definitely just give us a laundry list of benefits but how do you articulate that in a way you prove those benefits are worth the cost and what are the costs? What are the tradeoffs? [0:27:36.4] KN: Yeah, I mean I think this is such a great question because in my career, I have worked the world’s most paranoid software as a service shop where I mean everything we did, we baked like emergency disaster recovery into it, every layer of everything we did, and I have also worked at shops that are like, “No, we ain’t got time for that, like hurry up and get your code moved and pushed to production,” and I mean I think there is pros and cons to each. But I think, you know, as you look at the value you have in your application, you are going to come up with some sort of way of concretely measuring that, of saying like, “This is an application that brings in 500 bucks a month,” or whatever, and depending on that cost or how much your application is worth to you is going to depend on I think how seriously you take it. For instance, a WordPress blog is going to probably not have the same level of observability with concerns than like maybe a bank routing system have. So I think as your application gets more and more valuable your need for observability and your need for these tools is going to go up more. [0:28:37.6] DC: I agree. I think from the perspective of like, how do you convince, maybe an existing engineering culture to make this jump, to introduce these ideas? I think that that is a tricky question because effectively what you are trying to do is kind of enable that cultural shift that we were talking about before, about like, what tools would set up the culture to succeed as they build out these applications and distributed systems that are going to make up or that are going to comprise the basis of what your product is, right? What tool? And getting to that, coming at that from a SRE perspective that needs air cover to be able to actually have those tough conversations with your developers and say, “Look, this is why we do it that way and this is something I can help you do but fundamentally, we need to instrument this code in a way that we can actually observe it and to understand how it is actually operating when we start before we can actually open the front door and let some of that crazy – and let the Internet in,” right? We need to be able to understand how and when the doors fall off and if we are not working with our developers who are more focused on understanding, does this function do what it says on the box? Rather than, is this function implemented in a way that might admit events or metrics, right? This is a completely different set of problems from the developer’s perspective. I have seen a couple of different implementations of how to implement this within an organization and one of them is Facebook’s idea of product engineering or I think it is called product engineering or production engineering, one of the two, and so this idea is that you might have somebody who’s similar in some ways to an SRE. Somebody who understands the infrastructure and understands how to build applications that will reside upon it and is actually embedded with your developer team to say, “You know, before we can legit sign off on this thing, here are the things that this application must have to be able to wire into to enable us to operate this app so that we can observe it and monitor it. Do all the things that we need to do.”And the great part about that is that it means that you are teaming with the developer team, you have some engineering piece that is teaming with the developer team and enabling them to understand why these tools are there and what they’re for and really – and promoting that engagement. [0:30:59.9] CC: And getting to that place is an interesting proposition isn’t it? Because, as a developer, even as a developer, I see the world moving more and more towards developer taking on the ship of the apps and knowing more, more layers of this stack, and if I am a developer and I want to implement, incorporate these practices then I need to convince some one, either a developer, or whoever is in charge of monitoring it and making sure the system is up and running right? [0:31:33.0] KN: Yeah. [0:31:34.3] CC: So one way to go about quantifying the need for that is to say, “Well over the last month we spent X amount of hours trying to find a bug in production,” and that X is a huge number. So you can bring that number and say, “This is how much the number costs in engineering hours,” but on the other hand, you don’t want to be the one to say that it takes you a 100 hours to find one little bug in production, do you? [0:32:06.2] KN: Yeah, I mean I feel like this is why agile teams are so successful because baked into how you do your work is sort of this implicit way of tracking your time and your progress. So at the end of the day, if you do spend a 100 hours of work trying to find a bug, it is sort of like, that is the team’s hours that is not your hours and you sort of get this data for free at the end of every sprint. [0:32:28.1] CC: Yeah. [0:32:28.5] DC: What you brought up is actually another cultural piece of that that I think is a problem. It has to be – I think that frequently we assume there are many – let me put this differently. I have seen companies where in the culture is somewhat damming for people who spend a lot of time trying to troubleshoot something that they wrote and that is a terrible pattern because it means that the people who are out there writing the code, who are just trying to get across the finish line with the thing that needs to be in production, right, have now this incredible pressure on them to not make a mistake and that is not okay. We are all here to make mistakes. That is what we do professionally, is make mistakes and the rest is just the gravy, you know what I mean? And so yeah, it makes me nuts that there are organizations that are like that. I feel like we really just in it and what is awesome about this is I see that narrative raising up within the ecosystem that I, you know, around cloud native architectures and other things like that, is that like, you know, you are hired to do a hard job and if we come down on you for thinking that that is a hard job then we are messing up. You are not messing up. [0:33:37.2] CC: Yeah absolutely. Building software is very hard and complex. So if you are not making mistakes, you either are not human or you are not making enough changes and in today’s world, we still have humans making software instead of robots. We are not there yet but it is a very risky proposition not to be making continuous changes because you will be left behind. [0:34:04.7] KN: Yeah, I feel like there is definitely something to be said about empathy for software engineers. It is very easy to be like, “Oh my gosh you spent a 100 hours looking at this one bug to save 20 dollars, how dare you?” but it is also a lot harder to be like, “Oh you poor thing, you had to dig through a 100 million lines of somebody else’s code in order to find this bug and it took you a 100 hours and you did all of that just to fix this one little bug, how awesome are you?” And I feel like that is where we get into the team dynamic of are we a blame-centric team? Do we try to assign blame to a certain person or do we look at this as a team’s responsibility, like this is our code and poor Carlisia over here had to go dig through this code that hasn’t been touched in 10 years,” or whatever. [0:34:54.6] CC: Another layer to that is that in my experience, I have never done anything in software or looked at any codes or brought up any system that as trivial as the end result was and especially in relation with the time spent, it has never happened that it wasn’t a huge amount of education that I got to reuse in future work. So does that make sense? [0:35:20.2] DC: Yeah and that is what I was referring to is around being able to build up the intuition around how these systems operate like if the longer, the more time you spend in the trenches working on those things, right? If you are enabled leveraging technologies like observability and chaos into the grain to troubleshoot, to come up with a hypothesis about how this would break when this happens, and test it and view the result and come up with a new hypothesis and continue down that path, you will automatically, I mean like, by your nature, build a better intuition yourself around how all of these system operate. It doesn’t matter whether it is the application you are working on or some other application, you are going to be able to build up their intuition for how to understand and characterize systems in general. You’ll be a better engineer for distributed systems if you are in a culture that is blameless that gives you tools to experiment and gives you tools to validate those experiments and come up with new ones, you know? [0:36:21.3] CC: I am going to challenge you and then I am going to agree with you so hang on, okay? So I am going to challenge you, so we are saying that observability, which actually boils down to using automated tools to do all of this work for us that we don’t have to dig in manually on a case by case basis, no that’s wrong? [0:36:43.4] DC: No, I am saying observability is a set of tools that you can use to observe the interactions and behavior of distributed systems. [0:36:53.4] CC: Okay but with automated tools right? [0:36:55.2] DC: The automation piece isn’t really I mean do you want to take this one Kris? [0:36:59.5] KN: Yeah, I mean I think like they certainly can be automated. I just don’t think that there is a hard bit of criteria that says every one needs to be automated. Like there ain’t nothing wrong with SSH-ing into a server and running a debug script or something if you are having a really bad day. [0:37:12.3] CC: Okay but let me go with my theory, just pretend it is because it will sound better. All right, so let us say, not to exclude the option to do it manually too if you want, but let’s say we have these wonderful tools that can automate a bunch of this work for us and we get to look at it from a high level. So what I am thinking is whereas before, if we didn’t have or use those tools or we are not using those tools, we have to do a lot of that work manually. We have to look at a lot more different places and I will challenge you that we develop even more intuition that way. So we are decreasing the level of intuition that we develop potentially by using the tools. Now, I am going to agree with you. It was just a rationale that I had to follow. I agree with you, it definitely helps to develop intuition but it is a better quality of intuition because now you can hold these different pieces in your hand because you are looking at it at this higher level. Because when you look at the details, you look at thing at this view – at least I am like that. It is like, “Okay, can I hold this one thing, it is big already in my head,” and then for me when I switch context and go look at something else, you know what I looked at over there, and it is hard to, really hard to keep track and really wasteful for – it is impossible to keep all of it in your mind, right? And let’s say you have to go through the whole debugging process all over again. If I don’t have notes, it will be like just the first time because I can’t possibly remember. I mean I have been in situations of having to debug different systems and okay, now third time around I am taking notes because the fourth time is just going to be so painful. So having tools that lets us look at things at a higher level I think has the additional benefit of helping us understand the system and hold it together in our heads because okay, we definitely don’t know the little details of how these are happening behind the scene. But how useful is that anyway? I’d much rather know how the whole system works together, points of failure like I can visualize, right? [0:39:30.1] DC: Yeah. [0:39:30.7] KN: I have a question for everyone. Following up on Carlisia’s how she challenged you and then agreed with you, I really want to ask this question because I think Carlisia’s answer is going to be different than Duffy’s and I think that is going to say a lot about the different ways that we are thinking about observability here and it is really fascinating if you think about it. So have either of you worked in a shop before where you had ‘the guy’? You know, that one person who just knew the code base inside and out, he had been around for forever, he was a dinosaur and whatever something went wrong you’re like, “We got to get this guy on the phone,” and he would come in and be like, “Oh it is this one line and this one thing that it would take you six months to figure out but let me just fix this really quick,” bam-bam-bam-bam and production is back online. [0:40:14.1] CC: Oh code base guy, the system admins guy, like something that is not my app but the system broke, you get that person who knew every like could take one second to figure out what the problem was. [0:40:28.9] KN: Have you seen that before though? Like that one person who just have so much tribal knowledge. [0:40:33.3] CC: Yeah, absolutely. [0:40:34.3] KN: Yeah, Duffy what about you? [0:40:36.0] DC: Absolutely. I have both been that guy and seen that guy. [0:40:38.8] CC: I have never been that person. [0:40:40.2] DC: In lots of shops. [0:40:42.4] KN: Well what I am kind of digging at here is I think observability, and I mean this in the nicest way possible to all of our folks at home who are actively playing the role of ‘the guy’, I think observability kind of makes that problem go away, right? [0:40:56.8] DC: I think it normalizes it to your point. I think that it basically gives you – I think you’re onto it. I think that I agree with you but I think that fundamentally what happens is through tooling like chaos engineering, through tooling like observability, you are normalizing what it looks like to teach anybody to be like that person, right? But that is the key takeaway is like, to Carlisia’s point she might – actually, you know Kris and I, I promise that we will approach some complex distributed systems problem fundamentally differently, right? If somebody has a broken Kubernetes cluster, Kris and I are both going to approach that same problem and we will likely both be able to solve that problem but we are going to approach it in different ways and I think that the benefit of having common tooling with which to experiment and understand and observe the behavior of these distributed systems means that, you know, we can normalize what it looks like to be a developer and have a theory about how the system is breaking or would break, and having some way of actually validating that through the use of observability and perhaps chaos engineering depending, and that means that we are turning keys over, turning the keys to the castle over. There is no more bust test. You don’t have to worry about what happens to me at the end of the day. We all have this common receptacle. [0:42:16.7] CC: You could go on vacation. [0:42:18.1] DC: Yeah. [0:42:19.0] CC: No but this is the most excellent point, I am glad you brought it up Nova because what both of you said is absolutely true. I mean, give me a better documentation and I don’t need you anymore because I can be self-sufficient. [0:42:33.2] DC: Exactly. [0:42:34.9] KN: Yeah so when we’re – [0:42:35.5] CC: If you told me to observe where things went wrong and again I go back to that what I said, more and more developers are having to take being asked, I mean some developers are proactively taking on the shift and in other cases they’d been asked to take more ownership of the whole stack and then say from the application level down the stack and, but you gave me tools to observe where things went wrong beyond my code as a developer, I am not going to call the guy. [0:43:07.3] KN: Yeah. [0:43:08.4] CC: So the level of self-sufficient – [0:43:10.3] KN: The guy doesn’t want you to call him. [0:43:12.5] CC: So it provides – and then the decision – benefit, we could say, is provide the engineer an additional level of self-sufficiency. [0:43:22.0] KN: Yeah, I mean teach someone to fish, give someone a fish. [0:43:25.1] CC: Yeah. [0:43:25.6] KN: Yeah. [0:43:26.4] DC: Exactly. All right, well that was a great conversation on observability and we talked about a bunch of different topics. This is Duffy and I had a great time in this session and thanks. [0:43:38.1] CC: Yeah, we are super glad to be here today. Thanks for listening. Come back next week. [0:43:43.4] KN: Thanks for joining everyone and I apologize again to all of our ‘guys’ at home listening. Hopefully we can help you with observability along the way to get everybody’s job a little bit easier. [0:43:53.8] CC: And I want to say you know for the girls, we know that you are all there too. That is just a joke. [0:43:59.9] KN: Oh yeah, I was totally at it for a while. Good show everyone. [0:44:05.3] DC: All right, cheers. [0:44:06.8] KN: Cheers. [END OF INTERVIEW] [0:44:08.7] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter https://twitter.com/ThePodlets and on the https://thepodlets.io, where you will find transcripts and show notes. We’ll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

In this episode of The Podlets Podcast, we are diving into contracts and some of the building blocks of the Cloud-Native application. The focus is on the importance of contracts and how API's help us and fit into the cloud native space. We start off by considering the role of the API at the center of a project and some definitions of what we consider to be an API in this sense. This question of API-first development sheds some light onto Kubernetes and what necessitated its birth. We also get into picking appropriate architecture according to the work at hand, Kubernetes' declarative nature and how micro-services aid the problems often experienced in more monolithic work. The conversation also covers some of these particular issues, while considering possible benefits of the monolith development structure. We talk about company structures, Conway's Law and best practices for avoiding the pitfalls of these, so for all this and a whole lot more on the subject of API's and contracts, listen in with us, today! Note: our show changed name to The Podlets. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback and episode suggestions: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Josh Rosso Duffie Cooley Patrick Barker Key Points From This Episode: • Reasons that it is critical to start with APIs at the center. • Building out the user interface and how the steps in the process fit together. • Picking the way to approach your design based on the specifics of that job. • A discussion of what we consider to qualify as an API in the cloud-native space. • The benefit of public APIs and more transparent understanding. • Comparing the declarative nature of Kubernetes with more imperative models. • Creating and accepting pods, querying APIs and the cycle of Kubernetes. • The huge impact of the declarative model and correlation to other steps forward. • The power of the list and watch pattern in Kubernetes. • Discipline and making sure things are not misplaced with monoliths.• How micro-services goes a long way to eradicate some of the confusion that arises in monoliths. • Counteracting issues that arise out of a company's own architecture. • The care that is needed as soon as there is any networking between services. • Considering the handling of an API's lifecycle through its changes. • Independently deploying outside of the monolith model and the dangers to a system.• Making a service a consumer of a centralized API and flipping the model. Quotes: “Whether that contract is represented by an API or whether that contract is represented by a data model, it’s critical that you have some way of actually defining exactly what that is.” — @mauilion [0:05:27] “When you just look at the data model and the concepts, you focus on those first, you have a tendency to decompose the problem.” — @pbarkerco [0:05:48] “It takes a lot of discipline to really build an API first and to focus on those pieces first. It’s so tempting to go right to the UI. Because you get these immediate results.” — @pbarkerco [0:06:57] “What I’m saying is, you shouldn’t do one just because you don’t know how to do the others, you should really look into what will serve you better.” — @carlisia [0:07:19] Links Mentioned in Today’s Episode: The Podlets on Twitter — https://twitter.com/thepodlets Nicera — https://www.nicera.co.jp/ Swagger — https://swagger.io/tools/swagger-ui/ Jeff Bezos — https://www.forbes.com/profile/jeff-bezos/ AWS — https://aws.amazon.com/ Kubernetes — https://kubernetes.io/ Go Language — https://golang.org/ Hacker Noon — https://hackernoon.com/ Kafka — https://kafka.apache.org/ etcd — https://etcd.io/ Conway’s Law — https://medium.com/better-practices/how-to-dissolve-communication-barriers-in-your-api-development-organization-3347179b4ecc Java — https://www.java.com/ Transcript: EPISODE 03 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.2] D: Good afternoon everybody, my name is Duffy and I’m back with you this week. We also have Josh and Carlisia and a new member of our cast, Patrick Barker. [0:00:49.4] PB: Hey, I’m Patrick, I’m an upstream contributor to Kubernetes. I do a lot of stuff around auditing. [0:00:54.7] CC: Glad to be here. What are we going to talk about today? [0:00:57.5] D: This week, we’re going to talk about some of the building blocks of a cloud native application. This week we’re going to kind of focus on contracts and how API’s kind of help us and why they’re important to cloud native ecosystem. Usually, with these episodes, we start talking about the problem first and then we kind of dig into why this particular solution, something like a contract or an API is important. And so, to kind of kick that of, let’s talk about maybe this idea of API-first development and why that’s important. I know that Josh and Patrick both and Carlisia have all done some very interesting work in this space as far as developing your applications with that kind of a model in mind. Let’s open the floor. [0:01:34.1] PB: It’s critical to build API-centric. When you don’t build API-centric, most commonly, you’ll see a cross ecosystem building UI centric, it’s very tempting to do this sort of thing because UI’s are visually enticing and they’re kind of eye candy. But when you don’t go to API-centric and you go that direction, you kind of miss the majority of use cases down the line which are often around an SCK, just ended up being more often than not the flows that are the most useful to people but they’re kind of hard to see it to be getting. I think going and saying we’re building a product API-first is really saying, we understand that this is going to happen in the future and we’re making this a principle early, we’re going to enforce these patterns early, so that we develop a complete product that could be used in many fashions. [0:02:19.6] J: I’ve seen some of that in the past as well working for a company called Nicera, which is a network virtualization company. We really focused on providing an API that would be between you and your network infrastructure and I remember that being really critical that we define effectively what would be the entire public API for that product out in front and then later on, we figured out what obviously to learn this semantics of that sort, to be able to build a mental model around what that API might be, that’s where the UI piece comes in. That was an interesting experiment and like, we ended up actually kind of creating what was the kind of creating what was kind of the – an early version of the Swagger UI in which you basically had a UI that would allow you to explore and introspect and play with, all of those different API objects but it wasn’t a UI in the sense that you know, it had like a constrained user story that was trying to be defined, that was my first experience where I was working with a product that had an API-first model. [0:03:17.0] CC: I had to warm up my brain, I think about why do we build API’s to begin with before I could think why API-first is of a benefit and where the benefits are. And I actually looked up something today and it’s this Jeff Bezos mandate, I had seen this before, right? I mean, why do we view the API’s? API what you’re talking about is data transfer, right? Taking data from over here and sending it over there or you’re making that available so somebody can fetch it. It’s communication. Why do we build API? To make it easier to do that so you can automate, you can expose it, you can gate it with some security, right? Authentication, all of those things and with every increasing amount of data, this becomes more and more relevant and I think when Patrick was saying, when you do it API first, you’re absolutely focusing on making it all of those characteristics a priority, making that work well. If you want to make it pretty, okay, you can take that data in. Transforming some other way to make your presentation pretty, to display on the mobile device or whatever. [0:04:26.4] PB: Yeah, I think another thing with inserting the API design upfront in the software development lifecycle, at least in my experience has been – it allows you to sort of gather feedback from who your consumers will be early on before you worry about the intricacies of all the implementation details, right? I guess with Nicera’s instant stuff, I wonder when you all made that contract, were you pushing out a Swagger UI or just general API documentation before you had actually implemented the underlying pieces or did that all happen together? [0:04:58.1] D: With an API-first, we didn’t build out the UI until after the fact so even to the point where we would define a new object in that API, like a distributed logical router for example. We would actually define that API first and we would have test plants for it and all of that stuff and t hen we would surface it in the UI part of it and that’s a great point. I will say that it is probably to your benefit in the long run to define what all of the things that you’re going to be concerned with are out front. And if you can do that tin a contractual basis, whether that contract is represented by an API or whether that contract is represented by a data model, it’s critical that you have some way of actually defining exactly what that is so that you can also support things like versioning and being able to actually modify that contract as you move forward. [0:05:45.0] PB: I think another important piece here, too, is when you just look at the data model and the concepts, you focus on those first, you have a tendency to more decompose the problem, right? You start to look at things and you break it down better into individual pieces that combine better and you end up with more use cases and you end up with a more useable API. [0:06:03.2] D: That’s a good point. Yeah, I think one of the key parts of this contract is kind of like what you’re trying to solve and it’s always important, you know? I think that, when I talk about API-first development, it is totally kind of in line with that, you have to kind of think about what all the use cases are and if you’re trying to develop a contract that might satisfy multiple use cases, then you get this great benefit of thinking of it as you can kind of collapse a lot of the functionality down into a more composable API, rather than having to solve each individual use cases and kind of a myopic way. [0:06:34.5] CC: Yeah, it’s the concept of reusability, having the ability of making things composable, reusable. [0:06:40.7] D: I think we probably all seen UI’s that gets stuck in exactly that pattern, to Patrick’s point. They try to solve the user story for the UI and then on the backend, you’re like, why do we have two different data models for the same object, it doesn’t make sense. We have definitely seen that before. [0:06:57.2] PB: Yeah, I’ve seen that more times than not, it takes a lot of discipline to really build a UI or an API, you know, first to focus on those pieces first – it’s so tempting to go right to the UI because you get these immediate results and everyone’s like – you really need to bring that back, it takes discipline to focus on the concepts first but it’s just so important to do. [0:07:19.5] CC: I guess it really depends on what you are doing too. I can see all kinds of benefits for any kind of approach. But I guess, one thing to highlight is that different ways of doing it, you can do a UI-first, presentation first, you can do an API-first and you can do a model-first so those are three different ways to approach the design and then you have to think well, what I’m saying is, you shouldn’t do one just because you don’t know how to do the others, you should really look into what will serve you better. [0:07:49.4] J: Yeah, with a lot of this talk about API’s and contracts, obviously in software, there’s many levels of contracts we potentially work on, right? There’s the higher level, potential UI stuff and sometimes there’s a lower level pieces with code. Perhaps if you all think it’s a good idea, we could start with talking about what we consider to be an API in the cloud native space and what we’re referring to. A lot of the API’s we’ve described so far, if I heard everyone correctly, they sounded like they were more so API, as describing perhaps a web service of sorts, is that fair? [0:08:18.8] PB: That’s an interesting point to bring up. I’m definitely describing the consumption model of a particular service. I’m referring to that contract as an infrastructure guy, I want to be able to consume an API that will allow me to model or create infrastructure. I’m thinking of it from that perspective. If AWS didn’t have an API, I probably wouldn’t have adopted it, like the UI is not enough to do this job, either, like I need something that I could tie to better abstractions, things like terraform and stuff like that. I’m definitely kind of picturing it from that perspective. But I will add one other interesting point to this which is that in some cases, to Josh’s point, these things are broken up into public and private API’s, that might be kind of interesting to dig into. Why you would model it that way. There are certainly different interactions between composed services that you’re going to have to solve for. It’s an interesting point. [0:09:10.9] CC: Let’s hold that thought for a second. We are acknowledging that there are public and private API’s and we could talk about why their services work there. Other flavors of API’s also, you can have for example, a web service type of API and you can have a command line API, right? You can see a line on top of a web service API which is the crazy like, come to mind, Kubernetes but they have different shapes and different flavors even though they are accessing pretty much the same functionality. You know, of course, they have different purposes and you have to see a light and another one, yet, is the library so in this case, you see the calls to library which calls the web service API but like Duffy is saying, it’s critical sometimes to be able to have this different entry points because each one has its different advantages like a lot of times, it’s way faster to do things on the command line than it is to be a UI interface on the web that would access that web API which basically, you do want to have. Either your Y interface or CLA interface for that. [0:10:21.5] PB: What’s interesting about Kubernetes too and what I think they kind of introduced and someone could correct me if I’m wrong but is this kid of concept of a core generative type and in Kubernetes, it ends up being this [inaudible]. From the [inaudible], you’re generating out the web API and the CLI and the SCK and they all just come from this one place, it’s all code gen out of that. Kubernetes is really the first place I’ve seen do that but it’s really impressive model because you end up with this nice congruence across all your interfaces. It just makes the product really rockable, you can understand the concepts better because everywhere you go, you end up with the same things and you’re interacting with them in the same way. [0:11:00.3] D: Which is kind of the defining of type interface that Kubernetes relates to, right? [0:11:04.6] PB: Obviously, Kubernetes is incredibly declarative and we could talk a bit about declarative versus imperative, almost entirely declarative. You end up with kind of a nice, neat clear model which goes out to YAML and you end up a pretty clean interface. [0:11:19.7] D: If we’re going to talk about just the API as it could be consumed by other things. I think we’re kind of talking a little bit about the forward facing API, this is one of those things that I think Kubernetes does differently than pretty much any other model that I’ve seen. In Kubernetes, there are no hidden API’s, there’s not private API. Everything is exposed all the time which is fascinating. Because it means that the contract has to be solid for every consumer, not just the ones that are public but also anything that’s built on the back end of Kubernetes, the Kublet, controller manager, all of these pieces are going to be accessing the very same API that the user does. I’ve never seen another application built this way. In most applications, what I see is actually that they might define an API between particular services that you might have a contract between those particular services. Because this is literally — to Carlisia’s point, in most of the models that I’ve seen API’s are contract written, this is about how do I get data or consume data or interact with data, between two services and so there might be a contract between that service and all of its consumers, rather than between the course or within all of the consumers. [0:12:21.7] D: Like you said, Kubernetes is the first thing I’ve seen that does that. I’m pulling an API right now, there’s a strong push of internal API’s for it. But we’re building on top a Kubernetes product and it’s so interesting how they’ve been able to do that, where literally every API is public and it works well, there really aren't issues with it and I think it actually creates a better understanding of the underlying system and more people can probably contribute because of that. [0:12:45.8] J: On that front, I hope this is a good segue but I think it would be really interesting to talk about that point you made Patrick, around declarative versus imperative and how the API we’re discussing right now with Kubernetes in particular, it’s almost entirely declarative. Could you maybe expand on that a bit and compare the two? [0:13:00.8] PB: It’s interesting thing that Kubernetes has really brought to the forefront – I don’t know if there’d be another notable declarative API be terraform. This notion of you just declare state within a file and in some capacity, you just apply that up to a server and then that state is acted on by a controller and it brings us straight to fruition. I mean, that’s almost indicative of Kubernetes at this point I think. It’s so ingrained into the product and it’s one of the first things to kind of do that and that it’s almost what you think of when you think of Kubernetes. And with the advent of CRD’s and what not, that’s now, they want to be extended out to really in the use case you would have, that would fit this declarative pattern of just declaring to say which it turns out there’s a ton of use cases and that’s incredibly useful. Now, they’re kind of looking at, in core Kubernetes, could we add imperative functionality on top of the declarative resources, which is interesting too. They’re looking at that for V2 now because there are limitations, there are some things that just do fit in to declarative pattern perfectly that would fit just the standard rest. You end up some weird edges there. As they’re going towards V2, they’re starting to look at could we mix imperative and declarative, which is and even maybe more interesting idea if you could do that right. [0:14:09.3] CC: In the Kubernetes world, what would that look like? [0:14:11.3] PB: Say you have an object that just represents something like on FOO, you have a YAML file and you're declaring FOO to be some sort of thing, you could apply that file and then now that state exist within the system and things noticed that that state of change that they’re acting on that state, there are times when you might want that FOO to have another action. Besides just applying states, you may want it to have some sort of capability on top of the point, let’s say, they’re quite a few use cases that come in where that turns into a thing. It’s something to explore, it’s a bit of a Pandora’s box if you will because where does that end. Kubernetes is kind of nice that it does enforce constraints at this core level and it produces these really kind of deep patterns within the system that people will find kind of easy to understand at least at a high level. Granted, you go deep into it, it gets highly complex but enforcing like name spaces as this concept of just a flat name space with declarative resources within it and then declarative resources themselves just being confined to the standard rest verbs, is a model that people I think understand well. I think this is part of the success for Kubernetes is just that people could get their hands around that model. It’s also just incredibly useful. [0:15:23.7] D: Another way to think about this is like, you probably seen articles out there that kind of describe the RESTful model and talking about whether REST can be transactional. Let’s talk a little bit about what that means. I know the implementation of an API pattern or an interface pattern might be. That the client sends information to the server and that the server locks that client connection until it’s able to return the result, whatever that result is. Think of this, in some ways, this is very much like a database, right? As a client of a database, I want to insert a row into a database, the database will lock that row, it will lock my connection, it will insert that row and it will return success and in this way, it’s synchronous, right? It’s not trying to just accept the change, it just wants to make sure that it returns to a persisted that change to the database before, letting go of the connection. This pattern is probably one of the most common patterns in interfaces in the world like it is way super common. But it’s very different than the restful pattern or some of the implementations of a restful pattern. In which what we say, especially in this declarative model, right? In a declarative model, the contract is basically, I’m going to describe a thing and you're going to tell me when you understand the thing I want to describe. It’s asynchronous. For example, if I were interacting with Kubernetes and I said, cube kettle create pod, I would provide the information necessary to define that pod declaratively and I would get back from the API server 200 okay, pod has been accepted. It doesn’t mean to it's been created. It means it’s been accepted as an object and persisted to disk. Now, to understand from a declarative perspective, where I am in the life cycle of managing that pod object, I have to query that API again. Hey, this pod that I ask you to make, are you done making it and how does this work and where are you in that cycle of creating that thing? This is where I like within Kubernetes, we have the idea of a speck which defines all of the bits that are declaratively described and we have the idea of a status which describes what we’ve been up to around that declarative object and whether we’ve actually successfully created it or not. I would argue that from a cloud native perspective that declarative model is critical to our success. Because it allows us to scale and it allows us to provide an asynchronous API around those objects that we’re trying to interact with and it really changes the game as far as like, how we go about implementing those inputs. [0:17:47.2] CC: This is so interesting, it was definitely a mind bender for me when I started developing against Kubernetes. Because what do you mean you’ve returned the 200 okay, and the thing is not created yet. When does it get created? It’s not hard to understand but I was so not used to that model. I think it gives us a lot of control. So it is very interesting that way and I think you might be right, Duffy, that it might be critical to the success of native apps because it is always like the way I am thinking about it right now just having heard you is almost like with all the models, let’s say you are working with a database in that transactional system. The data has be inserted and that system decides to retry or not once the transaction is complete as we get a result back. With a Kubernetes model or cloud native model, I don’t know what, which is both a proper things to say, the control is with us. We send the request, Kubernetes is going to do its thing, which allows us to move on too, which is great, right? Then I can check for the result, when I want to check and then I can decide what to do with the results when I want to do anything with it if it all, I think it gives us a lot more control as developers. [0:19:04.2] D: Agreed. And I think another thing that has stuck in my head around this model whether it would be declared over imperative is that I think that Go Lang itself has actually really enabled us to adopt that asynchronous model around things that threads are first class, right? You can build a channel to handle each individual request, that you are not in this world where all transactions have to stop until this one is complete and then we’ll take the next one out of queue and do that one. We're no longer in that kind of a queue model, we can actually handle these things in parallel quite a bit more. It makes you think differently when you are developing software. [0:19:35.9] J: It’s scary too that you can check this stuff into a repo. The advent of Git Ops is almost parallel to the advent of Kubernetes and Terra Form and that you can now have this state that is source controlled and then you just apply it to the system and it understands what to do with it and how to put all of the pieces together that you gave it, which is a super powerful model. [0:19:54.7] D: There is a point to that whole asynchronous model. It is like the idea of the API that has a declarative or an imperative model and this is an idea and distributed system that is [inaudible]. It is like edge triggering or level triggering but definitely recommend looking up this idea. There is a great article on it on Hack Noon and what they highlight is that the pure abstract perspective there is probably no difference between edge and level triggering. But when you get down to the details especially with distributed systems or cloud native architectures, you have to take into account the fact that there is a whole bunch of disruption between your services pretty much all the time and this is the challenge of distributed systems in general, when you are defining a bunch of unique individual systems that need to interact and they are going to rely on an unreliable network and they are going to rely on unreliable DNS. And they’re going to rely on all kinds of things that are going to jump in the way of between these communication models. And the question becomes how do you build a system that can be resilient to those interruptions. The asynchronous model absolutely puts you in that place, where if you are in that situation wherein you say, “Create me a pod.” And that pod object is persisted and now you can have something else to do the work that will reconcile that declared state with the actual state until it works. It will just keep trying and trying and trying until it works. In other models, you basically say, “Okay, well what work do I have to do right now and I have to focus on doing this work until it stops.” What happens if the process itself dies? What happens if any of the interruptions that we talk about happen? Another good example of this is the Kafka model versus something like a watch on etcd, right? In Kafka, you have these events that you are watching for. And if you weren’t paying attention when that event went by, you didn’t get that event. It is not still there. It is gone now whereas like with etcd and models like that, what you are saying is I need to reconcile my expectancy of the world with what the desired thing is. And so I am no longer looking for events. I am looking for a list of work that I have to reconcile to, which is a very different model for these sorts of things. [0:21:47.9] J: In Kubernetes, it becomes the informer pattern. If you all don’t know, which is basically at the core of the informer is just this concepts of list and watch where you are just watching for changes but every so often you list as well in case you missed something. I would argue that that pattern is so much more powerful than the Kafka model you’re just going to skin as well because like you mentioned, if you missed an event in Kafka somehow, someway is very difficult to reconcile that state. Like you mentioned, your entire system can go down in a level set system. You bring it back up and because it is level set, everything just figures itself out, which is a lot nicer than your entire system going down in an edge-based system and trying to figure out how to put everything back together yourself, which is not a fun time, if you have ever done it. [0:22:33.2] D: These are some patterns in the contracts that we see in the cloud native ecosystem and so it is really interesting to talk about them. Did you have another point Josh around API’s and stuff? [0:22:40.8] J: No, not in particular. [0:22:42.2] D: So I guess we give into like what some of the forms of these API’s to talk about. We could talk about RESTful API’s versus to TIPC-based API’s or maybe even just interfaces back and forth between modular code and how that helped you architect things. One of the things I’ve had conversations with people around is we spend a lot of our time conditioning our audience when in cloud native architecture to the idea that monliths are bad, bad, bad and they should never do them. And that is not necessarily true, right? And I think it is definitely worth talking through like why we have these different opinions and what they mean. When I have that conversation with customers, frequently a monolith makes sense because as long as you’re able to build modularity into it and you are being really clear about the interfaces back and forth between those functions with the idea that if you have to actually scale traffic to or from this monolith. If the function that you are writing needs to be effectively externalized in such a way that can handle an amount of work that will surpass what the entire monolith can handle. As long as you are really clear about the contract that you are defining between those functions then later on, when it comes to a time to externalize those functions and embrace kind of a more microservices based model mainly due to traffic reload or any of the other concerns that kind of drive you toward a cloud native architecture, I think you are in a better spot and this is definitely one of the points of the contract piece that I wanted to raise up. [0:24:05.0] CC: I wonder though how hard it is for people to keep that in mind and follow that intention. If you have to break things into micro services because you have bottlenecks in your monolith and maybe have to redo the whole thing, once you have the micro services, you have gone through the exercise of deciding, you know this goes here, these goes there and once you have the separate modules it is clear where they should go. But when you have a monolith it is so easy to put things in a place where they shouldn’t be. It takes so much discipline and if you are working on a team that is greater than two, I don’t know. [0:24:44.3] PB: There are certain languages that lend themselves to these things like when you are writing Java services or there are things where it is easy to — when writing even quickly, rapidly prototyping an application that has multiple functions to be careful about those interfaces that you are writing, like Go because it is a strongly type language kind of forces you into this, right? There are some other languages that are out that make it difficult to be sloppy about those interfaces. And I think that is inherently a good thing. But to your point like you are looking at some of the larger monoliths that are out there. It is very easy to fall into these patterns where instead of an asynchronous API or an asynchronous interface, you have just a native interface and you are a asynchronous interface in which you expect that I would be able to call this functional and put something in there. I will get the result back and that is a pattern for monoliths. Like that is how we do it in monoliths. [0:25:31.8] CC: Because you say in there also made me think of the Conway’s Law because when we separate these into micro services and I am not saying micro services is right for everything for every team and every company. But I am just saying if you are going through that exercise of separating things because you have bottlenecks then maybe in the future you have to put them elsewhere. Externalize them like you said. If you think if the Conway’s Law if you have a big team, everybody working on that same monolith that is when things are in depth in the place that they shouldn’t be. The point of micro services is not just to technically separate things but to allow people to work separately and that inter-team communication is going to be reflected in the software that they are creating but because they are forced to communicate and hopefully they do it well that those micro services should be well-designed but if you have a monolith and everyone working on the same project, it gets more confusing. [0:26:31.4] D: Conway’s Law as an overview is basically that an organization will build software and laid out similar to the way the thought musician itself is architected. So if everybody in the entire company is working on one thing and they are really focused on doing that one thing, you’d better build a monolith. If you have these groups that are disparate and are really focused on some subset of work and need to communicate with each other to do that thing then you are going to build something more similar or maybe more capable as a micro service. That is a great point. So actually one of the things about [inaudible] that I found so fascinating with it, it would be a 100 people and we were everywhere. So communication became a problem that absolutely had to be solved or we wouldn’t be able to move forward as a team. [0:27:09.5] J: An observation that I had in my past life helping folks, breaking apart Java monoliths like you said Duffy, assume they had really good interfaces and contracts right? And that made it a lot easier to find the breaking points for their API’s to pull those API’s out into a different type of API. They went from this programmatic API, that was in the JBM where things were just intercommunicating to an API that was based on a web service. And an interesting observation I oftentimes found was that people didn’t realize that in removing complexity from within the app to the network space that oftentimes caused a lot of issues and I am not trying to down API’s because obviously we are trying to talk about the benefits of them but it is an interesting balancing act. Oftentimes when you are working with how to decouple a monolith, I feel like you actually can go too far with it. It can cause some serious issues. [0:27:57.4] D: I completely agree with that. That is where I wanted to go with the idea of why we say that building a monolith is bad and like with the challenges of breaking those monoliths apart later. But you are absolutely right. When you are going to introduce the wild chaos that is a network between your services, are you going to externalize functions and which means that you have to care a lot more about where you store a state because that state is no longer shared across all of the things. It means that you have to be really super careful about how you are modeling that. If you get to the point where this software that you built that is a monolith that is wildly successful and all of its consumers are networked based, you are going to have to come around on that point of contracts. Another thing that we haven’t really talked on so much is like we all agree that maybe like an API for say the consumer model is important. We have talked a little bit about whether private API’s or public API’s make sense. We described one of the whacky things that Kubernetes does, which is that there are no private API’s. It is all totally exposed all the time. I am sure that all of us have seen way more examples of things that do have a private API mainly because perhaps the services are trained. Service A always fact to service B. Service B has an API that it may be a private API. You are never going to expose to your external customers only to service A or to consumers of that internal API. One of the other things that we should talk about is when you are starting to think about these contracts. One of the biggest and most important bits is how you handle the lifecycle of those API’s, as they change right? Like I say add new features or functionality or as I deprecate old features and functionality, what are my concerns as it relates to this contract. [0:29:33.5] CC: Tell me and take my money. [0:29:37.6] D: I wish there was like a perfect answer. But I am pretty convinced that there are no perfect answers. [0:29:42.0] J: I spent a lot of time in the space recently and I have researched it for like a month or so and honestly, there are no perfect answers to try to version an API. Every single on of them has horrible potential consequences to it. The approach Kubernetes took is API evolution, where basically all versions of the API have to be backwards compatible and they basically all translate to what is an internal type in Kubernetes and everything has to be translatable back to that. This is nice for reasons. It is also very difficult to deal with at times because if you add things to an API, you can’t really every remove them without a massive amount of deprecation effort basically moderating the usage of that API specifically and then somehow deprecating it. It is incredibly challenging. [0:30:31.4] PB: I think it is 1-16 in which they finally turn off a lot of the deprecated API’s that Kubernetes had. So a lot of this stuff that has been moved for some number of versions off to different spaces for example deployments used to be extensions and now they are in apps. They have a lot of these things. Some of the older API’s are going to be turned off by default in 1-16 and I am really interested to see how this plays out you know from kind of a chaos level perspective. But yeah you’re right, it is tough. Having that backwards compatibility definitely means that the contract is still viable for your customers regardless of how old their client side looks like but this is kind of a fingernail problem, right? You are going to be in a situation where you are going to be holding those translations to that stored object for how many generations before you are able to finally get rid of some of those old API’s that you’ve have obviously moved on from. [0:31:19.6] CC: Deprecating an end point is not reviewed at all and ideally like better with, you would be able to monitor the usage of the end point and see as you intend deprecating is the usage is going lower and if there is anything you can do to accelerate that, which actually made me think of a question I have for you guys because I don’t know the answer to this. Do we have access to the end points usage, the consumption rate of Kubernetes end points by any of the cloud service providers? It would be nice if we did. [0:31:54.9] D: Yeah, there would be no way for us to get that information right? The thing about Kubernetes is something that you are going to run on your own infrastructure and there is no phone home thing like that. [0:32:03.9] CC: Yeah but the cost providers could do that and provide us a nice service to the community. [0:32:09.5] D: They could that is a very good point. [0:32:11.3] PB: [inaudible] JKE, it could expose some of the statistics around those API end points. [0:32:16.2] J: I think the model right now is they just ping the community and say they are deprecating it and if a bunch of people scream, they don’t. I mean that is the only way to really know right now. [0:32:27.7] CC: The squeaky wheels get the grease kind of thing. [0:32:29.4] J: Yeah. [0:32:30.0] D: I mean that is how it turns out. [0:32:31.4] J: In regarding versioning, taking out of Kubernetes for a second, I also think this is one of the challenges with micro service architectures, right? Because now you have the ability to independently deploy a service outside of the whole monolith and if you happen to break something that cracks contractually you said you would and people just didn’t pay attention or you accidentally broke it not knowing, it can cause a lot of rift in a system. So versioning becomes a new concern because you are no longer deploying a massive system. You are deploying bits of it and perhaps versioning them and releasing them at different times. So again, it is that added complexity. [0:33:03.1] CC: And then you have this set of versions talk to this set of versions. Now you have a matrix and it is very complicated. [0:33:08.7] PB: Yeah and you do somewhat have a choice. You can’t have each service independently versioned or you could go with global versioning, where everything within V1 could talk to everything else than V1. But it's an interesting point around breakage because tools like GRPC kind of enforce you to where you cannot break the API, through just how the framework itself is built and that’s why you see GRPC in a lot of places where you see micro services just because it helps get the system stable. [0:33:33.1] D: Yeah and I will call back to that one point again, which I think is actually one of Josh’s points. If you are going to build multiple services and you are building an API between them then that means the communication path might be service A to service B and service B to service A. You are going to build this crazy mesh in which you have to define an API in each of these points to allow for that consumption or that interaction data. And one of the big takeaways for me in studying the cloud native ecosystem is that if you could define that API and that declarative state as a central model to all of your services then you can flip this model on its head instead of actually trying to define an API between in front of a service. You can make that service a consumer of a centralized API and now you have one contract to right and one contract to standby and all of those things that are going to do work are going to pull down from that central API. And do the work and put back into that central API the results, meaning that you are flipping this model on its head. You are no longer locking until service B can return the result to you. You are saying, “Service B here is a declarative state that I want you to accomplish and when you are done accomplishing it, let me know and I will come back for the results,” right? And you could let me know in an event stream. You can let me know by updating a status object that I am monitoring. There’s lots of different ways for you to let me know that service B is done doing the work but it really makes you think about the architecture of these distributed systems. It is really one of the big highlights for me personally when I look at the way that Kubernetes was architected. Because there are no private API’s. Everything talks to the API server. Everything that is doing work regardless of what data it’s manipulating but it is changing or modifying. It has to adhere to that central contract. [0:35:18.5] J: And that is an interesting point you brought up is that Kubernetes in a way is almost a monolith, in that everything passes through the API server, all the data leaves in this central place but you still have those distributed nature too, with the controllers. It is almost a mix of the patterns in some ways. [0:35:35.8] D: Yeah, I mean thanks for the discussion everybody that was a tremendous talk on contracts and API’s. I hope everybody got some real value out of it. And this is Duffy signing off. I will see you next week. [0:35:44.8] CC: This is great, thank you. [0:35:46.5] J: Cheers, thanks. [0:35:47.8] CC: Bye. [END OF INTERVIEW] [0:35:49.2] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter https://twitter.com/ThePodlets and on the https://thepodlets.io website where you will find transcripts and show notes. We’ll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.

In this episode, we dive into the exciting world of container orchestration in Kubernetes. We have all heard about container orchestration, but to truly understand this concept, we have to first understand what containers are and why they started! From definitions of containers and how they fit into the bigger cloud landscape, down to the nitty-gritty’s of managing and scaling container orchestration; this episode gives you strong foundation to better understand the functions and impacts of container orchestration today. Container orchestration in Kubernetes is so popular today but it can be difficult to how whether container orchestration is right for you. These are just some of the questions and topics we get into today, and if you’re looking for a solid base to begin your container orchestration process or enquiry – this is the episode for you! Note: our show changed name to The Podlets. Follow us: https://twitter.com/thepodlets Website: https://thepodlets.io Feeback: info@thepodlets.io https://github.com/vmware-tanzu/thepodlets/issues Hosts: Carlisia Campos Josh Rosso Nicholas Lane Key Points From This Episode: • Discover why container orchestration first came about.• Find out exactly what a container is and how it functions.• Using a container versus a virtual machine or process.• Managing container orchestration on a large scale.• Learn how container orchestration acts on information.• Managing actual state and expected state in container orchestration. • The key benefits of adopting container orchestration.• The key difference between container orchestrators.• A declarative way to approach resource limiting.• How to distinguish between the project and the product.• What is it that makes Kubernetes so popular today?• How to make an informed decision about using Kubernetes.• Find out when you should not be using container orchestration. Quotes: “The orchestration part is really just dictating behavior and state.” — @carlisia [0:05:43] “If you already use Kubernetes that would be like trying to plow the field with a nuclear bomb.” — Nicholas @apinick [0:42:10] Links Mentioned in Today’s Episode: Velero — https://github.com/vmware-tanzu/veleroYoutube Premium — https://www.youtube.com/premiumKubeCon China — https://01.org/events/2019/open-source-summit-china-kubecon/ cloudnativeconSteven Wong — https://twitter.com/cantbewongCloud Native Social Hour — https://www.youtube.com/watch?v=wxBxcdeMOYEKube Janitor — https://github.com/theMagicalKarp/kube-janitorDocker — https://www.docker.com/Mesosphere — https://d2iq.com/Red Hat — https://www.redhat.com/enKubernetes VS Docker Swarm — https://thenewstack.io/kubernetes-vs-docker-swarm-whats- the-difference/Kubernetes Slack Channel — http://slack.k8s.io/Kubelets Cloud Native Podcast — http://cloudnativepodcast.com/The Podlets on Twitter — https://twitter.com/thepodlets Transcript: EPISODE 02 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.3] NL: Hello and welcome back to The Podlets Podcast, Episode Two, Container Orchestration. My name is Nicholas and joining me today this week are Carlisia and Josh. [0:00:50.9] CC: Hello. [0:00:50.9] NL: Hello. [0:00:52.0] CC: Good to be here again. [0:00:53.7] NL: Yeah. How was your week, everyone? [0:00:55.6] CC: Very good, lots of work. [0:00:57.3] NL: Yeah, anything exciting happening in the world of Velero? [0:01:00.1] CC: Yes, we just got our alpha release for version 1.0 and we are looking for testers, yeah, we want testers. [0:01:08.2] NL: Awesome. [0:01:09.2] JR: I’ve been traveling a lot but it’s been good, we’re doing a lot of interesting work with some Kubernetes cluster running in an on premise datacenter which is something we see less and less, now that the cloud providers are kind of taking on their different offering. So it’s cool to hop back to kind of the bare metal and virtualization space and play around there. [0:01:27.6] NL: That’s cool. I’ve actually got a question for you guys, kind of irrespective of container orchestration, but how do you guys manage travel, right? How do you keep yourself entertained, how do you keep yourself happy while you’re traveling? For me, it’s a lot of podcasts which is great, now that I’m doing a podcast. [0:01:43.0] CC: Yeah, I do podcasts. I signed up for YouTube premium so I can download videos. I watch the movies on the plane, I have a kindle with lots of books. [0:01:56.2] NL: Yeah, that’s nice. [0:01:57.4] CC: Or I just sleep. [0:01:59.0] NL: I wish I could. [0:01:59.9] JR: Yeah, sleep is always the first goal, but I also signed up for YouTube Premium and the offline feature is fantastic so there’s so much good info on YouTube, you know? It’s great to like – go to the KubeCon Playlist and just choose offline and then you have all that time in the plane to really sift through talks and what not. It’s been really cool. [0:02:18.9] CC: Exactly. [0:02:19.8] NL: That’s a great idea. I’ve actually not used YouTube Premium for that. I’ve only ever used it for like meditation tracks, to use on the airplane. I spend some time in the plane kind of just in my own head a little bit kind of doing some internal self-care if you will. [0:02:34.0] CC: Nice. [0:02:34.7] NL: But that gets boring.EPISODE 02 [INTRODUCTION] [0:00:08.7] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concepts, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.3] NL: Hello and welcome back to The Podlets Podcast, Episode Two, Container Orchestration. My name is Nicholas and joining me today this week are Carlisia and Josh. [0:00:50.9] CC: Hello. [0:00:50.9] NL: Hello. [0:00:52.0] CC: Good to be here again. [0:00:53.7] NL: Yeah. How was your week, everyone? [0:00:55.6] CC: Very good, lots of work. [0:00:57.3] NL: Yeah, anything exciting happening in the world of Velero? [0:01:00.1] CC: Yes, we just got our alpha release for version 1.0 and we are looking for testers, yeah, we want testers. [0:01:08.2] NL: Awesome. [0:01:09.2] JR: I’ve been traveling a lot but it’s been good, we’re doing a lot of interesting work with some Kubernetes cluster running in an on premise datacenter which is something we see less and less, now that the cloud providers are kind of taking on their different offering. So it’s cool to hop back to kind of the bare metal and virtualization space and play around there. [0:01:27.6] NL: That’s cool. I’ve actually got a question for you guys, kind of irrespective of container orchestration, but how do you guys manage travel, right? How do you keep yourself entertained, how do you keep yourself happy while you’re traveling? For me, it’s a lot of podcasts which is great, now that I’m doing a podcast. [0:01:43.0] CC: Yeah, I do podcasts. I signed up for YouTube premium so I can download videos. I watch the movies on the plane, I have a kindle with lots of books. [0:01:56.2] NL: Yeah, that’s nice. [0:01:57.4] CC: Or I just sleep. [0:01:59.0] NL: I wish I could. [0:01:59.9] JR: Yeah, sleep is always the first goal, but I also signed up for YouTube Premium and the offline feature is fantastic so there’s so much good info on YouTube, you know? It’s great to like – go to the KubeCon Playlist and just choose offline and then you have all that time in the plane to really sift through talks and what not. It’s been really cool. [0:02:18.9] CC: Exactly. [0:02:19.8] NL: That’s a great idea. I’ve actually not used YouTube Premium for that. I’ve only ever used it for like meditation tracks, to use on the airplane. I spend some time in the plane kind of just in my own head a little bit kind of doing some internal self-care if you will. [0:02:34.0] CC: Nice. [0:02:34.7] NL: But that gets boring. [0:02:36.0] CC: I meditate too, it’s great. [0:02:38.2] NL: Yeah, it’s good. All right, anything interesting in the cloud native space that you guys have found in the last week? [0:02:43.6] CC: I have a talk that was accepted for KubeCon China. [0:02:47.4] NL: Awesome, congratulations. [0:02:49.6] JR: Congrats. [0:02:50.6] CC: Yeah, it’s a joint talk with Steven Wong also from Thea Moore. We’re going to talk about data recovery, data protection, recovery, migration in Velero. [0:03:03.9] NL: That’s great. He’s been coming to the Cloud Native Social Hour pretty regularly. That’s awesome to see some more cross interaction. [0:03:11.6] CC: Yeah, he is awesome, so knowledgeable. [0:03:14.0] NL: Great. And Josh? [0:03:15.5] JR: Very cool. I was actually looking this week since I’m in kind of the Kubernetes mindset, for something that can kind of add a TTL to any Kubernetes resource. So think of something like a service account in Kubernetes and I want to attach a TTL to it such that in four hours, it effectively got swept up and is no longer existent in the system. There’s some interesting ways that actually Kube ADM, one of the bootstrapping tools, does this. I was trying to kind of replicate that for their tokens, there’s a project by one of these Landau folks. Jacobs, I don’t know if that’s his last of first name, sorry in advance for butchering it, but he’s got a project called Kube Janitor that does effectively that. With annotations, you can put a TTL on them, your resources and then Kube Janitor will just come through and sweep that up. Which I thought a really cool idea. That was an interesting thing that I saw, it’s no new news, I think it’s been around for a while but it’s the first time that I had run into it. [0:04:07.6] NL: Nice. For me, our cohost Duffy, turned me on to a tool called Chaos Blade. Recently, I’ve been getting more and more into Chaos engineering and this is apparently an easy to use Chaos engineering toolkit. Something I’ve only just started looking at but I’m pretty excited. I’ll probably play around with that a bit more. [0:04:25.2] JR: Cool, awesome. [0:04:26.9] NL: Yeah, this week on the podcast, we are talking about container orchestration and kind of what that is, right? For me, container orchestration is the idea that you need your workloads to run somewhere but you don’t necessarily need to care where they’re running and the way that this has been done traditionally, prior to container orchestration, was like scheduling VM’s or making sure these processes run on certain computers, right? There’s a lot of automation around that like, when containers came around, we needed some way to make sure that they’re running and it also enabled us to not need to care so much about how things get started in all that. Everything was kind of packaged in a container I think. They need to just be some way to run them. That’s kind of where container orchestration came in, is that kind of your guys’ take on that as well? [0:05:18.3] CC: Yeah, basically, when we say we are orchestrating containers, we basically tell them how to behave, right? For example, I have this container here and I’m going to declare that if it fails, I want it to come back up in this container over there, if you fail just keep that state, don’t do anything and then I might say hey, I want two of you, three of you, I want to – the orchestration part is really just dictating behavior and state. [0:05:48.6] NL: Yeah, absolutely. [0:05:49.9] JR: Yeah. I think one interesting thing that came with the advent of containers is, we used to have this notion of you know, what server is my application going to land on or then eventually, you know, what virtual machine is my app eventually going to land on and we think kind of in this units of virtual machines and the paradigm shift a bit, at least in my experience has been now that you have the container unit and you can run many of those on one virtual machine, right? Your concern about orchestration is not just putting it on machine A and putting it on machine B but it’s kind of like packing multiple of this containers, perhaps on the same virtual machine or same host. The orchestration notion is beyond just the conventional system construct of a different host each time, it’s really interesting. [0:06:34.0] NL: Yeah, I think it might be important for us actually to take a step back. I realized I kind of jumped right into it, but we should probably settle what a container is, right? Before we can talk about how we can orchestrate them. A container is basically just a tar ball honestly. That is a packaged application with the instructions for it to run on any system that can accept that tar ball. Containers are broken down into a couple of Linux constructs, C groups and name space, so C groups four, making sure the process runs in its own dedicated memory and then or just like isolated memory. Then name spaces for things like network isolation. So that the network traffic that’s going on in the container doesn’t cross over to other processes. Very controlled process initiation based on these instruction. That’s kind of what a container is, a lot of people think that they’re like, kind of like a VM, I’ve heard that a few times where like, “Oh how do I deploy it?” What’s the VMDK for a container? It’s just a process that runs on a computer in a very controlled fashion, that’s literally it. [0:07:43.9] JR: Yeah, it’s kind of interesting to think like, at what point in which we kind of started using containers and seeing containers. I’d be curious for either two of you, Carlisia especially, what was your first exposure to the unit of a container and why were you starting to consider using a container versus just a virtual machine or a process? [0:08:03.0] CC: Frankly, I don’t remember. My first time seeing a container has been a long time but I don’t remember. But probably maybe trying to do some application like some toy application that – an example application. I remember that I was working on an application that we had the option to stuff it into a container as well, but I personally didn’t make the development. I wasn’t using it for development. My first usage of container really was about three years ago when I was working for CDN and a CDN as you might imagine has many different parts, so it has very low-level software running to higher level software, right? Really, sometimes, well, not sometimes, it has kernel level applications in systems, and it has API level system. For you to develop one part of it is it was really handy to be able to stuff our different systems into containers and have containers stuck to each other. We weren’t using the introduction. This was for development, but it was amazing, it was fantastic, we would have applications developed and go. Different systems that needed to talk to each other and we would have applications in C and I think that is to remember but it was amazing. Everything in containers and then we have a tool as well, they were sort of like Kubernetes, it wasn’t Kubernetes. It was developed in house. That orchestrated all of these things and you know, we simply failed, bringing back up and did a bunch of other things as well. I cannot explain the difference of working like that. It’s so much faster and so I could be a lot more autonomous, being able to run everything myself. I didn’t depend on having access to its server. I ran everything on my laptop, it was fantastic. [0:10:17.6] NL: Awesome. The first time I ran into a container was back when I was working for Red Hat, right when Open Shift Three CEO came out, that’s when Open Shift kind of moved from the in-house version of Open Shift to adopting Kubernetes. I had been working mostly in the virtualization like infrastructure world like doing a Red Hat enterprise virtualization manager, which is kind of like a Red Hat take on B Sphere, you know, kind of. I was very used to virtualization and spinning things up. There is some aspects of creating a VM and creating a container that were very similar. It took me awhile for my brain to click. Once I started using open chip to kind of click into like, “Oh this is how they’re different, right?” Whereas, if you’ve just started looking at it, “Well what’s kind of the difference?” They’re all just like, in my command line, they all just come up as like lists of units, right? This is a processing unit, that’s a processing unit right there. They’re kind of similar but once you start really getting into the use of it, it was so much different. I had heard like during this process of switching over to these two tools, I had heard of Docker and I was like, it’s something I’ll take a look at and finally, by shifting over to it, I finally was starting to – like oh this is what docker is, this is how we use these and then like, kind of digging into containers there. It was an interesting switch from an infrastructure standpoint to like, this is how people use containers and then that kind of actually started getting me into development. Now that I didn’t have to care about all this overhead of like where do I put my application, if I want my application around on my computer versus your computer, how do I make sure that the packages are the same bubble? Once there was that easy way to kind of say, I just want this run everywhere, no matter what, hopefully, that really just like, fascinated me and it kind of took off from there. Josh, what about you? [0:12:11.6] JR: Yeah, my experience wasn’t to dissimilar. What was interesting is the space I was working in was a lot of legacy Java applications, so we kind of came into containers probably a little bit later than what some of you all did. What was always interesting about it is, you know, we started to really see the value of containers just like Carlisia was saying, we started packaging these apps up and they ran the same in every environment and just really changed our workflow around. Initially, it was just like, let’s figure out a way to simply start these containers on different hosts, whether it be like Answerable or even someone going out a host and typing Docker Run, you know, that was how we got these processes to start. As the adoption of containers grew and more and more containers started to come to life in this company, the need for orchestration finally became obvious, right?I had heard about this project called Kubernetes, I’d heard a bit about Swarm, Mesos and it was always just like I don’t understand why you’d ever need something this complex, right? But eventually you hit this like inflection point where it just becomes insanely obvious, that your life is potentially going to be just chaos without something that can actually figure out, hey, you need to run this container, let me figure out where to put it and make sure that it starts. I thought that was like a really interesting progression. It used to be really hard also to navigate the options because there were a lot of options and there still are, there’s Swarm Kubernetes, Open Shift, Mesos, so on and so forth. [0:13:31.9] NL: Yeah, that’s actually a good point to what I’m talking about is that, container orchestration, it seems like we’re all kind of building up to the same point where when containers were kind of taking off, everyone started to see like this is great. But how do I do this at scale? Even like remotely at scale. A bunch of people started doing their own thing. So there was Kubernetes, which is the open source version aboard with some changes to make a more friendly for other people, there’s Docker, Docker Swarm and then Mesos, Rancher. But then, Carlisia, your team had their own orchestration, a lot of other companies have their own orchestration as well so it’s not just – you don’t need like this project to do or any of these projects to do container orchestration. You can do it on your own if you need to, right? For example, you could take a look at Uber, they aren’t using a project, they’ve rolled their own container orchestration at scale and I think that’s the same, that’s crazy to me but that’s awesome for them to have pulled that off, right? [0:14:29.4] CC: Yeah, absolutely. When I think of container orchestration, there is the management part and the scaling part because when you think about management for example, I might need a whole set of services to be up and running before I can run the set of services. The orchestration is going to manage that for me. Make sure that the services come up, they’re up and now this set gets kicked off. If I don’t need to scale, I still need to do this, right? There is usually some sort of dependency. Then in the scaling part which is also – I mean, it’s important for a lot of companies but it’s not important for a lot of companies smaller sized companies, right? [0:15:18.7] JR: Maybe we can talk a bit about what kind of information container orchestration works with to determine what it should do, if that sort of makes sense? Like what kinds of things are we telling these systems about and then what is it doing to act on that information? [0:15:38.0] NL: Yeah, please, go ahead and dive into that a bit more. [0:15:41.0] JR: Yeah, I guess it seems like the common approach that we run into, at least with Kubernetes and I think it’s true for a lot of these different systems, is the notion of reconciling state, right? We start of kind of with declarative definition if you will of what we want the world to look like and that could be some app running with some amount of replicas and you want it to have a certain amount of CPU and memory available. Then, these orchestrators usually can just take that declarative notion and sort of act on it, right? I know Nicholas, you’re really close to Kubernetes, would you want to speak to like how exactly it acts on those things like when you give it that declarative API object? What it’s going to do behind the scenes? [0:16:24.6] NL: Yeah, in Kubernetes, there’s a couple of different systems at play. This is something that I find really fascinating. There’s a lot of reconciliation loops in many different places. In Kubernetes when you first declare to Kubernetes that you want something to happen, you talk to the API server. The API server then modifies the etcd data store, right? The data store is just, simply ley value pair brain, it’s like the brain of your Kubernetes, right? Only the API server, as far as I’m aware and remembering off the top of my head, that’s the only thing that actually directly communicates to the etcd server. That might be incorrect but for the purpose of this – [0:17:04.3] CC: I think that’s correct. [0:17:06.8] NL: Okay, good. I was suddenly second guessing myself. The API directly can be considered sort of make the changes. Then the controller manager is in a reconciliation loop, saying like, here’s what I think the world looks like and if the world changes based on what etcd is saying, the controller manager maintains actual state and etcd controls expected states. This is where we want to be. If actual state and expected state are different, the controller manager reconciles that. Either it will delete something or add something to the cluster at large to make sure that that state exists. [0:17:47.6] CC: Based on what’s in the etcd database? [0:17:50.4] NL: Yes, exactly. It will – the controller manager, based on all the many controllers that are just themselves reconciliation nubs, if any of them are you know, different, it will then kick of something to the schedule which will then inform the various nodes in the cluster, what changes they need to do to reconcile state. Those changes occur, control managers sees that actual state matches expected state and everyone’s fat, dumb and happy. [0:18:17.3] CC: We actually didn’t talk much about other container orchestrators other than Kubernetes but I’m wondering because I’m not familiar with any others, but others come to mind, Docker, Swarm and Mesosphere, do they operate in the same way? [0:18:36.7] NL: Josh, I think you had some more experience than I did with at least, I believe it was Mesosphere? [0:18:41.8] JR: No, unfortunately not. [0:18:43.1] NL: I thought – okay, I thought that you had used in your previous life, you’d use at least one other? [0:18:49.0] JR: No, we did some small proof of concepts on Swarm but we never go very far along with it. [0:18:54.1] NL: Yeah, I actually, to be honest, I don’t really know much of the difference between like Rancho Lab, Mesosphere and Docker Swarm. I believe that they all act very similarly to Kubernetes but in slightly different way and this is something that I meant to take a look at before, talking about it but I just ran out of time, I’ll be honest. [0:19:12.8] CC: I guess we’re going to need Part Two to this episode. [0:19:15.7] NL: This is a big topic, we’ll definitely have to come back and kind of launch on this a bit more. I think they’re all orchestration and all these orchestrators work in the same function, right? Or the same fashion. There’s what you want to happen, what actually exists, how do we get that change to occur, right? [0:19:33.8] CC: Was that what you meant, Josh? [0:19:35.4] JR: Yeah, exactly. I think the one thing to add too is the systems are generally making like really informed decisions when trying to reconcile desired state. By really informed decisions, I mean, they’re obviously aware of a lot about the compute resources available to them. One big benefit that adopting container orchestration gives you is things like the scheduler are able to look into the system and understand, hey, based on resources I have available in this area, it would be smarter for me to start more containers over here versus over here, right? When you have these larger complex things and you’re trying to kind of think of all your resources as kind of like a sea of compute. The container orchestration is not only able to get you to a desired state but also to do it in a way that is, at least in most cases, as desirable as possible, right? As far as using resources effectively and a term that we often times throughout there, which is Vin Packing, right? The idea of ensuring that we can know the resources a container needs and pack them together really tightly, so that we’re utilizing the potential hardware or cloud resources that we’re paying for every month. A lot of times, the adoption of container orchestration is this really elegant way to move our workloads around but at the same time, it’s a way to really utilize the things we’re paying for and potentially cut costs over time as well. [0:20:57.5] CC: Yeah, this is one thing that I find fascinating with at least Kubernetes because I haven’t used the other orchestrators. We can boot up let’s say, four machines, and four instances of a machine and deploy Kubernetes on it and tell Kubernetes, “I want these many nodes, these many pods and have this container with apps obviously, or services running in the containers.” I don’t need to specify even where anything’s going to go. It just spreads the load and keeps managing and monitoring and managing what needs to go where to better utilize the instances. [0:21:46.2] NL: I think that’s actually an important distinction between the different container orchestrators that exist out there. If I recall correctly, I believe that Mesosphere has a mechanism that can kind of better load balance your containers that are running in the cluster. At least better than – it can make a kind of a more informed decision on like the state of the cluster and where it took place things than Kubernetes does and that might be one of the key differences between the two. That’s something that I hear a lot in the Kubernetes community. Someone’s like, “I noticed that all of my resources are kind of being put on to one computer and then the rest of them aren’t even being utilized at all, what’s up with that?” I think there’s something there that’s important to understand which is the Vin packing that Josh was talking about. Also, I pointed like that because on my screen, Josh is right next to me but that might not be the case so I might just look like I’m pointing out from the space.vIt’s important to know that from the capacity of at least in Kubernetes and like most of these orchestrators, if there are resources to be utilized, the orchestrator doesn’t care for the most part. Mesosphere has the ability to kind of load balance, as I said but as long as the resources that are available on one computer are the same as any other computer. If one of them is getting like super utilized and the other ones aren’t, it doesn’t really matter, it doesn’t affect the functionality of the cluster at all, right? One meg here and one meg there, essentially the same. [0:23:11.7] CC: What does the orchestrator do when let’s say I have four instances and I have what I have, I stuffed a bunch of consigners in there and I’m thinking, for this instance, this will give me plenty of memory but I have a leaky app and all of a sudden, my RAM blows up. What happens? [0:23:34.1] NL: This actually ties into why I look into an orchestration from a cloud native perspective. This is kind of where, container orchestration is cloud native. It takes into account the elastic nature of your resources. If you have this application that’s blowing up, either you can have limits to how many resources the application can utilize, or you can use auto scaling. In Kubernetes, we have something called Horizontal Plot Auto Scaling and some of the other tools, I’m sure they have the same, but the idea is like, as you’re using more and more resources in the pod, it is taking up this much memory. It then needs to create a new pod, right? Or a new container, right? So a new container needs to get orchestrated and then another one, another one, another one. Now, if you have a really aggressive application that is acting kind of maliciously that’s not great because it will take up all the resources in your cluster and that’s not good. But if you just have a very spiky application, it could grow with its needs and then come back down, and no one has to know about it essentially. Your orchestrator can make that happen for you. I think that is really cool. [0:24:40.9] CC: It is and what if I am reaching the limits of my resources. I mean there are only so many pods that can stuff in four instances or two instances. So what if I am reaching the limits of my resources? What happens then? How is an orchestrator going to help me? [0:24:58.6] JR: Yeah, the nice thing is we can – and most of these orchestrators set some type of parameter around potential CPU that we want to make available in memory, that we want to make available for the app and what is nice about this is at least speaking to Kubernetes, and I am sure it is similar for others, just using some of the underlying technologies that are existent in Linux like Nicholas has mentioned C groups. We have the ability when CPU gets too high to potentially throttle it and slow it down or at least limit the amount of CPU it can use in given cycles and with memory, if we start over committing, we now have the ability to potentially kill the application if it is starting to take up more memory than it actually should be allowed to take up. What’s interesting about Kubernetes and other orchestrators is their self-healing model is that sometimes when apps are doing really bad things, like leaking memory all over the place, you might not detect it right away, right? Because it is actually going to potentially limit or kill the app and self-heal it by bringing it back up. So it might seem like your app is still online and you don’t necessarily realize that under the hood, Kubernetes was actually restarting it and trying to continually bring it to a state of health, right? So you have a lot of abilities. It is like everything that Nicholas just said about reading how much information or resource the app is taking and potentially scaling based on that, or even setting like hard limits to say, “I want to throttle my app or even potentially kill my app if it starts to act badly and use up more than it should.” So it is a really cool kind of declarative way to approach resource limiting. [0:26:29.2] NL: And that is actually something that I don’t think a lot of people including myself work on that much is the throttling aspect, right? Most people are like, “Okay, well whatever. Just take up as many resources as we need.” That’s what it’s there for but maybe you shouldn’t always be doing that. Not every application needs to expand horizontally or vertically, if it’s safe or said. It could be that the application is acting poorly, and they need to be like, “No, you actually don’t need that many resources.” [0:26:56.4] CC: So let us say any or all of these things are happening, throttling and self-healing, how could I know? I mean I am asking this question, but I know the answer. I mean what tools do people actually use to be informed and notified of these events? [0:27:15.9] NL: So this is something I think we are going to get on another episode but just to come breach this into something that – I am also very excited. I am just a very excitable person really, I’m like, people say I’m just like a puppy and they’re not wrong. [0:27:27.1] CC: That’s why you’re here Nic. [0:27:29.0] NL: What? Me? Who said that? Is observation or observability. So monitoring, alerting, inflecting into your cluster to know what is happening, right? So you could, as Josh was saying like under the hood these things could be happening and the orchestrator is reconciling your cluster and your resource utilization for you and you might not know it but if you have observation and you have monitoring going on, you could see like, “Oh hey this pod is like restarting every 20 minutes.” Like it shouldn’t, it doesn’t need to restart every 20 minutes, like clearly the application is still running. So that is not a bad thing but maybe we should fix that, right? So you can be aware of what’s going on, right? [0:28:10.3] CC: And I know that there are tools that provide monitoring and observation but Kubernetes itself doesn’t provide that, right? Those are things that we hook into Kubernetes. [0:28:21.0] NL: Yes. Yeah that is correct because Kubernetes, and like any of these other orchestraters, are doing what they should be doing, which is being the best orchestrator they could. Having like that package now that you are getting into something that is more like a product and there is nothing wrong with products but that is not what these projects are here to be, right? [0:28:40.2] CC: How do you distinguish between the project and the product? [0:28:43.0] NL: Now that is interesting. Josh, you want to take this one? [0:28:45.7] CC: You opened the door. [0:28:47.0] JR: Yeah, I’ll start and then I actually think Nicholas, you might be the best one to speak to this with your background in Open Shift quite frankly, right? So it is kind of like these orchestrators are primitive in a way for how we eventually build a platform and that platform is a larger thing that includes potential monitoring, maybe plugins to continuous integration and continuous delivery. There’s a lot of groups or companies that have kind of that whole story or at least parts of that story, packaged up together, right? I mean we do it at the MWare with some of our enterprise offerings around TKS and then Open Shift, at least in my mind, it does that as well. Maybe Nicholas you can speak to that a little bit? [0:29:27.3] NL: Yeah, so from an Open Shift perspective at least when I was using it, it was trying to be everything you would need to monitor, or to not monitor, but to run a container orchestration system, right? So it has a Docker registry built it. It has monitoring built it. It has some rudimentary charge built in, Ingress, all of these things that don’t necessarily come with Kubernetes like the option Kubernetes. It has a solution around that. And I think that is the difference between like project or just an orchestrator and a product. A product is trying to solve a grander enterprise problem versus a project or in this case, an orchestrator, is going to solve one problem and that problem is how do I get these containers to run in a way that my customers – not my customers really, my users expect them to run. [0:30:18.3] CC: Yeah, fair enough. [0:30:19.5] NL: And what do you think on the topic, Carlisia? [0:30:22.0] CC: Oh it sounds an awful lot to me like a product is you get money for it and the projects you don’t. [0:30:28.1] NL: That is actually very – you’re right, honestly that is really the main distinction. One of them is money based. [0:30:34.6] CC: But your description, the descriptions you make for it are very valid – it is very valid because a project by itself may not have enough value for let’s say companies and bundling this project with that project and the other project, which ultimately you’re building the product with a purpose, right? You will have a purpose with that product to have a specific audience for their product set of users. So it is very distinct from taking one part of that product and calling it a product because maybe it is not enough to address and solve problems. [0:31:20.3] JR: Yeah, I think that is an important distinction. It is almost like what Nicholas and I were talking about was more about the distinction between what an orchestrator is and what a full platform would be, right? And I think to Carlisia’s point about how we plug in the monitoring and stuff is really important because just like we were talking about with the cloud native landscape in our last podcast, Kubernetes is just one piece of the overall puzzle. Kubernetes isn’t your whole platform start to finish, right? It is just the container orchestration portion and you have a lot to build and hook into that to make it a full platform that your company might be onboarding developer workloads onto it. It is just really one piece of that overall puzzle. [0:32:01.0] CC: That is beautifully put Josh. [0:32:02.9] NL: Yeah, very nicely put. So I’ve got a question for you guys. We’ve been beating around the bush as it were but to me, it seems apparent that in the world of container orchestration, Kubernetes has come out on top. That isn’t to say that it’s the end, right? There could be something that comes out that actually beats Kubernetes, right? But for now, it seems like everyone is looking at Kubernetes and I am curious why it is that you think that – you all might think that Kubernetes took the top space. [0:32:32.7] CC: I am scratching my chin. [0:32:35.1] NL: Scratching chin emoji. [0:32:38.3] JR: Exactly, one thing for sure is I just think Kubernetes did the community thing really, really well and not that it is all about community. It is obviously about technical choices and things of that nature but I think they did, not to say they’re perfect, but they did a really good job of being very inclusive and getting people to join this community and give feedback and the structure of the special interest groups where people get together and focus on various areas of Kubernetes, like scheduling or cluster life cycle and things like that. And it is interesting because the community just grew so quickly in my mind, that it just made this massive push into the market because there were so many humans behind it pushing it along. So I think at least among other things, community was one of the biggest. [0:33:24.0] CC: I can’t say that I was paying attention in monitoring that space so I don’t know. Of course, I can make guesses, what Josh just said sounds very plausible that he had Google behind it. I am sure it didn’t hurt. Not that we need to be fan boys and fan girls of Google but having a company like that sponsor and put resources behind the project gives a signal that “Okay, this is going to be here for a while.” Even though Google has a reputation of discontinuing things, but at the same time, I think that is significant. What else? Definitely the community. I didn’t follow the community from the beginning so only this last year and something, almost two years that I have been working with Velero, that I get to see how the community is and it’s amazing. It’s crazy, so organized. Yeah and it is not perfect, nothing is perfect but it’s incredible. The enthusiasm and the organization and the transparency, it is amazing. [0:34:36.6] NL: Yeah, absolutely and I agree with actually both of your points. It’s corporate sponsorship not just Google I mean, I’ll get to this in a second and the community as well and also some of the functionality. But it was both the corporate sponsorship of Google and Red Hat in the early days and not to tap my old you know, “Yeah we did it.” But it Red Hat had a big play into early Kubernetes as a supporter and so what that did is establish, “Hey, Kubernetes is at least enterprised.” An enterprise perspective project, right? It is not just, “Hey, this is some open source project. It may or may not work. If it doesn’t, you are on your own.” If you had a company like Google and Red Hat who are both endorsing this project, suddenly enterprises were more interested in taking it onboard, like it was more of a viable concept. [0:35:29.4] CC: I’m glad you are here Nic to correct me and make that addition. [0:35:34.4] NL: Oh well, yeah I was not correcting you at all. I think – [0:35:37.2] CC: No because I didn’t clue into the fact that – I mean I see Red Hat all over the place but I don’t know the dimension of involvement that they had from the beginning, because at the beginning, I was an outsider to all of this. [0:35:50.8] NL: Yeah, so for perspective, Open Shift 300, which is when I first started getting into it, is based on Kubernetes 1-2, which is pretty early. They were big like they put a lot of resources into the development of the community and for the development of the functionality that exists, right? The horizontal pod auto-scale that we still use today is due in the large part to the contributions of Red Hat, right? The engineering at Red Hat is responsible for that piece, among other things. And so with them at play, kind of getting their community and Google’s community coming together and then able to organize this community that I think is a big piece of what took this off or what allowed Kubernetes to take off. That is how grammar works. There is also some pieces of functionality that I think were novel to Kubernetes in the early days, things like Ingres. The way the Kubelet worked was actually kind of unique, like how low level the commands that are being issued by the Kubelet were pretty unique. And so it allowed for people to adopt it like the things that were happening from the Kubelet perspective like changes to your IP tables, running a container, changing the C groups and all of these things, those are all well known by people at the time and so there wasn’t anything like arcane happening. It was just, “Hey, this process just runs these commands and that is how it reconciles say, right? And so I think that that kind of functionality really got people to trust what was happening. And so, you know it’s like I think the trust and transparency are the big things that people keyed into. The trust comes from the enterprise sponsorship, and also the fact that what was happening from a rudimentary standpoint was pretty simple and so people could wrap their heads around it and then transparency was having this community. Everything happens in the open, everything is recorded and accessible by everyone, right? It wasn’t just like some behind the scenes things happened. [0:37:50.7] JR: Yeah and I think that piece is super important, like Nicholas and I, we came from Coreless or our lineage around like open source Kubernetes is not too dissimilar. We spend a lot of time working with customers in pure upstream open source Kubernetes and actually taking some of their issues and requirements to the community and then helping shape the direction of Kubernetes in micro kind of ways, but still important ways to that company. And I think companies seeing through the CNCF and seeing through just community leadership and involvement that the things that they care about aren’t just going through a single vendor to make a decision as to whether that thing should be included, but it is being part of a larger community discussion, breeds a lot of confidence in this project in the long term. I think at the end of the day, we started with there will be container orchestrator that many of us use. Or maybe there will be a couple, right? There is no question, we need to solve container orchestration as an overall problem and companies are at this point where they are still placing a bet on what they want to use and because of the community, because of the involvement, because of the ability to adopt the project to potential business requirements, I feel like more large and small medium organizations are willing to put their money on Kubernetes as a whole. And I don’t think they felt as confident finding some other projects, like Open Stack and then historical Masos perhaps. I am just projecting based on conversations I’ve had but that is why I think a lot of folks are really excited about the future for Kubernetes. [0:39:21.3] NL: Yeah that is an excellent point. [0:39:23.0] CC: Let’s stick with the theme of projecting in the future and we are going to have to wrap up soon otherwise it is going to be a two hour, I mean this could be a two-hour show. But let’s not make our audience go through that. We’ll have part two. What about, we talk all the time, and people who are in this area, we talk about all the time how everybody knows Kubernetes and da-da-da, but I want to challenge you two, do you think that everybody knows Kubernetes? Everybody knows the purpose of Kubernetes, everybody knows if they should be using Kubernetes or not, how are people able to make an informed decision if they should be using Kubernetes? Because I don’t think everybody knows Kubernetes. I think the majority of companies, in terms of volume, because smaller companies I would guess they outnumber the bigger companies and technologists – I think a lot of people are not clear on what this is. That’s why we are here but what do we tell them? We have to have an episode to discover that, now that I am thinking about it, but we could wrap this up with some seed ideas for that. [0:40:46.9] NL: So that is a great idea and something that I’ve been playing around with introducing myself, is when do you not use container orchestration, right? Just because container orchestration exists doesn’t mean necessarily – [0:40:58.0] CC: If you don’t have containers. [0:40:59.1] NL: Yeah, one, if you don’t have containers. That is another starter. [0:41:02.5] CC: It is a real legit thing to say because some people ask me, “Should I start with Kubernetes or containers?” that’s the level of education that we must provide. [0:41:15.5] NL: Yeah, absolutely and that is something we actually run into a lot in the field is when we are engaging with our customers, part of our job is to help containerize their applications if they are not already there. Trying to help them do that in a logical manner. But for instance, to give an example, my fiancé’s company uses Docker, but they don’t use Kubernetes or any kind of orchestration because they don’t need to. Like the amount of the resources that they are going to be using and the amount of and the type of work that they are doing, it doesn’t make sense to use an orchestrator. I have actually talked to some of the engineers about it because they were like, “Oh tell me about this Kubernetes thing” and I’m like, “This is what it is” blah-blah-blah. I finally came up with a metaphor where it’s like your company uses the containers as a shovel. If we brought it into like, let us say we’re ploughing a field, right? You’ve got a plow, if you already use Kubernetes that would be like trying to plow the field with a nuclear bomb. It is way more complicated than you need to do. Sure, you can clear a lot of land with a giant bomb but that is way more than you guys need, right? And I think that for me that’s the drawing line. It is like if the complexity makes sense for you to do like, if you’re trying to all of a sudden, establish a farm. Not to say that you should use a bomb to plow land but hey, if you need to clear a lot of land a bomb can work, right? That is a terrible metaphor, I am sorry. That went off the rails really fast. [0:42:41.3] CC: It wasn’t too bad. [0:42:43.8] NL: I actually think that. [0:42:46.7] JR: I think, one thing that I will say and this is coming from experience working with organizations is let’s assume that you have justified Kubernetes for yourself, and by the way, I super echo everything that Nicholas just said, you have to be really careful and determine do you actually need to take this thing on? Because it is so hard to do in a lot of ways, right? But let us assume you have taken it on. I think an interesting thing to have empathy about as often times infrastructure dev ops people is you might know Kubernetes really, really well. But that doesn’t mean your thousands and thousands of developers have any idea what Kubernetes is at all and that is a massive disconnect we see in organizations all the time where they are trying to onboard folks onto Kubernetes, and they haven’t fully abstracted Kubernetes away, which some companies do and that could be a really good pattern too. Like developers deploy their apps, they don’t even know Kubernetes are running them under the hood. That is a really neat pattern as well. But assuming they are just trying to bring developers onto Kubernetes, they don’t really have the same amount of empathy for them and they just think like, “This should be really easy. It is just a bunch of yemel files, you’ll figure it out,” but they totally forget about all of the complexities that they originally learned about. Like how does pod to pod networking work and things like that. I just think that to your question Carlisia, it is interesting because one massive group in a company can know a lot about Kubernetes and forget what it was like to learn how something like Kubernetes or container orchestration worked. I think a lot of that is bridging the gap and really having some amount of education to bring everyone up to speed, even in the same organization. [0:44:21.2] CC: I am dying to have an episode on just that alone because it is quite challenging. When you are faced with Kubernetes, I mean the very first thing is that there are terminologies that you haven’t seen before and they’re like, “How does that map to what I already know?” and then sometimes it doesn’t map. It is completely new so. [0:44:43.1] JR: Yeah and when the benefits aren’t super obvious to you, it is really hard to get bought in and be willing to invest your own time and energy into it, right? And we forget that it is just not super obvious why Kubernetes makes sense for a lot of folks. [0:44:56.6] NL: Yeah, absolutely. That is a good point that even I sometimes forget like when someone says, “Well, why would I want Kubernetes?” I’m like, “Why wouldn’t you want Kubernetes?” like duh, it works so well in my brain why don’t you get it? But it is good to take a step back out of yourself and you know, be empathetic to the people you’re talking about in the community. I think Carlisia, you mentioned that we should be wrapping this up pretty soon and I think I totally agree. Before we go, I want to say if you want to contribute to any container orchestration about Kubernetes in specific since that is the one we want to work with the most, we totally encourage you to start contributing to these projects. Like with Kubernetes, we have the Kubernetes-Kubernetes Repo that has a lot of information on how to start contributing. I believe that Mesosphere has their own repos and the information online available for them. And I don’t know, I am not sure if there is much in a way of Docker or Swarm anymore that you can contribute to. I am not sure, but for Kubernetes, we have the Kubernetes-Kubernetes Repo and the Kubernetes Slack channel K8S at slack.k8s.io. Please join us and start talking about your container orchestration journey. [0:46:08.6] CC: And Kates by the way is K8S and I am going to say that because at some events and some people were up in the stage and they’re like, “Kates this, Kates that” and I am sitting with someone in the back and I’m like, “Who’s Kate?” [0:46:24.3] NL: Or I have seen people who are like, “K-eight-S” is the acronym and what that means is that there is eight letters between K and S in Kubernetes. That is all that means. I have seen some people do K8 and it drives me up every wall. I actually start constructing walls and it continues to drive me up them. I am in an infinite regression of walls. [0:46:44.6] CC: All right everybody, thank you for listening. It’s great that you are here, and we are going to be back with more cloud native goodness. [0:46:53.8] NL: Yeah, absolutely. All right, cheers. [0:46:56.0] JR: Thanks. [0:46:56.8] CC: Goodbye. [END OF INTERVIEW] [0:46:58.7] ANNOUNCER: Thank you for listening to the Podlets Cloud Native Podcast. Find us on Twitter @thepodlets and on the Podlets.io website where you will find transcripts and show notes. We’ll be back next week. Stay tuned by subscribing. [END] [0:02:36.0] CC: I meditate too, it’s great. [0:02:38.2] NL: Yeah, it’s good. All right, anything interesting in the cloud native space that you guys have found in the last week? [0:02:43.6] CC: I have a talk that was accepted for KubeCon China. [0:02:47.4] NL: Awesome, congratulations. [0:02:49.6] JR: Congrats. [0:02:50.6] CC: Yeah, it’s a joint talk with Steven Wong also from Thea Moore. We’re going to talk about data recovery, data protection, recovery, migration in Velero. [0:03:03.9] NL: That’s great. He’s been coming to the Cloud Native Social Hour pretty regularly. That’s awesome to see some more cross interaction. [0:03:11.6] CC: Yeah, he is awesome, so knowledgeable. [0:03:14.0] NL: Great. And Josh? [0:03:15.5] JR: Very cool. I was actually looking this week since I’m in kind of the Kubernetes mindset, for something that can kind of add a TTL to any Kubernetes resource. So think of something like a service account in Kubernetes and I want to attach a TTL to it such that in four hours, it effectively got swept up and is no longer existent in the system. There’s some interesting ways that actually Kube ADM, one of the bootstrapping tools, does this. I was trying to kind of replicate that for their tokens, there’s a project by one of these Landau folks. Jacobs, I don’t know if that’s his last of first name, sorry in advance for butchering it, but he’s got a project called Kube Janitor that does effectively that. With annotations, you can put a TTL on them, your resources and then Kube Janitor will just come through and sweep that up. Which I thought a really cool idea. That was an interesting thing that I saw, it’s no new news, I think it’s been around for a while but it’s the first time that I had run into it. [0:04:07.6] NL: Nice. For me, our cohost Duffy, turned me on to a tool called Chaos Blade. Recently, I’ve been getting more and more into Chaos engineering and this is apparently an easy to use Chaos engineering toolkit. Something I’ve only just started looking at but I’m pretty excited. I’ll probably play around with that a bit more. [0:04:25.2] JR: Cool, awesome. [0:04:26.9] NL: Yeah, this week on the podcast, we are talking about container orchestration and kind of what that is, right? For me, container orchestration is the idea that you need your workloads to run somewhere but you don’t necessarily need to care where they’re running and the way that this has been done traditionally, prior to container orchestration, was like scheduling VM’s or making sure these processes run on certain computers, right? There’s a lot of automation around that like, when containers came around, we needed some way to make sure that they’re running and it also enabled us to not need to care so much about how things get started in all that. Everything was kind of packaged in a container I think. They need to just be some way to run them. That’s kind of where container orchestration came in, is that kind of your guys’ take on that as well? [0:05:18.3] CC: Yeah, basically, when we say we are orchestrating containers, we basically tell them how to behave, right? For example, I have this container here and I’m going to declare that if it fails, I want it to come back up in this container over there, if you fail just keep that state, don’t do anything and then I might say hey, I want two of you, three of you, I want to – the orchestration part is really just dictating behavior and state. [0:05:48.6] NL: Yeah, absolutely. [0:05:49.9] JR: Yeah. I think one interesting thing that came with the advent of containers is, we used to have this notion of you know, what server is my application going to land on or then eventually, you know, what virtual machine is my app eventually going to land on and we think kind of in this units of virtual machines and the paradigm shift a bit, at least in my experience has been now that you have the container unit and you can run many of those on one virtual machine, right? Your concern about orchestration is not just putting it on machine A and putting it on machine B but it’s kind of like packing multiple of this containers, perhaps on the same virtual machine or same host. The orchestration notion is beyond just the conventional system construct of a different host each time, it’s really interesting. [0:06:34.0] NL: Yeah, I think it might be important for us actually to take a step back. I realized I kind of jumped right into it, but we should probably settle what a container is, right? Before we can talk about how we can orchestrate them. A container is basically just a tar ball honestly. That is a packaged application with the instructions for it to run on any system that can accept that tar ball. Containers are broken down into a couple of Linux constructs, C groups and name space, so C groups four, making sure the process runs in its own dedicated memory and then or just like isolated memory. Then name spaces for things like network isolation. So that the network traffic that’s going on in the container doesn’t cross over to other processes. Very controlled process initiation based on these instruction. That’s kind of what a container is, a lot of people think that they’re like, kind of like a VM, I’ve heard that a few times where like, “Oh how do I deploy it?” What’s the VMDK for a container? It’s just a process that runs on a computer in a very controlled fashion, that’s literally it. [0:07:43.9] JR: Yeah, it’s kind of interesting to think like, at what point in which we kind of started using containers and seeing containers. I’d be curious for either two of you, Carlisia especially, what was your first exposure to the unit of a container and why were you starting to consider using a container versus just a virtual machine or a process? [0:08:03.0] CC: Frankly, I don’t remember. My first time seeing a container has been a long time but I don’t remember. But probably maybe trying to do some application like some toy application that – an example application. I remember that I was working on an application that we had the option to stuff it into a container as well, but I personally didn’t make the development. I wasn’t using it for development. My first usage of container really was about three years ago when I was working for CDN and a CDN as you might imagine has many different parts, so it has very low-level software running to higher level software, right? Really, sometimes, well, not sometimes, it has kernel level applications in systems, and it has API level system. For you to develop one part of it is it was really handy to be able to stuff our different systems into containers and have containers stuck to each other. We weren’t using the introduction. This was for development, but it was amazing, it was fantastic, we would have applications developed and go. Different systems that needed to talk to each other and we would have applications in C and I think that is to remember but it was amazing. Everything in containers and then we have a tool as well, they were sort of like Kubernetes, it wasn’t Kubernetes. It was developed in house. That orchestrated all of these things and you know, we simply failed, bringing back up and did a bunch of other things as well. I cannot explain the difference of working like that. It’s so much faster and so I could be a lot more autonomous, being able to run everything myself. I didn’t depend on having access to its server. I ran everything on my laptop, it was fantastic. [0:10:17.6] NL: Awesome. The first time I ran into a container was back when I was working for Red Hat, right when Open Shift Three CEO came out, that’s when Open Shift kind of moved from the in-house version of Open Shift to adopting Kubernetes. I had been working mostly in the virtualization like infrastructure world like doing a Red Hat enterprise virtualization manager, which is kind of like a Red Hat take on B Sphere, you know, kind of. I was very used to virtualization and spinning things up. There is some aspects of creating a VM and creating a container that were very similar. It took me awhile for my brain to click. Once I started using open chip to kind of click into like, “Oh this is how they’re different, right?” Whereas, if you’ve just started looking at it, “Well what’s kind of the difference?” They’re all just like, in my command line, they all just come up as like lists of units, right? This is a processing unit, that’s a processing unit right there. They’re kind of similar but once you start really getting into the use of it, it was so much different. I had heard like during this pr

Welcome to the first episode of The Podlets Podcast! On the show today we’re kicking it off with some introductions to who we all are, how we got involved in VMware and a bit about our career histories up to this point. We share our vision for this podcast and explain the unique angle from which we will approach our conversations, a way that will hopefully illuminate some of the concepts we discuss in a much greater way. We also dive into our various experiences with open source, share what some of our favorite projects have been and then we define what the term “cloud native” means to each of us individually. The contribution that the Cloud Native Computing Foundation (CNCF) is making in the industry is amazing, and we talk about how they advocate the programs they adopt and just generally impact the community. We are so excited to be on this podcast and to get feedback from you, so do follow us on Twitter and be sure to tune in for the next episode! Note: our show changed name to The Podlets. Follow us: https://twitter.com/thepodlets Hosts: Carlisia Campos Kris Nóva Josh Rosso Duffie Cooley Nicholas Lane Key Points from This Episode: An introduction to us, our career histories and how we got into the cloud native realm. Contributing to open source and everyone’s favorite project they have worked on. What the purpose of this podcast is and the unique angle we will approach topics from. The importance of understanding the “why” behind tools and concepts. How we are going to be interacting with our audience and create a feedback loop. Unpacking the term “cloud native” and what it means to each of us. Differentiating between the cloud native apps and cloud native infrastructure. The ability to interact with APIs as the heart of cloud natives. More about the Cloud Native Computing Foundation (CNCF) and their role in the industry. Some of the great things that happen when a project is donated to the CNCF. The code of conduct that you need to adopt to be part of the CNCF. And much more! Quotes: “If you tell me the how before I understand what that even is, I'm going to forget.” — @carlisia [0:12:54] “I firmly believe that you can't – that you don't understand a thing if you can't teach it.” — @mauilion [0:13:51] “When you're designing software and you start your main function to be built around the cloud, or to be built around what the cloud enables us to do in the services a cloud to offer you, that is when you start to look at cloud native engineering.” — @krisnova [0:16:57] Links Mentioned in Today’s Episode: Kubernetes — https://kubernetes.io/The Podlets on Twitter — https://twitter.com/thepodlets VMware — https://www.vmware.com/Nicholas Lane on LinkedIn — https://www.linkedin.com/in/nicholas-ross-laneRed Hat — https://www.redhat.com/CoreOS — https://coreos.com/Duffie Cooley on LinkedIn — https://www.linkedin.com/in/mauilionApache Mesos — http://mesos.apache.org/Kris Nova on LinkedIn — https://www.linkedin.com/in/kris-novaSolidFire — https://www.solidfire.com/NetApp — https://www.netapp.com/us/index.aspxMicrosoft Azure — https://azure.microsoft.com/Carlisia Campos on LinkedIn — https://www.linkedin.com/in/carlisiaFastly — https://www.fastly.com/FreeBSD — https://www.freebsd.org/OpenStack — https://www.openstack.org/Open vSwitch — https://www.openvswitch.org/Istio — https://istio.io/The Kublets on GitHub — https://github.com/heptio/thekubeletsCloud Native Infrastructure on Amazon — https://www.amazon.com/Cloud-Native-Infrastructure-Applications-Environment/dp/1491984309Cloud Native Computing Foundation — https://www.cncf.io/Terraform — https://www.terraform.io/KubeCon — https://www.cncf.io/community/kubecon-cloudnativecon-events/The Linux Foundation — https://www.linuxfoundation.org/Sysdig — https://sysdig.com/opensource/falco/OpenEBS — https://openebs.io/Aaron Crickenberger — https://twitter.com/spiffxp Transcript: [INTRODUCTION] [0:00:08.1] ANNOUNCER: Welcome to The Podlets Podcast, a weekly show that explores Cloud Native one buzzword at a time. Each week, experts in the field will discuss and contrast distributed systems concept, practices, tradeoffs and lessons learned to help you on your cloud native journey. This space moves fast and we shouldn’t reinvent the wheel. If you’re an engineer, operator or technically minded decision maker, this podcast is for you. [EPISODE] [0:00:41.3] KN: Welcome to the podcast. [0:00:42.5] NL: Hi. I’m Nicholas Lane. I’m a cloud native Architect. [0:00:45.0] CC: Who do you work for, Nicholas? [0:00:47.3] NL: I've worked for VMware, formerly of Heptio. [0:00:50.5] KN: I think we’re all VMware, formerly Heptio, aren’t we? [0:00:52.5] NL: Yes. [0:00:54.0] CC: That is correct. It just happened that way. Now Nick, why don’t you tell us how you got into this space? [0:01:02.4] NL: Okay. I originally got into the cloud native realm working for Red Hat as a consultant. At the time, I was doing OpenShift consultancy. Then my boss, Paul, Paul London, left Red Hat and I decided to follow him to CoreOS, where I met Duffie and Josh. We were on the field engineering team there and the sales engineering team. Then from there, I found myself at Heptio and now with VMware. Duffie, how about you? [0:01:30.3] DC: My name is Duffie Cooley. I'm also a cloud native architect at VMware, also recently Heptio and CoreOS. I've been working in technologies like cloud native for quite a few years now. I started my journey moving from virtual machines into containers with Mesos. I spent some time working on Mesos and actually worked with a team of really smart individuals to try and develop an API in front of that crazy Mesos thing. Then we realized, “Well, why are we doing this? There is one that's called Kubernetes. We should jump on that.” That's the direction in my time with containerization and cloud native stuff has taken. How about you Josh? [0:02:07.2] JR: Hey, I’m Josh. I similar to Duffie and Nicholas came from CoreOS and then to Heptio and then eventually VMware. Actually got my start in the middleware business oddly enough, where we worked on the Egregious Spaghetti Box, or the ESB as it’s formally known. I got to see over time how folks were doing a lot of these, I guess, more legacy monolithic applications and that sparked my interest into learning a bit about some of the cloud native things that were going on. At the time, CoreOS was at the forefront of that. It was a natural progression based on the interests and had a really good time working at Heptio with a lot of the folks that are on this call right now. Kris, you want to give us an intro? [0:02:48.4] KN: Sure. Hi, everyone. Kris Nova. I've been SRE DevOps infrastructure for about a decade now. I used to live in Boulder, Colorado. I came out of a couple startups there. I worked at SolidFire, we went to NetApp. I used to work on the Linux kernel there some. Then I was at Deis for a while when I first started contributing to Kubernetes. We got bought by Microsoft, left Microsoft, the Azure team. I was working on the original managed Kubernetes there. Left that team, joined up with Heptio, met all of these fabulous folks. I think, I wrote a book and I've been doing a lot of public speaking and some other junk along the way. Yeah. Hi. What about you, Carlisia? [0:03:28.2] CC: All right. I think it's really interesting that all the guys are lined up on one call and all the girls on another call. [0:03:34.1] NL: We should have probably broken it up more. [0:03:36.4] CC: I am a developer and have always been a developer. Before joining Heptio, I was working for Fastly, which is a CDN company. They’re doing – helping them build the latest generation of their TLS management system. At some point during my stay there, Kevin Stuart was posting on Twitter, joined Heptio. At this point, Heptio was about, I don't know, between six months in a year-old. I saw those tweets go by I’m like, “Yeah, that sounds interesting, but I'm happy where I am.” I have a very good friend, Kennedy actually. He saw those tweets and here he kept saying to me, “You should apply. You should apply, because they are great people. They did great things. Kubernetes is so hot.” I’m like, “I'm happy where I am.” Eventually, I contacted Kevin and he also said, “Yeah, that it would be a perfect match.” two months later decided to apply. The people are amazing. I did think that Kubernetes was really hard, but my decision-making went towards two things. The people are amazing and some people who were working there I already knew from previous opportunities. Some of the people that I knew – I mean, I love everyone. The only thing was that it was an opportunity for me to work with open source. I definitely could not pass that up. I could not be happier to have made that decision now with VMware acquiring Heptio, like everybody here I’m at VMware. Still happy. [0:05:19.7] KN: Has everybody here contributed to open source before? [0:05:22.9] NL: Yup, I have. [0:05:24.0] KN: What's everybody's favorite project they've worked on? [0:05:26.4] NL: That's an interesting question. From a business aspect, I really like Dex. Dex is an identity provider, or a middleware for identity provider. It provides an OIDC endpoint for multiple different identity providers. You can absorb them into Kubernetes. Since Kubernetes only has an OIDC – only accepts OIDC job tokens for authentication, that functionality that Dex provides is probably my favorite thing. Although, if I'm going to be truly honest, I think right now the thing that I'm the most excited about working on is my own project, which is starting to join like me, joining into my interest in doing Chaos engineering. What about you guys? What’s your favorite? [0:06:06.3] KN: I understood some of those words. NL: Those are things we'll touch on on different episodes. [0:06:12.0] KN: Yeah. I worked on FreeBSD for a while. That was my first welcome to open source. I mean, that was back in the olden days of IRC clients and writing C. I had a lot of fun, and still I'm really close with a lot of folks in the FreeBSD community, so that always has a special place in my heart, I think, just that was my first experience of like, “Oh, this is how you work on a team and you work collaboratively together and it's okay to fail and be open.” [0:06:39.5] NL: Nice. [0:06:40.2] KN: What about you, Josh? [0:06:41.2] JR: I worked on a project at CoreOS. Well, a project that's still out there called ALB Ingress controller. It was a way to bring the AWS ALBs, which are just layer 7 load balancers and take the Kubernetes API ingress, attach those two together so that the ALB could serve ingress. The reason that it was the most interesting, technology aside, is just it went from something that we started just myself and a colleague, and eventually gained community adoption. We had to go through the process of just being us two worrying about our concerns, to having to bring on a large community that had their own business requirements and needs, and having to say no at times and having to encourage individuals to contribute when they had ideas and issues, because we didn't have the bandwidth to solve all those problems. It was interesting not necessarily from a technical standpoint, but just to see what it actually means when something starts to gain traction. That was really cool. Yeah, how about you Duffie? [0:07:39.7] DC: I've worked on a number of projects, but I find that generally where I fit into the ecosystem is basically helping other people adopt open source technologies. I spent a quite a bit of my time working on OpenStack and I spent some time working on Open vSwitch and recently in Kubernetes. Generally speaking, I haven't found myself to be much of a contributor to of code to those projects per se, but more like my work is just enabling people to adopt those technologies because I understand the breadth of the project more than the detail of some particular aspect. Lately, I've been spending some time working more on the SIG Network and SIG-cluster-lifecycle stuff. Some of the projects that have really caught my interest are things like, Kind which is Kubernetes in Docker and working on KubeADM itself, just making sure that we don't miss anything obvious in the way that KubeADM is being used to manage the infrastructure again. [0:08:34.2] KN: What about you, Carlisia? [0:08:36.0] CC: I realize it's a mission what I'm working on at VMware. That is coincidentally the project – the open source project that is my favorite. I didn't have a lot of experience with open source, just minor contributions here and there before this project. I'm working with Valero. It's a disaster recovery tool for Kubernetes. Like I said, it's open source. We’re coming up to version 1 pretty soon. The other maintainers are amazing, super knowledgeable and very experienced, mature. I have such a joy to work with them. My favorites. [0:09:13.4] NL: That's awesome. [0:09:14.7] DC: Should we get into the concept of cloud native and start talking about what we each think of this thing? Seems like a pretty loaded topic. There are a lot of people who would think of cloud native as just a generic term, we should probably try and nail it down here. [0:09:27.9] KN: I'm excited for this one. [0:09:30.1] CC: Maybe we should talk about what this podcast show is going to be? [0:09:34.9] NL: Sure. Yeah. Totally. [0:09:37.9] CC: Since this is our first episode. [0:09:37.8] NL: Carlisia, why don't you tell us a little bit about the podcast? [0:09:40.4] CC: I will be glad to. The idea that we had was to have a show where we can discuss cloud native concepts. As opposed to talking about particular tools or particular project, we are going to aim to talk about the concepts themselves and approach it from the perspective of a distributed system idea, or issue, or concept, or a cloud native concept. From there, we can talk about what really is this problem, what people or companies have this problem? What usually are the solutions? What are the alternative ways to solve this problem? Then we can talk about tools that are out there that people can use. I don't think there is a show that approaches things from this angle. I'm really excited about bringing this to the community. [0:10:38.9] KN: It's almost like TGIK, but turned inside out, or flipped around where TGIK, we do tools first and we talk about what exactly is this tool and how do you use it, but I think this one, we're spinning that around and we're saying, “No, let's pick a broader idea and then let's explore all the different possibilities with this broader idea.” [0:10:59.2] CC: Yeah, I would say so. [0:11:01.0] JR: From the field standpoint, I think this is something we often times run into with people who are just getting started with larger projects, like Kubernetes perhaps, or anything really, where a lot of times they hear something like the word Istio come out, or some technology. Often times, the why behind it isn't really considered upfront, it's just this tool exists, it's being talked about, clearly we need to start looking at it. Really diving into the concepts and the why behind it, hopefully will bring some light to a lot of these things that we're all talking about day-to-day. [0:11:31.6] CC: Yeah. Really focusing on the what and the why. The how is secondary. That's what my vision of this show is. [0:11:41.7] KN: I like it. [0:11:43.0] NL: That's something that really excites me, because there are a lot of these concepts that I talk about in my day-to-day life, but some of them, I don't actually think that I understand pretty well. It's those words that you've heard a million times, so you know how to use them, but you don't actually know the definition of them. [0:11:57.1] CC: I'm super glad to hear you say that mister, because as a developer in many not a system – not having a sysadmin background. Of course, I did sysadmin things as a developer, but not it wasn't my day-to-day thing ever. When I started working with Kubernetes, a lot of things I didn't quite grasp and that's a super understatement. I noticed that I mean, I can ask questions. No problem. I will dig through and find out and learn. The problem is that in talking to experts, a lot of the time when people, I think, but let me talk about myself. A lot of time when I ask a question, the experts jump right to the how. What is this? “Oh, this is how you do it.” I don't know what this is. Back off a little bit, right? Back up. I don't know what this is. Why is this doing this? I don't know. If you tell me the how before I understand what that even is, I'm going to forget. That's what's going to happen. I mean, it’s great you're trying to make an effort and show me the how to do something. This is personal, the way I learn. I need to understand the how first. This is why I'm so excited about this show. It's going to be awesome. This is what we’re going to talk about. [0:13:19.2] DC: Yeah, I agree. This is definitely one of the things that excites me about this topic as well, is that I find my secret super power is troubleshooting. That means that I can actually understand what the expected relationships between things should do, right? Rather than trying to figure out. Without really digging into the actual problem of stuff and what and the how people were going, or the people who were developing the code were trying to actually solve it, or thought about it. It's hard to get to the point where you fully understand that that distributed system. I think this is a great place to start. The other thing I'll say is that I firmly believe that you can't – that you don't understand a thing if you can't teach it. This podcast for me is about that. Let's bring up all the questions and we should enable our audience to actually ask us questions somehow, and get to a place where we can get as many perspectives on a problem as we can, such that we can really dig into the detail of what the problem is before we ever talk about how to solve it. Good stuff. [0:14:18.4] CC: Yeah, absolutely. [0:14:19.8] KN: Speaking of a feedback loop from our audience and taking the problem first and then solution in second, how do we plan on interacting with our audience? Do we want to maybe start a GitHub repo, or what are we thinking? [0:14:34.2] NL: I think a GitHub repo makes a lot of sense. I also wouldn't mind doing some social media malarkey, maybe having a Twitter account that we run or something like that, where people can ask questions too. [0:14:46.5] CC: Yes. Yes to all of that. Yeah. Having an issue list that in a repo that people can just add comments, praises, thank you, questions, suggestions for concepts to talk about and say like, “Hey, I have no clue what this means. Can you all talk about it?” Yeah, we'll talk about it. Twitter. Yes. Interact with those on Twitter. I believe our Twitter handle is TheKubelets. [0:15:12.1] KN: Oh, we already have one. Nice. [0:15:12.4] NL: Yes. See, I'm learning something new already. [0:15:15.3] CC: We already have. I thought you were all were joking. We have the Kubernetes repo. We have a github repo called – [0:15:22.8] NL: Oh, perfect. [0:15:23.4] CC: heptio/thekubelets. [0:15:27.5] DC: The other thing I like that we do in TGIK is this HackMD thing. Although, I'm trying to figure out how we could really make that work for us in a show that's recorded every week like this one. I think, maybe what we could do is have it so that when people can listen to the recording, they could go to the HackMD document, put questions in or comments around things if they would like to hear more about, or maybe share their perspectives about these topics. Maybe in the following week, we could just go back and review what came in during that period of time, or during the next session. [0:15:57.7] KN: Yeah. Maybe we're merging the HackMD on the next recording. [0:16:01.8] DC: Yeah. [0:16:03.3] KN: Okay. I like it. [0:16:03.6] DC: Josh, you have any thoughts? Friendster, MySpace, anything like that? [0:16:07.2] JR: No. I think we could pass on MySpace for now, but everything else sounds great. [0:16:13.4] DC: Do we want to get into the meat of the episode? [0:16:15.3] KN: Yeah. [0:16:17.2] DC: Our true topic, what does cloud native mean to all of us? Kris, I'm interested to hear your thoughts on this. You might have written a book about this? [0:16:28.3] KN: I co-authored a book called Cloud Native Infrastructure, which it means a lot of things to a lot of people. It's one of those umbrella terms, like DevOps. It's up to you to interpret it. I think in the past couple of years of working in the cloud native space and working directly at the CNCF as a CNCF ambassador, Cloud Native Computing Foundation, they're the open source nonprofit folks behind this term cloud native. I think the best definition I've been able to come up with is when you're designing software and you start your main function to be built around the cloud, or to be built around what the cloud enables us to do in the services a cloud to offer you, that is when you start to look at cloud native engineering. I think all cloud native infrastructure is, it's designing software that manages and mutates infrastructure in that same way. I think the underlying theme here is we're no longer caddying configurations disk and doing system D restarts. Now we're just sending HTTPS API requests and getting messages back. Hopefully, if the cloud has done what we expect it to do, that broadcast some broader change. As software engineers, we can count on those guarantees to design our software around. I really think that you need to understand that it's starting with the main function first and completely engineering your app around these new ideas and these new paradigms and not necessarily a migration of a non-cloud native app. I mean, you technically could go through and do it. Sure, we've seen a lot of people do it, but I don't think that's technically cloud native. That's cloud alien. Yeah. I don't know. That's just my thought. [0:18:10.0] DC: Are you saying that cloud native approach is a greenfield approach generally? To be a cloud native application, you're going to take that into account in the DNA of your application? [0:18:20.8] KN: Right. That's exactly what I'm saying. [0:18:23.1] CC: It's interesting that never said – mentioned cloud alien, because that plays into the way I would describe the meaning of cloud native. I mean, what it is, I think Nova described it beautifully and it's a lot of – it really shows her know-how. For me, if I have to describe it, I will just parrot things that I have read, including her book. What it means to me, what it means really is I'm going to use a metaphor to explain what it means to me. Given my accent, I’m obviously not an American born, and so I'm a foreigner. Although, I do speak English pretty well, but I'm not native. English is not my native tongue. I speak English really well, but there are certain hiccups that I'm going to have every once in a while. There are things that I'm not going to know what to say, or it's going to take me a bit long to remember. I rarely run into not understanding it, something in English, but it happens sometimes. That's the same with the cloud native application. If it hasn't been built to run on cloud native platforms and systems, you can migrate an application to cognitive environment, but it's not going to fully utilize the environments, like a native app would. That's my take. [0:19:56.3] KN: Cloud immigrant. [0:19:57.9] CC: Cloud immigrant. Is Nick a cloud alien? [0:20:01.1] KN: Yeah. [0:20:02.8] CC: Are they cloud native alien, or cloud native aliens. Yeah. [0:20:07.1] JR: On that point, I'd be curious if you all feel there is a need to discern the notion of cloud native infrastructure, or platforms, then the notion of cloud native apps themselves. Where I'm going with this, it's funny hearing the Greenfield thing and what you said, Carlisia, with the immigration, if you will, notion. Oftentimes, you see these very cloud native platforms, things, the amount of Kubernetes, or even Mesos or whatever it might be. Then you see the applications themselves. Some people are using these platforms that are cloud native to be a forcing function, to make a lot of their legacy stuff adopt more cloud native principles, right? There’s this push and pull. It's like, “Do I make my app more cloud native? Do I make my infrastructure more cloud native? Do I do them both at the same time?” Be curious what your thoughts are on that, or if that resonates with you at all. [0:21:00.4] KN: I've got a response here, if I can jump in. Of course, Nova with opinions. Who would have thought? I think what I'm hearing here, Josh is as we're using these cloud native platforms, we're forcing the hand of our engineers. In a world where we may be used to just send this blind DNS request out so whatever, and we would be ignorant of where that was going, now in the cloud native world, we know there's the specific DNS implementation that we can count on. It has this feature set that we can guarantee our software around. I think it's a little bit of both and I think that there is definitely an art to understanding, yes, this is a good idea to do both applications and infrastructure. I think that's where you get into this what it needs to be a cloud native engineer. Just in the same traditional legacy infrastructure stack, there's going to be good engineering choices you can make and there's going to be bad ones and there's many different schools of thought over do I go minimalist? Do I go all in at once? What does that mean? I think we're seeing a lot of folks try a lot of different patterns here. I think there's pros and cons though. [0:22:03.9] CC: Do you want to talk about this pros and cons? Do you see patterns that are more successful for some kinds of company versus others? [0:22:11.1] KN: I mean, I think going back to the greenfield thing that we were talking about earlier, I think if you are lucky enough to build out a greenfield application, you're able to bake in greenfield infrastructure management instead as well. That's where you get these really interesting hybrid applications, just like Kubernetes, that span the course of infrastructure and application. If we were to go into Kubernetes and say, “I wanted to define a service of type load balancer,” it’s actually going to go and create a load balancer for you and actually mutate that underlying infrastructure. The only way we were able to get that power and get that paradigm is because on day one, we said we're going to do that as software engineers; taking the infrastructure where you were hidden behind the firewall, or hidden behind the load balancer in the past. The software would have no way to reason about it. They’re blind and greenfield really is going to make or break your ability to even you take the infrastructure layers. [0:23:04.3] NL: I think that's a good distinction to make, because something that I've been seeing in the field a lot is that the users will do cloud native practices, but they’ll use a tool to do the cloud native for them, right? They'll use something along the lines of HashiCorp’s Terraform to create the VMs and the load balancers for them. It's something I think that people forget about is that the application themselves can ask for these resources as well. Terraform is just using an API and your code can use an API to the same API, in fact. I think that's an important distinction. It forces the developer to think a little bit like a sysadmin sometimes. I think that's a good melding of the dev and operations into this new word. Regrettably, that word doesn't exist right now. [0:23:51.2] KN: That word can be cloud native. [0:23:53.3] DC: Cloud here to me breaks down into a different set of topics as well. I remember seeing a talk by Brandon Phillips a few years ago. In his talk, he was describing – he had some numbers up on the screen and he was talking about the fact that we were going to quickly become overwhelmed by the desire to continue to develop and put out more applications for our users. His point was that every day, there's another 10,000 new users of the Internet, new consumers that are showing up on the Internet, right? Globally, I think it's something to the tune of about 350,000 of the people in this room, right? People who understand infrastructure, people who understand how to interact with applications, or to build them, those sorts of things. There really aren't a lot of people who are in that space today, right? We're surrounded by them all the time, but they really just globally aren't that many. His point is that if we don't radically change the way that we think about the development as the deployment and the management of all of these applications that we're looking at today, we're going to quickly be overrun, right? There aren't going to be enough people on the planet to solve that problem without thinking about the problem in a fundamentally different way. For me, that's where the cloud native piece comes in. With that, comes a set of primitives, right? You need some way to automate, or to write software that will manage other software. You need the ability to manage the lifecycle of that software in a resilient way that can be managed. There are lots of platforms out there that thought about this problem, right? There are things like Mesos, there are things like Kubernetes. There's a number of different shots on goal here. There are lots of things that I've really tried to think about that problem in a fundamentally different way. I think of those primitives that being able to actually manage the lifecycle of software, being able to think about packaging that software in such a way that it can be truly portable, the idea that you have some API abstraction that brings again, that portability, such that you can make use of resources that may not be hosted on your infrastructure on your own personal infrastructure, but also in the cloud, like how do we actually make that API contract so complete that you can just take that application anywhere? These are all part of that cloud native definition in my opinion. [0:26:08.2] KN: This is so fascinating, because the human race totally already learned this lesson with the Linux kernel in the 90s, right? We had all these hardware manufacturers coming out and building all these different hardware components with different interfaces. Somebody said, “Hey, you know what? There's a lot of noise going on here. We should standardize these and build a contract.” That contract then implemented control loops, just like in Kubernetes and then Mesos. Poof, we have the Linux kernel now. We're just distributed Linux kernel version 2.0. The human race is here repeating itself all over again. [0:26:41.7] NL: Yeah. It seems like the blast radius of Linux kernel 2.0 is significantly higher than the Linux kernel itself. That made it sound like I was like, pooh-poohing what you're saying. It’s more like, we're learning the same lesson, but at a grander scale now. [0:27:00.5] KN: Yeah. I think that's a really elegant way of putting it. [0:27:03.6] DC: You do raise a good point. If you are embracing on a cloud native infrastructure, remember that little changes are big changes, right? Because you're thinking about managing the lifecycle of a thousand applications now, right? If you're going full-on cloud native, you're thinking about operating at scale, it's a byproduct of that. Little changes that you might be able to make to your laptop are now big changes that are going to affect a fleet of thousand machines, right? [0:27:30.0] KN: We see this in Kubernetes all the time, where a new version of Kubernetes comes out and something totally unexpected happens when it is ran at scale. Maybe it worked on 10 nodes, but when we need to fire up a thousand nodes, what happens then? [0:27:42.0] NL: Yeah, absolutely. That actually brings up something that to me, defines cloud native as well. A lot of my definition of cloud native follows in suit with Kris Nova's book, or Kris Nova, because your book was what introduced me to the phrase cloud native. It makes sense that your opinion informs my opinion, but something that I think that we were just starting to talk about a little bit is also the concept of stability. Cloud native applications and infrastructure means coding with instability in mind. It's not being guaranteed that your VM will live forever, because it's on somebody else's hardware, right? Their hardware could go down, and so what do you do? It has to move over really quickly, has to figure out, have the guarantees of its API and its endpoints are all going to be the same no matter what. All of these things have to exist for the code, or for your application to live in the cloud. That's something that I find to be very fascinating and that's something that really excites me, is not trying to make a barge, but rather trying to make a schooner when you're making an app. Something that can, instead of taking over the waves, can be buffeted by the waves and still continue. [0:28:55.6] KN: Yeah. It's a little more reactive. I think we see this in Kubernetes a lot. When I interviewed Joe a couple years ago, Joe Beda for the book to get a quote from him, he said, this magic phrase that has stuck with me over the past few years, which is “goal-seeking behavior.” If you look at a Kubernetes object, they all use this concept in Go called embedding. Every Kubernetes object has a status in the spec. All it is is it’s what's actually going on, versus what did I tell it, what do I want to go on. Then all we're doing is just like you said with your analogy, is we're just trying to be reactive to that and build to that. [0:29:31.1] JR: That's something I wonder if people don't think about a lot. They don't they think about the spec, but not the status part. I think the status part is as important, or more important maybe than the spec. [0:29:41.3] KN: It totally is. Because I mean, a status like, if you have one potentiality for status, your control loop is going to be relatively trivial. As you start understanding more of the problems that you could see and your code starts to mature and harden, those statuses get more complex and you get more edge cases and your code matures and your code hardens. Then we can take that and globally in these larger cloud native patterns. It's really cool. [0:30:06.6] NL: Yeah. Carlisia, you’re a developer who's now just getting into the cloud native ecosystem. What are your thoughts on developing with cloud native practices in mind? [0:30:17.7] CC: I’m not sure I can answer that. When I started developing for Kubernetes, I was like, “What is a pod?” What comes first? How does this all fit together? I joined the project [inaudible 00:30:24]. I don't have to think about that. It's basically moving the project along. I don't have to think what I have to do differently from the way I did things before. [0:30:45.1] DC: One thing that I think you probably ran into in working with the application is the management of state and how that relates to – where you actually end up coupling that state. Before in development, you might just assume that there is a database somewhere that you would have to interact with. That database is a way of actually pushing that state off of the code that you're actually going to work with. In this way, that you might think of being able to write multiple consumers of state, or multiple things that are going to mutate state and all share that same database. This is one of the patterns that comes up all the time when we start talking about cloud native architectures, is because we have to really be very careful about how we manage that state and mainly, because one of the other big benefits of it is the ability to horizontally scale things that are going to mutate, or consume state. [0:31:37.5] CC: My brain is in its infancy as it relates to Kubernetes. All that I see is APIs all the way down. It's just APIs all the way down. It’s not very different than as a developer for me, is not very much more complex than developing against the database that sits behind. Ask me again a year from now and I will have a more interesting answer. [0:32:08.7] KN: This is so fascinating, right? I remember a couple years ago when Kubernetes was first coming out and listening to some of the original “Elders of Kubernetes,” and even some of the stuff that we were working on at this time. One of the things that they said was we hope one day, somebody doesn't have to care about what's passed these APIs and gets to look at Kubernetes as APIs only. Then they hear that come from you authentically, it's like, “Hey, that's our success statement there. We nailed it.” It's really cool. [0:32:37.9] CC: Yeah. I don’t understood their patterns and I probably should be more cognizant about these patterns are, even if it's just to articulate them. To me, my day-to-day challenge is understanding the API, understanding what library call do I make to make this happen and how – which is just programming 101 almost. Not different from any other regular project. [0:33:10.1] JR: Yeah. That is something that's nice about programming with Kubernetes in mind, because a lot of times you can use the source code as documentation. I hate to say that particularly is a non-developer. I'm a sysadmin first getting into development and documentation is key in my mind. There's been more than a few times where I'm like, “How do I do this?” You can look in the source code for pretty much any application that you're using that's in Kubernetes, or around the Kubernetes ecosystem. The API for that application is there and it'll tell you what you need to do, right? It’s like, “Oh, this is how you format your config file. Got it.” [0:33:47.7] CC: At the same time, I don't want to minimize that knowing what the patterns are is very useful. I haven't had to do any design for Valero for our projects. Maybe if I had, I would have to be forced to look into that. I'm still getting to know the codebase and developing features, but no major design that I had to lead at least. I think with time, I will recognize those patterns and it will make it easier for me to understand what is happening. What I was saying is that not understanding the patterns that are behind the design of those APIs doesn't preclude me at all so call against it, against them. [0:34:30.0] KN: I feel this is the heart of cloud native. I think we totally nailed it. The heart of cloud native is in the APIs and your ability to interact with the APIs. That's what makes it programmable and that's what makes – gives you the interface for you and your software to interact with that. [0:34:45.1] DC: Yeah, I agree with that. API first. On the topic of cloud native, what about the Cloud Native Computing Foundation? What are our thoughts on the CNCF and what is the CNCF? Josh, you have any thoughts on that? [0:35:00.5] JR: Yeah. I haven't really been as close to the CNCF as I probably should, to be honest with you. One of the great things that the CNCF has put together are programs around getting projects into this, I don't know if you would call it vendor neutral type program. Maybe somebody can correct me on that. Effectively, there's a lot of different categories, like networking and storage and runtimes for containers and things of that nature. There's a really cool landscape that can show off a lot of these different technologies. A lot of the categories, I'm guessing we'll be talking about on this podcast too, right? Things like, what does it mean to do cloud native networking and so on and so forth? That's my purview of the CNCF. Of course, they put on KubeCon, which is the most important thing to me. I'm sure someone else on this call can talk deeper at an organization level what they do. [0:35:50.5] KN: I'm happy to jump in here. I've been working with them for I think three years now. I think first, it's important to know that they are a subsidiary of the Linux Foundation. The Linux Foundation is the original open source, nonprofit here, and then the CNCF is one of many, like Apache is another one that is underneath the broader Linux Foundation umbrella. I think the whole point of – or the CNCF is to be this neutral party that can help us as we start to grow and mature the ecosystem. Obviously, money is going to be involved here. Obviously, companies are going to be looking out for their best interest. It makes sense to have somebody managing the software that is outside, or external of these revenue-driven companies. That's where I think the CNCF comes into play. I think that's its main responsibility is. What happens when somebody from company A and somebody from Company B disagree with the direction that the software should go? The CNCF can come in and say, “Hey, you know what? Let's find a happy medium in here and let's find a solution that works for both folks and let's try to do this the best we can.” I think a lot of this came from lessons we learned the hard way with Linux. In a weird way, we did – we are in version 2.0, but we were able to take advantage of some of the priority here. [0:37:05.4] NL: Do you have any examples of a time in the CNCF jumped in and mediated between two companies? [0:37:11.6] KN: Yeah. I think the steering committee, the Kubernetes steering committee is a great example of this. It's a relatively new thing. It hasn't been around for a very long time. You look at the history of Kubernetes and we used to have this incubation process that has since been retired. We've tried a lot of solutions and the CNCF has been pretty instrumental and guiding the shape of how we're going to manage, solve governance for such a monolithic project. As Kubernetes grows, the problem space grows and more people get involved. We're having to come up with new ways of managing that. I think that's not necessarily a concrete example of two specific companies, but I think that's more of as people get involved, the things that used to work for us in the past are no longer working. The CNCF is able to recognize that and guide us out of that. [0:37:57.2] DC: Cool. That’s such a very good perspective on the CNCF that I didn't have before. Because like Josh, my perspective with CNCF was well, they put on that really cool party three times a year. [0:38:07.8] KN: I mean, they definitely are great at throwing parties. [0:38:12.6] NL: They are that. [0:38:14.1] CC: My perspective of the CNCF is from participating in the Kubernetes meetup here in San Diego. I’m trying to revive our meetup, which is really hard to do, but different topic. I know that they try to make it easier for people to find meetups, because they have on meetup.com, they have an organization. I don't know what the proper name is, but if you go there and you put your zip code, you'll find any meetup that's associated with them. My meetup here in San Diego is associated, can be easily found. They try to give a little bit of money for swags. We also give out ads for meetup. They offer help for finding speakers and they also have a speaker catalog on their website. They try to help in those ways, which I think is very helpful, very valuable. [0:39:14.9] DC: Yeah, I agree. I know about CNCF, mostly just from interacting with folks who are working on its behalf. Working at meeting a bunch of the people who are working on the Kubernetes project, on behalf of the CNCF, folks like Ihor and people like that, which are constantly amazingly with the amount of work that they do on behalf of the CNCF. I think it's been really good seeing what it means to provide governance over a project. I think that really highlights – that's really highlighted by the way that Kubernetes itself has managed. I think a lot of us on the call have probably worked with OpenStack and remember some of the crazy battles that went on between vendors around particular components in that stack. I've yet to actually really see that level of noise creep into the Kubernetes situation. I think squarely on the CNCF around managing governance, and also run the community for just making it accessible enough thing that people can plug into it, without actually having to get into a battle about taking ownership of CNI, for example. Nobody should own CNI. That should be its own project under its own governance. How you satisfy the needs for something like container networking should be a project that you develop as a company, and you can make the very best one that you could make it even attract as many customers to that as you want. Fundamentally, the way that your interface to that major project should be something that is abstracted in such a way that it isn't owned by any one company. There should be a contact in an API, that sort of thing. [0:40:58.1] KN: Yeah. I think the best analogy I ever heard was like, “We’re just building USB plugs.” [0:41:02.8] DC: That's actually really great. [0:41:05.7] JR: To that point Duffie, I think what's interesting is more and more companies are looking to the CNCF to determine what they're going to place their bets on from a technology perspective, right? Because they've been so burned historically from some project owned by one vendor and they don't really know where it's going to end up and so on and so forth. It's really become a very serious thing when people consider the technologies they're going to bet their business on. [0:41:32.0] DC: Yeah. When a project is absorbed into the CNCF, or donated to the CNCF, I guess. There are a number of projects that this has happened to. Obviously, if you see that iChart that is the CNCF landscape, there's just tons of things happening inside of there. It's a really interesting process, but I think that from my part, I remember recently seeing Sysdig Falco show up in that list and seeing them donate – seeing Sysdig donate Falco to the CNCF was probably one of the first times that I've actually have really tried to see what happens when that happens. I think that some of the neat stuff here that happens is that now this is an open source project. It's under the governance of the CNCF. It feels to me more an approachable project, right? I don't feel I have to deal with Sysdig directly to interact with Falco, or to contribute to it. It opens that ecosystem up around this idea, or the genesis of the idea that they built around Falco, which I think is really powerful. What do you all think of that? [0:42:29.8] KN: I think, to look at it from a different perspective, that's one example of when the CNCF helps a project liberate itself. There's plenty of other examples out there where the CNCF is an opt-in feature, that is only there if we need it. I think cluster API, which I'm sure we're going to talk about this in a later episode. I mean, just a quick overview is a lot of different vendors implementing the same API and making that composable and modular. I mean, nowhere along the way in the history of that project has the CNCF had to come and step in. We’ve been able to operate independently of that. I think because the CNCF is even there, we all are under this working agreement of we're going to take everybody's concerns into consideration and we're going to take everybody’s use case in some consideration, work together as an ecosystem. I think it's just even having that in place, whether or not you use it or not is a different story. [0:43:23.4] CC: Do you all know any project under the CNCF? [0:43:26.1] KN: I have one. [0:43:27.7] JR: Well, I've heard of this one. It's called Kubernetes. [0:43:30.1] CC: Is it called Kubernetes or Kubernetes? [0:43:32.8] JR: It’s called Kubernetes. [0:43:36.2] CC: Wow. That’s not what Duffie thinks. [0:43:38.3] DC: I don’t say it that way. No, it's been pretty fascinating seeing just the breadth of projects that are under there. In fact, I was just recently noticing that OpenEBS is up for joining the CNCF. There seems to be – it's fascinating that the things that are being generated through the CNCF and going through that life cycle as a project sometimes overlap with one another and it's very – it seems it's a delicate balance that the CNCF would have to play to keep from playing favorites. Because part of the charter of CNCF is to promote the project, right? I'm always curious to see and I'm fascinated to see how this plays out as we see projects that are normally competitive with one another under the auspice of the same organization, like a CNCF. How do they play this in such a way that they remain neutral, even it would – it seems like it would take a lot of intention. [0:44:39.9] KN: Yeah. Well, there's a difference between just being a CNCF project and being an official project, or a graduated project. There's different tiers. For instance, Kubicorn, a tool that I wrote, we just adopted the CNCF, like I think a code of conduct and there was another file I had to include in the repo and poof, were magically CNCF now. It's easy to get onboard. Once you're onboard, there's legal implications that come with that. There totally is this tier ladder stature that I'm not even super familiar with. That’s how officially CNCF you can be as your product grows and matures. [0:45:14.7] NL: What are some of the code of conduct that you have to do to be part of the CNCF? [0:45:20.8] KN: There's a repo on it. I can maybe find it and add it to the notes after this, but there's this whole tutorial that you can go through and it tells you everything you need to add and what the expectations are and what the implications are for everything. [0:45:33.5] NL: Awesome. [0:45:34.1] CC: Well, Valero is a CNCF project. We follow the what is it? The covenant? [0:45:41.2] KN: Yeah, I think that’s what it is. [0:45:43.0] CC: Yes. Which is the same that Kubernetes follows. I am not sure if there can be others that can be adopted, but this is definitely one. [0:45:53.9] NL: Yeah. According to Aaron Crickenberger, who was the Release Lead for Kubernetes 1.14, the CNCF code of conduct can be summarized as don't be a jerk. [0:46:06.6] KN: Yeah. I mean, there's more to it than that, but – [0:46:10.7] NL: That was him. [0:46:12.0] KN: Yeah. This is something that I remember seeing an open source my entire career, open source comes with this implication of you need to be well-rounded and polite and listen and be able to take others’ just thoughts and concerns into consideration. I think we just are getting used to working like that as an engineering industry. [0:46:32.6] NL: Agreed. Yeah. Which is a great point. It's something that I hadn't really thought of. The idea of development back in the day, it seems like before, there was such a thing as the CNCF are cloud native. It seemed that things were combative, or people were just trying to push their agenda as much as possible. Bully their way through. That doesn't seem that happens as much anymore. Do you guys have any thoughts on that? [0:46:58.9] DC: I think what you're highlighting is more the open source piece than the cloud native piece, which I – because I think that when you're working – open source, I think has been described a few times as a force multiplier for software development and software adoption. I think of these things are very true. If you look at a lot of the big successful closed source projects, they have – the way that people in this room and maybe people listening to this podcast might perceive them, it's definitely just fundamentally differently than some open source project. Mainly, because it feels it's more of a community-driven thing and it also feels you're not in a place where you're beholden to a set of developers that you don't know that are not interested in your best, and in what's best for you, or your organization to achieve whatever they set out to do. With open source, you can be a part of the voice of that project, right? You can jump in and say, “You know, it would really be great if this thing has this feature, or I really like how you would do this thing.” It really feels a lot more interactive and inclusive. [0:48:03.6] KN: I think that that is a natural segue to this idea of we build everything behind the scenes and then hey, it's this new open source project, that everything is done. I don't really think that's open source. We see some of these open source projects out there. If you go look at the git commit history, it's all everybody from the same company, or the same organization. To me, that's saying that while granted the source code might be technically open source, the actual act of engineering and architecting the software is not done as a group with multiple buyers into it. [0:48:37.5] NL: Yeah, that's a great point. [0:48:39.5] DC: Yeah. One of the things I really appreciate about Heptio actually is that all of the projects that we developed there were – that the developer chat for that was all kept in some neutral space, like the Kubernetes Slack, which I thought was really powerful. Because it means that not only is it open source and you can contribute code to a project, but if you want to talk to people who are also being paid to develop that project, you can just go to the channel and talk to them, right? It's more than open source. It's open community. I thought that was really great. [0:49:08.1] KN: Yeah. That's a really great way of putting it. [0:49:10.1] CC: With that said though, I hate to be a party pooper, but I think we need to say goodbye. [0:49:16.9] KN: Yeah. I think we should wrap it up. [0:49:18.5] JR: Yeah. [0:49:19.0] CC: I would like to re-emphasize that you can go to the issues list and add requests for what you want us to talk about. [0:49:29.1] DC: We should also probably link our HackMD from there, so that if you want to comment on something that we talked about during this episode, feel free to leave comments in it and we'll try to revisit those comments maybe in our next episode. [0:49:38.9] CC: Exactly. That's a good point. We will drop a link the HackMD page on the corresponding issue. There is going to be an issue for each episode, so just look for that. [0:49:51.8] KN: Awesome. Well, thanks for joining everyone. [0:49:54.1] NL: All right. Thank you. [0:49:54.6] CC: Thank you. I'm really glad to be here. [0:49:56.7] DC: Hope you enjoyed the episode and I look forward to a bunch more. [END OF EPISODE] [0:50:00.3] ANNOUNCER: Thank you for listening to The Podlets Cloud Native Podcast. Find us on Twitter https://twitter.com/ThePodlets and on the https://thepodlets.io, where you'll find transcripts and show notes. We'll be back next week. Stay tuned by subscribing. [END]See omnystudio.com/listener for privacy information.