Podcasts about Hippogriff

Make your summer enjoyable and delicious by signing up for HelloFresh at http://hellofresh.com/jvsb10fm and get TEN FREE MEALS with a FREE ITEM FOR LIFE! Show your colors AND your fandom! Grab the SCB Pride Month shirt now!

Help MuggleCast grow! Become a MuggleCast Member and get great benefits like Bonus MuggleCast! On this week's edition we discuss the best use of 8 hours to tell the story of the first season of the upcoming TV show! Patreon.com/MuggleCast  Grab official merch! MuggleCastMerch.com Pick up overstock merch from years past, including our 19th Anniversary Shirt! MuggleMillennial.Etsy.com On this week's episode, we're trying our best to keep morale high as Harry is having a complete meltdown! Join Andrew, Eric and Micah as they debate Dumbledore's hands-off approach and navigate the highs and lows of Christmas on The Closed Ward. News: Warner Bros. finally confirms six actors for the new Harry Potter TV series Chapter by Chapter continues with Harry Potter and the Order of the Phoenix, Chapter 23: Christmas on The Closed Ward. MuggleCast last discussed this chapter on Episode 460! Harry feels violated and is completely spiraling! Who is going to save him? Why should Harry listen to or even trust Phineas Nigellus Black? The Power of Connection: Hermione & Ginny's Breakthrough Do we feel Harry is even capable of asking for help given his upbringing? Buckbeak: The Emotional Support Hippogriff The Lighter Side of Remus Lupin Santa Sirius has forgotten all about his Christmas elf Kreacher Should we feel sorry for Lockhart? The Longbottoms & The Harsh Realities of War Bubblegum Wrapper Theories Galore! Most Valuable Gift: What was the best Christmas gift someone received in this chapter? Lynx Line: What is something someone gave you that has no (or little) monetary value, but which you hold absolutely dear and would be devastated to part with? What's the story associated with it? Quizzitch: What actor/comedian once eschewed autographs in favor handing out cards reading “This certifies that you have had a personal encounter with me and that you found me warm, polite, intelligent, and funny”? On Bonus MuggleCast, the hosts try their hands (literally) at a game of Skribbl! Learn more about your ad choices. Visit megaphone.fm/adchoices

HIPPOGRIFFHello, amazing artists!

Welcome to Fanatical Fics and Where to Find Them! Fic:https://www.fanfiction.net/s/3278890/1/Harmony-Hippogriff-at-HogwartsRec:https://archiveofourown.org/works/93763Please leave a review! We'll take kind and constructive… but no flames! You can also help us by suggesting this podcast to every person you've ever met!To see a full list of episodes and recommendations, grab some podcast swag, get that sweet patreon link, or to submit a story you've read visit our website at https://www.fanaticalfics.com/As always, feel free to shoot us an email at fanaticalfics@gmail.com with any questions, comments, or suggestions.You can also find us on social media!Instagram: https://www.instagram.com/fanaticalfics Hosted on Acast. See acast.com/privacy for more information.

Welcome to Fanatical Fics and Where to Find Them! Follow Professional Talkers to get updates on the upcoming show: https://www.patreon.com/professionaltalkersFics:https://www.fanfiction.net/s/3278890/1/Harmony-Hippogriff-at-HogwartsPlease leave a review! We'll take kind and constructive… but no flames! You can also help us by suggesting this podcast to every person you've ever met!To see a full list of episodes and recommendations, grab some podcast swag, get that sweet patreon link, or to submit a story you've read visit our website at https://www.fanaticalfics.com/As always, feel free to shoot us an email at fanaticalfics@gmail.com with any questions, comments, or suggestions.You can also find us on social media!Instagram: https://www.instagram.com/fanaticalfics Hosted on Acast. See acast.com/privacy for more information.

Join Harry, Ron and Hermione on a thrilling adventure. Oh no Ron has been dragged into the shrieking shack by a black dog! Sirius Black appears in the shrieking shack and wants to kill not Harry but Rons rat scabbers?! Lupin isn't a human and scabbers isn't a rat! Most surprisingly Sirius Black the mass murderer isn't guilty either! How many more surprises will be thrown at you? How will Buck beak the Hippogriff be revived? And the most important question, will you tap on this new podcast to find out? Let's find out how the third book in the Harry Potter series (Harry Potter and the Prisoner of Azkaban) will end. The thrilling four podcast book comes to an exiting, epic end!

In episode 473 the "survival experts" have a look at not one but TWO films this week. The directors cut of Rebel Moon. Chris watched the much longer Part One Chalice of Blood and Jack watched the slightly longer Part Two Rebel Moon Curse of Forgiveness. E-mail your survival suggestions to us at wecouldsurvivethat@gmail.com or Twitter @WeCouldSurvive or find older episode on Youtube at: https://www.youtube.com/channel/UCXAa8-wNqv1G14ts_DHenkg/feed

Chapter 14 - Snape's GrudgeQ1 - What should the procedure be after Sirius Black has been caught sneaking into Hogwarts again?Q2 - Why hadn't Sirius killed Ron and then gone to Harry?“But I gotta tell yeh, I thought you two'd value yer friend more'n broomsticks or rats. Tha's all.”Ron had to hold onto the fence to keep himself standing, he was laughing so hard.Q3 - What is the hardest you have ever laughed?“So, everyone from the Minister of Magic downwards has been trying to keep famous Harry Potter safe from Sirius Black. But famous Harry Potter is a law unto himself. Let the ordinary people worry about his safety! Famous Harry Potter goes where he wants to, with no thought for the consequences.”Q4 - Does Snape have a point about Harry here?“I told you to shut up about my dad!” Harry yelled. “I know the truth, all right? He saved your life! Dumbledore told me! You wouldn't even be here if it weren't for my dad!” Snapes sallow skin had gone the colour of sour milk. “And did the Headmaster tell you the circumstances in which your father saved my life?” he whispered. “Or did he consider the details too unpleasant for precious Potter's delicate ears?”Q5 - What do you think of Snape and James Potter?Q6 - Who do you think made the Marauders Map?Your parents gave their lives to keep you alive, Harry. A poor way to repay them — by gambling their sacrifice for a bag of magic tricks.”Chapter 15 - The Quidditch Final“There'll be an appeal, though, there always is. Only I can't see any hope…nothing will have changed.” “Yeah it will,” said Ron fiercely. “You won't have to do all the work alone this time, Hermione. I'll help.” Hermione flung her arms around Ron's neck and broke down completely. Ron, looking quite terrified, patted her very awkwardly on the top of the head. Finally, Hermione drew away.Q1 - Do you think Ron and Hermione or Harry and Hermione are a better couple?Harry and Ron both made furious moves toward Malfoy, but Hermione got there first — SMACK! She had slapped Malfoy around the face with all the strength she could muster.Q2 - How do we like aggressive Hermione?“Fine!” she repeated swinging the bag over her shoulder and almost knocking Ron off his chair, “I give up! I'm leaving!” And to the whole class's amazement, Hermione strode over to the trapdoor, kicked it open, and climbed down the ladder out of sight.PREDICTION - ONE WILL LEAVE THE NUMBER FOREVER!Q3 - What do you think is the deal with crookshanks and the grim?The Gryffindor crowd were screaming themselves hoarse — Gryffindor were sixty points in the lead, and if Harry caught the snitch now, the cup was theirs.Q4 - How bad of a seeker is Malfoy?Q5 - Do you guys like quidditch now?Chapter 16 - Professor Trelawney's PredictionEven Fred and George Weasley had been spotted working.Q1 - What do you think differentiates between a charm and a spell?Their second from last exam, on Thursday morning, was Defense Against the Dark Arts. Professor Lupin had compiled the most unusual exam any of them had ever taken; a sort of obstacle course outside in the sun, where they had to wade across a deep padding pool containing a Grindylow, cross a series of potholes full of Red Caps, squish their way across a patch of marsh, ignoring the misleading directions from a Hinkypunk, then climb into an old trunk and battle with a new Boggart.Q2 - Which of professor Lupins final exam obstacles would trip you up?“Hermione!” said Lupin, startled. “What's the matter?” “P-P-Professor McGonagall!” Hermione gasped, pointing into the trunk. “Sh-she said I'd failed everything!”Q3 - If you had to make up a prediction, what would you say?Her eyes started to roll. Harry stood there in a panic. She looked as though she was about to have some sort of seizure. He hesitated, thinking of running to the hospital wing - and then Professor Trelawney spoke again, in the same harsh voice, quite unlike her own: “The Dark Lord lies alone and friendless, abandoned by his followers. His servant has been chained these twelve years. Tonight, before midnight, the servant will break free and set out to rejoin his master. The Dark Lord will rise again with his servant's aid, greater and more terrible than ever before. Tonight ... before midnight .. the servant will set out ... to rejoin … his master.”Q4 - What are your thoughts about this prediction?Q5 - Is Trelawney a true seer?Hermione didn't wait for the rest of his sentence; she strode across the room, pushed the fat lady's portrait open and vanished from sight.Q6 - Has Hermione changed at all from this book to the previous two?“Ron, I - I don't believe it - it's Scabbers!”There was a jumble of indistinct male voices, a silence and then, without warning, the unmistakable swish and thud of an ax.Chapter 17 - Cat, Rat, and DogBut before they could cover themselves again, before they could even catch their breath, they heard the soft pounding of gigantic paws. Something was bounding towards them out of the dark -an enormous, pale-eyed, jet-black dog. Harry reached for his wand, but too late the dog had made an enormous leap and its front paws hit him on the chest. He keeled over backwards in a whirl of hair; he felt its hot breath, saw inch-long teeth.Q1 - What is the worst injury you've had?Q2 -  If your best friend was in the most haunted place in the world, would you go rescue them?Harry and Hermione dashed across to him. 'Ron- are you OK?' Where's the dog?' "Not a dog; Ron moaned. His teeth were gritted with pain. 'Harry, it's a trap -' What 'He's the dog…. he's an Animagus. Ron was staring over Harry's shoulder. Harry wheeled around. With a snap, the man in the shadows closed the door behind them. A mass of filthy, matted hair hung to his elbows. If eyes hadn't been shining out of the deep, dark sockets, he might have been a corpse. The waxy skin was stretched so tightly over the bones of his face, it looked like a skull. His yellow teeth were bared in a grin. It was Sirius black.Q3 - Did you suspect the dog was Sirius black?“If you want to kill Harry, you'll have to kill us, too!” he said fiercely, though the effort of standing up had drained him of still more color, and he swayed slightly as he spoke.Q3 - How has Ron's character progressed over these books?Harry raised the wand. Now was the moment to do it. Now was the moment to avenge his mother and father. He was going to kill Black. He had to kill Black. This was his chance..Q4 - Is Harry's anger justified here? Q5 - Why is Black being so unclear here?Then lupin spoke, in an odd voice, a voice that shook with some suppressed emotion. “Where is he, Sirius?” Harry looked quickly at Lupin. He didn't understand what Lupin meant. Who was Lupin talking about? He turned to look at Black again. Black's face was quite expressionless. For a few seconds, he didn't move at all. Then, very slowly, he raised his empty hand, and pointed straight at Ron. Mystified, Harry glanced around at Ron, who looked bewildered. “But then .. Lupin muttered, staring at Black so intently it seemed he was trying to read his mind. Why hasn't he shown himself before now? Unless - Lupin's eyes suddenly widened, as though he was seeing something beyond Black, something none of the rest could see, unless he was the one ... unless you switched. without telling me?” Very slowly, his sunken gaze never leaving Lupin's face. Black nodded.Q6 - Do you understand what they are talking about here?Her voice wavering out of control, and all the time you've been his friend!' You're wrong,' said Lupin. I haven't been Sirius's friend for twelve years, but I am now ... let me explain. 'NO!' Hermione screamed, Harry, don't trust him, he's been helping Black get into the castle, he wants you dead too - he's a werewolf!” What's my rat got to do with anything?' "That's not a rat.' croaked Sirius Black suddenly. 'What d'you mean of course he's a rat - No, he's not;' said Lupin quietly. 'He's a wizard.' An Animagus, said Black, 'by the name of Peter Pettigrew.Chapter 18 - Moony, Wormtail, Padfoot, and Prongs“They've — got — a — right — to — know — everything.” Lupin panted, still trying to restrain Black. “Ron's kept him as a pet! There are parts of it even I don't understand! And Harry — you owe Harry the truth, Sirius!” Black stopped struggling, though his hollowed eyes were still fixed on Scabbers, who was clamped tightly under Ron's bitten, scratched and bleeding hands. “All right, then,” said Black, without taking his eyes off the rat. “Tell them whatever you like. But make it quick, Remus. I want to commit the murder I was imprisoned for…”Q1 - Who is right here? Lupin for waiting, or Black for wanting to kill Peter first?Q2 - Why didn't Scabbers try to escape earlier on?“Because ... because people would know if Peter Pettigrew had been an Animagus. We did Animagi in class with Professor McGonagall. And I looked them up when I did my homework - the Ministry keeps tabs on witches and wizards who can become animals; there's a register showing what animal they become, and their markings and things…and I went and looked Professor Mcgonagall up on the register, and there have only been seven Animagi this century, and pettigrews name wasn't on the list.”Q3 - I ask again, what would you be if you could be an Animagus?“It seemed impossible that I would be able to come to Hogwarts. Other parents weren't likely to want their children exposed to me. But then Dumbledore became Headmaster and he was sympathetic.”REDDIT - alphaWLFgang1 - So if we were looking at it from that perspective then I'd be a Nebelung Cat (according to Wizarding World) But, I mean if I had to choose I'd want to be something that could fly. So like an Eagle or maybe a Hippogriff would be sick too if that's even possible? I'd want to fly or swim! So maybe a shark? I'd want to be an animal that can either fly/swim under water. I'd want to fly so I can travel faster places and swim because I love the water.Q4 - Was Dumbledore right to let Lupin come to school?Q5 - Why does Hogwarts have such a rag-tag bunch of teachers?“Now, my three friends could hardly fail to notice that I disappeared once a month. I made up all sorts of stories. I told them my mother was ill, and that I had to go home to see her…I was terrified they would desert me the moment they found out what I was. But of course, they, like you, Hermione, worked out the truth and they didn't desert me at all. Instead they did something for me that would make my transformations the best times of my life. They became animagi.” Q6 - Do you know and understand how truly difficult it is to become an Animagus?Q7 - What was the craziest/most dangerous thing you've done in your life?Q8 - Have you ever put someone else's life in danger?“Sirius thought it would be — er — amusing, to tell Snape all he had to do was prod the knot on the tree-trunk with a long stick, and he'd be able to get in after me. Well, of course, Snape tried it — if he'd got as far as this house, he'd have met a fully grown werewolf — but your father, who'd heard what Sirius had done, went after Snape and pulled him back, at great risk to his life.”Q9 - Do you have any sympathy for Snape?

Chapter 5 - The DementorThey headed down to breakfast, where Mr Weasley was reading the front page of the Daily Prophet with a furrowed brow and Mrs Weasley was telling Hermione and Ginny about a Love Potion she made as a young girl. All three of them were rather giggly.Q1-  Do you think Mrs Weasley gave the love potion to Mr Weasley?“Harry, I knew you were, well, made of stronger stuff than Fudge seems to think, and I'm obviously pleased that you're not scared, but —”“Harry, swear to me you won't go looking for Black.”... “Why would I go looking for someone who wants to kill me?” said Harry blankly.Q2 - Why does Mr Weasley give Harry this advice to not go looking for Black?Q3 - Would you ever want to visit the Honeydukes and the Shrieking Shack?Q4 - Why would a building be more haunted than other buildings?And then, from far away, he heard screaming, terrible, terrified, pleading screams. He wanted to help whoever it was, he tried to move his arm, but couldn't…a thick white fog was swirling around him, inside him.Q5 - What are your thoughts on Dementors?Q6 - What do you think Hermione and McGonagall talked about?“It is not the nature of a Dementor to understand pleading or excuses.”Q7 - Do you think Hagrid will be a good teacher?Harry climbed the spiral stairs with no thought in his head except how glad he was to be back. They reached their familiar, circular dormitory with its five four-poster beds and Harry, looking around, felt he was home at last.Chapter 6 - Talons and Tea Leaves“Dad had to go out to Azkaban one time, remember, Fred? And he said it was the worst place he'd ever been. He came back all weak and shaking…They suck the happiness out of a place, Dementors. Most of the prisoners go mad in there.”Q1 - What do you think of Dementors sucking the happiness out of a place?“How're you supposed to be in three classes at once?” “Don't be silly,” said Hermione shortly. “Of course I won't be in three classes at once.”Q2 - Do you think Hermione could handle the workload of three classes at once?Q3 - Would you want to take a Divination class?PREDICTION #1 - Nevilles Grandma is Ill.PREDICTION #2 - Parvati must beware a red-haired man.PREDICTION #3 - February will bring the flu…she will lose her voice.PREDICTION #4 - By Easter one of them will leave forever.PREDICTION #5 - Lavender's dread will happen Friday the sixteenth of October.PREDICTION #6 - Neville breaking the pink cup. HAPPENEDQ4 - What are your thoughts on Trelawney?When Harry and Ron had had their teacups filled, they went back to their table and tried to drink the scalding tea quickly.Q5 - Can you drink scalding hot tea/coffee?“Right, you've got a wonky sort of cross.” he said, consulting Unfogging the Future. “That means you're going to have ‘trials and suffering' — sorry about that — but there's a thing that could be the sun. Hang on, that means ‘great happiness' .. so you're going to suffer but be very happy…”PREDICTION #7 - Harry has a deadly enemy.PREDICTION #8 - The club…an attack.PREDICTION #9 - The skull…danger in his path.PREDICTION #10 - The grim…an omen of death.“You'll forgive me for saying so, my dear, but I perceive very little aura around you. Very little receptivity to the resonances of the future.”PREDICTION #11 - Neville will be late to the next class.He hardly heard what Professor McGonagall was telling them about Animagi (wizards who could transform at will into animals), and wasn't even watching when she transformed herself in front of their eyes into a tabby cat with spectacle markings around her eyes…”Really what's gotten into you all today? … Not that it matters, but that's the first time my transformation's not gotten applause from a class.”“There is no need to tell me any more, Miss Granger. Tell me, which of you will be dying this year?”“I shall not conceal from you that I have very little patience with it [Divination]. True Seers are very rare, and Professor Trelawney…”Q6 - Thoughts on McGonagall's thoughts on Trelawney?Q7 - What do you think about Animagi?Q8 - If you were part of the Magical World, would you be more like Ron or Hermione with superstitions and grims?Q9 - Should Hagrid have left a note with the students for how they should open their books?Q10 - Would you want to ride a broomstick or a Hippogriff?“No one?” said Hagrid with a pleading look. “I'll do it,” said Harry.Q11 - Is Hagrid a good teacher?Q12 - Is Sirius Black that serious that Hagrid is upset at the trio for visiting him in the dark?Chapter 7 - The Boggart in the WardrobeQ1 - Have you ever milked an injury?Q2 - Would you be good at potions?“Have you heard? Daily Prophet this morning — they reckon Sirius Black's been sighted…Not too far from here.”Q3 - Do you think the Ministry of Magic should have someone in the 911 Dispatch office? Or do you think they can bug certain words…like Sirius Black?“Of course, if it was me,” he said quietly, “I'd have done something before now. I wouldn't be staying in school like a good boy, I'd be out there looking for him.” “What are you talking about, Malfoy?” said Ron roughly. “Don't you know, Potter?”Q4 - What doesn't Harry know that Malfoy knows about Sirius Black?Q5 - What is wrong with Snape to be testing on Neville's toad?Q6 - How do you think Hermione is taking all these classes?“Good afternoon,” he said, “Would you please put all your books back in your bags. Today's will be a practical lesson. You will only need your wands.”…”when their old teacher brought pixies…” LONG LIVE LOCKHART!“I was hoping that Neville would assist me with the first stage of the operation,” he said, “and I am sure he will perform it admirably.”Q7 - What are your first thoughts on Lupin?Q8 - What would the Boggart take shape if you confronted it? (aka, what frightens you?)“It's always best to have company when you're dealing with a Boggart.”“I would like all of you to take a moment now to think of the thing that scares you most, and imagine how you might force it to look comical?”Q9 - How would you change your biggest fear into something humorous?Q10 - What is Lupin afraid of?“What would it have been for you?” said Ron, sniggering. “A piece of homework that only got nine out of ten?”Chapter 8 - Flight of the Fat LadyQ1 - How do you think the Fat Lady feels that she is just known as the Fat Lady?“Well, look at it logically,” said Hermione turning to the rest of the group. “I mean, Binky didn't even die today, did he, Lavender just got the news today —” Lavender wailed loudy, “—and she can't have been dreading it, because it's come as a real shock—” “Don't mind Hermione, Lavender,” said Ron loudly, “she doesn't think other people's pets matter very much.”Q2 - Is Hermione a good pet owner?Q3 - If you started a magical shop in Hogsmeade, what would it be?He thought for a moment of telling Lupin about the dog he'd seen in Magnolia Crescent, but decided not to. He didn't want Lupin to think he was a coward.Q4 - What is this potion that Snape gives Lupin?Q5 - How is fearing fear most wise?“He got very angry when she wouldn't let him in, you see.” Peeves flipped over, and grinned at Dumbledore from between his own legs. “Nasty tempers he's got, that Sirius Black.”Q6 - How good is the end of this chapter?!

Dear ‘Arry and Ron, I hope this letter finds yeh well! Things 'round Hogwarts are as as wild as a bunch of maturing mandrakes in the same pot, I tell yeh. Speaking of which, Sirius Black is hot on the wormtail of that good fer nothin' Peter Pettigrew for rattin' out Lilly and James Potter. I hear CHO CHANG has made her presence felt; ‘Arry's stomach is likely bucking like a hacked off Hippogriff. Lastly, you two OAFS cut Hermione some slack. If she doesn't slap ye' both, I WILL. Gotta go, me marzipan is likely burning. -Hagrid wuz here

In episode 460, the "survival experts" take the time to watch the second part of Zack Snyders Star Wars film. Chris happily returns to the rebel moon and Jack unhappily returns to do a lot of slow motion farming.  E-mail your survival suggestions to us at wecouldsurvivethat@gmail.com or Twitter @WeCouldSurvive or find older episode on Youtube at: https://www.youtube.com/channel/UCXAa8-wNqv1G14ts_DHenkg/feed

Can YOU dance like a Hippogriff? Nananananananana! It's Yule Ball season and this week, we're fraternizing with the enemy, proving our emotional span is greater than a teaspoon and dodging Moody's creepy eye, all while trying our best to keep our dates happy and turn it up on the dance floor! Chapter-by-Chapter continues with Goblet of Fire, Chapter 23: The Yule Ball 7-Word Summary: Hormones fly amok at Hogwarts' Big Holiday Main Discussion #1: Hermione's Chapter Would anyone that Hermione took to the ball have been free of Ron's criticism? The Courting of Hermione Granger What do we think of Viktor Krum's approach... and sheer vulnerability? We react to Hermione's decision to have Madam Pomfrey shrink her teeth Main Discussion #2: Hurricane Ron Animal Magnetism, baby! What makes Ron suddenly so full of himself? Sour face, sour mood: Ron casts a massive black cloud over the entire Yule Ball Could Ron be jealous his two besties are a part of the Elite Champions date club? Low Blow, Ron! Does Hermione truly not have Harry's best interests at heart? We debate if Harry did enough to defuse the tension between Ron and Hermione Odds and Ends cover Sirius' knife, Dobby's Christmas gift, Barty Crouch Jr.'s mysterious absence, Snape and Karkaroff's odd relationship, Beetle Rita and more! Next week: Goblet of Fire, Chapter 24: Rita Skeeter's Scoop Quizzitch question: According to Rita Skeeter, what department at the Ministry would be particularly interested in Hagrid's Blast-Ended Skrewts? Visit MuggleCast.com for episode transcripts, social media links, our full episode archive, our favorite episodes, and to contact us! Check out the MuggleCast Overstock Store to grab MuggleCast t-shirts, signed album art, and so much more! Supplies are limited, so act fast and help us support the show! For twice-monthly Bonus MuggleCast, as well as other great benefits, including the chance to co-host the show, a new physical gift every year, and a video message from one of the four of us visit Patreon.com/MuggleCast! Learn more about your ad choices. Visit megaphone.fm/adchoices

Mooncalves and dragons and nifflers- oh my! This week, we're talking about the non-human side of the Wizarding World. What are the ethics of eating magical animals among witches and wizards? What exactly makes a magical creature different from, say, a squirrel? What does a Hippogriff have in common with our old dog Joe? Also- the mysterious ocean Bloop, Teen Wolf, and Macho Man Randy Savage. Support the showSupport FFH on Patreon: patreon.com/thefoxandthefoxhoundFollow us!IG: @thefoxandthefoxhoundTikTok: @thefoxandthefoxhound

In episode 446, the "survival experts" are in a galaxy far far away but not the galaxy you are thinking of. Chris has to fight a big spider lady that wondered in off the set of Doctor Who and Jack tries to fly a hippogriff that got lost from the Harry Potter set and ended up in space. E-mail your survival suggestions to us at wecouldsurvivethat@gmail.com or Twitter @WeCouldSurvive or find older episode on Youtube at: https://www.youtube.com/channel/UCXAa8-wNqv1G14ts_DHenkg/feed

800 Celebration: Time flies when you are having fun! We are celebrating EPISODE 800! It's time for a POWER HOUR CELEBRATION! Rebel Moon: Are we Zack Snyder apologists!? We report in with our review of Rebel Moon which could be categorized as Rebel Moon Cope. A Magic Moment: We go over Mike's recent MAGIC MOMENT at Avenger's Campus and then shit gets REAL. LET'S JUST TALK!, BOOGIE NIGHTS!, DON CHEADLE!, EPISODE 800!, 4 MORE YEARS!, JOE BERRY!, WRESTLING!, MIKE RETURNS!, JAKE LEAVES!, CALLED IN!, MILESTONE!, CELEBRATION!, POWER HOUR!, VOICEMAIL!, NEW YEAR'S CELEBRATION!, PATREON!, VGS!, TOP GAMES OF 2022!, GINO!, ADIDAS TRACK SUIT!, SPIRIT JERSEY!, CANDY!, POWER HOUR!, SHOTS!, GIFTS!, LASERDISCS!, RAIDERS OF THE LOST ARK!, SPEED!, PULP FICTION!, SLAYER!, BUSH!, MACHINEHEAD!, SCOTT PILGRIM!, ANIME!, NETFLIX!, CHEYENNE!, WALTON GOGGINS!, FALLOUT!, TV SHOW!, GORY!, BOASTY!, DETROIT LIONS!, ZACK SNYDER!, STYLE!, WATCHMEN!, 300!, DAWN OF THE DEAD!, MAN OF STEEL!, REBEL MOON!, A CHILD OF FIRE!, SUCKER PUNCH!, KUROSAWA!, SEVEN SAMURAI!, BUILD A TEAM!, OCEAN'S 11!, STAR WARS!, PHANTOM MENACE!, ANAKIN!, WATTO!, TARAK!, BLOODAXE!, HIPPOGRIFF!, KICKSTART MY HEART!, MOTLEY CRUE!, COREY FELDMAN!, COMEBACK KING!, JEDI!, LIGHTSABER!, RAY FISHER!, PRODUCTION DESIGN!, SPEED RAMPS!, ACTION!, SLICK!, THE RAID!, JOHN WICK!, ANIME!, METAL GEAR SOLID!, JAPANESE!, KEVIN!, TWFS!, OBEY!, ANDRE THE GIANT HAS A POSSE!, THEME PARKS!, MAGIC MOMENT!, AVENGER'S CAMPUS!, ANT-MAN!, BUG'S LIFE!, THOR!, CAPTAIN AMERICA!, NEW AVENGER!, NO FRIENDS!, ALONE!, SPIDER-MAN!, RYAN!, CALL IN!, PODCAST ELITE!,  You can find the videos from this episode at our Discord RIGHT HERE!

The Latent Space crew will be at NeurIPS on Tuesday! Reach out with any parties and papers of interest. We have also been incubating a smol daily AI Newsletter and Latent Space University is making progress.Good open models like Llama 2 and Mistral 7B (which has just released an 8x7B MoE model) have enabled their own sub-industry of finetuned variants for a myriad of reasons:* Ownership & Control - you take responsibility for serving the models* Privacy - not having to send data to a third party vendor* Customization - Improving some attribute (censorship, multiturn chat and chain of thought, roleplaying) or benchmark performance (without cheating)Related to improving benchmark performance is the ability to use smaller (7B, 13B) models, by matching the performance of larger models, which have both cost and inference latency benefits.Core to all this work is finetuning, and the emergent finetuning library of choice has been Wing Lian's Axolotl.AxolotlAxolotl is an LLM fine-tuner supporting SotA techniques and optimizations for a variety of common model architectures:It is used by many of the leading open source models:* Teknium: OpenHermes, Trismigestus, CollectiveCognition* OpenOrca: Mistral-OpenOrca, Mistral-SlimOrca* Nous Research: Puffin, Capybara, NousHermes* Pygmalion: Mythalion, Pygmalion* Eric Hartford: Dolphin, Samantha* DiscoResearch: DiscoLM 120B & 70B* OpenAccess AI Collective: Manticore, Minotaur, Jackalope, HippogriffAs finetuning is very formatting dependent, it also provides prompt interfaces and formatters between a range of popular model formats from Stanford's Alpaca and Steven Tey's ShareGPT (which led to Vicuna) to the more NSFW Pygmalion community.Nous Research MeetupWe last talked about Nous at the DevDay Recap at the e/acc “banger rave”. We met Wing at the Nous Research meetup at the a16z offices in San Francisco, where they officially announced their company and future plans:Including Nous Forge:Show NotesWe've already covered the nuances of Dataset Contamination and the problems with “Open Source” in AI, so we won't rehash those topics here but do read/listen to those if you missed it.* Axolotl GitHub and Discord* The Flan paper and dataset* StackLlama model and blogpost* Multipack paper* Our episode with Tri Dao* Mamba state space models - Tri Dao and Albert GuTimestamps* [00:00:00] Introducing Wing* [00:02:34] SF Open Source AI Meetup* [00:04:09] What is Axolotl?* [00:08:01] What is finetuning?* [00:08:52] Open Source Model Zoo* [00:10:53] Benchmarks and Contamination* [00:14:29] The Case for Open Source AI* [00:17:34] Orca and OpenOrca* [00:23:36] DiscoLM and Model Stacking* [00:25:07] Datasets and Evals over Models* [00:29:15] Distilling from GPT4* [00:33:31] Finetuning - LoRA, QLoRA, ReLoRA, GPTQ* [00:41:55] Axolotl vs HF Transformers* [00:48:00] 20x efficiency with StackLlama and Multipack* [00:54:47] Tri Dao and Mamba* [00:59:08] Roadmap for Axolotl* [01:01:20] The Open Source AI CommunityTranscript[00:00:00] Introducing Wing Lian[00:00:00] ​[00:00:00] swyx: Welcome to Latent Space, a special edition with Wing Lien, but also with our new guest host, Alex. Hello, hello. Welcome, welcome. Again, needs no introduction. I think it's like your sixth time on Latent Space already. I think so, yeah. And welcome, Wing. We just met, but you've been very prolific online. Thanks for having me.[00:00:30] Yeah. So you are in town. You're not local. You're in town. You're from Minneapolis?[00:00:35] Wing Lian: Annapolis. Annapolis. It's funny because a lot of people think it's Indianapolis. It's I've got Minneapolis, but I used to live out at least in the San Francisco Bay Area years ago from like 2008 to 2014. So it's fairly familiar here.[00:00:50] swyx: Yep. You're the maintainer of Axolotl now, which we'll get into. You're very, very prolific in the open source AI community, and you're also the founder of the Open Access AI Collective. Yeah. Cool. Awesome. Maybe we can go over a little bit of your backgrounds into tech and then coming into AI, and then we'll cover what[00:01:06] Wing Lian: happens and why you're here.[00:01:08] Yeah. So. Back on tech, so I started years ago, I started way back when I was scraping, Apartment websites for listings and then, and then building like SEO optimized pages and then just throwing Google AdSense on it.[00:01:24] And that got me through like college basically. Is[00:01:27] swyx: that decent money? And what year[00:01:28] Wing Lian: was this? Like 2004, 2005. Yeah, that's decent money. It's like thousand bucks a month. But as a college student, that's like. Gravy. Really good money, right? So, and then there's just too much competition It's just sort of like died off. I was writing stuff in like Perl back then using like like who nobody hosted anything on Perl anymore, right? Still did a little bit more like computer tech support and then software, and web more professionally.[00:01:54] So I spent some time working on applications in the blood industry. I came out to San Francisco for, I was at SGN, so Social Gaming Network, as a startup. They started doing, with Facebook apps, and then they pivoted into doing mobile apps. And then, from there, I spent time.[00:02:14] I've quite a few more startups since then and in the last few years I've been in the music space So like I was at United Masters for a while and then past year I've been at SoundCloud, but not doing that anymore and now that I have a lot more time It's just like all right.[00:02:30] We're going full bore on axolotl and we're gonna we're gonna crush AI So yeah,[00:02:34] SF Open Source AI Meetup[00:02:34] swyx: totally you so you're here in town for the open source. Yeah, I meet up that we had yesterday Yep, yeah, that was amazing. Yeah, it was a big collection. Olama, Noose Research, Alignment Lab, Anyone else that I missed? I mean, Jeremy Howard is his own thing.[00:02:47] Yeah.[00:02:49] And Alex, you're also there. You love to bring SF to the world. Your takes?[00:02:55] Alex Volkov: It's incredible that we recorded a Thursday Eye episode after that one. And LDJ, who's usually co hosts Thursday Eye, just like briefly mentioned, Oh yeah, I talked about it.[00:03:04] Like, I saw Karpathy, and then I talked to Jeremy Howard, and the guy from Mistral came in, and it's like, He's talking about all these, titans of industry, basically, that outside of SF, You just don't meet casually hanging out in the same space. You can't, pull somebody. He ran into the Laylow from Mistral, he ran into him while, drinking water.[00:03:20] He didn't even know he was there. It's just, that type of stuff is really hard to find outside of SF. So, absolutely, absolutely great. And also, presentations from Alignment Labs, presentations from News Research, news issues, talked about. Forge, and some of[00:03:33] swyx: the other stuff they announced. We can say now they're officially a company.[00:03:36] I met Technium.[00:03:37] He[00:03:37] Alex Volkov: came over here. He didn't want to get recorded. But maybe.[00:03:41] Wing Lian: We'll wear him down at some point. Yeah, I'm excited for Forge. They've positioned it as this agentic sort of framework where it's just Drag and drop things and, fill in text with where you want to inject different variables and it opens up all of these potentials for data pipelines now, right?[00:03:56] And using your own local LLMs and not relying on GPT 4 or anything like that. Yeah, yeah,[00:04:02] swyx: good stuff. Okay, so let's maybe go into the Axolotl origin story and then we have, we have some intro or background.[00:04:09] What is Axolotl?[00:04:09] swyx: To do on like the open source model universe and also on fine tuning, but maybe just, since you're talking about your personal journey, what was your personal journey into[00:04:18] Wing Lian: axolotl?[00:04:19] Yeah, so my personal journey started like back in mid March, completely unrelated to AI and axolotl. And it really started, I fell while skiing, I torqued. Great 3 MCL sprain and being sort of like an active person that can no longer be active because the two, couldn't play soccer, because that is requires to have having knees until I, it's healed.[00:04:42] So I. I decided I needed to find something to do to take up my free time. And that became, well, let's learn how to train in, these language models. It was everywhere. So I was like, all right, I'm just going to sit down, learn. I think I used like other, I think I was using like Alpacalora.[00:05:00] Cause I think the Alpaca paper had just came out, come out then. So I was like using Alpacalora repo and sort of like learning how to use like. None of us were like GPU rich back then, and none of us, most of us still we're still all GPU poor, but I was doing what was it, like 4 bit, Alpaca Lord, there was like a 4 bit version where we were doing quant, or 8, no, 8 bit quantizations, and then I think they had released QLOR a little bit later, and I think right when, before QLOR came out, I was already starting to do fine tunes, but having this need to sort of like mix data sets together, and If you've ever looked at all the various different datasets available on HuggingFace, they all have various different prompt formats, and, it's sort of a nightmare, and then I think the other piece is if you've ever tried to fine tune, at least Back then probably the ecosystem's a little better now.[00:05:54] Everybody required that you say, alright, you put your hyperparameters as command line arguments. And so it's always like, well, I now have to go copy and paste my previous thing and to change things out. And I really wanted it. to be in a YAML file because it was more portable and reproducible.[00:06:09] So I was doing that and then the QLOR paper came out. Tim Dettmer announced that and then somebody looked it up for me yesterday and it's like between that announcement it took us seven days to get that integrated into Axolotl, right? Which is like, it's not. I wouldn't say it's really fast, but in a manner that, is in a, a reusable framework, I think it was quite the accomplishment then.[00:06:33] And so we started, picking up traction with people there. And then it's just been building models, and then just iterating what my needs are. So, yeah. Excellent. Yeah. I[00:06:44] Alex Volkov: want to ask, for folks who are listening who never heard of Axolotl, now do you describe how you got there?[00:06:49] Can you, how do you summarize this for folks who maybe haven't fine tuned anything. They know about open source LLM exists, they maybe know like LLAML, what's XLR for somebody who doesn't know. I've never heard of a data set curation[00:07:01] Wing Lian: creation before. We sort of have to take a step back and understand that, when you've got these language models, you have what I think most people refer to as like base models, also known as like foundational models, right?[00:07:15] Where some benefactor, whether it's Meta or Mistral or whoever, has gone and spent all this money. To train these models on huge corpuses of text, right? And these, these corpuses, they're generally good across lots of different things, but they're really good at just saying, talking on and on and on, but they're not good at, following instructions or having chats or anything like that.[00:07:40] So, when you think about fine tuning, it's like Saying, all right, we have this really sort of good generalized, text completion thing, and I want to turn it into something that I can talk to or have, follow instructions. So, I think fine tuning is probably best defined in like that.[00:07:58] swyx: Okay, got it.[00:07:59] And we actually[00:08:01] What is finetuning?[00:08:01] swyx: Do want to make sure that we have like an overall introduction to fine tuning for people because again like trying to make sure that we bring everyone along in this, in this journey. We already went into Loras and QLoras without explaining what[00:08:12] Wing Lian: they are. Oh yes, yes, sorry.[00:08:14] swyx: And so I will put things in my words and you can correct me as, as, as my I'll be the village idiot here.[00:08:21] So, so fine tuning is basically sort of grabbing an open source model off the shelf, and then basically doing further training on it with a custom dataset of your own. Primarily, people use it, think about it as fine tuning for JSON output, or fine tuning for a style of response. Let's say you wanted to tell jokes, or be funny, or be short, or whatever.[00:08:43] Just the open source AI community has really fine tuned in all sorts of different manner. I think we'll go over those those things now. Let's go over those things now, and then we'll talk about fine tuning methods.[00:08:52] Open Source Model Zoo[00:08:52] swyx: So there's a universe of people who fine tune stuff. Yesterday in your slides, you had, I'll just list some of these and then we'll maybe go through some of them, right?[00:08:59] So Technium is personally leading Open Hermes, which is I think the sort of premier model out of the news. news community. There's OpenOrca, which you had a hand in. News, the news research itself also has Capybara and Puffin and all the others. There's Pygmalion, which I've never messed with.[00:09:14] Eric Hartford, I am aware of his Uncensored Models and his Samantha Models. Disco Research with Disco LM. And then you personally have done Manticore, Minotaur, Jackalope, and Hippogriff. What should people know about all these names? Being part of AI Twitter is seeing all these things and going dude, I'm being DDoS'ed by all these things and I don't know how different they are.[00:09:32] What should people know? Yeah, so[00:09:34] Wing Lian: I think on a lot of these models, generally, we like to think of those as sort of general models, so If you think about it, what is GPT 4, what is Chad GPT? It's a good general model, and then, One of the services I think that OpenAI offers is like these fine tunings where you're a business and you have very specific business use cases and you might fine tune for that use case.[00:10:00] All of these models are really just general use case that you can then go and maybe Fine tune another lore over it for your use cases, but they tend to be good. With good being relative, it's open source. Open source AI is still sort of is infancy. So, good is, it's pretty reasonable.[00:10:18] It's probably still better than most, high schoolers at answering questions and being able to like figure things out and, and reasoning skills and math and those sorts of things, right?[00:10:27] swyx: And also as measured on the Hugging[00:10:29] Wing Lian: Face leaderboard. Yes, well, that's like a whole other discussion, right, there's a whole other, group of people who, and I, I mostly agree with them that, benchmarks can be, are pretty bogus these days, LM says, I think they published something recently where, even if you think the dataset's not contaminated, you can go and, find contamination And maybe we should step back and say what contamination is, right?[00:10:53] Benchmarks and Contamination[00:10:53] Wing Lian: So we have all of these data, when you go and do these benchmarks, there's a specific data set where there are these questions and usually it's multiple choice. And what can happen is, well, sometimes someone It puts the question, maybe maliciously, maybe accidentally, into the training dataset, and now the, the, your model knows how to answer the test questions really well, but it doesn't, it hasn't generalized the ability to actually do that[00:11:20] Alex Volkov: right.[00:11:21] We've seen some folks competitively announce models that are like the best at that leaderboard, but then it's, it's quite obvious that, In open source? Yeah, and in that leaderboard, for Hugging Face specific, I don't know if LMCs, if that had suffered, but we, there's been some models that seem to have been competitively trained and some leakage happened into their,[00:11:41] swyx: like, supposal.[00:11:43] I understand, once there's been a credible assertion, Hugging Face actually does take them down, right? Yeah, yeah,[00:11:48] Alex Volkov: which is really hard to know, right?[00:11:50] swyx: It's really hard to know, sometimes it's like a pure accident,[00:11:52] Alex Volkov: it's oh, oops. You're going through a mixer. I think, a responsible So acknowledgement, that this kind of happened to you is also important.[00:11:58] I saw LDJ from news research can acknowledge that. Because many of these datasets are collections of other datasets. There's a bunch of people are baking, basically. It's alchemy. Right. And so sometimes you don't know. Sometimes you pull an open source dataset and they announce, oh, you know what, actually, the MMLU benchmark which we used to Specifically identify models that did go into this data set, that then went into that data set.[00:12:22] So sometimes it's actually an accident and folks take it down. But I've seen some competitive folks who want to put their name out there because people are starting to notice which is the top[00:12:30] swyx: model. For those who want a fun take on this so the file one dataset. FindOne model from Microsoft was accused of being contaminated.[00:12:37] And I saw this joke paper that was fantastic. It was called, training on the test set is all you need. It's a super small model that just memorizes everything. It was fantastic. So yeah, contamination, I think we've actually covered it in a previous episode before. So we're good. But again, I want to give people a map into the open source AI model, the universe.[00:12:57] And Alex, you can also jump in here because you guys have spent a lot more time with them than I have. So, what should people know about Technium? What should people know about Noose? And then we can go down the list. Yeah,[00:13:05] Wing Lian: I think so. I think if we start with, Technium. When you talk to him, he's gonna say, I think, I think his response is that he wants to build GP4 on his laptop, right?[00:13:14] So, very, very good at building general models. I think with Noose, Noose Research, they're looking at more, sort of, More, more research focused things, like their Yarn models, I don't, I don't, they didn't actually train their, they have their own trainer for their Yarn models, but So they did not use Xlato for that one?[00:13:30] They didn't use that, but like Is that, you don't have support for it? I think we do support Yarn, I think, I'd have to double check that answer. Yeah, I'm just kind of curious what you can and cannot support, and Yeah, I mean, Yarn is supportable, it's basically, I think it's just replacing, I think, the rope part of that, so Yeah, not, not a big deal.[00:13:48] Yeah, it's not a big deal, it's just I haven't gotten to it, not enough people have asked, I think a lot of people have asked for other things, so it's just, squeaky wheel, right? I think at the end of the day, people are like building these data sets and I think if you sort of map things chronologically, these make more sense because it's like, how do we incrementally improve all of these models?[00:14:07] So a lot of these models are just incremental improvements over the last thing, right? Whether it is sort of through methods of how do we, how did we curate the data set? How did we improve the quality of the data set? So, you maybe LDJ talked about it right on I think for, for Capybara and Puffin, like how those, those were very specific dataset curation techniques that he works on.[00:14:29] The Case for Open Source AI[00:14:29] Alex Volkov: So there's, folks are doing this for dataset curation. Folks are doing this for skillset building as well. Definitely people understand that open source is like very important, especially after the, the, the, the, the march, the debacle, the OpenAI weekend that we all had. And people started noticing that even after developer day in OpenAI, the APIs went out.[00:14:48] And then after that, the whole leadership of the company is swiftly changed and people, there was worries about, you know. How can people continue building AI products based on these like shaky grounds that turned attention definitely to Technium at least in open RMS I started seeing this more and more on Twitter, but also other models and many companies They're gonna start with open AI just to get there quick, and then they they think about okay Maybe I don't want to share my knowledge.[00:15:13] Maybe I don't want to sign up for Microsoft. Maybe they will change their terms and conditions so What else is out there? They turned to other companies. Up until yesterday, Google was nowhere to be found. We've talked about Gemini a little bit before in a previous And you can tune in[00:15:26] swyx: to[00:15:26] Alex Volkov: Thursday Eye.[00:15:26] Yeah, you can tune in to Thursday Eye. We covered the Gemini release a little bit. And but many are turning into the open source community and seeing that Meta released and continues to release and commit to open source AI. Mistral came out and the model is way smaller than LLAMA and performs Significantly better.[00:15:43] People play with OpenRMS, which is currently techniums based, news researched, sourced, axolotl trained OpenRMS, I assume, right? And then they play with this and they see that, okay, this is like GPT 3. 5 quality. We had GPT 4. 5 birthday just a week ago. A week ago, a year ago, a week ago, we never, interacted with these models of this caliber.[00:16:04] And now there's one open source, one that's on my laptop, completely offline, that, I can continue improving for my use cases. So enterprises, companies are also noticing this. And the open source community folks are building the skill set, not only the data sets. They're building the actual kind of, here's how we're going to do this, with Axelotl, with these data sets.[00:16:21] The curation pieces. Now. Interesting. There's like recipes of curation. The actual model training is kind of a competitive thing where people go and compete on these leaderboards that we talked about, the LMC arena, and that recently added open air and recently added open chat and a bunch of other stuff that are super cool.[00:16:37] The hug and face open source leaderboard. And so there's a competitive aspect to this. There's the open source. Aspect to this, like Technium says, I want GPT 4 on my laptop. There's the, let me build a skill set that potentially turns into a company, like we saw with Noose. Noose just, started organizing, a bunch of people on Discord, and suddenly, they're announcing their company.[00:16:54] It's happening across all these modalities, and suddenly all these people who saw these green pastures and a fairly quick way to, hey, here's a cool online community I can, start doing cool stuff with. You mentioned the same in the beginning, right? Like, after your accident, what's cool, let me try this out.[00:17:08] Suddenly I start noticing that there's a significant movement of interest in enterprising companies into these areas. And, this skill set, these data sets, and this community is now very Very important, important enough to create an event which pulls in Andrei Karpathy from OpenAI to come and see what's new Jeremy Howard, like the event that we just talked about, people are flying over and this is just a meetup.[00:17:28] So, definitely, the community is buzzing right now and I think Axelot is a big piece as well.[00:17:34] Orca and OpenOrca[00:17:34] Wing Lian: Cool. Maybe we can talk about like Orca real quick, Orca, OpenOrca rather, I think there was a lot of buzz when, the first Orca paper came out. And just briefly, what is Orca? Yeah, Orca was basically having traces of like chain of thought reasoning, right?[00:17:48] So they go and they, they distill sort of GPT 4. They take, they take a sampling of data from the Flan dataset. Maybe we can like add some show notes in the Flan dataset. Yeah, but we've covered it. Okay, cool. Use GPT 4 to say, all right, explain this in a step by step reasoning, right?[00:18:06] And then you take that and you, they train the model and it showed, very good improvements across a lot of benchmarks. So OpenOrca was sort of the open reproduction of that since Microsoft Research never released that particular data set. And going back to sort of the Hugging Face leaderboard thing, those models did really well.[00:18:23] And then I think, so sort of the follow up to that was SlimOrca, right? I think Going into and building the OpenOrca dataset, we never really went in and, validated the actual answers that GPT 4 gave us, so what we did was one from OpenChat actually cross referenced the original Flan, the original Flan response, the human responses, the correct answers with the dataset, and then I went and took it and sent all of, both of them to GPT 4 and said, is this answer mostly correct, right?[00:18:54] Yeah. And then we were able to filter the dataset from, At least of the GPT 4 only answers from like 800, 000 to like 500, 000 answers or rows and then, and then retrain the model and it had the same performance as the original model to within I think, 0. 1 percent here about, and 30 percent less data.[00:19:13] So, yeah. Okay.[00:19:15] swyx: Interesting. So, I mean, there's, there's so much there that I want to highlight, but yeah. Orca is interesting. I do want people to know about it. Putting chain of thought into the data set like it's just makes a ton of sense one thing I think it would be helpful for people to scope thing these things out is how much data are we talking about when when you When people are fine tuning and then how much time or resources or money does it take to train to fine[00:19:36] Wing Lian: tune?[00:19:37] Yeah, so I think there's a little bit of overlap there with sort of like fine tuning techniques, but let's say Orca and I think even Hermes, they're both relatively large data sets like 10 billion tokens. Yeah. So large data sets being or the original Orca was, or the original open Orca was 800,000 rows.[00:19:55] I believe it was somewhere in the ballpark of like a gigabyte of data, of gigabyte, of text data. And I, I don't. I believe, Hermes was, is like a quarter million rows of data, I don't know the actual byte size on that particular one. So, going and training a, let's, let's say everybody's training 7 billion Mistral right now, right?[00:20:15] So, to tri I, I believe to fine tune 7 billion Mistral on, let's say, 8 A6000s, which have 48 gigabytes of VRAM, I believe, It takes about 40 hours, so 40, and then that's, depending on where you get your compute, 40 times 6, so it's like 500 to fine tune that model, so, and, and that's assuming you get it right the first time, right?[00:20:44] So, you know.[00:20:45] swyx: Is, is that something that X. Lotto handles, like, getting it right the first[00:20:48] Wing Lian: time? If you talk to anybody, it's like you've probably tried at least three or four runs or experiments to like find the right hyperparameters. And after a while you sort of have a feel for like which, where you need your hyperparameters to be.[00:21:04] Usually you might do like a partial training run, do some benchmark. So I guess for Al Farouk, whether you're going by his. This is Jeremy, he's, his actual name, or his twitter handle. He released the Dharma dataset, which is basically a subset of all the benchmarks. And Axolotl actually supports, you know taking that subset and then just running many benchmarks across your model every time you're doing an evaluation so you can sort of like see sort of relative it's not going to be the actual benchmark score, but you can get ideas alright, is this benchmark improving, is this benchmark decreasing, based on, you know Wait,[00:21:39] swyx: why don't you run the full benchmark?[00:21:41] What, what, what The[00:21:42] Wing Lian: full benchmarks take Take a long time. Significant, yeah, significant amount of time. Yeah. And Okay, so that's like[00:21:48] swyx: mini MMLU. Yeah. Like,[00:21:49] Wing Lian: mini BigBench or whatever. Yep, exactly.[00:21:51] Alex Volkov: It's really cool. We, when I joined Web2Masters just recently, and one of the things that I try to do is hey I'm not, I'm a software engineer by trade, I don't have an MLE background, But I joined a company that does primarily MLE, and I wanted to learn from the community, Because a lot of the open source community, they use weights and biases, And the benchmark that you said that Pharrell did, remind me of the name, sorry.[00:22:13] Dharma? Dharma, yeah, yeah. So Luigi showed me how Dharma shows inside the dashboard. In Wi and Biases dashboard and so you can actually kinda see the trending run and then you can see per each kind of iteration or, or epoch or you can see the model improving trending so you can on top of everything else.[00:22:29] The wi and biases gives like hyper parameter tracking, which like you, you started with common line and that's really hard to like remember. Also the Dharma data set, like the quick, the mini orca mini, you mini many different things. It's pretty cool to like visualize them as well. And I, I heard that he's working on a new version of, of Dharma, so Dharma 2, et cetera.[00:22:47] So hopefully, hopefully we'll see that soon, but definitely it's hard, right? You start this training around, it said like 40, 50 hours. Sometimes, sometimes it's like your SSHing into this machine. You, you start a process, you send it with God and you just go about your day, collecting data sets, and then you have to return.[00:23:04] And the whole process of instrumentation of this is still a little bit like squeaky but definitely. Tuning performance, or like grabbing performance in the middle of this, like with Dharma and some other tools, is very helpful to know that you're not wasting precious resources going somewhere you shouldn't go.[00:23:21] Yeah.[00:23:22] swyx: Yeah. Very cool. Maybe I'll, I'll, before we go into like sort of more details on fine tuning stuff, I just wanted to round out the rest of the Excel autoverse. There's, there's still Eric Hartford stuff. I don't know if you want to talk about Pygmalion, Disco, anything that you know about[00:23:35] Wing Lian: those, those things.[00:23:36] DiscoLM and Model Stacking[00:23:36] Wing Lian: Yeah, I think like one of the, definitely one of the more interesting ones was like the Disco 120b, right? Yeah, I know nothing about it. Yeah. So, so. Alpen from Pygmalion AI, right, so they, so Pygmalion is a sort of a, it's, it's, they have their own community, a lot of it is based around, roleplay models, those sorts of things, and Alpen, like, put together, merged together Llama270B, so, and Alpen, like, put together, merged together Llama270B, so, I don't remember how he stacked them together, whether he merged the layers in between. There's a whole, there's a whole toolkit for that by Charles Goddard, where you can like take a single model and like stack them together or multiple models merge.[00:24:18] That's like a whole other talk and a whole other tool set, but was able to create this 120. Billion parameter model out of a LAMA two 70 B. And then I believe the, yeah, disco is a fine tune of, of the, the, the sort of the base one 20 B is, I believe Goliath one 20 B. So, and, and what are the[00:24:37] swyx: headline results that people should know about[00:24:39] Wing Lian: disco?[00:24:39] I think for the headline results, I, I've, I haven't played with it personally because it's. It's a very large model and there's a lot of GPU, right? But, like, from what I've heard anecdotally, it performs really well. The responses are very good. Even with, like, just, even the base model is a lot better than, Llama70b.[00:24:57] So, and we, I think generally everybody's like, we would all love to fine tune Llama70b, but it's just, it's so much, it's so much memory, so much compute, right?[00:25:07] Datasets and Evals over Models[00:25:07] Wing Lian: I[00:25:07] Alex Volkov: want to touch on this point because the interesting thing That comes up out of being in this ecosphere and being friends with open source folks, tracking week to week state of the art performance on different models.[00:25:19] First of all, a lot of the stuff that the folks do a couple of weeks ago, and then something like Mistral comes out, and a lot of the stuff back then, Doesn't technically make sense anymore. Like the artifacts of that work, the actual artifacts, they don't no longer make sense. They're like lower on the on, on the hug and face leaderboard or lower on LM CS leaderboard.[00:25:36] But some of the techniques that people use, definitely the datasets. The datasets keep traveling, right? So open airmen, for example, is the dataset. The tum cleaned up for only. Open sourceable data that previously was just Hermes. And that, it was previously used to train Lama. And then once Mistral came out, it was used to train Mistral.[00:25:54] And then it became significantly better on the 7b base Mistral. So the data sets keep traveling, keep getting better a little bit here and there. And so the techniques improve as well. It looks like both things are simultaneously true. The artifacts of a month and a half ago. The, the actual models themselves, it's great the hug and face has them, because not every company can keep up with the next weeks', oh, I, I'll install this model instead, sell this model instead.[00:26:19] But the, the techniques and the, the dataset keep improving as we go further, and I think that's really cool. However, the outcome of this is that for a long time. For many, many people, including us, that we do this every week. We literally talk with people who release these models every week. It's really hard to know.[00:26:36] So, there's a few aspects of this. One, I think, like you said, the bigger model, the 70B models, you actually have to have somebody like Perplexity, for example, giving you access to the 70B really fast. Or you have to, like, Actually, find some compute, and it's expensive, especially for the bigger models. For example Falcon 180B came out, like the hugest open source model.[00:26:56] How do you evaluate this if you can't run it? Nobody liked it. It's really, so first of all, nobody liked it, but secondly, only the people who were able to find compute enough to run inference on this, they only had like, I can't run this on my laptop, and so that's why it's much easier, something like OpenRMS 7 to be, 7B, it's much easier, because you can run this on your MacBook.[00:27:14] It's much easier to evaluate. It's much easier to figure out the vibes, right? Everybody talks about the vibes as an evaluation check. If you're plugged in enough, if you follow the right people, if they say pretty much the same things all independently, then you run into a problem of whether they're repeating, and their stochastic parents are repeating the same thing, or they actually evaluated themselves.[00:27:31] Yeah, you never know. But, you never know, but like, I think on a large enough scale on Twitter, you start getting the feel. And we all know that like, OpenRMS is one of the top performing models, benchmarks, but also vibes. And I just wanted to highlight this vibes checks thing because you can have the benchmarks, you can have the evaluations, they potentially have contamination in them, potentially they not necessarily tell you the whole story because some models are good on benchmarks, but then you talk to them, they're not super helpful.[00:28:00] And I think it's a combination of the benchmarks, the leaderboards, the chatbot, because LMSys, remember, their ranking is not only based on benchmarks, it's also people playing with their arena stuff. People actually like humans, like, get two answers. I think they completely ignore benchmarks. Yeah, and then They only do ELO.[00:28:18] Oh, they do ELO completely, right? So that, for example, is just like people playing with both models and say, Hey, I prefer this one, I prefer that one. But also there's like some selection bias. The type of people who will go to LMCs to play with the models, they're a little bit specific in terms of like who they are.[00:28:33] It's very interesting. There's so many models. People are doing this in this way, that way. Some people are doing this for academic rigor only to test out new ideas. Some people are actually doing this like the Intel fine tunes of Mistral. Intel wanted to come out and show that their hardware approach is possible, Mistral, etc.[00:28:51] And it's really hard to know, like, what to pick, what to use. And especially on the bigger models, like you said, like the Llama 70B, the Falcon 180B. It's really because, like, who has the compute to validate those? So I would mention that, like, use with caution. Like, go and research and see if the biggest model that just released was actually worth the tokens and the money you spend on it.[00:29:12] To try and, if you're a business, to integrate it.[00:29:15] Distilling from GPT4[00:29:15] swyx: Since you said use of caution, I'll bring in one issue that has always been in the back of my mind whenever I look at the entire universe of open source AI models, which is that 95 percent of the data is derived from GPC 4, correct?[00:29:30] Which technically you can't use for commercial licenses,[00:29:34] Wing Lian: right?[00:29:35] swyx: What is the community's stance on this kind of stuff?[00:29:40] Wing Lian: I think from the community stance, like I feel like a lot of us are just experimenting, so for us, it's like, we're not going and building a product that we're trying to sell, right?[00:29:49] We're just building a product because we think it's interesting and we want to use it in our day to day lives, whether or not we try and integrate it. Personal use, yeah. Yeah, personal use, so like, as long as we're not selling it, yeah, it's fine. But[00:30:01] swyx: like, I as a company cannot just take OpenHermes and start serving[00:30:05] Alex Volkov: it and make money on it.[00:30:06] OpenHermes you can. Because the opening of OpenHermes, I think, is a clean up. That did after the regular Hermes, please folks, check your licenses before you listen to podcasts and say, Hey, I will tell you though, you could say the same thing about OpenAI. You could say the same thing kind of makes sense, where OpenAI or StabilityAI trains their diffusion model on a bunch of pictures on the internet, and then the court kind of doesn't strike down Sarah Silverman, I think, or somebody else, who came and said, hey, this has my work in it, because of the way how it processes, and the model eventually builds this knowledge into the model, and then it doesn't actually reproduce one to one what happened in the dataset.[00:30:45] You could claim the same thing for open source. Like, we're using And by we, I mean the, the open source community that I like happily report on uses GPT 4 to rank, for example, which is the better answer you, you, that's how you build one, one type of data set, right? Or DPO or something like this, you, you basically generate data set of like a question and four answers, for example, and then you go to GPT 4 and say, Hey, smartest model in the world right now, up to Gemini Ultra, that we should mention as well.[00:31:11] Which one of those choices is better? But the choices themselves are not necessarily written with GPT 4. Some of them may be, so there's like full syntactic datasets. But there's also, datasets are just ranked with GPT 4. But they're actually generated with a sillier model, or like the less important model.[00:31:25] The lines are very blurry as to what type of stuff is possible or not possible. And again, when you use this model that's up on Hug Face, the license says you can use this. OpenAI is not going to come after you, the user. If anything, OpenAI will try to say, hey, let's prevent this, this type of thing happening, and the brain, but I honestly don't think that they could know even, not that it makes it okay, it's just like, They also kind of do this with the Internet's archive, and also, I think that some of it is for use.[00:31:55] You use models to help you augment tasks, which is what GPT 4 lets you do.[00:32:00] swyx: Yeah, the worst thing that OpenAI can do is just kick you off OpenAI. That's because it's only enforced in the terms of service.[00:32:05] Alex Volkov: Sure, but just like to make sure, to clarify who they're going to kick out, they could kick out like News, for example, if news are abusing their service, a user of the open source, fully Apache 2 open source, for example, They won't get kicked out if they use both, just because they use both.[00:32:22] I don't believe so. I don't think OpenAI has a claim for that.[00:32:25] swyx: Well, we're not lawyers, but I just want to mention it for people to know it's an issue.[00:32:30] Wing Lian: And one of the things, like, I talked to someone recently, and I think that they also are like interested in it, but also to the point of like, right, if I use a model trained on data, using GPT for data, But I use that model to then regenerate new data.[00:32:46] Is that model, is that data okay? So like you start going down this whole rabbit hole. So yeah. All right.[00:32:53] swyx: Fantastic. Cool. Well, I think that's roughly highlights most of the open source universe. You also have your own models. Do you want to shout out any one of them? Yeah.[00:33:01] Wing Lian: I mean, I think like, I think Early on, Manicore got a lot of love.[00:33:04] I think it was mostly popular in, like, the roleplay communities. It was, it tended to be pretty truthful. It tended to be, like, have relatively good answers, depending on who you ask, right? But, I think for me, it was just, Releasing models was a way to try and, like, continue to build out the product, figure out what I needed to put into the product, how do I make it faster, and, if you've got to, like, go and debug your product, you may as well have it do something useful.[00:33:29] Awesome. So, yeah.[00:33:31] Finetuning - LoRA, QLoRA, ReLoRA, GPTQ[00:33:31] swyx: Okay, and then maybe we'll talk about just fine tuning techniques. So this is going to be a little bit more technical than just talking about model names and datasets. So we started off talking about LoRa, QLoRa. I just learned from your readme there's ReLoRa. Which I've never heard about.[00:33:45] Could you maybe talk about, like, just parameter efficient fine tuning that whole, that[00:33:50] Wing Lian: whole journey, like, what people should know. Yeah, so with parameter efficient fine tuning, I think the popular ones, again, being, let's, we'll start with lore, right? So, usually what you do is you freeze all the layers on your base, on the base model, and then you, at the same time, you sort of introduce additional Oh, this is tight.[00:34:08] No. You introduce, another set of layers over it, and then you train those, and it is done in a way that is mathematically possible, particularly with LORs that you can, then you, you, When you, when you train the model, you, you run your inputs through the base model, whose weights are frozen, but you, then you also run it through the additional weights, and then at the end you combine the weights, and then, and then, or you combine the weights to get your outputs, and then at the end, and when you're done training, you're left with this other set of weights, right, that are completely independent, and And then from that, what you can do is, some person smarter than I figured out, well, oh, they've done it in such a way that now I can merge these weights back into the original model without changing the architecture of the model, right?[00:35:03] So, so, that tends to be, like, the go to, and You're training much fewer parameters so that when you do that, yes, you still need to have all of the original weights, but you have a smaller gradient, you have a smaller optimizer state, and you're just training less weights, so you can tend to train those models on, like, much smaller GPUs.[00:35:27] swyx: Yeah. And it's roughly like, what I've seen, what I've seen out there is roughly like 1 percent the number of parameters that you're trading. Yeah, that sounds about right. Which is that much cheaper. So Axelotl supports full fine tune, LoRa, QLoRa,[00:35:40] Wing Lian: Q. Yes. So, so QLoRa is, is very similar to LoRa. The paper was, if I remember correctly, the paper was Rather, traditionally, most people who did Loras were, were, they were quant, they were putting the model weights in 8 bit, and then fine tune, parameter efficient fine tuning over the Lora weights, and then with QLora, they were quantizing all of those, they were then quantizing the weights down to 4 bit, right, and then I believe they were also training on all of the linear layers in the model.[00:36:15] And then with ReLore, that was an interesting paper, and then, I think, like, it got implemented. Some people in the community tried it, tried it out, and it showed that it didn't really have the impact that the paper indicated that it would. And from what I was told recently, that they re I guess they re released something for Relora, like, a few weeks ago, and that it's possibly better.[00:36:44] I personally haven't had the time. What was the[00:36:46] swyx: main difference,[00:36:47] Wing Lian: apart from quantization? I don't know. Okay. What was the main difference, sorry?[00:36:49] swyx: Apart from quantization, right? Like,[00:36:50] Wing Lian: Qlora's thing was, like, we'll just drop off some bits. With Relora, what they did was, you would go through, you would define some number of steps that you would train, like, your Lora with, or your Qlora.[00:37:01] Like, you could do Like, ReqLore, if you really wanted to, you would, you would train your LoRa for some number of steps, And then you would merge those weights into your base model, and then you would start over. So by starting, so, then by starting over, The optimizer has to find, like, sort of, re optimize again, and find what's the best direction to move in, and then do it all again, and then merge it in, do it all again, and theoretically, according to the paper, doing ReLore, you can do parameter efficient fine tuning, but still have sort of, like, the performance gains of doing a full fine tuning, so.[00:37:38] swyx: Yeah, and[00:37:39] Wing Lian: GPTQ? And GPTQ, so it's, I think with GPTQ, it's very similar to, more similar to QLore, where you're, it's mostly a quantization of the weights down to like 4 bit, where GPTQ is a very, is a specific methodology or implementation of quantization, so. Got it.[00:37:57] Alex Volkov: Wang, for, for folks who use Axolotl, your users, some people who maybe, Want to try it out?[00:38:03] And do they need to know the differences? Do they need to know the implementation details of QLora versus ReLora? Or is it okay for them to just know that Axolotl is the place that already integrated them? And if that's true, if that's all they need to know, how do they choose which method to use? Yeah,[00:38:22] Wing Lian: so I think like, I think most people aren't going to be using ReLora.[00:38:25] I think most people are going to be using either Lora or QLora. And I think they should have it. They should have an understanding of why they might want to use one over the other. Most people will say that with Qlora, the quality of the final model is not quite as good as like if you were to do a LoRa or a full fine tune, right?[00:38:44] Just because, you've quantized these down, so your accuracy is probably a little off, and so that by the time you've done the Qlora, you're not moving the weights how you would on a full fine tune with the full parameter weights.[00:38:56] Interesting.[00:38:57] swyx: Okay, cool. For people who are more interested, obviously, read the papers. I just wanted to give people, like, a high level overview of what these things are. And you've done people a service by making it easy for people to try it out. I'm going to, I'm going to also ask a question which I know to be wrong, but I'm curious because I get asked this all the time.[00:39:15] What is the difference between all these kinds of fine tunes[00:39:17] Wing Lian: and RLHF? Okay, between all of these sorts of fine tunes and RLHF. So all of these sorts of fine tunes are based, are, ideally, this, they are taking knowledge that the base model already knows about, and presenting it in a way to the model that you're having the model answer like, Use what it already knows to sort of answer in a particular way, whether it's, you're extracting general knowledge, a particular task, right?[00:39:44] Instruct, tune, chat, those sorts of things. And then generally with RLHF, so what is, let's go back, what is it? Reinforcement Learning with Human Feedback. So if we start with the human feedback part, What you're doing is you generally have, you have like a given prompt and then you, maybe you have one, maybe you have two, I think, like if you look at with Starling, you have like up to what, seven different, seven different possible responses, and you're sort of ranking those responses on, on some sort of metric, right, whether the metric is how much I, I might like that answer versus or I think with like starling is like how how how helpful was the answer how accurate was the answer how toxic was the answer those sorts of things on some sort of scale right and then using that to go back and like sort of Take a model and nudge it in the direction of giving that feedback, to be able to answer questions based on those preferences.[00:40:42] swyx: Yeah, so you can apply, and is it commutative? Can you apply fine tuning after and onto an RLHF model? Or should the RLHF apply, come in afterwards,[00:40:54] Wing Lian: after the fine tune? Um, I, yeah, I don't know that there's There's been enough research for one way or another, like, I don't know.[00:41:02] That's a question that's been asked on Discord. Yeah, like, I definitely would say I don't know the answer. Go and try it and report back to me and let me know so I can answer for the next guy.[00:41:10] swyx: It's shocking how much is still unknown about all these things. Well, I mean, that's what research is for, right?[00:41:16] Wing Lian: So actually I, I think I saw on the top of a leaderboard, it was a, it was a mytral base model, and they didn't actually fine tune it. They, or they, they just did RLH, they did like an RLHF fine tune on it using like, I don't, I don't recall which dataset, but it was like, and it benchmarked really well.[00:41:37] But yeah, you'd have to go and look at it. But, so it is interesting, like going back to that, it's like. Traditionally, most people will fine tune the model and then do like a DPO, PPO, some sort of reinforcement learning over that, but that particular model was, it seemed like they skipped like the supervised fine tuning or Scott.[00:41:55] Axolotl vs HF Transformers[00:41:55] swyx: Cool. One thing I did also want to comment about is the overall, like, landscape, competitive landscape, I don't know. Hugging Face Transformers, I think, has a PFT module.[00:42:05] Wing Lian: Yeah, yeah, the PEFT, the Parameter Efficient Fine Tuning, yep. Is that a competitor to you? No, no, so we actually use it. We're just a wrapper over sort of, sort of the HuggingFace stuff.[00:42:15] So, so that is their own sort of module where They have, taken the responsibility or yeah, the responsibility of like where you're doing these parameter efficient fine tuning methods and just sort of like, it is in that particular package where transformers is mostly responsible for sort of like the modeling code and, and the trainer, right.[00:42:35] And then sort of, there's an integration between the two and, there's like a variety of other fine tuning packages, I think like TRL, TRLX, that's the stability AI one. Yeah, I think TRL likes the stability, yeah, Carper, and TRL is a hugging face trainer. Even that one's just another wrapper over, over the transformers library and the path library, right?[00:43:00] But what we do is we have taken sort of those, yes, we've We also use that, but we also have more validation, right? So, there are some of us who have done enough fine tunes where like, Oh, this and this just don't go together, right? But most people don't know that, so like Example?[00:43:19] Like, people want to One and one doesn't go together. I don't have an example offhand, but if you turn this knob and this knob, right? You would think, all right, maybe this will work, but you don't know until you try. And then by the time you find out it doesn't work, it's like maybe five minutes later, it's failed.[00:43:34] It's failed in the middle of training or it's failed during the evaluation step. And you're like, ah, so we've, we've added a lot of, we've added a lot more validation in it. So that like, when you've, you've created your configuration, you run it through and now you say. The validation code says this is probably not right or probably not what you don't, not what you want.[00:43:52] So are you like a, you[00:43:53] swyx: do some linting of your YAML file?[00:43:56] Wing Lian: There, I guess you could call it linting, it's sort of like Is there a set of rules out[00:44:00] swyx: there somewhere? Yeah, there's a set of rules in there. That's amazing, you should write documentation like This rule is because, this user at this time, like, ran into this bug and that's what we invested in.[00:44:10] It's like a good collection[00:44:11] Wing Lian: of knowledge. Yeah, it is, and I guess like, if you really wanted to, like, figure it out, I guess you could, like, git blame everything, and But, yeah, it's, so, I think that's always a useful thing, it's like Because people want to experiment but they don't, people will get frustrated when you've experiment, you're experimenting and it breaks and you don't know why or you know why and you've just gone down the rabbit hole, right?[00:44:37] So, so I think that's one of the big features that's, that I think I find important because it's It prevents you from doing things you probably shouldn't have, and it, and sometimes we will let you do those things, but we'll try and warn, warn you that you've done that.[00:44:50] I[00:44:51] Alex Volkov: have a follow up question on this, actually, because yesterday we hung out to this open source event, and I spent time by you a couple times, like when people told you, oh, XLR, I use XLR, it's super cool, and then the first thing you asked is, like, immediately, like, what can we improve?[00:45:04] And yes, from multiple folks, and I think we talked about this a little bit, where there's It's a developer tool. It's like a machine learning slash developer tool. Your purpose in this is to help and keep people, as much as possible, like, Hey, here's the best set of things that you can use right now. The bear libraries are, or the bear trainer, for example, is a bear trainer.[00:45:28] And also, maybe we should talk about how fast you're implementing these things. So you mentioned the first implementation took a week or so. Now there's a core maintainer group, right? There's like, features are landing, like Qlora, for example. Neftune, I don't know if that's one example of something that people potentially said that it's going to be cool, and then eventually, like, one of those things that didn't really shake out, like, people quickly tested this out.[00:45:48] So, there's a ton of Wait, Neftune is cancelled? I don't know if it's fully canceled, but based on vibes, I heard that it's not that great. So like, but the whole point that I'm trying to make with Neftune as well is that being existing in the community of like XLR or like, I don't know, even following the, the GitHub options or following the Discord, it's a fairly good way to like, learn these, Kind of gut feelings that you just, you just said, right?[00:46:14] Like where this, maybe this knob, that knob doesn't work. Some of these are not written down. Some of these are like tribal knowledge that passes from place to place. Axel is like a great collection of many of them. And so, do you get That back also from community of folks who just use, like, how do you know who uses this?[00:46:30] I think that's still an issue, like, knowing if they trained with XLR or should they add this to things? Talk about, how do you get feedback and how else you should get feedback?[00:46:38] Wing Lian: Yeah, I mean, most of the feedback comes from the Discord, so people come in and , they don't get a training running, they run into, like, obscure errors or, errors that That's a lot of things that maybe, maybe as a product we could catch, but like, there's a lot of things that at some point we need to go and do and it's just on the list somewhere.[00:46:58] Right that's why when people come up, I'm like, what, what were your pain points? Because like, as a developer tool, if you're not happy with it, or you come in and in the first, Takes you 30 minutes and you're still not happy. You leave the tool and you may, you might move on maybe to a better tool, maybe to, one with less frustration, but it may not be as good, right?[00:47:17] So I'm trying to like, figure out, all right, how can I reduce all this frustration? Because like for me, I use it every day for the most part, right? And so I am blind to that, right? Mm-Hmm. . Mm-Hmm. . I just know, I, I go do this, this, and this. It pretty much mostly works, right? But, so I don't have sort of that, alright, that learning curve that other people are seeing and don't understand their pain points.[00:47:40] Yeah,[00:47:40] Alex Volkov: you don't have the The ability to onboard yourself as a new user completely new to the whole paradigm to like get into the doors of like, Oh, no, I don't even know how to like ask about this problem or error.[00:47:53] swyx: Cool. The last few things I wanted to cover was also just the more advanced stuff that you covered yesterday.[00:48:00] 20x efficiency with StackLlama and Multipack[00:48:00] swyx: So I'll just, caution this as like, yeah, this is more advanced. But you mentioned Stackllama and Multipack. What are they[00:48:06] Wing Lian: and what should people know? Yeah, so, so, Stack Llama was, that paper came out, so Stack Llama I think was like, two, two, two separate, two separate concepts that they announced, so the first one was They being hugging face.[00:48:20] Yeah, sorry, yes, they being hugging face, so the first one being sort of like, this idea of packing, like some packing sequences together, so like, if we think about training data, right, your training data is, let's say, to keep the math easy, let's say your training data is 500, We, we, we, we will use the terminology words.[00:48:39] Let's say your training data is 500 words long, and let's say your, your context length, you know how much data your, that your model can accept is like, or that you want feed into your model. It's, let's say, we won't use tokens again, we'll we'll use it is it's 4,000 tokens, right? So if you're training at 4K Con or four 4,000 4K contacts and you're only using 500 of it, you're sitting like with the other 1500.[00:49:05] 3, 500 words that you're not using, right? And typically that's either filled with these PAD tokens, so I think I made the analogy last night that it's like having sort of like a glass here you fill it up with a shot of liquor and then you're and that's your training data and then you just fill it up with more water and those are your PAD tokens and it's just, it doesn't do much, right?[00:49:27] It's still the same thing, but you still have to go through all of that to go through all your training data. And then, so what Stack Llama showed was you could just sort of take your training data, append the next row of training data until you filled that entire 4k context, so in this example, right, with 500 words to 4k, that's 8 rows of training data.[00:49:48] But, the problem with that is, is that with a lot of these transformer models, they're very much relying on attention, right? So, like, if you now have this sequence of words that now, in order for the, the model has seen all of these other words before, right? And then it sees another set of words, another set of words, but it's learning everything in context of all the words that it's seen before.[00:50:13] We haven't corrected the attention for that. And just real quickly, since I said that that paper was two concepts, the other one was, I believe it was like a reinforcement learning, but outside the scope of this. So going from that, I implemented that early on because I was like, Oh, wow, this is really great.[00:50:29] And. Yes, because it saves you a bunch of time, but the trade off is a little bit of accuracy, ultimately, but it still did pretty well. I think when I did Manicore, I think it used sort of that concept from Stack Llama of just sort of appending these sequences together, right? And then sort of the next evolution of that is Multipack, right?[00:50:51] So, there was a separate paper on that, it was, I believe it was referenced, it got referenced in the Orca paper, where you could, you could properly mask those out using like a, I think it was like a lower block triangular attention mask, and then sort of, so, So, there's that. I did try implementing that, manually recreating that mask, but then one from the OpenChat, so he was helping with OpenOrca as well, and he had done an implementation of Multipack, and where he used FlashAttention, so FlashAttention So that was released by TreeDAO, and it was this huge performance gain.[00:51:35] Everybody uses it now, even the Transformers library now, they've taken all of these, like, people are taking all of these models and sort of like, making it compatible with FlashAttention. But in Flash Tension, there is one particular implementation that lets you say, Well, I'm sending you all of these sequences like you would in Stack Llama, But let me send you another, another, Set of information about, this is where this set of sequences is, this is where the second set of sequences is.[00:52:06] So like, if it was like, 500 words long, and you stacked them all together, you would just send it a row of information that was like, 0, 500, 1000, 1500, etc, etc, out to 4000. And it would know, alright, I need to break this up, and then run the forward pass with it. And then it would be able to, and it was much more, much more performant.[00:52:29] And I think you end up seeing like 10x, 20x improvements over sort of, I mean, I think FlashAttention was like a 2x improvement, and then adding that with the Multipack, you start to see like, depending on, how much data you have, up to like a 20x improvement sometimes. 20x. 20x. Wow. Yeah.[00:52:48] And I only know the 20x because I, like, before last night, I was like, I re ran the alpaca, I looked up the alpaca paper because it was like, I just need a frame of reference where somebody did it, and I think they used eight A100s for three hours, and they said it cost them 100. I don't, I don't think eight A100s cost, I don't know how much it costs right now.[00:53:14] But I ended up rerunning it. Usually a dollar an hour, right? Yeah, so eight. The cheapest is like a[00:53:18] Alex Volkov: dollar, a dollar an hour for one.[00:53:20] Wing Lian: Yeah, so that's still like 24, 25. But maybe if you're going on Azure, maybe it's like, maybe it's 100 on Azure. I mean, it used to be more expensive, like, a year ago.[00:53:31] Yeah, and then, so I re ran it with sort of like, I turned on all of the optimizations just to see what it would be. And like, and usually Multipack is the biggest optimization, so Multipack with Flash Detention. And it, I think I spun it up on 8 L40s, and it ran, and I didn't let it run all the way through, I just grabbed the time, the estimated completion time, and it was like 30 minutes, so it would have cost like 4 or 5 to run the entire, like, reproduce the alpaca paper, right?[00:54:00] Which is crazy. It's crazy. 20x,[00:54:02] Alex Volkov: yeah. I want to ask about, like, you said you turned on all the optimization. Is that the yaml file with xlodl, you just go and like check off, like, I want this, I want that? Yeah, yeah,[00:54:10] Wing Lian: so there's like one particular yaml file in there, That, there's one particular YAML file in there that's like, it's under examples, llama2, fft, optimize.[00:54:20] So, I think someone had created one where they just turned, they put in all of the optimizations and turned them on. I mean, it actually, it does run, which is like, sort of surprising sometimes, because sometimes, you optimize this, optimize this, and sometimes they just don't work together, but, yeah.[00:54:36] Just turn the knobs on, and like, fine tuning should really just be that easy, right? I just want to flip the knob and move on with my life and not figure out how to implement it.[00:54:47] Tri Dao and Mamba[00:54:47] Alex Volkov: Specifically, the guy behind FlashAttention came up with something new. You want to talk about this a little bit? You want to briefly cover Mamba?[00:54:53] Yeah, let's talk about Mamba. Let's talk about Mamba. So, what is Mamba?[00:54:57] Wing Lian: Oh, gosh. I

We're diving back into the binge of Harry Potter and the Prisoner of Azkaban with chapter 6, Talons and Tealeaves. For this discussion I'm joined by Dan from Hogwarts: A Podcast (https://podcasters.spotify.com/pod/show/hogwarts-a-podcast). This is the chapter where Trelawney predicts Harry's death in Divination class because she sees a big black dog aka the Grim in a teacup. Then Hagrid makes Harry ride a Hippogriff named Buckbeak after he successfully pets him in Care of Magical Creatures. Unfortunately, Draco Malfoy isn't a fan of Harry getting attention or in paying attention to Hagrid's warnings. So, he insults Buckbeak and gets clawed up as a result of his actions. Throughout the discussion we'll rate Draco's ability as a school bully. Dan will defend Trelawney's abilities as a teacher and seer. We'll talk about Hagrid's qualification for teaching, Dumbledore's loose ship, how there must be good Slytherins we don't see and a lot of other fun stuff! Find Dan on Twitter or X: https://twitter.com/HogwartsAPod and Instagram: https://www.instagram.com/hogwartsapod/ For bonus episodes as well as many other benefits such as early access, show prep notes, personal shoutouts, and more BECOME A PATRON: https://www.patreon.com/BelatedBinge *Free trial available I'd also love for you to join the discussion, which you can do in 2 key ways: Leaving a voicemail on my website: https://www.belatedbinge.com/ Joining the new free Facebook Group for listeners of the show: https://www.facebook.com/groups/688037702835574/ Find Belated Binge on social media: Twitter: https://twitter.com/belatedbinge Instagram: https://instagram.com/belatedbinge Facebook: https://www.facebook.com/belatedbinge TikTok: www.tiktok.com/@belatedbinge YouTube: https://youtube.com/@belatedbinge Show theme song: 'Worth the Wait' by Zac S. Eagle: Spotify: https://open.spotify.com/track/6feyaCsf8OxQY8cumcQtox?si=9df77d0dee0742ab Apple: https://music.apple.com/us/album/worth-the-wait-single/1627714792 Thanks for listening!

We are deep into spooky and Swiftie season. Thanks to this week's Patreon supporter, Jemima! Happy birthday to Ann's dad too! Phoenix Register: MinaLima's pop-up House of MinaLima is open in Paris from October 27 to March 9! To celebrate the 25th anniversary of Harry Potter, and the release of MinaLima's Prisoner of Azkaban, the Franklin Roosevelt station in Paris has been decorated with huge Harry Potter illustrations. Check out the La Gazette article here. No owl post ☹ Tell us why your favorite public transit system should have a stop named after Newt Scamander! Main Discussion: theme parks! Amy needs immersion in her theme park experiences. Could we be getting a Wizarding World in Europe? More information on Universal's possible acquisition of PortAventura here. (Article by Shannen!) How many of us are roller coaster people? (Spoiler: none of us) Watch a Hippogriff lesson at Universal Osaka here. Shannen tells the story of a sneaky, magical wedding. Brief spoilers for Hagrid's Motorbike Adventures at 00:25:19 - 00:25:30. Pro-tip: ride the Hogwarts Express both ways! Watch Celestina Warbeck's Christmas show here. Epic Universe discussion, including pretty major spoilers, runs from 00:46:45 – 01:10:45. Specific ride spoilers from 00:53:20 – 1:00:51. Skip if you want to go into Epic Universe unspoiled! We get to armchair Imagineer! SpeakBeasty meetup in 2025! Newtcase: vampires! “He was born in 1776, as all Americans were.” Some owls have a scientific name of strigoi! We discuss our first vampires. Podcast Question: What do you expect from the Circus Arcanus show in Epic Universe?

The episode seems to not have pushed to Spotify so I'm reuploading it with a different title. Jackie Hatton joins me.Support Our Patreon!: https://patreon.com/harrypotterandtheboysSee omnystudio.com/listener for privacy information. --- Send in a voice message: https://podcasters.spotify.com/pod/show/harrypotterandtheboys/message

The quest to save little Abe takes a turn for the worse as Roni the Hippogriff makes an unexpected stop in a distant land.

Our favorite adventurers pull out all the stops in order to woo a Hippogriff for Heather Honeypot. Season 2 Pathfinder 1E Check out our Patreon! patreon.com/bugbearhigh

Hugo, a rather ordinary horse, is invited to meet up with the world's most extraordinary horses. Has there been a mistake? Written especially for this podcast by Alice. If you enjoyed this story, please do leave us a review. And, if you'd like to suggest an animal for a future Animal Tales story, you can do so by emailing podcast@animaltales.uk. We would love to hear from you. Become a PREMIUM Subscriber You can now enjoy Animal Tales by becoming a Premium Subscriber. This gets you: All episodes in our catalogue advert freeBonus Premium-only episodes (minimum of one per month) which will never be used on the main podcastWe guarantee to use one of your animal suggestions in a storyYou can sign up through Apple Podcasts or through Supercast and there are both monthly and yearly plans available. Discover a brand new story every Monday, Wednesday and Friday – just for you! You can find more Animal Tales at https://www.spreaker.com/show/animal-tales-the-kids-story-podcast

The third novel in the series, and the one that, like a Hippogriff, really spreads its wings and starts to fly.From the mass murderer trying to break into Hogwarts, to Harry noticing the pretty girl on the Quidditch team, the trio certainly are growing up.Today's root beer was not root beer, but Flying Cauldron butterscotch beer.Intro and Outro music by Stockmusic331 on Pond5

On episode 86 of the Sofa Kingdom Podcast Talk about the oncoming AI Revolution, Dune 2 Guardians of the Galaxy 3, Gran Turismo, Ultimate Alliance and Diablo 4,   Find more Sofa Kingdom Podcast at SofaKingdomOfc on instagram https://www.instagram.com/sofakingdomofc/ Audio podcast is on all major podcast platforms just search "Sofa Kingdom Podcast"     General topics this episode Wes Anderson Star Wars Galactic Menagerie, Star Wars visions, AI take over art, Dune 2, Joe Rogan Podcast, skier stuck in the snow, Guardians of the Galaxy vol 3, Tiny Tina's Wonderland, Gran Turismo, Chrono Odyssey, Marvel's Ultimate Alliance, Diablo 4, modding games,

Weer een volle aflevering met Universal-nieuws! Van Universal Studios Hollywood tot Universal Studios Japan komen aan bod in de Upper Lot Podcast

My True Love Gave to Me by notebookandink --- Send in a voice message: https://podcasters.spotify.com/pod/show/sep-fanfic-readings/message

My True Love Gave to Me by notebookandink --- Send in a voice message: https://podcasters.spotify.com/pod/show/sep-fanfic-readings/message

You're a podcaster, Ryan! I'mma, I'mma wHAT? You're a podcaster! And you are watching The Bells 'N Whistles Podcast baby! This week we try once again to find something to interest the Youtube Algorithm, so we take a little looksie at the creatures and spells of Hogwarts Legacy! SUPPORT US BY BUYING MERCH: https://my-store-7405526.creator-spring.com/listing/bellsnwhistles Please leave 5 stars and a review on Apple Podcasts and follow us on Spotify! https://rss.com/podcasts/bnwpod/ Subscribe to Bellhop Productions on YouTube: https://www.youtube.com/channel/UCC1dr278XrRkoZReO4QoBiA @BellhopProductions on IG and TikTok Check out our brand new channel 3 Ways to Survive: https://www.youtube.com/channel/UCfBAtsmIKSfute9zVzRjfwA For video game content, subscribe to Another F*cking Let's Play: https://www.youtube.com/channel/UCu1-XrrL5OT5MSO7LbhF_ZQ Love you! TIMECODES: 00:00-07:20: Intro & Nonsense 07:21-10:34: Set-Up 10:35-13:39: Hippogriff and Baby Hippogriff 13:40-15:10: Graphorn 15:11-16:28: Phoenix 16:29-17:47: Mooncalf 17:48-19:09: Giant Purple Toad and Diricawl 19:10-21:17: Kneazle and Fwooper 21:18-24:06: Niffler "Nitler" and Puffskein 24:07-24:50: Jabberknoll 24:51-27:45: Thestral 27:46-29:22: Unicorn 29:23-32:34: Deep Fakes and Internet Celebs 32:35-52:40: Spell Review 52:41-58:07: Harry Potter in the Real World 58:09-1:04:02: Wrap Up

Harry and Draco are competing in the annual Christmas Bake-off for the orphanage run by Sirius and Remus. After getting stuck behind the wards for the night, they come to some interesting conclusions. --- Send in a voice message: https://anchor.fm/etl-echo/message

Join Ellen and Keighty as we continue with Harry Potter and the Order of the Phoenix, covering the second half of Chapter Twenty-Three (Christmas on the Closed Ward) and the corresponding film scenes that slightly correspond with a much earlier chapter, but not really. While we love Harry Potter, we do not condone or agree with its creator's ignorant opinions. We stand with the trans community.  We post weekly podcast episodes comparing and contrasting the Harry Potter books to the movies, Potter Ponderings, a weekly trivia question (that can win you a sticker!), Sorting Hat Stories, and other Harry Potter related fun! Please subscribe and join us as we delve into our favorite book series and the films that brought them to life.  Follow us on Podbean: https://fawkessakepod.podbean.com/ to get the episodes as early as possible and get a leg up on answering the trivia question! Check out our website at ForFawkesSakePodcast.com for all of our latest news, merchandise, and more!  Find us at the handles below: Facebook, Instagram, Twitter, and TikTok: @FawkesSakePod Join us on TikTok to stitch Potter Pondering responses and see other random videos! Subscribe to us on YouTube to get access to our weekly episodes, vlogs, other random videos, and cooking show episodes. If you would like to share your own Sorting Hat Story with us to read on a future episode or have any other questions, email us at ForFawkesSakePodcast@gmail.com. Don't forget to subscribe so you can get the episodes sooner! If you have Apple, please Rate and Review us! (send us an email to let us know you did and we will also send you a sticker!) If you don't have Apple, you can leave us a recommendation on our Facebook page to get a sticker. We also have a Patreon Program. Become a patron for extra podcast perks, including access to Potterheads, A History, giveaways, a special Facebook group for patrons only, and more! Help us reach our monthly goals so we can create more content, t-shirts, hand-thrown mugs, other merchandise, and much more! Check it out here: https://www.patreon.com/fawkessakepod. As always, any support you can give is greatly appreciated!

Musique Mécanique par le Théâtre Électrique ::"Flight of the Hippogriff" by John Williams

David, Spencer and Abby continue our read-through of Chamber of Secrets! Before that, they address the Hippogriff in the room of Alan Rickman's Diary reveal, and other shenanigans. Enjoy!

 Chapter 18 - Moony, Wormtail, Padfoot, and Prongs“They've — got — a — right — to — know — everything.” Lupin panted, still trying to restrain Black. “Ron's kept him as a pet! There are parts of it even I don't understand! And Harry — you owe Harry the truth, Sirius!” Black stopped struggling, though his hollowed eyes were still fixed on Scabbers, who was clamped tightly under Ron's bitten, scratched and bleeding hands. “All right, then,” said Black, without taking his eyes off the rat. “Tell them whatever you like. But make it quick, Remus. I want to commit the murder I was imprisoned for…”Q1 - Who is right here? Lupin for waiting, or Black for wanting to kill Peter first?Q2 - Why didn't Scabbers try to escape earlier on?“Because ... because people would know if Peter Pettigrew had been an Animagus. We did Animagi in class with Professor McGonagall. And I looked them up when I did my homework - the Ministry keeps tabs on witches and wizards who can become animals; there's a register showing what animal they become, and their markings and things…and I went and looked Professor Mcgonagall up on the register, and there have only been seven Animagi this century, and pettigrews name wasn't on the list.”Q3 - I ask again, what would you be if you could be an Animagus?“It seemed impossible that I would be able to come to Hogwarts. Other parents weren't likely to want their children exposed to me. But then Dumbledore became Headmaster and he was sympathetic.”REDDIT - alphaWLFgang1 - So if we were looking at it from that perspective then I'd be a Nebelung Cat (according to Wizarding World) But, I mean if I had to choose I'd want to be something that could fly. So like an Eagle or maybe a Hippogriff would be sick too if that's even possible? I'd want to fly or swim! So maybe a shark? I'd want to be an animal that can either fly/swim under water. I'd want to fly so I can travel faster places and swim because I love the water.Q4 - Was Dumbledore right to let Lupin come to school?Q5 - Why does Hogwarts have such a rag-tag bunch of teachers?“Now, my three friends could hardly fail to notice that I disappeared once a month. I made up all sorts of stories. I told them my mother was ill, and that I had to go home to see her…I was terrified they would desert me the moment they found out what I was. But of course, they, like you, Hermione, worked out the truth and they didn't desert me at all. Instead they did something for me that would make my transformations the best times of my life. They became animagi.” Q6 - Do you know and understand how truly difficult it is to become an Animagus?Q7 - What was the craziest/most dangerous thing you've done in your life?Q8 - Have you ever put someone else's life in danger?“Sirius thought it would be — er — amusing, to tell Snape all he had to do was prod the knot on the tree-trunk with a long stick, and he'd be able to get in after me. Well, of course, Snape tried it — if he'd got as far as this house, he'd have met a fully grown werewolf — but your father, who'd heard what Sirius had done, went after Snape and pulled him back, at great risk to his life.”Q9 - Do you have any sympathy for Snape?Chapter 19 - The Servant of Lord Voldemort“YOU'RE PATHETIC!” Harry yelled. “JUST BECAUSE THEY MADE A FOOL OF YOU AT SCHOOL YOU WON'T EVEN LISTEN _” “SILENCE! I WILL NOT BE SPOKEN TO LIKE THAT!' Snape shrieked, looking madder than ever. "Like father, like son, Potter! I have just saved your neck, you should be thanking me on bended knee! You would have been well served if he'd killed you! You'd have died like your father, too arrogant to believe you might be mistaken in Black. Now get out of the way or I will make you.”Q1 - What do you think of both Harry and Snape's statements?Q2 - Were they right to attack a teacher?"Harry I as good as killed them; he croaked. I persuaded Lily and James to change to Peter at the last moment, persuaded them to use him as Secret Keeper instead of me ... I'm to blame, I know it ... the night they died, I'd arranged to check on Peter, make sure he was still safe, but when I arrived at his hiding place, he'd gone. Yet there was no sign of a struggle. It didn't feel right. I was scared. I set out for your parents' house straight away. And when I saw their house, destroyed, and their bodies - I realized what Peter must have done. What I'd done. His voice broke. He turned away.”Q3 - What do you think of this whole story so far?Q4 - Do you get the secret keeper thing still?Harry remembered what Mr Weasley had told Mrs Weasley. “The guards say he's been talking in his sleep …. always the same words... "He's at Hogwarts.” “It was as if someone had lit a fire in my head, and the Dementors couldn't destroy it…it wasn't a happy feeling…it was an obsession…but it gave me strength, it cleared my mind. So, one night when they opened my door to bring food, I slipped past them as a dog…it's so much harder for them to sense animal emotions that they were confused…I was thin, very thin…thin enough to slip through the bars…I swam as a dog back to the mainland I journeyed north and slipped into the Hogwarts grounds as a dog…I've been living in the Forest ever since…except when I come to watch the Quidditch, of course…you fly as well as your father did, Harry…He looked at Harry who did not look away. Believe me croaked black believe me. I never betrayed James and lily. I would have died before I betrayed them. And at long last, Harry believed him. Throat too tight to speak, he nodded. Q5 - Is Ron nuts for letting Scabbers sleep in his bed?Q6 - Why would Harry change his murderous tone after learning of all this, is he right to show mercy to Pettigrew?Q7 - What are your thoughts on Pettigrew, Lupin, James, and Sirius now?Chapter 20 - The Dementor's Kiss“But I'm also — I don't know if anyone's ever told you — I'm your godfather.” “Yeah, I knew that,” said Harry. “Well…your parents appointed me your guardian,” said Sirius stiffly. “If anything happened to them…” Harry waited. Did Sirius mean what he thought he meant? “I'll understand, of course, if you want to stay with your aunt and uncle,” said Sirius. “But…well…think about it. Once my name's cleared…if you wanted a…a different home…”Q1 - Is it at all crazy that Harry now is okay with living with Sirius?Q2 - Do you think it's feasible that someone like Lupin forgot to take his potion?Q3 - What did you think Harry's patronus was on finishing this chapter?Q4 - Who did you think cast the patronus on your first read?

Chapter 6 - Talons and Tea Leaves“Dad had to go out to Azkaban one time, remember, Fred? And he said it was the worst place he'd ever been. He came back all weak and shaking…They suck the happiness out of a place, Dementors. Most of the prisoners go mad in there.”Q1 - What do you think of Dementors sucking the happiness out of a place?“How're you supposed to be in three classes at once?” “Don't be silly,” said Hermione shortly. “Of course, I won't be in three classes at once.”Q2 - Do you think Hermione could handle the workload of three classes at once?Q3 - Would you want to take a Divination class?PREDICTION #1 - Neville's Grandma is Ill.PREDICTION #2 - Parvati must beware a red-haired man.PREDICTION #3 - February will bring the flu…she will lose her voice.PREDICTION #4 - By Easter one of them will leave forever.PREDICTION #5 - Lavender's dread will happen Friday the sixteenth of October.PREDICTION #6 - Neville breaking the pink cup. HAPPENEDQ4 - What are your thoughts on Trelawney?When Harry and Ron had had their teacups filled, they went back to their table and tried to drink the scalding tea quickly.Q5 - Can you drink scalding hot tea/coffee?“Right, you've got a wonky sort of cross,” he said, consulting Unfogging the Future. “That means you're going to have ‘trials and suffering' — sorry about that — but there's a thing that could be the sun. Hang on, that means ‘great happiness' .. so you're going to suffer but be very happy…”PREDICTION #7 - Harry has a deadly enemy.PREDICTION #8 - The club…an attack.PREDICTION #9 - The skull…danger in his path.PREDICTION #10 - The grim…an omen of death.“You'll forgive me for saying so, my dear, but I perceive very little aura around you. Very little receptivity to the resonances of the future.”PREDICTION #11 - Neville will be late to the next class.He hardly heard what Professor McGonagall was telling them about Animagi (wizards who could transform at will into animals), and wasn't even watching when she transformed herself in front of their eyes into a tabby cat with spectacle markings around her eyes…”Really what's gotten into you all today? … Not that it matters, but that's the first time my transformation's not gotten applause from a class.”“There is no need to tell me any more, Miss Granger. Tell me, which of you will be dying this year?”“I shall not conceal from you that I have very little patience with it [Divination]. True Seers are very rare, and Professor Trelawney…”Q6 - Thoughts on McGonagall's thoughts on Trelawney?Q7 - What do you think about Animagi?Q8 - If you were part of the Magical World, would you be more like Ron or Hermione with superstitions and grims?Q9 - Should Hagrid have left a note with the students for how they should open their books?Q10 - Would you want to ride a broomstick or a Hippogriff?“No one?” said Hagrid with a pleading look. “I'll do it,” said Harry.Q11 - Is Hagrid a good teacher?Q12 - Is Sirius Black that serious that Hagrid is upset at the trio for visiting him in the dark?Chapter 7 - The Boggart in the WardrobeQ1 - Have you ever milked an injury?Q2 - Would you be good at potions?“Have you heard? Daily Prophet this morning — they reckon Sirius Black's been sighted…Not too far from here.”Q3 - Do you think the Ministry of Magic should have someone in the 911 Dispatch office? Or do you think they can bug certain words…like Sirius Black?“Of course, if it was me,” he said quietly, “I'd have done something before now. I wouldn't be staying in school like a good boy, I'd be out there looking for him.” “What are you talking about, Malfoy?” said Ron roughly. “Don't you know, Potter?”Q4 - What doesn't Harry know that Malfoy knows about Sirius Black?Q5 - What is wrong with Snape to be testing on Neville's toad?Q6 - How do you think Hermione is taking all these classes?“Good afternoon,” he said, “Would you please put all your books back in your bags. Today will be a practical lesson. You will only need your wands.”…”when their old teacher brought pixies…” LONG LIVE LOCKHART!“I was hoping that Neville would assist me with the first stage of the operation,” he said, “and I am sure he will perform it admirably.”Q7 - What are your first thoughts on Lupin?Q8 - What would the Boggart take shape if you confronted it? (aka, what frightens you?)“It's always best to have company when you're dealing with a Boggart.”“I would like all of you to take a moment now to think of the thing that scares you most, and imagine how you might force it to look comical?”Q9 - How would you change your biggest fear into something humorous?Q10 - What is Lupin afraid of?“What would it have been for you?” said Ron, sniggering. “A piece of homework that only got nine out of ten?”

Reading: The Piebald Hippogriff by Karen Anderson Location: Ohio Weather: End of Summer Rains --- Support this podcast: https://podcasters.spotify.com/pod/show/stormy-weather-stories/support

Welcome to Harry Potter Theory. In today's video, we're going to be discussing wizarding transportation. Not via the floo network, not via apparition, and certainly not by Hippogriff - but by broomstick. Of course- in the muggle world a broomstick has a very different function, but to a witch or wizard- they see an opportunity to fly- a means of getting from point A to point B. But they're not only used in every day life- they're also used in sport- notably Quidditch. Without broomsticks the game of quidditch would cease to exist, and wouldn't that be awful. Brooms are brought to the forefront of our minds right near the beginning of the first book/film, when Hagrid first brings Harry to Diagon Alley. In complete awe of everything around him, Harry pauses for a moment as he walks past 'Quality Quidditch Supplies', Diagon Alley's premier broom shop. It's also in this moment that he overhears other young witches and wizards going berzerk over a broom in the window: "Wow! Look at it! The new Nimbus 2000!" And from that moment Harry was intrigued.Socials: https://linktr.ee/hptheory Learn more about your ad choices. Visit megaphone.fm/adchoices

在这个故事里，我们学8个单词1．chancellor 财政大臣2．prime 主要的3．minister 部长 大臣4．cathedral 大教堂5．Hippogriff 鹰头马身有翅膀怪兽6．Pebbly 多卵石的 有卵石花纹的7．quirky 离奇的 古怪的8．enchanting 迷人的 妩媚的9、Griffin 希腊神话中半狮半鹫的怪兽

On today's IGN The Fix: Games, Presented by GameStop, we're taking a look at new Hogwarts Legacy screenshots and a never-before-seen cinematic. The cinematic shows a group of dark wizards trying to steal a Hippogriff, while the screenshots showcase some of the character customization options coming to the game. If you really wanted to live out your Harry Potter dreams and attend the School of Witchcraft and Wizardry, this game's your best option. The hit platform fighter MultiVersus has been hit with another delay, this time the start of Season 1 is being pushed back, along with the debut of Morty from Rick & Morty. Sorry, no new date for Season 1 has been revealed. And today we're taking a look at the pre-built gaming PC options offered by GameStop.

Need a mount or to hunt down a low-level monster? Try a hippogriff. You might even find yourself a new mount if you are okay stealing a child. Is your rogue too noticeable at the fancy dinner parties? Try the Ring of Negative Charisma. Monster: hippogriff, MM pg 184Magic Item: Ring of Negative CharismaJust the Tip: Fleshing out a magic itemSocial Media:WebsiteInstagramTwitterEdited and Produced by Zach W.Music: "Unholy Knight (Intro) and Realizer (Outro)" Kevin MacLeod (incompetech.com)Licensed under Creative Commons: By Attribution 4.0 License. http://creativecommons.org/licenses/by/4.0/Logo designed by Graphite the Dragon See acast.com/privacy for privacy and opt-out information.

One gotta go Harry Potter magical creatures. Fox the Phoenix vs Fluffy the 3 headed dog vs Buckbeak the Hippogriff vs Aragog the Spider. Gerald had covid last week. The Soccer Champions Tour 2022. Real Madrid vs FC Barcelona and Juventus vs Chivas in Vegas. Gerald's Las Vegas Vacation. Klondike's Choco Taco discontinuad after 40 years. The Hypothetical Moment. --- Support this podcast: https://anchor.fm/theuneducatedduopodcast/support

This week we cover all types of weird mythical horses. Everything from the Uchchaihshravas to the Hippogriff. Which steed would you like to tame?

This week we get into another amazing chimera, the Hippogriff! This half eagle, half horse magical beast has humble origins in our world, but a unique history in both our culture and D&D. Reading List https://www.encounterthis.ca/blog Socials Patreon Twitter Instagram Facebook Sponsor Strange Fellows Brewing, a Vancouver based brewery whose Blackmail Stout might be single handily responsible for the formation of our show

In this comeback episode, Adelia and Colin encounter a visit to the hospital, and a reunion with two old friends. Thank you all for listening and be sure to head over to instagram (@ronandhermioneminustheromance), twitter (@ron_hermionemtr), email (vulchanovbones@gmail.com), or Spotify, and Apple Podcasts to leave a review! Also, head over to our website (ronandhermionemtr.buzzsprout.com) to check out our bios and share our podcast with others!

Lexman tells the true story of how he was attacked by a hippogriff and how he managed to use its natural instinct to his advantage.

Classic monsters, plus some thoughts on the same.  If you're enjoying the show, why not consider supporting it on Patreon? You'll get access to lots of new bonus content, including my other podcast, Patron Deities! The Pamphlet of Pantheons is now available for order at Rook's Press and Four Rogues Trading Company! Or buy a digital copy at Itch or Drivethru. Thanks to Ray Otus for our thumbnail image. The intro music is a clip from "Solve the Damn Mystery" by Jesse Spillane, used under a Creative Commons Attribution License.

In this episode of Discover Universal, Kari and David immerse themselves in The Wizarding World of Harry Potter at Universal Orlando Resort. Hold on tight as you ride along with them on Hagrid's Magical Creatures Motorbike Adventure and listen to Kari discover the magic of the wand experience at Ollivander's. Hear details about both of these immersive lands: Hogsmeade at Universal's Islands of Adventure and Diagon Alley at Universal Studios Florida, each with their own unique attractions to experience, environments to explore, treats to enjoy, and more. Whether you're a fan or a first time visitor to The Wizarding World, you'll learn some planning tips to enjoy everything these lands have to offer. Additional Content: Explore The Wizarding World of Harry Potter: https://www.universalorlando.com/web/en/us/universal-orlando-resort/the-wizarding-world-of-harry-potter/hub 360 VIDEO: Hagrid's Magical Creatures Motorbike Adventure | The Wizarding World of Harry Potter:https://www.youtube.com/watch?v=-PO9x-NEak8&t=1s Hagrid's Magical Creatures Motorbike Adventure Front Row POV Experience:https://www.youtube.com/watch?v=TdGbsggU2dg FULL Queue Walkthrough of Hagrid's Magical Creatures Motorbike Adventure: https://www.youtube.com/watch?v=d-PkTOPKGfg&t=4s Harry Potter and the Escape From Gringotts Hype Video: https://www.youtube.com/watch?v=y_KkH48UCFY The Wizarding World of Harry Potter Complete Guide:https://blog.discoveruniversal.com/guides-and-tips/ultimate-guide-wizarding-world-harry-potter-universal-orlando-resort/ Your Guide to the Details Fans Will Appreciate About Gringotts Bank in The Wizarding World of Harry Potter:https://blog.discoveruniversal.com/things-to-do/your-guide-to-the-details-fans-will-appreciate-about-gringotts-bank-in-the-wizarding-world-of-harry-potter/ Complete Guide to the Hogwarts Express in The Wizarding World of Harry Potter: https://blog.discoveruniversal.com/things-to-do/complete-guide-to-the-hogwarts-express-in-the-wizarding-world-of-harry-potter/ The Ultimate Guide to Hagrid's Magical Creatures Motorbike Adventure at Universal Orlando Resort:https://blog.discoveruniversal.com/things-to-do/the-ultimate-guide-to-hagrids-magical-creatures-motorbike-adventure-at-universal-orlando-resort/ Complete Guide to Flight of the Hippogriff at The Wizarding World of Harry Potter – Hogsmeade:https://blog.discoveruniversal.com/things-to-do/complete-guide-to-flight-of-the-hippogriff-at-the-wizarding-world-of-harry-potter-hogsmeade/ Shopping in The Wizarding World of Harry Potter at Universal Orlando Resort:https://blog.discoveruniversal.com/things-to-do/shopping-in-the-wizarding-world-of-harry-potter-at-universal-orlando-resort/ The Wizarding World of Harry Potter Food Guide:https://blog.discoveruniversal.com/eat-and-drink/the-wizarding-world-of-harry-potter-food-guide/ A Hufflepuff's Guide to The Wizarding World of Harry Potter:https://blog.discoveruniversal.com/guides-and-tips/hufflepuff-guide-to-the-wizarding-world-of-harry-potter/ A Ravenclaw's Guide to The Wizarding World of Harry Potter:https://blog.discoveruniversal.com/guides-and-tips/a-ravenclaws-guide-to-the-wizarding-world-of-harry-potter/ A Slytherin's Guide to The Wizarding World of Harry Potter: https://blog.discoveruniversal.com/guides-and-tips/slytherin-guide-the-wizarding-world-of-harry-potter/A Gryffindor's Guide to The Wizarding World of Harry Potter:https://blog.discoveruniversal.com/guides-and-tips/a-gryffindors-guide-to-the-wizarding-world-of-harry-potter-2/ Links: Blog: https://blog.DiscoverUniversal.com/Universal Orlando Resort: https://www.universalparks.com/en/usUniversal Parks and Resorts: https://www.universalorlando.com/web/en/usFacebook: https://www.facebook.com/UniversalOrlandoResortTwitter: https://twitter.com/UniversalORLInstagram: https://www.instagram.com/universalorlando/YouTube: https://www.youtube.com/c/UniversalOrlando/featuredMust be 21 years and older with valid Photo identification to purchase or consume alcoholic beverages.EDITOR'S NOTE: The availability of some offerings may be temporarily limited, please check with guest services for current details.Universal Express Unlimited - †Requires theme park admission. Valid only at participating rides and attractions in Universal Studios Florida and Universal's Islands of Adventure. Not valid at Universal's Volcano Bay. Excludes separately ticketed events. Park-to-Park admission required to board the Hogwarts™ Express. Benefit valid only for guests of Loews Royal Pacific Resort, Hard Rock Hotel® and Loews Portofino Bay Hotel, for the number of guests staying in the room for the length of hotel stay. Available during normal theme park operating hours only. Additional restrictions may apply and benefits are subject to change without notice.Early Park Admission - *Requires theme park admission. Early Park Admission begins up to one (1) hour prior to regular scheduled park opening to either Universal Studios Florida or Universal's Islands of Adventure as determined by Universal Orlando. Early Park Admission begins up to one (1) hour prior to regular scheduled park opening for Universal's Volcano Bay. Valid at select attractions at each park. Attractions are subject to substitutions without notice. Additional restrictions may apply.Copyright & Trademark

Musique Mécanique par le Théâtre Électrique ::"Flight of the Hippogriff" by John Williams

The final leg of our epic Florida trip takes Paul for the first time to Universal Studios - Islands of Adventure.  Yes, he was there to enjoy this stunning and perfectly themed park but he was also there to get three big credits (yes, he left Flight of the Hippogriff for another trip).  In this episode he will answer the following : Is Hagrid's Magical Creatures Motorbike Adventure worth the triple digit wait times? Is Incredible Hulk a launch or looping coaster? City Walk, could there be a better place to go on a Friday night? Which island is his favorite (it's a tie but not what you think they would be) Does he even know anything about Harry Potter and does it really matter at Universal? and of course, does VelociCoaster live up to the hype? Paul fought the President's Day crowds to explore this amazing park and he has some thoughts, but we all want to know yours.  Tell us your favorite island.  Your favorite flat ride?  Where do you like to eat at when there?  How can you let us know? Let's Get Social Instagram / Facebook - 125rollercoaster Twitter - 125rollercoast Do you like to get your coaster fix one minute at a time?  If so, check out our YouTube page at - 125 Roller Coaster Challenge We love shorts and this is your place to find them. Our coaster season is in full swing and we can't wait to see you in the queue in 2022.

Our friends Erin and Shavaun stop by to share a couple Mortalis beers to celebrate the upcoming third season of the Mimosas With My Besties podcast! We talk food, movies, and just overall have a fun hang! And we drank Mortalis' Hippogriff and Hydra Sandwich. How do we feel about marshmallow in beer? Why are cheddar bay biscuits the best beer pairing? All questions will be answered!