The Video Insiders

Jeremy Toeman LinkedIn profileAug X Labs website(After you sign-up for the Beta, when you are asked how you heard about Aug X Labs, enter "Heard about it on The Video Insiders Podcast")---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Stef van der Ziel LinkedIn profileJet-Stream website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Paul Markham LinkedIn profilePaul's Cloud Native Media meetups---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Tsahi Levent-Levi LinkedIn profileBlogGeek.me websiteTweaking WebRTC Video Quality Blog Post---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Jim Jachetta LinkedIn profileVidOvation website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Guido Meardi LinkedIn profileV-Nova websiteLC-EVC websiteV-Nova tools documentation---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Dom Robinson  LinkedIn profileGreening Of Streaming websiteEnd-to-end live streaming workflow in the organization's 6 month update---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Ramzi Khsib  LinkedIn profileAWS Elemental website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Jonas Birmé  LinkedIn profileEyevinn websiteWHIP protocolWHPP protocolStreaming Media Sweden 2022 Presentation on WHIP and WHPP---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Daniel Silhavy LinkedIn profileDASH Industry Forum websitedash.js open source5G-MAG reference tools and open sourceFraunhofer FOKUS Media Web Symposium---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Nicolas Weil LinkedIn profileThe eltrovemo BlogNicolas' DJ setsDASH-IF watermarking spec---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Yair Adato LinkedIn profileBria website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Xavier Leclercq LinkedIn profileBroadpeak website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Ali Begen LinkedIn profileAli Begen website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Eric Tang LinkedIn profileThe Livepeer project The Livepeer service---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Yurong Jiang LinkedIn profileLinkedIn Engineering blog post about video conferencing ---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Jan Ozer LinkedIn profileStreaming Learning Center websiteJan Ozer on Streaming Media MagazineJan Ozer on OTTVerse---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Loke Dupont LinkedIn profileTV2 website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Ryan Jespersen LinkedIn profileMillicast websiteIETF WHIP standardMedooze open sourceAndreessen Horowitz podcast on AI, WebRTC, Crypto, and Full Stack Startups---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Michael Schapira LinkedIn profileCompira Labs websiteCompira Labs Report: The State of Streaming 2021 Research from Princeton University and the University of Chicago that shows bigger bandwidth doesn't bring better QoE---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Yaron Zakai-Or LinkedIn profileArti website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Abe Peled LinkedIn profileSynamedia website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Oz Krakowski LinkedIn Profiledeebdub website and YouTube channel---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Todd Erdley LinkedIn Profile---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Ed Laczynski LinkedIn profile---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Nouar Hamze LinkedIn profile---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

LINKS:Alex Liu LinkedIn profileNETINT AV1 Hardware Encoding Announcement---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Anne Aaron LinkedIn ProfileTamar Shoham LinkedIn Profile---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Jeremy Toeman LinkedIn profileAI-generated Beatles AlbumTravis Scott Fortnite Concert---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Brian Alvarez LinkedIn profileVittorio Giovara Blog---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Pierre Seigneurbieux LinkedIn profileBlueJeans Website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Peter Guglielmino LinkedIn profileATSC 3.0 Website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Simon Westbroek LinkedIn profilePepijn Tijhuis LinkedIn profileUnified Streaming Website---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Thierry Fautier LinkedIn profile Harmonic websiteBen Mesander LinkedIn profile Wowza websiteWalid Hamri LinkedIn profile SeaChange websiteWade Wan LinkedIn profile Broadcom websiteOur previous panel on extending the life of H.264 is here---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Bruce Devlin LinkedIn profileBruce's consultancy, Mr. MXF, is hereThe IMF User GroupOur previous episode with Bruce is here---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Zoe Liu LinkedIn profileVisionular, Zoe's company, is here---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Gal Oz LinkedIn profilePixellot, Gal's company, is here ---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Emery Wells LinkedIn profileFrame.io, Emery's company, is here and it is hiring---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

Gregg Heil LinkedIn profileEncoding.com, Gregg's company, is hereBeamr CABR on encoding.com is here---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr

David Hassoun LinkedIn profileRealEyes Media, David's company, is here---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr---------------------------------------------------

Nick Krzemienski LinkedIn profileAwesome.Video, Nick's video streaming knowledge repo, is here---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr---------------------------------------------------

Avisar Ten-Ami LinkedIn profileJosh Barnard LinkedIn profilePankaj Topiwala LinkedIn profile---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news and participate in the discussion.Email thevideoinsiders@beamr.com to be a guest on the show.Learn more about Beamr---------------------------------------------------

Dan Rayburn LinkedIn profileCheck out Dan's Streaming Media BlogBe sure to bookmark OTT Video News (going live in April 2020)---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news.Email thevideoinsiders@beamr.com to be a guest on the show.Learn about Beamr---------------------------------------------------

James Cahall LinkedIn profileLearn about OTTera and Toon Goggles---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insider news.Email thevideoinsiders@beamr.com to be a guest on the show.Learn about Beamr---------------------------------------------------

Tal Zubalsky LinkedIn profileExperience interactive video magicYoni Bloch, CEO & Co-Founder tells the Eko story---------------------------------------------------Join our LinkedIn Group so that you can get the latest video insiders news.Email thevideoinsiders@beamr.com to be a guest on the show.Learn about Beamr--------------------------------------------------- 

Manuel Briand LinkedIn profile.Other episodes that may be of interest: THEOPlayer CTO interviewThe transition from broadcast to OTT--------------------------------------The Video Insiders LinkedIn Group is where thousands of your peers are discussing the latest video technology news and sharing best practices. Click here to joinWould you like to be a guest on the show? Email: thevideoinsiders@beamr.comLearn about Beamr--------------------------------------

Michael Drazin LinkedIn profileRelated episode: HDR from glass-to-glass--------------------------------------The Video Insiders LinkedIn Group is where thousands of your peers are discussing the latest video technology news and sharing best practices. Click here to joinWould you like to be a guest on the show? Email: thevideoinsiders@beamr.comLearn about Beamr--------------------------------------

Github repo: Objective Perceptual Analysis - Compression Karma PredictorChris Kennedy LinkedIn profileRelated episode: Inside Beamr's Content-Adaptive Bitrate Algorithms--------------------------------------The Video Insiders LinkedIn Group is where thousands of your peers are discussing the latest video technology news and sharing best practices. Click here to joinWould you like to be a guest on the show? Email: thevideoinsiders@beamr.comLearn about Beamr--------------------------------------

Download the Innovid 2020 State of Connected TV ReportLearn about InnvoidTal Chalozin LinkedIn profileRelated episode: Direct-to-consumer streaming service launches and first impressionsListen to Episode 20 for more information on interactive advertising and video monetization technology--------------------------------------The Video Insiders LinkedIn Group is where thousands of your peers are discussing the latest video technology news and sharing best practices. Click here to joinWould you like to be a guest on the show? Email: thevideoinsiders@beamr.comLearn about Beamr--------------------------------------TRANSCRIPT: (edited lightly to improve readability)Tal Chalozin:       00:00          Innovid is what we call a video marketing platform. It's a technology platform sold to marketers, brands executives, and agencies that lets them do three things. First and foremost what is called an ad server. It's a technology that actually streams the ad to every website. So if a marketer, let's say Chrysler, or Proctor & Gamble or Best Buy, or others is advertising on YouTube or Hulu or Fox or NBC or New York Times there's a centralized platform that you can actually manage the campaign, upload the MP4's and actually do the streaming and make decisions on now on which video file to serve. So right now we're very fortunate to be the largest video ad server in the world and in many other countries in the United States and many other countries that we operate in. Tal Chalozin:       00:51          A little over a third of all video ads in the United States are being streamed by Innovid. So if you tune into every website and every app, let's say Hulu, one out of three ads, and as a matter of fact on Hulu, it's probably even higher than that. Almost one of every two ads would be, one's coming from Innovid every day. We stream roughly 450 years' worth of ads. And this is just ads content. So we stream a lot of videos. To complete the story of our platform. At a core it's an ad server. And then on top of that there are two applications. One is around creative and the other one is around measurement. Announcer:          01:31          The video insiders is the show that makes sense of all that is happening in the world of online video as seen through the eyes of a second generation codec nerd and a marketing guy who knows what I-frames and macro blocks are. And here are your hosts, Mark Donnigan and Dror Gill. Dror Gill:          01:51          Today we have a very special guest and an old friend of mine Tal Chalozin who is the CTO of Innovid. Hi Tal. Welcome to The Video Insiders. Tal Chalozin:       01:59          Hello Dror. Hello Mark. Thanks for having me. It's a true honor. Mark Donnigan:      02:03          Yeah, welcome Tal. So tell us about Innovid. Tal Chalozin:       02:07          Innovid is a software company that I had the honor of starting together with my two friends and co founders, Zvika Netter our CEO and Zack Zigdon who runs all of our international business. And myself, it's a company that we started back in 2007. Before I explain what we do, just to take you back almost 13 years ago, this is the time after Google acquired YouTube and Hulu as a streaming site was kind of an inception mode. NBC and News Corp started this operation to bring streaming television into the internet. Tal Chalozin:       02:49          And what we said back then is that we believe that the future of television is over IP and to be streamed. We thought that when this would happen the one thing that we really want to tackle is the viewing experience around the advertising. Because it was clear that marketers and ad dollars take a very, very important part of the experience of television subsidizing content and creating the access to so many different people. But it's also clear that sitting through a pretty boring 30 second spot and that every person around the United States in a broadcast time window would see the exact same ad. It's kind of silly. And so we went on a journey to build a software that helps to create a better viewing experience around commercials. Tal Chalozin:       03:44          So we started with the technology, with technology that allows what is called in kind of layman terms virtual product placement. It was a computer vision technology that lets you process videos and reconstruct the 3D. So understanding occlusions and backgrounds and foregrounds and planes and allow you to render a product a 3D product in 3D images into the shot. And it looks like as if it was there while the content was shot while reproducing all the shades and lighting and again, occlusion and, and things like that. This was where we started. We got a bunch of patents. This is how we raised our A round back then. We got so many awards. It was awesome. But then what we learned is that it's amazing, but advertising is a business of scale for marketers to actually play. Tal Chalozin:       04:38          One of the main things that marketers gain out of television is a massive megaphone that lets you tell your story to millions, if not hundreds of millions of people in 30 seconds. So then we went on a journey to better learn this business and expanded more and more capability and fast forward to today. Innovid is what we call a video marketing platform. It's a technology platform sold to marketers, brands executives and agencies that lets them do three things. First and foremost what is called an ad server. It's a technology that actually streams the ad to every website. So if a marketer, let's say Chrysler or Procter and Gamble or Best Buy or others is advertising on YouTube or Hulu or Fox or NBC or New York times there's a centralized platform that you can actually manage the campaign, upload the MP4's and actually do the streaming and make decisions on which video file to serve to the individual that is streaming the content. Tal Chalozin:       05:48          So right now we're very fortunate to be the largest video ad server in the world. And in many other countries in the United States, many other countries that we operate in a little over a third of all video ads in the United States are being streamed by Innovid. At a core it's an ad server. And then on top of that, there are two applications. One is around creative and the other one is around measurement. Our headquarters in New York. There's 350 people, a big R&D center in Israel and then offices across the U S and in Europe. And in APAC. If you read the trades, it seems like the future of television has no ads. Disney Plus, Netflix, Amazon, Apple, all of the big services that made a lot of splash in the press toot the horn of no ads. Tal Chalozin:       06:43          This is very nice for marketing, but in reality advertising dollars pays the bills that makes so many pieces of content to be streamed. The subscription services could not really thrive on subscription alone, let alone when you're talking about a massive global service that would like to reach hundreds of millions of subscribers. You cannot do that only with subscribing. With subscription dollars or advertising is a very strong market and in the future will be that. Easy testament is that just last week NBC launched or Comcast launched there foray into that game called Peacock. And the main thing that they said is that, Hey there's so much noise around advertising, about no ads. This cannot work. We will include ads. Tal Chalozin:       07:36          And this makes to the second part of what I wanted to say about the future is that, but they put a lot of emphasis around ed experience. So it's not that you will see ads in the same way that you're used to watching television. There will still be ad breaks, but it will look and feel very, very different than what it used to be on television. And we play a very big role there and in other places. And we think that yes, the future of television is over the internet, over IP. The future of television is with ads, or at least in some capacity of it, but it would look and feel much different. Dror Gill:          08:14          I want to ask a question regarding the, the ad server component. And these ads go interleaved into content experiences sometimes before or after or during the actual streaming of the content. So how do you match the resolution and the quality of the ad that you provide to the actual content that is being streamed? Because I don't assume that somebody watching a 4K movie would like to be interrupted by like an, you know, an SD, low quality ad. It would probably be quite annoying. Tal Chalozin:       08:52          I have so many things to say about this stuff. First of all, before I answer exactly how we did it I can tell you that people think that the internet is so advanced in 2020 so all of this problem is practically solved. And there is no real problem to bring television over the internet and it's not really true. I'm sure you know you know, very well the general standard in the video ads industry right now is that we as the server that generated the file and hosts them, would create an XML template called vast V A S T and put multiple video renditions in a file and create a manifest that would have different renditions of and actually different encoders as well. Tal Chalozin:       09:44          Of the file. It used to be, we used to put FLV and other stuff. But right now it's all MP4 containers. But anyway, you put multiple renditions and then the actual player picks the right one and the player, essentially what it's doing is doing playlisting. So picking the right ad at the right time and there is a, in the last, the last few years, but honestly, just in the last year, there is a big change in the way video ads are being streamed. Moving from what used to be called CSAI client side ad insertion, AKA playlisting. So on the client you download some, some type of playlisting and then you just move between different files even if it's the main content - it doesn't matter the rendition, you would still switch between different files that you do progressive downloads for. Tal Chalozin:       10:45          Most of the very large sites and today apps are what is called SSAI server-side ad insertion. Essentially it doesn't matter what file we bring. You convert it into an HLS stream, create TS files, and then do kind of the, the term that everyone is using is manifest manipulation. So just manipulate the M3u8 and swap packets, TS files inside the M3u8. I hope that I don't need to explain everything that I'm just saying, but stop me if you want me to. So essentially let's say on Hulu, this is how it works. You will tune into a stream and you hit play on an episode of a, I don't know, The Good Wife on Hulu. What they will do, they will go, let's say this is 48 minutes of an episode or 21 minutes of an episode with multiple ads that need to be weaved throughout. So what they will do, they will do a server side call to all the different ads and then get either an MP4 and do just in time transcoding for it. Or, if it's pre-prepared, like a lot of the things that we do you would get the actual TS file and then just merge it into a single M3u8 with content TS files in the right rendition and the ads. Mark Donnigan:      12:09          So Tal, are you actually able to get the, you know, I'll call it the mezzanine file of the ad, and then you can create a high quality or at least the highest quality possible for the, you know, target resolution and bit rate or are you limited by the fact that sometimes, you know, you may get a mezzanine quality and other times it may just be a 1080p in which case Dror's example of like a 4K. You're just limited. I mean, you have the quality you have. So can you tell us, shed some light on that? Tal Chalozin:       12:43          It's a fascinating point. This is an uphill battle for us because we are, we're still an intermediary. We're not the post production shop at that makes the video file, so we're limited to whatever you would get. So yeah, the intention is to get a Pro Res or a mez file, mezzanine file, of the ad that allows us to do transcoding into whatever we want. But, that's not the reality all the time. In many cases we would get to your example, a 1080p is a good case. In some cases we get 720 and sometimes we even need to up convert it, which clearly is not really working. Tal Chalozin:       13:34          And the reality is that the 4K streaming of ad supported content is not a real thing as of right now. But, 1080p is definitely one that is. And again, we're in 2020 right now and you can open whatever app without naming names, but you can open one of the biggest apps out there and I'm sure you would get to an ad break and even an unaided eye can see that it's a totally different rendition of the ad, even different audio, let alone volume normalization. But even just the quality of the encoding is significantly different or lower than the actual content. And this is a common case or the state of the internet right now. Dror Gill:          14:24          But this is something you're trying to avoid? Tal Chalozin:       14:26          We're definitely trying to avoid the way that we're doing it is that if you think about it, there are two inputs to our system. One is the ad itself, literally, again the mez file, Pro Res, whatever container that is, an MP4. And then, what is called in ad terms a media plan. Media plan is saying that we are Chrysler, the campaign starts in this date and ends on this date, there is X number of million impressions on YouTube, then on Snapchat, then on Hulu, and then the full list. It's a very complicated meta data of the whole campaign. So those are the two inputs that we're getting. Historically that was just an upload. So in our system, you would go and just upload the files. Tal Chalozin:       15:13          More and more we're trying to get down to the source and create some type of an integration with the, with the DAM, the digital asset manager. Let's say, again, this is a Chrysler commercial, Chrysler 300 commercial. Someone actually did the post for it, and they do have the approved asset at the best quality possible. But those are not our customers. So sometimes we don't get access to that and we need to beg the customer to get that and try to explain what's the outcome if they don't get it. So what we're trying to do is to get down to the source as close as possible. So then that post-production shop would actually have an API to us, or even if they upload, they would upload the source and not have a downsample of it. Mark Donnigan:      16:05          So our audience, are largely encoding engineers, video engineers, and we just hear over and over again incredible frustration about this. Dror and I were just talking to a very large live sports streaming service last week and the person responsible for encoding was lamenting that whenever there's issues with quality, it's because he can't do any better. It's a source issue! The high quality asset exists. Why can't we get access to it so that we can provide an incredible advertising experience. And I'm just wondering, how do we fix this? Tal Chalozin:       16:50          How do we fix that? As more hours per day continues to pour into the connected, let's call it the connected television space, and as more and more ad dollars flow in there, and then more and more people cut their cord or shave their cord or are cord nevers and haven't even been exposed to traditional television, this becomes the norm and not the new thing. It's essentially a supply chain or a workflow problem because as you said, the file is there. It's not that someone is shooting on an SD camera and now you, you're stuck with a shitty file. People are using RED cameras to shoot it. So yeah, so it's more of a workflow problem. And this is what we set out to do is to just remove the clutter and connect everything in an industry that wasn't connected. Ads on television, still are being delivered predominantly through FedEx with cassette tapes that are being sent to local TV stations. Tal Chalozin:       17:50          This is still a thing. We're moving from this world and now talking about getting a mezzanine or 4K file. I'll tell you about one thing that I'm very keen on, is that another thing is getting the raw asset is one thing. And then another thing, if you look at it, there's multiple parties on the internet that are getting an asset and transcoding it. So let's say that we get the video file. Probably Facebook got the video file as well, maybe not through Innovid. And they also transcoded the video file and then YouTube or Tik Tok got the video file somehow. And then sometimes clients would use Innovid. Sometimes you would go directly into YouTube and upload the raw file. And maybe NBC would get it through some other distribution channel to the broadcast side. Tal Chalozin:       18:44          And then when they run it online, they would take the broadcast file and transcode it as well. So there was multiple people or organization that got the raw footage and then they're in charge of transcoding. This is pretty stupid. It should be some type of a centralized repository because there is an ID to every file and there is an initiative called the Ad ID to make sure that there will be a unified numbering system, and a catalog. And by virtue of that, meta data and tracking just in the ad space so in every ad and then not only did you have a catalog, you can access all different resolutions in a centralized place. So then if YouTube wants a a downsampled version, then you just pick the resolution you want. You don't take the raw and then encode it as well. Tal Chalozin:       19:32          There's an initiative. There are several companies trying to do that. It's kind of a hurding cats type of an initiative. But it's almost a necessity because unless you do that, you will always have those artifacts. Mark Donnigan:      19:46          Yeah, that's right. And that Ad ID in your experience does that travel, I'll use the word seamlessly, you know, between these various systems or is that even an issue of keeping that ad ID intact? Tal Chalozin:       19:59          You know that it is a meta data but in reality again, we are one of the largest platforms that actually accesses files and stream them out and encode them. Most people that do encoding do not carry on all the meta data. That's one thing. Second thing is that most people, actually, most platforms don't even look into that meta data. So don't even expose that or do anything with it. Tal Chalozin:       20:22          Several encoders do not put it in there. So right now, yes, it is there, but it's not fully available. So the solution that is used mostly right now, which you would laugh, is putting it in the actual file name. So literally as an unstructured text on the file before the dot and before you put an underscore and then the the actual file, which clearly doesn't carry through anywhere. So that's the reality again, right now in 2020. It's almost like Dror do you remember Yossi Vardi's example of pigeons carrying DVDs in order to transfer a lot of files, large files? Dror Gill:          21:04          He also did another experiment. He took a snail and he stuck a USB drive on the back of the snail. And then he had two computers connected with a crossed ethernet cable and he was trying to see how the data will go faster through the cable or the snail that is moving slowly between the computers with the USB drive on his back. And I'm sorry to say, but the snail won! Tal Chalozin:       21:28          The industry from the outside seems like, again, it all problems are solved, but it's far from it. You know, the Superbowl is coming up very soon and Fox is going to air the Superbowl and like every year you can access it in streaming as well. And it's still a discussion every year. Is the internet already for that? The term for ad serving in real time in the world of television is called DAI dynamic ad insertion. Every broadcaster that gets the right to stream the Superbowl is asking, are we ready or are we safe to do DAI for the ads or to play it safe are we gonna take the broadcast feed and then just retransmit? I Can tell you a funny story, that last year we did a really cool experiment. Tal Chalozin:       22:19          CBS had the rights for the Superbowl and they use a system that takes the SCTE tone and converts it into an ID3 tag for digital systems. And then on the ID3 we put the marker of the ads, we put the actual Innovid URL of the the ad that is about to play. Originally the system was architected for measurement. So you can do measurements from the client side. So there is something on the client side, gate the ID3 tag and then fire that just do an HTTP get call that URL in order to track track the ads from the client in the most accurate way. But then what we did last year together with CBS is add the ability to also run overlays on top of the video. Tal Chalozin:       23:09          So that URL was not just for measurement, but also downloaded graphics to be displayed as a kind of, as a transparent layer on top of that on the device itself. So if you stream live stream. This is not VOD or anything like that. You do live stream of the Superbowl. Last year many devices on CBS Sports had a small SDK that again, took the SCTE tone converted to an ID3 tag, get a URL for a PNG file or whatever that is rendered in near real time. And then every house on the United States gets something else. We did an experiment together with Pringles. The whole commercial was some type of a game with Pringles. So you would get a message that is tailored to you. Tal Chalozin:       24:00          So, it literally featured the name of your city on it. And then it allows it to use your remote, let's say Apple TV. You can use your remote to left and right to swipe and play some, some kind of a funky game as the ad was playing. So funny thing again, this is 2019. You would imagine that we would have that technology available. This is not rocket science. We're talking about a lot more advanced things on the internet. But even that was super revolutionary and this year this capability will not be available because the way that Fox works is different. But that count is super cutting edge. Mark Donnigan:      24:40          Now Tal, I know that you're working very closely with Roku, so why don't you share with us what you're doing with them. Mark Donnigan:      24:49          Share what you can and tell us about what's happening on the Roku platform because I think that's very important to all of us in, in streaming media streaming video. Tal Chalozin:       25:00          Roku is a streaming device. It is divided into two parts of their platform. One is a device a streaming stick and streaming box. But, Roku first and foremost is an operating system that runs on that device or licensed to TV manufacturers, to TV OEMs. And right now there's eleven OEMs that carries that. Anything from TCL, or Insignia, all the way to LG, and on some SKUs from Sharp as well. And by numbers, Roku is the largest television operating system right now in the United States. The most amount of TV's purchased in 2019 was Roku powered or TVs or streams were powered by Roku. Tal Chalozin:       25:47          So this is larger than Amazon Fire, way way larger than Apple TV or Xbox or PlayStation or whatnot. So this is, this is Roku. Back in the early, early days of Roku this dating back to, to 2014 or 15, we did the first advertising oriented deal with Roku to create a small library and SDK that would be part of their firmware that many years later, the name is Roku ad framework, or RAF. Which is a set of libraries that lets app developers, Roku app developers get access to to stuff they need to run ads inside the app without a lot of work that allows us to create a technology for like, for example, interactive television, something that can be done in a very scalable way because now every app on Roku has the ability to render ads that can have overlays. Tal Chalozin:       26:47          You can press the remote and you can purchase things or send things to your phone or whatever activity you would like. So this is the first thing we've done with Roku and enabled that technology at a mass scale. This is many, many years before Roku was a big success. But at the end of last year, in September we, together with Roku, we announced kind of the second, second act of the innovation on the future of television, which is around measurement. I mentioned at the beginning, the top of the, of the show that we have three parts to our platform, the ad server, which we talked a lot about, different tools around creative. And the third one would be measurement capabilities. On the measurement side this is an area that the television industry, we talked a lot about things that require innovation. Tal Chalozin:       27:41          Measurement is maybe at the top of the list cause right now measurement on television is dominated by a company called Nielsen, which I'm sure many people know that the way they measure television because of lack of connectivity is by putting a people meter or a device in people's home. In very, very few households in the United States that act as a sample or as a panel which presumably should represent every household in the United States. So there's roughly 20,000 families in the United States that represent the television ecosystem, which there is north of 100 million households in the United States. And maybe 80 or 90 million households that are watching broadcast television and they're being paneled by 20,000 that essentially measuring what do people actually watch. Tal Chalozin:       28:42          So, we want to change that. We, and many other important, an important point is that many other companies are, are at it. Because, it's obvious this needs to be changed. But we teamed up with Roku that every one of the devices that carries their operating system, so every one of those TVs that have Roku as an operating system have a small chip called ACR. Stands for automatic content recognition that essentially knows what you're watching. So it records everything that hits the glass. And it doesn't matter if it hits the glass because it's an app on the Roku platform, let's say Hulu or YouTube or Netflix, or you plugged in via HDMI, your set top box or you plugged in an antenna to to the TV or even you have a DVR or VHS plugged into your television. Tal Chalozin:       29:32          Doesn't matter if it's rendered on the screen, then Roku would know what it is. They do a second by second or almost a frame by frame to a catalog. And then know what exactly you're watching and at what time code. We can talk about privacy as well, which is a very important part of it. But this is all opted in. You don't have to contribute this data, but most people do. And then we get this data. We don't care about the individual household, but we can use that as you don't, you don't need a panel anymore where every television is telling you what exactly you're watching. So we are, we're on a mission to reinvent that television measurement in a much better way. Dror Gill:          30:15          That's really amazing. So the television is actually watching what you are watching. Even if it's not streamed through that Roku platform, it's watching everything that is projected to the screen and not only you know, like recording the pixels or they're actually using this automatic content recognition system. Analyzing and knowing what content, what piece of content this is, whether it's a live broadcast or a video on demand. It could be a DVD or a VHS, time shifted or it's an ad. Exactly. Mark Donnigan:      30:51          Where is that fingerprint happening Tal? Tal Chalozin:       31:01          And by the way, a disclaimer, I don't work for Roku and I don't know any internal data about Roku. We have a strong partnership with them. So Roku is unique technology. And by the way, other TV manufacturers are doing the same thing. This is not limited to Roku. Vizio who made a lot of noise around that as well. And many others, Sony and Toshiba and others. Are using similar technologies. What's on the device is mainly picking up multiple pixels, hashing it together and sending it to the cloud. The matching to the catalog is not happening on the device. There's clearly no need for that. And there are several companies that create this catalog and does essentially the pattern matching between the set of temporal data of that set of frames, consecutive frames to a catalog to know exactly what you're watching. Tal Chalozin:       31:55          Is it - what show? What episode? Is it an ad? So one thing is to know the catalog. The other one is to know what is on right now in every... It's a very complicated problem, because sometimes you are you, you may be watching it live. Again, tuning into, I dunno, ABC, but right now because that show is a local show, you would watch it streamed by the Kansas city, Missouri ABC affiliate and it's not a national show. So you can't really match it to a catalog and know is it live or not live? And then when it comes to ads, it gets even more complicated because some of the ads are inserted in real time. So you need to know that that ad is inserted in real time so then it doesn't impact the idea of the stream. You didn't really change the channel. It's just dynamic insertion. Dror Gill:          32:48          So doing all of this measurement, I think it probably puts a lot of responsibility on your part of the value chain on the software that you create, on the reports that you generate. Because based on this I guess is how the content providers get paid right. For showing those ads, as you said. Tal Chalozin:       33:12          We are what is called the system of record for billing. So I mentioned that roughly a third of the ads are being transacted by us. This is a very rough number because the dollars don't go through us. We're just creating the billing. We are the actual counter of something like $5 billion of of ad dollars. So again, YouTube and Snapchat and New York times and NBC and Fox and TubiTV and many other channels and apps are being paid based on our numbers. And in order for that, we need to do a lot of filtration, detecting what is fraud, and making sure there's no false positives, and and many other things like that. And for it, we go through an audit process. So Ernst & Young is the auditor and there's an organization called the Media Rating Council that we go through an audit every year to make sure that what we say we do, we actually do. Tal Chalozin:       34:12          And there's no there's no problems in the counting. And yeah, it happens all the time that we are counting, but also clearly broadcasters or apps would count for their own use as well. And sometimes, unfortunately, the numbers are not the same. So we would say that P&G ran 10 million ads and the broadcaster, NBC, Discovery, what have you, would say that actually it's 10 and a half million ads. So then they need to get paid more. But the way that the contract is written is that Innovid numbers because we're unbiased is what is what will dictate the payment. So you're like the gold standard in measurements. But it's a very interesting, a very interesting world. Tal Chalozin:       35:08          It's an ever changing world. So counting ads 10 years ago and counting ads today is a very, very different business. Mark Donnigan:      35:14          There's a lot of studies and I think you even have one that you can cite if you'd like to that say very clearly that consumers are not opposed to ads. This whole notion that people "hate ads" is actually not true. What they hate is a bad or an irrelevant experience. If the platform happens to know that I'm looking for a new car and I get served a great car ad, guess what? And especially if it piques my interest, that's actually a good experience. Tal Chalozin:       35:48          100%. Yeah. We always use exactly the same term that you mentioned. People don't hate ads, they just hate bad ads. And that's absolutely true. And when you ask people, when you again, when you read the trades, it looks like ads are a very gloomy thing. Tal Chalozin:       36:06          And then you go to platforms like, in my mind, Instagram is the best ad experience ever made. When you see ads on Instagram, it's significantly better. And it's not disruptive at all. You have your thumb there and you can continue scrolling. And then many, many people choose to actually watch that. So completely reverse model. It's not that I'm forced to watch the ad. I literally can continue scrolling the same way that I'm scrolling there. But people literally are choosing to watch that because it's good ads. Mark Donnigan:      36:43          This has been a really amazing discussion and you know we have to do a part two. Yeah, there are few issues we did not cover and we must cover them and it's really been fascinating. Yeah, absolutely. Thanks for joining us Tal. Tal Chalozin:       36:57          I'd love to, thank you so much. Thanks, Mark. Thanks Dror. Thanks everyone that listened. Thanks Beamr. Announcer:          37:04          Thank you for listening to The Video Insiders podcast, a production of Beamr limited. To begin using Beamr's codecs today. Go to beamr.com/free to receive up to 100 hours of no cost HEVC and H.264 transcoding every month.

Click to watch SPIE Future Video Codec Panel DiscussionRelated episode with Gary Sullivan at Microsoft: VVC, HEVC & other MPEG codec standardsInterview with MPEG Chairman Leonardo Charliogne: MPEG Through the Eyes of it's ChairmanLearn about FastDVO herePankaj Topiwala LinkedIn profile--------------------------------------The Video Insiders LinkedIn Group is where thousands of your peers are discussing the latest video technology news and sharing best practices. Click here to joinWould you like to be a guest on the show? Email: thevideoinsiders@beamr.comLearn more about Beamr--------------------------------------TRANSCRIPT:Pankaj Topiwala: 00:00 With H.264 H.265 HEVC in 2013, we were now able to do up to 300 to one to up to 500 to one compression on a, let's say a 4K video. And with VVC we have truly entered a new realm where we can do up to 1000 to one compression, which is three full orders of magnitude reduction of the original size. If the original size is say 10 gigabits, we can bring that down to 10 megabits. And that's unbelievable. And so video compression truly is a remarkable technology and you know, it's a, it's a marval to look at Announcer: 00:39 The Video Insiders is the show that makes sense of all that is happening in the world of online video as seen through the eyes of a second generation codec nerd and a marketing guy who knows what I-frames and macro blocks are. And here are your hosts, Mark Donnigan and Dror Gill. Speaker 3: 00:39 Dror Gill: 01:11 Today we're going to talk with one of the key figures in the development of a video codecs and a true video insider Pankaj Topiwala. Hello Pankaj and welcome to The Video Insiders podcast. Pankaj Topiwala: 01:24 Gentlemen. hello, and thank you very much for this invite. It looks like it's going to be a lot of fun. Mark Donnigan: 01:31 It is. Thank you for joining Pankaj. Dror Gill: 01:33 Yeah, it sure will be a lot of fun. So can you start by telling us a little bit about your experience in codec development? Pankaj Topiwala: 01:41 Sure, so, I should say that unlike a number of the other people that you have interviewed or may interview my background is fair bit different. I really came into this field really by a back door and almost by chance my degree PhD degree is actually in mathematical physics from 1985. And I actually have no engineering, computer science or even management experience. So naturally I run a small research company working in video compression and analytics, and that makes sense, but that's just the way things go in the modern world. But that the effect for me was a, and the entry point was that even though I was working in very, very abstract mathematics I decided to leave. I worked in academia for a few years and then I decided to join industry. And at that point they were putting me into applied mathematical research. Pankaj Topiwala: 02:44 And the topic at that time that was really hot in applied mathematics was a topic of wavelets. And I ended up writing and edited a book called wavelet image and video compression in 1998. Which was a lot of fun along with quite a few other co authors on that book. But, wavelets had its biggest contribution in the compression of image and video. And so that led me finally to enter into, and I noticed that video compression was a far larger field than image compression. I mean, by many orders, by orders of magnitude. It is probably a hundred times bigger in terms of market size than, than image compression. And as a result I said, okay, if the sexiest application of this new fangled mathematics could be in video compression I entered that field roughly with the the book that I mentioned in 1998. Mark Donnigan: 03:47 So one thing that I noticed Pankaj cause it's really interesting is your, your initial writing and you know, research was around wavelet compression and yet you have been very active in ISO MPEG, all block-based codecs. So, so tell us about that? Pankaj Topiwala: 04:08 Okay. Well obviously you know when you make the transition from working on the wavelets and our initial starting point was in doing wavelet based video compression. When I started first founded my company fastVDO in 1998, 1999 period we were working on wavelet based video compression and we, we pushed that about as much as we could. And at that, at one point we had what we felt was the world's best a video compression using wavelets in fact, but best overall. And it had the feature that you know, one thing that we should, we should tell your view or reader listeners is that the, the value of wavelets in particular in image coding is that not only can you do state of the art image coding, but you can make the bitstream what is called embedded, meaning you can chop it off at anywhere you like, and it's still a decodable stream. Pankaj Topiwala: 05:11 And in fact it is the best quality you can get for that bit rate. And that is a powerful, powerful thing you can do in image coding. Now in video, there is actually no way to do that. Video is just so much more complicated, but we did the best we could to make it not embedded, but at least scalable. And we, we built a scalable wavelet based video codec, which at that time was beating at the current implementations of MPEG4. So we were very excited that we could launch a company based on a proprietary codec that was based on this new fangled mathematics called wavelets. And lead us to a state of the art codec. The facts of the ground though is that just within the first couple of years of running our company, we found that in fact the block-based transformed codecs that everybody else was using, including the implementers of MPEG4. Pankaj Topiwala: 06:17 And then later AVC, those quickly surpassed anything we could build with with wavelets in terms of both quality and stability. The wavelet based codecs were not as powerful or as stable. And I can say quite a bit more about why that's true. If you want? Dror Gill: 06:38 So when you talk about stability, what exactly are you referring to in, in a video codec? Pankaj Topiwala: 06:42 Right. So let's let's take our listeners back a bit to compare image coding and video coding. Image coding is basically, you're given a set of pixels in a rectangular array and we normally divide that into blocks of sub blocks of that image. And then do transforms and then quantization and than entropy coding, that's how we typically do image coding. With the wavelet transform, we have a global transform. It's a, it's ideally done on the entire image. Pankaj Topiwala: 07:17 And then you could do it multiple times, what are called multiple scales of the wavelet transform. So you could take various sub sub blocks that you create by doing the wavelet transfer and the low pass high pass. Ancs do that again to the low low pass for multiple scales, typically about four or five scales that are used in popular image codecs that use wavelets. But now in video, the novelty is that you don't have one frame. You have many, many frames, hundreds or thousands or more. And you have motion. Now, motion is something where you have pieces of the image that float around from one frame to another and they float randomly. That is, it's not as if all of the motion is in one direction. Some things move one way, some things move other ways, some things actually change orientations. Pankaj Topiwala: 08:12 And they really move, of course, in three dimensional space, not in our two dimensional space that we capture. That complicates video compression enormously over image compression. And it particularly complicates all the wavelet methods to do video compression. So, wavelet methods that try to deal with motion were not very successful. The best we tried to do was using motion compensated video you know, transformed. So doing wavelet transforms in the time domain as well as the spatial domain along the paths of motion vectors. But that was not very successful. And what I mean by stability is that as soon as you increase the motion, the codec breaks, whereas in video coding using block-based transforms and block-based motion estimation and compensation it doesn't break. It just degrades much more gracefully. Wavelet based codecs do not degrade gracefully in that regard. Pankaj Topiwala: 09:16 And so we of course, as a company we decided, well, if those are the facts on the ground. We're going to go with whichever way video coding is going and drop our initial entry point, namely wavelets, and go with the DCT. Now one important thing we found was that even in the DCT- ideas we learned in wavelets can be applied right to the DCT. And I don't know if you're familiar with this part of the story, but a wavelet transform can be decomposed using bits shifts and ads only using something called the lifting transform, at least a important wavelet transforms can. Now, it turns out that the DCT can also be decomposed using lifting transforms using only bit shifts and ads. And that is something that my company developed way back back in 1998 actually. Pankaj Topiwala: 10:18 And we showed that not only for DCT, but a large class of transforms called lab transforms, which included the block transforms, but in particular included more powerful transforms the importance of that in the story of video coding. Is that up until H.264, all the video codec. So H.261, MPEG-1, MPEG-2, all these video codecs used a floating point implementation of the discrete cosign transform and without requiring anybody to implement you know a full floating point transform to a very large number of decimal places. What they required then was a minimum accuracy to the DCT and that became something that all codecs had to do. Instead. If you had an implementation of the DCT, it had to be accurate to the true floating point DCT up to a certain decimal point in, in the transform accuracy. Pankaj Topiwala: 11:27 With the advent of H.264, with H.264, we decided right away that we were not going to do a flooding point transform. We were going to do an integer transform. That decision was made even before I joined, my company joined, the development base, H.264, AVC, But they were using 32 point transforms. We found that we could introduce 16 point transforms, half the complexity. And half the complexity only in the linear dimension when you, when you think of it as a spatial dimension. So two spatial dimensions, it's a, it's actually grows more. And so the reduction in complexity is not a factor of two, but at least a factor of four and much more than that. In fact, it's a little closer to exponential. The reality is that we were able to bring the H.264 codec. Pankaj Topiwala: 12:20 So in fact, the transform was the most complicated part of the entire codec. So if you had a 32 point transform, the entire codec was at 32 point technology and it needed 32 points, 32 bits at every sample to process in hardware or software. By changing the transform to 16 bits, we were able to bring the entire codec to a 16 bit implementation, which dramatically improved the hardware implementability of this transfer of this entire codec without at all effecting the quality. So that was an important development that happened with AVC. And since then, we've been working with only integer transforms. Mark Donnigan: 13:03 This technical history is a really amazing to hear. I, I didn't actually know that Dror or you, you probably knew that, but I didn't. Dror Gill: 13:13 Yeah, I mean, I knew about the transform and shifting from fixed point, from a floating point to integer transform. But you know, I didn't know that's an incredible contribution Pankaj. Pankaj Topiwala: 13:27 We like to say that we've saved the world billions of dollars in hardware implementations. And we've taken a small a small you know, a donation as a result of that to survive as a small company. Dror Gill: 13:40 Yeah, that's great. And then from AVC you moved on and you continued your involvement in, in the other standards, right? That's followed. Pankaj Topiwala: 13:47 in fact, we've been involved in standardization efforts now for almost 20 years. My first meeting was a, I recall in may of 2000, I went to a an MPEG meeting in Geneva. And then shortly after that in July I went to an ITU VCEG meeting. VCEG is the video coding experts group of the ITU. And MPEG is the moving picture experts group of ISO. These two organizations were separately pursuing their own codecs at that time. Pankaj Topiwala: 14:21 ISO MPEG was working on MPEG-4 and ITU VCEG was working on H.263, and 263 plus and 263 plus plus. And then finally they started a project called 263 L for longterm. And eventually it became clear to these two organizations that look, it's silly to work on, on separate codecs. They had worked once before in MPEG-2 develop a joint standard and they decided to, to form a joint team at that time called the joint video team, JVT to develop the H.264 AVC video codec, which was finally done in 2003. We participate participated you know fully in that making many contributions of course in the transform but also in motion estimation and other aspects. So, for example, it might not be known that we also contributed the fast motion estimation that's now widely used in probably nearly all implementations of 264, but in 265 HEVC as well. Pankaj Topiwala: 15:38 And we participated in VVC. But one of the important things that we can discuss is these technologies, although they all have the same overall structure, they have become much more complicated in terms of the processing that they do. And we can discuss that to some extent if you want? Dror Gill: 15:59 The compression factors, just keep increasing from generation to generation and you know, we're wondering what's the limit of that? Pankaj Topiwala: 16:07 That's of course a very good question and let me try to answer some of that. And in fact that discussion I don't think came up in the discussion you had with Gary Sullivan, which certainly could have but I don't recall it in that conversation. So let me try to give for your listeners who did not catch that or are not familiar with it. A little bit of the story. Pankaj Topiwala: 16:28 The first international standard was the ITU. H.261 standard dating roughly to 1988 and it was designed to do only about 15 to one to 20 to one compression. And it was used mainly for video conferencing. And at that time you'd be surprised from our point of view today, the size of the video being used was actually incredibly tiny about QCIP or 176 by 144 pixels. Video of that quality that was the best we could conceive. And we thought we were doing great. And doing 20 to one compression, wow! Recall by the way, that if you try to do a lossless compression of any natural signal, whether it's speech or audio or images or video you can't do better than about two to one or at most about two and a half to one. Pankaj Topiwala: 17:25 You cannot do, typically you cannot even do three to one and you definitely cannot do 10 to one. So a video codec that could do 20 to one compression was 10 times better than what you could do lossless, I'm sorry. So this is definitely lossy, but lossy with still a good quality so that you can use it. And so we thought we were really good. When MPEG-1 came along in, in roughly 1992 we were aiming for 25 to one compression and the application was the video compact disc, the VCD. With H.262 or MPEG-2 roughly 1994, we were looking to do about 35 to one compression, 30 to 35. And the main application was then DVD or also broadcast television. At that point, broadcast television was ready to use at least in some, some segments. Pankaj Topiwala: 18:21 Try digital broadcasting. In the United States, that took a while. But in any case it could be used for broadcast television. And then from that point H.264 AVC In 2003, we jumped right away to more than 100 to one compression. This technology at least on large format video can be used to shrink the original size of a video by more than two orders of magnitude, which was absolutely stunning. You know no other natural signal, not speech, not broadband, audio, not images could be compressed that much and still give you high quality subjective quality. But video can because it's it is so redundant. And because we don't understand fully yet how to appreciate video. Subjectively. We've been trying things you know, ad hoc. And so the entire development of video coding has been really by ad hoc methods to see what quality we can get. Pankaj Topiwala: 19:27 And by quality we been using two two metrics. One is simply a mean square error based metric called peak signal to noise ratio or PSNR. And that has been the industry standard for the last 35 years. But the other method is simply to have people look at the video, what we call subjective rating of the video. Now it's hard to get a subjective rating. That's reliable. You have to do a lot of standardization get a lot of different people and take mean opinion scores and things like that. That's expensive. Whereas PSNR is something you can calculate on a computer. And so people have mostly in the development of video coding for 35 years relied on one objective quality metric called PSNR. And it is good but not great. And it's been known right from the beginning that it was not perfect, not perfectly correlated to video quality, and yet we didn't have anything better anyway. Pankaj Topiwala: 20:32 To finish the story of the video codecs with H.265 HEVC in 2013, we were now able to do up to 300 to one to up to 500 to one compression on let's say a 4K. And with VVC we have truly entered a new realm where we can do up to 1000 to one compression, which is three full orders of magnitude reduction of the original size. If the original size is say, 10 gigabits, we can bring that down to 10 megabits. And that's unbelievable. And so video compression truly is a remarkable technology. And you know, it's a, it's a marvel to look at. Of course it does not, it's not magic. It comes with an awful lot of processing and an awful lot of smarts have gone into it. That's right. Mark Donnigan: 21:24 You know Pankaj, that, is an amazing overview and to hear that that VVC is going to be a thousand to one. You know, compression benefit. Wow. That's incredible! Pankaj Topiwala: 21:37 I think we should of course we should of course temper that with you know, what people will use in applications. Correct. They may not use the full power of a VVC and may not crank it to that level. Sure, sure. I can certainly tell you that that we and many other companies have created bitstreams with 1000 to one or more compression and seeing video quality that we thought was usable. Mark Donnigan: 22:07 One of the topics that has come to light recently and been talked about quite a bit. And it was initially raised by Dave Ronca who used to lead encoding at Netflix for like 10 years. In fact you know, I think he really built that department, the encoding team there and is now at Facebook. And he wrote a LinkedIn article post that was really fascinating. And what he was pointing out in this post was, was that with compression efficiency and as each generation of codec is getting more efficient as you just explained and gave us an overview. There's a, there's a problem that's coming with that in that each generation of codec is also getting even more complex and you know, in some settings and, and I suppose you know, Netflix is maybe an example where you know, it's probably not accurate to say they have unlimited compute, but their application is obviously very different in terms of how they can operate their, their encoding function compared to someone who's doing live, live streaming for example, or live broadcast. Maybe you can share with us as well. You know, through the generation generational growth of these codecs, how has the, how has the compute requirements also grown and has it grown in sort of a linear way along with the compression efficiency? Or are you seeing, you know, some issues with you know, yes, we can get a thousand to one, but our compute efficiency is getting to the, where we could be hitting a wall. Pankaj Topiwala: 23:46 You asked a good question. Has the complexity only scaled linearly with the compression ratio? And the answer is no. Not at all. Complexity has outpaced the compression ratio. Even though the compression ratio is, is a tremendous, the complexity is much, much higher. And has always been at every step. First of all there's a big difference in doing the research, the research phase in development of the, of a technology like VVC where we were using a standardized reference model that the committee develops along the way, which is not at all optimized. But that's what we all use because we share a common code base. And make any new proposals based on modifying that code base. Now that code base is always along the entire development chain has always been very, very slow. Pankaj Topiwala: 24:42 And true implementations are anywhere from 100 to 500 times more efficient in complexity than the reference software. So right away you can have the reference software for say VVC and somebody developing a, an implementation that's a real product. It can be at least 100 times more efficient than what the reference software, maybe even more. So there's a big difference. You know, when we're developing a technology, it is very hard to predict what implementers will actually come up with later. Of course, the only way they can do that is that companies actually invest the time and energy right away as they're developing the standard to build prototype both software and hardware and have a good idea that when they finish this, you know, what is it going to really cost? So just to give you a, an idea, between, H.264 and Pankaj Topiwala: 25:38 H.265, H.264, only had two transforms of size, four by four and eight by eight. And these were integer transforms, which are only bit shifts and adds, took no multiplies and no divides. The division in fact got incorporated into the quantizer and as a result, it was very, very fast. Moreover, if you had to do, make decisions such as inter versus intra mode, the intra modes there were only about eight or 10 intra modes in H.264. By contrast in H.265. We have not two transforms eight, four by four and eight by, but in fact sizes of four, eight, 16 and 32. So we have much larger sized transforms and instead of a eight or 10 intra modes, we jumped up to 35 intra modes. Pankaj Topiwala: 26:36 And then with a VVC we jumped up to 67 intro modes and we just, it just became so much more complex. The compression ratio between HEVC and VVC is not quite two to one, but let's say, you know, 40% better. But the the complexity is not 40% more. On the ground and nobody has yet, to my knowledge, built a a, a, a fully compliant and powerful either software or hardware video codec for VVC yet because it's not even finished yet. It's going to be finished in July 2020. When it, when, the dust finally settles maybe four or five years from now, it will be, it will prove to be at least three or four times more complex than HEVC encoder the decoder, not that much. The decoder, luckily we're able to build decoders that are much more linear than the encoder. Pankaj Topiwala: 27:37 So I guess I should qualify as discussion saying the complexity growth is all mostly been in the encoder. The decoder has been a much more reasonable. Remember, we are always relying on this principle of ever-increasing compute capability. You know, a factor of two every 18 months. We've long heard about all of this, you know, and it is true, Moore's law. If we did not have that, none of this could have happened. None of this high complexity codecs, whatever had been developed because nobody would ever be able to implement them. But because of Moore's law we can confidently say that even if we put out this very highly complex VVC standard, someday and in the not too distant future, people will be able to implement this in hardware. Now you also asked a very good question earlier, is there a limit to how much we can compress? Pankaj Topiwala: 28:34 And also one can ask relatively in this issue, is there a limit to a Moore's law? And we've heard a lot about that. That may be finally after decades of the success of Moore's law and actually being realized, maybe we are now finally coming to quantum mechanical limits to you know how much we can miniaturize in electronics before we actually have to go to quantum computing, which is a totally different you know approach to doing computing because trying to go smaller die size. Well, we'll make it a unstable quantum mechanically. Now the, it appears that we may be hitting a wall eventually we haven't hit it yet, but we may be close to a, a physical limit in die size. And in the observations that I've been making at least it seems possible to me that we are also reaching a limit to how much we can compress video even without a complexity limit, how much we can compress video and still obtain reasonable or rather high quality. Pankaj Topiwala: 29:46 But we don't know the answer to that. And in fact there are many many aspects of this that we simply don't know. For example, the only real arbiter of video quality is subjective testing. Nobody has come up with an objective video quality metric that we can rely on. PSNR is not it. When, when push comes to shove, nobody in this industry actually relies on PSNR. They actually do subjective testing well. So in that scenario, we don't know what the limits of visual quality because we don't understand human vision, you know, we try, but human vision is so complicated. Nobody can understand the impact of that on video quality to any very significant extent. Now in fact, the first baby steps to try to understand, not explicitly but implicitly capture subjective human video quality assessment into a neural model. Those steps are just now being taken in the last couple of years. In fact, we've been involved, my company has been involved in, in getting into that because I think that's a very exciting area. Dror Gill: 30:57 I tend to agree that modeling human perception with a neural network seems more natural than, you know, just regular formulas and algorithms which are which are linear. Now I, I wanted to ask you about this process of, of creating the codecs. It's, it's very important to have standards. So you encode a video once and then you can play it anywhere and anytime and on any device. And for this, the encoder and decoder need to agree on exactly the format of the video. And traditionally you know, as you pointed out with all the history of, of development. Video codecs have been developed by standardization bodies, MPEG and ITU first separately. And then they joined forces to develop the newest video standards. But recently we're seeing another approach to develop codecs, which is by open sourcing them. Dror Gill: 31:58 Google started with an open source code, they called VP9 which they first developed internally. Then they open sourced it and and they use it widely across their services, especially in, YouTube. And then they joined forces with the, I think the largest companies in the world, not just in video but in general. You know those large internet giants such as Amazon and Facebook and and Netflix and even Microsoft, Apple, Intel have joined together with the Alliance of Open Media to jointly create another open codec called AV1. And this is a completely parallel process to the MPEG codec development process. And the question is, do you think that this was kind of a one time effort to, to to try and find a, or develop a royalty free codec, or is this something that will continue? And how do you think the adoption of the open source codecs versus the committee defined codecs, how would that adoption play out in the market? Pankaj Topiwala: 33:17 That's of course a large topic on its own. And I should mention that there have been a number of discussions about that topic. In particular at the SPIE conference last summer in San Diego, we had a panel discussion of experts in video compression to discuss exactly that. And one of the things we should provide to your listeners is a link to that captured video of the panel discussion where that topic is discussed to some significant extent. And it's on YouTube so we can provide a link to that. My answer. And of course none of us knows the future. Right. But we're going to take our best guesses. I believe that this trend will continue and is a new factor in the landscape of video compression development. Pankaj Topiwala: 34:10 But we should also point out that the domain of preponderance use preponderant use of these codecs is going to be different than in our traditional codecs. Our traditional codecs such as H.264 265, were initially developed for primarily for the broadcast market or for DVD and Blu-ray. Whereas these new codecs from AOM are primarily being developed for the streaming media industry. So the likes of Netflix and Amazon and for YouTube where they put up billions of user generated videos. So, for the streaming application, the decoder is almost always a software decoder. That means they can update that decoder anytime they do a software update. So they're not limited by a hardware development cycle. Of course, hardware companies are also building AV1. Pankaj Topiwala: 35:13 And the point of that would be to try to put it into handheld devices like laptops, tablets, and especially smartphones. But to try to get AV1 not only as a decoder but also as an encoder in a smartphone is going to be quite complicated. And the first few codecs that come out in hardware will be of much lower quality, for example, comparable to AVC and not even the quality of HEVC when they first start out. So that's... the hardware implementations of AV1 that work in real time are not going to be, it's going to take a while for them to catch up to the quality that AV1 can offer. But for streaming we, we can decode these streams reasonably well in software or in firmware. And the net result is that, or in GPU for example, and the net result is that these companies can already start streaming. Pankaj Topiwala: 36:14 So in fact Google is already streaming some test streams maybe one now. And it's cloud-based YouTube application and companies like Cisco are testing it already, even for for their WebEx video communication platform. Although the quality will not be then anything like the full capability of AV1, it'll be at a much reduced level, but it'll be this open source and notionally, you know, royalty free video codec. Dror Gill: 36:50 Notionally. Yeah. Because they always tried to do this, this dance and every algorithm that they try to put into the standard is being scrutinized and, and, and they check if there are any patents around it so they can try and keep this notion of of royalty-free around the codec because definitely the codec is open source and royalty free. Dror Gill: 37:14 I think that is, is, is a big question. So much IP has gone into the development of the different MPEG standards and we know it has caused issues. Went pretty smoothly with AVC, with MPEG-LA that had kind of a single point of contact for licensing all the essential patents and with HEVC, that hasn't gone very well in the beginning. But still there is a lot of IP there. So the question is, is it even possible to have a truly royalty free codec that can be competitive in, in compression efficiency and performance with the codec developed by the standards committee? Pankaj Topiwala: 37:50 I'll give you a two part answer. One because of the landscape of patents in the field of video compression which I would describe as being, you know very, very spaghetti like and patents date back to other patents. Pankaj Topiwala: 38:09 And they cover most of the, the topics and the most of the, the tools used in video compression. And by the way we've looked at the AV1 and AV1 is not that different from all the other standards that we have. H.265 or VVC. There are some things that are different. By and large, it resembles the existing standards. So can it be that this animal is totally patent free? No, it cannot be that it is patent free. But patent free is not the same as royalty free. There's no question that AV1 has many, many patents, probably hundreds of patents that reach into it. The question is whether the people developing and practicing AV1 own all of those patents. That is of course, a much larger question. Pankaj Topiwala: 39:07 And in fact, there has been a recent challenge to that, a group has even stood up to proclaim that they have a central IP in AV1. The net reaction from the AOM has been to develop a legal defense fund so that they're not going to budge in terms of their royalty free model. If they do. It would kill the whole project because their main thesis is that this is a world do free thing, use it and go ahead. Now, the legal defense fund then protects the members of that Alliance, jointly. Now, it's not as if the Alliance is going to indemnify you against any possible attack on IP. They can't do that because nobody can predict, you know, where somebody's IP is. The world is so large, so many patents in that we're talking not, not even hundreds and thousands, but tens of thousands of patents at least. Pankaj Topiwala: 40:08 So nobody in the world has ever reviewed all of those patent. It's not possible. And the net result is that nobody can know for sure what technology might have been patented by third parties. But the point is that because such a large number of powerful companies that are also the main users of this technology, you know, people, companies like Google and Apple and Microsoft and, and Netflix and Amazon and Facebook and whatnot. These companies are so powerful. And Samsung by the way, has joined the Alliance. These companies are so powerful that you know, it would be hard to challenge them. And so in practice, the point is they can project a royalty-free technology because it would be hard for anybody to challenge it. And so that's the reality on the ground. Pankaj Topiwala: 41:03 So at the moment it is succeeding as a royalty free project. I should also point out that if you want to use this, not join the Alliance, but just want to be a user. Even just to use it, you already have to offer any IP you have in this technology it to the Alliance. So all users around the world, so if tens of thousands and eventually millions of you know, users around the world, including tens of thousands of companies around the world start to use this technology, they will all have automatically yielded any IP they have in AV1, to the Alliance. Dror Gill: 41:44 Wow. That's really fascinating. I mean, first the distinction you made between royalty free and patent free. So the AOM can keep this technology royalty free, even if it's not patent free because they don't charge royalties and they can help with the legal defense fund against patent claim and still keep it royalty free. And, and second is the fact that when you use this technology, you are giving up any IP claims against the creators of the technology, which means that if any, any party who wants to have any IP claims against the AV1 encoder cannot use it in any form or shape. Pankaj Topiwala: 42:25 That's at least my understanding. And I've tried to look at of course I'm not a lawyer. And you have to take that as just the opinion of a video coding expert rather than a lawyer dissecting the legalities of this. But be that as it may, my understanding is that any user would have to yield any IP they have in the standard to the Alliance. And the net result will be if this technology truly does get widely used more IP than just from the Alliance members will have been folded into into it so that eventually it would be hard for anybody to challenge this. Mark Donnigan: 43:09 Pankaj, what does this mean for the development of so much of the technology has been in has been enabled by the financial incentive of small groups of people, you know, or medium sized groups of people forming together. You know, building a company, usually. Hiring other experts and being able to derive some economic benefit from the research and the work and the, you know, the effort that's put in. If all of this sort of consolidates to a handful or a couple of handfuls of, you know, very, very large companies, you know, does that, I guess I'm, I'm asking from your view, will, will video and coding technology development and advancements proliferate? Will it sort of stay static? Because basically all these companies will hire or acquire, you know, all the experts and you know, it's just now everybody works for Google and Facebook and Netflix and you know... Or, or do you think it will ultimately decline? Because that's something that that comes to mind here is, you know, if the economic incentives sort of go away, well, you know, people aren't going to work for free! Pankaj Topiwala: 44:29 So that's of course a, another question and a one relevant. In fact to many of us working in video compression right now, including my company. And I faced this directly back in the days of MPEG-2. There was a two and a half dollar ($2.50) per unit license fee for using MPEG-2. That created billions of dollars in licensing in fact, the patent pool, MPEG-LA itself made billions of dollars, even though they took only 10% of the proceeds, they already made billions of dollars, you know, huge amounts of money. With the advent of H.264 AVC, the patent license went not to from two and a half dollars to 25 cents a unit. And now with HEVC, it's a little bit less than that per unit. Of course the number of units has grown exponentially, but then the big companies don't continue to pay per unit anymore. Pankaj Topiwala: 45:29 They just pay a yearly cap. For example, 5 million or 10 million, which to these big companies is is peanuts. So there's a yearly cap for the big companies that have, you know, hundreds of millions of units. You know imagine the number of Microsoft windows that are out there or the number of you know, Google Chrome browsers. And if you have a, a codec embedded in the browser there are hundreds of millions of them, if not billions of them. And so they just pay a cap and they're done with it. But even then, there was up till now an incentive for smart engineers to develop exciting new ideas in a future video coding. But, and that has been up the story up till now. But when, if it happens that this AOM model with AV1 and then AV2, really becomes a dominant codec and takes over the market, then there will be no incentive for researchers to devote any time and energy. Pankaj Topiwala: 46:32 Certainly my company for example, can't afford to you know, just twiddle thumbs, create technologies for which there is absolutely no possibility of a royalty stream. So we, we cannot be in the business of developing video coding when video coding doesn't pay. So the only thing that makes money, is Applications, for example, a streaming application or some other such thing. And so Netflix and, and Google and Amazon will be streaming video and they'll charge you per stream but not on the codec. So that that's an interesting thing and it certainly affects the future development of video. It's clear to me it's a negative impact on the research that we got going in. I can't expect that Google and Amazon and Microsoft are going to continue to devote the same energy to develop future compression technologies in their royalty free environment that companies have in the open standards development technology environment. Pankaj Topiwala: 47:34 It's hard for me to believe that they will devote that much energy. They'll devote energy, but it will not be the the same level. For example, in developing a video standards such as HEVC, it took up to 10 years of development by on the order of 500 to 600 experts, well, let's say four to 500 experts from around the world meeting four times a year for 10 years. Mark Donnigan: 48:03 That is so critical. I want you to repeat that again. Pankaj Topiwala: 48:07 Well, I mean so very clearly we've been putting out a video codec roughly on the schedule of once every 10 years. MPEG-2 was 1994. AVC was 2003 and also 2004. And then HEVC in 2013. Those were roughly 10 years apart. But VVC we've accelerated the schedule to put one out in seven years instead of 10 years. But even then you should realize that we had been working right since HEVC was done. Pankaj Topiwala: 48:39 We've been working all this time to develop VVC and so on the order of 500 experts from around the world have met four times a year at all international locations, spending on the order of $100 million per meeting. You know so billions of dollars have been spent by industry to create these standards, many billions and it can't happen, you know without that. It's hard for me to believe that companies like Microsoft, Google, and whatnot, are going to devote billions to develop their next incremental, you know, AV1and AV2 AV3's. But maybe they will it just, that there's no royalty stream coming from the codec itself, only the application. Then the incentive, suppose they start dominating to create even better technology will not be there. So there really is a, a financial issue in this and that's at play right now. Dror Gill: 49:36 Yeah, I, I find it really fascinating. And of course, Mark and I are not lawyers, but all this you know, royalty free versus committee developed open source versus a standard those large companies who some people fear, you know, their dominance and not only in video codec development, but in many other areas. You know, versus you know, dozens of companies and hundreds of engineers working for seven or 10 years in a codec. So you know, it's really different approaches different methods of development eventually to approach the exact same problem of video compression. And, and how this turns out. I mean we, we cannot forecast for sure, but it will be very interesting, especially next year in 2020 when VVC is ratified. And at around the same time, EVC is ratified another codec from the MPEG committee. Dror Gill: 50:43 And then AV1, and once you know, AV1 starts hitting the market. We'll hear all the discussions of AV2. So it's gonna be really interesting and fascinating to follow. And we, we promise to to bring you all the updates here on The Video Insiders. So Pankaj I really want to thank you. This has been a fascinating discussion with very interesting insights into the world of codec development and compression and, and wavelets and DCT and and all of those topics and, and the history and the future. So thank you very much for joining us today on the video insiders. Pankaj Topiwala: 51:25 It's been my pleasure, Mark and Dror. And I look forward to interacting in the future. Hope this is a useful for your audience. If I can give you a one parting thought, let me give this... Pankaj Topiwala: 51:40 H.264 AVC was developed in 2003 and also 2004. That is you know, some 17 years or 16 years ago, it is close to being now nearly royalty-free itself. And if you look at the market share of video codecs currently being used in the market, for example, even in streaming AVC dominates that market completely. Even though VP8 and VP9 and VP10 were introduced and now AV1, none of those have any sizeable market share. AVC currently dominates from 70 to 80% of that marketplace right now. And it fully dominates broadcast where those other codecs are not even in play. And so they're 17, 16, 17 years later, it is now still the dominant codec even much over HEVC, which by the way is also taking an uptick in the last several years. So the standardized codecs developed by ITU and MPEG are not dead. They may just take a little longer to emerge as dominant forces. Mark Donnigan: 52:51 That's a great parting thought. Thanks for sharing that. What an engaging episode Dror. Yeah. Yeah. Really interesting. I learned so much. I got a DCT primer. I mean, that in and of itself was a amazing, Dror Gill: 53:08 Yeah. Yeah. Thank you. Mark Donnigan: 53:11 Yeah, amazing Pankaj. Okay, well good. Well thanks again for listening to the video insiders, and as always, if you would like to come on this show, we would love to have you just send us an email. The email address is thevideoinsiders@beamr.com, and Dror or myself will follow up with you and we'd love to hear what you're doing. We're always interested in talking to video experts who are involved in really every area of video distribution. So it's not only encoding and not only codecs, whatever you're doing, tell us about it. And until next time what do we say Dror? Happy encoding! Thanks everyone. 

Learn about Unified Patents hereCheck out Unified Patents Objective PAtent Landscape OPAL toolRead the Independent economic study for HEVC royaltiesShawn Ambwani LinkedIn profileRelated episode: VVC, HEVC & other MPEG codec standardsThe Video Insiders LinkedIn Group is where thousands of your peers are discussing the latest video technology news and sharing best practices. Click here to join --------------------------------------Would you like to be a guest on the show? Email: thevideoinsiders@beamr.comLearn more about Beamr-------------------------------------- TRANSCRIPT (edited slightly for readability)Narrator: 00:00 The Video Insiders is the show that makes sense of all that is happening in the world of online video as seen through the eyes of a second generation codec nerd and a marketing guy who knows what I-frames and macro blocks are. Here are your hosts, Mark Donnigan and Dror Gill. Mark Donnigan: 00:19 Well, welcome back to The Video Insiders. Dror, how you doing today? I'm doing great. How are you Mark? I am doing awesome. As always. I am super pleased to welcome Shawn Ambwani who is co-founder of Unified Patents and Shawn is gonna tell us all about what Unified Patents does and we're going to dive into, you know, just a really tremendous discussion. But Shawn, Welcome to the podcast! Shawn Ambwani: 00:46 Hey guys. Thanks Mark. Thanks for, for allowing me to participate on your wonderful podcast. I look at this as similar to 'All Things Considered' and 'How I built this', two of my favorite podcasts. Mark Donnigan: 01:01 Those are awesome podcasts by the way. What an honor? Yeah. Wow. The level that I expect you guys to be at in traffic very shortly. That's right. Well, we hope so to. Well, why don't you introduce yourself you know, and give us a quick snapshot of your background and then let's let's hear about what Unified Patents is doing. Shawn Ambwani: 01:23 It's kind of a, I have an interesting I mean some might say not so interesting, but I think it's interesting background related to this area since, you know, the first startup that I did and the second one were all related to MPEG4. So I co-founded a company called Envivio, which way back when was actually one of the original MPEG4 companies when they just had simple profile actually out there doing encoders and decoders. And then I went to a Korean company called NexStreaming, which actually still exists, which is doing encoders as well, but more for the mobile space and decoders. So it's an area I'm quite familiar with. I wasn't really being an attorney back then. Now I'm kind of more of an attorney than I was back then, but I tried to avoid being an attorney as much as possible in general. Shawn Ambwani: 02:16 And basically I helped co-found a company called Unified Patents. And what unified patents does is it gets contributions from member companies as well as it allows small companies to join for free and they participate in joining what we call zones. And these different zones are intended to protect against what we consider unsubstantiated or invalid patent assertions. And the goal of these zones is to deter those from occurring in the first place. So if you imagine the kind of a technology area, let it be content or let it be video codec in this case or other things as having a bunch of companies that have a common interest in maintaining, you know, patents and ensuring patents that are asserted in that space are valid, which means that no one invented the idea beforehand. And also that it's fairly priced and you know, people are explaining the rationale behind what they're doing and they're not basically just attempting to get people to settle out, not because the assertions are valid or good, but simply because the cost of litigation is so high when it comes to patents. Shawn Ambwani: 03:35 And we want to deter that type of activity because there's been a lot of investment in that activity so far. In fact, most litigation's are by NPE's and so Unified started by doing those zones and, and, and we've have a bunch of them now. We just launched an open source zone in fact, but with you know, Linux foundation and OIN and IBM and others and the video codec zone was something that we were thinking about for a long time. It's something that I'm very familiar with from my past dealings with MPEG LA and other pools. And it was a big issue I think. And it has been a constant issue, which is how do you deal with multiple pools or multiple people asking for money in a standard? How do you deal with the pricing of it? Especially if you're smaller entities and you don't have the information that may be larger companies might have. Shawn Ambwani: 04:26 How do you deal with that and how do you deal with all the invalid assertions that are being done or declarations that occur in this area? How do you figure out who you have to pay? And how much you have to pay. All these add a level of complexity to deploying these standards, which makes adoption harder and creates the uncertainty that causes people to go to proprietary solutions, which I think is a negative in the end. So that's why Unified Patents really created this area and created the video codec zone. And basically we've been pretty, I think, successful so far now actually going through and doing each one of the things we said we were going to do. Dror Gill: 05:06 So what, what are those things basically when you set up a zone and, and want to start finding those patterns that may be invalid how do you go about doing that? What is the process? Shawn Ambwani: 05:18 Yeah, so I mean there's two major things in the SEP zones that it's not, it's not just about finding invalid patents, although I can tell you it's relatively easy to find invalid patents in any of these zones. That's not a difficulty. The hard part is figuring out which ones to go to or which ones are going to be the most interesting to go after. And that takes a lot of art. Essentially identifying them, finding out good prior art that we feel comfortable with, hiring good counsel. There's all kinds of weighing mechanisms that go into it, who the entity is, how it came about, how old the patents are, where it came from. All of these variables go into that kind of equation of when we decide. What's kind of unique about the way we work is we work independently of our members, so our members are funding these activities and some join for free. Shawn Ambwani: 06:11 So we have a number of members of the video codecs and we use all this money and information in our activities to basically go back and decide what to do, with the objective of deterring you know, what we consider bad assertions in this space. And then that's one part of it. The other part of it is that SEP's are all about, you know, an area called FRAND, fair, reasonable, non-discriminatory. And part of all of that involves negotiation. And so what we provide are tools to allow companies to negotiate we think in a fair and more transparent way to licensors as to pricing. But also explaining why the pricing is the way it is. Because one of the problems that we've had in the big picture is that a lot of these licensors have been asking for money, whether their own pools, whether outside of pools, whatever. Shawn Ambwani: 07:12 But no one can really explain why the price is what it is. And I think that leads to a lot of people to just stop paying or stop wanting to get into licensing discussions. And that's not beneficial for the market. And so by explaining how the price comes out the way it is and providing a very, we consider, solid methodology for it, it allows our members but also licensors to better understand who owns what and how much value is in the standard. So what they should reasonably expect to get for that technology and how much licensees reasonably should expect to pay in order to deploy the technology. Mark Donnigan: 07:55 Now my question, you know, Shawn is when you are getting into these conversations with the parties or party that, you know, owns this IP and I'm speaking more around sort of the pricing and the model and that sort of thing. Are you then...is that information available to your members or is it more that you're sort of helping facilitate, helping bring some rationality, you know, so that then that body can turn around and make public: "Hey, great news!" We've decided that all digitally distributed content doesn't carry, you know, a royalty cost. What exactly, I guess what my question is, what exactly is, is your role then in informing the market? Shawn Ambwani: 08:40 I think that, well, I mean there's a number of things to talk about, but what's I think most important is that we, you know, we don't know necessarily what the right price is. We hired an outside economist to look into that and he came back with a pricing range in you know, a report that we gave the highlights too and there was some press over and it's on our website but you can also look at it through a number of particles and basically he came back with a price of between 8 cents and 28 cents I believe if I'm accurate. Is what he believed the estimate to be for the value of the technology including everything. And it ranged based on I think the device and like other factors and stuff like that. Now that high level information we provided publicly and in fact we provided the information on who made the report when it was created and what it was based on. Shawn Ambwani: 09:38 And we even provided kind of the overall methodology of how it was done, which is basically being used at a very high level. They used MPEG LA's AVC license as the starting point or the foundation for deciding what HEVC in this case, which is what he was looking at, pricing should be based on his expert analysis. And then he modified that based on switching costs based on the cost of bandwidth, the cost of storage and quality and other factors basically that are valuable. So, that's where we went. Now, what's important to understand is that we published that information so anyone could take a look at the, at the high level. And the methodology pretty much tells you the roadmap of where we started and how we ended up where we are. The other part is how do you decide who you have to pay and how much each person gets. Even assuming that you figure out that, let's say it's 25 cents, that you think the royalty rate should be for it. And I'm not saying that's the number, but everyone can decide on whatever number they feel comfortable. Our expert created this report and we published it. Other people can create other reports and I'm sure they have their own kind of versions. But what's important for us is that, you know, people should explain why they came out with their pricing. And unfortunately in pools and licensing organizations in general, that just doesn't happen. Dror Gill: 11:05 So basically you're finding economical reason behind a certain price for for this technology. In this case, HEVC. And now companies who want to use HEVC, how do they use this number? Because they have your number, which is the total, and then they have royalty rates that are asked that, you know, certain patent pools are asking and they add up to a different number that could be a higher number. So do they just you know, divide the number that they think is the right one among the different patent pools and pay them the amount they think they should pay or do they just use it as a negotiating tool when they talk to them and, and you know, and negotiate the actual world, the rates that they will have to pay? Shawn Ambwani: 11:52 By providing a lot of this information. Some of it publicly like economic report in some format. The hope is that smaller entities instead of rolling over when licensing people come by and say, Hey, take it or leave it, they really have an ability to make a fair response, a good faith response with information that allows them to then basically justify why they came up with a price and really push back and say, listen, you know, this is what my methodology came out to. Now. It could be right, could be wrong. You know, in the end in FRAND negotiations, I have to make a good faith offer. That's really the intent. So that's an important aspect of pushing back on this kind of, we think less information that is occurring in the marketplace and more fragmentation. And I think they're all interrelated because of the less information you have more fragmentation. Cause if everyone could agree on a price and everyone agreed that this is the fair value for the technology, there really wouldn't be multiple pools in my opinion or multiple licensors, because everyone would know what the number is. And so why would you separate? Dror Gill: 13:07 But basically you're saying that even if a patent pool set, the royalty rates and those royalty rates in some cases are public, at least for some of the patent pools, this is not what a licensor would pay. This is just kind of a starting point for a negotiation and you're providing tools for this type of negotiation. Shawn Ambwani: 13:23 We also think that validity is a big issue because none of these entities look at validity when they're incorporating patents into their pools or into their licensing. It's really up to the licensees or the people who are potentially taking the license to have the responsibility to go out and figure that out, which can be very, very costly. Dror Gill: 13:44 You assume they're valid, right? If they're licensing patents to you, you assume that they're licensing valid patents. Right? These are kind of, you know, respectable patent holders and patent pools. Why would they license something that's not valid? Shawn Ambwani: 13:57 I mean, it's a great point. I mean, the argument would be that they want to license patents. Mark Donnigan: 14:03 That's their business at the end of the day. Yeah. Shawn Ambwani: 14:08 Right. So, you know, if you had a car and you're trying to sell a car, you're going to accentuate the good things about the car. Not that it's a rebuilt or something like that or you know, like it's, it's been, you know, it's been in a crash or accident like, yeah. Like you're going to show what you want to show. Right. And that's natural in any of these cases. The unfortunate fact is that it's very costly to figure out that stuff and there's no really organization you'd think a licensing organization like MPEG LA or others. And I'm not saying MPEG LA is doing a bad job necessarily, I'm just pointing them out as an example, would do a better job of vetting to some degree on that type of activity. But they don't, and I think there's a number of reasons Mark Donnigan: 14:53 Why do they want to do that? I almost liken this to the 500 channel cable bundle of which there's about 15 high quality channels and there's 485 that are anywhere from just a, you know, not, not relevant, not interesting to, you know, to even lower quality than that, but, but you know, but Hey, I got a 500 channel bundle, right? So I feel like, wow, it must be worth $100 a month, you know, or whatever. Shawn Ambwani: 15:23 The idea that that licensing organizations like MPEG LA or (HEVC) Advance or other ones like that aren't doing it to the benefit of their licensors. It just seems ridiculous to me. I mean the people on their, on their management and the people who are actually owning that organization, typically it's managed and owned and administrative fees are paid to licensors. And traditionally the money flows one way from licensees to licensors. It's for the benefit of the licensors. And the rules that they put in are essentially to make sure that those guys are protected. They have no incentive in general of saying people's patents are invalid. And, and that's just a bad fact pattern for them. If basically they get back and say, Hey, listen this patent... Yeah, no, it's bad. Mark Donnigan: 16:16 Exactly. So, so in that context then it completely makes sense that they don't vet you know, at the level that you are and why, you know, Unified Patents needs to exist, you know, is because we need this sort of independent third party. I guess. I, I, you know, that's, that's out there doing this work. Now, Shawn, one of the things that I noticed is you're acting both against NPE's, so, non-practicing entities, and against SEP's. So standard essential patents. What are the issues with SEP's? Shawn Ambwani: 16:51 Well, I mean the general assumption has been, and I don't know where this assumption came from, was that standard essential patents or people who declare their patents to be standard essential are more likely to be valid than other patents. And in the real world where there's litigation and there's challenges and things get checked out or vetted essentially, adversarially, the reality is that standard essential patents in all the studies that have been done fair, far worse, than normal patents do on average. And you know, it's not shocking actually when you think about it. Obviously there's a lot of self selection here, but part of the reason why is, you know, when you're submitting into pools or in when you're getting these patents, when part of a standardization body or doing other activities, there's a lot of other people involved and it's usually built on other ideas that people have had in the past. Shawn Ambwani: 17:59 And it's not surprising that a lot of these patents have underlying ideas that had been done in the past or other people had brought up previously. Sometimes they weren't accepted, sometimes they were or sometimes they were put on hold. Who knows? But there's a lot of prior art oftentimes in these areas. These aren't open fields, these aren't brand new innovations that typically come up. And so that's not surprising. Now, you know, there's also a general belief that standard essential patterns are more valuable. And I think, you know, that's a pretty, I would say, you know, I dunno if it's absolutely valid, but it's not unreasonable to believe that if you declare a patent, as standardized, if you look at the average patent and compared to that patent, it's probably your, it's probably more valuable, at that point. Because you basically said it's part of a standard that people are probably going to adopt at that point versus a patent in general, which you never know most of the time, whether anyone's going to use that patent. Shawn Ambwani: 19:02 I mean the vast majority of patents are never actually used in any way whatsoever. They're not enforceable because they're just ideas that people have most of the time, and these patents are arguable more likely than not to be in a standard and that standard might or might not actually get used in the end. Inherently you get - they're more valuable. The problem is there's tons of over declaration that occurs in this area. There's very little incentive. I mean some places there's more of an incentive than others, but the way MPEG works specifically is that you can do blanket declarations and so you don't have to declare specific patents. And, other standards, you have to basically declare each individual patent that you have. So, I mean, there's all kinds of trade offs, and all these different things, but the reality is that no one really knows exactly how many patents need to be licensed. And that just creates a lot of uncertainty. And you know, a lot of companies who are trying to make money, not off products but off of doing licensing thrive on uncertainty because that's where they can make money. Is basically by, you know, saying, okay, well who knows what can happen, but if you take care of me now, I can make sure that I'm not going to cause you issues. Dror Gill: 20:23 Right. And that's why uncertainty is in the middle of FUD, fear, uncertainty, doubt, which is one of those tactics and uncertainty is definitely a big part of that. Shawn Ambwani: 20:33 Yeah. I mean, the other thing is that companies in general, it seems like a one way street a lot of the time, which is pretty unfortunate in that although I'm not sure if I have a good solution, you know, a lot of companies, the licensors have a way of getting together, agreeing on a price and then licensing through an organization like MPEG LA or others to do that type of activity or Velos (Media), or whatever it is. They choose, you know, they can select a price, they can work together, agree on a price. And the reason why they can do that according to the DOJ is because it's a different product than what's available before. So it decreases uncertainty by making it easier for people to take a license of convenience for that specific technology area. Dror Gill: 21:21 Otherwise, it might might've been considered price setting, Shawn Ambwani: 21:24 Right? Yeah. It would be considered price. It would be considered price setting. But in this case, the argument is always that you can always go to each individual company and get a license or negotiate a separate license. This is a license of convenience for this technology area from all these companies for one price. And that makes it a lot easier for people on both sides to be able to know exactly how much they're going to be getting and how much they're going to have to pay for clearing this risk. Which makes sense. I fundamentally have no problem with pools and what they do. The, the issue comes up is that a lot of these pools, A) don't talk about the pricing, they don't look at the validity. They don't really have a great essentially checking on top of it. And they're very much incentivized to help out the licensors, not the licensees figure stuff out. And what ends up happening is over time you kind of, and you have companies also that are not interested in making products, which is unfortunate. They're just interested in making money off of their licensing. Which is unfortunate because there's a lot of games that can be played in the standardization world to get your stuff in and then get your patents in basically. Mark Donnigan: 22:44 Well, it ultimately, it, it stops innovation. I mean, at the end of the day, you know, and one thing, and Dror and I have talked about this on episodes and we've certainly talked about this a lot, you know, privately within Beamr is, you know, it's a little bit mystifying as well because okay, so HEVC clearly was set back as a result of, of many issues. But you know, largely what we'd been talking about for the last 35 minutes and the adoption of HEVC. And yet these people, as you point out, the licensors, they don't make money if nobody's using the technology. So, so what's mystifying to us is that this is not, you know, it's not like somehow they're getting paid still. You know, even though the adoption of the technology is not in place or it's not being used, they're not getting paid. And so it seems like at some point, you know, a rational actor would stand up and say, wait a second, I'd rather get something rather than nothing! But, it's almost like they, they're not acting that way. Dror Gill: 23:46 But, but it did happen. They did reduce the royalty rate. Yes, yes, yes. Certainly. And they did come to their senses and they did put a cap and then initially it was uncapped and they did remove royalties from content. And you know, they did a lot of things in the right direction after the pressure from the market when they realized they're not going to get anything. And when AV1 started to happen, you know, and they were pressured by that, by a competing codec that was supposedly a royalty free and didn't have these issues. So I think the situation is improved. But you've launched a specific zone. It's called the video codec zone, but basically right now it deals only with HEVC. Shawn Ambwani: 24:33 A lot of these patents that we've challenged relate not just to HEVC but potentially to AV and other codecs like AVC as well. Cause there's such overlap between these things. That's why we generically call it a video codec zone. So, obviously a lot of the things that we've looked at in like the economic report and everything else and landscape, a lot of the focus has been on HEVC. Dror Gill: 24:59 So you examined the HEVC and and you saw this situation that you have three patent pools. One of them hasn't even announced the royalty rates and, and you have a lot of independent patent holders who claim to have standard essential patents for HEVC. And this is kind of your, you're opening a, a situation. So what, what was the first thing that, that you did, how did you start to, to approach the HEVC pattern topic and what actions did you take? Shawn Ambwani: 25:34 Like I said, we've done a bunch of different stuff. We had a submission repo called open, which where we collated all the prior art, not prior art, but submissions into the standard for HEVC and AVC and other standards from MPEG so people can make it easily searchable. In fact, 50% of the priority art that we got for our patent challenges came from the submission repo, which is great, which is basically, you know, previous submissions to the same standard. We have OPAL, which is our landscape tool. And then, you know, obviously we have OPEN which is our evaluation report that I mentioned for HEVC. And then we did a bunch of reviews of validity and challenged a bunch of patents in different licensing entities. I mean, Velos, I think they don't consider themselves a pool. Just to be clear. Dror Gill: 26:29 Because they actually own the patents. They've licensed those patents on their own? Shawn Ambwani: 26:34 Well, I think they just don't consider themselves tackling a patent pool in the way that MPEG LA and HEVC Advanced does simply that would throw them into a different bucket and they would have all kinds of requirements on them that they don't want basically. So you know when the DOJ kind of made the rules or kind of the lawyers decided what the right rules are to make it work, you know like you've got to show your stuff. Basically you got to show your price, you've got to make sure it's reasonable or it's, you know, like there's, there's no most favored nation clause. I mean there is a most favorite nation (MFN) let me rephrase this. So all these things to make sure that everything is very transparent in order to allow this kind of companies to get together and set a price for how much they want to license for it, which typically would have huge anti-competitive or antitrust issues. Right. They made all these rules and Velos I think would not consider themselves technically a patent pool like those guys because that would make them have the similar requirements. Dror Gill: 27:40 So they're like an independent patent holder? Shawn Ambwani: 27:42 I don't really know what they call themselves. I've definitely never heard them say that they're a patent pool. I've heard other people call them a patent pool. I probably have at some point, but I don't really know if they actually consider themselves a patent pool Dror Gill: 27:55 Because I noticed that your litigation was against the patent holders. Companies like GE and KBS and against Velos Media itself. Yeah. Shawn Ambwani: 28:07 Yeah. Well Velos is you know, an unusual beast in that it owns a number of patents that got transferred to it as well as it provides licenses to the people who participated. You know, the other patent holders in general are much more traditional in their patent pool type activity in that the patent holders are different from the people who are doing the licensing. Dror Gill: 28:28 And you're not suing the patent pools like MPEG LA and HEVC Advanced or not your targets? Shawn Ambwani: 28:32 Well, they don't own patents directly, so really nothing to do as far as I know. I mean, you could say, you know, part of it is we're challenging them to a certain degree on their pricing and kind of their whole model of not looking at validity by challenging some of their patents as well as, you know, putting them on notice that as they get more patents in, we might challenge further patents for validity. So why don't they do it ahead of time? I mean, the idea that, you know, validity is a victimless crime if you don't check for validity, it doesn't hurt anyone. It's just not true in my opinion. It's just not true because you are hurting the people who actually innovated. There's a set amount of money that goes to everyone. If you have a bunch of patents and they're just like, you're checking for essentiality before you allow a patent in, you check for validity because there's a bunch of patents that just aren't valid that shouldn't be, they should not be making money off of. It just incentivizes people to get more invalid patents in the same space that they can stick into a pool to get a bigger share of it, like a giant game. Right? Mark Donnigan: 29:43 Yeah, that's a really good point. I'm wondering what is the cost to test for essentiality? Is some of this just sort of practical like it's just either too time consuming or costly to test? Yeah. I mean Shawn Ambwani: 29:58 Esentiality is often times more expensive than validity in some cases, but I mean they do test for essentiality. The companies pay to have their own patents tested often times for essentiality, but there is no test for validity that they enforce. So no one actually does it. You know, if they did ask for it, I'm sure people in some cases would pay for it, but more importantly, people who didn't think their patents would be found valid, probably wouldn't submit them in the first place then. Then there would be, there'd be huge disincentive for people who had that risk of that happening. They just wouldn't submit it, which you know, obviously it's going to hurt the pool because they get less patents. And at the same time, the hope is that people will think twice before they submit stuff they know is crap. Anyway. Mark Donnigan: 30:43 So what is, what is your bar for determining low quality? I mean, what does that process look like? Shawn Ambwani: 30:52 We have a bunch of patents that come into our hopper that we're constantly looking at in every single zone that we're in and we're constantly looking and seeing if it's a valid patent or not. And there's multiple ways of doing that. We have crowd sourcing that we do for that. We just pay people, you know, in order to do prior art experts for example, to do prior art searches. You can prior art search infinitely long these, there's no stopping. You know, what you can do. But you know, in the end there's only so much you can reasonably expect to find. And so from my perspective, you know, there's definitely been situations where we've looked at patents and we've said, okay, we don't think we have good prior art. We're just not going to do anything about it. And that's okay. In fact, I mean it's okay if a licensing entity or licensor has a valid patent, that's perfectly fine with me. Shawn Ambwani: 31:49 I think if they have a valid patent, they should be able to make money off of it. I have no problem with, it should be a fair amount if it's in a standard based on FRAND principles, but in general, people should be able to make money off of a valid patent. The problem is is that a lot of people are making a lot of money, in my opinion, off of a lot of bad patents. And invalid patents, which hurts the people who actually do have good patents because they're getting crowded out, which is sad because that really is the disincentive for innovation then is when the people actually are innovating aren't making money off of it because they're getting crowded out by the people who are just playing a good game. Dror Gill: 32:25 You described earlier the, the process with a standard setting bodies such as MPEG where you declare your patents but you only, you can declare them as, as a pool or as a bunch of patents and not specifically, and then you can basically, Dror Gill: 32:40 You know, create a pool and charge as much as you want if it's under the FRAND principles. Do you think there's anything broken in the standard setting process itself? Do those committees need to do something else in order to make sure that when they create a standard, the situation of royalties of, of the situation of, of IP which is essential to that standard is more well known that you have less uncertainty in that IP? Shawn Ambwani: 33:09 Yeah, I'm not sure. I mean there's always ways of like tweaking the system. Every standards body has different ways of managing it. I mean the only really clean way of doing it is saying it's royalty free and having anyone who participates in the standard agree that it's royalty free. Anything above that, just you know, you can play all these different types of rules and machinations and 3GPP has their own and other people, organizations have their own. But in the end it ends up being the same issue of you know, under declaring over declaring - issues with essentiality, validity, all kinds of other things. So I'm not sure if you, unless you go to that binary level, how much, you know, changing that up is going to change things fundamentally. I think the more fundamental thing is that, you know, the idea that I think the fundamental reason why you have these patent pools and other things like that was to clear risk and decrease uncertainty. Unfortunately I'd say uncertainty is actually increasing in some of these cases not decreasing by all these different groups asking for money at this point, which is unfortunate. Dror Gill: 34:17 No, that's a very interesting insight, really. Mark Donnigan: 34:19 Hey, thanks for joining us, Shawn. This was really an amazing discussion and we definitely have to have a part two. Shawn Ambwani: 34:26 All right, well, thanks for your time, gentlemen. I really appreciate it. Dror Gill: 34:28 Thank you. Narrator: 34:30 Thank you for listening to The Video Insiders podcast, a production of Beamr Imaging limited. To begin using Beamr's codecs today, go to beamr.com/free to receive up to 100 hours of no cost HEVC and H.264 transcoding every month.