Podcasts about aws batch

In this episode we explore the new features of Seqera's Data Studios and Data Explorer, with Phil Ewels, Rob Newman and Rob Syme from Seqera. Discover how to use these tools for troubleshooting Nextflow pipelines, tertiary analysis and Nextflow development. We discuss the pain points that led to the creation of Data Studios and how it's designed to allow scientists to interactively and collaboratively work with data and complex workflows, without having to move large datasets around. Rob Syme wows us with another fantastic practical demonstration, setting up and using Data Studios to write and test a Nextflow pipeline in VSCode running on the cloud in a Data Studio environment, including running the Nextflow CLI with task submission to AWS Batch. We cover features like session persistence to save work states, and upcoming custom container support for your own specialized applications. Learn how these tools can enhance your computational biology projects and make seamless cloud integration a reality. 00:00 Channels Podcast 43: Data Studios 00:26 Introductions 01:54 Data Studios 04:51 Move the compute to the data 06:13 Real-time collaboration 06:47 Data Explorer 09:41 Access to public data 10:45 Data Explorer demo 13:56 Data Studios setup 20:17 Session persistance 22:52 Data Studios RStudio demo 28:24 Nextflow development in Data Studios 36:17 Future development 37:01 Custom containers 40:01 Boston Summit demo 44:01 Lifetime management 47:14 Wrap up

C'était le Summit AWS Paris cette semaine, plus de 9000 personnes ce sont croisées au Palais des congrès de Paris. AWS a annoncé la disponibilité de Bedrock sur la région eu-west-3 et du modèle Mistral Large. Dans cet épisode, je parlerai aussi de CloudFormation qui est 40% plus rapide pour déployer vos infrastructures - je vous explique comment. Il y a de l'info aussi sur AWS Batch, CodeBuild et sur InfluxDB.

C'était le Summit AWS Paris cette semaine, plus de 9000 personnes ce sont croisées au Palais des congrès de Paris. AWS a annoncé la disponibilité de Bedrock sur la région eu-west-3 et du modèle Mistral Large. Dans cet épisode, je parlerai aussi de CloudFormation qui est 40% plus rapide pour déployer vos infrastructures - je vous explique comment. Il y a de l'info aussi sur AWS Batch, CodeBuild et sur InfluxDB.

AWS Puts Up a New VPC Lattice to Ease the Growth of Your Connectivity AKA Welcome to April (how is it April already?) This week, Justin, Jonathan, and Matt are your guides through all the latest and greatest in Cloud news; including VPC Lattice from AWS, the one and only time we'll talk about Service Catalog, and an ultra premium DDoS experience. All this week on The Cloud Pod.  This week's alternate title(s): AWS Finally makes service catalogs good with Terraform Amazon continues to believe retailers with supply chain will give all their data to them Azure copies your data from S3… AWS copies your data from Azure Blobs… or how I set money on fire with data egress charges

On The Cloud Pod this week, Amazon announces Neptune Serverless, Google introduces Google Blockchain Node Engine, and we get some cost management updates from Microsoft. Thank you to our sponsor, Foghorn Consulting, which provides top notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you're having trouble hiring? Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week. General News [1:24]

Cloud Posse holds public "Office Hours" every Wednesday at 11:30am PST to answer questions on all things related to DevOps, Terraform, Kubernetes, CICD. Basically, it's like an interactive "Lunch & Learn" session where we get together for about an hour and talk shop. These are totally free and just an opportunity to ask us (or our community of experts) any questions you may have. You can register here: https://cloudposse.com/office-hoursJoin the conversation: https://slack.cloudposse.com/Find out how we can help your company:https://cloudposse.com/quizhttps://cloudposse.com/accelerate/Learn more about Cloud Posse:https://cloudposse.comhttps://github.com/cloudpossehttps://sweetops.com/https://newsletter.cloudposse.comhttps://podcast.cloudposse.com/[00:00:00] Intro[00:01:29] Spacelift Neutral Status Checks!https://github.com/cloudposse/infra-live/pull/184[00:03:35]  AWS Organizations now manages primary contact information for all accountshttps://aws.amazon.com/about-aws/whats-new/2022/10/aws-organizations-console-centrally-manage-primary-contact-information-aws-accounts/[00:04:26] AWS Batch now supports EKShttps://aws.amazon.com/about-aws/whats-new/2022/10/aws-batch-supports-amazon-eks/[00:05:38] Terraform: why data sources and filters are preferable over remote statehttps://devopsian.net/posts/terraform-data-sources-over-remote-state/[00:09:42] Advanced Terraform Manipulations: Filtering, Grouping, Transformationshttps://brendanthompson.com/posts/2022/10/terraform-for-expression [00:14:07] How are people enforcing MFA in AWS [with IAM users and not SSO]?[00:16:40] Any atmos questions from last week's demo?[00:18:43] Home Automation[00:34:54] How does someone get good at IAM? [00:58:58] Outro#officehours,#cloudposse,#sweetops,#devops,#sre,#terraform,#kubernetes,#awsSupport the show

Today on the show, the panel discusses the intersection of data and DevOps, including various topics such as data warehousing, analytics, and source data, DVC and AWS Batch, and Snowflake integrations. In this episode… Data warehousing Analytics with source data Data Version Control (DVC) AWS Batch Production database schemas  Real time data  Snowflake integrations Sponsors Top End Devs Coaching | Top End Devs Picks Jillian- iD Tech | Summer Coding Camp & Online Tech Programs for Kids & Teens Jonathan- The Timeless Way of Building Will- ShouldeRök™

Today on the show, the panel discusses the intersection of data and DevOps, including various topics such as data warehousing, analytics, and source data, DVC and AWS Batch, and Snowflake integrations. In this episode… Data warehousing Analytics with source data Data Version Control (DVC) AWS Batch Production database schemas  Real time data  Snowflake integrations Sponsors Top End Devs Coaching | Top End Devs Picks Jillian- iD Tech | Summer Coding Camp & Online Tech Programs for Kids & Teens Jonathan- The Timeless Way of Building Will- ShouldeRök™

Cosa sono i test genetici? Come si possono usare per stimare il rischio di sviluppare una particolare malattia? Cos'è il Polygenic Risk Score e quanti data occorre processare per effettuare analisi complesse sui dati genetici di un individuo? In questo episodio ospito Paolo Di Domenico, CTO di Allelica, per parlare di come Allelica utilizza servizi cloud come AWS Batch ed AWS Lambda per processare moli gigantesche di dati in parallelo, riuscendo ad offrire risultati nel giro di pochi minuti anziché di ore. Link: Allelica. Link: AWS Batch.

Gene therapy holds the promise to permanently cure diseases that have been considered life-long challenges. But the complexity of rewriting DNA is truly huge and lives in its own special kind of big-data world. On this episode, you'll meet David Born, a computational biologist who uses Python to help automate genetics research and helps move that work to production. Links from the show David on Twitter: @Hypostulate Beam Therapeutics: beamtx.com AWS Cloud Development Kit: aws.amazon.com/cdk Jupyter: jupyter.org $1,279-per-hour, 30,000-core cluster built on Amazon EC2 cloud: arstechnica.com Luigi data pipelines: luigi.readthedocs.io AWS Batch: aws.amazon.com/batch What is CRISPR?: wikipedia.org SUMMIT supercomputer: olcf.ornl.gov/summit Watch YouTube live stream edition: youtube.com Episode transcripts: talkpython.fm ---------- Stay in touch with us ---------- Subscribe on YouTube (for live streams): youtube.com Follow Talk Python on Twitter: @talkpython Follow Michael on Twitter: @mkennedy Sponsors Shortcut Talk Python Training AssemblyAI

About Gojko AdzicGojko Adzic is a partner at Neuri Consulting LLP. He one of the 2019 AWS Serverless Heroes, the winner of the 2016 European Software Testing Outstanding Achievement Award, and the 2011 Most Influential Agile Testing Professional Award. Gojko’s book Specification by Example won the Jolt Award for the best book of 2012, and his blog won the UK Agile Award for the best online publication in 2010.Gojko is a frequent speaker at software development conferences and one of the authors of MindMup and Narakeet.As a consultant, Gojko has helped companies around the world improve their software delivery, from some of the largest financial institutions to small innovative startups. Gojko specializes in agile and lean quality improvement, in particular impact mapping, agile testing, specification by example, and behavior driven development.Twitter: @gojkoadzicNarakeet: https://www.narakeet.comPersonal website: https://gojko.netWatch this video on YouTube: https://youtu.be/kCDDli7uzn8This episode is sponsored by CBT Nuggets: https://www.cbtnuggets.com/TranscriptJeremy: Hi everyone, I'm Jeremy Daly and this is Serverless Chats. Today my guest is Gojko Adzic. Hey Gojko, thanks for joining me.Gojko: Hey, thanks for inviting me.Jeremy: You are a partner at Neuri Consulting, you're an AWS Serverless Hero, you've written I think, what? I think 6,842 books or something like that about technology and serverless and all that kind of stuff. I'd love it if you could tell listeners a little bit about your background and what you've been working on lately.Gojko: I'm a developer. I started developing software when I was six and a half. My dad bought a Commodore 64 and I think my mom would have kicked him out of the house if he told her that he bought it for himself, so it was officially for me.Jeremy: Nice.Gojko: And I was the only kid in the neighborhood that had a computer, but didn't have any ways of loading games on it because he didn't buy it for games. I stayed up and copied and pasted PEEKs and POKEs in a book I couldn't even understand until I made the computer make weird sounds and print rubbish on the screen. And that's my background. Basically, ever since, I only wanted to build software really. I didn't have any other hobbies or anything like that. Currently, I'm building a product for helping tech people who are not video editing professionals create videos very easily. Previously, I've done a lot of work around consulting. I've built a lot of product that is used by millions of school children worldwide collaborate and brainstorm through mind-mapping. And since 2016, most of my development work has been on Lambda and on team stuff.Jeremy: That's awesome. I joke a little bit about the number of books that you wrote, but the ones that you have, one of them's called Running Serverless. I think that was maybe two years ago. That is an excellent book for people getting started with serverless. And then, one of my probably favorite books is Humans Vs Computers. I just love that collection of tales of all these things where humans just build really bad interfaces into software and just things go terribly.Gojko: Thank you very much. I enjoyed writing that book a lot. One of my passions is finding edge cases. I think people with a slight OCD like to find edge cases and in order to be a good developer, I think somebody really needs to have that kind of intent, and really look for edge cases everywhere. And I think collecting these things was my idea to help people first of all think about building better software, and to realize that stuff we might glance over like, nobody's ever going to do this, actually might cause hundreds of millions of dollars of damage ten years later. And thanks very much for liking the book.Jeremy: If people haven't read that book, I don't know, when did that come out? Maybe 2016? 2015?Gojko: Yeah, five or six years ago, I think.Jeremy: Yeah. It's still completely relevant now though and there's just so many great examples in there, and I don't want to spent the whole time talking about that book, but if you haven't read it, go check it out because it's these crazy things like police officers entering in no plates whenever they're giving parking tickets. And then, when somebody actually gets that, ends up with thousands of parking tickets, and it's just crazy stuff like that. Or, not using the middle initial or something like that for the name, or the birthdate or whatever it was, and people constantly getting just ... It's a fascinating book. Definitely check that out.But speaking of edge cases and just all this experience that you have just dealing with this idea of, I guess finding the problems with software. Or maybe even better, I guess a good way to put it is finding the limitations that we build into software mostly unknowingly. We do this unknowingly. And you and I were having a conversation the other day and we were talking about way, way back in the 1970s. I was born in the late '70s. I'm old but hopefully not that old. But way back then, time-sharing was a thing where we would basically have just a few large computers and we would have to borrow time against them. And there's a parallel there to what we were doing back then and I think what we're doing now with cloud computing. What are your thoughts on that?Gojko: Yeah, I think absolutely. We are I think going in a slightly cyclic way here. Maybe not cyclic, maybe spirals. We came to the same horizontal position but vertically, we're slightly better than we were. Again, I didn't start working then. I'm like you, I was born in late '70s. I wasn't there when people were doing punch cards and massive mainframes and time-sharing. My first experience came from home PC computers and later PCs. The whole serverless thing, people were disparaging about that when the marketing buzzword came around. I don't remember exactly when serverless became serverless because we were talking about microservices and Lambda was a way to run microservices and execute code on demand. And all of a sudden, I think the JAWS people realized that JAWS is a horrible marketing name, and decided to rename it to serverless. I think it most important, and it was probably 2017 or something like that. 2000 ...Jeremy: Something like that, yeah.Gojko: Something like that. And then, because it is a horrible marketing name, but it's catchy, it caught on and then people were complaining how it's not serverless, it's just somebody else's servers. And I think there's some truth to that, but actually, it's not even somebody else's servers. It really is somebody else's mainframe in a sense. You know in the '70s and early '80s, before the PC revolution, if you wanted to be a small software house or a small product operator, you probably were not running your own data center. What you would do is you would rent it based on paying for time to one of these massive, massive, massive operators. And in fact, we ended up with AWS being a massive data center. As far as you and I are concerned, it's just a blob. It's not a collection of computers, it's a data center we learn something from and Google is another one and then Microsoft is another one.And I remember reading a book about Andy Grove who was the CEO of Intel where they were thinking about the market for PC computers in the late '70s when somebody came to them with the idea that they could repurpose what became a 8080 processor. They were doing this I think for some Japanese calculator and then somebody said, "We can attach a screen to this and make this a universal computer and sell it." And they realized maybe there's a market for four or five computers in the world like that. And I think that that's ... You know, we ended up with four or five computers, it's just the definition of a computer changed.Jeremy: Right. I think that's a good point because you think about after the PC revolution, once the web started becoming really big, people started building data centers and collocation facilities like crazy. This is way before the cloud, and everybody was buying racks and Dell was getting really popular because people buying servers from Dell, and installing these in their data centers and doing this. And it just became this massive, whole industry built around doing that. And then you have these few companies that say, "Well, what if we just handled all that stuff for you? Rather than just racking stuff for you," but started just managing the software, and started managing the networking, and the backups, and all this stuff for you? And that's where the cloud was born.But I think you make a really good point where the cloud, whatever it is, Amazon or Google or whatever, you might as well just assume that that's just one big piece of processing that you're renting and you're renting some piece of that. And maybe we have. Maybe we've moved back to this idea where ... Even though everybody's got a massive computer in their pocket now, tons of compute power, in terms of the real business work that's being done, and the real global value, and the things that are powering global commerce and everything else like that, those are starting to move back to run in four, five, massive computers.Gojko: Again, there's a cyclic nature to all of this. I remember reading about the advent of power networks. Because before people had electric power, there were physical machines and movement through physical power, and there were water-powered plants and things like that. And these whole systems of shafts and belts and things like that powering factories. And you had this one kind of power load in a factory that was somewhere in the middle, and then from there, you actually have physical belts, rotating cogs in other buildings, and that was rotating some shafts that were rotating other cogs, and things like that.First of all, when people were able to package up electricity into something that's distributable, and they were running their own small electricity generators next to these big massive machines that were affecting early factories. And one of the first effects of that was they could reuse 30% of their factories better because it was up to 30% of the workspace in the factory that was taken up by all the belts and shafts. And all that movement was producing a lot of air movement and a lot of dust and people were getting sick. But now, you just plug a cable and you no longer have all this bad air and you don't have employees going sick and things like that. Things started changing quite a lot and then all of a sudden, you had this completely new revolution where you no longer had to operate your own electric generator. You could just plug in and get power from the network.And I think part of that is again, cyclic, what's happening in our industry now, where, as you said, we were getting machines. I used to make money as a Linux admin a long time ago and I could set up my own servers and things like that. I had a company in 2007 where we were operating our own gaming system, and we actually had physical servers in a physical server room with all the LEDs and lights, and bleeps, and things like that. Around that time, AWS really made it easy to get virtual machines on EC2 and I realized how stupid the whole, let's manage everything ourself is. But, we are getting to the point where people had to run their own generators, and now you can actually just plug into the electricity network. And of course, there is some standardization. Maybe U.S. still has 110 volts and Europe has 220, and we never really get global standardization there.But I assume before that, every factory could run their own voltage they wanted. It was difficult to manufacture for these things but now you have standardization, it's easier for everybody to plug into the ecosystem and then the whole ecosystem emerged. And I think that's partially what's happening now where things like S3 is an API or Lambda is an API. It's basically the electric socket in your wall.Jeremy: Right, and that's that whole Wardley maps idea, they become utilities. And that's the thing where if you look at that from an enterprise standpoint or from a small business standpoint if you're a startup right now and you are ordering servers to put into a data center somewhere unless you're doing something that's specifically for servers, that's just crazy. Use the cloud.Gojko: This product I mentioned that we built for mind mapping, there's only two of us in the whole company. We do everything from presales, to development testing supports, to everything. And we're competing with companies that have several orders of magnitude more employees, and we can actually compete and win because we can benefit from this ecosystem. And I think this is totally wonderful and amazing and for anybody thinking about starting a product, it's easier to start a product now than ever. And, another thing that's totally I think crazy about this whole serverless thing is how in effect we got a bookstore to offer that first.You mentioned the world utility. I remember I was the editor of a magazine in 2001 in Serbia, and we had licensing with IDG to translate some of their content. And I remember working on this kind of piece from I think PC World in the U.S. where they were interviewing Hewlett Packard people about utility computing. And people from Hewlett Packard back then were predicting that in a few years' time, companies would not operate their own stuff, they would use utility and things like that. And it's totally amazing that in order to reach us over there, that had to be something that was already evaluated and tested, and there was probably a prototype and things like that. And you had all these giants. Hewlett Packard in 2001 was an IT giant. Amazon was just up-and-coming then and they were a bookstore then. They were not even anything more than a bookstore. And you had, what? A decade later, the tables completely turned where HP's ... I don't know ...Jeremy: I think they bought Compaq at some point too.Gojko: You had all these giants, IBM completely missed it. IBM totally missed ...Jeremy: It really did.Gojko: ... the whole mobile and web and everything revolution. Oracle completely missed it. They're trying to catch up now but fat chance. Really, we are down to just a couple of massive clouds, or whatever that means, that we interact with as we're interacting with electricity sockets now.Jeremy: And going back to that utility comparison, or, not really a comparison. It is a utility now. Compute is offered as a utility. Yes, you can buy and generate compute yourself and you can still do that. And I know a lot of enterprises still will. I think cloud is like 4% of the total IT market or something. It's a fraction of it right now. But just from that utility aspect of it, from your experience, you mentioned you had two people and you built, is it MindMup.com?Gojko: MindMup, yeah.Jeremy: You built that with just two people and you've got tons of people using it. But just from your experience, especially coming from the world of being a Linux administrator, which again, I didn't administer ... Well, I guess I was. I did a lot of work in data centers in my younger days. But, coming from that idea and seeing how companies were building in the past and how companies are still building now, because not every company is still using the cloud, far from it. But not taking advantage of that utility, what are those major disadvantages? How badly do you think that's going to slow companies down that are trying to innovate?Gojko: I can give you a story about MindMup. You mentioned MindMup. When was it? 2018, there was the Intel processor vulnerabilities that were discovered.Jeremy: Right, yes.Gojko: I'm not entirely sure what the year was. A few years ago anyway. We got a email from a concerned university admin when the second one was discovered. The first one made all the news and a month later a second one was discovered. Now everybody knew that, they were in panic and things like that. After the second one was discovered, we got a email from a university admin. And universities are big users, they need to protect the data and things like that. And he was insisting that we tell him what our plan was for mitigating this thing because he knows we're on the cloud.I'm working on European time. The customer was in the U.S., probably somewhere U.S. Pacific because it arrived in the middle of the night. I woke up, I'm still trying to get my head around and drinking coffee and there's this whole sausage CV number that he sent me. I have no idea what it's about. I took that, pasted it into Google to figure out what's going on. The first result I got from Google was that AWS Lambda was already patched. Copy, paste, my day's done. And I assume lots and lots of other people were having a totally different conversation with their IT department that day. And that's why I said I think for products like the one I'm building with video and for the MindMup, being able to rent operations as a utility, but really totally rent ops as a utility, not have to worry about anything below my unique business level is really, really important.And yes, we can hire people to work on that it could even end up being slightly cheaper technically but in terms of my time and where my focus goes and my interruptions, I think deploying on a utility platform, whatever that utility platform is, as long as it's reliable, lets me focus on adding value where I can actually add value. That makes my product unique rather than the generic stuff.Jeremy: You mentioned the video product that you're working on too, and something that is really interesting I think too about taking advantage of the cloud is the scalability aspect of it. I remember, it was maybe 2002, maybe 2003, I was running my own little consulting company at the time, and my local high school always has a rivalry football game every Thanksgiving. And I thought it'd be really interesting if I was to stream the audio from the local AM radio station. I set up a server in my office with ReelCast Streaming or something running or whatever it was. And I remember thinking as long as we don't go over 140 subscribers, we'll be okay. Anything over that, it'll probably crash or the bandwidth won't be enough or whatever.Gojko: And that's just one of those things now, if you're doing any type of massive processing or you need bandwidth, bandwidth alone ... I remember T1 lines being great and then all of a sudden it was like, well, now you need a T3 line or something crazy in order to get the bandwidth that you need. Just from that aspect of it, the ability to scale quickly, that just seems like such a huge blocker for companies that need to order provision servers, maybe get a utility company to come in and install more bandwidth for them, and things like that. That's just stuff that's so far out of scope for building a business to me. At least building a software business or building any business. It's crazy.When I was doing consulting, I did a bit of work for what used to be one of the largest telecom companies in the world.Jeremy: Used to be.Gojko: I don't want to name names on a public chat. Somewhere around 2006, '07 let's say, we did a software project where they just needed to deploy it internally. And it took them seven months to provision a bunch of virtual machines to deploy it internally. Seven months.Jeremy: Wow.Gojko: Because of all the red tape and all the bureaucracy and all the wait for capacity and things like that. That's around the time where Amazon when EC2 became commercially available. I remember working with another client and they were waiting for some servers to arrive so they can install more capacity. And I remember just turning on the Amazon console. I didn't have anything useful to running it then but just being able to start up a virtual machine in about, I think it was less than half an hour, but that was totally fascinating back then. Here's a new Linux machine and in less than half an hour, you can use it. And it was totally crazy. Now we're getting to the point where Lambda will start up in less than 10 milliseconds or something like that. Waiting for that kind of capacity is just insane.With the video thing I'm building, because of Corona and all of this remote teaching stuff, for some reason, we ended up getting lots of teachers using the product. It was one of these half-baked experiments because I didn't have time to build the full user interface for everything, and I realized that lots of people are using PowerPoint to prepare that kind of video. I thought well, how about if I shorten that loop, so just take your PowerPoint and convert it into video. Just type up what you want in the speaker notes, and we'll use these neurometrics to generate audio and things like that. Teachers like it for one reason or the other.We had this influential blogger from Russia explain it on his video blog and then it got picked up, my best guess from what I could see from Google Translate, some virtual meeting of teachers of Russia where they recommended people to try it out. I woke up the next day, the metrics went totally crazy because a significant portion of teachers in Russia tried my tool overnight in a short space of time. Something like that, I couldn't predict it. It's lovely but as you said, as long as we don't go over a hundred subscribers, we're fine. If I was in a situation like that, the thing would completely crash because it's unexpected. We'd have a thing that's amazingly good for marketing that would be amazingly bad for business because it would crash all our capacity we had. Or we had to prepare for a lot more capacity than we needed, but because this is all running on Lambda, Fargate, and other auto-scaling things, it's just fine. No sweat at all. It was a lovely thing to see actually.Jeremy: You actually have two problems there. If you're not running in the cloud or not running on-demand compute, is the fact that one, you would've potentially failed, things would've fallen over and you would've lost all those potential customers, and you wouldn't have been able to grow.Gojko: Plus you've lost paying customers who are using your systems, who've paid you.Jeremy: Right, that's the other thing too. But, on the other side of that problem would be you can't necessarily anticipate some of those things. What do you do? Over-provision and just hope that maybe someday you'll get whatever? That's the crazy thing where the elasticity piece of the cloud to me, is such a no-brainer. Because I know people always talk about, well, if you have predictable workloads. Well yeah, I know we have predictable workloads for some things, but if you're a startup or you're a business that has like ... Maybe you'd pick up some press. I worked for a company that we picked up some press. We had 10,000 signups in a matter of like 30 seconds and it completely killed our backend, my SQL database. Those are hard to prepare for if you're hosting your own equipment.Gojko: Absolutely, not even if you're hosting your own.Jeremy: Also true, right.Gojko: Before moving to Lambda, the app was deployed to Heroku. That was basically, you need to predict how many virtual machines you need. Yes, it's in the cloud, but if you're running on EC2 and you have your 10, 50, 100 virtual machines, whatever running there, and all of a sudden you get a lot more traffic, will it scale or will it not scale? Have you designed it to scale like that? And one of the best things that I think Lambda brought as a constraint was forcing people to design this stuff in a way that scales.Jeremy: Yes.Gojko: I can deploy stuff in the cloud and make it all distributed monolith, so it doesn't really scale well, but with Lambda because it was so constrained when it launched, and this is one thing you mentioned, partially we're losing those constraints now, but it was so constrained when it launched, it was really forcing people to design things that were easy to scale. We had total isolation, there was no way of sharing things, there was no session stickiness and things like that. And then you have to come up with actually good ways of resolving that.I think one of the most challenging things about serverless is that even a Hello World is a distributed transaction processing system, and people don't get that. They think about, well, I had this DigitalOcean five-dollar-a-month server and it was running my, you know, Rails up correctly. I'm just going to use the same ideas to redesign it in Lambda. Yes, you can, but then you're not going to really get the benefits of all of this other stuff. And if you design it as a massively distributed transaction processing system from the start, then yes, it scales like crazy. And it scales up and down and it's lovely, but as Lambda's maturing, I have this slide deck that I've been using since 2016 to talk about Lambda at conferences. And every time I need to do another talk, I pull it out and adjust it a bit. And I have this whole Git history of it because I do markdown to slides and I keep the markdown in Git so I can go back. There's this slide about limitations where originally it's only ... I don't remember what was the time limitation, but something very short.Jeremy: Five minutes originally.Gojko: Yeah, something like that and then it was no PCI compliance and the retries are difficult, and all of this stuff basically became sold. And one of the last things that was there, there was don't even try to put it in a VPC, definitely, you can but it's going to take 10 minutes to start. Now that's reasonably okay as well. One thing that I remember as a really important design constraint was effectively it was a share nothing platform because you could not share data between two Lambdas running at the same time very easily in the same VM. Now that we can connect Lambdas to EFS, you effectively can do that as well. You can have two Lambdas, one writing into an EFS, the other reading the same EFS at the same time. No problem at all. You can pump it into a file and the other thing can just stay in a file and get the data out.As the platform is maturing, I think we're losing some of these design constraints, and sometimes constraints breed creativity. And yes, you still of course can design the system to be good, but it's going to be interesting to see. And this 15-minute limit that we have in Lamdba now is just an artificial number that somebody thought.Jeremy: Yeah, it's arbitrary.Gojko: And at some point when somebody who is important enough asks AWS to give them half-hour Lamdbas, they will get that. Or 24-hour Lambdas. It's going to be interesting to see if Lambda ends up as just another way of running EC2 and starting EC2 that's simpler because you don't have to manage the operating system. And I think the big difference we'll get between EC2 and Lambda is what percentage of ops your developers are responsible for, and what percentage of ops Amazon's developers are responsible for.Because if you look at all these different offerings that Amazon has like Lightsail and EC2 and Fargate and AWS Batch and CodeDeploy, and I don't know how many other things you can run code on in Lambda. The big difference with Lambda is really, at least until very recently was that apart from your application, Amazon is responsible for everything. But now, we're losing design constraints, you can put a Docker container in, you can be responsible for the OS image as well, which is a bit again, interesting to look at.Jeremy: Well, I also wonder too, if you took all those event sources that you can point at Lambda and you add those to Fargate, what's the difference? It seems like they're just merging into two very similar products.Gojko: For the video build platform, the last step runs in Fargate because people are uploading things that are massive, massive, massive for video processing, and just they don't finish in 15 minutes. I have to run to Fargate, and the big difference is the container I packaged up for Fargate takes about 40 seconds to actually deploy. A new event at the moment with the stuff I've packaged in Fargate takes about 40 seconds to deploy. I can optimize that, but I can't optimize it too much. Fargate is still order of magnitude of tens of seconds to process an event. I think as Fargate gets faster and as Lambda gets more of these capabilities, it's going to be very difficult to tell them apart I think.With Fargate, you're intended to manage the container image yourself. You're responsible for patching software, you're responsible for patching OS vulnerabilities and things like that. With Lambda, Amazon, unless you use a container image, Amazon is responsible for that. They come close. When looking at this video building for the first time, I was actually comparing code. I was considering using CodeBuild for that because CodeBuild is also a way to run things on demand and containers, and you actually can get quite decent machines with CodeBuild. And it's also event-driven, and Fargate is event-driven, AWS Batch is event-driven, and all of these things are converging to each other. And really, AWS is famous for having 10 products that do the same thing effectively and you can't tell them apart, and maybe that's where we'll end.Jeremy: And I'm wondering too, the thing that was great about Lambda, at least for me like you said, the shared nothing architecture where it was like, you almost didn't have to rely on anything other than the event that came in, and the processing of that Lambda function. And if you designed your systems well, you may have some bottleneck up front, but especially if you used distributed transactions and you used async invocations of downstream functions, where you could basically take some data that you needed to pass into it, and then you wouldn't necessarily need that to communicate with anything other than itself to process that data. The scale there was massive. You could just keep scaling and scaling and scaling. As you add things like EFS and that adds constraints in terms of the number of transactions and connections that, that can make and all those sort of things. Do these things, do they become less reliable? By allowing it to do more, are we building systems that are less reliable because we're not using some of those tried-and-true constraints that were there?Gojko: Possibly, but every time you add a new moving part, you create one more potential point of failure there. And I think for me, one of the big lessons when I was working on ... I spent a few years working on very high throughput transaction processing systems. That's why this whole thing rings a bell a lot. A lot of it really was how do you figure out what type of messages you send and where you send them. The craze of these messages and distributed transaction processing systems in early 2000s, created this whole craze of enterprise service buses later that came. We now have this... What is it called? It's not called enterprise service bus, it's called EventBridge, or something like that.Jeremy: EventBridge, yes.Gojko: That's effectively an enterprise service bus, it's just the enterprise is the Amazon cloud. The big challenge in designing things like that is decoupling. And it's realizing that when you have a complicated system like that, stuff is going to fail. And especially when we were operating around hardware, stuff is going to fail badly or occasionally, and you need to not bring the whole house down where some storage starts working a bit slower. You create circuit breakers, you create layers and layers of stuff that disconnect things. I remember when we were looking originally at Lambdas and trying to get the head around that and experimenting, should one Lambda call another? Or should one Lambda not call another? And things like that.I realized, let's say for now, until we realize we want to do something else, a Lambda should only ever talk to SNS and nothing else. Or SQS or something like that. When one Lambda completes, it's going to track a message somewhere and we need to design these messages to be good so that we can decouple different parts of the process. And so far, that helps too as a constraint. I think very, very few times we have one Lambda calling another. Mostly when we actually need a synchronized response back, and for security reasons, we wanted to isolate something to a single Lambda, but that's effectively just a black box security isolation. Since creating these isolation layers through messages, through queues, through topics, becomes a fundamental part of designing these systems.I remember speaking at the conference to somebody. I forgot the name of the person who was talking about airline. And he was presenting after me and he said, "Look, I can relate to a lot of what you said." And in the airline community basically they often talk about, apparently, I'm not an airline programmer, he told me that in the airline community, talk about designing the protocol being the biggest challenge. Once you design the protocol between your components, the message is who sends what where, you can recover from almost any other design flaw because it's decoupled so if you've made a mess in one Lambda, you can redesign that Lambda, throw it away, rewrite it, decouple things a different way. If the global protocol is good, you get all the flexibility. If you mess up the protocol for communication, then nothing's going to save you at the end.Now we have EFS and Lambda can talk to an EFS. Should this Lambda talk directly to an EFS or should this Lambda just send some messages to a topic, and then some other Lambdas that are maybe reserved, maybe more constrained talk to EFS? And again, the platform's evolved quite a lot over the last few years. One thing that is particularly useful in that regard is the SQS FIFO queues that came out last year I think. With Corona ...Jeremy: Yeah, whenever it was.Gojko: Yeah, I don't remember if it was last year or two years ago. But one of the things it allows us to do is really run lots and lots of Lambdas in parallel where you can guarantee that no two Lambdas access the same kind of business entity that you have in the same type. For example, for this mind mapping thing, we have lots and lots of people modifying lots and lots of files in parallel, but we need to aggregate a single map. If we have 50 people over here working with a single map and 60 people on a map working a different map, aggregation can run in parallel but I never ever, ever want two people modifying the same map their aggregation to run in parallel.And for Lambda, that was a massive challenge. You had to put Kinesis between Lambda and other Lambdas and things like that. Kinesis' provision capacity, it costs a lot, it doesn't auto-scale. But now with SQS FIFO queues, you can just send a message and you can say the kind of FIFO ID is this map ID that we have. Which means that SQS can run thousands of Lambdas in parallel but they'll never run more than one Lambda for the same map idea at the same time. Designing your protocols like that becomes how you decouple one end of your app that's massively scalable and massively parallel, and another end of your app that we have some reserved capacity or limits.Like for this kind of video thing, the original idea of that was letting me build marketing videos easier and I can't get rid of this accent. Unfortunately, everything I do sounds like I'm threatening someone to blackmail them. I'm like a cheap Bond villain, and that's not good, but I can't do anything else. I can pay other people to do it for me and we used to do that, but then that becomes a big problem when you want to modify tiny things. We paid this lady to professionally record audio for a marketing video that we needed and then six months later, we wanted to change one screen and now the narration is incorrect. And we paid the same woman again. Same equipment, same person, but the sound is totally different because two different equipment.Jeremy: Totally different, right.Gojko: You can't just stitch it up. Then you end up like, okay, do we go and pay for the whole thing again? And I realized the neurometric text-to-speech has learned so much that it can do English better than I can. You're a native English speaker so you can probably defeat those machines, but I can't.Jeremy: I don't know if I could. They're pretty good now. It's kind of scary.Gojko: I started looking at one like why don't they just put stuff in a Markdown and use Markdown to generate videos and things like that? All of these things, you get quota limits still. I thought we were limited on Google. Google gave us something like five requests per second in parallel, and it took me a really long time to even raise these quotas and things like that. I don't want to have lots of people requesting stuff and then in parallel trashing this other thing over there. We need to create these layers of running things in a decent limit, and I think that's where I think designing the protocol for this distributed system becomes an importance.Jeremy: I want to go back because I think you bring up a really good point just about a different type of architecture, or the architectural design of decoupling systems and these event-driven things. You mentioned a Lambda function processes something and sends it to SQS or sends it to SNS to it can do a fan-out pattern or in the case of the FIFO queue, doing an ordered pattern for sequential processing, which those were all great patterns. And even things that AWS has done, such as add things like Lambda destination. Now if you run an asynchronous Lambda function, you still have to write some code or you used to have to write some code that said, "When this is finished processing, now call some other component." And there's just another opportunity for failure there. They basically said, "Well, if it succeeds, then you can actually just forward it off to one of these other services automatically and we'll handle all of the retries and all the failures and that kind of stuff."And those things have been added in to basically give you that warm and fuzzy feeling that if an event doesn't reach where it's supposed to go, that some sort of cloud trickery will kick in and make sure that gets processed. But what that is introduced I think is a cognitive overload for a lot of developers that are designing these systems because you're no longer just writing a script that does X, Y, and Z and makes a few database calls. Now you're saying, okay, I've got to write a script that can massively scale and take the transactions that I need to maybe parallelize or that I maybe need to queue or delay or throttle or whatever, and pass those down to another subsystem. And then that subsystem has to pick those up and maybe that has to parallelize those or maybe there are failure modes in there and I've got all these other things that I have to think about.Just that effect on your average developer, I think you and I think about these things. I would consider myself to be a cloud architect, if that's a thing. But essentially, do you see this being I guess a wall for a lot of developers and something that really requires quite a bit of education to ramp them up to be able to start designing these systems?Gojko: One of the topics we touched upon is the cyclic nature of things, and I think we're going back to where moving from apps working on a single machine to client server architectures was a massive brain melt for a lot of people, and three-tier architectures, which is later, we're not just client server, but three-tier architectures ended up with their own host of problems and then design problems and things like that. That's where a lot of these architectural patterns and design patterns emerged like circuit breakers and things like that. I think there's a whole body of knowledge there for people to research. It's not something that's entirely new and I think you can get started with Lambda quite easily and not necessarily make a mess, but make something that won't necessarily scale well and then start improving it later.That's why I was mentioning that earlier in the discussion where, as long as the protocol makes sense, you can salvage almost anything late. Designing that protocol is important, but then we're going to good software design. I think teaching people how to do that is something that every 10 years, we have to recycle and reinvent and figure it out because people don't like to read books from more than 10 years ago. All of this stuff like designing fault tolerance systems and fail-safe systems, and things like that. There's a ton of books about that from 20 years ago, from 10 years ago. Amazon, for people listening to you and me, they probably use Amazon more for compute than they use for getting books. But Amazon has all these books. Use it for what Amazon was originally intended for and then get some books there and read through this stuff. And I think looking at design of distributed systems and stuff like that becomes really, really critical for Lambdas.Jeremy: Yeah, definitely. All right, we've got a few minutes left and I'd love to go back to something we were talking about a little bit earlier and that was everything moving onto a few of these major cloud providers. And one of the things, you've got scale. Scale is a problem when we talked about oh, we can spin up as many VMs as we want to, and now with serverless, we have unlimited capacity really. I know we didn't say that, but I think that's the general idea. The cloud just provides this unlimited capacity.Gojko: Until something else decides it's not unlimited.Jeremy: And that's my point here where every major cloud provider that I've been involved with and I've heard the stories of, where you start to move the needle at all, there's always an SA that reaches out to you and really wants to understand what your usage is going to be, and what your patterns were going to be. And that's because they need to make sure that where you're running your applications, that they provision enough capacity because there is not enough capacity, or there's not unlimited capacity in the cloud.Gojko: It's physically limited. There's only so much buildings where you can have data centers on the surface of Earth.Jeremy: And I guess that's where my question comes in because you always hear these things about lock-in. Like, well serverless, if you use Lambda, you're going to be locked in. And again, if you're using Oracle, you're locked in. Or, you're using MySQL you're locked in. Or, you're using any of the other things, you're locked in.Gojko: You're actually not locked in physically. There's a key and a lock.Jeremy: Right, but this idea of being locked in not to a specific cloud provider, but just locked into a cloud in general and relying on the cloud to do that scaling for you, where do you think the limitations there are?Gojko: I think again, going back to cyclic, cyclic, cyclic. The PC revolution started when a lot more edge compute was needed on mainframes, and people wanted to get stuff done on their own devices. And I think probably, if we do ever see the limitations of this and it goes into a next cycle, my best guess it's going to be driven by lots of tiny devices connected to a cloud. Not necessarily computers as we know computers today. I pulled out some research preparing for this from IDC. They are predicting basically from 18.3 zettabytes of data needed for IOT in 2019, to be 73.1 zettabytes by 2025. That's like times three in a space of six years. If you went to Amazon now and told them, "You need to have three times more data space in three years," I'm not sure how they would react to that.This stuff, everything is taking more and more data, and everything is more and more connected to the cloud. The impact of something like that going down now is becoming totally crazy. There was a case in 2017 where S3 started getting a bit more latency than usual in U.S. East 1, in I think February of 2018, or something like that. There were cases where people couldn't turn the lights on in their houses because the management software was working on S3 and depending on S3. Expecting S3 to be indestructible. Last year, in November, Kinesis pretty much went offline as far as everybody else outside AWS concerend for about 15 hours I think. There were people on Twitter that they can't go back into their house because their smart lock is no longer that smart.And I think we are getting to places where there will be more need for compute on the edge. First of all, there's going to be a lot more demand for data centers and cloud power and I think that's going to keep going on for the next five, ten years. But then people will realize they've hit some limitation of that, and they're going to start moving towards the edge. And we're going from mainframe back into client server computing I think. We're getting these products now. I assume most of your listeners have seen one like all these fancy ubiquity Wi-Fi thingies that are costing hundreds of dollars and they look like pieces of furniture that's just sitting discretely on the wall. And there was a massive security breach yesterday published. Somebody took their AWS keys and took all the customer data and everything.The big advantage over all the ugly routers was that it's just like a thin piece of glass that sits on your wall, and it's amazing and it looks good, but the reason why they could do a very thin piece of glass is the minimal amount of software is running on that piece of glass, the rest is running the cloud. It's not just locking in terms of is it on Amazon or Google, it's that it's so tightly coupled with something totally outside of your home, where your network router needs Amazon to be alive now in a very specific region of Amazon where everybody's been deploying for the last 15 years, and it's running out of capacity very often. Not very often but often enough.There's some really interesting questions that I guess we'll answer in the next five, ten years. We're on the verge of IOT I think exploding because people are trying to come up with these new products that you wouldn't even think before that you'd have smart shoes and smart whatnot. Smart glasses and things like that. And when that gets into consumer technology, we're no longer going to have five or ten computer devices per person, we'll have dozens and dozens of computing. I guess think about it this way, fifteen years ago, how many computer devices were you carrying with yourself? Probably mobile phone and laptop. Probably not more. Now, in the headphones you have there that's Bose ...Jeremy: Watch.Gojko: ... you have a microprocessor in the headphones, you have your watch, you have a ton of other stuff carrying with you that's low-powered, all doing a bit of processing there. A lot of that processing is probably happening on the cloud somewhere.Jeremy: Or, it's just sending data. It's just sending, hey here's the information. And you're right. For me, I got my Apple Watch, my thermostat is connected to Wi-Fi and to the cloud, my wife just bought a humidifier for our living room that is connected to Wi-Fi and I'm assuming it's sending data to the cloud. I'm not 100% sure, but the question is, I don't know why we need to keep track of the humidity in my living room. But that's the kind of thing too where, you mentioned from a security standpoint, I have a bunch of AWS access keys on my computer that I send over the network, and I'm assuming they're secure. But if I've got another device that can access my network and somebody hacked something on the cloud side and then they can get in, it gets really dangerous.But you're right, the amount of data that we are now generating and compute that we're using in the cloud for probably some really dumb things like humidity in my living room, is that going to get to a point where... You said there's going to be a limitation like five years, ten years, whatever it is. What does the cloud do then? What does the cloud do when it can no longer keep up with the pace of these IOT devices?Gojko: Well, if history is repeating and we'll see if history is repeating, people will start getting throttled and all of a sudden, your unlimited supply of Lambdas will no longer be unlimited supply of Lambdas. It will be something that you have to reserve upfront and pay upfront, and who knows, we'll see when we get there. Or, we get things that we have with power networks like you had a Texas power cut there that was completely severe, and you get a IT cut. I don't know. We'll see. The more we go into utility, the more we'll start seeing parallels between compute and power networks. And maybe power networks are something that you can look at and later name. That's why I think the next cycle is probably going to be some equivalent of client server computing reemerging.Jeremy: Yeah. All right, well, I got one more question for you and this is just something where it may be a little bit of a tongue-in-cheek question. Because we talked it a little bit ... we talked about the merging of Lambda, and of Fargate, and some of these other things. But just from your perspective, serverless in five years from now, where do you see that going? Do you see that just becoming the main ... This idea of utility computing, on-demand computing without setting up servers and managing ops and some of these other things, do you see that as the future of serverless and it just becoming just the way we build applications? Or do you think that it's got a different path?Gojko: There was a tweet by Simon Wardley. You mentioned Simon Wardley earlier in the talk. There was a tweet a few days ago where he mentioned some data. I'm not sure where he pulled it from. This might be unverified, but generally Simon knows what he's talking about. Amazon itself is deploying roughly 50% of all new apps they're building on serverless. I think five years from now, that way of running stuff, I'm not sure if it's Lambda or some new service that Amazon starts and gives it some even more confusing name that runs in parallel to everything. But, that kind of stuff where the operator takes care of all the ops, which they really should be doing, is going to be the default way of getting utility compute out.I think a lot of these other things will probably remain useful for specialists' use cases, where you can't really deploy it in that way, or you need more stability, or it's not transient and things like that. My best guess is first of all, we'll get Lambda's that run for longer, and I assume that after we get Lambdas that run for longer, we'll probably get some ways of controlling routing to Lambdas because you already can set up pre-provisioned Lambdas and hot Lambdas and reserved capacity and things like that. When you have reserved capacity and you have longer running Lambdas, the next logical thing there is to have session stickiness, and routing, and things like that. And I think we'll get a lot of the stuff that was really complicated to do earlier, and you had to run EC2 instances or you had to run complicated networks of services, you'll be able to do in Lambda.And Lambda is, I wouldn't be surprised if they launch a totally new service with some AWS cloud socket, whatever. Something that is a implementation of the same principle, just in a different way, that becomes a default we are running computer for lots of people. And I think GPUs are still a bit limited. I don't think you can run GPU utility anywhere now, and that's limiting for a whole host of use cases. And I think again, it's not like they don't have the technology to do it, it's just they probably didn't get around to doing it yet. But I assume in five years time, you'll be able to do GPUs on-demand, and processing GPUs, and things like that. I think that the buzzword itself will lose really any special meaning and that's going to just be a way of running stuff.Jeremy: Yeah, absolutely. Totally agree. Well, listen Gojko, thank you so much for spending the time chatting with me. Always great to talk with you.Gojko: You, too.Jeremy: If people want to get in touch with you, find out more about what you're doing, how do they do that?Gojko: Well, I'm very easy to find online because there's not a lot of people called Gojko. Type Gojko into Google, you'll find me. And gojko.networks, gojko.com works, gojko.org works, and all these other things. I was lucky enough to get all those domains.Jeremy: That's G-O-J-K-O ...Gojko: Yes, G-O-J-K-O.Jeremy: ... for people who need the spelling.Gojko: Excellent. Well, thanks very much for having me, this was a blast.Jeremy: All right, yeah. And make sure you check out ... You mentioned Narakeet. It's a speech thing?Gojko: Yeah, for developers that want to build videos without hassle, and want to put videos in continuous integration, and things like that. Narakeet, that's like parakeet with an N for narration. Check that out and thanks for plugging it.Jeremy: Awesome. And then, check out MindMup as well. Awesome stuff. I've got all the stuff in the show notes. Thanks again, Gojko.Gojko: Thank you. Bye-bye.

Links: Unconventional Guide to AWS Cost Management: https://www.duckbillgroup.com/resources/unconventional-guide-to-aws-cost-management/ Migrate from Oracle to Amazon Aurora: https://aws.amazon.com/getting-started/hands-on/migrate-oracle-to-amazon-aurora/ TranscriptCorey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Pete: Hello, and welcome to the AWS Morning Brief: Fridays From the Field. I am Pete Cheslock.Jesse: I’m Jesse DeRose.Pete: We’re coming at you again with some more listener questions from the Unconventional Guide to AWS Cost Management. I’m excited. People are listening to us, Jesse.Jesse: This is fantastic. I’m really excited that we have one fan. I’ve always wanted one fan.Pete: Well, two fans now. Maybe even more because we keep getting questions. And you can also be one of our Friends of the Pod by going to lastweekinaws.com/QA. And you can give us some feedback, you can give us a question and, like, will totally answer it because we like Friends of the Pod.Jesse: We may or may not enter you into a raffle to get a Members Only jacket that’s branded with ‘Friends with the Pod.’Pete: We should get some pins made, maybe.Jesse: Ohh…Pete: I think that's a good idea.Jesse: Yeah.Pete: So, what are we answering today, or attempting to answer for our listener, Jesse?Jesse: So today, we’ve got a really great question from [Godwin 00:01:20]. Thank you, Godwin, Godwin writes, “I truly believe that the system that I support is, like, a data hoarder. We do a lot of data ingestion, we recently did a lift-and-shift of the system to AWS, we use an Oracle database. The question is, how do I segregate the data and start thinking about moving it out of traditional relational databases and into other types of databases? Presently, our method is all types of data goes into a quote-unquote, ‘all-purpose database,’ and the database is growing quite fast. Where should I get started?”Pete: Well, I just want to commend you for a lift-and-shift into Amazon. That’s a Herculean feat, no matter what you’re lifting and shifting over. Hopefully, you have maybe started to decommission those original data centers and you don’t just have more data in twice as many locations.Jesse: [laugh]. But I also want to call out well done for thinking about not just the lift-and-shift, but the next step. I feel like that’s the thing that a lot of people forget about. They think about the lift-and-shift, and then they go, “Awesome. We’re hybrid. We’re in AWS, now. We’re in our data center. We’re good. Case closed.” And they forget that there’s a lot more work to do to modernize all those workloads in AWS, once you’ve lifted and shifted. And this is part of that conversation.Pete: Yeah, that’s a really good point because I know we’ve talked about this in the past, the lift-and-shift shot clock: when you don’t start migrating, start modernizing those applications to take advantage of things that are more cloud-native, the technical debt is really going to start piling up, and the folks that are going to manage that are going to get more burnt out, and it really is going to end poorly. So, the fact you’re starting to think about this now is a great thing. Also, what is available to you now that you’re on AWS is huge compared to a traditional data center.Jesse: Yeah.Pete: And that’s not just talking about the—I don’t even know if I’ve ever counted how many different databases exist on Amazon. I mean, they have a database for, at this point, every type of data. I mean, is there a type of data that they’re going to create, just so that they can create a database to put it into?Jesse: Wouldn’t surprise me at this point.Pete: They’ll find a way [laugh] to come up with that charge on your bill. But when it comes to Oracle, specifically Oracle databases, there’s obviously a big problem in not only the cost of the engine, running the database on a RDS or something to that effect, but you have licensing costs that are added into it as well. Maybe you have a bring-your-own-license or maybe you’re just using the off-the-shelf, but the off-the-shelf, kind of, ‘retail on-demand pricing’ RDS—I’m using air quotes for all these things, but you can’t see that—they will just have the licensing costs baked in as well. So, you’re paying for it—kind of—either way.Jesse: And I think this is something also to think about that we’ll dive into in a minute, but one of the things that a lot of people forget about when they move into AWS says that you’re not just paying for data sitting on a piece of hardware in a data center that’s depreciating, now. You’re paying for storage, you’re paying for I/O costs, you’re paying for data transfer, to Pete’s point, you’re also paying for some of the license as well, potentially. So, there’s lots of different costs associated with keeping an Oracle Database running in AWS. So, that’s actually probably the best place to start thinking about this next step about where to get started. Think about the usage patterns of your data.And this may be something that you need to involve engineering, maybe involve product for if they’re part of these conversations for storage of your product or your feature sets. Think about what are the usage patterns of your data?Pete: Yeah, exactly. Now, you may say to yourself, “Well, we’re on Oracle”—and I’m sure people listening are like, “Well, that’s your problem. You should just move off of Oracle.” And since you can’t go back in time and undo that decision—and the reality is, it probably was a good decision at the time. There’s a lot of businesses, including Amazon, who ran all of their systems on Oracle.And then migrated off of them. Understanding the usage patterns, what type of data is going into Oracle, I think is a big one. Because if you can understand the access patterns of the types of data that are going in, that can help you start peeling off where that data should go. Now, let’s say you’re just pushing all new data created. And we don’t even know what your data is, so we’re going to take some wild assumptions here on what you could possibly do—but more so just giving you homework, really—thinking about the type of data going in, right?If you’re just—“I’m pushing all of my data into this database because someday we might need to query it.” That’s actually a situation where you really want to start thinking of leveraging more of a data warehouse-style approach to it, where you have a large amount of data being created, you don’t know if you’re going to need to query it in the future, but you might want to glean some value out of that. Using S3, which is now available to you outside of your data center world, is going to be super valuable to just very cheaply shove data into S3, to be able to go back in later time. And then you can use things like Athena to ad hoc query that data, or leverage a lot of the ingestion services that exist to suck that data into other databases. But thinking about what’s being created, when it is going into places is a big first step to start understanding, well, how quickly does this data need to come back?Can the query be measured in many seconds? Can it be done ad hoc, like in Athena? Does it need to be measured in milliseconds? What’s the replication that needs to happen? Is this very valuable data that we need to have multiple backups on?Is it queried more than it’s created? Maybe you need to have multiple replica reader databases that are there. So, all these types of things of really understanding just what’s there to begin with, and it’s probably going to be in talking to a lot of engineering teams.Jesse: Yeah, you can think about this project in the same way that you might move from a monolith to a microservice architecture. So, if you’re moving from a monolith to a microservice architecture, you might start peeling away pieces of the monolith, one at a time. Pieces that can easily be turned into microservices that stand on their own within the cloud, even if they’re running on the same underlying infrastructure as the monolith itself within AWS. And then, as you can pull those pieces away, then start thinking about does this need to be in a relational database? Does this need to have the same amount of uptime and availability as the resources that are sitting in my Oracle Database right now?All those things that Pete just mentioned, start thinking about all of those components to figure out where best to pull off the individual components of data, and ultimately put them in different places within AWS. And to be clear, there’s lots of great guides on the internet that talk about moving from your Oracle database into, gosh, just about any database of choice. AWS even has specific instructions for this, and we’ll throw a link in the [show notes 00:09:02].They really, really want you to move this data to RDS Aurora. They go through painstaking detail to talk about using the AWS schema conversion tool to convert your schema over; they talk about the AWS database migration service to migrate the data over, and then they talk about performing post-migration activities such as running SQL queries for validating the object types, object count, things like that. I think that a lot of folks actually don’t know that the database migration service exists, and it’s something worth calling out as a really powerful tool.Pete: Yeah, the Amazon DMS service is honestly I think, a super-underrated service that people just don’t know about. It has the ability to replicate data from both on-premises databases to Amazon databases but also databases already running on Amazon. You could replicate from a database running on EC2 into Aurora. You could replicate that into S3—you know, replicate data into S3 that way, bringing things into sync—replicate that data into S3, and then maybe use it for other purposes. It can replicate data from DocumentDB into other sources.So, they’re clearly doing a big investment in there. And to Jesse’s point, yeah, Amazon really wants this data. So, talk to your account manager as you’re testing out some of these services. Do a small proof of concept, maybe, to see how well it works, if you can understand the queries, or you can point your application over at an Aurora database with some of this data migrated in; that’s a great way to understand how well this could work for your organization. But as Jesse mentioned, they do want that data in Aurora.So, if it turns out that you’re looking at your—you know, migrate some data in there, and it’s starting to work, and you’re kind of getting a feel for the engineering effort to migrate there, stop. Talk to your account manager before you spend any more money on Aurora because it’s very likely that they can put together a program—if a program doesn’t already exist—to incentivize you to move that data over; they can give you subject matter expertise; they can provide you credits to help you migrate that data over. Don’t feel like you have to do this on your own. You have an account team; you should definitely reach out to them, and they will provide you a lot of help to get that data in there. They’ve done it for many of their other clients, and they’re happy to do it for you because they know that, long term, when you move that data to Aurora, it’s going to be very sticky in Aurora.You’re probably not going to move off of there. It’s a long game for them; that’s how they play it. So, check out those services; that could be a really great way to help you get rid of your Oracle addiction.Jesse: Yeah, and if you’re able to, as we talked about earlier, if you’re able to identify workloads that don’t need to run in a relational database, or don’t need to run in, maybe, a database at all, for that matter, stick that data in S3. Call it a day. Put them on lifecycle management policies or different storage tiers, and use Athena for ad hoc queries, or maybe Redshift if you’re doing more data warehouse-style tasks. But if that data doesn’t need to live in a relational database, there are many cheaper options for that data.Pete: Exactly. But one last point I will make is don’t shove it into MongoDB just because you want to have schema-less, or—Jesse: Please.Pete: —think about what you’re going to use it for, think about what the data access patterns because there is a right place for your data. Don’t just jump into no-SQL just ‘cause because you’ll probably end up with a bigger problem. In the long run.Corey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com.Pete: So Jesse, I’m looking at our list of questions. And it turns out, we have another question.Jesse: Ohh.Pete: Two questions came in.Jesse: You like me, you really like me!Pete: It’s so great. Again, you can also send us a question, lastweekinaws.com/QA. You can go there, drop in a question and feel free to put your name. Or not; you can be anonymous, it’s totally fine. We’ll happily answer your question either way. So Jesse, who is our next question from? What is this one about?Jesse: This one’s from [Joseph 00:13:19]. They write in, “Hey, folks. Love the show. Longtime listener, first-time caller.” Thank you. “I would love to know how people manage their costs in AWS Batch. Jobs themselves can’t be tagged for cost allocation, which makes things a bit complicated.” Lord Almighty, yes, it does. “How best should I see if the jobs are right-sized? Are they over-provisioned in terms of memory or compute? What’s the best way to see if EC2 is my better choice, versus Fargate, versus other options? How can I tell if the batch-managed cluster itself is under-utilized?”Pete: Oof. This is a loaded question with a lot of variables.Jesse: Yeah. And so we’re going to break it down because there’s definitely a couple questions here. But I want to start off with what AWS Batch is, just really quick to make sure everybody’s on the same page here. AWS Batch, effectively, is a managed service in AWS that schedules it and runs your batch computing jobs on top of AWS compute resources. Effectively, it does a lot of the heavy lifting configuration for you so you can just focus on analyzing the results of those queries.Pete: Yeah, exactly. And Batch supports a really wide variety of tooling that can operate this, and that’s why it’s hard for us to give, specifically, how you might optimize this, but I think some of the optimizations actually mirror a lot of the optimizations we’ve done with optimizing EMR clusters and things of that nature, where you’re running these distributed jobs. And you want to make sure that if you’re running straight off of EC2 instances, then you want to make sure that they are essentially maxed out. If the CPU is anything less than 100% for an on-demand instance, then there’s wasted, or there’s opportunity for improvement. And so making sure that your jobs are sized appropriately and balancing out memory and CPU so that, effectively, you’re using all of the memory and all of the CPU, that’s a real basic first step.But honestly, a lot of folks kind of miss out on that. They just kind of run a job and go off and do their own thing. They never really go back and look at those graphs. You can go to CloudWatch, they’re all going to be there for you.Jesse: Yeah. And to this point, there’s always an opportunity to make these workloads more ephemeral. If you have the opportunity to make it more ephemeral, please, please, please, please, absolutely do so. Unless your batch job needs to run 24/7. We’ve seen that in a few cases where they have, essentially, clusters that are running 24/7, but they’re not actually utilized regularly; the workloads are only scheduled for a short amount of time.So, if you don’t need those batch jobs running 24/7, please, by all means, move to more ephemeral resources, like Fargate. Fargate on Spot, Spot Instances in general, or even Lambda, which AWS Batch now supports as well.Pete: Yeah, it has some step function support, which is pretty interesting. Yeah, this is a great opportunity to aggressively—aggressively—leverage Spots, if you’re not currently today. The reality is that check out Fargate on Spot if you don’t need, like, a custom operating system, you don’t need a custom EBS volume size. If you do, then EC2 on Spot is probably the best option that you really have. But really do not want to be running anything on on-demand instances. Even on-demand instances with a really good savings plan, you’re still leaving money on the table because Spot Instances are going to be a lot cheaper than even the best savings plan that’s out there.Jesse: And I think that’s a good point, too, Pete, which is if you do need to run these workloads on-demand, 24/7, think about if you can get away with using Spot Instances. If you can’t get away with using Spot Instances, at least purchase a savings plan if you don’t do anything else. If you take nothing else away from this, at least make sure that you have some kind of savings plan in place for these resources so that you’re not paying on-demand costs 24/7. But in most cases, you can likely make them more ephemeral, which is going to save you a lot more money in the long run.Pete: Yeah, exactly. That’s the name of the game. I mean, when we talk to folks on Amazon, the more ephemeral you can make your application—the more you can have it handle interruption—the less expensive it will be to operate. And that goes from everywhere from Spot Instances and how they’re priced, right? If you just get a normal Spot Instance, it will have a really aggressive discount on it if you need zero time in advance before interruption.So, if that instance can just go in at any second, then you’ll get the best discount on that Spot Instance. But if your app needs a little time, or runs for a defined period of time—let’s say your app runs for one hour—you can get a defined duration Spot of one hour, you’ll get a great discount still and you’ll only pay for however long you use it, but you will get that resource for one whole hour, and then you’ll lose it. If that’s still too aggressive, there’s configurable options up to six hours. Again, less discount, but more stability in that resource. So, that’s the trade-off you make when you move over to Spot Instances.Jesse: So, I also want to make sure that we get to the second part of this question, which is about attributing cost to your AWS Batch workloads. According to the AWS Batch documentation, you can tag AWS Batch compute environments, jobs, job definitions, and job queues, but you can’t propagate those tags to the underlying resources that actually run those jobs. Which to me, kind of just defeats the point.Pete: Yeah. [sigh]. Hashtag AWS wishlist here. You know, again, continuing to expand out tagging support for things that don’t support it. I know we’ve seen kind of weird inconsistencies, and just even, like, tagging ECS jobs and where you have to tag them for they’re to apply.So, I know it’s a hard problem, but obviously, it’s something that should be continually worked out on because, yeah, if you’re trying to attribute these costs, you’re left with the only option to run them in separate Amazon accounts, which solves this problem, but again, depending on your organization, could increase just the management overhead of those. But that is the ultimate way. I mean, that is the one way to ensure 100% of costs are encapsulated to a service is to have them run in a dedicated account. The downside being is that if you have a series of different jobs running across a different, maybe, business units, then obviously that’s going to break down super quick.Jesse: Yeah, and it’s also worth calling out that if there’s any batch jobs that need to send data to different places—maybe the batch job belongs to product A, but it needs to send data to product B—there’s going to be some amount of data transfer either across regionally or across accounts in order to share that data, depending on how your organization, how your products are set up. So, keep in mind that there are potentially some minor charges that may appear with this, but ultimately, if you’re talking about the best ways to really attribute costs for your AWS Batch workloads, linked accounts is the way to go.Pete: Yeah. If you need attribution down to the penny—some of our clients absolutely do. For invoicing purposes, they need attribution for business unit down to the penny. And if you’re an organization that needs that, then the only way to get that, effectively, is segmented accounts. So, keep that in mind.Again, until Amazon comes out with the ability to get a little bit more flexible tagging, but also, too, feel free to yell at your account manager—I mean, ask them nicely. They are people, too. But, you know, let them know that you want this. Amazon builds what the customers want, and if you don’t tell them that you want it, they’re not going to prioritize it. I’m not saying if you tell them, you’re going to get it in a couple of months, but you’re never going to get it if you don’t say anything. So, definitely let people know when there’s something that doesn’t work the way you expect it to.Jesse: Absolutely.Pete: Awesome. Wow. Two questions. I feel it’s like Christmas. Except—Jesse: [laugh].Pete: —it’s Christmas in almost springtime. It’s great. Well, again, you, too, can join us by being a Friend of the Pod, which Jesse really loves that one for some reason. [laugh].Jesse: Yeah. Don’t know why, but it’s going to be stuck in my brain.Pete: Exactly. You too can be a Friend of the Pod by going to lastweekinaws.com/QA and you can send us a question. We would love to spend some time in a future episode, answering them for you.If you’ve enjoyed this podcast, please go to lastweekinaws.com/review. Give it a five-star review on your podcast platform of choice, whereas if you hated this podcast, please go to lastweekinaws.com/review and give it a five-star rating on your podcast platform of choice and tell us why you want to be a Friend of the Pod. Thank you.Announcer: This has been a HumblePod production. Stay humble. 

最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS」 おはようございます、木曜日担当の小林です。 今日は 4/7 に出たアップデートをピックアップしてご紹介 感想は Twitter にて「#サバワ」をつけて投稿してください！ ■ トークスクリプト https://blog.serverworks.co.jp/everyday-aws-176 ■ UPDATE PICKUP  AWS Amplify Admin UIで新機能Data Managerが追加  AWS SystemsManagerのParameter storeでパラメータラベルの削除をサポート   Amazon CodeGuruの機能CodeGuru Reviewerで課金体系の変更を発表   Amazon CodeGuru ReviewerのPythonサポートが強化   AWS Batchでservice-linked roleをサポート   AWS GlueでAmazon Kinesis DataStreamsからのクロスアカウント読み取りをサポート   AWS ElementalMediaLiveでHTML5ベースのモーショングラフィックスオーバーレイ機能をサポート ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ

En nuestro episodio 4 nos visita Isaac Vallhonrat, Solutions Architect Especialista en la parte de computación de AWS basado en Barcelona. 
Nos viene a hablar de Spot Instances, qué son, qué no son y cómo poder utilizarlas en AWS. Isaac Vallhonrat: Isaac es arquitecto de soluciones especialista en soluciones de computación de AWS, basado en Barcelona. Como especialista, Isaac trabaja con clientes del sur de Europa ayudándoles a construir arquitecturas resilentes, escalables y a desplegar a gran escala con costes asequibles, utilizando Spot Instances.Rodrigo Asensio - @rasensioBasado en Barcelona, España, Rodrigo es responsable de un equipo de Solution Architecture del segmento Enterprise que ayuda a grandes clientes en Iberia a moverse al cloud y aprovechar sus beneficios.Links:Amazon EC2 Spot: https://aws.amazon.com/ec2/spot Las instancias de spot de Amazon EC2 le permiten aprovechar la capacidad de EC2 sin usar en la nube de AWS. Las instancias de spot están disponibles con un descuento de hasta el 90 % en comparación con los precios bajo demanda. Puede utilizar instancias de spot para diversas aplicaciones flexibles, sin estado y tolerantes a errores, como big data, cargas de trabajo en contenedores, CI/CD, servidores web, informática de alto rendimiento (HPC) y cargas de trabajo de prueba y desarrollo. Como las instancias de spot se integran perfectamente en otros servicios de AWS, incluidos Auto Scaling, EMR, ECS, CloudFormation, Data Pipeline y AWS Batch, puede elegir la manera de lanzar y mantener sus aplicaciones en ejecución en las instancias de spot.Amazon EC2 Spot Integrations Roadmap: https://github.com/aws/ec2-spot-instances-integrations-roadmap/projects/1 This is the experimental public roadmap for Amazon EC2 Spot Instances integrations with open-source software and frameworks. Knowing about our upcoming projects and priorities helps our customers plan. This repository contains information about what we are working on and allows all AWS customers to give direct feedback.EC2 Spot Workshops: https://ec2spotworkshops.com/ Serie de workshops de EC2 Spot Instances relacionados con machine learning, CI/CD, Autoscaling, Containers con EKS, Spark con EMR, simulaciones y varios más..

最新情報を "ながら" でキャッチアップ！ ラジオ感覚放送 「毎日AWS！」 おはようございます、サーバーワークスの加藤です。 今日は 10/7 に出たアップデート14件をご紹介。 感想は Twitter にて「#サバワ」をつけて投稿してください！ ■ UPDATE ラインナップ Amazon Connect がエージェント階層を用いて問い合わせの検索機能のユーザーアクセス制御ができるように AWS Batch がタグベースのアクセス制御をできるように Amazon WorkDocs が iOS アプリのダークモードをサポート AWS Elemental MediaConvert で InSyncFrameFormer フレームレート変換が利用可能に AWS Elemental MediaConvert でニールセンのオーディオ署名機能が利用可能に Amazon EBS CSI ドライバが AWS Outposts をサポート AWS Single Sign-On が PingFederate に対応 Amazon ElastiCache for Redis が Redis 6 をサポート Amazon CodeGuru Profiler がタグ付けをサポート Trailhead に新しいデジタルコース - AWS Cloud for Technical Professionals Amazon SageMaker の料金を最大18%引き下げ AWS Compute Optimizer のレコメンデーションデータを AWS KMS のカスタマーマスターキーを使って暗号化できるように AWS Database Migration Service がコミット日に基づくフォルダ分け機能をサポート Apache Flink Kinesis Consumer が拡張ファンアウトとHTTP/2データ取得APIをサポート ■ サーバーワークスSNS Twitter / Facebook ■ サーバーワークスブログ サーバーワークスエンジニアブログ

In this themed episode of AWS TechChat we explore methods you can use to make a step change in optimizing cost in your AWS account and we are not talking about powering off idle resources. We talk about the new AWS Graviton Processor in the form of the Amazon Elastic Compute Cloud (Amazon EC2) M6G, R6G and C6G. Currently in public preview, meaning you can request access. If your workload can make use of an aarch64 architecture, they can save you in the realm of 40% off your EC2 costs. We also talk about Amazon EC2 Spot Instances that lets you take advantage of unused Amazon EC2 capacity in the AWS Cloud. Spot Instances are available at up to a 90% discount compared to On-Demand price, which is something to hang your hat on. You can consume spot from AWS Fargate - Amazon Elastic Container Service (ECS), Amazon EC2, Application Load Balancer, Amazon Elastic Map Reduce (EMR), AWS CloudFormation, AWS Data Pipleline and AWS Batch. We then pivot to AWS Well-Architected Framework that provides best practices for designing and operating reliable, secure, efficient, and cost-effective systems in the cloud. We hone in on the cost optimization pillar before riffing over some tips from a field solution architect on what they look when trying to trim costs out of an account. Speakers: Shane Baldacchino - Solutions Architect, ANZ, AWS Peter Stanski - Head of Solution Architecture, AWS Resources: Amazon WorkSpaces https://aws.amazon.com/workspaces/pricing/?sc_icampaign=Launch_4027_Workspaces_June30 Amazon Chime https://aws.amazon.com/chime/pricing/?sc_icampaign=Launch_4027_Chime_June30 Working From Home? Here’s How AWS Can Help https://aws.amazon.com/blogs/aws/working-from-home-heres-how-aws-can-help/ AWS Digital Training https://aws.amazon.com/training/course-descriptions/ AWS Graviton Processor https://aws.amazon.com/ec2/graviton/ Amazon EC2 Spot Instances https://aws.amazon.com/ec2/spot/ AWS Well-Architected https://aws.amazon.com/architecture/well-architected/ Cost Optimization https://aws.amazon.com/pricing/cost-optimization/ AWS Events: AWSome Day Online Conference https://aws.amazon.com/events/awsome-day/awsome-day-online/ AWS Innovate AIML Edition on-demand https://aws.amazon.com/events/aws-innovate/machine-learning/ AWS Builders Online Series on-demand https://aws.amazon.com/events/builders-online-series/ AWS Events and Webinars https://aws.amazon.com/events/

스탠다드아웃 71번째 로그에서는 팟캐스트 녹음 환경, 아마존닷컴 반품 경험, gRPC 로드 밸런싱 대한 이야기를 나눴습니다. 참가자: @nacyo_t, @raccoonyy, @seapy, @ecleya 정기 후원 - stdout.fm are creating 프로그래머들의 팟캐스트 | Patreon 녹화 환경 소개 α6400 E-mount camera with APS-C Sensor | ILCE-6400 / ILCE-6400L / ILCE-6400M | Sony US FE 16–35 mm G Master Wide-Angle Zoom Lens | SEL1635GM | Sony US DSC-RX100 III Compact Digital Camera | Cyber-shot Pocket Camera | Sony US 4K HDR Camcorder with Fast Hybrid AF | 4K Handycam FDR-AX700 | Sony US FDR-AX60 | デジタルビデオカメラ Handycam ハンディカム | ソニー 홍진경 더 만두 리뷰를 빙자한 a6400 + SEL1635FE + MixPre-3 II + Shure SM57 - YouTube Good genes: Samsung NX500 review posted: Digital Photography Review 인사이트 번역가 모집 도서출판 인사이트 - BPF Performance Tools: Linux System and Application Observability 역자 모집(마감) 시스템 성능 분석과 최적화 - YES24 DTrace - Wikipedia Berkeley Packet Filter - Wikipedia Brendan Gregg’s Homepage 도서출판 인사이트 - Rust in Action 역자 모집(마감) 알라딘: 클린 아키텍처 알라딘: 클린 코드 Clean Code 기계는 어떻게 생각하는가? - YES24 ODK는 구인중 ODK Media HackerRank 유튜브 익스플로어 지원 중단 3월부터 인터넷 익스플로러로 유튜브 못 본다 : IT : 경제 : 뉴스 : 한겨레 WebP - Wikipedia VP9 - Wikipedia 오라클 신한은행 라이센스 분쟁 오라클 “신한은행, 수백억 내놔라” : 클리앙 AWS Outposts 개요 페이지 데이터 베이스 관리 시스템 | MySQL | Amazon Web Services Amazon Aurora 서버리스 - 온디맨드 Auto-scaling 관계형 데이터베이스 - AWS 판교 낙생지구 남판교에 1만가구 주거타운 뜬다…대장동 이어 낙생지구 개발 - 땅집고 > 투자리포트 아마존 반품 이야기 Dart: The World’s Smallest Laptop Adapter by FINsix — Kickstarter Wireless intrusion prevention system - Wikipedia 속도내는 스마트폰 열풍 구글 넥서스원 국내 첫 개통자나와 | 한경닷컴 아이패드 사용 불법이라더니..장관은 예외? - Chosunbiz > 테크 > ICT/미디어 해외 직구 되팔기 ‘불법’이라는 정부… 직구족들 “현실과 괴리” 불만 - 중앙일보 키크론 Keychron – 맥도 윈도우도 문제없다. 신고 안한 샤넬백, 공항서 걸릴 확률은? Smart Noise Cancelling Headphones 700 | Bose Alibaba.com: Manufacturers, Suppliers, Exporters & Importers from the world’s largest online B2B marketplace gRPC 로드밸런싱 gRPC 웹 서버 로드 밸런싱 | 서버 로드 밸런싱 | Amazon Web Services Envoy Proxy - Home Cloud Map – 클라우드 리소스를 위한 서비스 검색 AWS 라이트세일 Amazon Lightsail, 이제 EC2 업그레이드 경로 제공 DigitalOcean – The developer cloud Amazon EC2 Instance Comparison AWS Batch – 쉽고 효율적인 배치 컴퓨팅 기능 – AWS

Learn how Uber uses AWS Batch to run hundreds of thousands of autonomous vehicle simulations across as many vCPUs every day. Hear the story of how the company built a highly performant and scalable simulation pipeline on native AWS services.

Machine learning involves more than just training models; you need to source and prepare data, engineer features, select algorithms, train and tune models, and then deploy those models and monitor their performance in production. Learn how Amazon Consumer Payments uses Amazon SageMaker, AWS Glue, AWS Step Functions, Amazon API Gateway, AWS Lambda, AWS Batch, Amazon Elastic Container Registry (Amazon ECR), and AWS CloudFormation to automate a CI/CD framework for business-critical machine learning workloads at scale.

It's easier than ever to grow your compute capacity and enable new types of cloud computing applications while maintaining the lowest TCO by blending Amazon EC2 Spot, On-Demand, and Reserved Instance purchase models. In this session, learn how to use the power of Amazon EC2 with other AWS services and features such as Auto Scaling, Amazon ECS, Amazon EKS, Amazon EMR, and AWS Batch to programmatically optimize costs while maintaining high performance and availability- all without breaking a sweat.

Amazon Lex creates conversational interfaces powered by the same deep learning technologies used in Alexa. AWS Batch dynamically provisions the optimal quantity and type of compute resources based on the volume and specific resource requirements of the batch jobs submitted. In this chalk talk, learn how Amazon Lex uses Amazon ECS to dynamically run these batch jobs to create conversational bots.

Simon guides you through lots of new features, services and capabilities that you can take advantage of. Including the new AWS Backup service, more powerful GPU capabilities, new SLAs and much, much more! Chapters: Service Level Agreements 0:17 Storage 0:57 Media Services 5:08 Developer Tools 6:17 Analytics 9:54 AI/ML 12:07 Database 14:47 Networking & Content Delivery 17:32 Compute 19:02 Solutions 21:57 Business Applications 23:38 AWS Cost Management 25:07 Migration & Transfer 25:39 Application Integration 26:07 Management & Governance 26:32 End User Computing 29:22 Links: Topic || Service Level Agreements 0:17 Amazon Kinesis Data Firehose Announces 99.9% Service Level Agreement | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-kinesis-data-firehose-announces-99-9-service-level-agreement/ Amazon Kinesis Data Streams Announces 99.9% Service Level Agreement | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-kinesis-data-streams-announces-99-9-service-level-agreement/ Amazon Kinesis Video Streams Announces 99.9% Service Level Agreement | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-kinesis-video-streams-announces-99-9-service-level-agreement/ Amazon EKS Announces 99.9% Service Level Agreement | https://aws.amazon.com/about-aws/whats-new/2019/01/-amazon-eks-announces-99-9--service-level-agreement-/ Amazon ECR Announces 99.9% Service Level Agreement | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-ecr-announces-99-9--service-level-agreement/ Amazon Cognito Announces 99.9% Service Level Agreement | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-cognito-announces-99-9-service-level-agreement/ AWS Step Functions Announces 99.9% Service Level Agreement | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-step-functions-announces-service-level-agreement/ AWS Secrets Manager Announces Service Level Agreement | https://aws.amazon.com/about-aws/whats-new/2019/01/AWS-Secrets-Manager-announces-service-level-agreement/ Amazon MQ Announces 99.9% Service Level Agreement | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-mq-announces-service-level-agreement/ Topic || Storage 0:57 Introducing AWS Backup | https://aws.amazon.com/about-aws/whats-new/2019/01/introducing-aws-backup/ Introducing Amazon Elastic File System Integration with AWS Backup | https://aws.amazon.com/about-aws/whats-new/2019/01/introducing-amazon-elastic-file-system-integration-with-aws-backup/ AWS Storage Gateway Integrates with AWS Backup - Amazon Web Services | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-storage-gateway-integrates-with-aws-backup-to-protect-volume/ AWS Backup Integrates with Amazon DynamoDB for Centralized and Automated Backup Management | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-backup-integrates-with-amazon-DynamoDB-for-centralized-and-automated-backup-management/ Amazon EBS Integrates with AWS Backup to Protect Your Volumes | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-ebs-integrates-with-aws-backup-to-protect-your-volumes/ AWS Storage Gateway Volume Detach & Attach - Amazon Web Services | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-storage-gateway-introduces-volume-detach-and-attach-feature-/ AWS Storage Gateway - Tape Gateway Performance | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-storage-gateway-announces-increased-throughput-performance-for-tape-gateway/ Amazon FSx for Lustre Offers New Options and Faster Speeds for Working with S3 Data | https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-fsx-for-lustre-offers-new-options-and-faster-speeds/ Topic || Media Services 5:08 AWS Elemental MediaConvert Adds IMF Input and Enhances Caption Burn-In Support | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-elemental-mediaconvert-adds-imf-input-enhances-caption-burn-in-support/ AWS Elemental MediaLive Adds Support for AWS CloudTrail | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-elemental-medialive-adds-support-for-aws-cloudtrail/ AWS Elemental MediaLive Now Supports Resource Tagging | https://aws.amazon.com/about-aws/whats-new/2019/02/aws-elemental-medialive-now-supports-resource-tagging/ AWS Elemental MediaLive Adds I-Frame-Only HLS Manifests and JPEG Outputs | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-elemental-medialive-add-i-frame-only-hls-manifest-and-jpeg-outputs/ Topic || Developer Tools 6:17 Amazon Corretto is Now Generally Available | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-corretto-is-now-generally-available/ AWS CodePipeline Now Supports Deploying to Amazon S3 | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-codepipeline-now-supports-deploying-to-amazon-s3/ AWS Cloud9 Supports AWS CloudTrail Logging | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-cloud9-supports-aws-cloudtrail-logging/ AWS CodeBuild Now Supports Accessing Images from Private Docker Registry | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-codebuild-now-supports-accessing-images-from-private-docker-registry/ Develop and Test AWS Step Functions Workflows Locally | https://aws.amazon.com/about-aws/whats-new/2019/02/develop-and-test-aws-step-functions-workflows-locally/ AWS X-Ray SDK for .NET Core is Now Generally Available | https://aws.amazon.com/about-aws/whats-new/2019/02/aws-x-ray-net-core-sdk-generally-available/ Topic || Analytics 9:54 Amazon Elasticsearch Service doubles maximum cluster capacity with 200 node cluster support | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-elasticsearch-service-doubles-maximum-cluster-capacity-with-200-node-cluster-support/ Amazon Elasticsearch Service announces support for Elasticsearch 6.4 | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-elasticsearch-service-announces-support-for-elasticsearch-6-4/ Amazon Elasticsearch Service now supports three Availability Zone deployments | https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-elasticsearch-service-now-supports-three-availability-zone-deployments/ Now bring your own KDC and enable Kerberos authentication in Amazon EMR | https://aws.amazon.com/about-aws/whats-new/2019/01/now_bring_your_own_kdc_and_enable_kerberos_authentication_in_amazon_emr/ Source code for the AWS Glue Data Catalog client for Apache Hive Metastore is now available for download | https://aws.amazon.com/about-aws/whats-new/2019/02/source-code-for-the-aws-glue-data-catalog-client-for-apache-hive-metatore-is-now-available-for-download/ Topic || AI/ML 12:07 Amazon Comprehend is now Integrated with AWS CloudTrail | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-comprehend-is-now-integrated-with-aws-cloudtrail/ Object Bounding Boxes and More Accurate Object and Scene Detection are now Available for Amazon Rekognition Video | https://aws.amazon.com/about-aws/whats-new/2019/01/object-bounding-boxes-and-more-accurate-object-and-scene-detection-are-now-available-for-amazon-rekognition-video/ Amazon Elastic Inference Now Supports TensorFlow 1.12 with a New Python API | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-elastic-inference-supports-tensorflow-1-12-with-a-python-api/ New in AWS Deep Learning AMIs: Updated Elastic Inference for TensorFlow, TensorBoard 1.12.1, and MMS 1.0.1 | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-deep-learning-amis-now-support-elastic-inference-for-tensorflow-tensorboard1-12-1-mms101/ Amazon SageMaker Batch Transform Now Supports TFRecord Format | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-sagemaker-batch-transform-now-supports-tfrecord-format/ Amazon Transcribe Now Supports US Spanish Speech-to-Text in Real Time | https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-transcribe-now-supports-us-spanish-speech-to-text-in-real-time/ Topic || Database 14:47 Amazon Redshift now runs ANALYZE automatically | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-redshift-auto-analyze/ Introducing Python Shell Jobs in AWS Glue | https://aws.amazon.com/about-aws/whats-new/2019/01/introducing-python-shell-jobs-in-aws-glue/ Amazon RDS for PostgreSQL Now Supports T3 Instance Types | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-rds-postgresql-now-supports-t3-instance-types/ Amazon RDS for Oracle Now Supports T3 Instance Types | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-rds-for-oracle-now-supports-t3-instance-types/ Amazon RDS for Oracle Now Supports SQLT Diagnostics Tool Version 12.2.180725 | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-rds-oracle-now-supports-sqlt-diagnostics-tool-122180725/ Amazon RDS for Oracle Now Supports January 2019 Oracle Patch Set Updates (PSU) and Release Updates (RU) | https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-rds-oracle-supports-jan-2019-oracle-psu/ Amazon DynamoDB Local Adds Support for Transactional APIs, On-Demand Capacity Mode, and 20 GSIs | https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-dynamodb-local-adds-support-for-transactional-apis-on-demand-capacity-mode-and-20-gsis/ Topic || Networking & Content Delivery 17:32 Network Load Balancer Now Supports TLS Termination | https://aws.amazon.com/about-aws/whats-new/2019/01/network-load-balancer-now-supports-tls-termination/ Amazon CloudFront announces six new Edge locations across United States and France | https://aws.amazon.com/about-aws/whats-new/2019/02/cloudfront-feb2019-6locations/ AWS Site-to-Site VPN Now Supports IKEv2 | https://aws.amazon.com/about-aws/whats-new/2019/02/aws-site-to-site-vpn-now-supports-ikev2/ VPC Route Tables Support up to 1,000 Static Routes | https://forums.aws.amazon.com/ann.jspa?annID=6554 Topic || Compute 19:02 Announcing a 25% price reduction for Amazon EC2 X1 Instances in the Asia Pacific (Mumbai) AWS Region | https://aws.amazon.com/about-aws/whats-new/2019/02/announcing-a-25-percent-price-reduction-for-amazon-ec2-x1-instances-in-the-asia-pacific-mumbai-aws-region/ Amazon EKS Achieves ISO and PCI Compliance | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-eks-achieves-iso-and-pci-compliance/ AWS Fargate Now Has Support For AWS PrivateLink | https://aws.amazon.com/about-aws/whats-new/2019/02/aws-fargate-now-has-support-for-aws-privatelink/ AWS Elastic Beanstalk Adds Support for Ruby 2.6 | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-elastic-beanstalk-adds-support-for-ruby-26/ AWS Elastic Beanstalk Adds Support for .NET Core 2.2 | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-elastic-beanstalk-adds-support-for-net-core-22/ Amazon ECS and Amazon ECR now have support for AWS PrivateLink | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-fargate--amazon-ecs--and-amazon-ecr-now-have-support-for-aws/ GPU Support for Amazon ECS now Available | https://aws.amazon.com/about-aws/whats-new/2019/02/gpu-support-for-amazon-ecs-now-available/ AWS Batch now supports Amazon EC2 A1 Instances and EC2 G3s Instances | https://aws.amazon.com/about-aws/whats-new/2019/02/aws-batch-now-supports-amazon-ec2-a1-instances-and-ec2-g3s-insta/ Topic || Solutions 21:57 Deploy Micro Focus Enterprise Server on AWS with New Quick Start | https://aws.amazon.com/about-aws/whats-new/2019/01/deploy-micro-focus-enterprise-server-on-aws-with-new-quick-start/ AWS Public Datasets Now Available from UK Meteorological Office, Queensland Government, University of Pennsylvania, Buildzero, and Others | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-public-datasets-now-available/ Quick Start Update: Active Directory Domain Services on the AWS Cloud | https://aws.amazon.com/about-aws/whats-new/2019/02/quick-start-update-active-directory-domain-services-on-aws/ Introducing the Media2Cloud solution | https://aws.amazon.com/about-aws/whats-new/2019/01/introducing-the-media2cloud-solution/ Topic || Business Applications 23:38 Alexa for Business now offers IT admins simplified workflow to setup shared devices | https://aws.amazon.com/about-aws/whats-new/2019/01/alexa-for-business-now-offers-it-admins-simplified-workflow-to-s/ Topic || AWS Cost Management 25:07 Introducing Normalized Units Information for Amazon EC2 Reservations in AWS Cost Explorer | https://aws.amazon.com/about-aws/whats-new/2019/02/normalized-units-information-for-amazon-ec2-reservations-in-aws-cost-explorer/ Topic || Migration & Transfer 25:39 AWS Migration Hub Now Supports Importing On-Premises Server and Application Data to Track Migration Progress | https://aws.amazon.com/about-aws/whats-new/2019/01/AWSMigrationHubImport/ Topic || Application Integration 26:07 Amazon SNS Message Filtering Adds Support for Multiple String Values in Blacklist Matching | https://aws.amazon.com/about-aws/whats-new/2019/02/amazon-sns-message-filtering-adds-support-for-multiple-string-values-in-blacklist-matching/ Topic || Management & Governance 26:32 AWS Trusted Advisor Expands Functionality With New Best Practice Checks | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-trusted-advisor-expands-functionality/ AWS Systems Manager State Manager Now Supports Management of In-Guest and Instance-Level Configuration | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-systems-manager-state-manager-now-supports-management-of-in-guest-and-instance-level-configuration/ AWS Config Increases Default Limits for AWS Config Rules | https://aws.amazon.com/about-aws/whats-new/2019/01/aws-config-increases-default-limits-for-aws-config-rules/ Introducing AWS CloudFormation UpdateReplacePolicy Attribute | https://aws.amazon.com/about-aws/whats-new/2019/01/introducing-aws-cloudformation-updatereplacepolicy-attribute/ Automate WebSocket API Creation in Amazon API Gateway Using AWS CloudFormation | https://aws.amazon.com/about-aws/whats-new/2019/02/automate-websocket-api-creation-in-api-gateway-with-cloudformation/ AWS OpsWorks for Chef Automate and AWS OpsWorks for Puppet Enterprise Now Support AWS CloudFormation | https://aws.amazon.com/about-aws/whats-new/2019/02/aws-opsworks-for-chef-automate-and-aws-opsworks-for-puppet-enter/ Find And Update Access Keys, Password, And MFA Settings Easily Using The AWS Management Console | https://aws.amazon.com/about-aws/whats-new/2019/01/my-security-credentials/ Amazon CloudWatch Agent Adds Support for Procstat Plugin and Multiple Configuration Files | https://aws.amazon.com/about-aws/whats-new/2019/01/amazon-cloudwatch-agent-adds-support-for-procstat-plugin-and-multiple-configuration-files/ Improve Security Of Your AWS SSO Users Signing In To The User Portal By Using Email-based Verification | https://aws.amazon.com/about-aws/whats-new/2019/01/email-based-verification-for-sso/ Topic || End User Computing 29:22 Introducing Amazon WorkLink | https://aws.amazon.com/about-aws/whats-new/2019/01/introducing-amazon-worklink/ AppStream 2.0 enables custom scripts before session start and after session termination | https://aws.amazon.com/about-aws/whats-new/2019/02/appstream-2-0-enables-custom-scripts-before-session-start-and-af/

Simon takes you through the December updates to finish up 2018! Shownotes: Topic || Customer Engagement 0:23 Amazon Pinpoint Announces Event-Based Campaigns, Driving Personalization and Engagement | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-pinpoint-announces-event-based-campaigns-driving-personalization-and-engagement/ Amazon Pinpoint Announces a New Email Deliverability Dashboard to Help Customers Reach their Users' Inboxes | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-pinpoint-announces-a-new-email-deliverability-dashboard-to-help-customers-reach-their-users-inboxes/ Amazon Connect Adds New Contact API to Get Contact Attributes | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-connect-adds-new-contact-api-to-get-contact-attributes/ Topic || Storage 2:05 Amazon S3 Inventory adds Apache Parquet output format | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-s3-announces-parquet-output-format-for-inventory/ AWS Storage Gateway Increases File Gateway Performance - Amazon Web Services | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-storage-gateway-announces-increased-throughput-and-adds-new-/ Topic || Networking & Content Delivery 3:32 Amazon Virtual Private Clouds can now be shared with other AWS Accounts | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-virtual-private-clouds-can-now-be-shared-with-other-aws-accounts/ Introducing AWS Client VPN to Securely Access AWS and On-Premises Resources | https://aws.amazon.com/about-aws/whats-new/2018/12/introducing-aws-client-vpn-to-securely-access-aws-and-on-premises-resources/ New AWS Direct Connect locations in Silicon Valley and Stockholm | https://aws.amazon.com/about-aws/whats-new/2018/12/new-aws-direct-connect-locations-silicon-valley-stockholm/ Amazon CloudFront announces ten new Edge locations in North America, Europe, and Asia | https://aws.amazon.com/about-aws/whats-new/2018/12/cloudfront-dec2018-10-edge-locations/ Amazon API Gateway Simplifies Building Real-Time Two-Way Communication Applications with WebSocket APIs | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-api-gateway-launches-support-for-websocket-apis/ Amazon Route 53 Adds Alias Record Support For API Gateway and VPC Endpoints | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-route-53-adds-alias-record-support-for-api-gateway-and-vpc-endpoints/ Topic || Database 7:41 Introducing Workload Qualification Framework to Project Plan Your Database Migrations to AWS | https://aws.amazon.com/about-aws/whats-new/2018/12/introducing-workload-qualification-framework-to-plan-your-database-migration-projects/ AWS Database Migration Service Adds Support for Parallel Full Load and Enhanced LOB Migration | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-database-migration-service-adds-support-for-parallel-full-load/ Amazon RDS Enhances Automatic Minor Version Upgrades | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-rds-enhances-auto-minor-version-upgrades/ Amazon RDS for PostgreSQL Now Supports R5 Instance Types | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-rds-postgresql-now-supports-r5-instance-types/ Amazon RDS Supports Publishing PostgreSQL Log Files to Amazon CloudWatch Logs | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-rds-supports-postgresql-logfiles-publish-to-amazon-cloudwatch-logs/ Amazon RDS Performance Insights Supports Counter Metrics for Aurora PostgreSQL | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-rds-performance-insights-supports-counter-metrics-for-aurora-postgresql/ Amazon RDS for PostgreSQL Supports New Minor Versions 10.6, 9.6.11, 9.5.15, and 9.4.20 | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-rds-postgresql-supports-minor-version-106/ Amazon Aurora with PostgreSQL Compatibility Supports PostgreSQL 10.5 | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-aurora-postgresql-supports-postgresql-105/ Amazon Aurora with PostgreSQL Compatibility Adds Query Plan Management | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-aurora-postgresql-compatibility-adds-query-plan-management/ Announcing the New Amazon DynamoDB Key Diagnostics Library | https://aws.amazon.com/about-aws/whats-new/2018/12/announcing-the-new-amazon-dynamodb-key-diagnostics-library/ Amazon DynamoDB Increases the Number of Global Secondary Indexes and Projected Index Attributes You Can Create Per Table | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-dynamodb-increases-the-number-of-global-secondary-indexes-and-projected-index-attributes-you-can-create-per-table/ Amazon DynamoDB Accelerator (DAX) Adds Support for DynamoDB Transactions | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-dynamodb-accelerator-adds-support-for-dynamodb-transactions/ Amazon MQ Now Supports ActiveMQ Minor Version 5.15.8 | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-mq-now-supports-activemq-minor-version5-15-8/ Topic || Compute 14:13 Amazon ECR Console Version 2 | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-ecr-console-version-2/ Amazon ECR now allows Repository Tagging | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-ecr-now-allows-repository-tagging/ Amazon EC2 Introduces Partition Placement Groups | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-ec2-ntroduces-partition-placement-groups/ AWS Auto Scaling is Now Available in 8 more Regions Worldwide and Offers Predictive Scaling for Amazon EC2 | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-auto-scaling-is-now-available-in-8-more-regions-worldwide/ Amazon EC2 C5d, M5d, and R5d Instances are Now Available in Additional AWS Regions | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-ec2-c5d-m5d-and-r5d-instances-are-now-available-in-additional-aws-regions/ AWS Fargate Platform Version 1.3 Adds Secrets Support | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-fargate-platform-version-1-3-adds-secrets-support/ Amazon EKS Adds Managed Cluster Updates and Support for Kubernetes Version 1.11 | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-eks-adds-managed-cluster-updates-and-support-for-kubernetes/ AWS Server Migration Service Adds Support for Multi-Server Migration | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-server-migration-service-adds-support-for-multi-server-migration/ AWS Batch now supports Amazon EC2 C5n Instances Featuring 100 Gbps of Network Bandwidth | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-batch-now-supports-amazon-ec2-c5n-instances-featuring-100-gbps-of-network-bandwidth/ AWS Batch Now Supports Amazon EC2 P3dn Instances | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-batch-now-supports-amazon-ec2-p3dn-instances/ New AWS ParallelCluster Features | https://aws.amazon.com/about-aws/whats-new/2018/12/new-aws-parallelcluster-features/ New SAM PUBLISH Command Simplifies Publishing Applications to the AWS Serverless Application Repository | https://aws.amazon.com/about-aws/whats-new/2018/12/sam-publish-command-simplifies-publishing-apps-to-serverless-application-repository/ AWS Elastic Beanstalk Adds Tag-Based Permissions | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-elastic-beanstalk-adds-tag-based-permissions/ Topic || Developer Tools 20:39 AWS X-Ray Adds the Ability to Group Traces by Root Cause | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-xray-adds-the-ability-to-group-traces-by-root-cause/ AWS CodePipeline Supports VPC Endpoints | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-codepipeline-supports-vpc-endpoints/ AWS CloudFormation macros can now be used in templates with nested stacks | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-cloudformation-macros-can-now-be-used-in-templates-with-nest/ Quickly Create, Build, and Deploy Amazon Alexa Skills from AWS | https://aws.amazon.com/about-aws/whats-new/2018/12/quickly-create-build-and-deploy-amazon-alexa-skills-from-aws/ Topic || Machine Learning 22:07 Amazon Transcribe now supports speech-to-text in French, Italian, and Brazilian Portuguese | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-transcribe-now-supports-speech-to-text-in-french-italian-and-brazilian-portuguese/ Topic || Security, Identity and Compliance 22:27 AWS IAM Console Now Available In German, Portuguese, Spanish, Italian, and Traditional Chinese | https://aws.amazon.com/about-aws/whats-new/2018/12/iam-console-available-in-new-languages/ Automate AWS IAM Permissions Analysis Using the New IAM Access Advisor APIs | https://aws.amazon.com/about-aws/whats-new/2018/12/iam_access_advisor_apis/ Introducing Notifications for New Amazon GuardDuty Finding Types and Feature Releases | https://aws.amazon.com/about-aws/whats-new/2018/12/Introducing-Notifications-for-New-Amazon-GuardDuty-Finding-Types-and-Feature-Releases/ AWS Organizations Supports AWS License Manager Cross Account Sharing Capabilities | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-organizations-supports-aws-license-manager/ AWS Shield Adds Advanced DDoS Protection for AWS Global Accelerator | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-shield-adds-advanced-ddos-protection-for-aws-global-accelerator/ AWS Systems Manager Automation Now Supports at Scale Action | https://aws.amazon.com/about-aws/whats-new/2018/12/AWS-Systems-Manager-Automation-Now-Supports-at-Scale-Actions/ AWS Service Catalog – Integration with AWS Organizations | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-service-catalog-announces-integration-with-aws-organizations/ The AWS WAF Security Automations solution now includes a monitoring dashboard | https://aws.amazon.com/about-aws/whats-new/2018/12/the-aws-waf-security-automations-solution-now-includes-a-monitoring-dashboard/ Announcing rule group exception for Managed Rules for AWS WAF | https://aws.amazon.com/about-aws/whats-new/2018/12/announcing-rule-group-exception-for-managed-rules-for-aws-waf/ AWS Firewall Manager Available in Four Additional Regions | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-firewall-manager-now-available-in-four-more-regions/ Topic || Application Integration 26:59 Amazon SQS now Supports Amazon VPC Endpoints using AWS PrivateLink - Amazon Web Services | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-sqs-vpc-endpoints-aws-privatelink/ Amazon MQ Introduces Network of Brokers Feature | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-mq-introduces-network-of-brokers-feature/ Topic || Desktop & App Streaming 27:27 AppStream 2.0 introduces APIs to simplify app entitlements and enable delivery of virtualized apps | https://aws.amazon.com/about-aws/whats-new/2018/12/appstream-2-0-introduces-apis-to-simplify-app-entitlements-and-e/ Topic || Analytics 28:10 Support for Spark 2.4.0, and Hue 4.3.0 on Amazon EMR release 5.20.0 | https://aws.amazon.com/about-aws/whats-new/2018/12/support-for-spark-240-hue-430-on-amazon-emr-release-5200/ Amazon Redshift now runs VACUUM DELETE automatically | https://aws.amazon.com/about-aws/whats-new/2018/12/amazon-redshift-automatic-vacuum/ Topic || Internet of Things 29:41 Introducing AWS CloudFormation Template Support for AWS IoT Analytics | https://aws.amazon.com/about-aws/whats-new/2018/12/introducing-aws-cloudformation-template-support-for-aws-iot-analytics/ AWS IoT Device Defender Adds Support for Two New Security Metrics | https://aws.amazon.com/about-aws/whats-new/2018/12/aws-iot-device-defender-adds-support-for-two-new-security-metrics/ MediaTek MT7697H System on Chip is Qualified for Amazon FreeRTOS | https://aws.amazon.com/about-aws/whats-new/2018/12/mediatek-mt7697h-system-on-chip-qualified-amazon-freertos/ Topic || Other 30:35 Announcing Programmatic Access to AWS Pricing Information in China via the AWS Price List API | https://aws.amazon.com/about-aws/whats-new/2018/12/announcing-aws-price-list-api-availability-in-china/ Introducing the Media Services Application Mapper | https://aws.amazon.com/about-aws/whats-new/2018/12/introducing-the-media-services-application-mapper/ New Quick Start Deploys Varnish Cache Plus (VCP) on the AWS Cloud | https://aws.amazon.com/about-aws/whats-new/2018/12/new-quick-start-deploys-varnish-on-aws/ Announcing 15 Free Digital Training Courses on New AWS Services Launched at re:Invent 2018 | https://aws.amazon.com/about-aws/whats-new/2018/12/announcing-15-free-digital-training-courses-on-new-aws-services-launched-at-re-invent-2018/

Corteva Agriscience, the agricultural division of DowDuPont, produces as much DNA sequence data every six hours as existed in the entire public sphere in 2008. On-premises processing and storage could not scale to meet the business demand. Partnering with Sogeti (part of Capgemini), Corteva replatformed their existing Hadoop-based genome processing systems into AWS using a serverless, cloud-native architecture. In this session, learn how Corteva Agriscience met current and future data processing demands without maintaining any long-running servers by using AWS Lambda, Amazon S3, Amazon API Gateway, Amazon EMR, AWS Glue, AWS Batch, and more. This session is brought to you by AWS partner, Capgemini America. Complete Title: AWS re:Invent 2018: Petabytes of Data & No Servers: Corteva Scales DNA Analysis to Meet Increasing Business Demand (ENT218-S)

When analyzing information for fraud detection, tasks must be run periodically. When building a fraud detection system, start by preparing the data, and work with small chunks of data and run parallel jobs so your machine learning (ML) models can predict fraudulent activity. For that, you schedule computer resources and, of course, the script. With AWS Batch, you only worry about your application job and run it at scale. With containers, you think in small processes and let AWS Batch run them concurrently. In this session, learn to build a fraud detection system and integrate it with other AWS services. This session is part of re:Invent Developer Community Day, a series led by AWS enthusiasts who share first-hand, technical insights on trending topics.

Manufacturing companies in all sectors—including automotive, aerospace, semiconductor, and industrial manufacturing—rely on design and engineering software in their product development processes. These computationally-intensive applications—such as computer-aided design and engineering (CAD and CAE), electronic design automation (EDA), other performance-critical applications—require immense scale and orchestration to meet the demands of today's manufacturing requirements. In this session, you learn how to achieve the maximum possible performance and throughput from design and engineering workloads running on Amazon EC2, elastic GPUs, and managed services such as AWS Batch and Amazon AppStream 2.0. We demonstrate specific optimization techniques and share samples on how to accelerate batch and interactive workloads on AWS. We also demonstrate how to extend and migrate on-premises, high performance compute workloads with AWS, and use a combination of On-Demand Instances, Reserved Instances, and Spot Instances to minimize costs.

AWS Batch is a fully managed service that enables developers to easily and efficiently run batch computing workloads of any scale on AWS. AWS Batch automatically provisions the right quantity and type of compute resources needed to run your jobs. With AWS Batch, you don't need to install or manage batch computing software, so you can focus on analyzing results and solving problems. In this session, the principal product manager for AWS Batch, Jamie Kinney, describes the core concepts behind AWS Batch and details of how the service functions. The presenter then demonstrates the latest features of AWS Batch with relevant use cases and sample code before describing some of the upcoming features for the service. Finally, hear from AWS Batch customers as they describe why and how they are using AWS Batch. This portion of the talk is delivered by representatives from the University of Utah, Autodesk, and AdRoll.

Batch processing is useful to analyze large amounts of data. But configuring and scaling a cluster of virtual machines to process complex batch jobs can be difficult. In this talk, we'll show how to use containers on AWS for batch processing jobs that can scale quickly and cost-effectively. We will also discuss AWS Batch, our fully managed batch-processing service. You'll also hear from GoPro and Here about how they use AWS to run batch processing jobs at scale including best practices for ensuring efficient scheduling, fine-grained monitoring, compute resource automatic scaling, and security for your batch jobs.

In this episode Simon gets you caught up on some useful new services including those to help you with ETL, Migration and Data Leakage protection! Shownotes: AWS Glue: https://aws.amazon.com/blogs/aws/launch-aws-glue-now-generally-available/ AWS CloudHSM: https://aws.amazon.com/blogs/aws/aws-cloudhsm-update-cost-effective-hardware-key-management/ AWS Macie: https://aws.amazon.com/blogs/aws/launch-amazon-macie-securing-your-s3-buckets/ AWS IAM Console: https://aws.amazon.com/about-aws/whats-new/2017/07/the-aws-iam-console-now-remembers-your-preferences-for-table-column-selections-and-policy-viewing-and-editing/ AWS Migration Hub: https://aws.amazon.com/blogs/aws/aws-migration-hub-plan-track-enterprise-application-migration/ Amazon EFS Encryption at Rest: https://aws.amazon.com/blogs/aws/new-encryption-at-rest-for-amazon-elastic-file-system-efs/ AWS Batch and CloudFormation: https://aws.amazon.com/about-aws/whats-new/2017/08/aws-batch-adds-support-for-aws-cloudformation/ AWS SAM Local: https://aws.amazon.com/blogs/aws/new-aws-sam-local-beta-build-and-test-serverless-applications-locally/ EC2 Systems Manager Maintenance Windows: https://aws.amazon.com/blogs/mt/maintenance-windows-support-for-new-task-types-using-amazon-ec2-systems-manager/ AWS CloudTrail in Amazon Lex: https://aws.amazon.com/about-aws/whats-new/2017/08/aws-cloudtrail-integration-is-now-available-in-amazon-lex/

AWS Batch is a fully-managed service that enables developers, scientists, and engineers to easily and efficiently run batch computing workloads of any scale on AWS. AWS Batch automatically provisions compute resources and optimizes the workload distribution based on the quantity and scale of the workloads. With AWS Batch, there is no need to install or manage batch computing software, allowing you to focus on analyzing results and solving problems. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features, such as Amazon EC2, Spot Instances, and AWS Lambda. AWS Batch reduces operational complexities, saving time and reducing costs. In this session, Principal Product Managers Jamie Kinney and Dougal Ballantyne describe the core concepts behind AWS Batch and details of how the service functions. The presentation concludes with relevant use cases and sample code.