Oracle University Podcast

In this episode, Lois Houston and Nikita Abraham, along with Nick Wagner, Senior Director of Product Management, dive into the Replicat process in Oracle GoldenGate 23ai.   They discuss how Replicat applies changes to the target database, highlighting the different types: Classic, Coordinated, and Parallel Replicat.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode. ---------------------------------------------------------------- Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Hello and welcome to another episode of the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead: Editorial Services.  Nikita: Hi everyone! If you've been listening to us these last few weeks, you'll know we've been discussing the fundamentals of GoldenGate 23ai. Today is going to be all about the Replicat process. Again, this is something we've discussed briefly in earlier episodes, but just to recap, the Replicat process applies changes from the source database to the target database. It's responsible for reading trail files and applying the changes to the target system. 01:04 Lois: That's right, Niki. And we'll be chatting with Nick Wagner, Senior Director of Product Management for Oracle GoldenGate. Hi Nick! Thanks for joining us again today. Let's get straight into it. Can you give us an overview of the Replicat process? Nick: One thing that's very important is the Replicat is extremely chatty with that target database. So it's going to be going in and trying to make lots of little transactions on that system. The Replicat process only issues single row DML. So if you can imagine a source database that's generating hundreds of thousands of changes per second, we're going to have to have a Replicat process that can do 100,000 changes per second on that target site. That means that it's going to have to send a lot of little one record commands. And so we've got a lot of ways to optimize that. But in all situations you're really going to want very, very low ping time between that Replicat process and that target database. This often means that if you're going to be running GoldenGate in a cloud, you're going to want the Cloud GoldenGate environment to be running in that target data center, wherever that target database is. 02:06 Lois: What are the key characteristics of the process, Nick? Nick: Replicat process is going to read the changes from the trail file and then apply them to the target system, just like any database user would. It's not doing anything special where it's going under the covers and trying to apply directly to the database blocks. It's just applying regular standard insert, update, delete, and DDL statements to that target database. A single trail file does support high volume of data replication activity depending on the type of Replicat. Replicats do preserve the boundary of their transactions. So in the situations, by default, a transaction that's on the source, let's say five inserts followed by a commit will remain five inserts followed by a commit on the target site. There are some operations and changes that do affect this, but they're not turned on by default. There are things like group transactions that allows you to group multiple transactions into a single commit. This one could actually improve performance in some cases. We also have batch SQL that can change the boundaries of a transaction as well. And then in a Parallel Replicat, you actually have the ability to split a large transaction into multiple chunks and apply those chunks in Parallel. So again, by default, it's going to preserve the boundaries, but there are ways to change that. And then the Replicats use a checkpoint table to help with recovery and to know where they're applying data and what they've done. The other thing in here is, like an Extract process can write to multiple trails and write subsets of data to each one, a Replicat can only process a single set of trail files at once. So it's going to be attached to a specific trail file like trail file AB, and will only be able to read changes from trail file AB. If I have multiple trails that need to be applied into a target system, then I have to set up multiple Replicats to handle that. 03:54 Nikita: So, what are the different Replicat types, Nick? Nick: We have three types in the product today. We have Classic Replicat, which should really only be used for testing purposes or in environments that don't support any of the other specialized Replicats. We have Coordinated Replicat, which is a high speed apply mechanism to apply data into a target system. It does have some parallelism in it, but it's user defined parallelism. And then we have our flagship and that's Parallel Replicat. And this is the most performant lowest latency Replicat that we have. 04:25 Lois: Ok. Let's dive a little deeper into each of them, starting with the Classic Replicat. How does it work? Nick: It's pretty straightforward. You're going to have a process that reads the trail files, and then in a single threaded fashion it's going to take the trail file logical change record, convert it to an insert, update, or delete, and then apply it into that target database. Each transaction that it does is preceded by a change to the checkpoint table. So when the transaction that the Replicat is currently doing is committed, that checkpoint table update also gets committed. That way when the Replicat restarts, it knows exactly what transaction it left off and how it last applied the record. And all the Replicats work the same way with regards to checkpoint tables. They each have their own little method of ensuring that the transaction they're applying is also reflected within the checkpoint table so that when it restarts, it knows exactly where it happened. That way, if a Replicat dies in the middle of a transaction, it can be restarted without any duplicate data or without missing data. 05:29 Did you know that Oracle University offers free courses on Oracle Cloud Infrastructure? You'll find training on everything from multicloud, database, networking, and security to artificial intelligence and machine learning, all free for our subscribers. So, what are you waiting for? Pick a topic, head over to mylearn.oracle.com, and get started. 05:53 Nikita: Welcome back! Moving on, what about Coordinated Replicat? Nick: The Coordinated Replicat is going to read from a set of trail files. It's going to have multiple threads that do this. So you have your base thread, your coordinated thread that's going to be thread 1. It's going to process the data and apply it into that target database. You then have thread 2, 4, 5, 6, and so on. When you set up your Replicat parameter file for a Coordinated Replicat, the map commands that maps from one table on the source to a table on the target has an additional option. So you'll have an option called a range or thread range. With the range and thread range option, you can actually tell which table to go into which thread. 06:38 Lois: Can you give us an example of this? Nick: So I could say map Scott.M into thread 1 and I want Scott.Dept into thread 2. Well, this is fantastic until you realize that Scott.M and Scott.Dept have a foreign key between them or a child dependencies, parent-child relationships. What that means is that now I'm going to have to disable that foreign key on the target site, because there's no way for GoldenGate to coordinate the changes in one thread to another thread. And so you really have to be careful on how you pair your tables together. If you don't have any referential integrity on that target database, then you can use parallel coordinated Replicat to really high degrees of parallelism, and you get some very good performance out of it. Let's say that you have a table that's really got too much data for even a single thread to process, that's where the thread range comes in. And thread range command will use something like the table's primary key to split transactions on that table across multiple threads. So I can say, hey, take my table Scott.M and I want to spread transactions across threads 10, 11, 12, 13, and 14 and then spread them evenly based on the primary key. And Coordinated Replicat will do that. So you can get some very high performance numbers out of it and you can really fine tune the tables, especially if you know the amount of data coming into each one. While this does work great, we observed that a lot of customers really don't know their applications to that level of detail, and so we needed a different method to push data into that target database, where we could define the parallelism based on the database expectations. So instead of the customer having to try and figure out what are the parent-child relationships, why can't GoldenGate do it for me? And that led to Parallel Replicat.  08:26 Nikita: And what are the benefits and features of the Parallel Replicat process?  Nick: So Parallel Replicat has been around for quite a few years now. It supports most targets, it was Oracle initially, but now it's been expanded out to a lot of the non-Oracle targets and even some of the nonrelational database targets. It has absolutely the best performance of any Replicat process out there. You can use it to split large transactions as well. So if all of a sudden you have a batch job that does a million inserts followed by a single commit, I can split that across 10 threads, each thread doing 100,000 inserts. And it's aware of your transaction dependencies, that's the cool thing. So in Coordinated Replicat, you had to worry about how to split your tables up, in Parallel Replicat, we do it for you. 09:11 Lois: And how does Parallel Replicat work? Nick: So there's three main processes to the Parallel Replicat. You have your first is the mapper process. This is going to be responsible for taking the data out of the trail files and putting them into kind of our collator and scheduler box. As transactions go from the trail file, they get put into this box in memory where they're processed. There's a collator process that will look at these processes and go, OK, as they're coming in, let me read some of the data in them to determine how they can be applied in Parallel or not. And so the collator process understands the foreign key dependencies on that target database. And it's able to say, hey, I know that my two tables are these two tables, have a parent-child relationship, I need to make sure that changes on those tables go in the correct order. And so if all of a sudden you see an insert using the parent record and then another insert into the child record and they're mapped together, GoldenGate will ensure that those two transactions go serially and not parallel where they could get applied out of order. There's then a scheduler process that's going to look at this and say, OK, now that I'm taking transactions from the collator process, who's already identified whether or not transactions can be applied in parallel or serial, and I'm going to feed them off to applier processes that are ready and waiting for me to apply those changes into the database. And then the applier process is waiting for the scheduler process to send its transactions and say, OK, what's my next one? Where's the next transaction I should be working on and applying? And then the applier process is the one actually applying the changes into that target database, again, just using standard DML operations. So there's a lot of benefits to this one. You don't need to worry about your foreign key dependencies, you can leave all your foreign keys enabled. The collator process will actually use information within the trail file to determine which transactions can be applied in parallel, and which one needs to be applied serially. 11:13 Lois: Thank you, Nick, for this insightful conversation. There's loads more to discover about the Replicat process, and you can do that by heading over to mylearn.oracle.com and searching for the Oracle GoldenGate 23ai: Fundamentals course. Nikita: In our next episode, Nick will take us through managing Extract Trails and Files. Until then, this is Nikita Abraham… Lois: And Lois Houston, signing off! 11:37 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, Lois Houston and Nikita Abraham dive into key components of Oracle GoldenGate 23ai with expert insights from Nick Wagner, Senior Director of Product Management.   They break down the Distribution Service, explaining how it moves trail files between environments, replaces the classic extract pump, and ensures secure data transfer. Nick also introduces Target Initiated Paths, a method for connecting less secure environments to more secure ones, and discusses how the Receiver Service simplifies monitoring and management. The episode wraps up with a look into Initial Load, covering different methods for syncing source and target databases without downtime.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode. ----------------------------------------------------------------- Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and with me is Lois Houston, Director of Innovation Programs.  Lois: Hey there! Last week, we spoke about the Extract process and today we're going to spend time discussing the Distribution Path, Target Initiated Path, Receiver Server, and Initial Load. These are all critical components of the GoldenGate architecture, and understanding how they work together is essential for successful data replication. 00:58 Nikita: To help us navigate these topics, we've got Nick Wagner joining us again. Nick is a Senior Director of Product Management for Oracle GoldenGate. Hi Nick! Thanks for being with us today. To kick things off, can you tell us what the distribution service is and how it works? Nick: A distribution path is used when we need to send trail files between two different GoldenGate environments. The distribution service replaces the extract pump that was used in GoldenGate classic architecture. And so the distribution service will send the trail files as they're being created to that receiver service and it will write the trail files over on the target system. The distribution service works in a kind of a streaming fashion, so it's constantly pulling the trail files that the extract is creating to see if there's any new data. As soon as it sees new data, it'll packet it up and send it across the network to the receiver service. It can use a couple of different methods to do this. The most secure and recommended method is using a WebSocket secure connection or WSS. If you're going between a microservices and a classic architecture, you can actually tell the distribution service to send it using the classic architecture method. In that case, it's the OGG option when you're configuring the distribution service. There's also some unsecured methods that would send the trail files in plain text. The receiver service is then responsible for taking that data and rewriting it into the trail file on the target site. 02:23 Lois: Nick, what are some of the key features and responsibilities of the distribution service? Nick: It's responsible for command deployment. So any time that you're going to actually make a command to the distribution service, it gets handled there directly. It can handle multiple commands concurrently. It's going to dispatch trail files to one or more receiver servers so you can actually have a single distribution path, send trail files to multiple targets. It can provide some lightweight filtering so you can decide which tables get sent to the target system. And it also is integrated in with our data streams, our pub and subscribe model that we've added in GoldenGate 23ai. 03:01 Lois: Interesting. And are there any protocols to remember when using the distribution service? Nick: We always recommend a secure WebSocket. You also have proxy support for use within cloud environments. And then if you're going to a classic architecture GoldenGate, you would use the Oracle GoldenGate protocol. So in order to communicate with the distribution service and send it commands, you can communicate directly from any web browser, client software-- installation is not required-- or you can also do it through the admin client if necessary, but you can do it directly through browsers. 03:33 Nikita: Ok, let's move on to the target initiated path. Nick, what is it and what does it do essentially? Nick: This is used when you're communicating from a less secure environment to a more secure environment. Often, this requires going through some sort of DMZ. In these situations, a connection cannot be established from the less secure environment into the more secure environment. It actually needs to be established from the more secure environment out. And so if we need to replicate data into a more secure environment, we need to actually have the target GoldenGate environment initiate that connection so that it can be established.  And that's what a target-initiated path does. 04:12 Lois: And how do you set it up? Nick: It's pretty straightforward to set up. You actually don't even need to worry about it on the source side. You actually set it up and configure it from the target. The receiver service is responsible for receiving the trail file data and writing it to the local trail file. In this situation, we have a target-initiated path created. And so that receiver service is going to write the trail files locally and the replicat is going to apply that data into that target system. 04:37 Nikita: I also want to ask you about the Receiver service. What is it really? Nick: Receiver service is pretty straightforward. It's a centrally controlled service. It allows you to view the status of your distribution path and replaces target side collectors that were available in the classic architecture of GoldenGate. You can also get statistics about the receiver service directly from the web UI.  You can get detailed information about these paths by going into the receiver service and identifying information like network details, transfer protocols, how many bytes it's received, how many bytes it's sent out. If you need to issue commands from the admin client to the receiver service, you can use the info command to get details about it. Info all will tell you everything that's running. And you can see that your receiver service is up and running. 05:28 Are you working towards an Oracle Certification this year? Join us at one of our certification prep live events in the Oracle University Learning Community. Get insider tips from seasoned experts and learn from others who have already taken their certifications. Go to community.oracle.com/ou to jump-start your journey towards certification today! 05:53 Nikita: Welcome back. In the last section of today's episode, we'll cover what Initial Load is. Nick, can you break down the basics for us? Nick: So, the initial load is really used when you need to synchronize the source and target systems. Because GoldenGate is designed for 24/7 environments, we need to be able to do that initial load without taking downtime on the source. And so all the methods that we talk about do not require any downtime for that source database. 06:18 Lois: How do you do the initial load? Nick: So there's a couple of different ways to do the initial load. And it really depends on what your topology is. If I'm doing like-to-like replication in a homogeneous environment, we'll say Oracle-to-Oracle, the best options are to use something that's integrated with GoldenGate, some sort of precise instantiation method that does not require HandleCollisions. That's something like a database backup and restoring it to a specific SDN or CSN value using a Database Snapshot. Or in some cases, we can use Oracle Data Pump integration with GoldenGate. There are some less precise instantiation options, which do require HandleCollisions. We also have dissimilar initial load methods. And this is typically when you're going between heterogeneous environments. When my source and target databases don't match and there isn't any kind of fast unload or fast load utility that I could use between those two databases. In almost all cases, this does require HandleCollisions to be used. 07:16 Nikita: Got it. So, with so many options available, are there any advantages to using GoldenGate's own initial load method?  Nick: While some databases do have very good fast load and unload utilities, there are some advantages to using GoldenGate's own initial load method. One, it supports heterogeneous replication environments. So if I'm going from Postgres to Oracle, it'll do all the data type transformation, character set transformation for me. It doesn't require any downtime, if certain conditions are met.  It actually performs transformation as the data is loaded, too, as well as filtering. And so any transformation that you would be doing in your normal transaction log replication or CDC replication can also go through the same transformation for the initial load process. GoldenGate's initial load process does read directly from the source tables. And it fetches the data in arrays. It also uses parallel processing to speed up the replication. It does also handle activity on the source tables during the initial load process, so you do not need to worry about quiescing that source database. And a lot of the initial load methods directly built into GoldenGate support distributed application analytics targets, including things like Databricks, Snowflake, BigQuery. 08:28 Lois: And what about its limitations? Or to put it differently, when should users consider using different methods? Nick: So the first thing to consider is system proximity. We want to make sure that the two systems we're working with are close together. Or if not, how are we going to send the data across? One thing to keep in mind, when we do the initial load, the source database is not quiesced. So if it takes an hour to do the initial load or 10 hours, it really doesn't matter to GoldenGate. So that's something to keep in mind. Even though we talk about performance of this, the performance really isn't as critical as one might suspect. So the important thing about data system proximity is the proximity to the extract and replicat processes that are going to be pulling the data out and pushing it across. And then how much data is generated? Are we talking about a database that's just a couple of gigabytes? Or are we talking about a database that's hundreds of terabytes? Do we want to consider outage time? Would it be faster to take a little bit of outage and use some other method to move the data across? What kind of outage or downtime windows do we have for these environments? And then another consideration is disk space. As we're pulling the data out of that source database, we need to have somewhere to store it. And if we don't have enough disk space, we need to run to temporary space or to use multiple external drives to be able to support it. So these are all different considerations. 09:50 Nikita: I think we can wind up our episode with that. Thanks, Nick, for giving us your insights.  Lois: If you'd like to learn more about the topics we covered today, head over to mylearn.oracle.com and check out the Oracle GoldenGate 23ai: Fundamentals course. Nikita: In our next episode, Nick will take us through the Replicat process. Until then, this is Nikita Abraham… Lois: And, Lois Houston signing off! 10:14 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

The Extract process is the heart of Oracle GoldenGate 23ai, capturing data changes with precision. In this episode, Lois Houston and Nikita Abraham sit down with Nick Wagner, Senior Director of Product Management, to break down Extract's role, architecture, and best practices.   Learn how Extract works across different setups, from running on source databases to using a Hub model for greater flexibility. Additionally, understand how trail files, parameter files, and naming conventions impact performance.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.

Installing Oracle GoldenGate 23ai is more than just running a setup file—it's about preparing your system for efficient, reliable data replication. In this episode, Lois Houston and Nikita welcome back Nick Wagner to break down system requirements, storage considerations, and best practices for installing GoldenGate.   You'll learn how to optimize disk space, manage trail files, and configure network settings to ensure a smooth installation.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   -------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Hello and welcome to Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and I'm joined by Lois Houston, Director of Innovation Programs.  Lois: Hi there! Last week, we took a close look at the security strategies of Oracle GoldenGate 23ai. In this episode, we'll discuss all aspects of installing GoldenGate. 00:48 Nikita: That's right, Lois. And back with us today is Nick Wagner, Senior Director of Product Management for GoldenGate at Oracle. Hi Nick! I'm going to get straight into it. What are the system requirements for a typical GoldenGate installation? Nick: As far as system requirements, we're going to split that into two sections. We've got an operating system requirements and a storage requirements. So with memory and disk, and I know that this isn't the answer you want, but the answer is that it varies. With GoldenGate, the amount of CPU usage that is required depends on the number of extracts and replicats. It also depends on the number of threads that you're going to be using for those replicats. Same thing with RAM and disk usage. That's going to vary on the transaction sizes and the number of long running transactions. 01:35 Lois: And how does the recovery process in GoldenGate impact system resources?  Nick: You've got two things that help the extract recovery. You've got the bonded recovery that will store transactions over a certain length of time to disk. It also has a cache manager setting that determines what gets written to disk as part of open transactions. It's not just the simple answer as, oh, it needs this much space. GoldenGate also needs to store trail files for the data that it's moving across. So if there's network latency, or if you expect a certain network outage, or you have certain SLAs for the target database that may not be met, you need to make sure that GoldenGate has enough room to store its trail files as it's writing them. The good news about all this is that you can track it. You can use parameters to set them. And we do have some metrics that we'll provide to you on how to size these environments. So a couple of things on the disk usage. The actual installation of GoldenGate is about 1 to 1.5 gig in size, depending on which version of GoldenGate you're going to be using and what database. The trail files themselves, they default to 500 megabytes apiece. A lot of customers keep them on disk longer than they're necessary, and so there's all sorts of purging options available in GoldenGate. But you can set up purge rules to say, hey, I want to get rid of my trail files as soon as they're not needed anymore. But you can also say, you know what? I want to keep my trail files around for x number of days, even if they're not needed. That way they can be rebuilt. I can restore from any previous point in time. 03:15 Nikita: Let's talk a bit more about trail files. How do these files grow and what settings can users adjust to manage their storage efficiently? Nick: The trail files grow at about 30% to 35% of the generated redo log data. So if I'm generating 100 gigabytes of redo an hour, then you can expect the trail files to be anywhere from 30 to 35 gigabytes an hour of generated data. And this is if you're replicating everything. Again, GoldenGate's got so many different options. There's so many different ways to use it. In some cases, if you're going to a distributed applications and analytics environment, like a Databricks or a Snowflake, you might want to write more information to the trail file than what's necessary. Maybe I want additional information, such as when this change happened, who the user was that made that change. I can add specific token data. You can also tell GoldenGate to log additional records or additional columns to the trail file that may not have been changed. So I can always say, hey, GoldenGate, replicate and store the entire before and after image of every single row change to your trail file, even if those columns didn't change. And so there's a lot of different ways to do it there. But generally speaking, the default settings, you're looking at about 30% to 35% of the generated redo log value. System swap can fill up quickly. You do want this as a dedicated disk as well. System swap is often used for just handling of the changes, as GoldenGate flushes data from memory down to disk. These are controlled by a couple of parameters. So because GoldenGate is only writing committed activity to the trail file, the log reader inside the database is actually giving GoldenGate not only committed activity but uncommitted activity, too. And this is so it can stay very high speed and very low latency. 05:17 Lois: So, what are the parameters? Nick: There's a cache manager overall feature, and there's a cache directory. That directory controls where that data is actually stored, so you can specify the location of the uncommitted transactions. You can also specify the cache size. And there's not only memory settings here, but there's also disk settings. So you can say, hey, once a cache size exceeds a certain memory usage, then start flushing to disk, which is going to be slower. This is for systems that maybe have less memory but more high-speed disk. You can optimize these parameters as necessary. 05:53 Nikita: And how does GoldenGate adjust these parameters? Nick: For most environments, you're just going to leave them alone. They're automatically configured to look at the system memory available on that system and not use it all. And then as soon as necessary, it'll overflow to disk. There's also intelligent settings built within these parameters and within the cache manager itself that if it starts seeing a lull in activity or your traditional OLTP type responses to actually free up the memory that it has allocated. Or if it starts seeing more activity around data warehousing type things where you're doing large transactions, it'll actually hold on to memory a little bit longer. So it kinda learns as it goes through your environment and starts replicating data. 06:37 Lois: Is there anything else you think we should talk about before we move on to installing GoldenGate?  Nick: There's a couple additional things you need to think of with the network as well. So when you're deploying GoldenGate, you definitely want it to use the fastest network.  GoldenGate can also use a reverse proxy, especially important with microservices. Reverse proxy, typically we recommend Nginx. And it allows you to access any of the GoldenGate microservices using a single port.  GoldenGate also needs either host names or IP addresses to do its communication and to ensure the system is available. It does a lot of communication through TCP and IP as well as WSS. And then it also handles firewalls. So you want to make sure that the firewalls are open for ingress and egress for GoldenGate, too. There's a couple of different privileges that GoldenGate needs when you go to install it. You'll want to make sure that GoldenGate has the ability to write to the home where you're installing it. That's kind of obvious, but we need to say it anyways. There's a utility called oggca.sh. That's the GoldenGate Configuration Assistant that allows you to set up your first deployments and manage additional deployments. That needs permissions to write to the directories where you're going to be creating the new deployments. The extract process needs connection and permissions to read the transaction logs or backups. This is not important for Oracle, but for non-Oracle it is. And then we also recommend a dedicated database user for the extract and replicat connections. 08:15 Are you keen to stay ahead in today's fast-paced world? We've got your back! Each quarter, Oracle rolls out game-changing updates to its Fusion Cloud Applications. And to make sure you're always in the know, we offer New Features courses that give you an insider's look at all of the latest advancements. Don't miss out! Head over to mylearn.oracle.com to get started. 08:41 Nikita: Welcome back! So Nick, how do we get started with the installation?  Nick: So when we go to the install, the first thing you're going to do is go ahead and go to Oracle's website and download the software. Because of the way that GoldenGate works, there's only a couple moving parts. You saw the microservices. There's five or six of them. You have your extract, your replicat, your distribution service, trail files. There's not a lot of moving components. So if something does go wrong, usually it affects multiple customers. And so it's very important that when you go to install GoldenGate, you're using the most recent bundle patch. And you can find this within My Oracle Support. It's not always available directly from OTN or from the Oracle e-delivery website. You can still get them there, but we really want people going to My Oracle Support to download the most recent version. There's a couple of environment variables and certificates that you'll set up as well. And then you'll run the Configuration Assistant to create your deployments.  09:44 Lois: Thanks, Nick, for taking us though the installation of GoldenGate. Because these are highly UI-driven topics, we recommend that you take a deep dive into the GoldenGate 23ai Fundamentals course, available on mylearn.oracle.com. Nikita: In our next episode, we'll talk about the Extract process. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 10:08 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

GoldenGate 23ai takes security seriously, and this episode unpacks everything you need to know. GoldenGate expert Nick Wagner breaks down how authentication, access roles, and encryption protect your data.   Learn how GoldenGate integrates with identity providers, secures communication, and keeps passwords out of storage. Understand how trail files work, why they only store committed data, and how recovery processes prevent data loss.   Whether you manage replication or just want to tighten security, this episode gives you the details to lock things down without slowing operations.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Hello and welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead: Editorial Services.  Nikita: Welcome, everyone! This is our fourth episode on Oracle GoldenGate 23ai. Last week, we discussed the terminology, different processes and what they do, and the architecture of the product at a high level. Today, we have Nick Wagner back with us to talk about the security strategies of GoldenGate. 00:56 Lois: As you know by now, Nick is a Senior Director of Product Management for GoldenGate at Oracle. He's played a key role as one of the product designers behind the latest version of GoldenGate. Hi Nick! Thank you for joining us again. Can you tell us how GoldenGate takes care of data security? Nick: So GoldenGate authentication and authorization is done in a couple of different ways. First, we have user credentials for GoldenGate for not only the source and target databases, but also for GoldenGate itself. We have integration with third-party identity management products, and everything that GoldenGate does can be secured. 01:32 Nikita: And we must have some access roles, right? Nick: There's four roles built into the GoldenGate product. You have your security role, administrator, operator, and user. They're all hierarchical. The most important one is the security user. This user is going to be the one that provides the administrative tasks. This user is able to actually create additional users and assign roles within the product. So do not lose this password and this user is extremely important. You probably don't want to use this security user as your everyday user. That would be your administrator. The administrator role is able to perform all administrative tasks within GoldenGate. So not only can they go in and create new extracts, create new replicats, create new distribution services, but they can also start and stop them. And that's where the operator role is and the user role. So the operator role allows you to go in and start/stop processes, but you can't create any new ones, which is kind of important. So this user would be the one that could go in and suspend activity. They could restart activity. But they can't actually add objects to replication. The user role is really a read-only role. They can come in. They can see what's going on. They can look at the log files. They can look at the alerts. They can look at all the watches and see exactly what GoldenGate is doing. But they're unable to make any changes to the product itself. 02:54 Lois: You mentioned the roles are hierarchical in nature. What does that mean? Nick: So anything that the user role does can be done by the operator. Anything that the operator and user roles can do can be done by the administrator. And anything that the user, operator, and administrator roles do can be done by the security role. 03:11 Lois: Ok. So, is there a single sign-on available for GoldenGate? Nick: We also have a password plugin for GoldenGate Connections. A lot of customers have asked for integration with whatever their single sign-on utility is, and so GoldenGate now has that with GoldenGate 23ai. So these are customer-created entities. So, we have some examples that you can use in our documentation on how to set up an identity provider or a third-party identity provider with GoldenGate. And this allows you to ensure that your corporate standards are met. As we started looking into this, as we started designing it, every single customer wanted something different. And so instead of trying to meet the needs for every customer and every possible combination of security credentials, we want you to be able to design it the way you need it. The passwords are never stored. They're only retrieved from the identity provider by the plugin itself. 04:05 Nikita: That's a pretty important security aspect…that when it's time to authenticate a user, we go to the identity provider. Nick: We're going to connect in and see if that password is matching. And only then do we use it. And as soon as we detect that it's matched, that password is removed. And then for the extract and replicats themselves, you can also use it for the database, data source, and data target connections, as well as for the GoldenGate users. So, it is a full-featured plugin. So, our identity provider plugin works with IAM as well as OAM. These are your standard identity manager authentication methods. The standard one is OAuth 2, as well as OIDC. And any Identity Manager that uses that is able to integrate with GoldenGate. 04:52 Lois: And how does this work? Nick: The way that it works is pretty straightforward. Once the user logs into the database, we're going to hand off authentication to the identity provider. Once the identity provider has validated that user's identity and their credentials, then it comes back to GoldenGate and says that user is able to log in to either GoldenGate or the application or the database. Once the user is logged in, we get that confirmation that's been sent out and they can continue working through GoldenGate. So, it's very straightforward on how it works. There's also a nice little UI that will help set up each additional user within those systems. All the communication is also secured as well. So any communication done through any of the GoldenGate services is encrypted using HTTPS. All the REST calls themselves are all done using HTTPS as well. All the data protection calls and all the communication across the network when we send data across a distribution service is encrypted using a secure WebSocket. And there's also trail file encryption at the operating system level for data at REST. So, this really gives you the full level of encryption for customers that need that high-end security. GoldenGate does have an option for FIPS 140-2 compliance as well. So that's even a further step for most of those customers. 06:12 Nikita: That's impressive! Because we want to maintain the highest security standards, right? Especially when dealing with sensitive information. I now want to move on to trail files. In our last episode, we briefly spoke about how they serve as logs that record and track changes made to data. But what more can you tell us about them, Nick? Nick: There's two different processes that write to the trail files. The extract process will write to the trail file and the receiver service will write to the trail file. The extract process is going to write to the trail file as it's pulling data out of that source database. Now, the extract process is controlled by a parameter file, that says, hey, here's the exact changes that I'm going to be pulling out. Here's the tables. Here's the rows that I want. As it's pulling that data out and writing it to the trail files, it's ensuring that those trail files have enough information so that the replicat process can actually construct a SQL statement and apply that change to that target platform. And so there's a lot of ways to change what's actually stored in those trail files and how it's handled. The trail files can also be used for initial loads. So when we do the initial load through GoldenGate, we can grab and write out the data for those tables, and that excludes the change data. So initial loads is pulling the data directly from the tables themselves, whereas ongoing replication is pulling it from the transaction logs. 07:38 Lois: But do we need to worry about rollbacks? Nick: Our trail files contain committed data only and all data is sequential. So this is two important things. Because it contains committed data only, we don't need to worry about rollbacks. We also don't need to worry about position within that trail file because we know all data is sequential. And so as we're reading through the trail file, we know that anything that's written in a prior location in that trial file was committed prior to something else. And as we get into the recovery aspects of GoldenGate, this will all make a lot more sense. 08:13 Lois: Before we do that, can you tell us about the naming of trail files? Nick: The trail files as far as naming, because these do reside on the operating system, you start with a two-letter trail file abbreviation and then a nine-digit sequential value. So, you almost look at it as like an archive log from Oracle, where we have a prefix and then an affix, which is numeric. Same kind of thing. So, we have our two-letter, in this case, an ab, and then we have a nine-digit number. 08:47 Transform the way you work with Oracle Database 23ai! This cutting-edge technology brings the power of AI directly to your data, making it easier to build powerful applications and manage critical workloads. Want to learn more about Database 23ai? Visit mylearn.oracle.com to pick from our range of courses and enroll today! 09:12 Nikita: Welcome back! Ok, Nick. Let's get into the GoldenGate recovery process. Nick: When we start looking at the GoldenGate recovery process, it essentially makes GoldenGate kind of point-in-time like. So on that source database, you have your extract process that's going to be capturing data from the transaction logs. In the case of Oracle, the Oracle Database is actually going to be reading those transaction logs from us and passing the change records directly to GoldenGate. We call them an LCR, Logical Change Record. And so the integrated extract and GoldenGate, the extract portion tells the database, hey, I'm now going to be interested in the following list of tables. And it gives a list of tables to that internal component, the log mining engine within the database. And it says, OK, I'm now pulling data for those tables and I'm going to send you those table changes. And so as the extract process gets sent those changes, it's going to have checkpoint information. So not only does it know where it was pulling data from out of that source database, but what it's also writing to the trail file. The trail files themselves are all sequential and they have only committed data, as we talked about earlier. The distribution service has checkpoint information that says, hey, I know where I'm reading from in the previous trail file, and I know what I've sent across the network. The receiver service is the same thing. It knows what it's receiving, as well as what it's written to the trail file and the target system. The replicat also has a checkpoint. It knows where it's reading from in the trail file, and then it knows what it's been applying into that target database.  This is where things start to become a little complicated. Our replicat process in most cases are parallel, so it'll have multiple threads applying data into that target database. Each of those threads is applying different transactions. And because of the way that the parallelism works in the replicat process, you can actually get situations where one replicat thread might be applying a transaction higher than another thread. And so you can eliminate that sequential or serial aspect of it, and we can get very high throughput speeds to the replicat. But it means that the checkpoint needs to be kind of smart enough to know how to rebuild itself if something fails. 11:32 Lois: Ok, sorry Nick, but can you go through that again? Maybe we can work backwards this time?  Nick: If the replicat process fails, when it comes back up, it's going to look to its checkpoint tables inside that target database. These checkpoint tables keep track of where each thread was at when it crashed. And so when the replicat process restarts, it goes, oh, I was applying these threads at this location in these SCNs. It'll then go and read from the trail file and say, hey, let me rebuild that data and it only applies transactions that it hasn't applied yet to that target system. There is a synchronized replicat command as well that will tell a crashed replicat to say, hey, bring all your threads up to the same high watermark. It does that process automatically as it restarts and continues normal replication. But there is an option to do it just by itself too. So that's how the replicat kind of repairs and recovers itself. It'll simply look at the trail files. Now, let's say that the replicat crashed, and it goes to read from the trail files when it restarts and that trail profile is missing. It'll actually communicate to the distribution, or excuse me, to the receiver service and say, hey, receiver service, I don't have this trail file. Can you bring it back for me? And the receiver service will communicate downstream and say, hey, distribution service, I need you to resend me trail find number 6. And so the distribution service will resend that trail file so that the replicat can reprocess it. So it's often nice to have redundant environments with GoldenGate so we can have those trail files kind of around for availability. 13:13 Nikita: What if one of these files gets corrupted? Nick: If one of those trail files is corrupt, let's say that a trail file on the target site became corrupt and the replicat can't read from it for one reason or another. Simply stop the replicat process, delete the corrupt trail file, restart the replicat process, and now it's going to rebuild that trail file from scratch based on the information from the source GoldenGate environment. And so it's very recoverable. Handles it all very well. 13:40 Nikita: And can the extract process bounce back in the same way? Nick: The extract process can also recover in a similar way. So if the extract process crashes, when it restarts itself, there's a number of things that it does. The first thing is it has to rebuild any open transactions. So it keeps all sorts of checkpoint information about the oldest transaction that it's keeping track of, any open transactions that haven't been committed, and any other transactions that have been committed that it's already written to the trail file. So as it's reprocessing that data, it knows exactly what it's committed to trail and what hasn't been committed. And there's a number of ways that it does this.  There's two main components here. One of them is called bounded recovery. Bounded recovery will allow you to set a time limit on transactions that span a certain length of time that they'll actually get flushed out to disk on that GoldenGate Hub. And that way it'll reduce the amount of time it takes GoldenGate to restart the extract process. And the other component is cache manager. Cache manager stores uncommitted transactions. And so it's a very elegant way of rebuilding itself from any kind of failure. You can also set up restart profiles so that if any process does crash, the GoldenGate service manager can automatically restart that service an x number of times across y time span. So if I say, hey, if my extract crashes, then attempt to restart it 100 times every 5 seconds. So there's a lot of things that you can do there to make it really nice and automatic repair itself and automatically resilient.  15:18 Lois: Well, that brings us to the end of this episode. Thank you, Nick, for going through the security strategies and recovery processes in such detail. Next week, we'll look at the installation of GoldenGate. Nikita: And if you want to learn more about the topics we discussed today, head over to mylearn.oracle.com and take a look at the Oracle GoldenGate 23ai Fundamentals course. Until next time, this is Nikita Abraham… Lois: And Lois Houston signing off! 15:44 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, Lois Houston and Nikita Abraham, along with Nick Wagner, focus on GoldenGate's terminology and architectural evolution.   Nick defines source and target systems, which are crucial for data replication, and then moves on to explain the data extraction and replication processes.   He also talks about the new microservices architecture, which replaces the classic architecture, offering benefits like simplified management, enhanced security, and a user-friendly interface. Nick highlights how this architecture facilitates easy upgrades and provides a streamlined experience for administrators.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and with me is Lois Houston: Director of Innovation Programs. Lois: Hi there! Thanks for joining us again as we make our way through Oracle GoldenGate 23ai. Last week, we discussed all the new features introduced in 23ai and today, we'll move on to the terminology, the different processes and what they do, and the architecture of the product at a high level. 00:56 Nikita: Back with us is Nick Wagner, Senior Director of Product Management for Oracle GoldenGate. Hi Nick! Let's get into some of the terminology. What do we actually call stuff in GoldenGate? Nick: Within GoldenGate, we have our source systems and our target systems. The source is where we're going to be capturing data from, the targets, where we're going to be applying data into. And when we start talking about things like active-active or setting up GoldenGate for high availability, where your source can also be your target, it does become a little bit more complex. And so in some of those cases, we might refer to things as East and West, or America and Europe, or different versions of that. We also have a couple of different things within the product itself. We have what we call our Extract and our Replicat. The Extract is going to be the process that pulls the data out of the database, our capture technology. Our Replicat's going to be the one that applies the data into the target system, or you can also look at it as a push technology. We have what we call our Distribution Path. Our Distribution Path is going to be how we're sending the data across the network. A lot of times when customers run GoldenGate, they don't have the luxury of just having a single server of GoldenGate that can pull data from one database and push data into another one. They need to set up multiple hops of that data. And so in that case, we would use what we call a Distribution Path to send that data from one system to the next. We also have what we call a Target Initiated Path. It's kind of a subset of your Distribution Path, but it allows you to communicate from a less secure environment into a more secure environment.  02:33 Lois: Nick, what about parameter names. I've seen them in uppercase…title case…does that matter?  Nick: GoldenGate has a lot of parameters. This is something you'll see all over the place within GoldenGate itself.  These parameters are in your Extract and Replicat parameter files during your distribution path parameter files. Parameters for GoldenGate are case insensitive.  Within your own environments, you can set it up to have lowercase, mixed case, whatever you want, but just be aware that they are case insensitive. GoldenGate doesn't care, it's just for readability. And then we also have something called trail files. Trail files is where GoldenGate stores all the data before we're able to apply it into that target system. Think about it as our queuing mechanism, and we're queuing everything outside the database so that we're not overloading those database environments. And that's some of the terminology for the product itself. We also have microservices within GoldenGate. 03:31 Nikita: And at the heart of everything is the Service Manager, right? Talk to us about what it is and what it does. Nick: The service manager is responsible for making sure that everything else is up and running. If you are familiar with GoldenGate classic architecture, this is kind of similar to a GoldenGate manager where that process was there to make sure that processes were running the trail files, or excuse me, that certain error logs were getting written out. If a process went down, the manager would restart that process. The service manager is performing a lot of those same functions. Now attached to the service manager, we have our configuration service. This is new in GoldenGate 23ai. This configuration service is going to allow you to set up GoldenGate for highly available environments. So you can build HA into GoldenGate itself using the configuration service. 04:22 Lois: And what does this configuration service do? Nick: This configuration service essentially moves the checkpoint files that used to be on disk into a database so that everything can be stored inside of a database.  Also attached to the service manager, we have the performance metric service. This is a service that is going to be gathering all the performance metrics of GoldenGate. So it's going to tell you how fast things are going, what the latencies are, how many bytes per second we're reading from, the transaction logs or writing to our trail files. How quickly a distribution path is sending data across a network. If you want to know any of your lag information, you'll get it from the performance metrics server. We also have the receiver service and the distribution service. These two work hand in hand to establish network communication between two GoldenGate environments. So on what we call our source system, we have a distribution service that's going to send the data to our target system. On the target system, a receiver service is going to receive that data and then rewrite the trail files. We also have the administration service that's responsible for authentication and authorization of the users, as well as making sure that people have access to the right information. 05:33 Nikita: Ok. Moving on the deployment, how is GoldenGate actually deployed, Nick? Nick: GoldenGate is kinda nice. So the way that the product is installed is you install the GoldenGate environment and that's what we call our service manager deployment under a specific GoldenGate home. So the software binaries themselves get installed under a home, we'll say U01/OGG23AI. Now once I've installed GoldenGate once, that's my OGG home. I can now have any number of service managers and deployments tied to that same home.  06:11 Lois: Ok, let's work with an example to make this simpler. Let's say I've got a service manager that's going be responsible for three different deployments: Accounting, Finance, and Sales. Nick: Each of these deployments is going to reside in its own directory. Each of these deployments is going to have its own set of microservices. And so this also means that each of these deployments can have their own set of users. So the people that access the GoldenGate accounting deployment can be different than the ones that access the sales deployment. This means with this distribution of roles that I can have somebody come in and administer the sales database, but they wouldn't have any information or any access to accounting or finance. And this is very important, it allows you to really pull that information apart and separate it. Each of these environments also has their own set of parameter files, Extract process, Replicat process, distribution services, and everything. So it's a very nice way of splitting things up, but all having them tied to the same GoldenGate home system. And this home is very important. So I can take a deployment, let's say my finance deployment, and if I want to move it to a new GoldenGate home and that GoldenGate home is a different version, like let's say that my original home is 23.4, my new GoldenGate home is 23.7, I simply stop that GoldenGate deployment. I stopped at a finance deployment. I changed its OGG home from 23.4 to 23.7. I restart the deployment, that deployment is automatically upgraded to the new environment and attached to the new system. So it makes upgrading very, very simple, very easy, very elegant. 07:53 Nikita: Ok. So, we've spoken about the services…some of the terminology. Let's get into the architecture next. Nick: So when we talk about the architecture for GoldenGate, we used to have two different architectures. We had a classic architecture and a microservices architecture. Classic architecture was something that's been around since the very beginning of GoldenGate in the late '90s. We announced that, that architecture was deprecated in 19c. And Oracle deprecated means that feature is no longer going to be enhanced and it'll be patched selectively. And at some point in the future, it'll be entirely desupported. Well, GoldenGate 23ai is that future. And so in 23ai, the classic architecture is desupported, that means that it's no longer in the build at all. And so it's just microservices architecture. 08:41 Lois: Is there a tool to assist with this migration?  Nick: We do have a migration utility that will convert an old classic architecture into the new microservices architecture. But there is quite a bit of learning curve to the new microservices architecture. So it's important that we go through how it works in the changes. 09:04 Are you looking to optimize your implementation strategies and improve efficiency? We have a solution for you! Our new Oracle Fusion Cloud Applications Foundations training and certification program. You'll learn to leverage Oracle Modern Best Practice (OMBP) to re-imagine business processes using advanced technologies in Oracle Fusion Cloud Applications such as AI, mobile, analytics, and more. Visit mylearn.oracle.com to get started today.  09:37 Nikita: Welcome back! Nick, what are the benefits of this microservices architecture?  Nick: It's got that simplified lifecycle for patching and upgrading. A lot of the GoldenGate patches that you get, especially these bundle patches, are complete installs as well. So you can go into My Oracle Support and download a complete install of a patch and that way, you don't have to use old patch to apply them. The only time you'll be using old patch is for one-off patches or smaller patches that need to be applied to your GoldenGate system. The microservices product has the same trusted Capture and Apply process that Classic did. There's almost no changes between the two except on how they communicate with their parent processes. And so the same logic that you use to pull data from Oracle or to apply data into Oracle is all the same. 10:25 Lois: And has the interface been upgraded as well? Nick: We've added a really nice, easy to use web interface for the microservices version of GoldenGate. Not only is this web interface work with all your standard browsers, but it's also mobile friendly too. So I can actually control and administer GoldenGate right through my mobile device. It also has new secure remote administration. This is something that the classic architecture was really missing. And so in the classic architecture, to use the command line interface, you had to log into the database server where GoldenGate was installed. Now, the command line interface, as well as the web interface and the REST API, all use remote administration and authentication. So that means that I can install the new command line interface or what we call admin client on my laptop locally and I can connect to any GoldenGate deployment as long as I have the username and password for that deployment. It's also more secure. GoldenGate microservices can also be deployed on premise or in OCI as a service and now also on these third-party clouds like Azure and Google Cloud. And it's also easier for developers to integrate in with the APIs themselves. Everything that GoldenGate does through the admin client as well as the web UI can all be traced. The REST API calls for GoldenGate are all fully published so you can get them right directly from the documentation, you can build your own web interface if you want to. So it makes it very easy. The REST APIs are also streamlined. With a single REST API call, I can do something like add an Extract process, create it, set up my parameter file, and set up the trail files all with a single API command. Whereas in the past, it would require multiple command line interface commands to do that same thing. So it's extremely elegant, very advanced. 12:16 Nikita: What does the microservices architecture look like? I know it's a bit complicated when we're not actually looking at a diagram of it, but just a high level, can you explain the different parts of it? Nick: It's pretty straightforward. But essentially what you've got on each system is a service manager. That service manager is then going to have a number of processes or services beneath it. It'll have the configuration service that stores the checkpoint information for GoldenGate. It'll have the administrative service for the authentication and users, the distribution service to send the data across a network, a receiver service to receive that information, performance metrics to get the performance statistics out of GoldenGate. And then of course, you also have your Extracts and Replicats that capture and apply technology. Each of those Extracts and Replicats will then connect to a database on the Extract side of things. That Extract is going to write to trail files. Those trail files are then going to be sent across the network where they're rebuilt on the target system and the Replicat's going to consume them and apply them into the target database. So the Replicat behaves almost like an end user. So it's taking that trail file data and simply converting it to DML operations, insert, update, delete, or a DDL operation in the case of Oracle, alter table, create table, et cetera, to go into that target database.   13:39 Lois: To look at a diagram of this architecture and learn about it in more detail, check out the Oracle GoldenGate 23ai Fundamentals course on mylearn.oracle.com. So, Nick, if I'm looking to deploy GoldenGate, what should I primarily keep in mind? Nick: So as you go to install GoldenGate and you look at a deployment, there's a couple of important environment variables that you want to make sure you're aware of. So one of the first ones is your OGG_Home. This environment variable is extremely important. This is the location of the GoldenGate software itself. And I want to stress how important it is to always use version numbers when you're setting up your GoldenGate home. When you go to install the software, if you're installing GoldenGate 23.5, use 23.5 within the home directory structure. If you're installing GoldenGate 23.7, use 23.7 inside that directory structure. 14:33 Nikita: Right… that way I'll always know which versions are which, and it'll make it really easy to upgrade and move from one version to the next. Ok, got it. What else, Nick? Nick: There's a couple other important directories. You have your OGG_ETC_HOME. This is where things like the configuration files are going to reside, parameter files, all your certificates for security, including the wallets where we store the credentials for not only the database accounts, but also for the GoldenGate user accounts as well. We have our GoldenGate variable home directory or VAR home. This is where all the GoldenGate log files are residing. And these are the log files that allow you to see what's going on in GoldenGate for auditing purposes. Anytime anybody makes a change to GoldenGate, you're going to see information go into the log files on what was happening and how it was working and what they did, what time they did, what command they issued. Another big important feature about these log files is it also gives you error information and troubleshooting details. So if you ever need to find out what happened in GoldenGate, what went wrong, you would look at these log files to find out that information. And then you also have your OGG_DATA_HOME. This is where those trail files are going to go. Essentially, this is kind of the queuing or overflow for GoldenGate. There's a couple of other additional components. We've got the admin client. This is our command line utility. If you don't want to use a web browser or prefer a command line utility, you can use the admin client. The admin client is also fully scriptable. So if you wanted to write scripts that would go off and automate things in GoldenGate, you can do that. A lot of customers did that with GGSCI in the classic architecture. You can do the same thing now with the admin client. The other component is the microservices security authentication and authorization services. These handle communication security, especially making sure that any passwords or usernames and everything like that is all encrypted. And instead of using an actual username and password, everything through the product is going to be done through an alias. And then it also handles all the authorization authentication, permissions, user accountability, and roles within GoldenGate. 16:39 Lois: Anything else you'd like to talk about before we wrap up for today, Nick? Nick: I also wanted to take a minute to talk about the REST API. All the microservices provide REST APIs to administer them and all of these are fully documented. They can be used by any client that can make REST API calls. So if you wanted to use Python, cURL, a web browser, you can do that as well. They're all just HTTP or HTTPS calls, get, put, patch, the standard REST API standards. And then GoldenGate does provide our admin client as well as a WebUI that use these REST APIs under the covers if you ever wanted to get a more advanced look at how it works. 17:18 Nikita: Well, that's all the time we have for today. Thanks for joining us, Nick.  Lois: Yes, thanks Nick. We look forward to having you back next week to talk with us about security strategies and data recovery. Nikita: And if you want to learn more about the topics we discussed today, head over to mylearn.oracle.com and take a look at the Oracle GoldenGate 23ai Fundamentals course. Until next time, this is Nikita Abraham… Lois: And Lois Houston, signing off!  17:43 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, Lois Houston and Nikita Abraham continue their deep dive into Oracle GoldenGate 23ai, focusing on its evolution and the extensive features it offers. They are joined once again by Nick Wagner, who provides valuable insights into the product's journey.   Nick talks about the various iterations of Oracle GoldenGate, highlighting the significant advancements from version 12c to the latest 23ai release. The discussion then shifts to the extensive new features in 23ai, including AI-related capabilities, UI enhancements, and database function integration.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   -----------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Hello and welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead: Editorial Services.  Nikita: Hi everyone! Last week, we introduced Oracle GoldenGate and its capabilities, and also spoke about GoldenGate 23ai. In today's episode, we'll talk about the various iterations of Oracle GoldenGate since its inception. And we'll also take a look at some new features and the Oracle GoldenGate product family. 00:57 Lois: And we have Nick Wagner back with us. Nick is a Senior Director of Product Management for GoldenGate at Oracle. Hi Nick! I think the last time we had an Oracle University course was when Oracle GoldenGate 12c was out. I'm sure there's been a lot of advancements since then. Can you walk us through those? Nick: GoldenGate 12.3 introduced the microservices architecture. GoldenGate 18c introduced support for Oracle Autonomous Data Warehouse and Autonomous Transaction Processing Databases. In GoldenGate 19c, we added the ability to do cross endian remote capture for Oracle, making it easier to set up the GoldenGate OCI service to capture from environments like Solaris, Spark, and HP-UX and replicate into the Cloud. Also, GoldenGate 19c introduced a simpler process for upgrades and installation of GoldenGate where we released something called a unified build. This means that when you install GoldenGate for a particular database, you don't need to worry about the database version when you install GoldenGate. Prior to this, you would have to install a version-specific and database-specific version of GoldenGate. So this really simplified that whole process. In GoldenGate 23ai, which is where we are now, this really is a huge release.  02:16 Nikita: Yeah, we covered some of the distributed AI features and high availability environments in our last episode. But can you give us an overview of everything that's in the 23ai release? I know there's a lot to get into but maybe you could highlight just the major ones? Nick: Within the AI and streaming environments, we've got interoperability for database vector types, heterogeneous capture and apply as well. Again, this is not just replication between Oracle-to-Oracle vector or Postgres to Postgres vector, it is heterogeneous just like the rest of GoldenGate. The entire UI has been redesigned and optimized for high speed. And so we have a lot of customers that have dozens and dozens of extracts and replicats and processes running and it was taking a long time for the UI to refresh those and to show what's going on within those systems. So the UI has been optimized to be able to handle those environments much better. We now have the ability to call database functions directly from call map. And so when you do transformation with GoldenGate, we have about 50 or 60 built-in transformation routines for string conversion, arithmetic operation, date manipulation. But we never had the ability to directly call a database function. 03:28 Lois: And now we do? Nick: So now you can actually call that database function, database stored procedure, database package, return a value and that can be used for transformation within GoldenGate. We have integration with identity providers, being able to use token-based authentication and integrate in with things like Azure Active Directory and your other single sign-on for the GoldenGate product itself. Within Oracle 23ai, there's a number of new features. One of those cool features is something called lock-free reservation columns. So this allows you to have a row, a single row within a table and you can identify a column within that row that's like an inventory column. And you can have multiple different users and multiple different transactions all updating that column within that same exact row at that same time. So you no longer have row-level locking for these reservation columns. And it allows you to do things like shopping carts very easily. If I have 500 widgets to sell, I'm going to let any number of transactions come in and subtract from that inventory column. And then once it gets below a certain point, then I'll start enforcing that row-level locking. 04:43 Lois: That's really cool… Nick: The one key thing that I wanted to mention here is that because of the way that the lock-free reservations work, you can have multiple transactions open on the same row. This is only supported for Oracle to Oracle. You need to have that same lock-free reservation data type and availability on that target system if GoldenGate is going to replicate into it. 05:05 Nikita: Are there any new features related to the diagnosability and observability of GoldenGate?  Nick: We've improved the AWR reports in Oracle 23ai. There's now seven sections that are specific to Oracle GoldenGate to allow you to really go in and see exactly what the GoldenGate processes are doing and how they're behaving inside the database itself. And there's a Replication Performance Advisor package inside that database, and that's been integrated into the Web UI as well. So now you can actually get information out of the replication advisor package in Oracle directly from the UI without having to log into the database and try to run any database procedures to get it. We've also added the ability to support a per-PDB Extract.  So in the past, when GoldenGate would run on a multitenant database, a multitenant database in Oracle, all the redo data from any pluggable database gets sent to that one redo stream. And so you would have to configure GoldenGate at the container or root level and it would be able to access anything at any PDB. Now, there's better security and better performance by doing what we call per-PDB Extract. And this means that for a single pluggable database, I can have an extract that runs at that database level that's going to capture information just from that pluggable database. 06:22 Lois And what about non-Oracle environments, Nick? Nick: We've also enhanced the non-Oracle environments as well. For example, in Postgres, we've added support for precise instantiation using Postgres snapshots. This eliminates the need to handle collisions when you're doing Postgres to Postgres replication and initial instantiation. On the GoldenGate for big data side, we've renamed that product more aptly to distributed applications in analytics, which is really what it does, and we've added a whole bunch of new features here too. The ability to move data into Databricks, doing Google Pub/Sub delivery. We now have support for XAG within the GoldenGate for distributed applications and analytics. What that means is that now you can follow all of our MAA best practices for GoldenGate for Oracle, but it also works for the DAA product as well, meaning that if it's running on one node of a cluster and that node fails, it'll restart itself on another node in the cluster. We've also added the ability to deliver data to Redis, Google BigQuery, stage and merge functionality for better performance into the BigQuery product. And then we've added a completely new feature, and this is something called streaming data and apps and we're calling it AsyncAPI and CloudEvent data streaming. It's a long name, but what that means is that we now have the ability to publish changes from a GoldenGate trail file out to end users. And so this allows through the Web UI or through the REST API, you can now come into GoldenGate and through the distributed applications and analytics product, actually set up a subscription to a GoldenGate trail file. And so this allows us to push data into messaging environments, or you can simply subscribe to changes and it doesn't have to be the whole trail file, it can just be a subset. You can specify exactly which tables and you can put filters on that. You can also set up your topologies as well. So, it's a really cool feature that we've added here. 08:26 Nikita: Ok, you've given us a lot of updates about what GoldenGate can support. But can we also get some specifics? Nick: So as far as what we have, on the Oracle Database side, there's a ton of different Oracle databases we support, including the Autonomous Databases and all the different flavors of them, your Oracle Database Appliance, your Base Database Service within OCI, your of course, Standard and Enterprise Edition, as well as all the different flavors of Exadata, are all supported with GoldenGate. This is all for capture and delivery. And this is all versions as well. GoldenGate supports Oracle 23ai and below. We also have a ton of non-Oracle databases in different Cloud stores. On an non-Oracle side, we support everything from application-specific databases like FairCom DB, all the way to more advanced applications like Snowflake, which there's a vast user base for that. We also support a lot of different cloud stores and these again, are non-Oracle, nonrelational systems, or they can be relational databases. We also support a lot of big data platforms and this is part of the distributed applications and analytics side of things where you have the ability to replicate to different Apache environments, different Cloudera environments. We also support a number of open-source systems, including things like Apache Cassandra, MySQL Community Edition, a lot of different Postgres open source databases along with MariaDB. And then we have a bunch of streaming event products, NoSQL data stores, and even Oracle applications that we support. So there's absolutely a ton of different environments that GoldenGate supports. There are additional Oracle databases that we support and this includes the Oracle Metadata Service, as well as Oracle MySQL, including MySQL HeatWave. Oracle also has Oracle NoSQL Spatial and Graph and times 10 products, which again are all supported by GoldenGate. 10:23 Lois: Wow, that's a lot of information! Nick: One of the things that we didn't really cover was the different SaaS applications, which we've got like Cerner, Fusion Cloud, Hospitality, Retail, MICROS, Oracle Transportation, JD Edwards, Siebel, and on and on and on.  And again, because of the nature of GoldenGate, it's heterogeneous. Any source can talk to any target. And so it doesn't have to be, oh, I'm pulling from Oracle Fusion Cloud, that means I have to go to an Oracle Database on the target, not necessarily.  10:51 Lois: So, there's really a massive amount of flexibility built into the system.  11:00 Unlock the power of AI Vector Search with our new course and certification. Get more accurate search results, handle complex datasets easily, and supercharge your data-driven decisions. From now through May 15, 2025, we are waiving the certification exam fee (valued at $245). Visit mylearn.oracle.com to enroll. 11:26 Nikita: Welcome back! Now that we've gone through the base product, what other features or products are in the GoldenGate family itself, Nick? Nick: So we have quite a few. We've kind of touched already on GoldenGate for Oracle databases and non-Oracle databases. We also have something called GoldenGate for Mainframe, which right now is covered under the GoldenGate for non-Oracle, but there is a licensing difference there. So that's something to be aware of. We also have the OCI GoldenGate product. We are announcing and we have announced that OCI GoldenGate will also be made available as part of the Oracle Database@Azure and Oracle Database@ Google Cloud partnerships.  And then you'll be able to use that vendor's cloud credits to actually pay for the OCI GoldenGate product. One of the cool things about this is it will have full feature parity with OCI GoldenGate running in OCI. So all the same features, all the same sources and targets, all the same topologies be able to migrate data in and out of those clouds at will, just like you do with OCI GoldenGate today running in OCI.  We have Oracle GoldenGate Free.  This is a completely free edition of GoldenGate to use. It is limited on the number of platforms that it supports as far as sources and targets and the size of the database.  12:45 Lois: But it's a great way for developers to really experience GoldenGate without worrying about a license, right? What's next, Nick? Nick: We have GoldenGate for Distributed Applications and Analytics, which was formerly called GoldenGate for big data, and that allows us to do all the streaming. That's also where the GoldenGate AsyncAPI integration is done. So in order to publish the GoldenGate trail files or allow people to subscribe to them, it would be covered under the Oracle GoldenGate Distributed Applications and Analytics license. We also have OCI GoldenGate Marketplace, which allows you to run essentially the on-premises version of GoldenGate but within OCI. So a little bit more flexibility there. It also has a hub architecture. So if you need that 99.99% availability, you can get it within the OCI Marketplace environment. We have GoldenGate for Oracle Enterprise Manager Cloud Control, which used to be called Oracle Enterprise Manager. And this allows you to use Enterprise Manager Cloud Control to get all the statistics and details about GoldenGate. So all the reporting information, all the analytics, all the statistics, how fast GoldenGate is replicating, what's the lag, what's the performance of each of the processes, how much data am I sending across a network. All that's available within the plug-in. We also have Oracle GoldenGate Veridata. This is a nice utility and tool that allows you to compare two databases, whether or not GoldenGate is running between them and actually tell you, hey, these two systems are out of sync. And if they are out of sync, it actually allows you to repair the data too. 14:25 Nikita: That's really valuable…. Nick: And it does this comparison without locking the source or the target tables. The other really cool thing about Veridata is it does this while there's data in flight. So let's say that the GoldenGate lag is 15 or 20 seconds and I want to compare this table that has 10 million rows in it. The Veridata product will go out, run its comparison once. Once that comparison is done the first time, it's then going to have a list of rows that are potentially out of sync. Well, some of those rows could have been moved over or could have been modified during that 10 to 15 second window. And so the next time you run Veridata, it's actually going to go through. It's going to check just those rows that were potentially out of sync to see if they're really out of sync or not. And if it comes back and says, hey, out of those potential rows, there's two out of sync, it'll actually produce a script that allows you to resynchronize those systems and repair them. So it's a very cool product.  15:19 Nikita: What about GoldenGate Stream Analytics? I know you mentioned it in the last episode, but in the context of this discussion, can you tell us a little more about it?  Nick: This is the ability to essentially stream data from a GoldenGate trail file, and they do a real time analytics on it. And also things like geofencing or real-time series analysis of it.  15:40 Lois: Could you give us an example of this? Nick: If I'm working in tracking stock market information and stocks, it's not really that important on how much or how far down a stock goes. What's really important is how quickly did that stock rise or how quickly did that stock fall. And that's something that GoldenGate Stream Analytics product can do. Another thing that it's very valuable for is the geofencing. I can have an application on my phone and I can track where the user is based on that application and all that information goes into a database. I can then use the geofencing tool to say that, hey, if one of those users on that app gets within a certain distance of one of my brick-and-mortar stores, I can actually send them a push notification to say, hey, come on in and you can order your favorite drink just by clicking Yes, and we'll have it ready for you. And so there's a lot of things that you can do there to help upsell your customers and to get more revenue just through GoldenGate itself. And then we also have a GoldenGate Migration Utility, which allows customers to migrate from the classic architecture into the microservices architecture. 16:44 Nikita: Thanks Nick for that comprehensive overview.  Lois: In our next episode, we'll have Nick back with us to talk about commonly used terminology and the GoldenGate architecture. And if you want to learn more about what we discussed today, visit mylearn.oracle.com and take a look at the Oracle GoldenGate 23ai Fundamentals course. Until next time, this is Lois Houston… Nikita: And Nikita Abraham, signing off! 17:10 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In a new season of the Oracle University Podcast, Lois Houston and Nikita Abraham dive into the world of Oracle GoldenGate 23ai, a cutting-edge software solution for data management. They are joined by Nick Wagner, a seasoned expert in database replication, who provides a comprehensive overview of this powerful tool.   Nick highlights GoldenGate's ability to ensure continuous operations by efficiently moving data between databases and platforms with minimal overhead. He emphasizes its role in enabling real-time analytics, enhancing data security, and reducing costs by offloading data to low-cost hardware. The discussion also covers GoldenGate's role in facilitating data sharing, improving operational efficiency, and reducing downtime during outages.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode. ---------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead: Editorial Services with Oracle University, and with me is Lois Houston: Director of Innovation Programs. Lois: Hi everyone! Welcome to a new season of the podcast. This time, we're focusing on the fundamentals of Oracle GoldenGate. Oracle GoldenGate helps organizations manage and synchronize their data across diverse systems and databases in real time.  And with the new Oracle GoldenGate 23ai release, we'll uncover the latest innovations and features that empower businesses to make the most of their data. Nikita: Taking us through this is Nick Wagner, Senior Director of Product Management for Oracle GoldenGate. He's been doing database replication for about 25 years and has been focused on GoldenGate on and off for about 20 of those years.  01:18 Lois: In today's episode, we'll ask Nick to give us a general overview of the product, along with some use cases and benefits. Hi Nick! To start with, why do customers need GoldenGate? Nick: Well, it delivers continuous operations, being able to continuously move data from one database to another database or data platform in efficiently and a high-speed manner, and it does this with very low overhead. Almost all the GoldenGate environments use transaction logs to pull the data out of the system, so we're not creating any additional triggers or very little overhead on that source system. GoldenGate can also enable real-time analytics, being able to pull data from all these different databases and move them into your analytics system in real time can improve the value that those analytics systems provide. Being able to do real-time statistics and analysis of that data within those high-performance custom environments is really important. 02:13 Nikita: Does it offer any benefits in terms of cost?  Nick: GoldenGate can also lower IT costs. A lot of times people run these massive OLTP databases, and they are running reporting in those same systems. With GoldenGate, you can offload some of the data or all the data to a low-cost commodity hardware where you can then run the reports on that other system. So, this way, you can get back that performance on the OLTP system, while at the same time optimizing your reporting environment for those long running reports. You can improve efficiencies and reduce risks. Being able to reduce the amount of downtime during planned and unplanned outages can really make a big benefit to the overall operational efficiencies of your company.  02:54 Nikita: What about when it comes to data sharing and data security? Nick: You can also reduce barriers to data sharing. Being able to pull subsets of data, or just specific pieces of data out of a production database and move it to the team or to the group that needs that information in real time is very important. And it also protects the security of your data by only moving in the information that they need and not the entire database. It also provides extensibility and flexibility, being able to support multiple different replication topologies and architectures. 03:24 Lois: Can you tell us about some of the use cases of GoldenGate? Where does GoldenGate truly shine?  Nick: Some of the more traditional use cases of GoldenGate include use within the multicloud fabric. Within a multicloud fabric, this essentially means that GoldenGate can replicate data between on-premise environments, within cloud environments, or hybrid, cloud to on-premise, on-premise to cloud, or even within multiple clouds. So, you can move data from AWS to Azure to OCI. You can also move between the systems themselves, so you don't have to use the same database in all the different clouds. For example, if you wanted to move data from AWS Postgres into Oracle running in OCI, you can do that using Oracle GoldenGate. We also support maximum availability architectures. And so, there's a lot of different use cases here, but primarily geared around reducing your recovery point objective and recovery time objective. 04:20 Lois: Ah, reducing RPO and RTO. That must have a significant advantage for the customer, right? Nick: So, reducing your RPO and RTO allows you to take advantage of some of the benefits of GoldenGate, being able to do active-active replication, being able to set up GoldenGate for high availability, real-time failover, and it can augment your active Data Guard and Data Guard configuration. So, a lot of times GoldenGate is used within Oracle's maximum availability architecture platinum tier level of replication, which means that at that point you've got lots of different capabilities within the Oracle Database itself. But to help eke out that last little bit of high availability, you want to set up an active-active environment with GoldenGate to really get true zero RPO and RTO. GoldenGate can also be used for data offloading and data hubs. Being able to pull data from one or more source systems and move it into a data hub, or into a data warehouse for your operational reporting. This could also be your analytics environment too. 05:22 Nikita: Does GoldenGate support online migrations? Nick: In fact, a lot of companies actually get started in GoldenGate by doing a migration from one platform to another. Now, these don't even have to be something as complex as going from one database like a DB2 on-premise into an Oracle on OCI, it could even be simple migrations. A lot of times doing something like a major application or a major database version upgrade is going to take downtime on that production system. You can use GoldenGate to eliminate that downtime. So this could be going from Oracle 19c to Oracle 23ai, or going from application version 1.0 to application version 2.0, because GoldenGate can do the transformation between the different application schemas. You can use GoldenGate to migrate your database from on premise into the cloud with no downtime as well. We also support real-time analytic feeds, being able to go from multiple databases, not only those on premise, but being able to pull information from different SaaS applications inside of OCI and move it to your different analytic systems. And then, of course, we also have the ability to stream events and analytics within GoldenGate itself.  06:34 Lois: Let's move on to the various topologies supported by GoldenGate. I know GoldenGate supports many different platforms and can be used with just about any database. Nick: This first layer of topologies is what we usually consider relational database topologies. And so this would be moving data from Oracle to Oracle, Postgres to Oracle, Sybase to SQL Server, a lot of different types of databases. So the first architecture would be unidirectional. This is replicating from one source to one target. You can do this for reporting. If I wanted to offload some reports into another server, I can go ahead and do that using GoldenGate. I can replicate the entire database or just a subset of tables. I can also set up GoldenGate for bidirectional, and this is what I want to set up GoldenGate for something like high availability. So in the event that one of the servers crashes, I can almost immediately reconnect my users to the other system. And that almost immediately depends on the amount of latency that GoldenGate has at that time. So a typical latency is anywhere from 3 to 6 seconds. So after that primary system fails, I can reconnect my users to the other system in 3 to 6 seconds. And I can do that because as GoldenGate's applying data into that target database, that target system is already open for read and write activity. GoldenGate is just another user connecting in issuing DML operations, and so it makes that failover time very low. 07:59 Nikita: Ok…If you can get it down to 3 to 6 seconds, can you bring it down to zero? Like zero failover time?   Nick: That's the next topology, which is active-active. And in this scenario, all servers are read/write all at the same time and all available for user activity. And you can do multiple topologies with this as well. You can do a mesh architecture, which is where every server talks to every other server. This works really well for 2, 3, 4, maybe even 5 environments, but when you get beyond that, having every server communicate with every other server can get a little complex. And so at that point we start looking at doing what we call a hub and spoke architecture, where we have lots of different spokes. At the end of each spoke is a read/write database, and then those communicate with a hub. So any change that happens on one spoke gets sent into the hub, and then from the hub it gets sent out to all the other spokes. And through that architecture, it allows you to really scale up your environments. We have customers that are doing up to 150 spokes within that hub architecture. Within active-active replication as well, we can do conflict detection and resolution, which means that if two users modify the same row on two different systems, GoldenGate can actually determine that there was an issue with that and determine what user wins or which row change wins, which is extremely important when doing active-active replication. And this means that if one of those systems fails, there is no downtime when you switch your users to another active system because it's already available for activity and ready to go. 09:35 Lois: Wow, that's fantastic. Ok, tell us more about the topologies. Nick: GoldenGate can do other things like broadcast, sending data from one system to multiple systems, or many to one as far as consolidation. We can also do cascading replication, so when data moves from one environment that GoldenGate is replicating into another environment that GoldenGate is replicating. By default, we ignore all of our own transactions. But there's actually a toggle switch that you can flip that says, hey, GoldenGate, even though you wrote that data into that database, still push it on to the next system. And then of course, we can also do distribution of data, and this is more like moving data from a relational database into something like a Kafka topic or a JMS queue or into some messaging service. 10:24 Raise your game with the Oracle Cloud Applications skills challenge. Get free training on Oracle Fusion Cloud Applications, Oracle Modern Best Practice, and Oracle Cloud Success Navigator. Pass the free Oracle Fusion Cloud Foundations Associate exam to earn a Foundations Associate certification. Plus, there's a chance to win awards and prizes throughout the challenge! What are you waiting for? Join the challenge today by visiting visit oracle.com/education. 10:58 Nikita: Welcome back! Nick, does GoldenGate also have nonrelational capabilities?  Nick: We have a number of nonrelational replication events in topologies as well. This includes things like data lake ingestion and streaming ingestion, being able to move data and data objects from these different relational database platforms into data lakes and into these streaming systems where you can run analytics on them and run reports. We can also do cloud ingestion, being able to move data from these databases into different cloud environments. And this is not only just moving it into relational databases with those clouds, but also their data lakes and data fabrics. 11:38 Lois: You mentioned a messaging service earlier. Can you tell us more about that? Nick: Messaging replication is also possible. So we can actually capture from things like messaging systems like Kafka Connect and JMS, replicate that into a relational data, or simply stream it into another environment. We also support NoSQL replication, being able to capture from MongoDB and replicate it onto another MongoDB for high availability or disaster recovery, or simply into any other system. 12:06 Nikita: I see. And is there any integration with a customer's SaaS applications? Nick: GoldenGate also supports a number of different OCI SaaS applications. And so a lot of these different applications like Oracle Financials Fusion, Oracle Transportation Management, they all have GoldenGate built under the covers and can be enabled with a flag that you can actually have that data sent out to your other GoldenGate environment. So you can actually subscribe to changes that are happening in these other systems with very little overhead. And then of course, we have event processing and analytics, and this is the final topology or flexibility within GoldenGate itself. And this is being able to push data through data pipelines, doing data transformations. GoldenGate is not an ETL tool, but it can do row-level transformation and row-level filtering.  12:55 Lois: Are there integrations offered by Oracle GoldenGate in automation and artificial intelligence? Nick: We can do time series analysis and geofencing using the GoldenGate Stream Analytics product. It allows you to actually do real time analysis and time series analysis on data as it flows through the GoldenGate trails. And then that same product, the GoldenGate Stream Analytics, can then take the data and move it to predictive analytics, where you can run MML on it, or ONNX or other Spark-type technologies and do real-time analysis and AI on that information as it's flowing through.  13:29 Nikita: So, GoldenGate is extremely flexible. And given Oracle's focus on integrating AI into its product portfolio, what about GoldenGate? Does it offer any AI-related features, especially since the product name has “23ai” in it? Nick: With the advent of Oracle GoldenGate 23ai, it's one of the two products at this point that has the AI moniker at Oracle. Oracle Database 23ai also has it, and that means that we actually do stuff with AI. So the Oracle GoldenGate product can actually capture vectors from databases like MySQL HeatWave, Postgres using pgvector, which includes things like AlloyDB, Amazon RDS Postgres, Aurora Postgres. We can also replicate data into Elasticsearch and OpenSearch, or if the data is using vectors within OCI or the Oracle Database itself. So GoldenGate can be used for a number of things here. The first one is being able to migrate vectors into the Oracle Database. So if you're using something like Postgres, MySQL, and you want to migrate the vector information into the Oracle Database, you can. Now one thing to keep in mind here is a vector is oftentimes like a GPS coordinate. So if I need to know the GPS coordinates of Austin, Texas, I can put in a latitude and longitude and it will give me the GPS coordinates of a building within that city. But if I also need to know the altitude of that same building, well, that's going to be a different algorithm. And GoldenGate and replicating vectors is the same way. When you create a vector, it's essentially just creating a bunch of numbers under the screen, kind of like those same GPS coordinates. The dimension and the algorithm that you use to generate that vector can be different across different databases, but the actual meaning of that data will change. And so GoldenGate can replicate the vector data as long as the algorithm and the dimensions are the same. If the algorithm and the dimensions are not the same between the source and the target, then you'll actually want GoldenGate to replicate the base data that created that vector. And then once GoldenGate replicates the base data, it'll actually call the vector embedding technology to re-embed that data and produce that numerical formatting for you.  15:42 Lois: So, there are some nuances there… Nick: GoldenGate can also replicate and consolidate vector changes or even do the embedding API calls itself. This is really nice because it means that we can take changes from multiple systems and consolidate them into a single one. We can also do the reverse of that too. A lot of customers are still trying to find out which algorithms work best for them. How many dimensions? What's the optimal use? Well, you can now run those in different servers without impacting your actual AI system. Once you've identified which algorithm and dimension is going to be best for your data, you can then have GoldenGate replicate that into your production system and we'll start using that instead. So it's a nice way to switch algorithms without taking extensive downtime. 16:29 Nikita: What about in multicloud environments?  Nick: GoldenGate can also do multicloud and N-way active-active Oracle replication between vectors. So if there's vectors in Oracle databases, in multiple clouds, or multiple on-premise databases, GoldenGate can synchronize them all up. And of course we can also stream changes from vector information, including text as well into different search engines. And that's where the integration with Elasticsearch and OpenSearch comes in. And then we can use things like NVIDIA and Cohere to actually do the AI on that data.  17:01 Lois: Using GoldenGate with AI in the database unlocks so many possibilities. Thanks for that detailed introduction to Oracle GoldenGate 23ai and its capabilities, Nick.  Nikita: We've run out of time for today, but Nick will be back next week to talk about how GoldenGate has evolved over time and its latest features. And if you liked what you heard today, head over to mylearn.oracle.com and take a look at the Oracle GoldenGate 23ai Fundamentals course to learn more. Until next time, this is Nikita Abraham… Lois: And Lois Houston, signing off! 17:33 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Discover how Oracle APEX leverages OCI AI services to build smarter, more efficient applications. Hosts Lois Houston and Nikita Abraham interview APEX experts Chaitanya Koratamaddi, Apoorva Srinivas, and Toufiq Mohammed about how key services like OCI Vision, Oracle Digital Assistant, and Document Understanding integrate with Oracle APEX.   Packed with real-world examples, this episode highlights all the ways you can enhance your APEX apps.   Oracle APEX: Empowering Low Code Apps with AI: https://mylearn.oracle.com/ou/course/oracle-apex-empowering-low-code-apps-with-ai/146047/ Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Hello and welcome to the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead: Editorial Services. Nikita: Hi everyone! Last week, we looked at how generative AI powers Oracle APEX and in today's episode, we're going to focus on integrating APEX with OCI AI Services. Lois: That's right, Niki. We're going to look at how you can use Oracle AI services like OCI Vision, Oracle Digital Assistant, Document Understanding, OCI Generative AI, and more to enhance your APEX apps. 01:03 Nikita: And to help us with it all, we've got three amazing experts with us, Chaitanya Koratamaddi, Director of Product Management at Oracle, and senior product managers, Apoorva Srinivas and Toufiq Mohammed. In today's episode, we'll go through each Oracle AI service and look at how it interacts with APEX. Apoorva, let's start with you. Can you explain what the OCI Vision service is? Apoorva: Oracle Cloud Infrastructure Vision is a serverless multi-tenant service accessible using the console or REST APIs. You can upload images to detect and classify objects in them. With prebuilt models available, developers can quickly build image recognition into their applications without machine learning expertise. OCI Vision service provides a fully managed model infrastructure. With complete integration with OCI Data Labeling, you can build custom models easily. OCI Vision service provides pretrained models-- Image Classification, Object Detection, Face Detection, and Text Recognition. You can build custom models for Image Classification and Object Detection. 02:24 Lois: Ok. What about its use cases? How can OCI Vision make APEX apps more powerful? Apoorva: Using OCI Vision, you can make images and videos discoverable and searchable in your APEX app.  You can use OCI Vision to detect and classify objects in the images. OCI Vision also highlights the objects using a red rectangular box. This comes in handy in use cases such as detecting vehicles that have violated the rules in traffic images. You can use OCI Vision to identify visual anomalies in your data. This is a very popular use case where you can detect anomalies in cancer X-ray images to detect cancer. These are some of the most popular use cases of using OCI Vision with your APEX app. But the possibilities are endless and you can use OCI Vision for any of your image analysis. 03:29 Nikita: Let's shift gears to Oracle Digital Assistant. Chaitanya, can you tell us what it's all about? Chaitanya: Oracle Digital Assistant is a low-code conversational AI platform that allows businesses to build and deploy AI assistants. It provides natural language understanding, automatic speech recognition, and text-to-speech capabilities to enable human-like interactions with customers and employees. Oracle Digital Assistant comes with prebuilt templates for you to get started.  04:00 Lois: What are its key features and benefits, Chaitanya? How does it enhance the user experience? Chaitanya: Oracle Digital Assistant provides conversational AI capabilities that include generative AI features, natural language understanding and ML, AI-powered voice, and analytics and insights. Integration with enterprise applications become easier with unified conversational experience, prebuilt chatbots for Oracle Cloud applications, and chatbot architecture frameworks. Oracle Digital Assistant provides advanced conversational design tools, conversational designer, dialogue and domain trainer, and native multilingual support. Oracle Digital Assistant is open, scalable, and secure. It provides multi-channel support, automated bot-to-agent transfer, and integrated authentication profile. 04:56 Nikita: And what about the architecture? What happens at the back end? Chaitanya: Developers assemble digital assistants from one or more skills. Skills can be based on prebuilt skills provided by Oracle or third parties, custom developed, or based on one of the many skill templates available. 05:16 Lois: Chaitanya, what exactly are “skills” within the Oracle Digital Assistant framework?  Chaitanya: Skills are individual chatbots that are designed to interact with users and fulfill specific type of tasks. Each skill helps a user complete a task through a combination of text messages and simple UI elements like select list. When a user request is submitted through a channel, the Digital Assistant routes the user's request to the most appropriate skill to satisfy the user's request. Skills can combine multilingual NLP deep learning engine, a powerful dialogflow engine, and integration components to connect to back-end systems.  Skills provide a modular way to build your chatbot functionality. Now users connect with a chatbot through channels such as Facebook, Microsoft Teams, or in our case, Oracle APEX chatbot, which is embedded into an APEX application. 06:21 Nikita: That's fascinating. So, what are some use cases of Oracle Digital Assistant in APEX apps? Chaitanya: Digital assistants streamline approval processes by collecting information, routing requests, and providing status updates. Digital assistants offer instant access to information and documentation, answering common questions and guiding users. Digital assistants assist sales teams by automating tasks, responding to inquiries, and guiding prospects through the sales funnel. Digital assistants facilitate procurement by managing orders, tracking deliveries, and handling supplier communication. Digital assistants simplify expense approvals by collecting reports, validating receipts, and routing them for managerial approval. Digital assistants manage inventory by tracking stock levels, reordering supplies, and providing real-time inventory updates. Digital assistants have become a common UX feature in any enterprise application. 07:28 Want to learn how to design stunning, responsive enterprise applications directly from your browser with minimal coding? The new Oracle APEX Developer Professional learning path and certification enables you to leverage AI-assisted development, including generative AI and Database 23ai, to build secure, scalable web and mobile applications with advanced AI-powered features. From now through May 15, 2025, we're waiving the certification exam fee (valued at $245). So, what are you waiting for? Visit mylearn.oracle.com to get started today. 08:09 Nikita: Welcome back! Thanks for that, Chaitanya. Toufiq, let's talk about the OCI Document Understanding service. What is it? Toufiq: Using this service, you can upload documents to extract text, tables, and other key data. This means the service can automatically identify and extract relevant information from various types of documents, such as invoices, receipts, contracts, etc. The service is serverless and multitenant, which means you don't need to manage any servers or infrastructure. You can access this service using the console, REST APIs, SDK, or CLI, giving you multiple ways to integrate. 08:55 Nikita: What do we use for APEX apps?  Toufiq: For APEX applications, we will be using REST APIs to integrate the service. Additionally, you can process individual files or batches of documents using the ProcessorJob API endpoint. This flexibility allows you to handle different volumes of documents efficiently, whether you need to process a single document or thousands at once. With these capabilities, the OCI Document Understanding service can significantly streamline your document processing tasks, saving time and reducing the potential for manual errors. 09:36 Lois: Ok.  What are the different types of models available? How do they cater to various business needs? Toufiq: Let us start with pre-trained models. These are ready-to-use models that come right out of the box, offering a range of functionalities. The available models are Optical Character Recognition (OCR) enables the service to extract text from documents, allowing you to digitize, scan the documents effortlessly. You can precisely extract text content from documents. Key-value extraction, useful in streamlining tasks like invoice processing. Table extraction can intelligently extract tabular data from documents. Document classification automatically categorizes documents based on their content. OCR PDF enables seamless extraction of text from PDF files. Now, what if your business needs go beyond these pre-trained models. That's where custom models come into play. You have the flexibility to train and build your own models on top of these foundational pre-trained models. Models available for training are key value extraction and document classification. 10:50 Nikita: What does the architecture look like for OCI Document Understanding? Toufiq: You can ingest or supply the input file in two different ways. You can upload the file to an OCI Object Storage location. And in your request, you can point the Document Understanding service to pick the file from this Object Storage location.  Alternatively, you can upload a file directly from your computer. Once the file is uploaded, the Document Understanding service can process the file and extract key information using the pre-trained models. You can also customize models to tailor the extraction to your data or use case. After processing the file, the Document Understanding service stores the results in JSON format in the Object Storage output bucket. Your Oracle APEX application can then read the JSON file from the Object Storage output location, parse the JSON, and store useful information at local table or display it on the screen to the end user. 11:52 Lois: And what about use cases? How are various industries using this service? Toufiq: In financial services, you can utilize Document Understanding to extract data from financial statements, classify and categorize transactions, identify and extract payment details, streamline tax document management. Under manufacturing, you can perform text extraction from shipping labels and bill of lading documents, extract data from production reports, identify and extract vendor details. In the healthcare industry, you can automatically process medical claims, extract patient information from forms, classify and categorize medical records, identify and extract diagnostic codes. This is not an exhaustive list, but provides insights into some industry-specific use cases for Document Understanding. 12:50 Nikita: Toufiq, let's switch to the big topic everyone's excited about—the OCI Generative AI Service. What exactly is it? Toufiq: OCI Generative AI is a fully managed service that provides a set of state of the art, customizable large language models that cover a wide range of use cases. It provides enterprise grade generative AI with data governance and security, which means only you have access to your data and custom-trained models. OCI Generative AI provides pre-trained out-of-the-box LLMs for text generation, summarization, and text embedding. OCI Generative AI also provides necessary tools and infrastructure to define models with your own business knowledge. 13:37 Lois: Generally speaking, how is OCI Generative AI useful?  Toufiq: It supports various large language models. New models available from Meta and Cohere include Llama2 developed by Meta, and Cohere's Command model, their flagship text generation model. Additionally, Cohere offers the Summarize model, which provides high-quality summaries, accurately capturing essential information from documents, and the Embed model, converting text to vector embeddings representation. OCI Generative AI also offers dedicated AI clusters, enabling you to host foundational models on private GPUs. It integrates LangChain and open-source framework for developing new interfaces for generative AI applications powered by language models. Moreover, OCI Generative AI facilitates generative AI operations, providing content moderation controls, zero downtime endpoint model swaps, and endpoint deactivation and activation capabilities. For each model endpoint, OCI Generative AI captures a series of analytics, including call statistics, tokens processed, and error counts. 14:58 Nikita: What about the architecture? How does it handle user input? Toufiq: Users can input natural language, input/output examples, and instructions. The LLM analyzes the text and can generate, summarize, transform, extract information, or classify text according to the user's request. The response is sent back to the user in the specified format, which can include raw text or formatting like bullets and numbering, etc. 15:30 Lois: Can you share some practical use cases for generative AI in APEX apps?  Toufiq: Some of the OCI generative AI use cases for your Oracle APEX apps include text summarization. Generative AI can quickly summarize lengthy documents such as articles, transcripts, doctor's notes, and internal documents. Businesses can utilize generative AI to draft marketing copy, emails, blog posts, and product descriptions efficiently. Generative AI-powered chatbots are capable of brainstorming, problem solving, and answering questions. With generative AI, content can be rewritten in different styles or languages. This is particularly useful for localization efforts and catering to diverse audience. Generative AI can classify intent in customer chat logs, support tickets, and more. This helps businesses understand customer needs better and provide tailored responses and solutions. By searching call transcripts, internal knowledge sources, Generative AI enables businesses to efficiently answer user queries. This enhances information retrieval and decision-making processes. 16:47 Lois: Before we let you go, can you explain what Select AI is? How is it different from the other AI services? Toufiq: Select AI is a feature of Autonomous Database. This is where Select AI differs from the other AI services. Be it OCI Vision, Document Understanding, or OCI Generative AI, these are all freely managed standalone services on Oracle Cloud, accessible via REST APIs. Whereas Select AI is a feature available in Autonomous Database. That means to use Select AI, you need Autonomous Database.  17:26 Nikita: And what can developers do with Select AI? Toufiq: Traditionally, SQL is the language used to query the data in the database. With Select AI, you can talk to the database and get insights from the data in the database using human language. At the very basic, what Select AI does is it generates SQL queries using natural language, like an NL2SQL capability.  17:52 Nikita: How does it actually do that? Toufiq: When a user asks a question, the first step Select AI does is look into the AI profile, which you, as a developer, define. The AI profile holds crucial information, such as table names, the LLM provider, and the credentials needed to authenticate with the LLM service. Next, Select AI constructs a prompt. This prompt includes information from the AI profile and the user's question.  Essentially, it's a packet of information containing everything the LLM service needs to generate SQL. The next step is generating SQL using LLM. The prompt prepared by Select AI is sent to the available LLM services via REST. Which LLM to use is configured in the AI profile. The supported providers are OpenAI, Cohere, Azure OpenAI, and OCI Generative AI. Once the SQL is generated by the LLM service, it is returned to the application. The app can then handle the SQL query in various ways, such as displaying the SQL results in a report format or as charts, etc.  19:05 Lois: This has been an incredible discussion! Thank you, Chaitanya, Apoorva, and Toufiq, for walking us through all of these amazing AI tools. If you're ready to dive deeper, visit mylearn.oracle.com and search for the Oracle APEX: Empowering Low Code Apps with AI course. You'll find step-by-step guides and demos for everything we covered today.  Nikita: Until next week, this is Nikita Abraham… Lois: And Lois Houston signing off! 19:31 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.  

Get ready to explore how generative AI is transforming development in Oracle APEX. In this episode, hosts Lois Houston and Nikita Abraham are joined by Oracle APEX experts Apoorva Srinivas and Toufiq Mohammed to break down the innovative features of APEX 24.1. Learn how developers can use APEX Assistant to build apps, generate SQL, and create data models using natural language prompts.   Oracle APEX: Empowering Low Code Apps with AI: https://mylearn.oracle.com/ou/course/oracle-apex-empowering-low-code-apps-with-ai/146047/ Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Welcome back to another episode of the Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and I'm joined by Lois Houston, Director of Innovation Programs. Lois: Hi everyone! In our last episode, we spoke about Oracle APEX and AI. We covered the data and AI -centric challenges businesses are up against and explored how AI fits in with Oracle APEX. Niki, what's in store for today? Nikita: Well, Lois, today we're diving into how generative AI powers Oracle APEX. With APEX 24.1, developers can use the Create Application Wizard to tell APEX what kind of application they want to build based on available tables. Plus, APEX Assistant helps create, refine, and debug SQL code in natural language. 01:16 Lois: Right. Today's episode will focus on how generative AI enhances development in APEX. We'll explore its architecture, the different AI providers, and key use cases. Joining us are two senior product managers from Oracle—Apoorva Srinivas and Toufiq Mohammed. Thank you both for joining us today. We'll start with you, Apoorva. Can you tell us a bit about the generative AI service in Oracle APEX? Apoorva: It is nothing but an abstraction to the popular commercial Generative AI products, like OCI Generative AI, OpenAI, and Cohere. APEX makes use of the existing REST infrastructure to authenticate using the web credentials with Generative AI Services. Once you configure the Generative AI Service, it can be used by the App Builder, AI Assistant, and AI Dynamic Actions, like Show AI Assistant and Generate Text with AI, and also the APEX_AI PL/SQL API. You can enable or disable the Generative AI Service on the APEX instance level and on the workspace level. 02:31 Nikita: Ok. Got it. So, Apoorva, which AI providers can be configured in the APEX Gen AI service? Apoorva: First is the popular OpenAI. If you have registered and subscribed for an OpenAI API key, you can just enter the API key in your APEX workspace to configure the Generative AI service. APEX makes use of the chat completions endpoint in OpenAI. Second is the OCI Generative AI Service. Once you have configured an OCI API key on Oracle Cloud, you can make use of the chat models. The chat models are available from Cohere family and Meta Llama family.  The third is the Cohere. The configuration of Cohere is similar to OpenAI. You need to have your Cohere OpenAI key. And it provides a similar chat functionality using the chat endpoint.  03:29 Lois: What is the purpose of the APEX_AI PL/SQL public API that we now have? How is it used within the APEX ecosystem? Apoorva: It models the chat operation of the popular Generative AI REST Services. This is the same package used internally by the chat widget of the APEX Assistant. There are more procedures around consent management, which you can configure using this package. 03:58 Lois: Apoorva, at a high level, how does generative AI fit into the APEX environment? Apoorva: APEX makes use of the existing REST infrastructure—that is the web credentials and remote server—to configure the Generative AI Service. The inferencing is done by the backend Generative AI Service. For the Generative AI use case in APEX, such as NL2SQL and creation of an app, APEX performs the prompt enrichment.  04:29 Nikita: And what exactly is prompt enrichment? Apoorva: Let's say you provide a prompt saying "show me the average salary of employees in each department." APEX will take this prompt and enrich it by adding in more details. It elaborates on the prompt by mentioning the requirements, such as Oracle SQL syntax statement, and providing some metadata from the data dictionary of APEX. Once the prompt enrichment is complete, it is then passed on to the LLM inferencing service. Therefore, the SQL query provided by the AI Assistant is more accurate and in context.  05:15 Unlock the power of AI Vector Search with our new course and certification. Get more accurate search results, handle complex datasets easily, and supercharge your data-driven decisions. From now to May 15, 2025, we are waiving the certification exam fee (valued at $245). Visit mylearn.oracle.com to enroll. 05:41  Nikita: Welcome back! Let's talk use cases. Apoorva, can you share some ways developers can use generative AI with APEX? Apoorva: SQL is an integral part of building APEX apps. You use SQL everywhere. You can make use of the NL2SQL feature in the code editor by using the APEX Assistant to generate SQL queries while building the apps. The second is the prompt-based app creation. With APEX Assistant, you can now generate fully functional APEX apps by providing prompts in natural language. Third is the AI Assistant, which is a chat widget provided by APEX in all the code editors and for creation of apps. You can chat with the AI Assistant by providing your prompts and get responses from the Generative AI Services. 06:37 Lois: Without getting too technical, can you tell us how to create a data model using AI? Apoorva: A SQL Workshop utility called Create Data Model Using AI uses AI to help you create your own data model. The APEX Assistant generates a script to create tables, triggers, and constraints in either Oracle SQL or Quick SQL format. You can also insert sample data into these tables. But before you use this feature, you must create a generative AI service and enable the Used by App Builder setting. If you are using the Oracle SQL format, when you click on Create SQL Script, APEX generates the script and brings you to this script editor page. Whereas if you are using the Quick SQL format, when you click on Review Quick SQL, APEX generates the Quick SQL code and brings you to the Quick SQL page. 07:39 Lois: And to see a detailed demo of creating a custom data model with the APEX Assistant, visit mylearn.oracle.com and search for the "Oracle APEX: Empowering Low Code Apps with AI" course. Apoorva, what about creating an APEX app from a prompt. What's that process like? Apoorva: APEX 24.1 introduces a new feature where you can generate an application blueprint based on a prompt using natural language. The APEX Assistant leverages the APEX Dictionary Cache to identify relevant tables while suggesting the pages to be created for your application. You can iterate over the application design by providing further prompts using natural language and then generating an application based on your needs. Once you are satisfied, you can click on Create Application, which takes you to the Create Application Wizard in APEX, where you can further customize your application, such as application icon and other features, and finally, go ahead to create your application. 08:53 Nikita: Again, you can watch a demo of this on MyLearn. So, check that out if you want to dive deeper.  Lois: That's right, Niki. Thank you for these great insights, Apoorva! Now, let's turn to Toufiq. Toufiq, can you tell us more about the APEX Assistant feature in Oracle APEX. What is it and how does it work? Toufiq: APEX Assistant is available in Code Editors in the APEX App Builder. It leverages generative AI services as the backend to answer your questions asked in natural language. APEX Assistant makes use of the APEX dictionary cache to identify relevant tables while generating SQL queries. Using the Query Builder mode enables Assistant. You can generate SQL queries from natural language for Form, Report, and other region types which support SQL queries. Using the general assistance mode, you can generate PL/SQL JavaScript, HTML, or CSS Code, and seek further assistance from generative AI. For example, you can ask the APEX Assistant to optimize the code, format the code for better readability, add comments, etc. APEX Assistant also comes with two quick actions, Improve and Explain, which can help users improve and understand the selected code. 10:17 Nikita: What about the Show AI Assistant dynamic action? I know that it provides an AI chat interface, but can you tell us a little more about it?  Toufiq: It is a native dynamic action in Oracle APEX which renders an AI chat user interface. It leverages the generative AI services that are configured under Workspace utilities. This AI chat user interface can be rendered inline or as a dialog. This dynamic action also has configurable system prompt and welcome message attributes.  10:52 Lois: Are there attributes you can configure to leverage even more customization?  Toufiq: The first attribute is the initial prompt. The initial prompt represents a message as if it were coming from the user. This can either be a specific item value or a value derived from a JavaScript expression. The next attribute is use response. This attribute determines how the AI Assistant should return responses. The term response refers to the message content of an individual chat message. You have the option to capture this response directly into a page item, or to process it based on more complex logic using JavaScript code. The final attribute is quick actions. A quick action is a predefined phrase that, once clicked, will be sent as a user message. Quick actions defined here show up as chips in the AI chat interface, which a user can click to send the message to Generative AI service without having to manually type in the message. 12:05 Lois: Thank you, Toufiq and Apoorva, for joining us today. Like we were saying, there's a lot more you can find in the “Oracle APEX: Empowering Low Code Apps with AI” course on MyLearn. So, make sure you go check that out. Nikita: Join us next week for a discussion on how to integrate APEX with OCI AI Services. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 12:28 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.  

Lois Houston and Nikita Abraham kick off a new season of the podcast, exploring how Oracle APEX integrates with AI to build smarter low-code applications. They are joined by Chaitanya Koratamaddi, Director of Product Management at Oracle, who explains the basics of Oracle APEX, its global adoption, and the challenges it addresses for businesses managing and integrating data. They also explore real-world use cases of AI within the Oracle APEX ecosystem   Oracle APEX: Empowering Low Code Apps with AI: https://mylearn.oracle.com/ou/course/oracle-apex-empowering-low-code-apps-with-ai/146047/ Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   -----------------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Hello and welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead: Editorial Services. Nikita: Hi everyone! Thank you for joining us as we begin a new season of the podcast, this time focused on Oracle APEX and how it integrates with AI to help you create powerful applications. This season is for everyone—from beginners and SQL developers to DBA data scientists and low-code enthusiasts. So, if you're interested in using Oracle APEX to build low-code applications that have custom generative AI features, you'll want to stay tuned in. 01:07 Lois: That's right, Niki. Today, we're going to discuss Oracle APEX at a high level, starting with what it is. Then, we'll cover a few business challenges related to data and AI innovation that organizations face, and learn how the powerful combination of APEX and AI can help overcome these challenges.  01:27 Nikita: To take us through it all, we've got Chaitanya Koratamaddi with us. Chaitanya is Director of Product Management for Oracle APEX. Hi Chaitanya! For anyone new to Oracle APEX, can you explain what it is and why it's so widely used? Chaitanya: Oracle APEX is the world's most popular enterprise low code application platform. APEX enables you to build secure and scalable enterprise-scale applications with world class features that can be deployed anywhere, cloud or on-premises. And with APEX, you can build applications 20 times faster with 100 times less code. APEX delivers the most productive way to develop and deploy mobile and web applications everywhere. 02:18 Lois: That's impressive. So, what's the adoption rate like for Oracle APEX? Chaitanya: As of today, there are 19 million plus APEX applications created globally. 5,000 plus APEX applications are created on a daily basis and there are 800,000 plus APEX developers worldwide. 60,000 plus customers in 150 countries across various industry verticals. And 75% of Fortune 500 companies use Oracle APEX. 02:56 Nikita: Wow, the numbers really speak for themselves, right? But Chaitanya, why are organizations adopting Oracle APEX at this scale? Or to put it differently, what's the core business challenge that Oracle APEX is addressing? Chaitanya: From databases to all data, you know that the world is more connected and automated than ever. To drive new business value, organizations need to explore and exploit new sources of data that are generated from this connected world. That can be sounds, feeds, sensors, videos, images, and more. Businesses need to be able to work with all types of data and also make sure that it is available to be used together. Typically, businesses need to work on all data at a massive scale. For example, supply chains are no longer dependent just on inventory, demand, and order management signals. A manufacturer should be able to understand data describing global weather patterns and how it impacts their supply chains. Businesses need to pull in data from as many social sources as possible to understand how customer sentiment impacts product sales and corporate brands. Our customers need a data platform that ensures all this data works together seamlessly and easily. 04:38 Lois: So, you're saying Oracle APEX is the platform that helps businesses manage and integrate data seamlessly. But data is just one part of the equation, right? Then there's AI. How are the two related?  Chaitanya: Before we start talking about Oracle AI, let's first talk about what customers are looking for and where they are struggling within their AI innovation. It all starts with data. For decades, working with data has largely involved dealing with structured data, whether it is your customer records in your CRM application and orders from your ERP database. Data was organized into database and tables, and when you needed to find some insights in your data, all you need to do is just use stored procedures and SQL queries to deliver the answers. But today, the expectations are higher. You want to use AI to construct sophisticated predictions, find anomalies, make decisions, and even take actions autonomously. And the data is far more complicated. It is in an endless variety of formats scattered all over your business. You need tools to find this data, consume it, and easily make sense of it all. And now capabilities like natural language processing, computer vision, and anomaly detection are becoming very essential just like how SQL queries used to be. You need to use AI to analyze phone call transcripts, support tickets, or email complaints so you can understand what customers need and how they feel about your products, customer service, and brand. You may want to use a data source as noisy and unstructured as social media data to detect trends and identify issues in real time.  Today, AI capabilities are very essential to accelerate innovation, assess what's happening in your business, and most importantly, exceed the expectations of your customers. So, connecting your application, data, and infrastructure allows everyone in your business to benefit from data. 07:32 Raise your game with the Oracle Cloud Applications skills challenge. Get free training on Oracle Fusion Cloud Applications, Oracle Modern Best Practice, and Oracle Cloud Success Navigator. Pass the free Oracle Fusion Cloud Foundations Associate exam to earn a Foundations Associate certification. Plus, there's a chance to win awards and prizes throughout the challenge! What are you waiting for? Join the challenge today by visiting oracle.com/education. 08:06 Nikita: Welcome back! So, let's focus on AI across the Oracle Cloud ecosystem. How does Oracle bring AI into the mix to connect applications, data, and infrastructure for businesses? Chaitanya: By embedding AI throughout the entire technology stack from the infrastructure that businesses run on through the applications for every line of business, from finance to supply chain and HR, Oracle is helping organizations pragmatically use AI to improve performance while saving time, energy, and resources.  Our core cloud infrastructure includes a unique AI infrastructure layer based on our supercluster technology, leveraging the latest and greatest hardware and uniquely able to get the maximum out of the AI infrastructure technology for scenarios such as large language processing. Then there is generative AI and ML for data platforms. On top of the AI infrastructure, our database layer embeds AI in our products such as autonomous database. With autonomous database, you can leverage large language models to use natural language queries rather than writing a SQL when interacting with the autonomous database. This enables you to achieve faster adoption in your application development. Businesses and their customers can use the Select AI natural language interface combined with Oracle Database AI Vector Search to obtain quicker, more intuitive insights into their own data. Then we have AI services. AI services are a collection of offerings, including generative AI with pre-built machine learning models that make it easier for developers to apply AI to applications and business operations. The models can be custom-trained for more accurate business results. 10:17 Nikita: And what specific AI services do we have at Oracle, Chaitanya?  Chaitanya: We have Oracle Digital Assistant Speech, Language, Vision, and Document Understanding. Then we have Oracle AI for Applications. Oracle delivers AI built for business, helping you make better decisions faster and empowering your workforce to work more effectively. By embedding classic and generative AI into its applications, Fusion Apps customers can instantly access AI outcomes wherever they are needed without leaving the software environment they use every day to power their business. 11:02 Lois: Let's talk specifically about APEX. How does APEX use the Gen AI and machine learning models in the stack to empower developers. How does it help them boost productivity? Chaitanya: Starting APEX 24.1, you can choose your preferred large language models and leverage native generative AI capabilities of APEX for AI assistants, prompt-based application creation, and more. Using native OCI capabilities, you can leverage native platform capabilities from OCI, like AI infrastructure and object storage, etc. Oracle APEX running on autonomous infrastructure in Oracle Cloud leverages its unique native generative AI capabilities tuned specifically on your data. These language models are schema aware, data aware, and take into account the shape of information, enabling your applications to take advantage of large language models pre-trained on your unique data. You can give your users greater insights by leveraging native capabilities, including vector-based similarity search, content summary, and predictions. You can also incorporate powerful AI features to deliver personalized experiences and recommendations, process natural language prompts, and more by integrating directly with a suite of OCI AI services. 12:38 Nikita: Can you give us some examples of this? Chaitanya: You can leverage OCI Vision to interpret visual and text inputs, including image recognition and classification. Or you can use OCI Speech to transcribe and understand spoken language, making both image and audio content accessible and actionable. You can work with disparate data sources like JSON, spatial, graphs, vectors, and build AI capabilities around your own business data. So, low-code application development with APEX along with AI is a very powerful combination. 13:22 Nikita: What are some use cases of AI-powered Oracle APEX applications?  Chaitanya: You can build APEX applications to include conversational chatbots. Your APEX applications can include image and object detection capability. Your APEX applications can include speech transcription capability. And in your applications, you can include code generation that is natural language to SQL conversion capability. Your applications can be powered by semantic search capability. Your APEX applications can include text generation capability. 14:00 Lois: So, there's really a lot we can do! Thank you, Chaitanya, for joining us today. With that, we're wrapping up this episode. We covered Oracle APEX, the key challenges businesses face when it comes to AI innovation, and how APEX and AI work together to give businesses an AI edge.  Nikita: Yeah, and if you want to know more about Oracle APEX, visit mylearn.oracle.com and search for the Oracle APEX: Empowering Low Code Apps with AI course. Join us next week for a discussion on AI-assisted development in Oracle APEX. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 14:39 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this special episode of the Oracle University Podcast, Bill Lawson and Nikita Abraham chat with Peter Fernandez, Senior Director of Cloud Certification at Oracle University, about the exciting new Raise Your Game challenge. They discuss how the initiative is designed to enhance participants' skills in Oracle Fusion Cloud Applications and Oracle Cloud Success Navigator. They also cover key details about the challenge, such as how to get started, who can participate, the way it is structured, and the prizes up for grabs.   Raise Your Game: https://education.oracle.com/raise-your-game-saas Oracle University Learning Community:  https://education.oracle.com/ou-community LinkedIn:  https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, and the OU Studio Team for helping us create this episode.   ------------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Bill: Welcome to the Oracle University Podcast. I'm Bill Lawson, Senior Director of Cloud Applications Product Management with Oracle University, and with me is Nikita Abraham, Team Lead of Editorial Services. Nikita: Hi everyone! Last week, we concluded our three-part series on multicloud, and today, we're shifting gears and exploring an exciting new challenge that's been thrown down by Oracle University. To tell us all about it, we have Peter Fernandez joining us. Peter is Senior Director of Cloud Certification at Oracle University. Hi Peter! We're thrilled to have you with us today! Peter: Hi Niki, hi Bill! I'm delighted to be here. 01:02 Bill: So, Peter, let's get straight into it. What's this new challenge all about?  Peter: The challenge, which we're calling Raise Your Game, is an incredible opportunity for anyone looking to gain knowledge and gain professional skills about Oracle's Fusion Cloud Applications. We launched a skills challenge on Feb 14, and it will continue until May 15, 2025. This challenge encourages you to build expertise in two key areas: Oracle Fusion Cloud Applications and Oracle Cloud Success Navigator.  This training is geared towards anyone who could be a student in higher ed or someone pursuing a business degree, and Oracle customers and partners who are new to Oracle's Applications or experienced consultants implementing business applications.  01:55 Nikita: And how exactly does the challenge help in building this expertise? Peter: The challenge has two levels. In Level 1, you'll need to complete an Oracle Fusion Cloud Apps Foundations course and pass the corresponding exam. These courses are designed to deepen your understanding of the technology enablers in Oracle's Fusion Cloud Applications and learn about Oracle's Modern Best Practice, or OMBP. These are extremely helpful throughout all phases in the journey when implementing and using Oracle Fusion Cloud Applications. The Foundation training itself covers a wide range of topics, including core OMBP processes, key performance metrics, implementation considerations, and technology enablers like AI, ML, mobile, and analytics.  02:49 Bill: Before we move on, Peter, can you tell us more about Oracle Modern Best Practice? We discussed it a few weeks back, but for anyone who missed that episode, it'll be nice to get a quick refresher. Peter: Sure, Bill. Implementing Oracle Fusion Cloud Applications successfully is more than just technology—it's about following best practices that drive efficiency and success and tie back to business requirements.  Oracle Modern Best Practice represent years of accumulated experience, industry insights, and proven methodologies. It serves as a guiding framework for implementing efficient business processes within Oracle Fusion Cloud Applications. These best practices map the features and innovations within Oracle applications to the processes that customers perform every day, and that is key.  These curated, industry-leading practices detail how the features that we have built using the most modern technologies can be leveraged to optimize operations.  Having a solid grasp of an OMBP and its associated technology enablers will empower you to ensure smoother business operations and higher customer satisfaction. It will show you how to automate activities, streamline tasks, improve results, and set your team up for continued success.  The goal of these courses is to make it easy for implementers, global process owners, IT teams to identify every opportunity to improve an organization's business processes with Oracle Fusion Cloud Applications.  04:33 Bill: So, getting back to Level 1, what do I earn when I complete it? Peter: When you complete this level, you'll earn a Level 1 Oracle University Learning Community badge. This recognizes that you have foundational knowledge in your chosen Fusion application. 04:48 Bill: That sounds exciting. And then there's a Level 2? Peter: There is also a Level 2 and where things get even more exciting. You're going to take your knowledge to the next level by completing the Oracle Cloud Success Navigator Essentials course and passing the associated assessment.  This level in the challenge focuses on particularly applying the knowledge you gained in Level 1 where you'll explore the Oracle Cloud Success Navigator's features and functionality, and get the skills you need to lead organizations through their Oracle Fusion Cloud Applications implementation journey.  05:25 Nikita: And when I complete Level 2, I earn another badge? Peter: That's right, Niki. When you successfully complete Level 2, you'll earn a special Level 2 Oracle University Learning Community badge.  The goal of Raise Your Game is to reach the Summit by completing both Level 1 and Level 2 challenges with the fastest time and the highest pass scores. Both these combined determine your position on the leaderboard and your position in the Top 500, which will be awarded separate prizes at the end of the challenge.  05:59 Nikita: So, when you're done, you'll have both theoretical and practical knowledge. And I understand that there are some fantastic prizes up for grabs? Peter: Absolutely, Niki.  This not only helps with both theoretical but also practical knowledge. Learners also have a chance to be featured on the leaderboard in the Oracle University Learning Community. The leaderboard showcases the people who have achieved Level 1 and Level 2 with the fastest times and the highest scores. Along with the badges I told you about, at the end of the promotion, the top 500 people who complete both Level 1 and Level 2 with the fastest time and highest pass scores will receive an Oracle-branded cap, an Oracle Success Navigator pin, and a special Oracle University Community Success Navigator digital badge.  06:52 Bill: So, Peter, who can participate in this challenge, and are there any prerequisites? Peter: The challenge is open to anyone interested in expanding their knowledge of Oracle Cloud Applications. And while there are no strict prerequisites, a basic understanding of business concepts and some familiarity with Oracle Cloud Applications is recommended.  This will ensure that you're able to make the most of the learning materials and engage with the content effectively. You can always check the program overview on the website if you have more questions about this challenge. We've got an FAQ posted there that should answer most anything you are curious about. 07:31 Bill: That's good to know, Peter. And the fact that I get started no matter my level of experience is great news, too. Peter: Absolutely, Bill. Even if you are a beginner fresh out of college, or a seasoned pro like I mentioned earlier who has been implementing Oracle Fusion Cloud Applications (or other applications) for years, I would recommend the challenge and training to you.  Basically, this training is for everyone. This program provides foundational knowledge to improve the implementation approach using Oracle Modern Best Practices. Even those individuals that are certified in Cloud Applications will benefit from learning how these modern best practices fit into their work.  08:12 Nikita: Ok, Peter, I'm ready to do it. How do I get started with the challenge? Peter: That's great. The first step, of course, is to register. And you can do this by visiting oracle.com/education. That's Oracle's main site. oracle.com/education. And select the first tile that you'll see on the webpage, which is the Raise Your Game challenge. If you don't already have an Oracle MyLearn account, you'll need to create one and you'll be prompted to create one. This account gives you access to the Oracle MyLearning platform. Once you're registered, you'll have access to a curated list of learning paths and corresponding certifications. It's important that you review the official rules and promotion details before proceeding with the challenge.  09:12 Unlock the power of AI Vector Search with our new course and certification. Get more accurate search results, handle complex datasets easily, and supercharge your data-driven decisions. From now to May 15, 2025, we are waiving the certification exam fee (valued at $245). Visit mylearn.oracle.com to enroll. 09:40 Nikita: Welcome back! Ok, Peter, I've registered. What's next? Peter: After you're done registering, you need to select the Cloud Application course that aligns with your interests and goals. We have courses on four different areas: Human Capital Management, Enterprise Resource Planning, Supply Chain Management, and Customer Experience. Once you complete the training and certification, you're done with Level 1.  For Level 2, we have the Oracle Cloud Success Navigator Essentials course and assessment that you will need to complete. You can check your status on the leaderboard in the Oracle University Learning Community and share your progress on social media. Like I was saying, the time taken to complete each of these levels and the higher scores earned determines the Top 500 winners.  10:36 Bill: And the best part of this challenge is that it's completely free, right? Peter: Absolutely. There is no cost associated with participating in the skills challenge. It is completely free for anyone, anywhere in the world to participate as long as they comply with the official rules of the promotion. You can take any or all of the Foundation Associate certification exams at no cost. With multiple free attempts, there is no time limit for completing the exams, but to be eligible for the prizes, you must complete the exams and assessments by May 15, 2025. That's midnight GMT. 11:16 Nikita: What if someone doesn't pass the certification exam on their first attempt? Peter: If someone does not pass the certification exam on their first attempt, we understand that not everyone does. We've made provisions for that. If you don't pass the foundations associate certification exam, you have the option to retake the exam many times over. 11:37 Nikita: Now, Peter, let's say someone has already registered for the Fusion Cloud Applications Foundations Associate certification exam before joining the skills challenge. Will their exam be considered for the prizes? Peter: Well, that's a great question, Niki. If someone has already registered for the exam before joining the challenge, their exam will be considered for the prizes as long as they first join the skills challenge. This ensures that everyone who engages with the challenge has a fair chance to win. 12:06 Bill: Does course content consumed before the start of the challenge count towards the awards and badges? Peter: Unfortunately no, Bill. Any content consumed or purchased before Feb 14, 2025, that's again 12 AM GMT, does not apply retroactively to awards or prizes in the Raise Your Game challenge. We want everyone to start on an equal footing here. 12:29 Nikita: What about certifications earned before the challenge began? Peter: Again, certifications earned before Feb 14, 2025, again 12 AM GMT, do not qualify for the promotion. That ensures again that the challenge is fair for all participants. 12:48 Nikita: Now, Peter, how many free exam attempts do participants get as part of the challenge? Peter: Since all the Oracle Fusion Cloud Applications Foundations Associate Certification exams are free, there is no limit to the number of attempts. Participants can take these exams as many times as they need to. 13:05 Bill: And, Peter, say I want to take more than one of the Foundations courses and exams. Can I do that? Peter: Absolutely. This is a great way for someone to learn about the different areas of business that they may be familiar with. As I mentioned earlier, the Oracle Fusion Cloud Applications Foundations training is a program to provide you with knowledge of OMBPs, Oracle Modern Best Practices, that is, and Fusion Cloud Applications. So, it's a great opportunity to cross-skill. You can earn all four certifications if you choose. 13:38 Bill: Peter, thank you so much for joining us today and telling us all about this challenge. It is a really fantastic opportunity for everyone, whether you're new to Fusion Cloud Applications or an experienced implementation professional, to boost your Oracle Cloud Apps expertise. We're really excited to try it out ourselves! Peter: A sincere thank you to you, Bill and Niki. It's been an absolute pleasure. I'd really encourage everyone to jump on this challenge. It's a great way to enhance your learning journey and have some fun along the way. Nikita: I couldn't agree more! Thanks Peter. That's a wrap on this episode. Join us next week for another episode of the Oracle University Podcast. Until then, this is Nikita Abraham… Bill: And Bill Lawson, signing off! 14:19 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

The final episode of the multicloud series focuses on Oracle Database@Azure, a powerful cloud database solution. Hosts Lois Houston and Nikita Abraham, along with Senior Manager of CSS OU Cloud Delivery Samvit Mishra, discuss how this service allows customers to run Oracle databases within the Microsoft Azure data center, simplifying deployment and management. The discussion also highlights the benefits of native integration with Azure services, eliminating the need for complex networking setups.   Oracle Cloud Infrastructure Multicloud Architect Professional: https://mylearn.oracle.com/ou/course/oracle-cloud-infrastructure-multicloud-architect-professional-2025-/144474 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Hello and welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead: Editorial Services. Nikita: Hi everyone! For the last two weeks, we've been talking about different aspects of multicloud. In the final episode of this three-part series, Samvit Mishra, Senior Manager of CSS OU Cloud Delivery, joins us once again to tell us about the Oracle Database@Azure service. Hi Samvit! Thanks for being here today. Samvit: Hi Niki! Hi Lois! Happy to be back. 01:01 Lois: In our last episode, we spoke about the strategic partnership between Oracle and Microsoft, and specifically discussed the Oracle Interconnect for Azure.  Nikita: Yeah, and Oracle Database@Azure is yet another addition to this partnership. What can you tell us about this service, Samvit? Samvit: The Oracle Database@Azure service, which was made generally available in 2023, runs right inside the Microsoft Azure data center and uses Azure networking. The entire Oracle Cloud Database Service infrastructure resides in the Azure data center, while it is managed by an expert Oracle Cloud Infrastructure operations team.  It provides customers simple and secure access to Oracle Cloud database services within their chosen Azure deployment region, without getting into the complexity of managing networking between the cloud vendors. It is natively integrated with various Microsoft Azure services. This provides a seamless user experience when configuring and using the different Azure services with OCI Oracle database, since much of the complexity associated with the configuration is greatly simplified.  There is no need to set up a private interconnect between Microsoft Azure and OCI because the service itself resides within the Azure data center and uses the Azure network. This is very beneficial in terms of strategic deployment because customers can experience microseconds network latency between the endpoints, while receiving a high-performance database environment.  02:42 Nikita: How do I get started with the Oracle Database@Azure service? Samvit: You begin by purchasing the subscription from Oracle and setting up your billing account. Then you provision the database, resources, and service.  With that you are ready to configure your application to connect to the database and work on the remaining deployment. As you continue using the service, you can monitor the different resource metrics using the Azure monitoring services and analyze those logs using Azure Log Analytics.  03:15 Lois: So, the adoption is pretty easy, then. What about the responsibilities? Who is responsible for what? Samvit: The Oracle Cloud operations team is entirely responsible for managing the Exadata Database Infrastructure and the VM cluster resources that are provisioned in the Microsoft Azure data center.  Oracle is responsible for maintaining the service software and infrastructure by applying updates as they are released. Any issues arising from the OCI Database Service and the resources will be addressed by Oracle Support. You have to raise a support ticket for them to investigate and provide a resolution.  And as Azure customers, you have to do rightsizing, based on your workload needs, and provision the Exadata Database Infrastructure and VM cluster in the OCI pod within the Azure data center. You have to provision the database in Exadata Database Service, apply the database and system updates, and take advantage of the cloud automation to maintain and manage the database.  You have to load data, establish the connectivity, and support development on your database. As a customer, you monitor the database and infrastructure metrics and events, and also analyze those logs using the Microsoft Azure-provided native services.  04:42 Nikita: Samvit, what sort of challenges were being faced by customers that necessitated the creation of the Oracle Database@Azure service?  Samvit: A common deployment scenario in customer environments was that a lot of critical applications, which could be packaged applications, in-house applications, or customized third-party applications, used Oracle Database as their primary database solution.  These Oracle databases were deployed in Exadata Infrastructure on-premises or even in Enterprise Server hardware. Some customers evaluated and migrated many of their packaged and other applications to Microsoft Azure compute. Since Oracle Exadata was not supported in Azure, they had to configure a hybrid deployment in order to use Oracle databases that reside in the Exadata infrastructure on-premises.  They needed to configure a dedicated and secure network between the Azure data center and their on-premises data center. This added complexity, incurred high costs, had a latency effect, and was even unreliable. There were also cases where customers migrated Oracle databases on Enterprise Server on-premises to Oracle databases hosted on Azure compute.  This did not boost efficiency to a large scale. And those were the only options available when provisioning Oracle Database in Azure because Exadata was not available earlier in Azure. 06:18 Lois: And how has that been resolved now? Samvit: With the Oracle Database@Azure service, customer requirements have been aptly met by allowing them to host their Oracle databases on Exadata infrastructure, right next to their application in the Azure data center.  Customers, while migrating their applications to Azure compute, can also migrate their Oracle databases on-premises on Exadata infrastructure directly to Exadata Database Service in Azure. And Oracle databases that are on Enterprise Server on-premises can be consolidated directly into Exadata Database Service in Azure, providing them the benefits of scalability, security, performance, and availability, all that are inherent property of OCI Oracle Exadata Database Service.  Customers can see growth in the operational efficiency, saving on the overall cost.  07:17 Nikita: Can you take us through the process of deployment?  Samvit: It's quite simple, actually. First, you deploy the Exadata Database Service that is plugged into Azure VNET. Next, you provision the required number of databases, which might be migrated as is or with a consolidated exercise.  You can use any of the Oracle database tools or utilities to do the migration or even use the Oracle Zero Downtime Migration method to automate the entire Oracle database migration. Finally, migrate your enterprise application into the Azure environment.  Establish the required network configuration to allow communication between the migrated applications and Oracle databases.  And then you are all set to publish your application that is running entirely in Azure. You can leverage other Azure services, like monitoring, log analytics, Power BI, or DevOps tools, to enhance existing or even build and deploy newer enterprise applications that are powered by OCI Oracle Database Service in the back end.  08:25 Lois: What about multi-cloud deployment scenarios where applications reside in Azure, but the Oracle databases are deployed on third-party cloud providers, either as a native solution or in computes? Samvit: These Oracle databases can be migrated to Exadata Database Service in the Oracle Database@Azure service.  There is no need for the complex cross-cloud connectivity setup between the vendors. And at the same time, you experience the lowest latency between the application and the database deployment. 09:05 Want to learn how to design stunning, responsive enterprise applications directly from your browser with minimal coding? The new Oracle APEX Developer Professional learning path and certification enables you to leverage AI-assisted development, including generative AI and Database 23ai, to build secure, scalable web and mobile applications with advanced AI-powered features. From now through May 15, 2025, we're waiving the certification exam fee (valued at $245). So, what are you waiting for? Visit mylearn.oracle.com to get started today. 09:45 Nikita: Welcome back! Samvit, what's the onboarding process like? Samvit: You have to complete the onboarding process to use the service in Microsoft Azure. But before you do that, you first have to complete the subscription process. You must have an active Microsoft Azure account subscription that will be used for subscribing and onboarding the Oracle Database@Azure service.  To subscribe to Oracle Database@Azure, you need to purchase an Oracle Database@Azure private offer from Azure Marketplace. As a customer, you will first reach out to Oracle Sales and negotiate a price for the service. Oracle will provide you with the billing account ID and contact details of the person within the organization who will be handling the service.  After this, Oracle will create a private offer in Azure Marketplace.  10:40 Lois: Sorry to interrupt you, but what's a private offer? Samvit: That's alright, Lois. Private offers are basically solutions or services created for customers by a Microsoft partner, which, in this case, is Oracle. Purchase of those private offers happens from the private offer management page of Azure Marketplace.  But there is a prerequisite. The Azure account must be enabled to make private offer purchases on the subscription from Azure Marketplace. You can refer to the Azure documentation to enable the account, if it is not enabled. You review the offer terms and accept the purchase offer, which will take you to the Create Oracle Subscription page.  You validate the subscription and other particulars and proceed with the process. After the service is deployed, the purchase status of the private offer changes to subscribed.  There are a few points to note here. Billing and payment are done via Azure, and you can use Microsoft Azure Consumption Commitment.  You can also use your on-premises licenses with the Bring Your Own License option and the Unlimited License Agreements to pay towards your service consumption. And you also receive Oracle Support rewards for every dollar spent on the service.  12:02 Nikita: OK, now that I'm subscribed, what's next? Samvit: After you complete the subscription step, Oracle Database@Azure will appear as an Azure resource, just like any other Azure service, and you can move on to onboarding. Onboarding begins with the linking of your OCI account, which will be used for provisioning and managing database resources.  The account is also used for provisioning infrastructure and software maintenance updates for the database service. You can either provide an existing OCI account or create a new one. Then you set up Identity Federation between the Azure account and the OCI tenancy.  This can authenticate login to the OCI portal using Azure credentials, which you require while performing certain operations in OCI. For example, provisioning databases, getting infrastructure and software maintenance updates, and so on. This is an optional step, but it is recommended that you complete the Federation.  The last step is to authorize users by assigning groups and roles in order to have the needed privileges to perform different operations. For example, some groups of users can manage Exadata Database Service resources in Azure, while some can manage the databases in OCI.  You can refer to OCI documentation to get detailed descriptions of roles and group names. 13:31 Lois: Right. That will ensure you assign the correct permissions to the appropriate users. Samvit: Exactly. Assigning the correct roles and permissions to individuals inside the organization is a necessary step for transacting in the marketplace and guaranteeing a smooth purchasing experience. Azure Marketplace uses Azure Role-Based Access Control to enable you to acquire solutions certified to run on Azure. Those are then going to determine the purchasing privileges within the organization.  14:03 Nikita: There's so much more we can discuss about Oracle Database@Azure, but we have to stop somewhere! Thank you so much, Samvit, for joining us over these last three episodes. Lois: Yeah, it's been great to have you, Samvit. Samvit: Thank you for having me. Nikita: Remember, we also have the Oracle Database@Google Cloud service. So, if you want to learn about that, or even if you want to dive deeper into the topics we covered today, go to mylearn.oracle.com and search for the Oracle Cloud Infrastructure Multicloud Architect Professional course.  Lois: There are a lot of demonstrations that you'll surely find useful. Well, that's all we have for today. Until next time, this is Lois Houston… Nikita: And Nikita Abraham, signing off! 14:43 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Join Lois Houston and Nikita Abraham as they interview Samvit Mishra, Senior Manager of CSS OU Cloud Delivery, on Oracle Interconnect for Azure. Learn how this interconnect revolutionizes the customer experience by providing a direct, private link between Oracle Cloud Infrastructure and Microsoft Azure. From use cases to bandwidth considerations, get an in-depth look into how Oracle and Azure come together to create a unified cloud experience.   Oracle Cloud Infrastructure Multicloud Architect Professional: https://mylearn.oracle.com/ou/course/oracle-cloud-infrastructure-multicloud-architect-professional-2025-/144474 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------------   Episode transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead: Editorial Services with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hey there! Last week, we spoke about multicloud, discussing what it is, and the new partnerships we have with Microsoft Azure, Google Cloud, and Amazon Web Services. If you haven't gotten to the episode yet, we suggest you go back and listen to it before you dive into this one.  00:56 Nikita: Joining us again is Samvit Mishra, Senior Manager of CSS OU Cloud Delivery, and we're going to ask him about Oracle Interconnect for Azure. We'll look at the scenarios around Oracle Interconnect for Azure and talk about some considerations too. Hi  Samvit! Thanks for being with us today. Samvit: Hi Niki! Hi Lois!  Lois: Samvit, you introduced Oracle Interconnect for Azure last week, but tell us, how does it improve the customer experience? What benefits does it offer? 01:25 Samvit: Oracle Interconnect for Azure can be established with a one-time setup, eliminating the need for an intermediary network provider. This cross-cloud direct connection also helps you migrate to the cloud or build cloud-native applications by using the best of OCI and Microsoft Azure.  Now, because it is a private connection between Oracle Cloud Infrastructure and Microsoft Azure, you get consistent network performance… around 2 millisecond latency. The interconnect also enables joint customers to take advantage of a unified Identity and Access Management platform. So, you can set up single sign-on between Microsoft Azure and OCI for your Oracle applications, like PeopleSoft and e-Business Suite.  02:16 Nikita: That makes the integration pretty seamless, right? Samvit: Exactly, Niki. Having a federated single sign-on means you authenticate only once to access multiple applications, without signing in separately to access each application. And you also get a secure inter-cloud connection that bypasses the public internet.  02:38 Nikita: How extensive is the global reach of Oracle Cloud Infrastructure and Azure in terms of the number of cloud regions available? Samvit: OCI has the fastest growing network of global data centers, with 50 cloud regions available. And there are 12 Azure interconnect regions. For example, Ashburn in the US is an OCI-Azure interconnect region.  03:01 Lois: Samvit, what is the architecture of Oracle Interconnect for Azure like? How is data transferred securely between a Virtual Cloud Network in Oracle Cloud Infrastructure and a Virtual Network in Microsoft Azure? Samvit: A Virtual Network in a Microsoft Azure region is connected to a Virtual Cloud Network in an OCI region using a private interconnection composed of Azure ExpressRoute and OCI FastConnect. Now, on the OCI side, the FastConnect virtual circuit terminates at a dynamic routing gateway, which is attached to the Virtual Cloud Network. On the Microsoft Azure side, the ExpressRoute connection ends at a virtual network gateway, which is attached to a virtual network. So, traffic from Azure to OCI is routed through the virtual network gateway in Microsoft Azure to the dynamic routing gateway in OCI.  What's important to note is that in both directions, the traffic never leaves the private network.  04:05 Nikita: Wow, ok. Samvit, what are some common use cases of Oracle Interconnect for Azure? Can you give us an example of a supported deployment option? Samvit: We can have a .NET application running in Azure that can access an Oracle database in OCI. Similarly, you can also have custom cloud-native applications running on Azure using Oracle Autonomous Database on the OCI side.  04:29 Lois: And are there any prerequisites when you configure Oracle Interconnect for Azure? Samvit: Yes, there are. Remember, on the Azure side, you must have a virtual network with subnets and a virtual network gateway and on the OCI side, you must have a VCN with subnets and an attached dynamic routing gateway.  04:50 Lois: Let's talk about the networking components that are involved in each site of the connection. Can you run us through the comparison? Samvit: Now, if we talk about the virtual network component, on the OCI side, there is a Virtual Cloud Network and on the Azure side, there is a Virtual Network. From a virtual circuit standpoint, in OCI, there is the FastConnect virtual circuit… on the Azure side, there is the ExpressRoute circuit. When it comes to the gateway, on the OCI side, there is the dynamic routing gateway and on the Azure side, there is the virtual network gateway.  Similarly, for routing, there are route tables in OCI and Microsoft Azure. From a security standpoint, in OCI, you can configure security lists as well as network security groups and on the Azure side, you have network security groups.  05:44 Nikita: What are the benefits of this partnership?  Samvit: This partnership allows you to innovate using the best combination of Oracle's and Microsoft's cloud services based on their features, performance, and pricing. So, in a way, you can combine the capabilities of both cloud vendors.  06:01 Nikita: So, a one-stop shop. Samvit: Exactly, Niki. This partnership also gives you a highly optimized, secure, and unified cross-cloud experience so you can use the best of services from Oracle Cloud Infrastructure and Microsoft Azure. And the best part is you continue to leverage any existing investment in Oracle and Microsoft technologies.  06:24 Lois: I wanted to ask you about the typical scenarios where Oracle Interconnect for Azure is supported. Samvit: There are many scenarios where this Interconnect is supported. Let me run you through a couple of them. You could connect an OCI Virtual Cloud Network to an Azure Virtual Network. That's a scenario that is supported.  You could connect peered OCI VCNs in the same region to Azure. You could connect peered OCI VCNs in different regions to Azure. You could also connect services in Oracle Services Network to Azure. 06:59 Lois: And are there any scenarios where this interconnect is not supported? Samvit: When the scenario involves connecting an on-premises environment to Azure via OCI VCN and vice versa, that is not supported.  07:16 Unlock the power of AI Vector Search with our new course and certification. Get more accurate search results, handle complex datasets easily, and supercharge your data-driven decisions. From now through May 15, 2025, we are waiving the certification exam fee (valued at $245). Visit mylearn.oracle.com to enroll. 07:42 Nikita: Welcome back! I want to explore these scenarios in a little more detail, Samvit.  Samvit: OK. Imagine you have OCI on one side and Azure on the other. In this scenario, we have a dynamic routing gateway in OCI and a virtual network gateway in Azure. This is a basic configuration. With Oracle FastConnect and Azure ExpressRoute, customers can create a private interconnection between their OCI and Azure environments. Now in another scenario, we have VCNs in OCI, and they're peered together using a dynamic routing gateway. With this local peering, the peered VCN can talk to Azure through Oracle Interconnect for Azure.  Here's another scenario. We have VCNs in different OCI regions: one VCN in OCI Region 1 and another in OCI Region 2, with Azure sitting alongside. They have established a remote peering connection, and each VCN has its own dynamic routing gateway. Here's the kicker—the peered VCN in this architecture can also converse with Azure using the interconnect.  Now think about this scenario. We have the dynamic routing gateway, but we have also added a service gateway to the VCN in OCI. This service gateway allows your VCN to privately access specific Oracle services without exposing data to the public internet. No internet gateway or NAT gateway is required to reach those specific services. Now, traffic from the VCN to the Oracle Services Network travels over the Oracle network fabric and never traverses the internet. Using Oracle Interconnect for Azure, resources in Azure can also privately access resources in Oracle Services Network.  09:38 Nikita: What are the bandwidth and cost considerations? Samvit: Pricing is based solely on the port capabilities of OCI FastConnect and your ExpressRoute. One thing you need to understand is that the cost of FastConnect is the same across all OCI regions. And there are no separate ingress or egress data charges. The cost of Azure ExpressRoute varies across regions and Oracle recommends that you use the local setting, which has no separate ingress or egress charges. Azure ExpressRoute supports up to 10 GB as bandwidth. FastConnect is available in 1, 2, 5, or 10 Gbps. So, the recommendation here is to choose one of these matching bandwidth options under ExpressRoute.  10:27 Lois: Thank you, Samvit, for taking the time to talk to us about Oracle Interconnect for Azure. Samvit: Thank you for having me. Nikita: Remember, Oracle also offers an interconnect solution with Google Cloud, which is very similar to the one with Azure. It too provides a direct, high-performance, and secure network connection with Oracle Cloud Infrastructure. So, if you want to learn more about it, head over to mylearn.oracle.com and search for the Oracle Cloud Infrastructure Multicloud Architect Professional course.  Lois: In our next episode, we'll take a close look at Oracle Database@Azure service. Until then, this is Lois Houston… Nikita: And Nikita Abraham, signing off! 11:07 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.    

This week, hosts Lois Houston and Nikita Abraham are shining a light on multicloud, a game-changing strategy involving the use of multiple cloud service providers. Joined by Senior Manager of CSS OU Cloud Delivery Samvit Mishra, they discuss why multicloud is becoming essential for businesses, offering freedom from vendor lock-in and the ability to cherry-pick the best services. They also talk about Oracle's pioneering role in multicloud and its partnerships with Microsoft Azure, Google Cloud, and Amazon Web Services.   Oracle Cloud Infrastructure Multicloud Architect Professional: https://mylearn.oracle.com/ou/course/oracle-cloud-infrastructure-multicloud-architect-professional-2025-/144474 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, and the OU Studio Team for helping us create this episode.   -----------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Hello and welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me today is Nikita Abraham, Team Lead: Editorial Services. Nikita: Hi everyone! Today, we're moving on to multicloud. In our next three episodes, we'll be discussing what multicloud is and why there's so much of a buzz around it. With us is Samvit Mishra, Senior Manager of CSS OU Cloud Delivery. Hi Samvit! Thanks for joining us today. 00:55 Samvit: Hi Niki! Hi Lois! Happy to be here. Lois: So Samvit, we know that Oracle has been an early adopter of multicloud and a pioneer in multicloud services. But for anyone who isn't familiar with what multicloud is, can you explain what it means? Samvit: Absolutely, Lois. Multicloud is a very simple, basic concept. It is the coordinated use of cloud services from more than one cloud service provider.  01:21 Nikita: But why would someone want to use more than one cloud service provider? Samvit: There are many reasons why a customer might want to leverage two or more cloud service providers. First, it addresses the very real concern of mitigating or avoiding vendor lock-in. By using multiple providers, companies can avoid being tied down to one vendor and maintain their flexibility. 01:45 Lois: That's like not putting all your eggs in one basket, so to speak. Samvit: Exactly. Another reason is that customers want the best of breed. What that means is basically leveraging or utilizing the best product from one cloud service provider and pairing it against the best product from another cloud service provider. Getting a solution out of the combined products…out of the coordinated use of those services.  02:14 Nikita: So, it sounds like multicloud is becoming the new normal. And as we were saying before, Oracle was a pioneer in this space. But why did we embrace multicloud so wholeheartedly? Samvit: We recognized that our customers were already moving in this direction. Independent studies from Flexera found that 89% of the subjects of the study used multicloud. And we conducted our own study and came to similar numbers. Over 90% of our customers use two or more cloud service providers.  HashiCorp, the big infrastructure as code company, came to similar numbers as well, 94%. They basically asked companies if multicloud helped them advance their business goals. And 94% said yes. And all this is very recent data.  03:04 Lois: Can you give us the backstory of Oracle's entry into the multicloud space? Samvit: Sure. So back in 2019, Oracle and Microsoft Azure joined forces and announced the interconnect service between Oracle Cloud Infrastructure and Microsoft Azure. The interconnect was between Oracle's FastConnect and Microsoft Azure's ExpressRoute. This was a big step, as it allowed for a direct connection between the two providers without needing a third-party. And now we have several of our data centers interconnected already. So, out of the 48 regions, 12 of them are already interconnected. And more are coming. And you can very easily configure the interconnect.  This interconnectivity guarantees low latency, high throughput, and predictable performance. And also, on the OCI side, there are no egress or ingress charges for your data.  There's also a product called Oracle Database@Azure, where Oracle and Microsoft deliver Oracle Database services in Microsoft Azure data centers.  04:12  Lois: That's exciting! And what are the benefits of this product? Samvit: The main advantage is the co-location. Being co-located with the Microsoft Azure data center offers you native integration between Azure and OCI resources. No manual configuration of a private interconnect between the two providers is needed. You're going to get microsecond latency between your applications and the Oracle Database.  The OCI-native Exadata Database Service is available on Oracle Database@Azure. This enables you to get the highest level of Oracle Database performance, scalability, security, and availability. And your tech support can be provided either from Microsoft or from Oracle.  05:03 Unlock the power of AI Vector Search with our new course and certification. Get more accurate search results, handle complex datasets easily, and supercharge your data-driven decisions. From now through May 15, 2025, we are waiving the certification exam fee (valued at $245). Visit mylearn.oracle.com to enroll. 05:30 Nikita: Welcome back. Samvit, there have been some new multicloud milestones from OCI, right? Can you tell us about them?  Samvit: That's right, Niki. I am thrilled to share the latest news on Oracle's multicloud partnerships. We now have agreements with Microsoft Azure, Google Cloud, and Amazon Web Services.  So, as we were discussing earlier, with Azure, we have the Oracle Interconnect for Azure and Oracle Database@Azure. Now, with Google Cloud, we have the Oracle Interconnect for Google Cloud. And it is very similar to the Oracle Interconnect for Azure. With Google Cloud, we have physically interconnected data centers and they provide a sub-2 millisecond latency private interconnection. So, you can come in and provision virtual circuits going from Oracle FastConnect to Google Cloud Interconnect.  And the best thing is that there are no egress or ingress charges for your data. The way it is structured is you have your Oracle Cloud Infrastructure on one side, with your virtual cloud network, your subnets, and your resources. And on the other side, you have your Google Cloud router with your virtual private cloud subnet and your resources interconnecting.  You initiate the connectivity on the Google Cloud side, retrieve the service key and provide that service key to Oracle Cloud Infrastructure, and complete the interconnection on the OCI side. So, for example, our US East Ashburn interconnect will match with us-east4 on the Google Cloud side.  07:08 Lois: Now, wasn't the other major announcement Oracle Database@Google Cloud? Tell us more about that, please. Samvit: With Oracle Database@Google Cloud, you can run your applications on Google Cloud and the database as well inside the Google Cloud platform. That's the Oracle Cloud Infrastructure database co-located in Google Cloud platform data centers. It allows you to run native integration between GCP and OCI resources with no manual configuration of private interconnect between these two cloud service providers.  That means no FastConnect, no Interconnect because, again, the database is located in the Google Cloud data center. And you're going to get microsecond latency and the OCI native Exadata Database Service. So, you're going to gain the highest level of Oracle Database performance, scalability, security, and availability.  08:04 Lois: And how is the tech support managed? Samvit: The technical support is a collaboration between Google Cloud and Oracle Cloud Infrastructure. That means you can either have the technical support provided to completion by Google Cloud or by Oracle. One of us will provide you with an end-to-end solution.  08:22 Nikita: During CloudWorld last year, we also announced Oracle Database@AWS, right?  Samvit: Yes, Niki. That's where Oracle and Amazon Web Services deliver the Oracle Database service on Oracle Cloud Infrastructure in your AWS data center. This will provide you with native integration between AWS and OCI resources, with no manual configuration of private interconnect between AWS and OCI. And you're getting microsecond latency with the OCI-native Exadata Database Service.  And again, as with Oracle Database@Google Cloud and Oracle Database@Azure, you're gaining the highest level of Oracle Database performance, scalability, security, and availability. And the technical support is provided by either AWS or Oracle all the way to completion. Now, Oracle Database@AWS is currently available in limited preview, with broader availability in the coming months as it expands to new regions to meet the needs of our customers.  09:28 Lois: That's great. Now, how does Oracle fare when it comes to pricing, especially compared to our major cloud competitors?  Samvit: Our pricing is pretty consistent. You'll see that in all cases across the world, we have the less expensive solution for you and the highest performance as well.  09:45 Nikita: Let's move on to some use cases, Samvit. How might a company use the multicloud setup? Samvit: Let's start with the split-stack architecture between Oracle Cloud Infrastructure and Microsoft Azure.  Like I was saying earlier, this partnership dates back to 2019. And basically, we eliminated the FastConnect partner from the middle. And this will provide you with high throughput, low latency, and very predictable performance, all of this on highly available links. These links are redundant, ensuring business continuity between OCI and Azure.  And you can have your database on the OCI side and your application on Microsoft Azure side or the other way around. You can have SQL Server on Azure and the application running on Oracle Cloud Infrastructure. And this is very easy to configure.  10:34 Lois: It really sounds like Oracle is at the forefront of the multicloud revolution. Thanks so much, Samvit, for shedding light on this exciting topic.  Samvit: It was my pleasure.  Nikita: That's a wrap for today. To learn more about what we discussed, head over to mylearn.oracle.com and search for the Oracle Cloud Infrastructure Multicloud Architect Professional course. In our next episode, we'll take a close look at Oracle Interconnect for Azure. Until then, this is Nikita Abraham… Lois: And Lois Houston, signing off! 11:05 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this special episode of the Oracle University Podcast, hosts Lois Houston and Nikita Abraham dive into Oracle Fusion Cloud Applications and the new courses and certifications on offer. They are joined by Oracle Fusion Apps experts Patrick McBride and Bill Lawson who introduce the concept of Oracle Modern Best Practice (OMBP), explaining how it helps organizations maximize results by mapping Fusion Application features to daily business processes. They also discuss how the new courses educate learners on OMBP and its role in improving Fusion Cloud Apps implementations.   OMBP: https://www.oracle.com/applications/modern-best-practice/ Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and with me is Lois Houston, Director of Innovation Programs.  Lois: Hi everyone! For the last two months, we've been focusing on all things MySQL. But today, we wanted to share some really exciting news about new courses and certifications on Oracle Fusion Cloud Applications that feature Oracle Modern Best Practice, or OMBP, and Oracle Cloud Success Navigator.  00:57 Nikita: And to tell us more about this, we have two very special guests joining us today. Patrick McBride is a Senior Director from the Fusion Application Development organization. He leads the Oracle Modern Best Practice Program office for Oracle. And Bill Lawson is a Senior Director for Cloud Applications Product Management here at Oracle University. We'll first ask Patrick about Oracle Modern Best Practice and then move on to Bill for details about the new training and certification we're offering. Patrick, Bill, thanks for being here today.   Patrick: Hey, Niki and Lois, thanks for the invitation. Happy to be here.  Bill: Hi Niki, Lois. 01:32 Lois: Patrick, let's start with some basic information about what OMBP are. Can you tell us a little about why they were created? Patrick: Sure, love to. So, modern best practices are more than just a business process. They're really about translating features and technology into actionable capabilities in our product. So, we've created these by curating industry leading best practices we've collected from our customers over the years. And ensure that the most modern technologies that we've built into the Fusion Application stack are represented inside of those business processes. Our goal is really to help you as customers improve your business operations by easily finding and applying those technologies to what you do every day.  02:18 Nikita: So, by understanding these modern best practice and the technology that enables it, you're really unlocking the full potential of Fusion Apps.  Patrick: Absolutely. So, the goal is that modern best practice make it really easy for customers, implementers, partners, to see the opportunity and take action. 02:38 Lois: That's great. OK, so, let's talk about implementations, Patrick. How do Oracle Modern Best Practice support customers throughout the lifecycle of an Oracle Fusion Cloud implementation? Patrick: What we found during many implementers' journey with taking our solution and trying to apply it with customers is that customers come in with a long list of capabilities that they're asking us to replicate. What they've always done in the past. And what modern best practice is trying to do is help customers to reimage the art of the possible…what's possible with Fusion by taking advantage of innovative features like AI, like IoT, like, you know, all of the other solutions that we built in to help you automate your processes to help you get the most out of the solution using the latest and greatest technology. So, if you're an implementer, there's a number of ways a modern best practice can help during an implementation. First is that reimagine exercise where you can help the customer see what's possible. And how we can do it in a better way. I think more importantly though, as you go through your implementation, many customers aren't able to get everything done by the time they have to go live. They have a list of things they've deferred and modern best practices really establishes itself as a road map for success, so you can go back to it at the completion and see what's left for the opportunity to take advantage of and you can use it to track kind of the continuous innovation that Oracle delivers with every release and see what's changed with that business process and how can I get the most out of it. 04:08 Nikita: Thanks, Patrick. That's a great primer on OMBP that I'm sure everyone will find very helpful. Patrick: Thanks, Niki. We want our customers to understand the value of modern best practices so they can really maximize their investment in Oracle technology today and in the future as we continue to innovate. 04:24 Lois: Right. And the way we're doing that is through new training and certifications that are closely aligned with OMBP. Bill, what can you tell us about this? Bill: Yes, sure. So, the new Oracle Fusion Applications Foundations training program is designed to help partners and customers understand Oracle Modern Best Practice and how they improve the entire implementation journey with Fusion Cloud Applications. As a learner, you will understand how to adhere to these practices and how they promise a greater level of success and customer satisfaction. So, whether you're designing, or implementing, or going live, you'll be able to get it right on day one. So, like Patrick was saying, these OMBPs are reimagined, industry-standard business processes built into Fusion Applications. So, you'll also discover how technologies like AI, Mobile, and Analytics help you automate tasks and make smarter decisions. You'll see how data flows between processes and get tips for successful go-lives. So, the training we're offering includes product demonstrations, key metrics, and design considerations to give you a solid understanding of modern best practice. It also introduces you to Oracle Cloud Success Navigator and how it can be leveraged and relied upon as a trusted source to guide you through every step of your cloud journey, so from planning, designing, and implementation, to user acceptance testing and post-go-live innovations with each quarterly new release of Fusion Applications and those new features. And then, the training also prepares you for Oracle Cloud Applications Foundations certifications.                   05:55 Nikita: Which applications does the training focus on, Bill? Bill: Sure, so the training focuses on four key pillars of Fusion Apps and the associated OMBP with them. For Human Capital Management, we cover Human Resources and Talent Management. For Enterprise Resource Planning, it's all about Financials, Project Management, and Risk Management. In Supply Chain Management, you'll look at Supply Chain, Manufacturing, Inventory, Procurement, and more. And for Customer Experience, we'll focus on Marketing, Sales, and Service. 06:24 Lois: That's great, Bill. Now, who is the training and certification for?  Bill: That's a great question. So, it's really for anyone who wants to get the most out of Oracle Fusion Cloud Applications. It doesn't matter if you're an experienced professional or someone new to Fusion Apps, this is a great place to start. It's even recommended for professionals with experience in implementing other applications, like on-premise products. The goal is to give you a solid foundation in Oracle Modern Best Practice and show you how to use them to improve your implementation approach. We want to make it easy for anyone, whether you're an implementer, a global process owner, or an IT team employee, to identify every way Fusion Applications can improve your organization. So, if you're new to Fusion Apps, you'll get a comprehensive overview of Oracle Fusion Applications and how to use OMBP to improve business operations. If you're already certified in Oracle Cloud Applications and have years of experience, you'll still benefit from learning how OMBP fits into your work.  If you're an experienced Fusion consultant who is new to Oracle Modern Best Practice processes, this is a good place to begin and learn how to apply them and the latest technology enablers during implementations.  And, lastly, if you're an on-premise or you have non-Fusion consultant skills looking to upskill to Fusion, this is a great way to begin acquiring the knowledge and skills needed to transition to Fusion and migrate your existing expertise. 07:53 Raise your game with the Oracle Cloud Applications skills challenge. Get free training on Oracle Fusion Cloud Applications, Oracle Modern Best Practice, and Oracle Cloud Success Navigator. Pass the free Oracle Fusion Cloud Foundations Associate exam to earn a Foundations Associate certification. Plus, there's a chance to win awards and prizes throughout the challenge! What are you waiting for? Join the challenge today by visiting oracle.com/education. 08:27 Nikita: Welcome back! Bill, how long is it going to take me to complete this training program? Bill: So, we wanted to make this program detailed enough so our learners find it valuable, obviously. But at the same time, we didn't want to make it too long. So, each course is approximately 5 hours or more, and provides folks with all the requisite knowledge they need to get started with Oracle Modern Best Practice and Fusion Applications.  08:51 Lois: Bill, is there anything that I need to know before I take this course? Are there any prerequisites? Bill: No, Lois, there are no prerequisites. Like I was saying, whether you're fresh out of college or a seasoned professional, this is a great place to start your journey into Fusion Apps and Oracle Modern Best Practice. 09:06 Nikita: That's great, you know, that there are no barriers to starting. Now, Bill, what can you tell us about the certification that goes along with this new program?  Bill: The best part, Niki, is that it's free. In fact, the training is also free. We have four courses and corresponding Foundation Associate–level certifications for Human Capital Management, Enterprise Resource Planning, Supply Chain Management, and Customer Experience. So, completing the training prepares you for an hour-long exam with 25 questions. It's a pretty straightforward way to validate your expertise in Oracle Modern Best Practice and Fusion Apps implementation considerations. 09:40 Nikita: Ok. Say I take this course and certification. What can I do next? Where should my learning journey take me? Bill: So, you're building knowledge and expertise with Fusion Applications, correct? So, once you take this training and certification, I recommend that you identify a product area you want to specialize in. So, if you take the Foundations training for HCM, you can dive deeper into specialized paths focused on implementing Human Resources, Workforce Management, Talent Management, or Payroll applications, for example.  The same goes for other product areas. If you finish the certification for Foundations in ERP, you may choose to specialize in Finance or Project Management and get your professional certifications there as your next step. So, once you have this foundational knowledge, moving on to advanced learning in these areas becomes much easier. We offer various learning paths with associated professional-level certifications to deepen your knowledge and expertise in Oracle Fusion Cloud Applications. So, you can learn more about these courses by visiting oracle.com/education/training/ to find out more of what Oracle University has to offer. 10:43 Lois: Right. I love that we have a clear path from foundational-level training to more advanced levels. So, as your skills grow, we've got the resources to help you move forward.  Nikita: That's right, Lois. Thanks for walking us through all this, Patrick and Bill. We really appreciate you taking the time to join us on the podcast. Bill: Yeah, it's always a pleasure to join you on the podcast. Thank you very much. Patrick: Oh, thanks for having me, Lois. Happy to be here. Lois: Well, that's all the time we have for today. If you have questions or suggestions about anything we discussed today, you can write to us at ou-podcast_ww@oracle.com. That's ou-podcast_ww@oracle.com. Until next time, this is Lois Houston… Nikita: And Nikita Abraham, signing off! 11:29 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, Lois Houston and Nikita Abraham chat with MySQL expert Perside Foster on the importance of keeping MySQL performing at its best. They discuss the essential tools for monitoring MySQL, tackling slow queries, and boosting overall performance.   They also explore HeatWave, the powerful real-time analytics engine that brings machine learning and cross-cloud flexibility into MySQL.   MySQL 8.4 Essentials: https://mylearn.oracle.com/ou/course/mysql-84-essentials/141332/226362 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   ----------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me today is Nikita Abraham, Team Lead: Editorial Services. Nikita: Hey everyone! In our last two episodes, we spoke about MySQL backups, exploring their critical role in data recovery, error correction, data migration, and more. Lois: Today, we're switching gears to talk about monitoring MySQL instances. We'll also explore the features and benefits of HeatWave with Perside Foster, a MySQL Principal Solution Engineer at Oracle. 01:02 Nikita: Hi, Perside! We're thrilled to have you here for one last time this season. So, let's start by discussing the importance of monitoring systems in general, especially when it comes to MySQL. Perside: Database administrators face a lot of challenges, and these sometimes appear in the form of questions that a DBA must answer. One of the most basic question is, why is the database slow? To address this, the next step is to determine which queries are taking the longest. Queries that take a long time might be because they are not correctly indexed. Then we get to some environmental queries or questions. How can we find out if our replicas are out of date? If lag is too much of a problem? Can I restore my last backup? Is the database storage likely to fill up any time soon? Can and should we consider adding more servers and scaling out the system? And when it comes to users and making sure they're behaving correctly, has the database structure changed? And if so, who did it and what did they do? And more generally, what security issues have arisen? How can I see what has happened and how can I fix it? Performance is always at the top of the list of things a DBA worries about. The underlying hardware will always be a factor but is one of the things a DBA has the least flexibility with changing over the short time. The database structure, choice of data types and the overall size of retained data in the active data set can be a problem. 03:01 Nikita: What are some common performance issues that database administrators encounter? Perside: The sort of SQL queries that the application runs can be an issue. 90% of performance problems come from the SQL index and schema group.  03:18 Lois: Perside, can you give us a checklist of the things we should monitor? Perside: Make sure your system is working. Monitor performance continually. Make sure replication is working. Check your backup. Keep an eye on disk space and how it grows over time. Check when long running queries block your application and identify those queries. Protect your database structure from unauthorized changes. Make sure the operating system itself is working fine and check that nothing unusual happened at that level. Keep aware of security vulnerabilities in your software and operating system and ensure that they are kept updated. Verify that your database memory usage is under control. 04:14 Lois: That's a great list, Perside. Thanks for that. Now, what tools can we use to effectively monitor MySQL?     Perside: The slow query log is a simple way to monitor long running queries. Two variables control the log queries. Long_query_time. If a query takes longer than this many seconds, it gets logged. And then there's min_exam_row_limit. If a query looks at more than this many rows, it gets logged. The slow query log doesn't ordinarily record administrative statements or queries that don't use indexes. Two variables control this, log_slow_admin_statements and log_queries_not_using_indexes. Once you have found a query that takes a long time to run, you can focus on optimizing the application, either by limiting this type of query or by optimizing it in some way. 05:23 Nikita: Perside, what tools can help us optimize slow queries and manage data more efficiently? Perside: To help you with processing the slow query log file, you can use the MySQL dump slow command to summarize slow queries. Another important monitoring feature of MySQL is the performance schema. It's a system database that provides statistics of how MySQL executes at a low level. Unlike user databases, performance schema does not persist data to disk. It uses its own storage engine that is flushed every time we start MySQL. And it has almost no interaction with the storage media, making it very fast. This performance information belongs only to the specific instance, so it's not replicated to other systems. Also, performance schema does not grow infinitely large. Instead, each row is recorded in a fixed size ring buffer. This means that when it's full, it starts again at the beginning. The SYS schema is another system database that's strongly related to performance schema. 06:49 Nikita: And how can the SYS schema enhance our monitoring efforts in MySQL? Perside: It contains helper objects like views and stored procedures. They help simplify common monitoring tasks and can help monitor server health and diagnose performance issues. Some of the views provide insights into I/O hotspots, blocking and locking issues, statements that use a lot of resources in various statistics on your busiest tables and indexes. 07:26 Lois: Ok… can you tell us about some of the features within the broader Oracle ecosystem that enhance our ability to monitor MySQL? Perside: As an Oracle customer, you also have access to Oracle Enterprise Manager. This tool supports a huge range of Oracle products. And for MySQL, it's used to monitor performance, system availability, your replication topology, InnoDB performance characteristics and locking, bad queries caught by the MySQL Enterprise firewall, and events that are raised by the MySQL Enterprise audit. 08:08 Nikita: What would you say are some of the standout features of Oracle Enterprise Manager? Perside: When you use MySQL in OCI, you have access to some really powerful features. HeatWave MySQL enables continuous monitoring of query statistics and performance. The health monitor is part of the MySQL server and gathers raw data about the performance of queries. You can see summaries of this information in the Performance Hub in the OCI Console. For example, you can see average statement latency or top 100 statements executed. MySQL metrics lets you drill in with your own custom monitoring queries. This works well with existing OCI features that you might already know. The observability and management framework lets you filter by resource type and across several dimensions. And you can configure OCI alarms to be notified when some condition is reached. 09:20 Lois: Perside, could you tell us more about MySQL metrics? Perside: MySQL metrics uses the raw performance data gathered by the health monitor to measure the important characteristic of your servers. This includes CPU and storage usage and information relevant to your database connection and queries executed. With MySQL metrics, you can create your own custom monitoring queries that you can use to feed graphics. This gives you an up to the minute representation of all the performance characteristics that you're interested in. You can also create alarms that trigger on some performance condition. And you can be notified through the OCI alarms framework so that you can be aware instantly when you need to deal with some issue.  10:22 Are you keen to stay ahead in today's fast-paced world? We've got your back! Each quarter, Oracle rolls out game-changing updates to its Fusion Cloud Applications. And to make sure you're always in the know, we offer New Features courses that give you an insider's look at all of the latest advancements. Don't miss out! Head over to mylearn.oracle.com to get started. 10:47 Nikita: Welcome back! Now, let's dive into the key features of HeatWave, the cloud service that integrates with MySQL. Can you tell us what HeatWave is all about? Perside: HeatWave is the cloud service for MySQL. MySQL is the world's leading database for web applications. And with HeatWave, you can run your online transaction processing or OLTP apps in the cloud. This gives you all the benefits of cloud deployments while keeping your MySQL-based web application running just like they would on your own premises. As well as OLTP applications, you need to run reports with Business Intelligence and Analytics Dashboards or Online Analytical Processing, or OLAP reports. The HeatWave cluster provides accelerated analytics queries without requiring extraction or transformation to a separate reporting system. This is achieved with an in-memory analytics accelerator, which is part of the HeatWave service. In addition, HeatWave enables you to create Machine Learning models to embed artificial intelligence right there in the database. The ML accelerator performs classification, regression, time-series forecasting, anomaly detection, and other functions provided by the various models that you can embed in your architecture. HeatWave can also work directly with storage outside the database. With HeatWave Lakehouse, you can run queries directly on data stored in object storage in a variety of formats without needing to import that data into your MySQL database. 12:50 Lois: With all of these exciting features in HeatWave, Perside, what core MySQL benefits can users continue to enjoy? Perside: The reason why you chose MySQL in the first place, it's still a relational database and with full transactional support, low latency, and high throughput for your online transaction processing app. It has encryption, compression, and high availability clustering. It also has the same large database support with up to 256 terabytes support. It has advanced security features, including authentication, data masking, and database firewall. But because it's part of the cloud service, it comes with automated patching, upgrades, and backup. And it is fully supported by the MySQL team. 13:50 Nikita: Ok… let's get back to what the HeatWave service entails. Perside: The HeatWave service is a fully managed MySQL. Through the web-based console, you can deploy your instances and manage backups, enable high availability, resize your instances, create read replicas, and perform many common administration tasks without writing a single line of SQL. It brings with it the power of OCI and MySQL Enterprise Edition. As a managed service, many routine DBA tests are automated. This includes keeping the instances up to date with the latest version and patches. You can run analytics queries right there in the database without needing to extract and transform your databases, or load them in another dedicated analytics system. 14:52 Nikita: Can you share some common use cases for HeatWave? Perside: You have your typical OLTP workloads, just like you'd run on prem, but with the benefit of being managed in the cloud. Analytic queries are accelerated by HeatWave. So your reporting applications and dashboards are way faster. You can run both OLTP and analytics workloads from the same database, keeping your reports up to date without needing a separate reporting infrastructure. 15:25 Lois: I've heard a lot about HeatWave AutoML. Can you explain what that is? Perside: HeatWave AutoML enables in-database artificial intelligence and Machine Learning. Externally sourced data stores, such as sensor data exported to CSV, can be read directly from object store. And HeatWave generative AI enables chatbots and LLM content creation. 15:57 Lois: Perside, tell us about some of the key features and benefits of HeatWave. Perside: Autopilot is a suite of AI-powered tools to improve the performance and applicability of your HeatWave queries. Autopilot includes two features that help cut costs when you provision your service. There's auto provisioning and auto shape prediction. They analyze your existing use case and tell you exactly which shape you must provision for your nodes and how many nodes you need. Auto parallel loading is used when you import data into HeatWave. It splits the import automatically into an optimum number of parallel streams to speed up your import. And then there's auto data placement. It distributes your data across the HeatWave cluster node to improve your query retrieval performance. Auto encoding chooses the correct data storage type for your string data, cutting down storage and retrieval time. Auto error recovery automatically recovers a fail node and reloads data if that node becomes unresponsive. Auto scheduling prioritizes incoming queries intelligently. An auto change propagation brings data optimally from your DB system to the acceleration cluster. And then there's auto query time estimation and auto query plan improvement. They learn from your workload. They use those statistics to perform on node adaptive optimization. This optimization allows each query portion to be executed on every local node based on that node's actual data distribution at runtime. Finally, there's auto thread pooling. It adjusts the enterprise thread pool configuration to maximize concurrent throughput. It is workload-aware, and minimizes resource contention, which can be caused by too many waiting transactions. 18:24 Lois: How does HeatWave simplify analytics within MySQL and with external data sources? Perside: HeatWave in Oracle Cloud Infrastructure provides all the features you need for analytics, all in one system. Your classic OLTP application run on the MySQL database that you know and love, provision in a DB system. On-line analytical processing is done right there in the database without needing to extract and load it to another analytic system. With HeatWave Lakehouse, you can even run your analytics queries against external data stores without loading them to your DB system. And you can run your machine learning models and LLMs in the same HeatWave service using HeatWave AutoML and generative AI. HeatWave is not just available in Oracle Cloud Infrastructure. If you're tied to another cloud vendor, such as AWS or Azure, you can use HeatWave from your applications in those cloud too, and at a great price. 19:43 Nikita: That's awesome! Thank you, Perside, for joining us throughout this season on MySQL. These conversations have been so insightful. If you're interested in learning more about the topics we discussed today, head over to mylearn.oracle.com and search for the MySQL 8.4: Essentials course.  Lois: This wraps up our season on the essentials of MySQL. But before we go, we just want to remind you to write to us if you have any feedback, questions, or ideas for future episodes. Drop us an email at ou-podcast_ww@oracle.com. That's ou-podcast_ww@oracle.com. Nikita: Until next time, this is Nikita Abraham… Lois: And Lois Houston, signing off! 20:33 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Lois Houston and Nikita Abraham continue their conversation with MySQL expert Perside Foster, with a closer look at MySQL Enterprise Backup. They cover essential features like incremental backups for quick recovery, encryption for data security, and monitoring with MySQL Enterprise Monitor—all to help you manage backups smoothly and securely.   MySQL 8.4 Essentials: https://mylearn.oracle.com/ou/course/mysql-84-essentials/141332/226362 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead: Editorial Services  with Oracle University, and with me is Lois Houston, Director of Innovation Programs.  Lois: Hi there! Last week was the first of a two-part episode covering the different types of backups and why they're important. Today, we'll look at how we can use MySQL Enterprise Backup for efficient and consistent backups. 00:52 Nikita: And of course, we've got Perside Foster with us again to walk us through all the details. Perside, could you give us an overview of MySQL Enterprise Backup?  Perside: MySQL Enterprise Backup is a form of physical backup at its core, so it's much faster for large data sets than logical backups, such as the most commonly used MySQL Dump. Because it backs up the data files, it's non-locking and enables either complete system backup or partial backup, focusing only on specific databases. 01:29 Lois: And what are the benefits of using MySQL Enterprise Backup? Perside: You can back up to local storage or direct-to-common-cloud storage types. You can perform incremental backups, which can speed up your backup process greatly. Incremental backups enable point-in-time recovery. It's useful when you need to restore to a point in time before some application or human error occurred. Backups can be compressed to save archival storage requirements and encrypted for regulatory compliance and offline data security. 02:09 Nikita: So we know MySQL Enterprise Backup is an impressive tool, but could you talk more about some of the main features it supports for creating and managing backups? Specifically, which tools are integrated within MySQL Enterprise to support different backup scenarios? Perside: MySQL Enterprise Backup supports SBT, implemented by many common Tape storage systems. MySQL Enterprise Backup supports optimistic backup. This process deals with busy tables separately from the rest of the database. It can record changes that happen in the database during the backup for consistency. In a large data set, this can make a huge difference in performance. MySQL Enterprise Backup runs on all supported platforms. It's available when you have a MySQL Enterprise Edition license. And it comes with Enterprise Edition, but it also is available as a separate package. You can get the most recent version from eDelivery, where you can also get a trial version. If you need a previous release, you can get that from My Oracle Support. It's also available in all versions of MySQL, whether you run a Long-Term support version or an Innovation Release. For LTS releases, MySQL Enterprise Backup supports MySQL instances of the same LTS release. For Innovation releases, it supports the previous LTS release and any subsequent Innovation version within the same LTS family. 04:03 Nikita: How does MySQL Enterprise Monitor manage and track backup processes? Perside: MySQL Enterprise Monitor has a dashboard for monitoring MySQL Enterprise Backup. The dashboard monitors the health of backup process and usage throughout the entire Enterprise fleet, not just a single server. It supports drilling down into specific sub-operations within a backup job. You can see information about full backups, partial backups, and incremental backups. You can configure alerts that will notify you in the event of delays, failures, or backups that have not been performed in some configuration time period. 04:53 Lois: Ok…let's get into the mechanics. I understand that MySQL Enterprise Backup uses binary logs as part of its backup process. Can you explain how these logs fit into the bigger picture of maintaining database integrity? Perside: MySQL Enterprise Backup is a utility designed specifically for backing up MySQL systems in the most efficient and flexible way. At its simplest, it performs a physical backup of the data files, so it is fast. However, it also records the changes that were made during the time it took to do the backup. So, the result is that you get a consistent backup of the data at the time the backup completed. This backup is not tied to the host system and can be moved to other hosts. It can be used for archiving and is fully supported as part of the MySQL Enterprise Edition. It is, however, tied to the specific version of MySQL from which the backup was taken. So, you cannot use it for upgrades where the destination server is an upgrade from the source. For example, if you take a backup from MySQL 5.7, you can't directly restore it to MySQL 8.0. As a part of MySQL Enterprise Edition, it's not part of the freely available Community Edition. 06:29 Lois: Perside, how do MySQL's binary logs track changes over time? And why is this so critical for backups? Perside: The binary logs record changes to the database. These changes are recorded in a sequential set of files numbered incrementally. MySQL logs changes either in statement-based form, where each log entry records the statement that gives rise to the change, or in row-based form where the actual change row data is recorded. If you select mixed format, then MySQL records statements for most operations and records row for changes where the statement might result in a different row value each time it's run, for example, where there's a generated value like autoincrement. The current log file grows as changes are recorded. When it reaches its maximum configured size, that log file is closed, and the next sequential file is created for new logs. You can make this happen automatically by using the FLUSH BINARY LOGS command. This does not delete any existing log files. 07:59 Nikita: But what happens if you want to delete the log files? Perside: If you want to delete all log files, you can do so manually with the PURGE BINARY LOGS command, either specifying a file or a date time.  08:14 Lois: When it comes to tracking transactions, MySQL provides a couple of methods, right? Can you explain the differences between Global Transaction Identifiers and the traditional log file sequence? Perside: Log files positioning is one of two formats, either legacy, where you specify transactions with a log file in a sequence number, or by using global transaction identifiers, or GTIDs, where each transaction is identified with a universally unique identifier or UUID. When you apply a transaction to the source server, that is when the GTID is attached to the transaction. This makes it particularly useful in replication topologies so that each transaction is uniquely identified by both its server ID and the transaction sequence number. When such a transaction is replicated to other hosts, the transaction retains its original GTID so that you can track when that transaction has propagated to the replicas and has been applied. The global transaction identifier is unique across the entire network. 09:49 Have you mastered the basics of AI? Are you ready to take your skills to the next level? Unlock the potential of advanced AI with our OCI Generative AI Professional course and certification that covers topics like LLMs, the OCI Generative AI Service, and building Q&A chatbots for real-world applications. Head over to mylearn.oracle.comand find out more. 10:19 Nikita: Welcome back! Let's move on to replication. How does MySQL's legacy log format handle transactions, and what does that mean for replication timing across different servers? Perside: Legacy format binary logs are non-transactional. This means that a transaction made up of multiple modifications is logged as a sequence of changes. It's possible that different hosts in a replication network apply those changes at different times. Each server that uses legacy binary logging maintain the current applied log position as coordinates based on a combination of binary log files in the position within that log file. 11:11 Nikita: Troubleshooting with legacy logs can be quite complex, right? So, how does the lack of unique transaction IDs make it more difficult to address replication issues? Perside: Because each server has its own log with its own transactions, these modification could have entirely different coordinates, making it challenging to find the specific modification point if you need to do any deep dive troubleshooting, for example, if one replica fell partway through applying a transaction and you need to partially roll it back manually. On the other hand, when you enable GTIDs, the transaction applied on the source host has that globally unique identifier attached to the whole transaction as a sequence of unique IDs. When the second or subsequent servers apply those transactions, they have exactly the same identifier, making it both transaction-safe for MySQL and also easier to troubleshoot if you need to. 12:26 Lois: How can you use binary logs to perform a point-in-time recovery in MySQL? Perside: First, you restore the last full backup. Once you've restarted the restart server, find the current log position of that backup. Either it's GTID or log sequence number. The SHOW BINARY LOG STATUS command shows this information. Then you can use the MySQL binlog utility to replay events from the binary log files, specifying the start and stop position containing the range of log operations that you wish to apply. You can pipe the output of the MySQL bin log to the MySQL client if you want to execute the changes immediately, or you can redirect the output to a script file if you want to examine and perhaps edit the changes. 13:29 Nikita: And how do you save binary logs? Perside: You can save binary logs to use in disaster recovery, for point-in-time restores, or for incremental backups. One way to do this is to flush the logs so that the log file closes and ready for copying. And then copy it to a different server to protect against hardware media failures. You can also use the MySQL binlog utility to create a copy of a set of binary log files in the same format, but to a different file or set of files. This can be useful if you want to run MySQL binlog continuously, copying from the source server binary log to a new location, perhaps in network storage. If you do this, remember that MySQL binlog does not run as a service or daemon, so you'll need to monitor it to make sure it's running continually.  14:39 Lois: Can you take us through how the MySQL Enterprise Backup process works? What does it do when performing a backup? Perside: First, it performs a physical file copy of necessary data and log files. This can be done while the server is fully operational, and it has minimal impact on performance. Once this initial copy is taken, it applies a low impact backup lock on the instance. If you have any tables that are not using InnoDB, the backup cannot guarantee transaction-safe consistency for those tables. It applies a weed lock to those tables so that it can guarantee consistency. Then it briefly locks all logging activity to take a consistent view of the current coordinates of various logs. It releases the weed lock on non-transactional tables. Using the log coordinates that were taken earlier in the process, it gathers all logs for transactions that have occurred since then. Bear in mind that the backup process takes place while the system is active. So, for a consistent backup, it must record not only the data files, but all changes that occurred during the backup. Then it releases the backup lock. The last piece of information recorded is any metadata for the backup itself, including its timing and contents in the final redo log. That completes the backup operation. 16:30 Nikita: And where are the files stored? Perside: The files contained in the backup are saved to the backup location, which can be on the local system or in network storage. The files contained in the backup location include files from the MySQL data directory. Some raw files include InnoDB tablespace, plus any InnoDB file per table tablespace files, and InnoDB log files. Other files might include data files belonging to other storage engines, perhaps MyISAM files. The various log files in instance configuration files are also retained. 17:20 Lois: What steps do you follow to restore a MySQL Enterprise Backup, and how do you guarantee consistency, especially when dealing with incremental backups? Perside: To restore from a backup using MySQL Enterprise Backup, you must first remove any previous files from the data directory. The restore process will fail if you attempt to restore over an existing system or backup. Then you restore the database with appropriate options. If you only restore a single backup, you can use copy, back, and apply log to ensure that the restored system has a consistency state. If you perform a full backup in subsequent incremental backups, you might need to restore multiple times using copy-back, and then use copy-back-and-apply-log only for the final consistent restore operation. The restart server might be on the same host or might be a different host with different configuration. This means that you might have to change some configuration on the restored server, including the operating system ownership of the restored data directory and various MySQL configuration files. If you want to retain the MySQL configuration files from the source server to reproduce on a new server, you should copy those files separately. MySQL Enterprise Backup focuses on the data rather than the server configuration. It does, however, produce configuration files appropriate for the backup. These are similar to the MySQL configuration files, but only contain options relevant for the backup process itself. There's also variables that have been changed to non-default values and all global variable values. These files must be renamed and possibly edited before they are suitable to become configuration files in the newly restored server. For example, the mysqld-auto.cnf file contains a JSON-formatted set of persisted variables. The backup process stores this as the newly named backup mysqld-auto.cnf. If you want to use it in the restored server, you must rename it and place it in the appropriate location so that the restored server can read it. This also applies in part to the auto.cnf file, which contain identifying information for the server. If you are replacing the original server or restoring on the same host, then you can keep the original values. However, this information must be unique within a network. So, if you are restoring this backup to create a replica in a replication topology, you must not include that file and instead start MySQL without it so that it creates its own unique identifying information. 21:14 Nikita: Let's discuss securing and optimizing backups. How does MySQL Enterprise Backup handle encryption and compression, and what are the critical considerations for each? Perside: You can encrypt backups so that they are secure while moving them around or archiving them. The encrypt option performs the encryption. And you can specify the encryption key either on the command line as a string or a key file that has been generated with some cryptographic algorithm. Encryption only applies to image files, not to backup directories. You can also compress backup with different levels of compression, with higher levels requiring more CPU, but resulting in greater savings in storage. Compression only works with InnoDB data files. If your organization has media management software for performing backups, perhaps to a tape array, then you can use the SBT interface supported in MySQL Enterprise Backup. 22:34 Lois: Before we wrap up, could you share how MySQL Enterprise Backup facilitates the management of backups across a multi-server environment? Perside: As an enterprise solution, it's easy to run MySQL Enterprise Backup in a multi-server environment. We've already mentioned backing up to cloud storage, but you can, of course, back up to a directory or image on network storage that can be mounted locally, perhaps with NFS or some other file system. The "with time" option enables multiple backups within the same backup directory, where each in its own subdirectory named with the timestamp. This is especially useful when you want to run the same backup script repeatedly.  23:32 Lois: Thank you for that detailed overview, Perside. This wraps up our discussion of the various backup types, their pros and cons, and how to select the right option for your needs. In our next session, we'll explore the different MySQL monitoring strategies and look at the features as well as benefits of Heatwave. Nikita: And if you want to learn more about the topics we discussed today, head over to mylearn.oracle.com and take a look at the MySQL 8.4 Essentials course. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 24:06 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Join Lois Houston and Nikita Abraham as they kick off a two-part episode on MySQL backups with MySQL expert Perside Foster. In this conversation, they explore the critical role of backups in data recovery, error correction, data migration, and more.   Perside breaks down the differences between logical and physical backups, discussing their pros and cons, and shares valuable insights on how to create a reliable backup strategy to safeguard your data.   MySQL 8.4 Essentials: https://mylearn.oracle.com/ou/course/mysql-84-essentials/141332/226362 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead of Editorial Services.  Nikita: Hi everyone! This is Episode 6 in our series on MySQL, and today we're focusing on how to back up our MySQL instances. This is another two-parter and we've got Perside Foster back with us.    00:49 Lois: Perside is a MySQL Principal Solution Engineer at Oracle and she's here to share her insights on backup strategies and tools. In this episode, we'll be unpacking the types of backups available and discussing their pros and cons. Nikita: But first let's start right at the beginning. Perside, why is it essential for us to back up our databases? 01:10 Perside: The whole point of a database is to store and retrieve your business data, your intellectual property. When you back up your data, you are able to do disaster recovery so that your business can continue after some catastrophic event. You can recover from error and revert to a previous known good version of the data. You can migrate effectively from one system to another, or you can create replicas for load balancing or parallel system. You can retain data for archival purposes. Also, you can move large chunks of data to other systems, for example, to create a historical reporting application. And then you can create test environments for applications that are in development and that need real world test data. 02:10 Lois: Yes, and creating a robust backup strategy takes planning, doesn't it? Perside: As with any complex business critical process, there are challenges with coming up with a backup strategy that you can trust. This requires some careful planning. Any backup process needs to read the data. And in a production system, this will involve adding input/output operations to what might be an already busy system. The resources required might include memory or disk I/O operation and of course, you'll want to avoid downtime, so you might need to schedule the backup for a time when the system is not at peak usage. You'll also need to consider whether the backup is on network storage or some local storage so that you don't exceed limitations for those resources. It isn't enough just to schedule the backup. You'll also need to ensure that they succeed, which you can do with monitoring and consistency check. No backup is effective unless you can use it to restore your data, so you should also test your restore process regularly. If you have business requirements or regulatory commitments that control your data storage policies, you need to ensure your backup also align with those policies. Remember, every backup is a copy of your data at that moment in time. So it is subject to all of your data retention policies, just like your active data. 04:02 Nikita: Let's talk backup types. Perside, can you break them down for us? Perside: The first category is logical backup. A logical backup creates a script of SQL statements that will re-create the data structure and roles of the live database. Descript can be moved to another server as required. And because it's a script, it needs to be created by and executed on a running server. Because of this, the backup process takes up resources from the source server and is usually slower than a physical media backup.  04:45 Nikita: Ok… what's the next type? Perside: The next category is physical backup. This is a backup of the actual data file in the server. Bear in mind that the file copy process takes time, and if the database server is active during that time, then the later parts of the copy data will be inconsistent with those parts copied earlier. Ideally, the file must be stable during the backup so that the database state at the start of the copy process is consistent with the state at the end. If there is inconsistency in the data file, then MySQL detects that when the server starts up and it performs a crash recovery. From MySQL's perspective, there is no difference between a database backup copied from a running server and restarting a server after a crash. In each case, the data files were not saved in a consistent state and crash recovery can take a lot of time on large databases. 06:02 Lois: I see… how can MySQL Enterprise Backup help with this? Perside: MySQL Enterprise Backup has features that enable a consistent backup from a running server. If you create file system copies, either by copying the data files or by performing a file system snapshot, then you must either shut the server down before the copy and undergo crash recovery on the server that starts with those copied files. 06:35 Lois: And aside from logical and physical backups, are there other techniques to back up data? Perside: The binary log enables point-in-time recovery. You can enable replication in a couple of ways. If you start replication and then stop it at a particular time, the replica effectively contains a live backup of the data at the time that you stopped replication. You can also enable a defined replication lag so that the replica is always a known period of time behind the production database. You can also use transportable tablespaces, which are tables or sets of tables in a specific file that you can copy to another server. 07:34 AI is being used in nearly every industry…healthcare, manufacturing, retail, customer service, transportation, agriculture, you name it! And it's only going to get more prevalent and transformational in the future. It's no wonder that AI skills are the most sought-after by employers. If you're ready to dive in to AI, check out the OCI AI Foundations training and certification e that's available for free! It's the perfect starting point to build your AI knowledge. So, get going! Head over to mylearn.oracle.com to find out more. 08:14 Nikita: Welcome back! I want to return to the topic of crafting an effective backup strategy. Perside, any advice here? Perside: We can use the different backup types to come up with an effective backup strategy based on how we intend to restore the data. A full backup is a complete copy of the database at some point in time. This can take a lot of time to complete and to restore. An incremental backup contains only the changes since the last backup, as recorded in the binary log files. To restore an incremental backup, you must have restored the previous full backup and any incremental backups taken since then. For example, you might have four incremental backups taken after the last full backup. Each incremental backup contains only the changes since the previous backup. If you want to restore to the point at which you took the fourth incremental backup, then you must restore the full backup and each incremental backup in turn. A differential backup contains all changes since the last full backup. It contains only those portions of the database that are different from the full backup. Over time, the differential backup takes longer because it contains more changes. However, it is easier to restore because if you want to restore to the point at which you took a particular differential backup, you must restore the last full backup and only the differential backup that you require. You can ignore the intermediate differential backups. 10:13 Lois: Can you drill into the different types of backups and explain how each technique is used in various situations? Perside: One of the physical backup techniques is taking a snapshot of the storage medium. The advantages of a snapshot include its quickness. A snapshot is quick to create and restore. It is well-suited to situations where you need to quickly revert to a previous version of the database. For example, in a development environment. A storage snapshot is often a feature of the underlying file system. Linux supports logical volume management or LVM, and many storage area networks or network-attached storage platforms have native snapshot features. You can also use a storage snapshot to supplement a more scheduled logical backup structure. This way, the snapshot enables quick reversion to a previous type, and the logical backup can be used for other purposes, such as archiving or disaster recovery. 11:28 Nikita: Are there any downsides to using snapshots? Perside: First one includes issues with consistency. Because taking a snapshot is quick and does not cause a database performance hit, you might take the snapshot while the system is running. When you restore such a snapshot, MySQL must perform a crash recovery. If you want a consistent snapshot, you must shut down MySQL in advance. Another problem is that the snapshot is a copy of the file system and not of the database. So if you want to transfer it to another system, you must create a database backup from the storage. This adds step in time. A snapshot records the state of the disk at a specific point in time. Initially, the snapshot is practically empty. When a data page changes, the original version of that page is written to the snapshot. Over time, the snapshot storage grows as more data pages are modified. So multiple snapshots result in multiple writes whenever a snapshot data page is changed. To avoid performance deterioration, you should remove or release snapshots when they are no longer in use. Also, because snapshots are tied to the storage medium, they're not suited to moving backups between systems. 13:03 Lois: How about logical backups? How do we create those? Perside: The mysqldump utility has long been a standard way to create logical backups. It creates a script made up of the SQL statement that creates the data and structure in a database or server.  13:21 Nikita: Perside, what are the advantages and disadvantages of mysql dump? Perside: It is an excellent solution for preserving the database structure or for backing up small databases. Logical backups naturally require that the server is running. And they use system resources to produce the SQL statements, so they are less likely for very large databases. The output is a human-readable text file with SQL statements that you can edit as a text file. It can be managed by a source code management system. This allows you to maintain a known good version of the database structure, one that matches your application source code version, which can also include sample data. The mysqldump disadvantages are it needs to run against an active server. So if your production server is busy, you must take action to ensure a consistent backup. This requires locking tables or using the single transaction option, which can result in application delays as the backup completes in a consistent way. Mysqldump does not track changes since the last backup, so it has no way of recording only those rows that have changed. This means it's not suited to perform differential or incremental backups. The scripts must be executed against a running server, so it is slower to restore than using a data dump or physical backup. Additionally, if the database structure has indexes of foreign keys, these conditions must be checked and updated as the data is imported. You can disable these checks during the import but must handle any risks that come from doing so. Because the backup is nothing more than an SQL script, it is easy to restore. You can simply use the MySQL client or any other client tool that can process scripts.  15:46 Nikita: Is there an alternative tool for logical backups? Perside: MySQL Shell is another utility that supports logical backup and restore. Unlike mysqldump, it dumps data in a form that can be processed in parallel, which makes it much faster to use for larger data sets. This enables it to export to or import from remote storage where it can stream data without requiring the whole file before starting the input. It can process multiple chunks of imported data in parallel, and you can monitor progress as it completes. You can also pause import and resume later. For example, in the event of network outage. You can dump and restart table structure, including indexes and primary keys. The utilities in MySQL Shell are exposed through functions. The dumpInstance and dumpSchema utilities back up the whole server or specified schemas respectively. And loadDump is how you restore from such a dump. 17:07 Lois: Thanks for that rundown, Perside! This concludes our first part on MySQL backups. Next week, we'll take a look at advanced backup methods and the unique features of MySQL Enterprise Backup. Nikita: And if you want to learn more about everything we discussed today, head over to mylearn.oracle.com and explore the MySQL 8.4 Essentials course. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 17:37 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Picking up from Part 1, hosts Lois Houston and Nikita Abraham continue their deep dive into MySQL security with MySQL Solution Engineer Ravish Patel. In this episode, they focus on user authentication techniques and tools such as MySQL Enterprise Audit and MySQL Enterprise Firewall.   MySQL 8.4 Essentials: https://mylearn.oracle.com/ou/course/mysql-84-essentials/141332/226362 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative  podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi everyone! Last week, we began exploring MySQL security, covering regulatory compliance and common security threats.  Nikita: This week, we're continuing the conversation by digging deeper into MySQL's user authentication methods and taking a closer look at some powerful security tools in the MySQL Enterprise suite. 00:57 Lois: And we're joined once again by Ravish Patel, a MySQL Solution Engineer here at Oracle. Welcome, Ravish! How does user authentication work in MySQL? Ravish: MySQL authenticates users by storing account details in a system database. These accounts are authenticated with three elements, username and hostname commonly separated with an @ sign along with a password.  The account identifier has the username and host. The host identifier specifies where the user connects from. It specifies either a DNS hostname or an IP address. You can use a wild card as part of the hostname or IP address if you want to allow this username to connect from a range of hosts. If the host value is just the percent sign wildcard, then that username can connect from any host. Similarly, if you create the user account with an empty host, then the user can connect from any host.  01:55 Lois: Ravish, can MySQL Enterprise Edition integrate with an organization's existing accounts?  Ravish: MySQL Enterprise authentication integrates with existing authentication mechanisms in your infrastructure. This enables centralized account management, policies, and authentication based on group membership and assigned corporate roles, and MySQL supports a wide range of authentication plugins. If your organization uses Linux, you might already be familiar with PAM, also known as Pluggable Authentication Module. This is a standard interface in Linux and can be used to authenticate to MySQL. Kerberos is another widely used standard for granting authorization using a centralized service. The FIDO Alliance, short for Fast Identify Online, promotes an interface for passwordless authentication. This includes methods for authenticating with biometrics RUSB security tokens. And MySQL even supports logging into centralized authentication services that use LDAP, including having a dedicated plugin to connect to Windows domains. 03:05 Nikita: So, once users are authenticated, how does MySQL handle user authorization? Ravish: The MySQL privilege system uses the GRANT keyword. This grants some privilege X on some object Y to some user Z, and optionally gives you permission to grant the same privilege to others. These can be global administrative privileges that enable users to perform tasks at the server level, or they can be database-specific privileges that allow users to modify the structure or data within a database. 03:39 Lois: What about database privileges? Ravish: Database privileges can be fine-grained from the largest to the smallest. At the database level, you can permit users to create, alter, and delete whole databases. The same privileges apply at the table, view, index, and stored procedure levels. And in addition, you can control who can execute stored procedures and whether they do so with their own identity or with the privileges of the procedure's owner. For tables, you can control who can select, insert, update, and delete rows in those tables. You can even specify the column level, who can select, insert, and update data in those columns. Now, any privileged system carries with it the risk that you might forget an important password and lock yourself out. In MySQL, if you forget the password to the root account and don't have any other admin-level accounts, you will not be able to administer the MySQL server. 04:39 Nikita: Is there a way around this? Ravish: There is a way around this as long as you have physical access to the server that runs the MySQL process. If you launch the MySQL process with the --skip grant tables option, then MySQL will not load the privilege tables from the system database when it starts. This is clearly a dangerous thing to do, so MySQL also implicitly disables network access when you use that option to prevent users from connecting over the network. When you use this option, any client connection to MySQL succeeds and has root privileges. This means you should control who has shell access to the server during this time and you should restart the server or enable privileged system with the command flush privileges as soon as you have changed the root password. The privileges we have already discussed are built into MySQL and are always available. MySQL also makes use of dynamic privileges, which are privileges that are enabled at runtime and which can be granted once they are enabled.  In addition, plugins and components can define privileges that relate to features of those plugins. For example, the enterprise firewall plugin defines the firewall admin privilege and the audit admin privilege is defined by the enterprise audit plugin.  06:04 Are you working towards an Oracle Certification this year? Join us at one of our certification prep live events in the Oracle University Learning Community. Get insider tips from seasoned experts and learn from others who have already taken their certifications. Go to community.oracle.com/ou to jump-start your journey towards certification today! 06:28 Nikita: Welcome back! Ravish, I want to move on to MySQL Enterprise security tools. Could you start with MySQL Enterprise Audit? Ravish: MySQL Enterprise Audit is an extension available in Enterprise Edition that makes it easier to comply with regulations that require observability and control over who does what in your database servers. It provides visibility of connections, authentication, and individual operations. This is a necessary part of compliance with various regulations, including GDPR, NIS2, HIPAA, and so on. You can control who has access to the audited events so that the audits themselves are protected. As well as configuring what you audit, you can also configure rotation policies so that unmonitored audit logs don't fill up your storage space. The configuration can be performed while the server is running with minimal effect on production applications. You don't need to restart the server to enable or disable auditing or to change the filtering options. You can output the audit logs in either XML or JSON format, depending on how you want to perform further searching and processing. If you need it, you can compress the logs to save space and you can encrypt the logs to provide address protection of audited identities and data modifications. The extension is available either as a component or if you prefer, as the legacy plugin. 07:53 Lois: But how does it all work? Ravish: Well, first, as a DBA, you'll enable the audit plugin and attach it to your running server. You can then configure filters to audit your connections and queries and record who does what, when they do it, and so on. Then once the system is up and running, it audits whenever a user authenticates, accesses data, or even when they perform schema changes. The logs are recorded in whatever format that you have configured. You can then monitor the audited events at will with MySQL tools such as Workbench or with any software that can view and manipulate XML or JSON files. You can even configure Enterprise Audit to export the logs to an external Audit Vault, enabling collection, and archiving of audit information from all over your enterprise. In general, you won't audit every action on every server. You can configure filters to control what specific information ends up in the logs. 08:50 Nikita: Why is this sort of filtering necessary, Ravish? Ravish: As a DBA, this enables you to create a custom designed audit process to monitor things that you're really interested in. Rules can be general or very fine grained, which enables you to reduce the overall log size, reduces the performance impact on the database server and underlying storage, makes it easier to process the log file once you've gathered data, and filters are configured with the easily used JSON file format. 09:18 Nikita: So what information is audited? Ravish: You can see who did what, when they did it, what commands they use, and whether they succeeded. You can also see where they connected from, which can be useful when identifying man in the middle attacks or stolen credentials. The log also records any available client information, including software versions and information about the operating system and much more. 09:42 Lois: Can you tell us about MySQL Enterprise Firewall, which I understand is a specific tool to learn and protect the SQL statements that MySQL executes? Ravish: MySQL Enterprise Firewall can be enabled on MySQL Enterprise Edition with a plugin. It uses an allow list to set policies for acceptable queries. You can apply this allow list to either specific accounts or groups. Queries are protected in real time. Every query that executes is verified per server and checked to make sure that it conforms to query structures that are defined in the allow list. This makes it very useful to block SQL injection attacks. Only transactions that match well-formed queries in the allow list are permitted. So any attempt to inject other types of SQL statements are blocked. Not only does it block such statements, but it also sends an alert to the MySQL error log in real time. This gives you visibility on any security gaps in your applications. The Enterprise Firewall has a learning mode during which you can train the firewall to identify the correct sort of query. This makes it easy to create the allow list based on a known good workload that you can create during development before your application goes live. 10:59 Lois: Does MySQL Enterprise Firewall operate seamlessly and transparently with applications? Ravish: Your application simply submits queries as normal and the firewall monitors incoming queries with no application changes required. When you use the Enterprise Firewall, you don't need to change your application. It can submit statements as normal to the MySQL server. This adds an extra layer of protection in your applications without requiring any additional application code so that you can protect against malicious SQL injection attacks. This not only applies to your application, but also to any client that configured user runs. 11:37 Nikita: How does this firewall system work?  Ravish: When the application submits a SQL statement, the firewall verifies that the statement is in a form that matches the policy defined in the allow list before it passes to the server for execution.  It blocks any statement that is in a form that's outside of policy.  In many cases, a badly formed query can only be executed if there is some bug in the application's data validation. You can use the firewall's detection and alerting features to let when it blocks such a query, which will help you quickly detect such bugs, even when the firewall continues to block the malicious queries. 12:14 Lois: Can you take us through some of the encryption and masking features available in MySQL Enterprise Edition?  Ravish: Transparent data encryption is a great way to protect against physical security disclosure. If someone gains access to the database files on the file system through a vulnerability of the operating system, or even if you've had a laptop stolen, your data will still be protected. This is called Data at Rest Encryption. It protects not only the data rows in tablespaces, but also other locations that store some version of the data, such as undo logs, redo logs, binary logs and relay logs. It is a strong encryption using the AES 256 algorithm. Once we enable transparent data encryption, it is, of course, transparent to the client software, applications, and users. Applications continue to submit SQL statements, and the encryption and decryptions happen in flight. The application code does not need to change. All data types, table structure, and database names remain the same. It's even transparent to the DBAs. The same data types, table structure, and so on is still how the DBA interacts with the system while creating indexes, views, and procedures. In fact, DBAs don't even need to be in possession of any encryption keys to perform their admin tasks. It is entirely transparent. 13:32 Nikita: What kind of management is required for encryption? Ravish: There is, of course, some key management required at the outside. You must keep the keys safe and put policies in place so that you store and rotate keys effectively, and ensure that you can recover those keys in the event of some disaster. This key management integrates with common standards, including KMIP and KMS. 13:53 Lois: Before we close, I want to ask you about the role of data masking in MySQL. Ravish: Data masking is when we replace some part of the private information with a placeholder. You can mask portions of a string based on the string position using the letter X or some other character. You can also create a table that contains a dictionary of suitable replacement words and use that dictionary to mask values in your data. There are specific functions that work with known formats of data, for example, social security numbers as used in the United States, national insurance numbers from the United Kingdom, and Canadian social insurance numbers. You can also mask various account numbers, such as primary account numbers like credit cards or IBAN numbers as used in the European Bank system. There are also functions to generate random values, which can be useful in test databases. This might be a random number within some range, or an email address, or a compliant credit card number, or social security number. You can also create random information using the dictionary table that contains suitable example values. 14:58 Nikita: Thank you, Ravish, for taking us through MySQL security. We really cannot overstate the importance of this, especially in today's data-driven world.  Lois: That's right, Niki. Cyber threats are increasingly sophisticated these days. You really have to be on your toes when it comes to security. If you're interested in learning more about this, the MySQL 8.4 Essentials course on mylearn.oracle.com is a great next step.  Nikita: We'd also love to hear your thoughts on our podcast so please feel free to share your comments, suggestions, or questions by emailing us at ou-podcast_ww@oracle.com. That's ou-podcast_ww@oracle.com. In our next episode, we'll journey into the world of MySQL backups. Until then, this is Nikita Abraham… Nikita: And Lois Houston, signing off! 15:51 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Security takes center stage in this episode as Lois Houston and Nikita Abraham are joined by MySQL Solution Engineer Ravish Patel. Together, they explore MySQL's security features, addressing key topics like regulatory compliance.   Ravish also shares insights on protecting data through encryption, activity monitoring, and access control to guard against threats like SQL injection and malware.   MySQL 8.4 Essentials: https://mylearn.oracle.com/ou/course/mysql-84-essentials/141332/226362 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative  podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me today is Nikita Abraham, Team Lead of Editorial Services. Nikita: Hey everyone! In our last episode, we took a look at MySQL database design. Today is the first of a two-part episode on MySQL security.  Lois: In Part 1, we'll discuss how MySQL supports regulatory compliance and how to spot and handle common security risks.  00:55 Nikita: Joining us today is Ravish Patel, a MySQL Solution Engineer at Oracle. Hi Ravish! Let's start by talking about how MySQL supports regulatory compliance. 01:06 Ravish: Some of the most important international regulations that we have surrounding data and organizations include the GDPR, HIPAA, Sarbanes-Oxley, the UK Data Protection Act, and the NIS2. Although each regulatory framework differs in the details, in general, you must be able to comply with certain key requirements and all of which are enabled by MySQL. First, you must be able to monitor user activity on the system, which includes keeping track of when new users are created, when the schema changes, and when backups are taken and used. You must protect data, for example, by ensuring that databases that are stored on disk are encrypted at REST and ensuring that only authorized users have privileges to access and modify the data. You must have the appropriate retention policies in place for your data, ensuring that backups are held securely and used only for the purpose intended. You must be able to audit access to the data so that you can trace which users gained access to records or when they were modified. All of these facilities are available in MySQL, either as part of the core community edition features or made available through enterprise features. 02:21 Lois: What kind of risks might we encounter, Ravish, and how can we address them? Ravish: As your system grows in complexity, you're likely going to have more risks associated with it. Some of those risks are associated with the human factors that come with any computer system. These might be errors that are introduced when people perform work on the system, either administrative work on the environment or database or work that developers and testers perform when working on a changing system. You might even have malicious users trying to exploit the system or good faith users or support staff who make changes without proper consideration or protection from knock-on effects. At the foundation are the necessary components of the system, each of which might be vulnerable to human error or malicious actors. Every piece of the system exposes possible risks, whether that's the application presented to users, the underlying database, the operating system or network that it works on, or processes such as backups that place copies of your data in other locations. More complex environments add more risks. High availability architectures multiply the number of active systems. Consolidating multiple application databases on a single server exposes every database to multiple vectors for bugs and human error. Older, less well supported applications might give more challenges for maintenance. Engaging external contractors might reduce your control over authorized users. And working in the cloud can increase your network footprint and your reliance on external vendors.  03:53 Nikita: What are risks that specifically impact the database? Ravish: The database server configuration might not be optimal. And this can be changed by users with proper access. To mitigate this risk, you might enable version control of the configuration files and ensure that only certain users are authorized. Application and administrator accounts might have more data privileges than required, which adds risk of human error or malicious behavior. To mitigate this, you should ensure that users are only granted necessary permissions. In particular, structural modifications and administrative tasks might be more widely accessible than desired. Not every developer needs full administrative rights on a database. And certainly, an application should not have such privileges. You should limit administrative privileges only to those users who need that authorization. 04:45 Nikita: Okay, quick question, Ravish. How do authentication and password security fit into this picture? Ravish: Authentication is often a weak point. And password security is one of the most common issues in large applications. Ensure that you have strong password policies in place. And consider using authentication mechanisms that don't solely rely on passwords, such as pass-through authentication or multifactor authentication. 05:11 Lois: So, it sounds like auditing operations are a critical part of this process, right? Ravish: When something bad happens, you can only repair it or learn from it if you know exactly what has happened and how. You should ensure that you audit key operations so you can recover from error or malicious actions. If a developer laptop is lost or stolen or someone gains access to an underlying operating system, then your data might become vulnerable. You can mitigate this by encrypting your data in place.  This also applies to backups and, where possible, securing the connection between your application and the database to encrypt data in flight. 05:54 Did you know that Oracle University offers free courses on Oracle Cloud Infrastructure? You'll find training on everything from multicloud, database, networking, and security to artificial intelligence and machine learning, all free for our subscribers. So, what are you waiting for? Pick a topic, head over to mylearn.oracle.com and get started. 06:18 Nikita: Welcome back! Before the break, we touched on the importance of auditing. Now, Ravish, what role does encryption play in securing these operations? Ravish: Encryption is only useful if the keys are secure. Make sure to keep your encryption assets secure, perhaps by using a key vault. Every backup that you take contains a copy of your data. If these backups are not kept securely, then you are at risk, just as if your database wasn't secure. So keep your backups encrypted. 06:47 Lois: From what we've covered so far, it's clear that monitoring is essential for database security. Is that right? Ravish: Without monitoring, you can't track what happens on an ongoing basis. For example, you will not be aware of a denial-of-service attack until the application slows down or becomes unavailable. If you implement monitoring, you can identify a compromised user account or unusual query traffic as it happens. A poorly coded application might enable queries that do more than they should. A database firewall can be configured to permit only queries that conform to a specific pattern. 07:24 Nikita: There are so many potential types of attacks out there, right? Could you tell us about some specific ones, like SQL injection and buffer overflow attacks? Ravish: A SQL injection attack is a particular form of attack that modifies a SQL command to inject a different command to the one that was intended by the developer. You can configure an allow list in a database firewall to block such queries and perform a comprehensive input validation inside the application so that such queries cannot be inserted. A buffer overflow attack attempts to input more data than can fit in the appropriate memory location. These are usually possible when there is an unpatched bug in the application or even in the database or operating system software. Validation and the database firewall can catch this sort of attack before it even hits the database. And frequent patching of the platforms can mitigate risks that come from unpatched bugs. Malicious acts from inside the organization might also be possible. So good access control and authorization can prevent this. And monitoring and auditing can detect it if it occurs. 08:33 Lois: What about brute force attacks? How do they work? Ravish: A brute force attack is when someone tries passwords repeatedly until they find the correct one. MySQL can lock out an account if there have been too many incorrect attempts. Someone who has access to the physical network on which the application and database communicate can monitor or eavesdrop that network. However, if you encrypt the communications in flight, perhaps by using TLS or SSL connections, then that communication cannot be monitored. 09:04 Nikita: How do the more common threats like malware, Trojan horses, and ransomware impact database security? Ravish: Malware, ransomware, and Trojan horses can be a problem if they get to the server platforms or if client systems are compromised and have too much permissions. If the account that is compromised has only limited access and if the database is encrypted in place, then you can minimize the risks associated even if such an event occurs. There are also several risks directly associated with people who want to do the harm. So it's vital to protect personal information from any kind of disclosure, particularly sensitive information, such as credit card numbers. Encryption and access control can protect against this. 09:49 Lois: And then there are denial-of-service and spoofing attacks as well, right? How can we prevent those? Ravish: A denial-of-service attack prevents users from accessing the system. You can prevent any single user from performing too many queries by setting resource users limits. And you can limit the total number of connections as well. Sometimes, a user might gain access to a privileged level that is not appropriate. Password protection, multifactor authentication, and proper access control will protect against this. And auditing will help you discover if it has occurred. A spoofing attack is when an attacker intercepts and uses information to authenticate a user. This can be mitigated with strong access control and password policies. An attacker might attempt to modify or delete data or even auditing information. Again, this can be mitigated with tighter access controls and caught with monitoring and auditing. If the attack is successful, you can recover from it easily if you have a strong backup strategy in place. 10:50 Nikita: Ravish, are there any overarching best practices for keeping a database secure? Ravish: The MySQL installation itself should be kept up-to-date. This is the easiest if you install from a package manager on Windows or Linux. Your authentication systems should be kept strong with password policies or additional authentication systems that supplement or replace passwords entirely. Authorization should be kept tightly controlled by minimizing the number of active accounts and ensuring that those accounts have only the minimal privileges. You should control and monitor changes on the system. You can limit such changes with the database firewall and with tight access controls and observe changes with monitoring, auditing, and logging. Data encryption is also necessary to protect data from disclosure. MySQL supports encryption in place with Transparent Data Encryption, also known as TDE, and a variety of encryption functions and features. And you can encrypt data in flight with SSL or TLS. And of course, it's not just about the database itself but how it's used in the wider enterprise. You should ensure that replicas are secure and that your disaster recovery procedures do not open up to additional risks. And keep your backups encrypted. 12:06 Lois: Is there anything else we should keep in mind as part of these best practices? Ravish: The database environment is also worth paying attention to. The operating system and network should be as secure as you can keep them.  You should keep your platform software patched so that you are protected from known exploits caused by bugs. If your operating system has hardening guidelines, you should always follow those. And the Center of Internet Security maintains a set of benchmarks with configuration recommendations for many products designed to protect against threats. 12:38 Nikita: And that's a wrap on Part 1! Thank you, Ravish, for guiding us through MySQL's role in ensuring compliance and telling us about the various types of attacks. If you want to dive deeper into these topics, head over to mylearn.oracle.com to explore the MySQL 8.4 Essentials course. Lois: In our next episode, we'll continue to explore how user authentication works in MySQL and look at a few interesting MySQL Enterprise security tools that are available. Until then, this is Lois Houston…  Nikita: And Nikita Abraham, signing off! 13:12 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Explore the essentials of MySQL database design with Lois Houston and Nikita Abraham, who team up with MySQL expert Perside Foster to discuss key storage concepts, transaction support in InnoDB, and ACID compliance. You'll also get tips on choosing the right data types, optimizing queries with indexing, and boosting performance with partitioning.   MySQL 8.4 Essentials: https://mylearn.oracle.com/ou/course/mysql-84-essentials/141332/226362 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative  podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Lois: Hello and welcome to the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me today is Nikita Abraham, Team Lead of Editorial Services. Nikita: Hi everyone! Last week, we looked at installing MySQL and in today's episode, we're going to focus on MySQL database design. Lois: That's right, Niki. Database design is the backbone of any MySQL environment. In this episode, we'll walk you through how to structure your data to ensure smooth performance and scalability right from the start.   00:58 Nikita: And to help us with this, we have Perside Foster joining us again. Perside is a MySQL Principal Solution Engineer at Oracle. Hi Perside, let's start with how MySQL handles data storage on the file system. Can you walk us through the architecture? Perside: In the MySQL architecture, the storage engine layer is part of the server process. Logically speaking, it comes between the parts of the server responsible for inputting, parsing, and optimizing SQL and the underlying file systems. The standard storage engine in MySQL is called InnoDB. But other storage engines are also available. InnoDB supports many of the features that are required by a production database system. Other storage engines have different sets of features. For example, MyISAM is a basic fast storage engine but has fewer reliability features. NDB Cluster is a scalable distributed storage engine. It runs on multiple nodes and uses additional software to manage the cluster.  02:21 Lois: Hi Perside! Going back to InnoDB, what kind of features does InnoDB offer? Perside: The storage engine supports many concurrent users. It also keeps their changes separate from each other. One way it achieves this is by supporting transactions. Transactions allows users to make changes that can be rolled back if necessary and prevent other users from seeing those changes until they are committed or saved persistently. The storage engine also enables referential integrity. This is to make sure that data in a dependent table refers only to valid source data. For example, you cannot insert an order for a customer that does not exist. It stores raw data on disk in a B-tree structure and uses fast algorithms to insert rows in the correct place. This is done so that the data can be retrieved quickly. It uses a similar method to store indexes. This allows you to run queries based on a sort order that is different from the row's natural order. InnoDB has its own buffer pool. This is a memory cache that stores recently accessed data. And as a result, queries on active data are much faster than queries that read from the disk. InnoDB also has performance features such as multithreading and bulk insert optimization. 04:13 Lois: So, would you say InnoDB is generally the best option? Perside: When you install MySQL, the standard storage engine is InnoDB. This is generally the best choice for production workloads that need both reliability and high performance. It supports transaction syntax, such as commit and rollback, and is fully ACID compliant. 04:41 Nikita: To clarify, ACID stands for Atomicity, Consistency, Isolation, and Durability. But could you explain what that means for anyone who might be new to the term? Perside: ACID stands for atomic. This means your transaction can contain multiple statements, but the transaction as a whole is treated as one change that succeeds or fails. Consistent means that transactions move the system from one consistent state to another. Isolated means that changes made during a transaction are isolated from other users until that transaction completes. And durable means that the server ensures that the transaction is persisted or written to disk once it completes. 05:38 Lois: Thanks for breaking that down for us, Perside. Could you tell us about the data encryption and security features supported by InnoDB? Perside: InnoDB supports data encryption, which keeps your data secure on the disk. It also supports compression, which saves space at the cost of some extra CPU usage. You can configure an InnoDB cluster of multiple MySQL server nodes across multiple hosts to enable high availability. Transaction support is a key part of any reliable database, particularly when multiple concurrent users can change data. By default, each statement commits automatically so that you don't have to type commit every time you update a row. You can open a transaction with the statement START TRANSACTION or BEGIN, which is synonymous. 06:42 Nikita: Perside, what exactly do the terms "schema" and "database" mean in the context of MySQL, and how do they relate to the storage structure of tables and system-level information? Perside: Schema and database both refer to collections of tables and other objects. In some platform, a schema might contain databases. In MySQL, the word schema is a synonym for database. In InnoDB and some other storage engines, each database maps to a directory on the file system, typically in the data directory. Each table has rows data stored in a file. In InnoDB, this file is the InnoDB tablespace, although you can choose to store tables in other tablespaces. MySQL uses some databases to store or present system-level information. The MySQL and information schema databases are used to store and present structural information about the server, including authentication settings and table metadata. You can query performance metrics from the performance schema and sys databases. If you have configured a highly available InnoDB cluster, you can examine its configuration from the MySQL InnoDB cluster metadata database. 08:21 Lois: What kind of data types does MySQL support? Perside: MySQL supports a number of data types with special characteristics. BLOB stands for Binary Large Object Block. Columns that specify this type can contain large chunks of binary data. For example, JPG pictures or MP3 audio files. You can further specify the amount of storage required by specifying the subtype-- for example, TINYBLOB or LONGBLOB. Similarly, you can store large amounts of text data in TEXT, TINYTEXT, and so on. These types, BLOB and TEXT, share the same characteristic, that they are not stored in the same location as other data from the same row. This is to improve performance because many queries against the table do not query BLOB or TEXT data contained within the table. MySQL supports geographic or spatial data and queries on that data. These include ways to represent points, lines, polygons, and collections of such elements. The JSON data type enables you to use MySQL as a document store. A column of this type can contain complete JSON documents in each row. And MySQL has several functions that enable querying and searching for values within such documents.  10:11 Adopting a multicloud strategy is a big step towards future-proofing your business and we're here to help you navigate this complex landscape. With our suite of courses, you'll gain insights into network connectivity, security protocols, and the considerations of working across different cloud platforms. Start your journey to multicloud today by visiting mylearn.oracle.com. 10:38 Nikita: Welcome back. Perside, how do indexes improve the performance of MySQL queries? Perside: Indexes make it easier for MySQL to find specific rows. This doesn't just speed up queries, but also ensures that newly inserted rows are placed in the best position in the data file so that future queries will findthem quickly. 11:03 Nikita: And how do these indexes work exactly? Perside: Indexes work by storing the raw data or a subset of the raw data in some defined order. An index can be ordered on some non-unique value, such as a person's name. Or you can create an index on some value that must be unique within the table, such as an ID. The primary index, sometimes called a clustered index, is the complete table data stored on a unique value called a Primary Key. 11:38 Lois: Ok. And what types of indices are supported by InnoDB? Perside: InnoDB supports multiple index types. Raw data in most secondary indexes are stored in a BTREE structure. This stores data in specific buckets based on the index key using fixed-size data pages. HASH indexes are supported by some storage engines, including the memory storage engine. InnoDB has an adaptive HASH feature, which kicks in automatically for small tables and workloads that benefits from them. Spatial data can be indexed using the RTREE structure.  12:25 Nikita: What are some best practices we should follow when working with indexes in MySQL? Perside: First, you should create a Primary Key for each table. This value is unique for each row and is used to order the row data. InnoDB doesn't require that tables have an explicit Primary Key, but if you don't set one, it creates a hidden Primary Key. Each secondary index is a portion of the data ordered by some other column. And internally, each index entry uses the Primary Key as a lookup back to the rest of the row. If your Primary Key is large or complex, this increases the storage requirement of each index. And every time you modify a row, MySQL must update every affected index in the background. The more indexes you have on a table, the slower every insert operation will be. This means that you should only create indexes that improve query performance for your specific workload. The sys schema in MySQL Enterprise Monitor have features to identify indexes that are unused. Use prefix and compound keys to reduce indexes. A prefix key contains only the first part of a string. This can be particularly useful when you have large amounts of text in an index key and want to index based on the first few characters. A compound key contains multiple columns, for example, last name and first name. This also speeds up queries where you're looking for only those values because the secondary index can fulfill the query without requiring a lookup back to the primary indexes.  14:35 Lois: Before we let you go, can you explain what table partitioning is? Perside: Table partitioning is enabled by using a plugin. When you partition a table, you divide its content according to certain rules. You might store portions of the table based on the range of values in a column. For example, storing all sales for 2024 in a single partition. A partition based on a list enables you to store rows with specific values in the partition column. When you partition by hash or key, you distribute rows somewhat evenly between partitions. This means that you can distribute a large table across multiple disks, or you can place more frequently accessed data on faster storage. Explain works with partitioning. Simply prefix any query that uses partition data, and the output shows information about how the optimizer will use the partition. Partitioning is one of the features that is only fully supported in Enterprise Edition. 15:57 Lois: Perside, thank you so much for joining us today. In our next episode, we'll dive deep into MySQL security. Nikita: And if you want to learn more about what we discussed today, visit mylearn.oracle.com and search for the MySQL 8.4: Essentials course. Until next week, this is Nikita Abraham… Lois: And Lois Houston signing off! 16:18 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, Lois Houston and Nikita Abraham discuss the basics of MySQL installation with MySQL expert Perside Foster. Perside covers every key step, from preparing your environment and selecting the right software, to installing MySQL, setting up secure initial user accounts, configuring the system, and managing updates efficiently.   MySQL 8.4 Essentials: https://mylearn.oracle.com/ou/course/mysql-84-essentials/141332/226362 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative  podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Nikita: Welcome back to another episode of the Oracle University Podcast. I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and I'm joined by Lois Houston, Director of Innovation Programs. Lois: Hi everyone! In our last episode, we spoke about Oracle MySQL ecosystem and its various components. We also discussed licensing, security, and some key tools. What's on the agenda for today, Niki? 00:52 Nikita: Well Lois, today, we're going beyond tools and features to talk about installing MySQL. Whether you're setting up MySQL for the first time or looking to understand its internal structure a little better, this episode will be a valuable guide.  Lois: And we're lucky to have Perside Foster back with us. Perside is a MySQL Principal Solution Engineer at Oracle. Hi Perside! Say I wanted to get started and install MySQL. What factors should I keep in mind before I do that?  01:23 Perside: The first thing to consider is the environment for the database server. MySQL is supported on many different Linux distributions. You can also run it on Windows or Apple macOS.  You can run MySQL on a variety of host platforms. You can use dedicated servers in a server room or virtual machines in a data center. Developers might prefer to deploy on Docker or Kubernetes containers. And don't forget, you can deploy HeatWave, the MySQL cloud version, in many different clouds. MySQL has great multithreading capability. It also has support for Non-Uniform Memory Access or NUMA. This is particularly important if you run large systems with hundreds of concurrent connections. MySQL storage engine, InnoDB, makes effective use of your available memory. It stores your active data in a buffer pool. This greatly improves access time compared to reading straight from disk. Of course, SSDs and other solid state media are much faster than hard disks. But don't forget, MySQL can make full use of that performance benefit too. Redundancy is very important for the MySQL server. Hardware with redundant power supply, storage media, and network connections can make all the difference to your uptime. Without redundancy, a single point of failure will bring down the server if it fails. 03:26 Nikita: Got it. Perside, from where can I download the different editions of MySQL? Perside: Our most popular software is the MySQL Community Edition. It is available at no cost for mysql.com for many platforms. This version is why MySQL is the most popular database for web application. And it is also open source. MySQL Enterprise Edition is the commercial edition. It is fully supported by Oracle. You can get it from support.oracle.com as an Oracle customer. If you want to try out the enterprise features but are not yet a customer, you can get the latest version of MySQL as a trial edition from edelivery.oracle.com. Because MySQL is open source, you can get the source code from either mysql.com or GitHub. Most people don't need the source. But any developer who wants to modify the code or even contribute back to the project are welcome to do so.  04:43 Lois: Perside, can you walk us through MySQL's release model? Perside: This is divided into LTS and Innovation releases, each with a different target audience. LTS stands for long-term support. MySQL 8.4 is an LTS release and will be supported for several years. LTS releases are feature-stable. When you install an LTS release, you can apply future bug fixes and security patches without changing any behavior in the product. The bug fixes and security patches are designed to be backward compatible. This means you can upgrade easily from previous releases. LTS releases come every two years. This allows you to maintain a stable system without having to change your underlying application too frequently. You will not be forced to upgrade after two years. You can continue to enjoy support for an LTS release for up to eight years. Along with LTS releases, we also have Innovation releases. These contain the latest leading-edge features that are developed even in the middle of an LTS cycle. You can upgrade from LTS to Innovation and back again, depending on which features you require in your application. Innovation releases have a much more rapid cadence. You can get the latest features every quarter. This means Innovation releases are supported only for their specific release. So, if you're on the Innovation track, you must upgrade more frequently. All editions of MySQL are shipped as both LTS and Innovation releases. This includes the self-managed editions and also HeatWave in the cloud. You can treat both LTS and Innovation releases as production-ready. This means they are generally available releases. Innovation does not mean beta quality software. You get the same quality support from Oracle whether you're using LTS or Innovative software. The MySQL client software and other tools will operate with both LTS and innovation releases.  07:43 Nikita: What are connectors in the context of MySQL? Perside: Connectors are the language-specific software component that connects your application to MySQL. You should use the latest version of connectors. Connectors are also production-ready, generally available software. They will work with any version of MySQL that is supported at the time of the connector's release. 08:12 Nikita: How does MySQL integrate with Docker and other container platforms? Perside: You might already be familiar with the Docker store. It is used for getting containerized images of software. As an Oracle customer, you might be familiar with My Oracle Support. It provides support and updates for all supported Oracle software in patches.  MySQL works well with virtualization and container platform, including Docker. You can get images from the Community Edition on Docker Hub. If you are an Enterprise Edition customer, you can get images from the Docker store for MySQL Oracle Support or from Oracle container's registry.  09:04 Lois: What resources are available for someone who wants to know more about MySQL? Perside: MySQL has detailed documentation. You should familiarize yourself with the documentation as you prepare to install MySQL. The reference manual for both Community and Enterprise editions are available at the Developer Zone at dev.mysql.com. Oracle customers also have access to the knowledge base at support.oracle.com. It contains support information on use cases and reference architectures. The product team regularly posts announcements and technical articles to several blogs. These blogs often contain pre-release announcements of upcoming features to help you prepare for your next project. Also, you'll find deep dives into technical topics and complex problems that MySQL solves. This includes some problems specific to highly available architecture. We also feature individual blogs from high profile members of our team. These include the MySQL Community evangelist lefred. He posts about upcoming events and interesting features. Also, Dimitri Kravchuk offers blogs that provide deep dives into performance. 10:53 Nikita: Ok, now that I have all this information and am prepped and ready, how do I actually install MySQL on my operating system? What's the process like? Perside: You can install MySQL on various operating system, depending on your needs. These might include several distributions of Linux or UNIX, Windows, Mac OS, Oracle Linux based on the Unbreakable Enterprise Kernel, Solaris, and freeBSD. As always, the MySQL documentation provides full details on supported operating system. It also provides the specific installation steps for each of the operating system. Plus, it tells you how to perform the initial configuration and further administrative steps.  If you're installing on Windows, you have a couple of options. First, the MySQL Installer utility is the easiest way to install MySQL. It installs MySQL and performs the initial configuration based on options that you choose at installation time. It includes not only the MySQL server, but also the most important connectors, the MySQL Shell Client, MySQL Workbench Client with user interface and common utilities for troubleshooting and administration. It also installs several sample databases and models and documentation. It's the easiest way to install MySQL because it uses an installation wizard. It lets you select your installation target location, what components to install, and other options. 12:47 Lois: But what if I want to have more control?  Perside: For more control over your installation, you can install MySQL from the binary zip archive. This does not include sample or supporting tools and connectors, but only contains the application's binaries, which you can install anywhere you want. This means that the initial configuration is not performed by selecting an option through a wizard. Instead, you must configure the Windows service and MySQL configuration file yourself. Linux installation is more varied. This is because of the different distribution and also because of its terms of flexibility. On many distributions of Linux, you can use the package manager native to that distribution. For example, you can use the yum package manager in all Oracle Linux to install RPM files. You can also use a binary archive to install only the files. To decide which method you want to use, it's based on several factors. How much you know about MySQL files and configuration and the operating system on which you're going to do the installation? Any applicable standard or operating procedures within your own company's IT infrastructure, how much control do you need over this installation and how flexible a method do you need? For example, the RPM package for Oracle Linux, it installs the file in specific locations and with a specific service, MySQL user account. 14:54 Transform the way you work with Oracle Database 23ai! This cutting-edge technology brings the power of AI directly to your data, making it easier to build powerful applications and manage critical workloads. Want to learn more about Database 23ai? Visit mylearn.oracle.com to pick from our range of courses and enroll today! 15:18 Nikita: Welcome back! Is there a way for me to extend the functionality of MySQL beyond its default capabilities? Perside: Much of MySQL's behavior is standard and always exists when you install the server. However, you can configure some additional behaviors by extending MySQL with plugins or components. Plugins operate closely with the server and by calling APIs exposed by the server, they add features by providing extra functions or variables. Not only do they add variables, they can also interact with the servers on global variables and functions. That makes them work as if they are dynamically loadable parts of the server itself. Components also extend functionality, but they are separate from the server and extend its functionality through a service-based architecture. You can also extend MySQL in other ways-- by creating stored procedures, triggers, and functions with standard SQL and MySQL extensions to that language, or by creating external dynamically loaded user-defined functions. 16:49 Lois: Perside, can we talk about the initial user accounts? Perside: A MySQL account identifier is more than just a username and password. It consists of three elements, two that identify the account, and one that is used for authentication. The three elements are the username, it's used to log in from the client; the hostname element, it identifies a computer or set of computers; and the password, it must be provided to gain access to MySQL. The hostname is a part of the account identifier that controls where the user can log in. It is typically a DNS computer name or an IP address. You can use a wildcard, which is the percentage sign to allow the name user to log in from any connected host, or you can use the wildcard as part of an IP address to allow login from a limited range of IP addresses. 17:58 Nikita: So, what happens when I install MySQL on my computer? Perside: When you first install MySQL on your computer, it installs several system accounts. The only user account that you can log in to is the administrative account. That's called the root account. Depending on the installation method that you use, you'll either see the initial root password on the console as you install the server, or you can read it from the log file. For security reasons, the password of a new account, such as the root account must change. MySQL prevents you from executing any other operation with that account until you have changed the password.  18:46 Lois: What are the system requirements for installing and running MySQL? Perside: The MySQL service must run as a system-level user. Each operating system has its own method for creating such a user. All operating system follows the same general principles. However, when using the MySQL installer on Windows or the RPM package installation on Oracle Linux, each installation process creates and configure the system-level user. 19:22 Lois: Perside, since MySQL is always evolving, how do I upgrade it when newer versions become available?  Perside: When you upgrade MySQL, you have to bring the server down so that the upgrade process can replace all of the relevant binary executable files. And if necessary, update the data and configuration to suit the new software. The safest thing to do is to back up your whole MySQL environment. This includes not only your data in the files, such as binaries and configuration files, but also logical elements, including triggers, stored procedures, user configuration, and anything else that's required to rebuild your system. The upgrade process gives you two main options. An in-place upgrade uses your existing data directory. After you shut down your MySQL server process, you either replace the package or binaries with new versions, or you install the new binary executables in a new location and point your symbolic links to this new location. The server process detects that the data directory belongs to an earlier version and performs any required upgrade checks. 20:46 Lois: Thank you, Perside, for taking us through the practical aspects of using MySQL. If you want to learn about the MySQL architecture, visit mylearn.oracle.com and search for the MySQL 8.4: Essentials course.  Nikita: Before you go, we wanted to take a minute to thank you for taking the Oracle University Podcast survey that we put out at the end of last year. Your insights were invaluable and will help shape our future episodes. Lois: And if you missed taking the survey but have feedback to share, you can write to us at ou-podcast_ww@oracle.com. That's ou-podcast_ww@oracle.com. We'd love to hear from you. Join us next week for a discussion on MySQL database design. Until then, this is Lois Houston… Nikita: And Nikita Abraham signing off! 21:45 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Join hosts Lois Houston and Nikita Abraham as they kick off a new season exploring the world of MySQL 8.4. Together with Perside Foster, a MySQL Principal Solution Engineer, they break down the fundamentals of MySQL, its wide range of applications, and why it's so popular among developers and database administrators. This episode also covers key topics like licensing options, support services, and the various tools, features, and plugins available in MySQL Enterprise Edition.   ------------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative  podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Lois: Hello and welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead: Editorial Services. Nikita: Happy New Year, everyone! Thank you for joining us as we begin a new season of the podcast, this time focused on the basics of MySQL 8.4. If you're a database administrator or want to become one, this is definitely for you. It's also great for developers working with data-driven apps or IT professionals handling MySQL installs, configurations, and support. 01:03 Lois: That's right, Niki. Throughout the season, we'll be delving into MySQL Enterprise Edition and covering a range of topics, including installation, security, backups, and even MySQL HeatWave on Oracle Cloud.  Nikita: Today, we're going to discuss the Oracle MySQL ecosystem and its various components. We'll start by covering the fundamentals of MySQL and the different licenses that are available. Then, we'll explore the key tools and features to boost data security and performance. Plus, we'll talk a little bit about MySQL HeatWave, which is the cloud version of MySQL.  01:39 Lois: To take us through all of this, we've got Perside Foster with us today. Perside is a MySQL Principal Solution Engineer at Oracle. Hi Perside! For anyone new to MySQL, can you explain what it is and why it's so widely used? Perside: MySQL is a relational database management system that organizes data into structured tables, rows, and columns for efficient programming and data management. MySQL is transactional by nature. When storing and managing data, actions such as selecting, inserting, updating, or deleting are required. MySQL groups these actions into a transaction. The transaction is saved only if every part completes successfully. 02:29 Lois: Now, how does MySQL work under the hood? Perside: MySQL is a high-performance database that uses its default storage engine, known as InnoDB. InnoDB helps MySQL handle complex operations and large data volumes smoothly. 02:49 Nikita: For the unversed, what are some day-to-day applications of MySQL? How is it used in the real world? Perside: MySQL works well with online transaction processing workloads. It handles transactions quickly and manages large volumes of transaction at once. OLTP, with low latency and high throughput, makes MySQL ideal for high-speed environments like banking or online shopping. MySQL not only stores data but also replicates it from a main server to several replicas. 03:31 Nikita: That's impressive! And what are the benefits of using MySQL?  Perside: It improves data availability and load balancing, which is crucial for businesses that need up-to-date information. MySQL replication supports read scale-out by distributing queries across servers, which increases high availability. MySQL is the most popular database on the web. 04:00 Lois: And why is that? What makes it so popular? What sets it apart from the other database management systems? Perside: First, it is a relational database management system that supports SQL. It also works as a document store, enabling the creation of both SQL and NoSQL applications without the need for separate NoSQL databases. Additionally, MySQL offers advanced security features to protect data integrity and privacy. It also uses tablespaces for better disk space management. This gives database administrators total control over their data storage. MySQL is simple, solid in its reliability, and secure by design. It is easy to use and ideal for both beginners and professionals. MySQL is proven at scale by efficiently handling large data volumes and high transaction rates. MySQL is also open source. This means anyone can download and use it for free. Users can modify the MySQL software to meet their needs. However, it is governed by the GNU General Public License, or GPL. GPL outlines specific rules for its use. MySQL offers two major editions. For developers and small teams, the Community Edition is available for free and includes all of the core features needed. For large enterprises, the Commercial Edition provides advanced features, management tools, and dedicated technical support. 05:58 Nikita: Ok. Let's shift focus to licensing. Who is it useful for?  Perside: MySQL licensing is essential for independent software vendors. They're called ISVs. And original manufacturers, they're called OEMs. This is because these companies often incorporate MySQL code into their software products or hardware system to boost the functionality and performance of their product. MySQL licensing is equally important for value-added resellers. We call those VARs. And also, it's important for other distributors. These groups bundle MySQL with other commercially licensed software to sell as part of their product offering. The GPL v.2 license might suit Open Source projects that distribute their products under that license.   07:02 Lois: But what if some independent software vendors, original manufacturers, or value-add resellers don't want to create Open Source products. They don't want their source to be publicly available and they want to keep it private? What happens then? Perside: This is why Oracle provides a commercial licensing option. This license allows businesses to use MySQL in their products without having to disclose their source code as required by GPL v2. 07:33 Nikita: I want to bring up the robust support services that are available for MySQL Enterprise. What can we expect in terms of support, Perside?  Perside: MySQL Enterprise Support provides direct access to the MySQL Support team. This team consists of experienced MySQL developers, who are experts in databases. They understand the issues and challenges their customers face because they, too, have personally tackled these issues and challenges. This support service operates globally and is available in 29 languages. So no matter where customers are located, Oracle Support provides assistance, most likely in their preferred language. MySQL Enterprise Support offers regular updates and hot fixes to ensure that the MySQL customer systems stays current with the latest improvements and security patches. MySQL Support is available 24 hours a day, 7 days a week. This ensures that whenever there is an issue, Oracle Support can provide the needed help without any delay. There are no restrictions on how many times customers can receive help from the team because MySQL Enterprise Support allows for unlimited incidents. MySQL Enterprise Support goes beyond simply fixing issues. It also offers guidance and advice. Whether customers require assistance with performance tuning or troubleshooting, the team is there to support them every step of the way.  09:27 Lois: Perside, can you walk us through the various tools and advanced features that are available within MySQL? Maybe we could start with MySQL Shell. Perside: MySQL Shell is an integrated client tool used for all MySQL database operations and administrative functions. It's a top choice among MySQL users for its versatility and powerful features. MySQL Shell offers multi-language support for JavaScript, Python, and SQL. These naturally scriptable languages make coding flexible and efficient. They also allow developers to use their preferred programming language for everything, from automating database tasks to writing complex queries. MySQL Shell supports both document and relational models. Whether your project needs the flexibility of NoSQL's document-oriented structures or the structured relationships of traditional SQL tables, MySQL Shell manages these different data types without any problems. Another key feature of MySQL Shell is its full access to both development and administrative APIs. This ability makes it easy to automate complex database operations and do custom development directly from MySQL Shell. MySQL Shell excels at DBA operations. It has extensive tools for database configuration, maintenance, and monitoring. These tools not only improve the efficiency of managing databases, but they also reduce the possibility for human error, making MySQL databases more reliable and easier to manage.  11:37 Nikita: What about the MySQL Server tool? I know that it is the core of the MySQL ecosystem and is available in both the community and commercial editions. But how does it enhance the MySQL experience? Perside: It connects with various devices, applications, and third-party tools to enhance its functionality. The server manages both SQL for structured data and NoSQL for schemaless applications. It has many key components. The parser, which interprets SQL commands. Optimizer, which ensures efficient query execution. And then the queue cache and buffer pools. They reduce disk usage and speed up access. InnoDB, the default storage engine, maintains data integrity and supports robust transaction and recovery mechanism. MySQL is designed for scalability and reliability. With features like replication and clustering, it distributes data, manage more users, and ensure consistent uptime. 13:00 Nikita: What role does MySQL Enterprise Edition play in MySQL server's capabilities? Perside: MySQL Enterprise Edition improves MySQL server by adding a suite of commercial extensions. These exclusive tools and services are designed for enterprise-level deployments and challenging environments. These tools and services include secure online backup. It keeps your data safe with efficient backup solutions. Real-time monitoring provides insight into database performance and health. The seamless integration connects easily with existing infrastructure, improving data flow and operations. Then you have the 24/7 expert support. It offers round the clock assistance to optimize and troubleshoot your databases. 14:04 Lois: That's an extensive list of features. Now, can you explain what MySQL Enterprise plugins are? I know they're specialized extensions that boost the capabilities of MySQL server, tools, and services, but I'd love to know a little more about how they work. Perside: Each plugin serves a specific purpose. Firewall plugin protects against SQL injection by allowing only pre-approved queries. The audit plugin logs database activities, tracking who accesses databases and what they do. Encryption plugin secures data at rest, protecting it from unauthorized access. Then we have the authentication plugin, which integrates with systems like LDAP and Active Directory for control access. Finally, the thread pool plugin optimizes performance in high load situation by effectively controlling how many execution threads are used and how long they run. The plugin and tools are included in the MySQL Enterprise Edition suite. 15:32 Join the Oracle University Learning Community and tap into a vibrant network of over 1 million members, including Oracle experts and fellow learners. This dynamic community is the perfect place to grow your skills, connect with likeminded learners, and celebrate your successes. As a MyLearn subscriber, you have access to engage with your fellow learners and participate in activities in the community. Visit community.oracle.com/ou to check things out today! 16:03 Nikita: Welcome back! We've been going through the various MySQL tools, and another important one is MySQL Enterprise Backup, right?  Perside: MySQL Enterprise Backup is a powerful tool that offers online, non-blocking backup and recovery. It makes sure databases remain available and performs optimally during the backup process. It also includes advanced features, such as incremental and differential backup. Additionally, MySQL Enterprise Backup supports compression to reduce backups and encryptions to keep data secure. One of the standard capabilities of MySQL Enterprise Backup is its seamless integration with media management software, or MMS. This integration simplifies the process of managing and storing backups, ensuring that data is easily accessible and secure. Then we have the MySQL Workbench Enterprise. It enhances database development and design with robust tools for creating and managing your diagram and ensuring proper documentation. It simplifies data migration with powerful tools that makes it easy to move databases between platforms. For database administration, MySQL Workbench Enterprise offers efficient tools for monitoring, performance tuning, user management, and backup and recovery. MySQL Enterprise Monitor is another tool. It provides real-time MySQL performance and availability monitoring. It helps track database's health and performance. It visually finds and fixes problem queries. This is to make it easy to identify and address performance issues. It offers MySQL best-practice advisors to guide users in maintaining optimal performance and security. Lastly, MySQL Enterprise Monitor is proactive and it provides forecasting. 18:40 Lois: Oh that's really going to help users stay ahead of potential issues. That's fantastic! What about the Oracle Enterprise Manager Plugin for MySQL? Perside: This one offers availability and performance monitoring to make sure MySQL databases are running smoothly and efficiently. It provides configuration monitoring. This is to help keep track of the database settings and configuration. Finally, it collects all available metrics to provide comprehensive insight into the database operation. 19:19 Lois: Are there any tools designed to handle higher loads and improve security? Perside: MySQL Enterprise Thread Pool improves scalability as concurrent connections grows. It makes sure the database can handle increased loads efficiently. MySQL Enterprise Authentication is another tool. This one integrates MySQL with existing security infrastructures. It provides robust security solutions. It supports Linux PAM, LDAP, Windows, Kerberos, and even FIDO for passwordless authentication. 20:02 Nikita: Do any tools offer benefits like customized logging, data protection, database security? Perside: The MySQL Enterprise Audit provides out-of-the-box logging of connections, logins, and queries in XML or JSON format. It also offers simple to fine-grained policies for filtering and log rotation. This is to ensure comprehensive and customizable logging. MySQL Enterprise Firewall detects and blocks out of policy database transactions. This is to protect your data from unauthorized access and activities. We also have MySQL Enterprise Asymmetric Encryption. It uses MySQL encryption libraries for key management signing and verifying data. It ensures data stays secure during handling. MySQL Transparent Data Encryption, another tool, provides data-at-rest encryption within the database. The Master Key is stored outside of the database in a KMIP 1.1-compliant Key Vault. That is to improve database security. Finally, MySQL Enterprise Masking offers masking capabilities, including string masking and dictionary replacement. This ensures sensitive data is protected by obscuring it. It also provides random data generators, such as range-based, payment card, email, and social security number generators. These tools help create realistic but anonymized data for testing and development. 22:12 Lois: Can you tell us about HeatWave, the MySQL cloud service? We're going to have a whole episode dedicated to it soon, but just a quick introduction for now would be great. Perside: MySQL HeatWave offers a fully managed MySQL service. It provides deployment, backup and restore, high availability, resizing, and read replicas, all the features you need for efficient database management. This service is a powerful union of Oracle Infrastructure and MySQL Enterprise Edition 8. It combines robust performance with top-tier infrastructure. With MySQL HeatWave, your systems are always up to date with the latest security fixes, ensuring your data is always protected. Plus, it supports both OLTP and analytics/ML use cases, making it a versatile solution for diverse database needs. 23:22 Nikita: So to wrap up, what are your key takeways when it comes to MySQL? Perside: When you use MySQL, here is the bottom line. MySQL Enterprise Edition delivers unmatched performance at scale. It provides advanced monitoring and tuning capabilities to ensure efficient database operation, even under heavy loads. Plus, it provides insurance and immediate help when needed, allowing you to depend on expert support whenever an issue arises. Regarding total cost of ownership, TCO, this edition significantly reduces the risk of downtime and enhances productivity. This leads to significant cost savings and improved operational efficiency. On the matter of risk, MySQL Enterprise Edition addresses security and regulatory compliance. This is to make sure your data meets all necessary standards. Additionally, it provides direct contact with the MySQL team for expert guidance. In terms of DevOps agility, it supports automated scaling and management, as well as flexible real-time backups, making it ideal for agile development environments. Finally, concerning customer satisfaction, it enhances application performance and uptime, ensuring your customers have a reliable and smooth experience. 25:18 Lois: Thank you so much, Perside. This is really insightful information. To learn more about all the support services that are available, visit support.oracle.com. This is the central hub for all MySQL Enterprise Support resources.  Nikita: Yeah, and if you want to know about the key commercial products offered by MySQL, visit mylearn.oracle.com and search for the MySQL 8.4: Essentials course. Join us next week for a discussion on installing MySQL. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 25:53 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Want to quickly provision your autonomous database? Then look no further than Oracle Autonomous Database Serverless, one of the two deployment choices offered by Oracle Autonomous Database.   Autonomous Database Serverless delegates all operational decisions to Oracle, providing you with a completely autonomous experience.   Join hosts Lois Houston and Nikita Abraham, along with Oracle Database experts, as they discuss how serverless infrastructure eliminates the need to configure any hardware or install any software because Autonomous Database handles provisioning the database, backing it up, patching and upgrading it, and growing or shrinking it for you.   Survey: https://customersurveys.oracle.com/ords/surveys/t/oracle-university-gtm/survey?k=focus-group-2-link-share-5   Oracle MyLearn: https://mylearn.oracle.com/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X (formerly Twitter): https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Rajeev Grover, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started. 00:26 Lois: Hello and welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead: Editorial Services. Nikita: Hi everyone! We hope you've been enjoying these last few weeks as we've been revisiting our most popular episodes of the year.  Lois: Today's episode is the last one in this series and is a throwback to a conversation on Autonomous Databases on Serverless Infrastructure with three experts in the field: Hannah Nguyen, Sean Stacey, and Kay Malcolm. Hannah is a Staff Cloud Engineer, Sean is the Director of Platform Technology Solutions, and Kay is Vice President of Database Product Management. For this episode, we'll be sharing portions of our conversations with them.  01:14 Nikita: We began by asking Hannah how Oracle Cloud handles the process of provisioning an  Autonomous Database. So, let's jump right in! Hannah: The Oracle Cloud automates the process of provisioning an Autonomous Database, and it automatically provisions for you a highly scalable, highly secure, and a highly available database very simply out of the box. 01:35 Lois: Hannah, what are the components and architecture involved when provisioning an Autonomous Database in Oracle Cloud? Hannah: Provisioning the database involves very few steps. But it's important to understand the components that are part of the provisioned environment. When provisioning a database, the number of CPUs in increments of 1 for serverless, storage in increments of 1 terabyte, and backup are automatically provisioned and enabled in the database. In the background, an Oracle 19c pluggable database is being added to the container database that manages all the user's Autonomous Databases. Because this Autonomous Database runs on Exadata systems, Real Application Clusters is also provisioned in the background to support the on-demand CPU scalability of the service. This is transparent to the user and administrator of the service. But be aware it is there. 02:28 Nikita: Ok…So, what sort of flexibility does the Autonomous Database provide when it comes to managing resource usage and costs, you know… especially in terms of starting, stopping, and scaling instances? Hannah: The Autonomous Database allows you to start your instance very rapidly on demand. It also allows you to stop your instance on demand as well to conserve resources and to pause billing. Do be aware that when you do pause billing, you will not be charged for any CPU cycles because your instance will be stopped. However, you'll still be incurring charges for your monthly billing for your storage. In addition to allowing you to start and stop your instance on demand, it's also possible to scale your database instance on demand as well. All of this can be done very easily using the Database Cloud Console. 03:15 Lois: What about scaling in the Autonomous Database? Hannah: So you can scale up your OCPUs without touching your storage and scale it back down, and you can do the same with your storage. In addition to that, you can also set up autoscaling. So the database, whenever it detects the need, will automatically scale up to three times the base level number of OCPUs that you have allocated or provisioned for the Autonomous Database. 03:38 Nikita: Is autoscaling available for all tiers?  Hannah: Autoscaling is not available for an always free database, but it is enabled by default for other tiered environments. Changing the setting does not require downtime. So this can also be set dynamically. One of the advantages of autoscaling is cost because you're billed based on the average number of OCPUs consumed during an hour. 04:01 Lois: Thanks, Hannah! Now, let's bring Sean into the conversation. Hey Sean, I want to talk about moving an autonomous database resource. When or why would I need to move an autonomous database resource from one compartment to another? Sean: There may be a business requirement where you need to move an autonomous database resource, serverless resource, from one compartment to another. Perhaps, there's a different subnet that you would like to move that autonomous database to, or perhaps there's some business applications that are within or accessible or available in that other compartment that you wish to move your autonomous database to take advantage of. 04:36 Nikita: And how simple is this process of moving an autonomous database from one compartment to another? What happens to the backups during this transition? Sean: The way you can do this is simply to take an autonomous database and move it from compartment A to compartment B. And when you do so, the backups, or the automatic backups that are associated with that autonomous database, will be moved with that autonomous database as well. 05:00 Lois: Is there anything that I need to keep in mind when I'm moving an autonomous database between compartments?  Sean: A couple of things to be aware of when doing this is, first of all, you must have the appropriate privileges in that compartment in order to move that autonomous database both from the source compartment to the target compartment. In addition to that, once the autonomous database is moved to this new compartment, any policies or anything that's defined in that compartment to govern the authorization and privileges of that said user in that compartment will be applied immediately to that new autonomous database that has been moved into that new compartment. 05:38 Nikita: Sean, I want to ask you about cloning in Autonomous Database. What are the different types of clones that can be created?  Sean: It's possible to create a new Autonomous Database as a clone of an existing Autonomous Database. This can be done as a full copy of that existing Autonomous Database, or it can be done as a metadata copy, where the objects and tables are cloned, but they are empty. So there's no rows in the tables. And this clone can be taken from a live running Autonomous Database or even from a backup. So you can take a backup and clone that to a completely new database. 06:13 Lois: But why would you clone in the first place? What are the benefits of this?  Sean: When cloning or when creating this clone, it can be created in a completely new compartment from where the source Autonomous Database was originally located. So it's a nice way of moving one database to another compartment to allow developers or another community of users to have access to that environment. 06:36 Nikita: I know that along with having a full clone, you can also have a refreshable clone. Can you tell us more about that? Who is responsible for this? Sean: It's possible to create a refreshable clone from an Autonomous Database. And this is one that would be synced with that source database up to so many days. The task of keeping that refreshable clone in sync with that source database rests upon the shoulders of the administrator. The administrator is the person who is responsible for performing that sync operation. Now, actually performing the operation is very simple, it's point and click. And it's an automated process from the database console. And also be aware that refreshable clones can trail the source database or source Autonomous Database up to seven days. After that period of time, the refreshable clone, if it has not been refreshed or kept in sync with that source database, it will become a standalone, read-only copy of that original source database. 07:38 Nikita: Ok Sean, so if you had to give us the key takeaways on cloning an Autonomous Database, what would they be?  Sean: It's very easy and a lot of flexibility when it comes to cloning an Autonomous Database. We have different models that you can take from a live running database instance with zero impact on your workload or from a backup. It can be a full copy, or it can be a metadata copy, as well as a refreshable, read-only clone of a source database. 08:12 Did you know that Oracle University offers free courses on Oracle Cloud Infrastructure? You'll find training on everything from cloud computing, database, and security to artificial intelligence and machine learning, all of which is available free to subscribers. So, get going! Pick a course of your choice, get certified, join the Oracle University Learning Community, and network with your peers. If you're already an Oracle MyLearn user, go to MyLearn to begin your journey. If you have not yet accessed Oracle MyLearn, visit mylearn.oracle.com and create an account to get started.  08:50 Nikita: Welcome back! Thank you, Sean, and hi Kay! I want to ask you about events and notifications in Autonomous Database. Where do they really come in handy?  Kay: Events can be used for a variety of notifications, including admin password expiration, ADB services going down, and wallet expiration warnings. There's this service, and it's called the notifications service. It's part of OCI. And this service provides you with the ability to broadcast messages to distributed components using a publish and subscribe model. These notifications can be used to notify you when event rules or alarms are triggered or simply to directly publish a message. In addition to this, there's also something that's called a topic. This is a communication channel for sending messages to subscribers in the topic. You can manage these topics and their subscriptions really easy. It's not hard to do at all. 09:52 Lois: Kay, I want to ask you about backing up Autonomous Databases. How does Autonomous Database handle backups? Kay: Autonomous Database automatically backs up your database for you. The retention period for backups is 60 days. You can restore and recover your database to any point in time during this retention period. You can initiate recovery for your Autonomous Database by using the cloud console or an API call. Autonomous Database automatically restores and recovers your database to the point in time that you specify. In addition to a point in time recovery, we can also perform a restore from a specific backup set.  10:37 Lois: Kay, you spoke about automatic backups, but what about manual backups?  Kay: You can do manual backups using the cloud console, for example, if you want to take a backup say before a major change to make restoring and recovery faster. These manual backups are put in your cloud object storage bucket. 10:58 Nikita: Are there any special instructions that we need to follow when configuring a manual backup? Kay: The manual backup configuration tasks are a one-time operation. Once this is configured, you can go ahead, trigger your manual backup any time you wish after that. When creating the object storage bucket for the manual backups, it is really important-- so I don't want you to forget-- that the name format for the bucket and the object storage follows this naming convention. It should be backup underscore database name. And it's not the display name here when I say database name. In addition to that, the object name has to be all lowercase. So three rules. Backup underscore database name, and the specific database name is not the display name. It has to be in lowercase. Once you've created your object storage bucket to meet these rules, you then go ahead and set a database property. Default_backup_bucket. This points to the object storage URL and it's using the Swift protocol. Once you've got your object storage bucket mapped and you've created your mapping to the object storage location, you then need to go ahead and create a database credential inside your database. You may have already had this in place for other purposes, like maybe you were loading data, you were using Data Pump, et cetera. If you don't, you would need to create this specifically for your manual backups. Once you've done so, you can then go ahead and set your property to that default credential that you created. So once you follow these steps as I pointed out, you only have to do it one time. Once it's configured, you can go ahead and use it from now on for your manual backups. 13:00 Lois: Kay, the last topic I want to talk about before we let you go is Autonomous Data Guard. Can you tell us about it? Kay: Autonomous Data Guard monitors the primary database, in other words, the database that you're using right now.  13:14 Lois: So, if ADB goes down… Kay: Then the standby instance will automatically become the primary instance. There's no manual intervention required. So failover from the primary database to that standby database I mentioned, it's completely seamless and it doesn't require any additional wallets to be downloaded or any new URLs to access APEX or Oracle Machine Learning. Even Oracle REST Data Services. All the URLs and all the wallets, everything that you need to authenticate, to connect to your database, they all remain the same for you if you have to failover to your standby database. 13:58 Lois: And what happens after a failover occurs? Kay: After performing a failover, a new standby for your primary will automatically be provisioned. So in other words, in performing a failover your standby does become your new primary. Any new standby is made for that primary. I know, it's kind of interesting. So currently, the standby database is created in the same region as the primary database. For better resilience, if your database is provisioned, it would be available on AD1 or Availability Domain 1. My secondary, or my standby, would be provisioned on a different availability domain. 14:49 Nikita: But there's also the possibility of manual failover, right? What are the differences between automatic and manual failover scenarios? When would you recommend using each? Kay: So in the case of the automatic failover scenario following a disastrous situation, if the primary ADB becomes completely unavailable, the switchover button will turn to a failover button. Because remember, this is a disaster. Automatic failover is automatically triggered. There's no user action required. So if you're asleep and something happens, you're protected. There's no user action required, but automatic failover is allowed to succeed only when no data loss will occur. For manual failover scenarios in the rare case when an automatic failover is unsuccessful, the switchover button will become a failover button and the user can trigger a manual failover should they wish to do so. The system automatically recovers as much data as possible, minimizing any potential data loss. But you can see anywhere from a few seconds or minutes of data loss. Now, you should only perform a manual failover in a true disaster scenario, expecting the fact that a few minutes of potential data loss could occur, to ensure that your database is back online as soon as possible.  16:23 Lois: We hope you've enjoyed revisiting some of our most popular episodes over these past few weeks. We always appreciate your feedback and suggestions so remember to take that quick survey we've put out. You'll find it in the show notes for today's episode. Thanks a lot for your support. We're taking a break for the next two weeks and will be back with a brand-new season of the Oracle University Podcast in January. Happy holidays, everyone! Nikita: Happy holidays! Until next time, this is Nikita Abraham... Lois: And Lois Houston, signing off! 16:56 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Redwood is a state-of-the-art graphical interface that defines the look and feel of the new Oracle Cloud Redwood Applications.   In this episode, hosts Lois Houston and Nikita Abraham, along with Senior Principal OCI Instructor Joe Greenwald, take a closer look at the intent behind the design and development aspects of the new Redwood experience. They also explore Redwood page templates and components.   Survey: https://customersurveys.oracle.com/ords/surveys/t/oracle-university-gtm/survey?k=focus-group-2-link-share-5   Developing Redwood Applications with Visual Builder: https://mylearn.oracle.com/ou/learning-path/developing-redwood-applications-with-visual-builder/112791   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X (formerly Twitter): https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started. 00:26 Nikita: Hello and welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead: Editorial Services with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi everyone! Thanks for joining us for this Best of 2024 series, where we're playing for you our four most popular episodes of the year.    Nikita: Today's episode is #3 of 4 and is a throwback to another conversation with Joe Greenwald, our Senior Principal OCI Instructor. We asked Joe about Oracle's Redwood design system and how it helps us create stunning, world-class enterprise applications and user experiences.  01:04 Lois: Yeah, Redwood is the basis for all the new Oracle Cloud Applications being re-designed, developed, and delivered. Joe is the best person to ask about all of this because he's been working with our Oracle software development tools since the early 90s and is responsible for OU's Visual Builder Studio and Redwood course content. So, let's dive right in! Joe: Hi Lois. Hi Niki. I am excited to join you on this episode because with the release of 24A Fusion applications, we are encouraging all our customers to adopt the new Redwood design system and components, and take advantage of the world-class look and feel of the new Redwood user experience. Redwood represents a new approach and direction for us at Oracle, and we're excited to have our customers benefit from it. 01:49 Nikita: Joe, you've been working with Oracle user interface development tools and frameworks for a long time. How and why is Redwood different? Joe: I joined Oracle in 1992, and the first Oracle user interface I experienced was Oracle Forms. And that was the character mode. I came from a background of Smalltalk and its amazing, pioneering graphical user interface (GUI) design capabilities. I worked at Apple and I developed my own GUIs for a few years on PCs and Macs. So, Character Mode Forms, what we used to call DMV (Department of Motor Vehicles) screens, was a shock, to say the least. Since then, I've worked with almost every user interface and development platform Oracle has created: Character Mode Forms, GUI Forms, Power Objects, HyperCard on the Macintosh, that was pre-OS X by the way, Sedona, written in native C++ and ActiveX and OLE, which didn't make it to a product but appeared in other things later, ADF Faces, which uses Java to generate HTML pages, and APEX, which uses PL/SQL to generate HTML pages. And I've worked with and wrote training classes for Java Swing, an excellent GUI framework for event-driven desktop and enterprise applications, but it wasn't designed for the web. So, it's with pleasure that I introduce you to the Redwood design system, easily the best effort I've ever seen, from the look and feel of holistic user-goal-centered design philosophy and approach to the cutting-edge WYSIWYG design tools.  03:16 Lois: Joe, is Redwood just another set of styles, colors, and fonts, albeit very nice-looking ones? Joe: The Redwood platform is new for Oracle, and it represents a significant change, not just in the look and feel, colors, fonts, and styles, I mean that too, but it's also a fundamental change in how Oracle is creating, designing, and imagining user interfaces. As you may be aware, all Oracle Cloud Applications are being re-designed, re-engineered, and re-rebuilt from the ground up, with significant changes to both back-end and front-end architectures. The front end is being redesigned, re-developed, and re-created in pure HTML5, CSS3, and JavaScript using Visual Builder Studio and its design-time browser-based Integrated Development Environment. The back end is being re-architected, re-designed, and implemented in a modern microservice architecture for Oracle Cloud using Kubernetes and other modern technologies that improve performance and work better in the cloud than our current legacy architecture. The new Oracle Cloud Applications platform uses Redwood for its design system—its tools, its patterns, its components, and page templates. Redwood is a richer and more productive platform to create solutions while still being cost-effective for Oracle. It encourages a transformation of the fundamental user experience, emphasizing identifying, meeting, and understanding end users' goals and how the applications are used.  04:39 Nikita: Joe, do you think Oracle's user interface has been improved with Redwood? In what ways has the UI changed? Joe: Yes, absolutely. Redwood has changed a lot of things. When I joined Oracle back in the '90s, there was effectively no user interface division or UI team. Everybody just did their own thing. There was no user interface lab—and that was started in the mid-‘90s—and I was asked to give product usability feedback and participate in UI tests and experiments in those labs. I also helped test the products I was teaching at the time. I actually distinctly remember having to take a week to train users on Oracle's Designer CASE tool product just to prep the participants enough to perform usability testing. I can still hear the UI lab manager shaking her head and saying any product that requires a week of training to do usability testing has usability issues! And if you're like me and you've been around Oracle long enough, you know that Oracle's not always been known for its user interfaces and been known to release products that look like they were designed by two or more different companies. All that has changed with Redwood. With Redwood, there's a new internal design group that oversees the design choices of all development teams that develop products. This includes a design system review and an ongoing audit process to ensure that all the products being released, whether Fusion apps or something else, all look and feel similar so it looks like it's designed by a single company with a single thought in mind. Which it is. There's a deeper, consistent commitment in identifying user needs, understanding how the applications are being used, and how they meet those user needs through things like telemetry: gathering metrics from the actual components and the Redwood system itself to see how the applications are being used, what's working well, and what isn't.  This telemetry is available to us here at Oracle, and we use it to fine tune the applications' usability and purpose.  06:29 Lois: That's really interesting, Joe. So, it's a fundamental change in the way we're doing things. What about the GUI components themselves? Are these more sophisticated than simple GUI components like buttons and text fields? Joe: The graphical components themselves are at a much higher level, more comprehensive, and work better together. And in Redwood, everything is a component. And I'm not just talking about things like input text fields and buttons, though it applies to these more fine-grained components as well. Leveraging Oracle's deep experience in building enterprise applications, we've incorporated that knowledge into creating page templates so that the structure and look and feel of the page is fixed based on our internal design standards. The developer has control over certain portions of it, but the overall look and feel of the page is controlled by Oracle. So there is consistency of look and feel within and across applications. These page templates come with predefined functionalities: headers, titles, properties, and variables to manipulate content and settings, slots for other components to hold like search fields, collections, contextual information, badges, and images, as well as primary and secondary actions, and variables for events and event handling through Visual Builder action chains, which handle the various actions and processing of the request on the page. And all these page templates and components are responsive, meaning they respond to the change in the size of the page and the orientation. So, when you move from a desktop to a handheld mobile device or a tablet, they respond appropriately and consistently to deliver a clean, easy-to-use interface and experience. 08:03 Nikita: You mentioned WYSIWYG design tools and their integration with Visual Builder Studio's integrated development environment. How does Redwood work with Visual Builder Studio? Joe: This is easily one of my most favorite aspects about Redwood and the integration with Visual Builder Studio Designer. The components and page templates are responsive at runtime as well as responsive at design time! In over 30 years of working with Oracle software development products, this is the first development system and integrated development environment I've seen Oracle produce where what you see is what you get at design time. Now, with products such as Designer and JDeveloper ADF Faces and even APEX—all those page-generation types of products—you have to generate the page, deploy it, and only then can you view the final page to see whether it meets the needs of your user interface. For example, with Designer, there were literally hundreds of configuration parameters that you could set to control how forms and reports looked when they were generated —down to how many buttons on a row or how many rows to a page, that sort of thing, all done in text mode. Then you'd generate and run the page to see what the result was and then go back and modify things until you got what you wanted. I remember hearing the product managers for Oracle ADF Faces being asked…well, a customer asked, “What happens if I put this component here and this component here? What will the page look like?” and they'd say, “I don't know. Render the page and let's see.” That's just crazy talk. With Redwood and its integration with Visual Builder Studio Designer, what you see on the page at design time is literally what you get. And if I make the page narrower or I even convert it to a mobile display while in the Designer itself, I immediately see what the page looks like in that new mode. Everything just moves accordingly, at design time. For example, when changing to a mobile UI, everything stacks up nicely; the components adjust to the page size and change right there in the design environment. Again, I can't emphasize enough the simple luxury of being able to see exactly what the user is going to see on my page and having the ability to change the resolution, orientation, and screen size, and it changes right there immediately in my design environment. 10:06 Lois: I'm intrigued by the idea of page templates that are managed by Oracle but still leave room for the developer to customize aspects of the look and feel and functionality. How does that work? Joe: Well, the page templates themselves represent the typical pages you would most likely use in an enterprise application. Things like a welcome page, a search page, and edit and create pages, and a couple of different ways to display summary information, including foldout pages, though this is not an exhaustive list of course. Not only do they provide a logical and complete starting point for the layout of the page itself, but they also include built-in functionality. These templates include functionality for buttons, primary and secondary actions, and areas for holding contextual information, badges, avatars, and images. And this is all built right into the page, and all of them use variables to describe the contents for the various parts, so the contents can change programmatically as the variables' contents change, if necessary. 11:04 Do you have an idea for a new course or learning opportunity? We'd love to hear it! Visit the Oracle University Learning Community and share your thoughts with us on the Idea Incubator. Your suggestion could find a place in future development projects. Visit mylearn.oracle.com to get started.  11:24 Nikita: Welcome back! So, Joe, let's say I'm a developer. How do I get started working with Redwood? Joe: One of the easiest ways to do it is to use Visual Builder Studio Designer and create a new visual application. If you're creating a standalone, bespoke custom application, you can choose a Redwood starter template, which will include all the Redwood components and page templates automatically. Or, if you're extending and customizing an Oracle Fusion application, Redwood is already included.  Either way, when you create a new page, you have a choice of different page templates—welcome page templates, edit pages, search pages, etc. —and all you have to do is choose a page that you want and begin configuring it. And actually if you make a mistake, it's easy to switch page templates. All the components, page templates, and so on have documentation right there inside Visual Builder Studio Designer, and we do recommend that you read through the documentation first to get an understanding of what the use case for that template is and how to use it. And some components are more granular, like a collection container which holds a collection of rows of a list or a table and provides capabilities like toolbars and other actions that are already built and defined. You decide what actions you want and then use predefined event listeners that are triggered when an event occurs in the application—like a button being clicked or a row being selected—which kicks off a series of actions to be performed. 12:42 Lois: That sounds easy enough if you know what you're doing. Joe, what are some of the more common pages and what are they used for?  Joe: Redwood page templates can be broken down into categories. There are overview templates like the welcome page template, which has a nice banner, colors, and illustrations that can be used for a welcoming page—like for entering a new application or a new logical section of the application. The dashboard landing page template displays key information values and their charts and graphs, which can come from Oracle Analytics, and automatically switches the display depending on which set of data is selected.  The detail templates include a general overview, which presents read-only information related to a single record or resource. The item overview gives you a small panel to view summary information (for example, information on a customer) and in the main section, you can view details like all the orders for that customer. And you can even navigate through a set of customers, clicking arrows for next-previous navigation. And that's all built in. There's no programming required. The fold-out page template folds out horizontally to show you individual panels with more detail that can be displayed about the subject being retrieved as well as overflow and drill-down areas. And there's a collection detail template that will display a list with additional details about the selected item (for example, an order and its order line items). The smart search page does exactly what it says. It has a search component that you use to filter or search the data coming back from the REST data sources and then display the results in a list or a table. You define the filter yourself and apply it using different kinds of comparators, so you can look for strings that start with certain values or contain values, or numerical values that are equal to or less than, depending on what you're filtering for. And then there are the transactional templates, which are meant to make changes. This includes both the simple create and edit and advanced create and edit templates.  The simple create and edit page template edits a single record or creates a single record. And the advanced page template works well if you're working with master-detail, parent-child type relationships. Let's say you want to view the parent and create children for it or even create a parent and the children at the same time. And there's a Gantt chart page for project management–type tracking and a guided process page for multiple-step processes and there's a data management page template specifically for viewing and editing data collections like Excel spreadsheets. 14:55 Nikita: You mentioned that there's a design system behind all this. How is this used, and how does the customer benefit from it? Joe: Redwood comprises both a design system and a development system. The design system has a series of steps that we follow here at Oracle and can suggest that you, our customers and partners, can follow as well. This includes understanding the problem, articulating the vision for the page and the application (what it should do), identifying the proper Redwood page templates to use, adding detail and refining the design and then using a number of different mechanisms, including PowerPoint or Figma design tools to specify the design for development, and then monitor engagement in the real world. These are the steps that we follow here at Oracle. The Redwood development process starts with learning how to use Redwood components and templates using the documentation and other content from redwood.oracle.com and Visual Builder Studio. Then it's about understanding the design created by the design team, learning more about components and templates for your application, specifically the ones you're going to use, how they work, and how they work together. Then developing your application using Visual Builder Studio Designer, and finally improving and refining your application. Now, right now, as I mentioned, telemetry is available to us here at Oracle so we can get a sense of the feedback on the pages of how components are being used and where time is being spent, and we use that to tune the designs and components being used. That telemetry data may be available to customers in the future. Now, when you go to redwood.oracle.com, you can access the Redwood pattern book that shows you in detail all the different page templates that are available: smart search page, data grid, welcome page, dashboard landing page, and so on, and you can select these and read more about them as well as the actual design specifications that were used to build the pages—defining what they do and what they respond to. They provide a lot of detailed information about the templates and components, how they work and how they're intended to be used. 16:50 Lois: That's a lot of great resources available. But what if I don't have access to Visual Builder Studio Designer? Can I still see how Redwood looks and behaves? Joe: Well, if you go to redwood.oracle.com, you can log in and work with the Redwood reference application, which is a live application working with live data. It was created to show off the various page templates and components, their look and feel and functionality from the Redwood design and development systems. This is an order management application, so you can do things like view filtered pending orders, create new orders, manage orders, and view information about customers and inventory. It uses the different page templates to show you how the application can perform. 17:29 Nikita: I assume there are common aspects to how these page templates are designed, built, and intended to be used. Is that a good way to begin understanding how to work with them? Becoming familiar with their common properties and functionality? Joe: Absolutely! Good point! All pages have titles, and most have primary and secondary actions that can be triggered through a variety of GUI events, like clicking a button or a link or selecting something in a list or a table. The transactional page templates include validation groups that validate whether the data is correct before it is submitted, as well as a message dialog that can pop up if there are unsaved changes and someone tries to leave the page. All the pages can use variables to display information or set properties and can easily display specific contextual information about records that have been retrieved, like adding the Order Number or Customer Name and Number to the page title or section headers. 18:18 Lois: If I were a developer, I'd be really excited to get started! So, let's say I'm a developer. What's the best way to begin learning about Redwood, Joe? Joe: A great place to start learning about the Redwood design and development system is at the redwood.oracle.com page I mentioned. We have many different pages that describe the philosophy and fundamental basis for Redwood, the ideas and intent behind it, and how we're using it here at Oracle. It also has a list of all the different page templates and components you can use and a link to the Redwood reference application where you can sign in and try it yourself. In addition, we at Oracle University offer a course called Design and Develop Redwood Applications, and in there, we have both lecture content as well as hands-on practices where you build a lightweight version of the Redwood reference application using data from the Fusion apps application, as well as the pages that I talked about: the welcome page, detail pages, transactional pages, and the dashboard landing page.  And you'll see how those pages are designed and constructed while building them yourself.  It's very important though to take one of the free Visual Builder Developer courses first: either Build Visual Applications Using Visual Builder Studio and/or Develop Fusion Applications Using Visual Builder Studio before you try to work through the practices in the Redwood course because it uses a lot of Visual Builder Designer technology.  You'll get a lot more out of the Redwood practices if you understand the basics of Visual Builder Studio first. The Build Visual Applications Using Visual Builder Studio course is probably a better place to start unless you know for a fact you will be focusing on extending Oracle Fusion Applications using Visual Builder Studio. Now, a lot of the content is the same between the two courses as they share much of the same technology and architectures. 19:58 Lois: Ok, so Build Visual Applications Using Visual Builder Studio and Develop Fusion Applications Using Visual Builder Studio…all on mylearn.oracle.com and all free for anyone who wants to take them, right? Joe: Yes, exactly. And the free Redwood learning path leads to an Associate certification. While our courses are a great place to start in preparing for your certification exam, they are not, of course, by themselves sufficient to pass and you will want to study and be familiar with the redwood.oracle.com content as well. The learning path is free, but you do have to pay for the certification exam. Nikita: We hope you enjoyed that conversation. A quick reminder about the short survey we've created to gather your insights and suggestions for the podcast. It's really quick. Just click the link in the show notes to complete the survey. Thank you so much for helping us make the show better. Join us next week for another throwback episode. Until then, this is Nikita Abraham... Lois: And Lois Houston, signing off! 20:58 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

What do you need to start customizing the next generation of Oracle Fusion Apps? How do you create new pages for business processes? What level of expertise do you require for this?   Join Lois Houston and Nikita Abraham as they get answers to all these questions and more from Senior Principal OCI Instructor Joe Greenwald.   Survey: https://customersurveys.oracle.com/ords/surveys/t/oracle-university-gtm/survey?k=focus-group-2-link-share-5   Develop Fusion Applications Using Visual Builder Studio: https://mylearn.oracle.com/ou/course/develop-fusion-applications-using-visual-builder-studio/138392/   Build Visual Applications Using Oracle Visual Builder Studio: https://mylearn.oracle.com/ou/course/build-visual-applications-using-oracle-visual-builder-studio/137749/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X (formerly Twitter): https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started. 00:26 Lois: Welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead of Editorial Services. Nikita: Hi there! You're listening to our Best of 2024 series, where over the next few weeks, we'll be revisiting four of our most popular episodes of the year. Lois: Today's episode is #2 of 4, and we're throwing it back to another episode with our friend and Senior Principal OCI Instructor Joe Greenwald. This episode is all about extending Oracle Cloud Applications that are being built using Visual Builder for the front-end. 01:04 Nikita: Right, Lois. We began by asking Joe to explain what's happening with the redesign and re-architecture of Oracle Cloud Applications using Visual Builder Studio, or VBS.  Joe: That's right, Niki. Oracle is redesigning and rebuilding its entire suite of Fusion Cloud Applications, over 330 different products, utilizing over 60,000 engineers — that is “60,” not “16”—at Oracle to develop the next generation of Oracle Fusion Applications. What's most exciting is that the same tools the engineers are using to accomplish this are available to our partners and our customers to use to extend the functionality and capabilities of Fusion Applications to meet their custom needs and processes.  01:45 Lois: That's pretty awesome! We want to use this time today to ask you about extensions, the types of extensions you can create, and how to use Visual Builder Studio to create those extensions. Nikita: Yeah, can we start with you telling us what an extension is? I've gotten the sense that Oracle uses the term extension as both a noun and a verb and that's a bit confusing to me. Joe: Yeah, good catch, Niki. Yes, Oracle does use the term extension in two ways: both as a noun and a verb. As a noun, an extension is a container for the code changes that you make to your applications. Basically, it's a Git repository that Oracle creates and manages for you. So, the extension container holds the code changes you make to your page layouts: the fields, their positioning, showing and hiding fields, that sort of thing, as well as page functionality. These code changes you make are stored in the extension and it is this extension with your code changes that is merged with the main Git branch eventually and then deployed using continuous integration/continuous deployment jobs defined in Visual Builder Studio, which manages the project and its assets. Your extension is a Git branch that is an asset of the project. Once your extension code is merged with the main branch and deployed, then the next time someone brings up the application, they'll see the changes you've made in the app. 02:59 Lois: And as a verb? Joe: As a verb, extension means to extend the functionality and the look and feel of the application, though I prefer the term customization or configuration to describe this aspect, as the documentation does, and to avoid confusion, though I'll admit I'm not always consistent about the terms I use. 03:16 Lois: What types of customizations, or extensions, and I'm using the verb now, are available for Fusion Apps in Visual Builder Studio? Joe: There are three different ways Fusion Apps can be customized effectively, configured, or extended. The first way is what we call a basic extension, where you're rearranging hiding, or showing, or moving around fields and sections on the page that have been set up to be extendable by the Fusion Application development teams. Things like hiding fields, showing fields, hiding sections, showing sections…  03:45 Nikita: So fairly basic actions… Joe: Yeah exactly and they can be done in Visual Builder Studio Designer by people with minimal VB training, Visual Builder training. And, most recently, if you have access to it, you can do it in the new Express mode, where the page shows you just those things you can work with and just the tools you need to work with the page. This is new and makes it much easier for folks who are not highly technical to make basic changes to the page layout. 04:09 Lois: People like me! That sounds easy enough. Joe: And the next type of extension is more of an intermediate change and requires some training with Visual Builder Studio because you're creating rules that govern the display of layouts based on certain conditions on the page. These are highly flexible, powerful, and useful for creating customized page layouts based on a variety of factors from page size and orientation to the role of the person using it to values in the actual fields on the page itself. These rules can be combined to create complex rule-based conditions that display exactly what the user should see, given the conditions of the page and their role. I would also include making changes to action chains, which execute sequences of behaviors and navigation, and the actual structure of the application, but this is more advanced.  Lastly, is creating mashup applications, which are stand-alone Visual Builder visual applications, which use data from Fusion apps, and customer data sources, like their own database tables, and potentially third-party APIs to create brand new pages and applications with new functionality, new processes, new procedures, new displays, all of which look just like Fusion Applications and use the same data as Fusion applications. 05:18 Lois: Joe, how do I get started if I want to extend a page?  Joe: The easiest way to do it is to open a page in Fusion Applications and then select Edit Page in Visual Builder Studio from the Profile menu. You're then prompted for a project to hold the Git repository for the extension container. And since there's probably already one that exists, after you select the project, an extension Git container is assigned to you. Unless this is the very first time the application has been extended in which case it creates an extension for you. When creating customizations or configurations, we recommend that each application be done in its own separate project. So, for example, if you're working on Customer Experience Sales, you might do it in Project A and if you're working on extensions with HCM, you might do it in Project B. And if you decide to create your own pages and flows in your own app, you might do that in Project C.  06:04 Nikita: But why do you need to do this? Joe: That's just to keep things nice and separate and organized. The tool, Visual Builder Studio, doesn't really care, but it makes for cleaner development and can help with the management of the development teams. 06:14 Nikita: Ok, Joe, I have a question. How do I know if the page I'm on in Fusion Apps can be edited in Visual Builder? I know there are a lot of legacy pages still out there and they can co-exist with the new VB-based pages. Joe: If the URL of the page you're on has the word /Redwood in it instead of /faces, then you know this is a page that was created using Visual Builder Studio and you'll be able to extend it and make changes to it using the Edit in Visual Builder Studio option. So, if you select Edit in Visual Builder Studio, then the page you are on opens inside Visual Builder Studio Designer and you can make changes to any part of the page that has been explicitly enabled for extension by the development team. 06:53 Lois: That's an important part, right? The application is not extendable by default.  Joe: That's right, Lois. It is all locked down and you can't make any changes to it by default. The development team must specifically enable certain parts of the page: sections, fields, layouts, variables, types, action chains, etc. as extendable for you to be able to make changes to it. This ensures the changes the development team makes to the application in the future won't break your extensions. And conversely, the development team can choose to not extend portions that they do not want you to touch or mess with. Then if they do change that bit of the app in the future, it won't break the application and you won't get a big surprise. So, using the Edit page in Visual Builder Studio, you can make both basic changes, like moving, showing, and hiding fields and sections, as well as the more intermediate types of configurations, like using dynamic components to create rule-based layouts that change dynamically based on several conditions such as page size, roles of the user, and field values on the page itself. 07:51 Nikita: What happens if two developers make changes and essentially overwrite each other's customizations — say one hides a field and another later exposes it? Joe: Well, whoever commits their changes and deploys last wins. The other developer's changes get overwritten. So, this is something the team would want to consider carefully. It is possible to roll back to an earlier version if one must. And this can be done in Visual Builder Studio — the part that manages project assets like Git repositories. And there are Oracle blog posts about how to do that if you're interested in learning more. 08:20 Lois: Joe, earlier you mentioned creating new pages and flows, but so far you've only talked about modifying existing extendable pages. How do I create new pages and flows? Joe: In a Visual Builder extension, a set of pages and flows is called an App UI. When I use the terms pages and flows, what I'm talking about is a set of pages that are logically related—whatever logical means to the designer and developer—in a group called a flow that you can navigate between. But you can also navigate between flows and even between applications. So, without getting too technical, each application has a default flow, which has a default page where that flow starts when the app first comes up. So, you can think of an App UI as a collection of flows and their pages, and a URL that accesses the default flow and its default page. That's the page you would see first when accessing that URL. Of course, this can be configured and changed by the developer, as needed. Now, when Oracle creates the original application (for example, digital sales, helpdesk, or something like that), we create an App UI, which contains the pages and flows for that application and is the “entry point” into the app, accessing that App UI's default flow and its default page and then things flow on from there. Partners and customers can create their own application extensions that are dependent on an Oracle application and even create their own App UI – their own sets of pages and flows to accommodate their own processing and workflow needs. This gives them the ability to add their own processes and rules, and still leverage and navigate to the core application that Oracle built. For example, say Oracle delivered digital sales as an Oracle Cloud Application built using Visual Builder to a customer and the customer needs to add a few pages to do some validation or other type of business processing before entering the digital sales application. What the customer does, in this case, is create a new extension of the Oracle Digital Sales app and an App UI of their own, which would be the set of pages and flows that contain the processing they want to start with before then navigating into the digital sales app to use Oracle's application. 10:22 Nikita: Wait, did I hear that correctly? We're creating an extension of an extension or creating an extension on an existing extension? Joe: I know, right? I realize this can sound confusing the first time you hear it or the second time or even the third time. It took me a while to get my head around what they're talking about. Let's start with a Fusion application. In a Fusion application, everything is an extension of something. This is just how the code base and the architecture are organized and how they manage the Git repositories and the code base itself. So, Oracle created a base application called the Unified App. The Unified Application contains the basic page structure and common functionality needed for all applications. For example, it contains the header at the top that has the profile and the footer at the bottom of the page that has that little Ask Oracle icon. Within that page, between the header and the footer, are the pages that are created by the developers, whether they be Oracle engineers or partners or customers. They display the contents of the page with the data and the layouts and all of that. In a sense, you can think of the Unified App as an index page, the starting page of the web application. Though that's not completely true technically, it's good enough for illustrative purposes. So, Oracle starts with the Unified App and then a development team extends that Unified App to build their product. This is how digital sales did it. This is how customer experience did it. This is how helpdesk did it. They start with the Unified App and they extend that and create an App UI that contains the flows and pages for their specific application, and then add functionality for all the pages and flows, as needed for the design. Partners and customers can then create a new extension that extends the Oracle Application and add their own App UI and their own URL if they want their pages accessed first, before navigating to the Oracle application. For example, if the digital sales application has functionality you'd like to leverage, like it has data services or fragments or page layouts that you want to reuse or other things, you extend the digital sales application, and this extension holds your code changes. You could then create a new App UI, and once deployed, users can use that URL for the new App UI to access your new pages. And your page can then navigate to the Oracle app when it needs to. Though I will say to date, we're really not seeing much demand for this particular use case, but it is possible. 12:33 Lois: Is that the only option available to customers and partners—to extend an existing Oracle application? Joe: No, Lois. We're seeing customers and partners create brand new Fusion applications of their own, based on the Unified App Oracle created. In a sense, doing the same thing that our development teams here are doing.  Remember, I said an Oracle development team starts with the Unified App, which has common functionality and look and feel for all applications, and then extends that to add business rules processing, flows, App UI, whatever they need for their specific Oracle application. We're seeing our partners and customers wanting to build their own applications. Maybe a customer or partner wants to create a Time & Expense application and leverage the Fusion application data and the APIs available, but define their own flows, their own pages, their own processing. This is very easy to do. They'd start by extending the Unified App just like the Oracle development teams do, and then build their own App UI and within that, their own flows, pages, and custom processing. The nice thing about it is that the application looks and works and feels just like a Fusion application and it appears alongside other Fusion applications, because it is a Fusion application. 13:43 Did you know that the Oracle University Learning Community regularly holds live events hosted by Oracle expert instructors. Find out how to prepare for your certification exams. Learn about the latest technology advances and features. Ask questions in real time and learn from an Oracle subject matter expert. From Ask Me Anything about certification to Ask the Instructor coaching sessions, you'll be able to achieve your learning goals for 2024 in no time. Join a live event today and witness firsthand the transformative power of the Oracle University Learning Community. Visit mylearn.oracle.com to get started.  14:24 Nikita: Welcome back! So Joe, it sounds like there are two different paths or life cycles to create extensions for future applications in Visual Builder Studio. Is that correct? Joe: Yes, exactly. So one path to extending the functionality of Fusion apps is to edit the page in Visual Builder Studio, which opens the page in Visual Builder Designer, and you then make changes to the existing pages, depending on what the development team has made extendable.  14:49 Nikita: But you can't create new pages and flows in this scenario, right? Joe: This is strictly about modifying an existing page. The other path is creating a new application extension, which is a new application from scratch or extending an existing Oracle application or even an existing partner or customer application. Again, we're not seeing this typically being done too much. Most partners and customers create new applications or make customizations to existing pages. But the architecture does support it. So, your partner might create a new application based on the production app released by Oracle, and you could extend their application. Or a development team at your site could extend Oracle's application and you could then extend that team's application. This is mechanically possible, although I question the use case behind that. Usually, we see our apps being extended – becoming a dependency when there's code that can be leveraged or reused for a new app and its new App UI. 15:40 Lois: Joe, what did you mean when you say one extension is a dependency of another? Can you talk a bit about dependencies, what that means, how it looks to the developer? Joe: When you extend an application, it becomes a dependency to your application, and you get access to all the resources within that dependency that are marked as extendable by the developer who created that extension. Most useful are things like service connections to REST APIs from Fusion apps data sources, reusable code fragments, and layouts that you can leverage in those cases where you want to create a new App UI. When an extension is listed as a dependency, you'll see this graphically in Visual Builder Studio Designer. When you see an extension listed as a dependency, it means you can reference any of that extension's resources that have been marked extendable by the developer. Recall all resources are closed off or hidden by default, but development teams can mark resources as open to being extended and reused, and then you can see and use those resources. So, you can easily add and remove extensions as dependencies in Visual Builder Designer as needed. Now, this can be a nice way to modularize and reuse your resources and assets. To summarize: I can modify an existing page – this is most common, extend an existing application and create a new App UI, which is not common, or I can extend the unified app to create a new app and a new App UI and add other extensions as dependences, as needed, to leverage their services, fragments, and layouts when building my own pages – this is pretty common as well. 17:04 Nikita: There's one thing I'd like to come back to, Joe. You mentioned something called a mashup application earlier. Can you tell us a little more about that? Joe: To recap: I mentioned a couple of different ways that you can extend Fusion applications. One is changing layouts or creating rule-based layouts. You can also extend existing apps and create your own App UI on top of them or create your own Fusion app from scratch. But these are Fusion apps and they have restrictions.  These can only run within the Fusion applications ecosystem, which means they can only be accessed by people who are registered in the Fusion application ecosystem, and there are some other restrictions (for example, in terms of the APIs you can access). And you also have no access to customer data tables. Mashup applications use the stand-alone Visual Builder Cloud Service, which enables you to create custom visual applications. These are visual applications that run outside the Fusion apps ecosystem. Users only need to be identified to the Identity Cloud Service, IDCS, and then they can get access to these mashup apps, depending on the roles and privileges given to them, of course. These mashup applications can access Fusion apps API data, as well as customer database tables, Excel spreadsheet data, CSV files, and third-party APIs. And all this data can appear on the same page, in the same app, using the same Redwood components, so they look and work just like Fusion applications. 18:22 Lois: I know in the past there's been some friction to making changes in Fusion applications. Partner and customer developers use different tools than the ones Oracle engineers use and there have been some deployment issues. To wrap up things, can you tell us why customers should use Visual Builder Studio to customize Fusion apps? Joe: Glad to, Lois. The big benefit to customers is that they are using the exact same tools, Visual Builder Designer for page design work and Visual Builder Studio for project and code management, to build the customizations and extensions that Oracle is using to create the applications and extensions that are delivered to them. I can't emphasize enough how big a deal this is and how wonderful it is for the customer. We're constantly making the Visual Builder Designer interface easier and easier to work with. We're currently releasing a new version of Visual Builder Designer—the Express mode version. This version of Designer is lightweight and has only the necessary features required to allow you to make changes to pages and layouts, and create and manage dynamic rule-based layouts. If you need more (for example, you need to create service connections, fragments, and do a lot more of that type of advanced work), then use the advanced version of the Designer. Both are available to you, assuming that your user has the appropriate permission and the Fusion app you are using has implemented Express Designer. 19:37 Lois: OK Joe, what courses does Oracle University offer for me if I wanted to learn more about developing extensions for Fusion apps and creating mashup apps using Visual Builder Studio? Joe: Oracle University has several courses. We have the Develop Visual Applications Using Visual Builder Studio, which focuses on creating the stand-alone custom bespoke mashup visual applications. We also have our Design and Develop Redwood Applications course, which goes into detail about working with the Redwood page templates and components. All these courses are free and available today. And all you need to do is log in to mylearn.oracle.com to get started. 20:10 Nikita: We hope you enjoyed that conversation. Just a quick reminder before we close about the short survey we've put together to get your thoughts on the podcast. It'll take just a few minutes and will help us make the podcast even better. Just click the link in the show notes to participate. Join us next week for another throwback episode. Until then, this is Nikita Abraham... Lois: And Lois Houston, signing off! 20:33 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

The next generation of front-end user interfaces for Oracle Fusion Applications is being built using Visual Builder Studio and Oracle JavaScript Extension Toolkit. However, many of the terms associated with these tools can be confusing.   In this episode, Lois Houston and Nikita Abraham are joined by Senior Principal OCI Instructor Joe Greenwald. Together, they take you through the different terminologies, how they relate to each other, and how they can be used to deliver the new Oracle Fusion Applications as well as stand-alone, bespoke visual web applications.   Survey: https://customersurveys.oracle.com/ords/surveys/t/oracle-university-gtm/survey?k=focus-group-2-link-share-5   Develop Fusion Applications Using Visual Builder Studio: https://mylearn.oracle.com/ou/course/develop-fusion-applications-using-visual-builder-studio/138392/   Build Visual Applications Using Oracle Visual Builder Studio: https://mylearn.oracle.com/ou/course/build-visual-applications-using-oracle-visual-builder-studio/137749/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X (formerly Twitter): https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started. 00:26 Nikita: Hello and welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi there! If you've been following along with us, you'll know that we've had some really interesting seasons this year. We covered Autonomous Database, Artificial Intelligence, Visual Builder Studio and Redwood, OCI Container Engine for Kubernetes, and Oracle Database 23ai New Features. Nikita: And we've had some pretty awesome special guests. Do go back and check out those episodes if any of those topics interest you. 01:04 Lois: As we close out the year, we thought this would be a good time to revisit some of our best episodes. Over the next few weeks, you'll be able to listen to four of our most popular episodes of the year.  Nikita: Right, this is the best of the best–according to you–our listeners.   Lois: Today's episode is #1 of 4 and is a throwback to a discussion with Senior Principal OCI Instructor Joe Greenwald on Visual Builder Studio. Nikita: We asked Joe about Visual Builder Studio and Oracle JavaScript Extension Toolkit, also known as JET. Together, they form the basis of the technology for the next generation of front-end user interfaces for Oracle Fusion Applications, as well as many other Oracle applications, including most Oracle Cloud Infrastructure (OCI) interfaces. 01:48 Lois: We looked at the different terminologies and technologies, how they relate to each other, and how they deliver the new Oracle Fusion Applications and stand-alone, bespoke visual web applications.  So, let's dive right in. Nikita: Joe, I'm somewhat thrown by the terminology around Visual Builder, Visual Studio, and JET. Can you help streamline that for us? Lois: Yeah, things that are named the same sometimes refer to different things, and sometimes things with a different name refer to the same thing. 02:18 Joe: Yeah, I know where you're coming from. So, let's start with Visual Builder Studio. It's abbreviated as VBS and can go by a number of different names. Some of the most well-known ones are Visual Builder Studio, VBS, Visual Builder, Visual Builder Stand-Alone, and Visual Builder Cloud Service. Clearly, this can be very confusing. For the purposes of these episodes as well as the training courses I create, I use certain definitions.  02:42 Lois: Can you take us through those? Joe: Absolutely, Lois. Visual Builder Studio refers to a product that comes free with an OCI account and allows you to manage your project-related assets. This includes the project itself, which is a container for all of its assets. You can assign teams to your projects, as well as secure the project and declare roles for the different team members. You manage GIT repositories with full graphical and command-line GIT support, define package, build, and deploy jobs, and create and run continuous integration/continuous deployment graphical and code-managed pipelines for your applications. These can be visual applications, created using the Visual Builder Integrated Development Environment, the IDE, or non-visual apps, such as Java microservices, docker builds, NPM apps, and things like that. And you can define environments, which determine where your build jobs can be deployed. You can also define issues, which allow you to identify, track, and manage things like bugs, defects, and enhancements. And these can be tracked in code review merge requests and build jobs, and be mapped to agile sprints and scrum boards. There's also support for wikis for team collaboration, code snippets, and the management of the repository and the project itself. So, VBS supports code reviews before code is merged into GIT branches for package, build, and deploy jobs using merge requests. 04:00 Nikita: OK, what exactly do you mean by that? Joe: Great. So, for example, you could have developers working in one GIT branch and when they're done, they would push their private code changes into that remote branch. Then, they'd submit a merge request and their changes would be reviewed. Once the changes are approved, their code branch is merged into the main branch and then automatically runs a CI/CD package (continuous integration/continuous deployment) package, build, and deploy job on the code. Also, the CI/CD package, build, and deploy jobs can run against any branches, not just the main branch. So Visual Builder Studio is intended for managing the project and all of its assets. 04:37 Lois: So Joe, what are the different tools used in developing web applications? Joe: Well, Visual Builder, Visual Builder Studio Designer, Visual Builder Designer, Visual Builder Design-Time, Visual Builder Cloud Service, Visual Builder Stand-Alone all kind of get lumped together. You can kinda see why. What I'm referring to here are the tools that we use to build a visual web application composed of HTML5, CSS3, JavaScript, and JSON (JavaScript Object Notation) for metadata. I call this Visual Builder Designer. This is an Integrated Development Environment, it's the “IDE” which runs in your browser. You use a combination of drag and drop, setting properties, and writing and modifying custom and generated code to develop your web applications. You work within a workspace, which is your own private copy of a remote Git branch. When you're ready to start development work, you open an existing workspace or create a new one based on a clone of the remote branch you want to work on. Typically, a new branch would be created for the development work or you would join an existing branch. 05:38 Nikita: What's a workspace, Joe? Is it like my personal laptop and drive? Joe: A workspace is your own private code area that stores any changes you make on the Oracle servers, so your code changes are never lost—even when working in a browser-based, network-based tool. A good analogy is, say I was working at home on my own machine. And I would make a copy of a remote GIT branch and then copy that code down to my local machine, make my code changes, do my testing, etc. and then commit my work—create a logical save point periodically—and then when I'm ready, I'd push that code up into the remote branch so it can be reviewed and merged with the main branch. My local machine is my workspace. However, since this code is hosted up by Oracle on our servers, and the code and the IDE are all running in your browser, the workspace is a simulation of a local work area on your own computer. So, the workspace is a hosted allocation of resources for you that's private. Other people can't see what's going on in your workspace. Your workspace has a clone of the remote branch that you're working with and the changes you make are isolated to your cloned code in your workspace. 06:41 Lois: Ok… the code is actually hosted on the server, so each time you make a change in the browser, the change is written back to the server? Is it possible that you might lose your edits if there's a networking interruption? Joe: I want to emphasize that while I started out not personally being a fan of web-based integrated development environments, I have been using these tools for over three years and in all that time, while I have lost a connection at times—networks are still subject to interruptions—I've never lost any changes that I've made. Ever. 07:11 Nikita: Is there a way to save where you are in your work so that you could go back to it later if you need to? Joe: Yes, Niki, you're asking about commits and savepoints, like in a Git repository or a Git branch. When you reach a logical stopping or development point in your work, you would create a commit or a savepoint. And when you're ready, you would push that committed code in your workspace up to the remote branch where it can be reviewed and then eventually merged, usually with the main Git branch, and then continuous integration/continuous package and deployment build jobs are run. Now, I'm only giving you a high-level overview, but we cover all this and much more in detail with hands-on practices in our Visual Builder developer courses. Right now, I'm just trying to give you a sense of how these different tools are used. 07:52 Lois: Yeah, that makes sense, Joe. It's a lot to cover in a short amount of time. Now, we've discussed the Visual Builder Designer IDE and workspace. But can you tell us more about Visual Builder Cloud Service and stand-alone environments? What are they used for? What features do they provide? Are they the same or different things? Joe: Visual Builder Cloud Service or Visual Builder Stand-Alone, as it's sometimes called, is a service that Oracle hosts on its servers. It provides hosting for the deployed web application source code as well as database tables for business objects that we build and maintain to store your customer data. This data can come from XLS or CSV files, or even your own Oracle database customer table data. A custom REST proxy makes calls to external third-party REST services on your behalf and supports several popular authentication mechanisms. There is also integration with the Identity Cloud Service (IDCS) to manage users and their access to your web apps. Visual Builder Cloud Service is a for-fee product. You pay licensing fees for how much you use because it's a hosted service. Visual Builder Studio, the project asset management aspect I discussed earlier, is free with a standard OCI license. Now, keep in mind these are separate from something like Visual Builder Design Time and the service that's running in Fusion application environments. What I'm talking about now is creating stand-alone, bespoke, custom visual applications. These are applications that are built using industry-standard HTML5, CSS3, JavaScript, and JSON for metadata and are hosted on the Oracle servers.  09:30 Are you looking for practical use cases to help you plan and apply configurations that solve real-world challenges?  With the new Applied Learning courses for Cloud Applications, you'll be able to practically apply the concepts learned in our implementation courses and work through case studies featuring key decisions and configurations encountered during a typical Oracle Cloud Applications implementation. Applied learning scenarios are currently available for General Ledger, Payables, Receivables, Accounting Hub, Global Human Resources, Talent Management, Inventory, and Procurement, with many more to come!  Visit mylearn.oracle.com to get started. 10:12 Nikita: Welcome back! Joe, you said Visual Builder Cloud Service or Stand-Alone is a for-fee service. Is there a way I can learn about using Visual Builder Designer to build bespoke visual applications without a fee? Joe: Yes. Actually, we've added an option where you can run the Visual Builder Designer and learn how to create web apps without using the app hosting or the business object database that stores your customer data or the REST proxy for authentication or the Identity Cloud Service. So you don't get those features, but you can still learn the fundamentals of developing with Visual Builder Designer. You can call third-party APIs, you can download the source, and run it locally, for example, in a Tomcat server. This is a great and free way to learn how to develop with the Visual Builder Designer. 10:55 Lois: Joe, I want to know more about the kinds of apps you can build in VB Designer and the capabilities that VB Cloud Service provides. Joe: Visual Builder Designer allows you to build custom, bespoke web applications made of interactive webpages; flows of pages for navigation; events that respond when things happen in the app, for example, GUI events like a button is clicked or values are entered into a text field; variables to store the state of the application and the ability to make REST calls, all from your browser. These applications have full access to the Oracle Fusion Applications APIs, given that you have the right security permissions and credentials of course. They can access your customer business data as business objects in our internally hosted database tables or your own customer database tables. They can access third-party APIs, and all these different data sources can appear in the same visual application, on the same page, at the same time. They use the Identity Cloud Service to identify which users can log in and authenticate against the application. And they all use the new Redwood graphical user interface components and page templates, so they have the same look and feel of all Oracle applications. 12:02 Nikita: But what if you're building or extending Oracle Fusion Applications? Don't things change a little bit? Joe: Good point, Niki. Yes. While you still work within Visual Builder Studio, that doesn't change, VBS maintains your project and all your project-related assets, that is still the same. However, in this case, there is no separate hosted Visual Builder Cloud Service or Stand-Alone instance. In this case, Visual Builder is hosted inside of Fusion apps itself as part of the installation. I won't go into the details of how the architecture works, but the Visual Builder instance that you're running your code against is part of Fusion applications and is included in the architecture as well as the billing. All your code changes are maintained and stored within a single container called an extension. And this extension is a Git repository that is created for you, or you can create it yourself, depending on how you choose to work within Visual Builder Studio. You create an extension to hold the source code changes that provide a customization or configuration. This means making a change to an existing page or a set of pages or even adding new pages and flows to your Oracle Fusion Applications. You use Visual Builder Studio and Visual Builder Designer in a similar way as to how you would use them for bespoke stand-alone visual applications. 13:12 Lois: I'm trying to envision how this workflow is used. How is it different from bespoke VB app development? Or is it different at all? Joe: So, recall that the Visual Builder Designer is effectively the Integrated Development Environment, the IDE, where you make your code changes by working with both the raw HTML5, CSS3, and JavaScript code, if need be, or the Page Designer for drag and drop, and setting properties and then Live mode to test your work. You use a version of VB Designer to view and modify your customizations, and the code is stored in a Git repository called an extension. So, in that sense, the work of developing pages and flows and such is the same. You still start by creating or, more typically, joining a project and then either create a new extension from scratch or base it on an existing application, or go directly to the page that you want to edit and, on that page, select from your profile menu to edit in Visual Builder Studio. Now, this is a different lifecycle path from bespoke visual applications. With them, you're not extending an app or modifying individual pages in the same way. You get a choice of which project you want to add your extension to when you're working with Fusion apps and potentially which repository to store your customizations, unless one already exists and then it's assigned automatically to hold your code changes. So you make your changes and edits to the portions of the application that have been opened for extensibility by the development team. This is another difference. Once you make your code changes, the workflow is pretty much the same as for a bespoke visual application: do your development work, commit your changes, push your changes to the remote branch. And then typically, your code is reviewed and if the code passes and is approved, it's merged with the main branch. Then, the package and deploy jobs run to deploy the main code to the production environment or whatever environment you're targeting. And once the package and deploy jobs complete, the code base is updated and users who log in see the changes that you've made. 15:03 Nikita: You mentioned creating apps that combine data from Fusion cloud, applications, customer data, and third-party APIs into one page. Why is it necessary? Why can't you just do all that in one Fusion Applications extension? Joe: When you create extensions, you are working within the Oracle Fusion Applications ecosystem, that's what they actually call it, which includes a defined a set of users who have been predefined and are, therefore, known to Fusion Applications. So, if you're a user and you're not part of that Fusion Apps ecosystem, you can't access the pages. Period. That's how Fusion Apps works to maintain its security and integrity. Secondly, you're working pretty much solely with the Fusion Applications APIs data sources coming directly from Fusion Applications, which are also available to you when you're creating bespoke visual apps. When you're working with Fusion Applications in Visual Builder, you don't have access to these business objects that give you access to your own customer database data through Visual Builder-generated REST APIs. Business objects are available only to bespoke visual applications in the hosted VB Cloud Service instance. So, your data sources are restricted to the Oracle Fusion Applications APIs and some third-party APIs that work within a narrow set of authentication mechanisms currently, although there are plans to expand this in the future. A mashup app that allows you now to access all these data sources while creating apps that leverage the Redwood Component System, so they look and work like Fusion Apps. They're a highly popular option for our partners and customers. 16:28 Lois: So, to review, we have two different approaches. You can create a visual application using the for-fee, hosted Visual Builder Cloud Service/Stand-Alone or the one that comes with Oracle Integration Cloud, or you can use the extension architecture for Fusion applications, where you use the designer and create your extensions, and the code is delivered and deployed to Fusion applications code. You haven't talked about JET yet though, Joe. What is that? Joe: So, JET is an abbreviation. It stands for Oracle JavaScript Extension Toolkit and JET is the underlying technology that makes Visual Builder, visual applications, and Visual Builder Extensions for Fusion Applications possible. Oracle JavaScript Extension Toolkit provides a module-based, open-source toolkit that leverages modern JavaScript, TypeScript, CSS3, and HTML5 to deliver web applications. It's targeted at JavaScript developers working on client-side applications. It is not for backend development.  It's a collection of popular, powerful JavaScript libraries and a set of Oracle-contributed JavaScript libraries that make it very simple, easy, and efficient to build front-end applications that can consume and interact with Oracle products and services, especially Oracle Cloud services, but of course it can work with any type of third-party API. 17:44 Nikita: How are JET applications architected, Joe, and how does that relate to Visual Builder pages and flows? Joe: The architecture of JET applications is what's called a single page architecture. We've all seen these. These are where you have a single webpage—think of your index page that provides the header and footer for your webpage—and then the middle portion or the middle content of the page, represented by modules, allow you to navigate from one page or module to another. It also provides the data mapping so that the data elements in the variables and the state of the application, as well as the graphical user interface elements that provide the fields and functionality for the interface for the application, these are all maintained on the client side. If you're working in pure JET, then you work with these modules at the raw JavaScript code level. And there are a lot of JavaScript developers who want to work like this and create their custom applications from the code up, so to speak. However, it also provides the basis for Visual Builder visual applications and Fusion Apps visual extensions in Visual Builder. 18:41 Lois: How does JET support VB Apps? You didn't talk much about having to write a bunch of JavaScript and HTML5, so I got the impression that this is all done for you by VB Designer? Joe: Visual Builder applications are composed of HTML5, CSS3, and JavaScript code that is usually generated by the developer when she drags and drops components on to the page designer canvas or sets properties or creates action chains to respond to events. But there's also a lot of JavaScript object notation (JSON) metadata created at the time that describes the pages, the flows, the navigation, the REST services, the variables, their data types, and other assets needed for the app to function. This JSON metadata is translated at runtime using a large JavaScript extension toolkit library called the Visual Builder Runtime that runs in the browser and real time translates the metadata and other assets in the Visual Builder source code into JET code and assets, which are actually executed at runtime. And it's very quick, very fast, very efficient, and provides a layer of abstraction between the raw JET code and the Visual Builder architecture of pages, flows, action chains for executing code and events to handle things that occur in the user interface, including saving the state in variables that are mapped to GUI components. For example, if you have an Input text component, you need to have a variable to store the value that was entered into that Input text component between page refreshes. The data can move from the Input text component to the variable, and from the variable to that Input text component if it's changed programmatically, for example. So, JET manages binding these data values to variables and the UI components on the page. So, a change to a variable value or a change to the contents of the component causes the others to change automatically. Now, this is only a small part of what JET and the frameworks and libraries it uses do for the applications. JET also provides more complex GUI components like lists and tables, and selection lists, and check boxes, and all the sorts of things you would expect in a modern GUI application. 20:37 Nikita: You mentioned a layer of abstraction between Visual Builder Studio Designer and JET. What's the benefit of working in Visual Builder Designer versus JET itself? Joe: The benefit of Visual Builder is that you work at a higher level of abstraction than having to get down into the more detailed levels of deep JavaScript code, working with modules, data mappings, HTML code, single page architecture navigation, and the related functionalities. You can work at a higher level, a graphical level, where you can drag and drop things onto a design canvas and set properties. The VB architecture insulates you from the more technical bits of JET. Now, this frees the developer to concentrate more on application and page design, implementing logic and business rules, and creating a pleasing workflow and look and feel for the user. This keeps them from having to get caught up in the details of getting this working at the code level. Now if needed, you can write custom JavaScript, HTML5, and CSS3 code, though much less than in a JET app, and all that is part of the VB application source, which becomes part of the code used by JET to execute the application itself. And yet it all works seamlessly together. 21:38 Lois: Joe, I know we have courses in JavaScript, HTML, and CSS. But does a developer getting ready to work in Visual Builder Designer have to go take those courses first or can they start working in VB Designer right away? Joe: Yeah, that question does often comes up: Do I need to learn JET to work with Visual Builder? No, you don't. That's all taken care for you in the products themselves. I don't really think it helps that much to learn JET if you are going to be a VB developer. In some ways, it could even be a bit distracting since some of things you learn to do in JET, you would have to unlearn or not do so much because of what VB does it for you. The things you would have to do manually in code in JET are done for you. This is why we call VB a low code development tool. I mean, you certainly can if you want to, but I would spend more time learning about the different GUI components, page templates, the Visual Builder architecture — events, action chains, and the data provider variables and types. Now, I know JET myself. I started with that before learning Visual Builder, but I use very little of my JET knowledge as a VB developer. Visual Builder Designer provides a nice, abstracted, clean layer of modern visual development on top of JET, while leveraging the power and flexibility of JET and keeping the lower-level details out of my way. 22:49 Nikita: Joe, where can I go to get started with Visual Builder? Joe: Well, for more information, I recommend you take a look at our Develop Fusion Applications course if you're working with Fusion Applications and Visual Builder Studio. The other course is Develop Visual Applications with Visual Builder Studio and that's if you're creating stand-alone bespoke applications. Both these courses are free. We also have a comprehensive course that covers JavaScript, HTML5, and CSS3, and while it's not required that you take that to be successful, it can be helpful down the road. I would also say that some basic knowledge of HTML5, CSS3, and JavaScript will certainly support you and serve you well when working with Visual Builder. You learn more as you go along and you find that you need to create more sophisticated applications. I would also mention that a lot of the look and feel of the applications in Visual Builder visual applications and Fusion apps extensions and customizations come through JET components, JET styles, and JET variables, and CSS variables, so that's something that you would want to pursue at some point. There's a JET cookbook out there. You can search for Oracle JET and look for the JET cookbook and that's a good introduction to all of that. 23:50 Nikita: We hope you enjoyed that conversation. To learn about some of the courses Joe mentioned, visit mylearn.oracle.com to get started. Lois: Before we wrap up, we've got a favor to ask. We've created a short survey to capture your thoughts on the podcast. It'll only take a few minutes of your time. Just click the link in the show notes and share your feedback. We want to make sure we're delivering the best experience possible so don't hesitate to let us know what's on your mind! Thanks for your support. Join us next week for another throwback episode. Until then, this is Lois Houston… Nikita: And Nikita Abraham, signing off! 24:30 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this special episode of the Oracle University Podcast, Lois Houston and Nikita Abraham, along with Principal HCM Instructor Jeff Schuster, delve into the intersection of HCM and AI, exploring the practical applications and implications of this technology in human resources. Jeff shares his insights on bias and fairness, the importance of human involvement, and the need for explainability and transparency in AI systems. The discussion also covers the various AI features embedded in HCM and their impact on talent acquisition, performance management, and succession planning.  Oracle AI in Fusion Cloud Human Capital Management: https://mylearn.oracle.com/ou/learning-path/oracle-ai-in-fusion-cloud-human-capital-management-hcm/136722 Oracle Fusion Cloud HCM: Dynamic Skills: https://mylearn.oracle.com/ou/course/oracle-fusion-cloud-hcm-dynamic-skills/116654/ Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ Twitter: https://twitter.com/Oracle_Edu Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode. -------------------------------------------------------- 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started!  00:26 Lois: Hello and welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs here at Oracle University, and with me, is Nikita Abraham, Team Lead of Editorial Services. Nikita: Hi everyone! Last week's conversation was all about Oracle Database 23ai backup and recovery, where we dove into instance recovery and effective recovery strategies. Today's episode is a really special one, isn't it, Lois? 00:53 Lois: It is, indeed, Niki. Of course, all of our AI episodes are special. But today, we have our friend and colleague Jeff Schuster with us. I think our listeners are really going to enjoy what Jeff has to share with us. Nikita: Yeah definitely! Jeff is a Principal HCM Instructor at Oracle University. He recently put together this really fantastic course on MyLearn, all about the intersection of HCM and AI, and that's what we want to pick his brain about today. Hi Jeff! We're so excited to have you here.  01:22 Jeff: Hey Niki! Hi Lois! I feel special already. Thanks you guys so much for having me. Nikita: You've had a couple of busy months, haven't you?  01:29 Jeff: I have! It's been a busy couple of months with live classes. I try and do one on AI in HCM at least once a month or so so that we can keep up with the latest/greatest stuff in that area. And I also got to spend a few days at Cloud World teaching a few live classes (about artificial intelligence in HCM, as a matter of fact) and meeting our customers and partners. So yeah, absolutely great week. A good time was had by me.  01:55 Lois: I'm sure. Cloud World is such a great experience. And just to clarify, do you think our customers and partners also had a good time, Jeff? It wasn't just you, right? Jeff: Haha! I don't think it was just me, Lois. But, you know, HCM is always a big deal, and now with all the embedded AI functionality, it really wasn't hard to find people who wanted to spend a little extra time talking about AI in the context of our HCM apps. So, there are more than 30 separate AI-powered features in HCM. AI features for candidates to find the right jobs; for hiring managers to find the right candidates; skills, talent, performance management, succession planning— all of it is there and it really covers everything across the Attract/Grow/Keep buckets of the things that HR professionals do for a living. So, anyway, yeah, lots to talk about with a lot of people! There's the functional part that people want to know about—what are these features and how do they work? But obviously, AI carries with it all this cultural significance these days. There's so much uncertainty that comes from this pace of development in that area. So in fact, my Cloud World talk always starts with this really silly intro that we put in place just to knock down that anxiety and get to the more practical, functional stuff. 03:11 Nikita: Ok, we're going to need to discuss the functional stuff, but I feel like we're getting a raw deal if we don't also get that silly intro. Lois: She makes a really good point.  Jeff: Hahaha! Alright, fair enough. Ok, but you guys are gonna have to imagine I've got a microphone and a big room and a lot of echo. AI is everywhere. In your home. In your office. In your homie's home office. 03:39 Lois: I feel like I just watched the intro of a sci-fi movie. Jeff: Yeah. I'm not sure it's one I'd watch, but I think more importantly it's a good way to get into discussing some of the overarching things we need to know about AI and Oracle's approach before we dive into the specific features, so you know, those features will make more sense when we get there?  03:59  Nikita: What are these “overarching” things?  Jeff: Well, the things we work on anytime we're touching AI at Oracle. So, you know, it starts with things like Bias and Fairness. We usually end up in a pretty great conversation about things like how we avoid bias on the front end by making sure we don't ingest things like bias-generating content, which is to say data that doesn't necessarily represent bias by itself, but could be misused. And that pretty naturally leads us into a talk about guardrails. Nikita: Guardrails? Jeff:  Yeah, you can think of those as checkpoints. So, we've got rules about ingestion and bias. And if we check the output coming out of the LLM to ensure it complied with the bias and fairness rules, that's a guardrail. So, we do that. And we do it again on the apps side. And so that's to say, even though it's already been checked on the AI side, before we bring the output into the HCM app, it's checked again. So another guardrail.  04:58 Lois: How effective is that? The guardrails, and not taking in data that's flagged as bias-generating? Jeff: Well, I'll say this: It's both surprisingly good, and also nowhere near good enough.  Lois: Ok, that's as clear as mud. You want to elaborate on that?  Jeff: Haha! I think all it means is that approach does a great job, but our second point in the whole “standards” discussion is about the significance of having a human in the loop. Sometimes more than one, but the point here is that, particularly in HCM, where we're handling some really important and sensitive data, and we're introducing really powerful technology, the H in HCM gets even more important. So, throughout the HCM AI course, we talk about opportunities to have a human in the loop. And it's not just for reviewing things. It's about having the AI make suggestions, and not decisions, for example. And that's something we always have a human in the loop for all the time. In fact, when I started teaching AI for HCM, I always said that I like to think of it is as a great big brain, without any hands.  06:00 Nikita: So, we're not talking about replacing humans in HCM with AI.                                                                         Jeff: No, but we're definitely talking about changing what the humans do and why it's more important than ever what the humans do. So, think of it this way, we can have our embedded AI generate this amazing content, or create really useful predictions, whatever it is that we need. We can use whatever tools we want to get there, but we can still expect people to ask us, “Where did that come from?” or “Does this account for [whatever]?”. So we still have to be able to answer that. So that's another thing we talk about as kind of an overarching important concept: Explainability and Transparency. 06:41 Nikita: I'm assuming that's the part about showing our work, right? Explaining what's being considered, how it's being processed, and what it is that you're getting back. Jeff: That's exactly it. So we like to have that discussion up front, even before we get to things like Gen and Non-Gen AI, because it's great context to have in mind when you start thinking about the technology. Whenever we're looking at the tech or the features, we're always thinking about whether people are appropriately involved, and whether people can understand the AI product as well as they need to.  07:11 Lois: You mentioned Gen and Non-Gen AI. I've also heard people use the term “Classic AI.” And lately, a lot more about RAG and Agents. When you're teaching the course, does everybody manage to keep all the terminology straight? Jeff: Yeah, people usually do a great job with this. I think the trick is, you have to know that you need to know it, if that makes sense.  Lois: I think so, but why don't you spell it out for us. Jeff: Well, the temptation is sometimes to leave that stuff to the implementers or product developers, who we know need to have a deep understanding of all of that. But I think what we've learned is, especially because of all the functional implications, practitioners, product owners, everybody needs to know it too. If for no other reason so they can have more productive conversations with their implementers. You need to know that Classic or Non-Generative AI leverages machine learning, and that that's all you need in order to do some incredibly powerful things like predictions and matching. So in HCM, we're talking about things like predicting time to hire, identifying suggested candidates for job openings, finding candidates similar to ones you already like, suggesting career paths for employees, and finding recommended successors. All really powerful matching stuff. And all of that stuff uses machine learning and it's certainly AI, but none of that uses Generative AI to do that because it doesn't need to. 08:38 Nikita: So how does that fit in with all the hype we've been hearing for a long time now about Gen AI and how it's such a transformative technology that's going to be more impactful than anything else? Jeff: Yeah, and that can be true too. And this is what we really lean into when we do the AI in HCM course live. It's much more of a “right AI for the right job” kind of proposition. Lois: So, just like you wouldn't use a shovel to mix a cake. Use the right tool for the job. I think I've got it. So, the Classic AI is what's driving those kinds of features in HCM? The matching and recommendations?  Jeff: Exactly right. And where we need generative content, that's where we add on the large language model capability. With LLMs, we get the ability to do natural language processing. So it makes sense that that's the technology we'd use for tasks like “write me a job description” or “write me performance development tips for my employee”. 09:33 Nikita: Ok, so how does that fit in with what Lois was asking about RAG and Agents? Is that something people care about, or need to? Jeff: I think it's easiest to think about those as the “what's next” pieces, at least as it relates to the embedded AI. They kind of deal with the inherent limitations of Gen and Non-Gen components. So, RAG, for example - I know you guys know, but your listeners might not...so what's RAG stand for? Lois & Nikita: Retrieval. Augmented. Generation. Jeff: Hahaha! Exactly. Obviously. But I think everything an HCM person needs to know about that is in the name. So for me, it's easiest to read that one backwards. Retrieval Augmented Generation. Well, the Generation just means it's more generative AI. Augmented means it's supplementing the existing AI. And Retrieval just tells you that that's how it's doing it. It's going out and fetching something it didn't already have in order to complete the operation. 10:31 Lois: And this helps with those limitations you mentioned? Nikita: Yeah, and what are they anyway?  Jeff: I think an example most people are familiar with is that large language models are trained on this huge set of information. To a certain point. So that model is trained right up to the point where it stopped getting trained. So if you're talking about interacting with ChatGPT, as an example, it'll blow your doors off right up until you get to about October of 2023 and then, it just hasn't been trained on things after that. So, if you wanted to have a conversation about something that happened after that, it would need to go out and retrieve the information that it needed. For us in HCM, what that means is taking the large language model that you get with Oracle, and using retrieval to augment the AI generation for the things that the large language model wouldn't have had.  11:22 Nikita: So, things that happened after the model was trained? Company-specific data? What kind of augmenting are you talking about? Jeff: It's all of that. All those things happen and it's anything that might be useful, but it's outside the LLM's existing scope. So, let's do an example. Let's say you and Lois are in the market to hire someone. You're looking for a Junior Podcast Assistant. We'd like the AI in HCM to help, and in order to do that, it would be great if it could not just generate a generic job description for the posting, but it could really make it specific to Oracle. Even better, to Oracle University.  So, you'd need the AI to know a few more things in order to make that happen. If it knows the job level, and the department, and the organization—already the job posting description gets a lot better. So what other things do you think it might need to know? 12:13 Lois: Umm I'm thinking…does it need to account for our previous hiring decisions? Can it inform that at all? Jeff: Yes! That's actually a key one. If the AI is aware not only of all the vacancies and all of the transactional stuff that goes along with it (like you know who posted it, what's its metadata, what business group it was in, and all that stuff)...but it also knows who we hired, that's huge. So if we put all that together, we can start doing the really cool stuff—like suggesting candidates based not only on their apparent match on skills and qualifications, but also based on folks that we've hired for similar positions. We know how long it took to make those hires from requisition open to the employee's first start date. So we can also do things like predicting time to hire for each vacancy we have with a lot more accuracy. So now all of a sudden, we're not just doing recruiting, but we have a system that accounts for “how we do it around here,” if that makes any sense.  But the point is, it's the augmented data, it's that kind of training that we do throughout ingestion, going out to other sources for newer or better information, whatever it is we need. The ability to include it alongside everything that's already in the LLM, that's a huge deal.  13:31  Nikita: Ok, so I think the only one we didn't get to was Agents. Jeff: Yeah, so this one is maybe a little less relevant in HCM—for now anyway. But it's something to keep an eye on. Because remember earlier when I described our AI as having a great big brain but no hands?  Lois: Yeah... Jeff: Well, agents are a way of giving it hands. At least for a very well-defined, limited set of purposes. So routine and repetitive tasks. And for obvious reasons, in the HCM space, that causes some concerns. You don't want, for example, your AI moving people forward in the recruiting process or changing their status to “not considered” all by itself. So going forward, this is going to be a balancing act. When we ask the same thing of the AI over and over again, there comes a point where it makes sense to kind of “save” that ask. When, for example, we get the “compare a candidate profile to a job vacancy” results and we got it working just right, we can create an agent. And just that one AI call that specializes in getting that analysis right. It does the analysis, it hands it back to the LLM, and when the human has had what they need to make sure they get what they need to make a decision out of it, you've got automation on one hand and human hands on the other...hand. 14:56 Have you mastered the basics of AI? Are you ready to take your skills to the next level? Unlock the potential of advanced AI with our OCI Generative AI Professional course and certification that covers topics like large language models, the OCI Generative AI Service, and building Q&A chatbots for real-world applications. Head over to mylearn.oracle.com to find out more. 15:26 Nikita: Welcome back! Jeff, you've mentioned the “Time to Hire” feature a few times? Is that a favorite with people who take your classes? Jeff: The recruiting folks definitely seem to enjoy it, but I think it's just a great example for a couple of reasons. First, it's really powerful non-generative AI. So it helps emphasize the point around the right AI for the right job. And if we're talking about things in chronological order, it's something that shows up really early in the hire-to-retire cycle. And, you know, just between us learning nerds, I like to use Time to Hire as an early example because it gets folks in the habit of working through some use cases. You don't really know if a feature is going to get you what you need until you've done some of that. So, for example, if I tell you that Time to Hire produces an estimated number of days to your first hire. And you're still Lois, and you're still Niki, and you're hiring for a Junior Podcast Assistant. So why do you care about time to hire? And I'm asking you for real—What would you do with that prediction if you had it?  16:29 Nikita: I guess I'd know how long it is before I can expect help to arrive, and I could plan my work accordingly. Jeff: Absolutely. What else. What could you do with a prediction for Time to Hire? Lois: Think about coverage? Jeff: Yeah! Exactly the word I was looking for. Say more about that.  Lois: Well, if I know it's gonna be three months before our new assistant starts, I might be able to plan for some temporary coverage for that work. But if I had a prediction that said it's only going to be two weeks before a new hire could start, it probably wouldn't be worth arranging temporary coverage. Niki can hold things down for a couple of weeks. Jeff: See, I'm positive she could! That's absolutely perfect! And I think that's all you really need to have in terms of prerequisites to understand any of the AI features in HCM. When you know what you might want to do with it, like predicting the need for temp cover, and you've got everything we talked about in the foundation part of the course—the Gen and the Classic, all that stuff, you can look at a feature like Time to Hire and then you can probably pick that up in 30 seconds. 17:29 Nikita: Can we try it? Jeff: Sure! I mean, you know, we're not looking at screens for this conversation, but we can absolutely try it. You're a recruiter. If I tell you that Time to Hire is a feature that you run into on the job requisition and it shows you just a few editable fields, and then of course, the prediction of the number of days to hire—tell me how you think that feature is going to work when you get there. Lois: So, what are the fields? And does it matter? Jeff: Probably not really, but of course you can ask. So, let me tell you. Ready? The fields—they are these. Requisition Title, Location, and Education Level.  Nikita: Ok, well, I have to assume that as I change those things… like from a Junior Podcast Assistant to a Senior Podcast Assistant, or change the location from Redwood Shores to Detroit, or change the required education, the time to hire is going to change, right?  Jeff: 100%, exactly. And it does it in real time as you make those changes to those values. So when you pick a new location, you immediately get a new number of days, so it really is a useful tool. But how does it work? Well, we know it's using a few fields from the job requisition, but that's not enough. Besides those fields, what else would you need in order to make this prediction work? 18:43 Lois: The part where it translates to a number of days. So, this is based on our historic hiring data? How long it took us to hire a podcast assistant the last time? Jeff: Yep! And now you have everything you need. We call that “historic data from our company” bit “ingestion,” by the way. And there's always a really interesting discussion around that when it comes up in the course. But it's the process we use to bring in the HCM data to the AI so it can be considered or predictions exactly like this. Lois: So it's the HCM data making the AI smarter and more powerful. Nikita: And tailored. Jeff: Exactly, it's all of that. And obviously, the HCM is better because we've given it the AI. But the AI is also better because it has the HCM in it. But look, I was able to give you a quick description of Time to Hire, and you were able to tell me what it does, which data it uses, and how it works in just a few seconds. So, that's kind of the goal when we teach this stuff. It's getting everybody ready to be productive from moment #1 because what is it and how does it work stuff is already out of the way, you know?  19:52 Lois: I do know! Nikita: Can we try it with another one? Jeff: Sure! How about we do...Suggested Candidates. Lois: And you're going to tell us what we get on the screen, and we have to tell you how it works, right? Jeff: Yeah, yeah, exactly. Ok—Suggested Candidates. You're a recruiter or a hiring manager. You guys are still looking for your Junior Podcast Assistant. On the requisition, you've got a section called Suggested Candidates. And you see the candidate's name and some scores. Those scores are for profile match, skills match, experience match. And there's also an overall match score, and the highest rated people you notice are sorted to the top of the list. So, you with me so far?  Lois: Yes! Jeff: So you already know that it's suggesting candidates. But if you care about explainability and transparency like we talked about at the start, then you also care about where these suggested candidates came from. So let's see if we can make progress against that. Let's think about those match scores. What would you need in order to come up with match scores like that? 20:54 Nikita: Tell me if I'm oversimplifying this, but everything about the job on the requisition, and everything about the candidate? Their skills and experience? Jeff: Yeah, that's actually simplified pretty perfectly. So in HCM, the candidate profile has their skills and experience, and the req profile has the req requirements.  Lois: So we're comparing the elements of the job profile and the person/candidate profile. And they're weighted, I assume? Jeff: That's exactly how it works. See, 30 seconds and you guys are nailing these! In fairness, when we discuss these things in the course, we go into more detail. And I think it's helpful for HCM practitioners to know which data from the person and the job profiles is being considered (and sometimes just as important, which is not being considered). And don't forget we're also considering our ingested data. Our previously selected candidates. 21:45 Lois: Jeff, can I change the weighting? If I care more about skills than experience or education, can I adjust the weighting and have it re-sort the candidates? Jeff: Super important question. So let me give you the answer first, which is “no.” But because it's important, I want to tell you more. This is a discussion we have in the class around Oracle's Embedded vs. Custom AI. And they're both really important offerings. With Embedded, what we're talking about are the features that come in HCM like any other feature. They might have some enablement steps like profile options, and there's an activation panel. But essentially, that's it. There's no inspection panel for you to open up and start sticking your screwdriver in there and making changes. Believe it or not, that's a big advantage with Embedded AI, if you ask me anyway.  Nikita: It's an advantage to not be able to configure it? Jeff: In this context, I think you can say that it is. You know, we talk about the advantages about the baked-in, Embedded AI in this course, but one of the key things is that it's pre-built and pre-tested. And the big one: that it's ready to use on day one. But one little change in a prompt can have a pretty big butterfly effect across all of your results. So, Oracle provides the Embedded AI because we know it works because we've already tested it, and it's, therefore, ready on day one. And I think that story maybe changes a little bit when you open up the inspection panel and bust out that screwdriver. Now you're signing up to be a test pilot. And that's just fundamentally different than “pre-built and ready on day one.” Not that it's bad to want configuration. 23:24 Lois: That's what the Custom AI path and OCI are about though, right? For when customers have hyper-specific needs outside of Oracle's business processes within the apps, or for when that kind of tuning is really required. And your AI for HCM course—that focuses on the Embedded AI instead of Custom, yes? Jeff: That is exactly it, yes. Nikita: You said there are about 30 of these AI features across HCM. So, when you teach the course, do you go through all of them or are there favorites? Ones that people want to spend more time on so you focus on those? Jeff: The professional part of me wants to tell you that we do try to cover all of them, because that explainability and transparency business we talked about at the beginning. That's for real, so I want our customers to have that for the whole scope.  24:12  Nikita: The professional part? What's the other part?  Jeff: I guess that's the part that says sure, we need to hit all of them. But some of them are just inherently more fun to work on. So, it's usually the learners who drive that in the live classes when they get into something, that's where we spend the most time. So, I have my favorites too. The learners have their favorites. And we spend time where it's everybody's favorite. Lois: Like where? Jeff: Ok, so one is far from the most complex one, but I think it's really elegant in its simplicity. And it's the Celebrate feature, where we do employee recognition. There's an AI Assist available there. So when it's time to recognize a colleague, you just need to enter the headline or the title, and the AI takes it from there and just writes up the recognition. 24:56 Lois: What about that makes it a good example, Jeff? You said it's elegant. What do you mean?  Jeff: I think it's a few things. So, start with the prompt. It's just the one line—just the headline. And that's your one input. So, type in the headline, get the recognition below. It's a great demonstration of not just the simplicity, but the power we get out of that simplicity. I always ask it to recognize my employees for implementing AI features in Oracle HCM, just to see what it comes up with. When it tells the employee that they're helping the company by automating routine tasks, bringing efficiency to the HR department, and then launches into specific examples of how AI features help in HCM, it really is pretty incredible. So, it's a simple demo, but it explains a lot about how the Gen AI works. Lois: That's really cool. 25:45 Nikita: So this one is generative AI. It's using the large language model to create the recognition based on the prompt, which is basically just whatever you entered in the headline. But how does that help explain how Gen AI works in HCM? Jeff: Well, let's take our simple prompt for example. There's a lot happening behind the scenes. It's taking our prompt, it's doing its LLM thing, but before it's done, it's creating the results in a very specific way. An employee recognition reads really differently than a job description. So, I usually describe this as the hidden part of our prompt. The visible part is what we typed. But it needs to know things like our desired output format. Make sure to use the person's name, summarize the benefits, and be sure to thank them for their contribution, that kind of stuff. So, those things are essentially hard-coded into the page. And that's to say, this is another area where we don't get an inspection panel that lets us go in and tweak the prompt.  26:42 Nikita: And that's generally how generative AI works? Jeff: Pretty much. Wherever you see an AI Assist button in HCM, that's more or less what's going on. And so when you get to some of the other more complex features, it's helpful to know that that is what's going on.  Lois: Like where? Jeff: Well, it works that way for the About Me part of your employee profile, for goal creation in performance, and I think a really great example is in performance, where managers are providing the competency development tips. So the prompt there is a little more complex there because it involves the employee's proficiency rating instead of free text. But still, pretty straightforward. You're gonna click AI Assist and it's gonna generate all the development tips for any specific competency listed for that employee. Good development tips. Five of them. Nicely formatted with bullet points. And these aren't random words assembled by an AI. So they conform to best practices in the development of competencies. So, something is telling the LLM to give us results that are that good, in that particular way.  So, it's just another good example of the work AI is doing while protected behind the inspection panel that doesn't exist. So, the coding of that page, in combination with what the LLM generates and the agent that it uses, is what produces the result. That's generally the approach. In the class, we always have a good time digging into what must be going on behind that inspection panel. Generally speaking, the better feel we have for what's going on on these pages, the better we're able to get the results we want, even without having that screwdriver out. 28:21  Nikita: So it's time well-spent, looking at all the individual features? Jeff: I think so, especially if you're anticipating really using any of them. So, the good news is, once you learn a few of them and how they work, and what they're best at, you stop being surprised after a while. But there are always tips and tricks. And like we talked about at the top, explainability and transparency are absolutely key. So, as much as I'm not a fan of the phrase, I do think this is kind of a “knowledge is power” kind of situation. 28:51 Nikita: Sadly, we're just about out of time for this episode.  Lois: That's too bad, I was really enjoying this. Jeff, you were just talking about knowledge—where can we get more?  Jeff: Well, like you mentioned at the start, check out the AI in HCM course on MyLearn. It's about an hour and a half, but it really is time well spent. And we get into detail on everything the three of us discussed here today, and then we have demoscussions of every feature where we show them and how they work and which data they're using and a whole bunch more. So, there's that. Plus, I hear the instructor is excellent. Lois: I can vouch for that! Jeff: Well, then you should definitely look into Dynamic Skills. Different instructor. But we have another course, and again I think about an hour and a half, but when you're done with the AI course, I always feel like Dynamic Skills is where you really wanna go next to really flesh out all the Talent Management ideas that got stirred up while you were having a great time in the AI course.  And then finally, the live classes. It's always really fun to take live questions while we talk about AI in HCM.   29:54 Nikita: Thanks, Jeff! This has been really interesting.  Lois: Yeah, thanks for being here, Jeff. We've loved having you on. Jeff: Thank you guys so much for having me. It's been a pleasure.  Lois: If you want to learn more about what we discussed, go to the show notes for today's episode. You'll find links to the AI for Human Capital Management and Dynamic Skills courses that Jeff mentioned so you can check them out. You can also head over to mylearn.oracle.com to find the live sessions for MyLearn subscribers that Jeff conducts. Nikita: Join us next week as we kick off our “Best of 2024” season, where we'll be revisiting some of our most popular episodes of the year. Until then, this is Nikita Abraham…  Lois: And Lois Houston, signing off!   30:35 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Lois Houston and Nikita Abraham continue their deep dive into Oracle Database 23ai backup and recovery strategies with Senior Principal Database & MySQL Instructor Bill Millar.   Picking up from Part 1, they explore critical concepts such as instance recovery, checkpoint processes, and the role of redo log files. Bill shares insights into complete and incomplete recovery, flashback technologies, and lots more.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-backup-and-recovery/141127/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   Twitter: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Radhika Banka, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 
00:26 Nikita: Welcome back to the Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi everyone! Last week, we had a fantastic chat with Bill Millar, our Senior Principal Database & MySQL Instructor. We dug into the basics of backup and recovery. We touched on everything from a DBA's role in preventing data loss to handling different types of failures, and even some common mistakes that tend to pop up when managing a database. Nikita: Yeah, if you missed that episode, definitely go back and check it out. It's packed with useful info, especially if you're in charge of keeping databases safe. 01:10 Lois: Today, we're picking up where we left off. We're going to ask Bill about instance recovery and recovery strategies. Bill, can you kick things off by explaining what instance recovery is? Bill: You can understand instant recovery by becoming familiar with the checkpoint process, the redo log files, and the role of the log writer with the redo log files. Automatically instance or crash recovery. What is it doing? What are the phases of instance recovery? How we possibly can tune that instance recovery. We can use the mean time to recovery advisor that can help us determine how we might tune the instance recovery. 01:51 Nikita: OK, so let's go through some of these concepts and procedures you mentioned. What is the checkpoint process responsible for exactly? Bill: The checkpoint process itself, it's responsible for updating the data file headers with checkpoint information. When a checkpoint is taken, it is going to write into the controlfiles. It tells the DB writer to write. DB writer writes to the data files, and the checkpoint is also annotated in the data files. So updating controlfiles with that checkpoint information also, controlfiles and database files. It signals that DB writer at full check points again, hey, it's time to write. So that way, it has the latest data written to the data files. The controlfile and datafiles, those are in sync with that. 02:40 Lois: Bill, what about the log writer process and the redo log files? Bill: With the log writer process and the redo log files, the redo log files record the changes to the database himself. It should be multiplexed. 02:53 Nikita: What do you mean by that? Bill: More than one redo log group. Now, the redo log groups, it is recommended that they should be multiplexed. Each group member should be on a different disk or in a different disk group if you're using ASM. 03:10 Nikita: And why is that, Bill? Bill: Because if I lose one, if I lose one redo log group, one member, I can continue to operate with just the one. If I only have one redo log group member and the system comes around and tries to write to it, then my system is going to come to a halt. So the log writer is going to write to those redo logs whenever somebody does a commit. When that redo log buffer is 1/3 full or every three seconds and before DB writer writes. So those are the four mechanism that tells log writer to write from that log buffer to the redo log files. And it'll also write, when we do a shut down, all the buffers will be flushed. And so that way, everything will be in sync when the system is shut down. 04:01 Lois: What are the different modes of operation for a database, Bill? And how do these modes impact the recovery capabilities of the database? Bill: So we have two different modes we can operate in. One is called NOARCHIVELOG mode. It is the default. ARCHIVELOG mode, highly encouraged. But not every environment has to be in ARCHIVELOG mode. 04:21 Nikita: So with ARCHIVELOG mode… Bill: Closed database. You have to close it, recover to the last backup. That's as far as I can go. Actually, I could, depending on what happens, I might be able to apply some redo. Suitable for training and test environments or for data warehouses, we don't have a lot of frequent changes. It's mainly bulk loading data at night and querying during the day. So it might be appropriate for that. Because ARCHIVELOG mode, it is a little overhead. Yes. So with that database, it goes down while it's open. The system, when it comes up, it can recover to the last committed transaction. And this is usually the mode we want to operate in for production environments. So we have that data in the buffer cache. We have that redo being buffered. We have the undo tablespace, keeping track of what the data was before a change. The redo keeps track of what was the change. And if we're in ARCHIVELOG mode, as we switch from one redo log to another, we will generate what's referred to as archived log files, and that's what allows us to do a complete recovery. 05:33 Lois: What happens in the case of automatic instance recovery? Bill: For an automatic instance recovery or crash recovery, our system went down unexpectedly. Because it did not do a clean shutdown, the buffers were not flushed. Everything was not synchronized. So the datafile, controlfile, everything is out of sync. 05:53 Nikita: So, how do the files get synchronized then? Bill: It uses the redo log groups to synchronize the files. It's going to roll forward. It rolls forward the changes that were made. So due to different distinct operations. Roll forward applies committed and uncommitted data. And the redo does not keep track of what was committed and uncommitted. It'll keep track of, hey, I had this transaction, hey, here's a commit for that transaction. But hey, I have a transaction. That was never uncommitted. That's the job of the undo. But rolls forward all those changes. And then anything that did not actually receive a commit, it will roll back the uncommitted data, return to the original state. And that is the job of the undo tablespace. 06:37 Lois: Bill, is it possible to tune instance recovery for better performance? Bill: You can try to tune this instance recovery. Tuning it is touchy. Be careful because you can cause more harm than what you think you might be doing good. The instance recovery, what we're doing, we're trying to-- the transactions between checkpoints. When was the last checkpoint? Because the items between the checkpoints, that's what has to be reapplied. So the last checkpoint to the last redo log, what is that time frame there between those? Well, what we're going to do, we're going to try to control that. We're going to try to control the difference between the checkpoint and the end of the redo log. There is a mean-time recovery advisor. You specify the desired times in seconds or minutes that how often you want that checkpoint to occur. There is a parameter, FAST_START_MTTR parameter that you can set. The default value is zero saying, hey, I'm going to let the system take care of it. And the maximum you can set it is to one hour. 07:46 Nikita: And why 1 hour?
 Bill: The reason being, if I set that to one hour and I have a lot of activity, how long is it going to take? How many transactions can happen within that hour? Yeah, I'm not doing a checkpoint as often, so I'm eliminating that workload. But if it has to recover, how long is it going to take? If I set it too small, the system says, hey, right now, it's going to take me 19 seconds based off statistics. If I said, OK, I want it in five seconds. So what does that mean? Every five seconds, I'm saying do a checkpoint. So what is it doing? OK, time to do a checkpoint. OK, time to go ahead and OK, DB writer write. OK, log writer write. OK, let me update the datafiles and the controlfiles. So you're just thrashing your system. So be careful if you decide to try to manually tune it. And when you go out and look at this mean time to recover, and even if you do it through the command line, you'll see that, that value is most likely going to change throughout the day, depending on the workload that you have. 08:46 Lois: How does the process of restoring and recovering data typically work? Bill: So when we restore, we're restoring our datafiles. All the datafiles, tablespace, controlfiles, archived redo log, server parameter file. Then when we recover, it involves depending on the backup that we use and other factors in there, it is going to apply the redo. So automatically done by RMAN. So I tell it, this is what I want to do. Hey, I want to restore a database. OK, RMAN says, all right, what backup are you going to use? What is it I need to restore? And then we tell it to recover. OK, I know what I need to use to recover. So RMAN can do the work for you. So when we restore and recover due to a manual process and there's different methods that we can use, and depending on the failure, we'll drive what type of restore and recovery we might perform. 09:40 Are you looking for practical use cases to help you plan and apply configurations that solve real-world challenges? With the new Applied Learning courses for Cloud Applications, you'll be able to practically apply the concepts learned in our implementation courses and work through case studies featuring key decisions and configurations encountered during a typical Oracle Cloud Applications implementation. Applied learning scenarios are currently available in General Ledger, Payables, Receivables, Accounting Hub, Global Human Resources, Talent Management, Inventory, and Procurement, with many more to come! Visit mylearn.oracle.com to get started. 10:22 Nikita: Welcome back! Can you talk about the different types of recovery scopes, Bill? How do they compare? Bill: Recovery can have two kinds of scope. All right. One is the complete recovery. We are getting the database back to the current time of the crash with no loss of data. We're going to again bring everything back to the present. Incomplete or point-in-time recovery. We're going to take a database or maybe a tablespace or even a table back to a point-in-time in the past. So from the time that we select to take it to recover, everything that was done after that is null and void, is gone missing. That's why it's called incomplete recovery, because it's not complete. 11:09 Lois: What are the steps that take place during complete recovery? Bill: We restore the datafiles. Changes are applied. We're applying the redo. The datafiles contained committed and uncommitted transactions. The undo is applied. Anything that did not receive an actual commit will take back to the original value. And we have our datafiles recovered. 11:33 Nikita: And what about point-in-time recovery? Bill: Point-in-time recovery, very similar. We're going to restore the datafiles from as far back as necessary. Changes are applied. So the data files are going to contain the committed and uncommitted up to that point-in-time. Database is open, that redo, that undo, anything that did not actually receive a commit. The undo is applied. The point-in-time recovered is complete. We're not applying all the redo, all the changes, only up to the time that we specify. 12:08 Lois: Are there any features that can make point-in-time recovery quicker? Bill: We also have the ability to use flashback database. It is an optional feature. And it can be a quick way to do that point-in-time recovery. It is an alternative to that database point-in-time recovery we just looked at. Faster. No restore is required. It's going to rewind the database. It does require some configuration in the environment. We do have to set up in order to use flashback database. 12:41 Nikita: I want to talk about Oracle's data protection solutions, particularly when it comes to backup and recovery or disaster recovery. Bill: So for physical data protection-- backup and recovery objective. Yep, that works for both physical and logical. My recovery time, hours to days. Possibly minutes to hours for the logical. And Oracle solution, we have the Recovery Manager that's out of the box, RMAN. Oracle Secure Backup, that is Oracle's media management library system backing up to tape. The logical protection, yes, flashback technologies can help me take care of that very easily. For disaster recovery, physical data protection, recovery time objective, seconds to minutes. We're not going to accomplish that with RMAN. You're going to want to use our Data Guard with the Active Data Guard feature to be able to switch over to a standby database within seconds of a failure. 13:41 Lois: Why would someone choose to use flashback technologies for recovery, Bill? Bill: With the flashback technologies, we can use it for viewing data as past dates. What did it look like? We can wind the database back and forth in time. Assist users in an error analysis and recovery, because we have different technologies. This flashback query, version query, transaction query, those allow me to view what was the value of a row at a time. I can even see what were the changes to a row over a period of time? I can also view the query that caused that change. For error recovery, I can back out a transaction. I can take a table back to a non-current time. I can also flashback a table that was dropped. And I can also take an entire database by using flashback. So the different recovery options I might have with the flashback technology. 14:44 Lois: Thank you so much, Bill. These last two episodes have been so insightful, right Niki? Nikita: I couldn't agree more, Lois! If you want to know more about backup and recovery configuration and other concepts, visit mylearn.oracle.com and search for the Oracle Database 23ai: Backup and Recovery course. Our upcoming episode is a very special one, where we'll be discussing Oracle AI in Fusion Cloud Human Capital Management. So, watch out for that! Until next week, this is Nikita Abraham… Lois: And Lois Houston, signing off!
 15:16 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this two-part special, Lois Houston and Nikita Abraham delve into the critical topic of backup and recovery in Oracle Database 23ai.   Together with Bill Millar, Senior Principal Database & MySQL Instructor, they discuss the role of database administrators, strategies for protecting data, and dealing with various types of data failure.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-backup-and-recovery/141127/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   Twitter: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Radhika Banka, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------- Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi there! For the last two weeks, we've been having really exciting discussions on Oracle AI Vector Search. We covered the fundamentals, benefits, the vector workflow, and lots more. Today, we're going to talk about backup and recovery in Oracle Database 23ai with Bill Millar. If you've been listening this season, you'll know that Bill is a Senior Principal Database & MySQL Instructor with Oracle University. Nikita: In this two-part special, we'll dive into some of the things you need to know about backup and recovery, especially if you're a database and backup admin. So, if you're the person in charge of keeping data safe and handling disaster recovery, this is definitely worth your time. 01:20 Lois: That's right, Niki. Hi Bill, thanks for joining us again. What's the role of a Database Administrator, or DBA, when it comes to backup and recovery? Bill: The DBA is typically responsible for ensuring the database is open and available when needed and at times you need to work with system administrators and other people within your organization to achieve that. But we want to try to protect the database from failure wherever possible. We want to increase the mean time between failures. Hopefully, we don't have failures, and we have to increase that time. But it might mean that we need to ensure we have redundant hardware and that in place, again, maybe out of the realm of the DBA, but people within your organization can help with that. We want to protect those critical components by using the redundancy. And we want to decrease the mean time to recover. Failures happen, but how fast can we get access back to that data after that failure. The faster we can do it, the happier customers are. Minimize the loss of data. It's never good to lose data, especially in a critical environment, but maybe in test and development, maybe not so bad.  02:39 Nikita: How do we ensure a separation of duties for backup and recovery processes? Bill: For a separation of duties, we do have a user called SYSBACKUP. It has the privileges that's required to perform backup and recoveries, the privilege to connect and execute the commands in what we refer to as RMAN, our Recovery Manager. As I said, it has permissions for backup and recovery because you do need to shut down the database, start up the database, those type of things. We're able to connect to that closed database to try to troubleshoot it, to get it to the open state again. It does not include any privileges to access data. The SYSBACKUP user is created when we install the database, when we create the database. We can use it explicitly for privileged user connection. It allows us to connect to the database. So RMAN connects as SYSBACKUP. 03:37 Lois: Bill, what should people keep in mind when figuring out what's considered critical data? Bill: You want to try to identify your critical data. Some data might be highly required to access and make sure we don't lose don't lose data, but then you might have some environments. OK, I don't need to have them up and running as fast. If we lose a little data, it may not hurt, but we want to identify the difference in the different data that we have on different environments. So we want to also prioritize that critical data, which data do we need access to first because how much will the company lose per hour of downtime because we can't do business. We want to make sure the access data protection requirements. Not everybody has access to everything. And there are different types of disaster that can happen that are going to be totally out of your control. There's the physical disaster, a hurricane or tornado, outages, power outages, component failures, failures within the building itself, corruption of data because of some of these failures. And then, the most dreaded one, the one that happens most often, usually those human errors, the logical errors, where the data is just bad, we are able to access and everything. It's just that something has changed that shouldn't have been changed. We want to make sure we access our recovery requirements.  05:04 Lois: So, what are they? What are those requirements? Bill: We want to base that requirement based on how critical is that data, how soon do we need to have access to that? What is our recovery point objective? Do we have a tolerance for any type of data loss? How frequently should we backups? How often they should be taken? What type of backups will be another thing we'll want to figure out? Is point-in-time recovery required? Are we able to or do we ever need to go back to a previous point in time to do something? It's not always just recovery for a database failure. We might need to do a recovery point in time to a different system so we can investigate something. What is my recovery time objective? Again, what is the tolerance for the downtime? How long can I be down? The downtime, the biggest part of when a system goes down is trying to identify what is the problem, then next is what is going to be my plan to recover, and then perform in the recovery. We might have a tiered required time objective based off of critical data, and then depending on the failure. Is that failure at the entire database? Is it just a tablespace? Is it just a table? Is it just a row? That also determines how long it takes to recover and what type of recovery we might try to perform. What is my backup retention policy? Do I have a requirement to where I have to have my backups off site? And it doesn't mean like back in the old days of mainframe computers, you'd back up to tape and you'd take those tapes off site. You might still do that today. Or, am I backing up to a cloud environment? So what do I need to have for that? What about long-term backups? We work with our day-to-day backups, but there's those backups that require for longer, archives like end of year backups. Some places require to keep their end of year backup for like 10 years. How are we going to handle that? So these are some of the things that we have to think about when we start talking about backup and recovery. 07:23 Did you know that the Oracle University Learning Community regularly holds live events hosted by Oracle expert instructors. Find out how to prepare for your certification exams. Learn about the latest technology advances and features. Ask questions in real time and learn from an Oracle subject matter expert. From Ask Me Anything about certification to Ask the Instructor coaching sessions, you'll be able to achieve your learning goals for 2024 in no time. Join a live event today and witness firsthand the transformative power of the Oracle University Learning Community. Visit mylearn.oracle.com to get started. 08:04 Nikita: Welcome back! Bill, I want to talk about the different failures that can occur in an Oracle database. How would you categorize them? Bill: There are different category of failure. This is not an all-inclusive list by any means. It's just something that possibly can happen. So they can usually be divided into different categories like statement failure. All right. When doing a select and insert, update, delete, the statement itself fails. A user process fails. Single database session fails for some reason. Network failure, connectivity is lost. The user error, probably one of the most common ones we have to deal with. A user successfully completes an operation, but that operation was erroneous. They dropped the wrong table, updated the wrong row. Then there's the instance failure. The database itself shuts down unexpectedly. And then media failure, usually a hard failure of our disk. Something of memory, something failed and caused an error. 09:12 Lois: Ok. I want to dive a little deeper into each of these categories that you mentioned. Let's start with statement failures. What are typical problems that one might face? Bill: Attempts to enter invalid data into a table. They're trying to put a numeric field in a date field, and usually just working with the user is going to correct that. Is that the DBA responsible? Yes, no, maybe. They attempt to form operations with insufficient privileges. Attempts to allocate space that fails, well, that depends on are they going-- do they have unlimited storage or do they have a limit? Logic errors in the application. Well, that's where we're going to have to work with those developers to try to correct those type of errors. 09:59 Nikita: What about user process failures? Bill: User performs an abnormal disconnect, doesn't close out properly. It can cause something to hang up or even possibly erroneous data to be updated. A user session is abnormally terminated. Well, usually, we don't have to try to resolve those user type errors, but something we might need to look into. A user experiences a program error that terminates the session. Again, usually it's the application developers, but it's something as a DBA, we might want to keep an eye on. Is it the same person? Is it from the same location? Is it the same module within that application? Maybe there's some things we can help to identify what the possible problem can be. 10:43 Nikita: Bill, tell us about common issues that can lead to network failures. What can we do to mitigate these problems and ensure network resilience? Bill: The listener fails. Well, we can connect a backup listener and configure how it can connect time failover can work. A network interface card fails. Well, again, we're not the hardware people, but can we work with our network, our server team, whatever, to possibly have redundant network cards? The network connection fails itself. Can we configure a backup network connection? 11:18 Lois: And what about user errors? How can we recover from those types of scenarios? Bill: The user inadvertently deletes or modifies data. Well, we have some things we'll look at as far as like rollback a transaction along with the dependent transactions. Rewind that table back to where it should have been. You're also can use LogMiner. You can look at our redo logs to try to figure out where that bad transaction was. User drops a table inadvertently. Well, we can recover the table from the recycle bin if we have the recycle bin on or we may need to recover from a backup. 11:56 Nikita: What are common causes of instance failures, Bill? Bill: The dreaded power outage. Well, hopefully, we have some type of up system to keep us running, even if it's not for continuous operation. Maybe if it's just to allow us to gracefully take a system down. The dreaded hardware failure. If you have a way to predict a hardware failure, you can make a lot of money. Always happens at the most inopportune times. But then again, do we have redundant hardware? Do we have something in place to help allow us to continue to operate in case of a hardware failure? Failure of one of the critical background processes. Why did it fail? We can go out. We can look at our alert log, we have trace files. And then we have, you have the Enterprise Manager Cloud Control. We can do the same thing as looking at the alert log and trace files. But the Enterprise Manager Cloud Control gives us a GUI interface to allow us to do that. 12:53 Lois: Before we let you go, Bill, can you tell us about media and data failures? Bill: Failure of a disk drive, failure of a disk controller, deletion or corruption of a file needed for database operation, well, this is the dreaded media failure. So we're going to restore from a backup. If we need to move, we can move a data file to a different location. We can notify, hey, here's that new location. And then recover by applying any of the incremental backups, any of the redo to get it back to where it should be. And then we have the data failures. We can't access the component, missing data files at OS level. And maybe our system administrators deleted something thinking it wasn't needed, or maybe even a developer on a development type system. Don't have the right permissions. Tablespace is offline. Well, why is it offline? Did somebody took the wrong tablespace offline? We have physical corruptions, block checksum failures. It's inconsistent between the header and footer. Invalid block header field values, like all of them are zeroed out. Then we have the logical corruptions, inconsistent dictionary, corrupt row piece, the inconsistencies, a control file not synchronized with the data files, usually because we recovered something and didn't do it the right way. I/O failures, maybe we just exceeded the number of open files that we're allowed to have. Maybe it's just a network or an I/O error itself. And these are different types of failures that you might experience. Again, it's not an all-inclusive list. It's just a few examples. 14:41 Nikita: I know you said it's not an all-inclusive list and you were just giving us a few examples, but that seemed quite thorough! Thank you so much, Bill, for walking us through all of that today! Lois: Yeah, I totally agree! Thanks Bill! For more on what we discussed today, visit mylearn.oracle.com. Search for the Oracle Database 23ai: Backup and Recovery course. Next week, we'll get into instance recovery and recovery strategies. Until then, this is Lois Houston… Nikita: And Nikita Abraham, signing off! 15:15 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

This week, Lois Houston and Nikita Abraham continue their exploration of Oracle AI Vector Search with a deep dive into vector indexes and memory considerations.   Senior Principal APEX and Apps Dev Instructor Brent Dayley breaks down what vector indexes are, how they enhance the efficiency of search queries, and the different types supported by Oracle AI Vector Search.   Oracle Database 23ai: Oracle AI Vector Search Fundamentals: https://mylearn.oracle.com/ou/course/oracle-database-23ai-oracle-ai-vector-search-fundamentals/140188/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   Twitter: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Radhika Banka, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started!   00:26 Nikita: Welcome back to the Oracle University Podcast! I'm Nikita Abraham, Team Lead of Editorial Services at Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi everyone! Last week was Part 1 of our discussion on Oracle AI Vector Search. We talked about what it is, its benefits, the new vector data type, vector embedding models, and the overall workflow. In Part 2, we're going to focus on vector indices and memory. 00:56 Nikita: And to help us break it all down, we've got Brent Dayley back with us. Brent is a Senior Principal APEX and Apps Dev Instructor with Oracle University. Hi Brent! Thanks for being with us today. So, let's jump right in! What are vector indexes and how are they useful? Brent: Now, vector indexes are specialized indexing data structures that can make your queries more efficient against your vectors. They use techniques such as clustering, and partitioning, and neighbor graphs. Now, they greatly reduce the search space, which means that your queries happen quicker. They're also extremely efficient. They do require that you enable the vector pool in the SGA. 01:42 Lois: Brent, walk us through the different types of vector indices that are supported by Oracle AI Vector Search. How do they integrate into the overall process? Brent: So Oracle AI Vector Search supports two types of indexes, in-memory neighbor graph vector index. HNSW is the only type of in-memory neighbor graph vector index that is supported. These are very efficient indexes for vector approximate similarity search. HNSW graphs are structured using principles from small world networks along with layered hierarchical organization. And neighbor partition vector index, inverted file flat index, is the only type of neighbor partition index supported. It is a partition-based index which balances high search quality with reasonable speed. 02:35 Nikita: Brent, you mentioned that enabling the vector pool in the SGA is a requirement when working with vector indexes. Can you explain that process for us? Brent: In order for you to be able to use vector indexes, you do need to enable the vector pool area. And in order to do that, what you need to do is set the vector memory size parameter. You can set it at the container database level. And the PDB inherits it from the CDB. Now bear in mind that the database does have to be balanced when you set the vector pool. 03:12 Lois: Ok. Are there any other considerations to keep in mind when using vector indices? Brent: Vector indexes are stored in this pool, and vector metadata is also stored here. And you do need to restart the database. So large vector indexes do need lots of RAM, and RAM constrains the vector index size. You should use IVF indexes when there is not enough RAM. IVF indexes use both the buffer cache as well as disk. 03:42 Nikita: And what about memory considerations? Brent: So to remind you, a vector is a numerical representation of text, images, audio, or video that encodes the features or semantic meaning of the data, instead of the actual contents, such as the words or pixels of an image. So the vector is a list of numerical values known as dimensions with a specified format. Now, Oracle does support the int8 format, the float32 format, and the float64 format. Depending on the format depends on the number of bytes. For instance, int8 is one byte, float32 is four bytes. Now, Oracle AI Vector Search supports vectors with up to 65,535 dimensions. 04:34 Lois: What should we know about creating a table with a vector column? Brent: Now, Oracle Database 23ai does have a new vector data type. The new data type was created in order to support vector search. The definition can include the number of dimensions and can include the format. Bear in mind that either one of those are optional when you define your column. The possible dimension formats are int, float 32, and float 64. Float 32 and float 64 are IEEE standards, and Oracle Database will automatically cast the value if needed. 05:18 Nikita: Can you give us a few declaration examples? Brent: Now, if we just do a vector type, then the vectors can have any arbitrary number of dimensions and formats. If we describe the vector type as vector * , *, then that means that vectors can have an arbitrary number of dimensions and formats. Vector and vector * , * are equivalent. Vector with the number of dimensions specified, followed by a comma, and then an asterisk, is equivalent to vector number of dimensions. Vectors must all have the specified number of dimensions, or an error will be thrown. Every vector will have its dimension stored without format modification. And if we do vector asterisk common dimension element format, what that means is that vectors can have an arbitrary number of dimensions, but their format will be up-converted or down-converted to the specified dimension element format, either INT8, float 32, or float 64. 06:25 Working towards an Oracle Certification this year? Take advantage of the Certification Prep live events in the Oracle University Learning Community. Get tips from OU experts and hear from others who have already taken their certifications. Once you're certified, you'll gain access to an exclusive forum for Oracle-certified users. What are you waiting for? Visit mylearn.oracle.com to get started.   06:52 Nikita: Welcome back! Brent, what is the vector constructor and why is it useful? Brent: Now, the vector constructor is a function that allows us to create vectors without having to store those in a column in a table. These are useful for learning purposes. You use these usually with a smaller number of dimensions. Bear in mind that most embedding models can contain thousands of different dimensions. You get to specify the vector values, and they usually represent two-dimensional like xy coordinates. The dimensions are optional, and the format is optional as well. 07:29 Lois: Right. Before we wrap up, can you tell us how to calculate vector distances? Brent: Now, vector distance uses the function VECTOR_DISTANCE as the main function. This allows you to calculate distances between two vectors and, therefore, takes two vectors as parameters. Optionally, you can specify a metric. If you do not specify a metric, then the default metric, COSINE, would be used. You can optionally use other shorthand functions, too. These include L1 distance, L2 distance, cosine distance, and inner product. All of these functions also take two vectors as input and return the distance between them. Now the VECTOR_DISTANCE function can be used to perform a similarity search. If a similarity search query does not specify a distance metric, then the default cosine metric will be used for both exact and approximate searches. If a similarity search does specify a distance metric in the VECTOR_DISTANCE function, then an exact search with that distance metric is used if it conflicts with the distance metric specified in a vector index. If the two distance metrics are the same, then this will be used for both exact as well as approximate searches. 08:58 Nikita: I was wondering Brent, what vector distance metrics do we have access to? Brent: We have Euclidean and Euclidean squared distances. We have cosine similarity, dot product similarity, Manhattan distance, and Hamming similarity. Let's take a closer look at the first of these metrics, Euclidean and Euclidean squared distances. This gives us the straight-line distance between two vectors. It does use the Pythagorean theorem. It is sensitive to both the vector size as well as the direction. With Euclidean distances, comparing squared distances is equivalent to comparing distances. So when ordering is more important than the distance values themselves, the squared Euclidean distance is very useful as it is faster to calculate than the Euclidean distance, which avoids the square root calculation. 09:58 Lois: And the cosine similarity metrics? Brent: It is one of the most widely used similarity metrics, especially in natural language processing. The smaller the angle means they are more similar. While cosine distance measures how different two vectors are, cosine similarity measures how similar two vectors are. Dot product similarity allows us to multiply the size of each vector by the cosine of their angle. The corresponding geometrical interpretation of this definition is equivalent to multiplying the size of one of the vectors by the size of the projection of the second vector onto the first one or vice versa. Larger means that they are more similar. Smaller means that they are less similar. Manhattan distance is useful for describing uniform grids. You can imagine yourself walking from point A to point B in a city such as Manhattan. Now, since there are buildings in the way, maybe we need to walk down one street and then turn and walk down the next street in order to get to our result. As you can imagine, this metric is most useful for vectors describing objects on a uniform grid such as city blocks, power grids, or perhaps a chessboard. 11:27 Nikita: And finally, we have Hamming similarity, right? Brent: This describes where vector dimensions differ. They are binary vectors, and it tells us the number of bits that require change to match. It compares the position of each bit in the sequence. Now, these are usually used in order to detect network errors. 11:53 Nikita: Brent, thanks for joining us these last two weeks and explaining what Oracle AI Vector Search is. If you want to learn more about what we discussed today, visit mylearn.oracle.com and search for the Oracle Database 23ai: Oracle AI Vector Search Fundamentals course.   Lois: This concludes our season on Oracle Database 23ai New Features for administrators. In our next episode, we're going to talk about database backup and recovery, but more on that later! Until then, this is Lois Houston… Nikita: And Nikita Abraham signing off! 12:29 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, Senior Principal APEX and Apps Dev Instructor Brent Dayley joins hosts Lois Houston and Nikita Abraham to discuss Oracle AI Vector Search. Brent provides an in-depth overview, shedding light on the brand-new vector data type, vector embeddings, and the vector workflow.   Oracle Database 23ai: Oracle AI Vector Search Fundamentals: https://mylearn.oracle.com/ou/course/oracle-database-23ai-oracle-ai-vector-search-fundamentals/140188/   Oracle Database 23ai: SQL Workshop: https://mylearn.oracle.com/ou/course/oracle-database-23ai-sql-workshop/137830/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   Twitter: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Radhika Banka, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started!   00:26 Lois: Hello and welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs here at Oracle University. Joining me as always is our Team Lead of our Editorial Services, Nikita Abraham. Nikita: Hi everyone! Thanks for tuning in over the last few months as we've been discussing all the Oracle Database 23ai new features. We're coming to the end of the season, and to close things off, in this episode and the next one, we're going to be talking about the fundamentals of Oracle AI Vector Search. In today's episode, we'll try to get an overview of what vector search is, why Oracle Vector Search stands out, and dive into the new vector data type. We'll also get insights into vector embedding models and the vector workflow. 01:11 Lois: To take us through all of this, we're joined by Brent Dayley, who is a Senior Principal APEX and Apps Development Instructor with Oracle University. Hi Brent! Thanks for joining us today. Can you tell us about the new vector data type? Brent: So this data type was introduced in Oracle Database 23ai. And it allows you to store vector embeddings alongside other business data. Now, the vector data type allows a foundation to store vector embeddings. 01:42 Lois: And what are vector embeddings, Brent? Brent: Vector embeddings are mathematical representations of data points. They assign mathematical representations based on meaning and context of your unstructured data. You have to generate vector embeddings from your unstructured data either outside or within the Oracle Database. In order to get vector embeddings, you can either use ONNX embedding machine learning models or access third-party REST APIs. Embeddings can be used to represent almost any type of data, including text, audio, or visual, such as pictures. And they are used in proximity searches. 02:28 Nikita: Hmmm, proximity search. And similarity search, right? Can you break down what similarity search is and how it functions? Brent: So vector data tends to be unevenly distributed and clustered into groups that are semantically related. Doing a similarity search based on a given query vector is equivalent to retrieving the k nearest vectors to your query vector in your vector space. What this means is that basically you need to find an ordered list of vectors by ranking them, where the first row is the closest or most similar vector to the query vector. The second row in the list would be the second closest vector to the query vector, and so on, depending on your data set. What we need to do is to find the relative order of distances. And that's really what matters rather than the actual distance. Now, similarity searches tend to get data from one or more clusters, depending on the value of the query vector and the fetch size. Approximate searches using vector indexes can limit the searches to specific clusters. Exact searches visit vectors across all clusters. 03:44 Lois: Ok. I want to move on to vector embedding models. What are they and why are they valuable? Brent: Vector embedding models allow you to assign meaning to what a word, or a sentence, or the pixels in an image, or perhaps audio. It allows you to quantify features or dimensions. Most modern vector embeddings use a transformer model. Bear in mind that convolutional neural networks can also be used. Depending on the type of your data, you can use different pretrained open source models to create vector embeddings. As an example, for textual data, sentence transformers can transform words, sentences, or paragraphs into vector embeddings. 04:33 Nikita: And what about visual data? Brent: For visual data, you can use residual network also known as ResNet to generate vector embeddings. You can also use visual spectrogram representation for audio data. And that allows us to use the audio data to fall back into the visual data case. Now, these can also be based on your own data set. Each model also determines the number of dimensions for your vectors. 05:02 Lois: Can you give us some examples of this, Brent? Brent: Cohere's embedding model, embed English version 3.0, has 1,024 dimensions. Open AI's embedding model, text-embedding-3-large, has 3,072 dimensions. 05:24 Want to get the inside scoop on Oracle University? Head over to the Oracle University Learning Community. Attend exclusive events. Read up on the latest news. Get first-hand access to new products. Read the OU Learning Blog. Participate in Challenges. And stay up-to-date with upcoming certification opportunities. Visit mylearn.oracle.com to get started.  05:50 Nikita: Welcome back! Let's now get into the practical side of things. Brent, how do you import embedding models? Brent: Although you can generate vector embeddings outside the Oracle Database using pre-trained open source embeddings or your own embedding models, you also have the option of doing those within the Oracle Database. In order to use those within the Oracle Database, you need to use models that are compatible with the Open Neural Network Exchange Standard, or ONNX, also known as Onyx. Oracle Database implements an Onyx runtime directly within the database, and this is going to allow you to generate vector embeddings directly inside the Oracle Database using SQL. 06:35 Lois: Brent, why should people choose to use Oracle AI Vector Search? Brent: Now one of the biggest benefits of Oracle AI Vector Search is that semantic search on unstructured data can be combined with relational search on business data, all in one single system. This is very powerful, and also a lot more effective because you don't need to add a specialized vector database. And this eliminates the pain of data fragmentation between multiple systems. It also supports Retrieval Augmented Generation, also known as RAG. Now this is a breakthrough generative AI technique that combines large language models and private business data. And this allows you to deliver responses to natural language questions. RAG provides higher accuracy and avoids having to expose private data by including it in the large language model training data. 07:43 Nikita: In the last part of our conversation today, I want to ask you about the Oracle AI Vector Search workflow, starting with generating vector embeddings. Brent: Generate vector embeddings from your data, either outside the database or within the database. Now, embeddings are a mathematical representation of what your data meaning is. So what does this long sentence mean, for instance? What are the main keywords out of it? You can also generate embeddings not only on your typical string type of data, but you can also generate embeddings on other types of data, such as pictures or perhaps maybe audio wavelengths. 08:28 Lois: Could you give us some examples? Brent: Maybe we want to convert text strings to embeddings or convert files into text. And then from text, maybe we can chunk that up into smaller chunks and then generate embeddings on those chunks. Maybe we want to convert files to embeddings, or maybe we want to use embeddings for end-to-end search. Now you have to generate vector embeddings from your unstructured data, either outside or within the Oracle Database. You can either use the ONNX embedding machine learning models or you can access third-party REST APIs. You can import pre-trained models in ONNX format for vector generation within the database. You can download pre-trained embedding machine learning models, convert them into the ONNX format if they are not already in that format. Then you can import those models into the Oracle Database and generate vector embeddings from your data within the database. Oracle also allows you to convert pre-trained models to the ONNX format using Oracle machine learning for Python. This enables the use of text transformers from different companies. 09:51 Nikita: Ok, so that was about generating vector embeddings. What about the next step in the workflow—storing vector embeddings? Brent: So you can create one or more columns of the vector data type in your standard relational data tables. You can also store those in secondary tables that are related to the primary tables using primary key foreign key relationships. You can store vector embeddings on structured data and relational business data in the Oracle Database. You do store the resulting vector embeddings and associated unstructured data with your relational business data inside the Oracle Database. 10:30 Nikita: And the third step is creating vector indexes? Brent: Now you may want to create vector indexes in the event that you have huge vector spaces. This is an optional step, but this is beneficial for running similarity searches over those huge vector spaces. So once you have generated the vector embeddings and stored those vector embeddings and possibly created the vector indexes, you can then query your data with similarity searches. This allows for Native SQL operations and allows you to combine similarity searches with relational searches in order to retrieve relevant data. 11:15 Lois: Ok. I think I've got it. So, Step 1, generate the vector embeddings from your unstructured data. Step 2, store the vector embeddings. Step 3, create vector indices. And Step 4, combine similarity and keyword search. Brent: Now there is another optional step. You could generate a prompt and send it to a large language model for a full RAG inference. You can use the similarity search results to generate a prompt and send it to your generative large language model in order to complete your RAG pipeline. 11:59 Lois: Thank you for sharing such valuable insights about Oracle AI Vector Search, Brent. We can't wait to have you back next week to talk about vector indices and memory. Nikita: And if you want to know more about Oracle AI Vector Search, visit mylearn.oracle.com and check out the Oracle Database 23ai: Oracle AI Vector Search Fundamentals course. Lois: Yes, and if you're serious about advancing in your development journey, we recommend taking the Oracle Database 23ai SQL workshop. It's designed for those who might be familiar with SQL from other database platforms or even those completely new to SQL. Nikita: Yeah, we'll add the link to the workshop in the show notes so you can find it easily. Until next week, this is Nikita Abraham… Lois: And Lois Houston signing off! 12:45 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, Lois Houston and Nikita Abraham explore the Automatic Transaction Quarantine feature with Senior Principal Database & MySQL Instructor, Bill Millar. Bill explains that this feature isolates transactions that could potentially cause system crashes, preventing them from impacting the entire container database. They also discuss the key advantages of automatic transaction quarantine in maintaining database stability and availability.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/140830/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead: Editorial Services with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi there! In our last episode, we looked at an Oracle Database 23ai new feature called Automatic Transaction Rollback, and we spoke about why it is such an important feature for database administrators. 00:51 Nikita: Today, we're going to talk about another new feature called Automatic Transaction Quarantine. We'll discuss what it is, go through the steps to monitor and identify quarantine transactions, explore how an issue is resolved once a quarantined transaction has been identified, and end by looking at quarantined transaction escalation, and how it helps to protect not only your PDB, but also your container database. Lois: Back with us is Bill Millar, our Senior Principal Database & MySQL Instructor with Oracle University. Hi Bill! What is automatic transaction quarantine and why do we need it? 01:27 Bill: The good news is that starting in 23c with the database quarantines, it's going to isolate a transaction or transactions that could possibly cause a system crash, so you can avoid crashes. It's going to isolate those transactions that potentially could cause a problem. However, those transactions must be manually resolved by the DBA so that the row locks are released from those bad transactions. A transaction recovery basically is going to isolate failure and also identify what is the cause of that corruption. So when a system restarts, transaction can fail to recover while the other transactions can be recovered. So with the transaction recovery, basically, we know when the system recovers, the SMON is going to use the redo and the undo. 02:27 Nikita: Can you explain that in a little more detail? How does transaction recovery work and why is it so critical for database stability? Bill: It does the redo to roll forward the database. However, at that point, it'll go ahead and open the database, allow it to start being used while it is applying the undo. And when it cannot apply that undo, that's when the system is going to mark that transaction as bad for that. That is what is transaction recovery. Whereas instance recovery is basically the same thing, except now you're in a RAC environment. And it's unable to be recovered on one of the instances within your RAC environment. Because it can be, it'll have those rows locked, and it can affect the other instances. So SMON might be unable to perform that recovery, so it could cause that PDB or the CDB to crash. OK, now, nobody can access any information. So once if that entire container crashes, recovery is going to stop. If it has a bad transaction, recovery stops. So it might be because of physical data, might be because of the index is corrupt, might be logical corruption. So it stops that interactive transaction recovery process. So not only does it stop the recovery of the transaction that is trying to be recovered by SMON, it's going to stop the rest of the inactive transactions. Those row locks are held. And it can impact critical operations. Yeah, if my system can't do anything, yes, it's going to have an impact. The DBAs must resolve what is that bad transaction, how to get rid of it, how we're going to get around it? 04:12 Lois: Bill, what's the workflow a DBA would follow when a transaction is quarantined? Bill: So in the system, when that transaction recovery failure is, OK, I've found this dead transaction. I'm going to quarantine. I'm going to say, hey, you have something you need to take care of for that. So it's not recovered by the SMON. So what's going to happen? So there is also is going to be a limit. So if it does reach that limit and the limit is three, then you're going to have to step in and try to take care of that very quickly. The shut down abort will be performed on the PDB. So the good news there is that it's going to keep it from impacting the entire container. If the limit isn't reached, well, then, OK, hey, we have this bad transaction that's going to quarantine, is going to populate. There's a couple of views that you can go out and look at. There's a CDB quarantine transactions or a DBA quarantine transactions. Those views you can look at. And then once we determine that, what are we going to do to try to recover it? If we're going to try to recover it, then we can go ahead and drop that bad transaction. It'll help free up the rows. That way, everything can start working again. That PDB can be opened. 05:30 Nikita: What can you tell us about monitoring quarantined transactions? What specific views or logs should DBAs monitor? Bill: So you can view. You'll see these quarantine transactions in several different places. One is the alert queue. It's going to be sent to the alert queue. That is what is going to notify Enterprise Manager Cloud Control, also populates it within the AWR. Back in 21c, we added the attention log. It shows critical events. Hey, you need to take a look at this. It also can populate it. It will populate it to the alert log. So remember you have the V$DIAG_ALERT that you can look at. Or, if you're familiar with or you use the ADRCI, automatic diagnostic repair recovery advisor, so you can also look at the alert log there. So there are two new views, the CDB_QUARANTINED_TRANSACTION, the DBA_QUARANTINE_TRANSACTIONS working with multi-tenant. The CDB, I can see all the quarantine transactions from the root container, the DBA_QUARANTINE_TRANSACTIONS what I see if I'm in a specific PDB. But it's going to give me the information. 06:52 Lois: What about resolving quarantined transactions? Bill: Monitoring is a must to be able to identify, hey, we have bad transactions that we need to-- quarantine transactions we need to take care of. You can apply the appropriate MOS note if you're not sure what to do. Like anything else, if something happens-- and hopefully, you're not getting quarantined transactions daily or anything like that. But once we start doing a few things, we remember how to do them. 07:21 Lois: And, how do we take care of this? Bill: Well, you always have the ability to go to My Oracle Support. There is a view called-- that CDB quarantine transaction that we talked about that we can look at, hey, here's the reason. And we might use that to go out there and search My Oracle Support and/or contact Oracle Support. 07:49 Do you have an idea for a new course or learning opportunity? We'd love to hear it! Visit the Oracle University Learning Community and share your thoughts with us on the Idea Incubator. Your suggestion could find a place in future development projects! Visit mylearn.oracle.com to get started.  08:09 Nikita: Welcome back! Bill, what are some of the common causes of quarantined transactions? Could you share some examples with us? And how do you resolve them? Bill: One could be physical corruptions. It could either be logical or physical. So maybe because media failed. Hardware bits get flipped. So that might be able to be easily fixed by using the RMAN Block Media Recovery. And that's the lowest level of recovery that we can apply. And then there's logical corruptions. This is the recommended order when trying to resolve logical corruptions. First level is the Block Media Recovery. And then, after that, if the Block Media Recovery fails, then possibly, how about re-creating that data segment? So either truncate or drop it, and then recover it from another source. So once you drop the segment, the transaction then is going to skip trying to recover it. It's no longer there. So it's, OK, hey, I'm successful now. And then, the last resort type method is to drop that undo segment. There's an offline rollback segment that you can use. But it's recommended-- it's best to avoid that-- again, kind of a last-ditch effort to try to fix something. There are other options that you might try. However, these options do end up being a loss of data. Why? Because we're going to do a point-in-time recovery. So we can go back to a table point-in-time recovery. So we start with the Block Media Recovery. OK, we can't. OK, so how about if we go back before that transaction and try to recover the table at that time? So it will be a loss of data. Then, the next level is, we can't do the table. Can we do the entire tablespace? That might be an option. Might flashback the database if we are using-- if we have Flashback Database on. Again, that's just another method of point-in-time recovery. And then also do a database point-in-time recovery. If we can do the database point-in-time recovery flashback at the PDB level, so that way it's not impacting the entire container, hopefully, we don't have to try to do a point-in-time recovery at the database level. So we wouldn't want to do that. That would something really drastic would have to happen to force us to do the entire container. But we want to do that at the PDB level. 10:54 Lois: Ok. So the issue is resolved. What happens next? Bill: So once we have the issue resolved that caused that, SMON is still going to try to do transaction recovery because why? That quarantined transaction says, hey, I've still got this bad transaction there. So once that transaction has been fixed, we need to drop that quarantined transaction. So that way, SMON says, hey, I have this transaction. I need to recover. SMON will keep from trying to do that. So there is a DDL command to drop that quarantined transaction. So remember, from the views, the quarantined transaction views, that's where we saw the undo segment. We saw the slot number. We saw the quarantined transaction slot number. So that way, we can drop that transaction by using that. 11:51 Nikita: How does the escalation process work for quarantined transactions? And why is it important to protect the PDB and the container database? Bill: So quarantined transaction escalation-- we might have multiple transactions fail, depending on the corruption level. It might have multiple blocks for that that have failed. So just to quarantine a bad transaction may not help whatsoever. It depends on what the root cause is for the failures and how many are happening at that time. So the database with these bad transactions will continuously run in an inconsistent state. So it could be dangerous if we have multiples of the same issue and that. So with that system running in an inconsistent state, things will continue to spread. Things will continue to get worse. That's why, once that level of 3 is reached, we go ahead, and we do a shut down abort on that PDB. Because if a transaction can't be recovered, there's no need in trying to do any other type of shutdown. So with this escalation process, it does benefit us because, again, SMON is going to continuously try to recover that bad transaction for that. OK, SMON's going to keep trying. It's not going to work. And at some point, it might cause it to crash. So by stopping it before it continues getting worse, damaging more, we're going to go ahead and say we're escalating this issue to where we're shutting down the PDB. Fault tolerance, so meaning that we have higher availability of the rest of the container. So it's not going to crash the entire container. So the PDB can continue to operate when we are trying to resolve transactions except in the case where it exceeds the amount, and it does a shutdown abort on that PDB. So with that escalation, we reach that limit of 3 for that. We do a Shutdown Abort on that PDB. That transaction recovery is disabled. OK. Don't try to recover any transactions. Why? Because we know we have a few of them. So it's shut down, so we're going to go out and look at our quarantine transactions views, what's the reason for that, how many do we have? And then, once we resolve the issue, we are going to enable recovery again because it turns off the recovery option before it allows us to open that PDB. It's not going to be in a consistent state, though. So now we can go ahead and alter the system and, OK, go ahead and allow recovery of transactions again. 14:42 Lois: Thank you, Bill, for walking us through the details of automatic transaction quarantine and telling us how to manage and resolve these complex scenarios. Nikita: Yeah, thanks Bill! To learn more about what we discussed today, visit mylearn.oracle.com and search for the Oracle Database 23ai New Features for Administrators course. Join us next week for a discussion on some more Oracle Database 23ai new features. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 15:13 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Join Lois Houston and Nikita Abraham as they discuss the Automatic Transaction Rollback feature with Senior Principal Database & MySQL Instructor, Bill Millar. Bill explains that in the 23ai release, transactions blocking other transactions can now be automatically rolled back, depending on certain parameters. Bill highlights the advantages of using automatic transaction rollback, which eliminates the time-consuming process of manually terminating blocking transactions. They also cover the workload reduction benefits for database administrators.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/140830/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   -------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Lois: Hello and welcome back to the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Team Lead of Editorial Services. Nikita: Hi everyone! Last week, we looked at two Oracle Database 23ai new features related to Data Manipulation Language, or DML. One was Unrestricted Parallel DMLs and the other was Unrestricted Direct Loads. Do check out that episode if you missed it. 00:56 Lois: Today, we have Senior Principal Database & MySQL Instructor, Bill Millar, with us. He's been on several times this season taking us through all the different 23ai new features. In this episode, we're going to ask him about the Automatic Transaction Rollback feature. Hi Bill! What is automatic transaction rollback and why is it an important feature for database administrators? 01:22 Bill: We can now have transactions that are blocking other transactions, depending on some settings, to automatically roll back. It does require some parameters to be set. Rows basically get locked in a single row. Each row is locked based off of what type of activity is being performed on that row, such as inserts, updates, deletes, merge, select for updates. 01:52 Nikita: And how were things before this feature? Bill: Traditionally, the database administrator had to research and manually terminate blocking transactions, or there are some things that resource manager might have been able to do. 02:05 Lois: This seems like such a game-changer for DBAs, Bill. So, how does it work? Bill: So there are some parameters that control the automatic rollback. One is the transaction priority. We're going to set that priority for a transaction either to medium, high, or low. We have the high priority wait target and a medium priority wait target that we can set. The high wait target will terminate if a medium transaction is blocking that high target based off of the values that we set, the medium transaction can be terminated. A medium transaction will terminate a low priority. So if a transaction designated as low exceeds the blocking time that we set for the medium priority wait time, then it'll be terminated. Whereas, the high priority will terminate both medium and low transactions. We have the rollback mode. We're either going to roll back or we're going to track, depending on what we're trying to do. 03:10 Nikita: So, if I decide that I want to use automatic transaction rollback… if I decide to implement it…I'll need to set those parameters, right? Bill: So we can set those at a session level. We also have some system level wait targets. What are the wait times for the medium, high transactions? How long they are going to wait for those lower transactions? And then we also have the rollback mode. Are we actually going to roll back or are we just going to track for right now? We have to determine what is going to be the wait times for those transactions that we want to wait before those lower transactions, priority transactions are rolled back? At that session level, we're going to set the session. High is the default. So if we want transactions to run at a lower, we have to set those. So we can set the medium or low because that's going to determine how they're rolled back. So, what is that rollback order? Again the low, we'll roll back any low that's blocking mediums. High, we'll roll back any mediums or lows that are blocking. So you do need to have the understanding of that application, and how critical are the different transactions, because if you start rolling back transactions, what? It does-- If you roll back the transactions, it does generate a little research, a little bit more work on why did that happen. 04:38 Lois: Yeah… you don't want to set it without really understanding what you're doing. Ok, so, what else do I need to know? Bill: So we do have the system level wait targets again. How long is the high priority transaction going to wait for a lower transaction before it rolls it back? How long that medium priority is going to wait? We use the ALTER SYSTEM SET command. It does have a range of values from one second to 2,147,483,647 seconds. That's like 68 years. Might not want to wait 68 years for a transaction to be rolled back. We can set it at the PDB level. Each pluggable can have a different value. And it can have a different value in the different RAC instances. We have those system level wait targets that we want to set. Automatic rollback. In order for it to function, all the parameters have to be set properly. What is that transaction priority? We saw the medium, high, low. What is the wait target? How long is the medium is going to wait? How long is the low is going to wait? We set that in seconds. The order of those transactions determine how they are terminated. 05:53 Lois: Earlier on, you mentioned rollback mode. Can you tell us a little more about it? Bill: So with that automatic rollback mode, there's only two valid values. It is considered advanced parameter. We can either set it in rollback, which is the default, or we can put it in track mode. Track mode gives us the ability to try it out. I guess you can say. It will say, hey, if I would have been running, if I would have been used, I would have terminated this transaction. It'll show me the number of times it would have happened for high priority, the number of times it would happen for a medium priority. It is modifiable in the PDB, but however, the track mode must be the same in each instance. So that rollback mode, again, that is the default value for that. So statistics are going to be available. So how many high priority rollbacks occurred? How many medium rollbacks occurred? In that track mode, I have to set that value. I do have to have the time set for how long is it going to wait for those, so the high and medium. And those priorities has to be set in the session. So statistics are available for the high and the medium in the track mode. Not only when we're actually rolling back, but also tracking. Again, this gives us the ability, by having it in the track mode, gives us the ability to do a little testing with it first. 07:27 The Oracle University Learning Community is an excellent place to collaborate and learn with Oracle experts and fellow learners. Grow your skills, inspire innovation, and celebrate your successes. All your activities, from liking a post to answering questions and sharing with others, will help you earn a valuable reputation, badges, and ranks to be recognized in the community. Visit mylearn.oracle.com to get started. 07:55 Nikita: Welcome back! Bill, when it comes to monitoring, how do you keep track of these rollbacks? Bill: For monitoring our rollback transactions, the data dictionary information is available to assist with monitoring our transaction priority. So from the V$TRANSACTIONS, there are columns available allowing us to do that. Based off that transaction priority shows what is the wait target for that. And then also each of the priority of those transactions. We can view this information, it will be populated to the alert log. So we can see that session ID, what was session ID of that? What was the transaction ID? What was the priority? What was the system identifier for that? It tells you-- even tells you the parameter and tells you what that wait time was set at. If it was a medium transaction that was terminated, it shows, OK, it was a medium. So we can view the alert log. And we can look for these terminations. Gives an idea of what's being done. 09:01 Nikita: And finally, what are the key advantages of using automatic transaction rollback? Bill: It eliminates a very manual process. It can be very time-consuming for the DBA to go out there and try to find what's the blocking session. Yep, I'll go ahead and do an ALTER SYSTEM. I'll kill that session trying to track it down, finding the views to look at it to say, OK, Yeah, this is the blocking one. I want to go ahead and take care of it. Resource manager doesn't really fully address blocking transactions. Some things that can do for that. We have the maximum estimate execution time. So that's the number in CPU seconds allowed for that call. It's terminated. It doesn't matter whether it's blocking another session or not in that case or even another transaction. It just says, OK, you exceeded this time. I'm going to terminate you. Then we also have the max idle time again. That's maximum session idle time. All right. You haven't been doing anything to session, we're going to terminate you. And then we have the MAX_IDLE_BLOCKER. That's the time duration of an idle session can block another session. Again, it's going to check OK, is the session actually idle? But these don't really address the issue of, hey, I have a higher priority transaction waiting for a lower transaction that's blocking it. 10:27 Lois: Thank you, Bill, for that breakdown. This feature is such a time saver. Nikita: Yeah, and such good way to reduce the manual workload for DBAs. Thanks Bill! Lois: To learn more about what we discussed today and view some of the demonstrations of this feature, visit mylearn.oracle.com and search for the Oracle Database 23ai New Features for Administrators course. Join us next week for a discussion on some more Oracle Database 23ai new features. Until then, this is Lois Houston… Nikita: And Nikita Abraham signing off! 11:02 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, hosts Lois Houston and Nikita Abraham discuss new features in Oracle Database 23ai related to Data Manipulation Language (DML). They are joined by Senior Principal Database & MySQL Instructor, Bill Millar, who explains the concept of unrestricted parallel DMLs and their importance in speeding up large operations and maintaining summary tables. The discussion then turns to unrestricted direct loads, examining the evolution of direct loads with 23ai and the broader impact of these changes.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/140830/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00  Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Principal Technical Editor with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi there! In our last episode, we discussed a ground-breaking caching solution in Oracle Database 23ai, known as True Cache. We spoke about its configuration and deployment, and explored how to apply True Cache to our applications. Nikita: Today, we're going to talk about two Oracle Database 23ai new features related to Data Manipulation Language, or DML. The first is Unrestricted Parallel DMLs and then we'll move on to Unrestricted Direct Loads. We'll talk about the situation prior to 23ai, identify the improvements that have been made, and look at their benefits. 01:15 Lois: And returning for another episode is Bill Millar, our Senior Principal Database & MySQL Instructor with Oracle University. Hi Bill! So, to start, can you explain what unrestricted parallel DMLs are and why they are important, especially in the context of Oracle Database? Bill: The Oracle Database allows DML statements such as inserts, updates, deletes, merge to be executed in parallel by breaking those statements into smaller task. These transactions can contain multiple DML statements. And they can modify multiple different tables. So transactions with the parallel DML is going to use the execution method by breaking up those large operations to execute the transaction in parallel. It helps speed up the large operations. And it's advantageous to data warehouse environments where we're maintaining like summary tables, historical tables. And even in OLTP systems, it can be beneficial for long-running batch jobs. The scale up. Well, it's basically dividing the executing SQL against those large tables and indexes into those smaller units of work. 02:36 Nikita: So, what were the limitations prior to 23ai? Bill: So once that object was modified by APLL statement, the object cannot be read or modified later in the same transaction. After that parallel DML modifies a table, there is no follow-on DML or query on the same table within that same transaction. If any attempt to access a table modified by that parallel statement, the transaction would be rejected. You're only allowed to query on those tables prior to that DML on that object itself. 03:16 Lois: Ok… So with these new improvements, I'm guessing some of these restrictions have been removed? Bill: In this case, in the same session, you can query the table multiple times. You can perform conventional DML on the same table within the same session. And you can also have multiple direct loads in the same session without having to do that commit. But there are still some restrictions with it. Heap tables only. You can't do it with any clustered tables or IOT, Index Organized Tables. Non-ASSM, the Automatic Segment Space Management tables. The temp table is not under ASSM. Why? Because it has to have uniform extents or any other tablespaces that you created with the uniform extents. So those restrictions still apply. So some of the improvements are some of the restrictions can help reduce the overhead. We can enable Parallel DML within that session. Allows the multiple operations on the same object. And it doesn't require that commit for each separate operation. Makes it a little bit easier to use by removing some of these limitations. Now users can run parallel DMLs and any combination of statements within that same transaction. And it can help simplify and speed up data loading analytic processes by making the database, the parallel execution and parallel queries, at the same time within that same session, again, eliminating having to do commits. 04:58 Nikita: Thanks for that summary of all the improvements, Bill. Now, how do you enable this? Is it enabled by default? Bill: To enable the Parallel DML mode, it is required for a session. It is disabled by default. That's because the Parallel DML and Serial DML, they have different locking, different ways to handle the transactions, different disk space requirements. When Parallel DML is enabled in a session, all DML statements are considered for parallel execution. Only a statement is considered for parallel execution when the Enable Parallel DML hint is used if I don't set it for a session. The sessions DML mode does not influence any parallelism of DDL statements. When the Parallel DML is disabled, no DML is executed in parallel, even if the hint is used. 05:59 Lois: Bill, I would like to dig a little deeper into the benefits. How do these lifted restrictions improve the overall performance and reduce overhead? Bill: There's no longer that requirement to commit everything separately. So that's going to reduce the overhead, not having to do the commit all the time. The scalability of accessing those large objects, executing parallel makes the decision support systems, those data warehouses and batch OLTP jobs or any other larger DML operation execute faster. By removing that one touch limitation, it allows the parallel DML statements to be read or modified by later statements of the same transaction in the same session. It's very similar to the non-parallel statements. And even OLTP systems can also benefit, for example, maintaining a larger operation, such as the creation of indexes, refreshing tables, or even creating summary tables. 07:14 Did you know that Oracle University offers free courses on Oracle Cloud Infrastructure? You'll find training on everything from cloud computing, database, and security to artificial intelligence and machine learning, all free to subscribers. So, what are you waiting for? Pick a topic, leverage the Oracle University Learning Community to ask questions, and then sit for your certification. Visit mylearn.oracle.com to get started. 07:42 Nikita: Welcome back! Let's move on to the next new feature, which is unrestricted direct loads. Bill, what was the situation with direct loads like, prior to 23ai? Bill: After a direct load and prior-- it was always prior to a commit, queries in additional DMLs were not allowed on that same table. You might encounter the ORA error, the 12838, saying, hey, you can't read or modify this in parallel. That's because the DML on that direct load had access to that and that session for that. So you might have received that error. The enq contention, the wait event for the direct load issue in a different session from the other sessions during the direct load is having to wait, because of that queuing that-- because a transaction gets that table, locks that information to keep that table from being modified until that direct load has actually committed. Within the same transaction, within the same session, trying to do multiple DMLs with the-- while it is being modified with the direct loads itself. Unlike conventional loads, the direct loads, as the new blocks and extents are added to the segment, the high water mark does not actually get moved until the actual commit itself. So that's why there is restrictions in the same session or even in other sessions to be able to do anything. So to prevent the errors, the applications had to do a commit immediately after that direct load to prevent those errors from happening. Well now, there are restrictions when that direct load was done prior to that commit for that. The same table in the same session, couldn't query, couldn't do any additional DMLs, couldn't do any additional parallel DMLs. And even in other sessions, queries were not allowed on the same tables that was in use by the other session. So no additional conventional DMLs, no additional parallel DMLs were allowed. 10:09 Lois: Ok.. it was restrictive in what could be done. So, how have direct loads evolved with the 23ai release? Bill: Some of those previous restrictions have been lifted in that same session with that same table. So now you can immediately-- and notice that we're talking here, same session, same table. All right. So you can query multiple times within that same session. You can perform additional DML and you can also do multiple direct loads in the same session without having to do that commit. However, there still are restrictions. It has to be a heap table. It does not work with index organized tables or clustered tables. And the tablespace, if it's not using the automatic segment space management, it cannot-- it does not apply to those either, or if tables with a uniform extents-- tablespace with uniform extents. That's why anything in the temporary table is also included. Why? Because the temporary tablespace has to be uniform extents. 11:17 Nikita: So, what are the restrictions lifted for different sessions on the same table? Bill: Sessions can query that table, can perform conventional DML on that, able to also concurrently perform a direct load, and I can roll back to a save point. So you can see those added features can be very beneficial. But there's still restrictions that apply. It still applies to heap tables only, and it still applies to only tablespaces that are using the automatic segment space management for that. Of course, that includes the temporary tablespace and it doesn't work with tablespaces that have uniform extents. Your application DML might need to query the data after that direct load without committing, applications that might need to modify data within that same transaction as that direct load. You can enable multiple append hint. So you can specify the hint in addition to pending hint to disable. You can specify the no multi-append hint to disable it. 12:27 Lois: Bill, what's the broader impact of these changes. How do these improvements make things more development-friendly? Bill: So changes to the direct load make things a little bit more development friendly by removing those directions after that direct load itself. So previous restrictions when loading-- querying the data kept us from doing multiple things at the same time. So now I can query on that table direct load from the same session, from a different session. I can do conventional DMLs on the table within the same session. It allows me to do a rollback on it. I can do direct loads on the same table within the same session. Again, I can also allow rollback to a save point. As long as my compatibility is set to 21.0.0.0, I will be able to go ahead and benefit from this feature. And there is no increase with it as far as the space usage or causing any fragmentation to the table. So that will not be an issue. 13:35 Nikita: Well, that's the end of our time together, but I want to thank you, Bill, for sharing your expertise with us. Lois: To learn more about what we discussed today, visit mylearn.oracle.com and search for the Oracle Database 23ai New Features for Administrators course. Join us next week for a discussion on some more Oracle Database 23ai new features. Until then, this is Lois Houston… Nikita: And Nikita Abraham signing off! 14:03 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Hosts Lois Houston and Nikita Abraham are joined by Senior Principal Database & MySQL Instructor Bill Millar who explains Oracle's newest caching solution called True Cache. Available in Oracle Database 23ai, True Cache is an automatically managed, in-memory, read-only cache that improves application performance dramatically. Bill provides an overview of its features and highlights the benefits of using True Cache.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/140830/   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Lois: Hello and welcome to the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Principal Technical Editor. Nikita: Hi everyone! Last week, we had quite a power-packed episode. We discussed the 23ai new feature for Automatic SQL Plan Management. We also looked at the 23ai automatic feature that enhances SecureFiles LOB Write Performance as well as the update to Wide Columns. 00:59 Lois: Yeah, and in today's episode, we will look at True Cache, another 23ai new feature. To tell us all about it, we have Bill Millar back with us. Bill is a Senior Principal Database & MySQL Instructor with Oracle University. We'll ask Bill to give us an overview of True Cache, talk about its configuration and deployment, and discuss how to apply True Cache to our applications.  Nikita: To kick things off, Bill, can you give us a high-level overview of what True Cache is? How does it differ from other caching solutions like Redis or Memcached? 01:35 Bill: True Cache is an in-memory cache. It is read-only. True Cache is deployed in front of a primary database, and it is automatically managed. It keeps the most frequently accessed data in the cache, and it keeps the cache consistent with the primary database. They call it diskless, but it's not. It does require some space for SP file, redo logs, control files, and such. But it's very similar to Active Data Guard.  The queries can be offloaded to the True Cache for faster query response. And the data in the query cache is consistent. Unlike other mid-tier caches like Redis or Memcached, a query to the True Cache returns only committed data, and the data is always consistent. It's secure. Why? Because we implement our Oracle database security policies and you can control access to the cache.  02:33 Lois: So, why should we use True Cache?  Bill: Improve application performance without having to rewrite any applications. That can save considerable amount of time, effort, and expense. Reduces the application response time. So the closer the True Cache is to the application, the faster the response. Now, you do need a large amount of memory. We're talking memory here. It's an in-memory storage area, and depending on how you configure it, you can have it shared, you can have it divided. We mentioned it's automatically maintained. So there's no application changes required, and it is transparent to the application. Again, simplifies that development and maintenance.  03:15 Nikita: How does it impact application performance, and what kind of scenarios would benefit the most from implementing True Cache? Bill: So at a high-level view, True Cache or primary database, the application configuration serves as other things that are going to decide where is it going to query the data from, from the True Cache or from the primary database.  The True Cache satisfies that query. And that's where the data will be fetched from. If not, then from the primary database. On start up, True Cache is empty. So it starts reading large chunks of data to populate the True Cache. So after a block is cached, then again, it can be automatically updated, apply the redo to it-- very similar to the Oracle Active Data Guard. In the data returned, it is always going to be consistent.  04:04 Lois: Is it going to be current data? Bill: Maybe, maybe not. If it's been updated in the primary, if they redo apply hasn't occurred yet, then it's not the most consistent. But as far as the query cache is concerned, it is the most current because we only display consistent. You can have multiple True Caches. You can save the same database application service to the True Cache as you can partition it.  04:28 Nikita: I'm curious about the memory requirements, Bill. How crucial is memory for True Cache's performance? Bill: You need to have significant amount of memory. Memory, memory, memory. So True Cache is completely memory, memory. So I want to have all my data possible in there. The more memory you have, the less likely something is going to age out. And of course, just like with the standard caching, you can also pin objects to stay in the True Cache.  Yeah, like I said, there are some requirements for storage, even though it's called diskless because of, again, redo log files, configuration files like the control files, SP file. And again it is read only.  05:11 Lois: Can you explain the differences between using physical and logical connections with True Cache? How does this impact the way applications interact with the database? Bill: So with using the True Cache, we have two physical connections, and we can have one to the primary database and one to the True Cache. Each connection has a database application service associated with it, and it's going to choose which connection to use based whether it's going to go to the True Cache or to the primary database.  The second way is the application maintains one logical connection that uses the application service for the primary database. It's the JDBC thin driver, starting with Oracle Database 23 is available. It's going to maintain the physical connections to the primary database and the True Cache itself. Now, the logical connection, the one logical and one physical, is for Java applications only.  Applications that work with JSON, we extend the HTTP entity tag support for that. So a database GET request to the True Cache is going to compute the ETag, insert it into the return document.  06:27 Nikita: But what happens if there's a mismatch when the modified document is put back into the primary database?  Bill: Well, then the database is going to verify. OK, what happens with that?  It's going to verify the document row still matches that ETag for that. If with that put command, let's say, I have new data here, the row is going to match that ETag that was automatically updated. If there's no match, another user has changed the data and the PUT request is rejected. So the PUT request can be retired using the new data.  07:05 Are you planning to become an Oracle Certified Professional this year? Whether you're a seasoned IT pro or just starting your career, getting certified can give you a significant boost. And don't worry, we've got your back! Join us at one of our cert prep live events in the Oracle University Learning Community. You'll get insider tips from seasoned experts and learn from other professionals' experiences. Plus, once you've earned your certification, you'll become part of our exclusive forum for Oracle-certified users. So, what are you waiting for? Head over to mylearn.oracle.com and create an account to jump-start your journey towards certification today! 07:48 Nikita: Welcome back! Now, how do you configure True Cache, Bill? Bill: You can configure True Cache one of two ways. You can either use the Database Configuration Assistant, which actually makes it a little simpler to configure it, and you can also manually create it.  You have some environment options. One is a uniform configuration where you can deploy identical True Cache that use the same database application service. Another way is partition configuration. The data is going to be divided across multiple True Caches, which, each cache is a different subset of the data. You can also deploy True Cache in a RAC environment. As one might expect, there are some additional configuration steps for a RAC environment.  You want to make sure you verify your configuration, that the database application services are working as expected after you configure it. And then, optionally, you can enable DML redirection. What that will do, it writes data to the primary database, and that data is automatically updated in the cache. It's very similar how to the Oracle Active Data Guard works. Because the DML redirection uses more resources, it's not recommended for update-intensive applications.  There is a parameter, a ADG_REDIRECT_DML initialization parameter, that you will set to True in order to do that.  09:18 Lois: Bill, what are the specific challenges or considerations that administrators should be aware of during the configuration process?  Bill: You do need to make sure your network is configured for True Cache in the primary database. So optionally, you can create a remote listener for high availability. But you create your True Cache. You go ahead, and make sure that you have your primary database. You want the network configuration for both of those. And then you create the True Cache. Once the True Cache is created, you're going to create the application services associated with the database. And then, you're going to start the database application services on the True Cache.  When it comes to naming the application service names, each primary database application is going to be associated with a corresponding True Cache application service. To help simplify things a little bit, in the naming convention, you'll notice in our examples-- for example, if we have SALES as the primary database service, then we have the True Cache, we have SALES_TC, standing for True Cache, so it's easily identified.  You don't have to do that, but it's kind of recommended to do that, some way that you're going to identify it. So we're going to start our True Cache services. And you only start the True Cache services on the True Cache instances. Because it's the database services on the database that you need to make sure are started. And they are read-only.  10:46 Lois: Are there some best practices for maximum availability architecture? Bill: Uniform configuration seems to be a popular one. Why? Because I am going to have the both True Caches can be shared. That way, hopefully, I'm getting full usage out of both. And maybe if I have one service going to one, it might be minimally used. Whereas, the other one might be over. Hey, I could use more memory over here.  We'll also recommend use the JDBC 23ai UCP, Universal Connection Pool, for the application. So that can lessen the impact. If one True Cache becomes unavailable, as far as, OK, I need to reroute over here-- benefit of uniform configuration also. Prepopulate the cache. You want to go ahead and run the critical workload for that. If you have a planned outage, and you need to shut down the True Cache, you want to make sure you stop the database application service on that True Cache.  And then, how are you going to design your True Cache? Are you going to partition it? Are you going to have uniform? Which partition option are you going to use? So you can try to design that to help minimize the number of fetches it has to do from the primary database. And the more you can keep in the True Cache, the better the performance is going to be.  12:09 Nikita: What do I need to keep in mind when it comes to managing True Cache? Bill: One thing you might need to do for managing the True Cache is to monitor the True Cache. There's a couple different ways that we can do it. One, you can use the V$ view, the V$TRUE_CACHE view. And, of course, you can always use the Automatic Workload Repository.  12:30 Lois: Bill, we already spoke about this a bit, but can you tell us more about using True Cache in an application? Bill: There's two ways of using True Cache, as we've seen, physical and logical. Physical, it's going to maintain two connections, front one to the primary database and one to the True Cache. The application can decide which connection to use, based off of what it is trying to do.  If it's just reading, long as it's for a service that's configured with True Cache, it can read the True Cache. If it's going to write something, it's going to update, insert, whatever the case might be, it's for the primary database. And you can use any existing client driver as long as you're using the physical connection method. Any programming language will also work.  With the one logical connection method, it uses the application service for the primary database. You're going to use the JDBC Thin driver, starting with 23ai. You can use it and it maintains the connection to the primary database and True Cache. This model only works with Java applications, though. It maintains the physical connections.  We're going to enable the driver connection. And then, we're going to set the read only. We're going to set it to read only, true. Read only, false, whatever the case might be. And the read only mode is false for a connection by default. False is the default. Java applications only.  14:14 Nikita: What are some best practices for load balancing in a uniform configuration? Bill: You have multiple--multiple True Caches. They're going to service the same database application. They're going to cache the same data. It's the listener that's going to distribute the load balances. So the listener will automatically distribute and load each session to each cache. It will do it randomly and it will do it based off a load. Where can it configure? Where can it send for the best performance.  To route the request to the best performing True Cache, you want to make sure that you are using the same listener. So that remote listener parameter should point to the same listener, which is also the primary database listener. Single instance primary database local listener or scan listener, whichever one you're using, points to the primary. For the application for the JDBC URL, should point to the primary database.  You'll remember that Thin driver is going to create that logical connection, and it's going to create the physical connection to the primary database into each True Cache. To simplify things and possibly avoid connection issues, you might consider using the LISTENER_NETWORK, so the initialization parameter instead of specifying the remote and local listener separately. Because with the local--with the listener networks, all listeners within the same network name will cross register.  15:44 Lois: Before we wrap up, are there any complementary features that you would recommend using alongside True Cache to further enhance performance or simplify management? Bill: There are features that can complement True Cache-- the server-side result set cache.  So you can create--you can go ahead and create the result set that's part of the library cache set aside, a portion of that. You're going to go in, you're going to configure what objects will use that. You can still use that even with True Cache.  There's also the KEEP Buffer Pool that can be used. It's a separate pool that you set aside as part of the buffer cache. And you want to make sure you size it so the object that you want to keep in memory in the buffer cache that you size it appropriately. But again, some configuration, you configure the key pool, plus also you go in and alter the objects to use it.  And then lastly, there's the database smart flash cache. So again, if your data doesn't fit into memory, you can expand the capacity of by adding flash devices. When you configure the flash cache, if you are using transparent data encryption data, the local flash devices is not supported. So if it's TD encrypted on the primary database, it's going to stay in the buffer cache of the primary database.  17:11 Nikita: Ok! I think we can close the episode with that. Thank you, once again, for joining us, Bill.  Lois: Yes thanks! We're learning so much from you. To learn more about what we discussed today, including the various configuration options that are available, visit mylearn.oracle.com and search for the Oracle Database 23ai New Features for Administrators course. Join us next week for a discussion on some more Oracle Database 23ai new features. Until then, this is Lois Houston… Nikita: And Nikita Abraham signing off! 17:46 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Join Lois Houston and Nikita Abraham, along with Senior Principal Database & Security Instructor Ron Soltani, as they discuss how the new Automatic SQL Plan Management feature in Oracle Database 23ai improves performance consistency and simplifies management. Then, Senior Principal Database & MySQL Instructor Bill Millar shares insights into two new features: one that enhances SecureFiles LOB Write Performance, improving read and write speeds, and another that increases the column limit in a table to 4,096, making it easier to handle complex data.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/137192/207062   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Principal Technical Editor with Oracle University, and joining me is Lois Houston, Director of Innovation Programs. Lois: Hi there! Last week, we looked at the Oracle Database 23ai enhancements that have been made to Hybrid Columnar Compression and Fast Ingest. In today's episode, we'll talk about the 23ai new feature for Automatic SQL Plan Management with Ron Soltani, a Senior Principal Database & Security Instructor with Oracle University.  01:01 Nikita: And later on, we'll be joined by Bill Millar, another Senior Principal Database & MySQL Instructor, who will tell us about the 23ai automatic feature that enhances SecureFiles LOB Write Performance. We'll also get him to talk about the Wide Columns update. So, let's get started. Hi Ron! What have been the common challenges with SQL plans and database performance? Ron: One of the problems that we have always had, if you remember, was when data changes, database setting configuration, parameter changes, SQL that were operating very well could now behave badly using the SQL plan that were associated to them. And remember, the same SQL plan generally Oracle likes to continuously reuse.  So the SQL plans were put in the baseline in the past, and we could have those SQL plan baseline, which are a set of approved plans to be used for a SQL from the SQL history stored in AWR, then could be used for the optimizer to choose from. However, which plan to choose and which one would be the best one to use, this is what the problem has been in managing the SQL plan baselines, and a lot of the operation would have been done manually.  02:22 Lois: And what have we done to overcome this?  Ron: So now this new system will going to perform all of those operations automatically for us. Now it can search the Automatic Workload Repository. It can find SQL plans for a particular SQL statement, then look for any alternative plans that may available in alternate sources like SQL tuning sets. And then validate those plans and see if those plans are going to be good and to be used as SQL plan baseline for executing SQL statement by the optimizer. 03:00 Nikita: So we now have the Automatic SQL Plan Management Evolve Advisor to help manage operations automatically, right? Can you tell us a little more about it? How does it ensure optimal performance? Ron: This is an automatic advisor that is created that can go look for different plans and validate the plans by examining them, making sure that they are not causing any regression compared to the previous operation, and then evolve that plan into a good baseline.  This simplifies management of the baseline repository for a SQL statement. So as data changes, as parameters changes, optimizer could come up with different type of plans that are set within this baseline that has been validated to be good baseline for each situational operation. So this way you reduce a lot of hard parsing operations.  04:00 Lois: And how does the SQL Evolve Advisor work, Ron? Ron: First, it will check the AWR to find what are the top SQLs that has been found. Then it will look to see if these top SQLs who did not perform well with the plan that they have, that's why they're top SQL, have other alternative plans that are stored in the SQL plan history, in AWR, or available in any other sources.  Then if it finds any additional plans, it will go ahead and add all of those plans into the plan history. So in the plan history, now you have accumulation of all the plans available in AWR and anything that has been brought from other sources. Then it will test every one of those plans and validate that by use of the plan, the SQL statement will not deprivate and get slower. The performance is either similar or actually better. So normally, there is a percentage that the SQL should improve. So we will then validate these baselines.  And finally, once the baselines or those plans have been validated, they will be accepted, and then they will be added as SQL plan baselines. They will remain in the statement history, in the AWR, and will be available for optimizer for the future use.  05:28 Nikita: What are the benefits of this? Ron: Number one is Autonomous Database. As you know, they want to automate all management, including management of the SQL execution due to changes that are happening for the application, for the data, or the database and its environment.  It totally eliminates any manual intervention for management of the statement, and it can transparently repair any statement that had been affected by a major change.  06:00 Lois: What sort of problems does this feature solve for us? Ron: Of course, this is a performance consistency. We want to make sure that every statement performed to its best performance and any specific changes that may impact those SQL statements would be taken into an account, and a better plan, if available, would then be available for use.  It also improves the application performance level, therefore database service level will get much improvement. And the SQL execution plans will be automatically managed behind the scene by expanding these baselines, by managing all of these baseline history and all of that that is managed by this automatic SQL plan management environment automatically.  06:50 Nikita: And when do we use this?  Ron: If there is a change in a database environment, like you add SGA, the change into the shared pool, change in the size of the buffer cache or any type of storage effects. So all of those can actually affect the SQL execution.  Now all of those changes, including data changes, can cause a SQL plan to not behave very well or behave as well as it was doing before. Therefore, if particular plans do not perform as well as they did before, that affects the performance of the application. This also affects the performance of the database and the instance.  07:35 Lois: So, how do we use this environment?  Ron: Well, best news that I have for you in that is that there is nothing manual needs to be done. All we need to do is, number one, make sure that we enable foreground automatic SQL plan management that we done through the package for the DBMS SPM for SQL plan management.  You will use the package with the configure option, and you enable the auto SPM evolve task, and you set it to auto. Once this is done, now the SQL evolve plan management and advisor are enabled, and they will then monitor your statements, review all of the top SQLs as they are found with all of the ADDM operation, and then do their work in looking for better plans and being able to maintain the SQL plan baselines we talked about.  Now for you to be able to view, monitor, and see how these operations are going, if it is enabled, you can take a look at the DBA SQL plan baseline's view. There are many, many columns in that particular baseline, and there are also columns that has been added that tell you where is the plan generated from, if a plan is approved, and any other user interaction with the plan or settings can then be verified using that DBA SQL plan baseline view.  09:13 Are you looking for practical use cases to help you plan and apply configurations that solve real-world challenges? With the new Applied Learning courses for Cloud Applications, you'll be able to practically apply the concepts learned in our implementation courses and work through case studies featuring key decisions and configurations encountered during a typical Oracle Cloud Applications implementation. Applied learning scenarios are currently available for General Ledger, Payables, Receivables, Accounting Hub, Global Human Resources, Talent Management, Inventory, and Procurement, with many more to come! Visit mylearn.oracle.com to get started. 09:54 Nikita: Welcome back! Let's bring Bill into the conversation. Hi  Bill! Can you tell us about the 23ai automatic feature that enhances SecureFiles LOB Write Performance?  Bill: The key here is that it is automatic and transparent. There's no parameters set. Nothing to configure in table, no hints, and nothing that you have to do with these improvements. It is tightly integrated with SecureFiles LOB infrastructure.  So now, multiple LOBs can be handled in a single transaction and can be buffered simultaneously. This will help with mixed workloads, switching between the LOBs that are writing in a single transaction. The PGA will adaptively resize based off the size for these large writes for the LOBs if you're using the No Cache option. Remember, no cache is going to bypass the buffer cache and does direct reads and writes from the PGA.  JSON type will be transformed into the OSON Oracle data type. It is an optimized native binary storage format for JSON data.  11:15 Lois: Ok. So, going forward, there will be better read and write performance for LOBs. Bill: Multiple LOBs in a single transaction can be buffered simultaneously, improving mixed workloads. We just talked about the PGA. Automatically, the buffer is automatically resized.  And the improved JSON support. The reason it will recognize, hey, this is a JSON data type. But traditionally, JSON data types were small.  So they were small to medium size. So the range from 32k to 32 meg was considered small to medium whereas LOBs were designed for data types larger than 100 meg. So by recognizing this a JSON data type, it can take advantage of the LOB architecture.  Other enhancements will also include the acceleration of compressed LOBs, the pen and compression caching, and improves the poor performance of your reads and writes to compressed LOBs. It's faster than previously.  12:24 Nikita: Bill, what do you think about the recent increase in the column limit? Previously, the limit was 1000 columns per table, which sometimes posed issues when migrating from other systems that allowed more than 1,000 columns, right?  Bill: Maybe because of workload requirements, the whole machine learning, the internet of things workloads, IOTs can have hundreds of thousands of attributes, dimensional attribute columns for that. And even our very own blockchain tables reserves up to 40 hidden virtual columns, so that takes away from the total amount.  Virtual columns count towards the column limits and some applications as they drop columns, what it does, it just converts them to unused, and it still applies towards the limit the number of columns that you can have to that limit. There were workarounds. However, they were most likely not the best way to do it, like column switching, table splitting for that. But big data really use cases, really saw where files have or required more than 1,000 columns.  13:42 Lois: So, now that we can have 4,096 columns in a table, I'm sure it's made handling complex data a lot easier.  Bill: So by increasing this, since other systems do support higher column limits, it can-- the increase can make migration from other systems easier and possibly even a little bit more attractive while it can make applications a little bit simpler because the 1,000 column limit was not always optimal for analytics. Where 1,000 might have been plenty for OLTP type environments, but not for the analytics, especially when it comes to machine learning and those internet of things that we talked about, where the previous workarounds, like splitting the tables, really caused more performance issue than anything else.  So we want to avoid those suboptimal workarounds. And the nice thing is there's no change to the SQL. So once you have that-- well, if we were doing SQL, if we had tables that were split and we're trying to do things that is actually going to help improve that SQL, now, we don't have multiple objects that we're dealing with.  14:57 Nikita: How do we actually go about increasing the column limit to 4,096? Bill: You do have to have the compatibility set to 23c. Why? Because it's a new feature. There is a new initialization parameter called Max columns, and you do set that. There's two different ways, two different values. We can set it to standard or we can set it to extended.  It is dynamic. When it's set to standard, it's only 1,000. When we set it to extended, it's going to allow the 4,096. It is modifiable at the PDB level. However, it will inherit what's at the root level, if it's not explicitly set at a PDB. It can't alter it in a session for that. And multiple instances of the RAC environment must use the same value.  Now one thing, notice that it cannot be set to standard if I created a table that had more than 1,000 columns. One thing that might get you, when you drop a table that has more 1,000 columns and you try to set it back to standard, it might tell you, hey, you have tables that have more than 1,000 columns. Don't forget your recycle bin unless you did a drop table purge.  16:09 Lois: Are there any performance considerations to keep in mind, Bill? Bill: There's really no DML or query performance degradation for the tables. However, it might require, as you would expect, the increase in memory when we have the new column limits. It might require additional shared pool, additional SGA with the additional columns, more buffer cache as we're bringing blocks in.  So that's shared pool along with the PGA. And also we can add in buffer cache in there, because that increased column count is going to be increase in the total PGA memory usage. And those are kind of expected for that. But the big advantage is it gives us the ability to eliminate some of these suboptimal workarounds that we had in the past.  17:02 Nikita: Ok! We covered a lot today so thank you Bill and Ron.  Lois: To learn more about what we discussed today, visit mylearn.oracle.com and search for the Oracle Database 23ai New Features for Administrators course. Join us next week for a discussion on some more Oracle Database 23ai new features. Until then, this is Lois Houston… Nikita: And Nikita Abraham signing off! 17:27 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, hosts Lois Houston and Nikita Abraham speak with Senior Principal Database & MySQL Instructor Bill Millar about the enhanced performance of Hybrid Columnar Compression, the different compression levels, and how to achieve the best compression for your tables. Then, they delve into Fast Ingest, what's new in Oracle Database 23ai, and the benefits of these improvements.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/137192/207062   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Lois: Hello and welcome to the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Principal Technical Editor. Nikita: Hi everyone! In our last episode, we spoke about the 23ai improvements in time and data handling and data storage with Senior Principal Instructor Serge Moiseev. Today, we're going to discuss the enhancements that have been made to the performance of Hybrid Columnar Compression. We'll look at how Hybrid Columnar Compression was prior to 23ai, learn about the changes that have been made, talk about how to use this compression in 23ai, and look at some performance factors. After that, we'll move on to Fast Ingest, the improvements in 23ai, and how it is managed. 01:15 Lois: Yeah, this is a packed episode and to take us through all this, we have Bill Millar back on the podcast. Bill is a Senior Principal Database & MySQL Instructor with Oracle University. Hi Bill! Thanks for joining us. So, let's start with how Hybrid Columnar Compression was prior to 23ai. What can you tell us about it? Bill: We support all kinds of platforms from the Database Enterprise Edition on up to the high engineered systems for that and even the Exadata Cloud at the Customer. We have four different levels of compression. One is considered the warehouse compression where we do a COLUMN STORE COMPRESS FOR QUERY LOW and COLUMN STORE COMPRESS FOR QUERY HIGH. The COLUMN STORE COMPRESS FOR QUERY HIGH is the default, unless another compression level is specified. With the archive compression, we have the COLUMN STORE COMPRESSED FOR ARCHIVE LOW and also COLUMN STORE COMPRESS FOR ARCHIVE HIGH. With the Hybrid Columnar Compression warehouse and archive, the array inserts are compressed immediately. But, however, some conditions have to be met. It has to be a locally-- to use these, it has to be a locally managed tablespace, the automatic segment space management. And compatibility level, at least 12 too or higher when these values have been introduced. There are different compressors that are used for the compression hidden from the customer. It just depends on what is selected as to what is going to be the compression that's going to be used for--  notice that with the COLUMN STORE FOR QUERY HIGH and for ARCHIVE LOW, the zlib compression method is used, whereas if you select the ARCHIVE HIGH, the Bzip2. And in 19C, we added the Zstandard. And it's available for the MEMORY COMPRESS FOR CAPACITY HIGH.  03:30 Nikita: So, what's happened in 23ai? Bill: When in 23c, to take advantage of the changes in compression, the compatibility level has to be set at least to 23.0.0 or higher. When a table is created or altered with the hybrid column compression, the Zstandard will automatically be selected. So it doesn't matter which one of the four you select, that will be the one that is selected. It is internally set transparent to the user. There is no new SQL format that has to be used in order for the Zstandard compression to be applied. And the Database Compatibility Mode has to be at least at 23.0.0 or higher. Only then can the format of the Hybrid Column Compression storage use that Zstandard compression. If we already have compressed data blocks in existing tables, they're going to remain in their original format.  04:31 Lois: And are the objects regenerated? Bill: If the objects are-- they might be regenerated if they were deleted in another operation. If you want to completely take advantage of the new compression, all you have to do is alter table move. And that's going to go ahead and trigger the recompression of that, whereas any newly created tables that are created will use the Zstandard by default. 05:00 Nikita: What are the performance factors we need to think about, Bill? Bill: There are some performance factors that we do need to consider, the ratio, the amount of space reduction in storage that we're going to achieve, the time spent compressing the data, the CPU cost to compress that data, and also, is there any decompression rate, time spent decompressing the data when we're doing queries on it? 05:24 Lois: And not all tables are equal, are they? Bill: Not all tables are equal. Some might get better performance by different compression level than others for that. So how we can basically have to test our results, there is a compression advisor that's available, that you can use to give you a recommendation on what compression to use. But only through testing can we really see the availability, the benefits of using that compression for an application. So best compression, just as in previous versions, the higher the compression levels, the more CPU it's going to use. The higher the compression level, the more space savings that we're going to achieve for that as we are doing those direct path inserts. So there's always that cost. 06:20 Did you know that the Oracle University Learning Community regularly holds live events hosted by Oracle expert instructors. Find out how to prepare for your certification exams. Learn about the latest technology advances and features. Ask questions in real time and learn from an Oracle subject matter expert. From Ask Me Anything about certification to Ask the Instructor coaching sessions, you'll be able to achieve your learning goals for 2024 in no time. Join a live event today and witness firsthand the transformative power of the Oracle University Learning Community. Visit mylearn.oracle.com to get started.  07:01 Nikita: Welcome back! Let's now move on to the enhancements that have been made to fast ingest. We'll begin with an overview of fast ingest, how to use it, and the improvements and benefits. And then we'll look at some features for managing fast ingest. Bill, why don't you start by defining fast ingest for us?  Bill: Traditionally the fast ingest, also referred to as deferred inserts, is faster than processing a single row at a time. It can support high-volume transactions like from the Internet of Things applications, where you have hundreds of thousands of items coming in trying to write to the database. They are faster, because the inserts don't use the traditional buffer cache. They use a pool that will size out of the large pool. And then they're later written to disk using the SMCO, the space management coordinator. Instead of using the buffer cache, they're going to write into an area of the large pool. The space management coordinator, it has these helper threads, however many-- that's just a number for that-- that will buffer. And as buffer is filled based off size of that algorithm, it will then write those deferred inserts into the database itself. 08:24 Lois: So, do deferred inserts support constraints? Bill: Deferred writes do support constraints in index just as for regular inserts. However, performance benchmarks that have been done recommend that you disable constraints, if you're going to use the fast ingest. 08:41 Lois: Can you tell us a bit about the streaming and ingest mechanism?  Bill: We declare a table with the memoptimize for write. We can do that in the create table statement, or we can alter the table for that. The data is written to the large pool, unlike traditionally writing items to the buffer cache. It's going to write to the ingest buffer, the large pool. And it's going to be drained. It's going to be written from that area by using those background processes to write to the actual database itself. So the very high throughput, since drainers issues deferred writes in large batches. So we're not having to wait especially for the buffer cache. OK, I need space. OK, I need to write. I need to free up blocks. Very ideal for these streaming inserts, sensor readings, alarms, door locks. Those type of things. 09:33 Nikita: How does performance improve with this? Bill: With the benchmarks we have done, we have found that the performance can be up to 75% faster by going ahead and doing the fast ingest versus traditional inserts. The 23 million inserts per second on a single X6-2 server with the benchmarks that we have. 09:58 Nikita: Are there any considerations to keep in mind? Bill: With the fast ingest, some things to consider for that. The written data, you might need to validate to make sure it's there. So you might have input files that are writing to that that are loading it. You might want to hang on to those, before that data destroyed. Have some kind of mechanism to validate, yes, it was written. There is a possible loss of data. Why? Because unlike the buffer cache that has the recovery mechanism with the redo and the undo, there is none with that large pool. So that's why if the system crashes, and the buffers haven't been flushed yet, then it's possible loss of data. There's no queries from the large pool meaning that if I want to query the information that the fast ingest is loading into the table, it doesn't go and see what's sitting in the buffer in the large pool like it does with the buffer cache. Index and constraints are checked but only at flush time. And the memoptimize pool size is a fixed amount of space that we're going to allocate-- of memory that we're going to allocate to use for the memoptimize write. We can enable a table for the fast ingest, enable with the memoptimize for write. We can create a table and do it. We can also alter a table. We already have a table existing. All we have to do is alter it. And we want to use that, the fast ingest, for these tables. 11:21 Lois: Do we have options for the writing operation, Bill? Bill: You do have options for the writing operation. We have the parameters, the memoptimize write where we can turn that on. We can also use it in a hint. It is set at the root level, it. Is not modifiable at the PDB level. It's set at the root level, It is a static parameter. We can also do things in our session. We want to verify, OK, is the memoptimize write on? We can verify a table is enabled. So with the fast ingest, the data inserts, you can also use a hint. You can also set this at a session level.  If you decide there's something that you don't want to use the memoptimize write for, then you can disable it for a table.  12:11 Nikita: Bill, what are some of the benefits of the enhancements made in 23ai? Bill: With some of the enhancements-- so now, some table attributes are now supported-- we can now have common default values for a column. We can use transparent data encryption. We can also use the fast inserts, any inline LOBs, along with virtual columns. We've also added partitioning support. We can do subpartitioning and we can also do interval partitioning, along with auto list. So we've added some items that previously prevented us from doing the fast inserts. It does provide additional flexibility, especially with the enhancements and the restrictions that we have removed. It allows to use that fast insert, especially in a data warehouse-type environment. It can also use-- in the Cloud, it can use encrypted tablespaces, because remember, in the Cloud, we always encrypt, by default, users' data. So now, it also gives us the ability to use it in that Cloud environment because of that change. We have faster background flushing for the loads.  13:36 Lois: And how is it faster now?  Bill: Because we bypassed the traditional buffer cache. Faster ingest for those direct ingest. So again, bypassing the traditional inserts and using the buffer cache gives the ability to bulk load into large pool, then flush to the database so that way, we have access to that data for possible faster analytics of those internet of things, especially when it comes to the temperature of the temperature sensors. We need to know when a temperature of something is out of bounds very quickly. Or maybe it's sensors for security. We need to know when there's a problem with the security. 14:20 Nikita: How difficult is it to manage this? Bill: Management is fairly simple. We have the MEMOPTIMIZE_WRITE_AREA_SIZE parameter that we're going to say-- it is dynamic. It does not require a restart. However, all instances in a RAC environment must have the same value. So we have the write area. What are we going to set? And then the MEMOPTIMIZE_WRITE, by default, it uses a hint. Or we can go ahead and we can just set that to all. It is allocated from the large pool. You manually set it. And we can see how much is actually being allocated to the pool. We can go out and look at our alert log for that information.  There's also a view. The MEMOPTIMIZE_WRITE_AREA has some columns. What is the total memory allocated for the large pool? How much is currently used by the fast ingest? How much free space? As you're using it, you might want to go out and do a little checking, or do you have enough space? Are you not allocating enough space? Or have you allocated too much?  It'll also show the total number of writes, and also, the number-- the writers is currently the users that are using it.  And the container ID, what is the container within that container database? What's the pluggable or pluggables that's using the fast ingest? There is a subprogram, the DBMS_MEMOPTIMIZE that we have access to that we possibly can use. So there are some procedures. Here, we can return the rows of the low and high water mark of the sequence numbers. And the key here is across all the sessions. We can see the high water mark, sequence number of the rows written to the large pool for the current session. And we can also flush all the ingest data from the large pool to disk for the current session. 16:26 Lois: What if I want to flush them all for all sessions?  Bill: Well, that's where we have the WRITE_FLUSH procedure. So it's going to flush the fast ingest data of the Memoptimize Rowstore from the large pool for all the sessions. As a DBA, that's one that you most likely will want to be using, especially if it's going to be before I do a shutdown or something along that line. 16:49 Nikita: Ok! On that note, I think we can end this episode. Thank you so much for taking us through all that, Bill. Lois: Yes, thanks Bill. If you want to learn more about what we discussed today, visit mylearn.oracle.com and search for Oracle Database 23ai New Features for Administrators. Join us next week for a discussion on some more Oracle Database 23ai new features. Until then, this is Lois Houston… Nikita: And Nikita Abraham signing off! 17:21 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, hosts Lois Houston and Nikita Abraham discuss improvements in time and data handling and data storage in Oracle Database 23ai. They are joined by Senior Principal Instructor Serge Moiseev, who explains the benefit of allowing databases to have their own time zones, separate from the host operating system. Serge also highlights two data storage improvements: Automatic SecureFiles Shrink, which optimizes disk space usage, and Automatic Storage Compression, which enhances database performance and efficiency. These features aim to reduce the reliance on DBAs and improve overall database management.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/137192/207062   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Principal Technical Editor with Oracle University, and joining me is Lois Houston, Director of Innovation Programs. Lois: Hi there! Over the past two weeks, we've delved into database sharding, exploring what it is, Oracle Database Sharding, its benefits, and architecture. We've also examined each new feature in Oracle Database 23ai related to sharding. If that sounds intriguing to you, make sure to check out those episodes. And just to remind you, even though most of you already know, 23ai was previously known as 23c. 01:04  Nikita: That's right, Lois. In today's episode, we're going to talk about the 23ai improvements in time and data handling and data storage with one of our Senior Principal Instructors at Oracle University, Serge Moiseev. Hi Serge! Thanks for joining us today. Let's start with time and data handling. I know there are two new changes here in 23ai: the enhanced time zone data upgrade and the improved system data and system timestamp data handling. What are some challenges associated with time zone data in databases? 01:37 Serge: Time zone definitions change from time to time due to legislative reasons. There are certain considerations. Changes include daylight savings time when we switch, include the activity that affects the Oracle Database time zone files.  Time zone files are modified and used by the administrators. Customers select the time zone file to use whenever it's appropriate. And customers can manage the upgrade whenever it happens.  The upgrades affect columns of type TIMESTAMP with TIME ZONE. Now, the upgrades can be online or offline.  02:24 Lois: And how have we optimized this process now?  Serge: Oracle Database 23c improves the upgrade by reducing the resources used, by selectively using the updates and minimizing the application impact. And only the data that has dependencies on the time zone would be impacted by the upgrade.  The optimization of the time zone file upgrade does not really change the upgrade process, so upgrade can be done offline. Database would be unavailable for a prolonged period of time, which is not optimal for today's database availability requirements.  Online upgrade, in this case, we want to minimize the application impact while the data is being upgraded. With the 23c database enhancement for time zone file change handling, the modified data is minimized, which means that the database updates only impacted rows. And it reduces the impact to the applications and other database operations.  03:40 Nikita: Serge, how does updating only the impacted rows improve the efficiency of the upgrade process? Serge: The benefits of enhanced timezone update include customers who manage large fleet of databases. They will benefit tremendously with a lower downtime. The DBAs will benefit due to the faster updates and less resource consumption needed to apply those updates. And that improves the efficiency of the update process.  Tables with no affected data are simply skipped and not touched. All results in the significant resource savings on the upgrade of the time zone files. It applies to all customers that utilize timestamp with time zone columns for their data storage.  04:32 Lois: Excellent! Now, what can you tell us about the improved system data and system timestamp data handling?  Serge: Date and time in Oracle databases depends on the system time as well as the database settings. System time now can be set as the local time zone for an individual database.  04:53 Nikita: How was it before this update? Serge: Before 23c, the time has always matched the time zone of the database host operating system. Now, imagine that we use either multitenant environments or cloud-based environments when the host OS system time zone is not really the same as the application that runs in a different geographic locality or affects data from other locations.  And system time obviously applies not only to the data stored and updated in the database rows but also to the scheduler, the flashback, to a place to materialized view refresh, Recovery Manager, and other time-sensitive features in the database itself.  Now, with the database time versus operating system time, there is a need to be more selective. It is desired that the applications use the same database time in the same time zone as the applications are actually being used in.  And multitenant and cloud databases will certainly experience a mismatch between the host operating system time zone, which is not local for the applications that run in some other geographical locations or not recognizing some, for example, daylight savings time.  So migration challenge is obviously present. If you want to migrate from a specific on-premises database to either multitenant or cloud, you would experience the host operating system time zone by default.  06:38 Lois: And that's obviously not convenient for the applications, right? Serge: Well, the database-specific time in Oracle Database 23c, any cloud database can set local time zone to whatever the customer's requirements are explicitly. And any pluggable database can also set its own local time zone to customer's requirements, not inheriting the time zone from the container database it is currently running in.  This simplifies migration to multitenant or cloud for applications that are time-sensitive. And it offers more intuitive, easier database monitoring, and development.  07:23 Working towards an Oracle Certification this year? Take advantage of the Certification Prep live events in the Oracle University Learning Community. Get tips from OU experts and hear from others who have already taken their certifications. Once you're certified, you'll gain access to an exclusive forum for Oracle-certified users. What are you waiting for? Visit mylearn.oracle.com to get started.   07:51 Nikita: Welcome back! Let's move on to the data storage improvements. We have two updates here as well, automatic secure file shrink and automatic storage compression. Let's start with the first one. But before we get into it, Serge, can you explain what SecureFiles are?  Serge: SecureFiles are the default storage mechanism for large objects in Oracle Database. They are strongly recommended by Oracle to store and manage large object data.  The LOBs are stored in segments. Those segments may incur large amounts of free space over time. Because of the updates to the LOB data, the fragmentation of the space used is growing depending, of course, on the frequency and the scope of the updates.  The storage efficiency could be improved by shrinking segments with the free space removed. And manual secure files shrinking has become available since Oracle Database 21c, requiring administrators to perform these tasks manually.  Traditional SecureFiles required the time-consuming DBA activities. DBAs would need to manually identify eligible LOB segments either using Segment Advisor or PL/SQL or built-in database views.  Once identified, the administrators would manually execute shrink operations on very large LOBs which takes too much time and may result in excessive disk space consumption. For example, code to operate this shrinking would look like ALTER TABLE some table SHRINK SPACE CASCADE.  That would shrink all LOB segments in a particular table. If you want to scope the shrinking to a single column, the code would be required to ALTER TABLE some table MODIFY LOB, followed by the column name SHRINK SPACE.  This affects only a single column in a table with LOBs.  10:01 Lois: So, how has automatic secure shrinking made things better? Serge: Automatic SecureFile shrink removes the emphasis from the DBAs to manually perform these tasks. And it results in the more optimal use of space over time.  It is integrated into the automated database maintenance tasks. The automation once enabled runs every 30 minutes, collects eligible LOB segments, and shrinks them offline. The execution time and freed space would vary depending on the fragmentation and the size of the LOBs. Each shrink execution may reclaim up to 5 gigabytes of unused disk space from each LOB segment that is idle.  On the high level, automatic SecureFile shrink improves the Oracle Database 23c storage usage efficiency. It is part of the ongoing Oracle Database improvement effort and transparently reclaims the free space with negligible to no impact on performance of the database operations.  Again, this is done in the background without affecting the running processes. It makes Oracle database 23c less dependent on the DBA activities while reducing the disk space required to store SecureFiles, reducing the usage of LOB segments.  Automatic securefile shrink runs incrementally in small steps over time. Some of the features are tunable. And it is supported for all types of large objects, storage, compressed, encrypted, and duplicated the object segments.  11:50 Nikita: Right, and note that this feature is turned on out-of-the-box in the Autonomous Database 23ai in Oracle Cloud. Now, let's talk about Automatic Storage Compression, Serge.  Serge: With Automatic Storage Compression and Automatic Clustering, the storage compression gives you the background compression functionality. Directly loaded data is first uncompressed to speed up the actual load process. Rows are then moved into hybrid columnar compression format in the background asynchronously.  The automatic clustering applies advanced heuristic algorithms to cluster the stored data depending on the workload and data access patterns and the data access is optimized to more efficiently make use of database table indices, zone maps, and join zone maps.  Automatic Storage Compression advantages include the improvements to Oracle Database 23c storage efficiency as well. It is part of the continuous improvement, part of the ongoing Oracle Database improvement effort. And it brings performance gains, speeds up uncompressed data loads while compressing in the background.  The latencies to load and compress data are because of that also reduced. With the hybrid columnar compression in particular, this works in combination.  And it results in less DBA activities, makes the Database Management less dependent on the DBA time and availability and effort.  Automatic Storage Compression performs operations asynchronously on the data that has already been loaded. To control Automatic Storage Compression on-premises, it must be enabled explicitly. And you have to have heatmap enabled on your Oracle Database objects.  Table must use hybrid columnar compression and be placed on the tablespace with the SEGMENT SPACE MANAGEMENT AUTO and allowing autoallocation. And this feature, again, is transparent for the Autonomous Database 23c in the Oracle Cloud. 14:21 Lois: Thanks for that quick rundown of the new features, Serge. We really appreciate you for taking us through them. To learn more about what we discussed today, visit mylearn.oracle.com and search for the Oracle Database 23ai New Features for Administrators course. Join us next week for a discussion on some more Oracle Database 23ai new features. Until then, this is Lois Houston… Nikita: And Nikita Abraham signing off! 14:50 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Join hosts Lois Houston and Nikita Abraham in Part 2 of the discussion on database sharding with Ron Soltani, a Senior Principal Database & Security Instructor. They talk about sharding native replication, directory-based sharding, and coordinated backup and restore for sharded databases, explaining how these features work and their benefits. Additionally, they explore the automatic bulk data move on sharding keys and the ability to split and move partition sets, highlighting the flexibility and efficiency they bring to data management.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/137192/207062   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative  podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Lois: Hello and welcome to the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Principal Technical Editor. Nikita: Hi everyone! In our last episode, we dove into database sharding and Oracle Database Sharding in particular. If you haven't listened to it yet, I'd suggest you go back and do so before you listen to this episode because it will give you a lot of context.  00:53 Lois: Right, Niki. Today, we will discuss all the 23ai new features related to database sharding. We will cover sharding native replication, directory-based sharding, coordinated backup and restore for sharded databases, and a few more.  Nikita: And we're so happy to have Ron Soltani back on the podcast. If you don't already know him, Ron is a Senior Principal Database & Security Instructor with Oracle University. Hi Ron! Let's talk about sharding native replication, which is RAFT-based, meaning that it is reliable and fault tolerant-based, usually providing subzero or subsecond zero data loss replication support. Tell us more about it, please.  01:33 Ron: This is completely transparent replication built in within Oracle sharding that duplicates data across the different shards. So data are generally put into chunks. And then the chunks are replicated either between three or five different shards, depending on how much of the fault tolerance is required.  This is completely provided by the Oracle sharding database, and does not require use of any other component like GoldenGate and Data Guard. So if you remember when we talked about the architecture, we said that each shard, each database can have a Data Guard component, whether through GoldenGate or whether through Data Guard to have a standby.  And that way support high availability with the sharding native replication, you don't rely on the secondary database. You actually-- the shards will back each other up by holding replicas and being able to globally manage the replica, make sure everything is preserved, and manage all of the fault operations.  Now this is a logical replication, generally consensus-based, kind of like different components all aware of each other. They know which component is good, depending on the load, depending on the failure. The sharded databases behind the scene decide who is actually serving the data to the client. That can provide subsecond failovers with zero data loss.  03:15 Lois: And what are the benefits of this? Ron: Major benefits for having sharding native replication is that it is completely transparent to the application or any of the structures. You just identify that you want to go ahead and use this replication and identify the replication factor. The rest is managed by the Oracle sharded database behind the scene.  It supports fast failover with zero data loss, usually subsecond failovers. And depending on the number of replicas, it can even tolerate multiple failures like two server failures.  And when the loads are submitted, the loads are also load-balanced across all of these shards based on where the data is located, based on the replicas. So this way, it can also provide you with a little bit of a better utilization of the hardware and load administration.  So generally, it's designed to help you keep your regular SQL-based databases without having to resolve to FauxSQL or NoSQL environment getting into other databases. 04:33 Nikita: So next is directory-based sharding. Can you tell us what directory-based sharding is, Ron?  Ron: Directory-based sharding basically allows the user to define the values that are used and combined for different partition, so better control, location of the data, in what partition, what shard. So this allows you to set up a good configuration.  Now, many times we may have a key that may not be large enough for hash partitioning to distribute the data enough. Sometimes we may not even know what keys are going to come in the future. And these need to be built in the future. So having to build these, you really don't want to have to go reorganize the whole data based on new hash functions, and so when data cannot be managed and distributed using hash partitioning or when we need full control over combination of where data exists.  05:36 Lois: Can you give us a practical example of how this works? Ron: So let's say our company is very small in three different countries. So I can combine those three countries into one single shard. And then have three other big countries, each one sitting in their own individual shards. So all of this done through this directory-based sharding. However, what is good about this is the directory is created, which is a table, created behind the scene, stored in the catalog, available to the client that is cached with them, used for connection mapping, used for data access. So it can give you a lot of very high-level benefits.  06:24 Nikita: Speaking of benefits, what are the key advantages of using directory-based sharding? Ron: First benefit allow you to group the data together based on the whatever values you want, depending on what location you want to put them as far as across the shards are concerned. So all of that is much better and easier controlled by us or by the designers. Now, this is when there is not enough values available. So when you're going to use hash-based partition, that would result into an uneven distribution of the data.  Therefore, we may be able to use this directory for better distribution of the data since we understand the data structure better than just the hash function. And having a specification where you can go ahead and create future component, future partitions, depending on how large they're going to be. Maybe you're creating them with an existing shard, later put them in another shard. So capability of having all of those controls become essential for management of this specific type of data.  If a shard value, the key value is required, for example, as we said, client getting too big or can use the key value, split it or get multiple key value. Combine them. Move data from one location to another. So all of these components maintain automatically behind the scene by us providing the changes. And then the directory sharding and then the sharded database manages all of the data structure, movement, everything behind the scene using some of the future functionalities. And finally, large chunk of data, all of that can then be moved from one location to another. This is part of the automatic chunk data move and whatnot, but utilized within the directory-based sharding to allow us the control of this data and how we're going to move and manage the data based on the load as the load or the size of the data changes.  08:50 Lois: Ron, what is the purpose of the coordinated backup and restore system in Oracle Database Sharding? Ron: So, basically when we talk about a coordinated backup and restore, remember in a sharded database, I have different databases. Each database is a shard. When you take a backup, each database creates its own backup.  So to have consistent data across all of the shards for the whole schema, it is extremely important for these databases to be coordinated when the backup is taken, when the restore is being done. So you have consistency of the data maintained across all of the shards.  09:28 Nikita: So, how does this coordination actually happen? Ron: You don't submit this through our main. You submit this through the Global Management tool that is used for the sharded database. And it's the Global Management tool that is actually submit your request to each database, but maintains the consistency of when the actual backup is taken, what SCN.  So that SCN coordination across all of the shards is then maintained for the backup so you can create a consistent backup or restore to a consistent point in time across the sharded database. So now this system was enhanced in 23C to support multiple destinations.  So you can now send your backup to an object store. You can send it to ZDLRA. You can send it to Amazon S3. So multiple locations can now be defined where you can send these backups to. You can also use multiple recovery catalogs.  So let's say I have data that is located on different countries and we have requirement that data for each country must stay in that country. So I need to also use a separate catalog to maintain that partition.  So now I can use multiple catalog and define which catalog is maintaining which partition to satisfy those type of requirements or any data administration requirement when it comes to backup recovery. In addition, you can also now specify different type of encryption to be used, whether you want to have different type of encryption algorithm for each of the databases that you're backing up that is maintained. It can be identified, and then set up for each one of those components.  So these advancements now allow you to manage this coordinated backup and restore with all of the various specific configuration that may be required based on the data organization. So the encryption, now can also be done across that, as I mentioned, for different algorithms. And you can define different components.  Finally, there is much better error handling and response available through this global system. Since things have been synchronized, you get much better information into diagnosing any issues.  12:15 Want to get the inside scoop on Oracle University? Head over to the Oracle University Learning Community. Attend exclusive events. Read up on  the latest news. Get first-hand access to new products. Read the OU Learning Blog. Participate in challenges. And stay up-to-date with upcoming certification opportunities. Visit www.mylearn.oracle.com to get started.  12:41 Nikita: Welcome back! Continuing with the updates… next up is the automatic bulk data move on sharding keys. Ron, can you explain how this works and why it's significant? Ron: And by the way, this doesn't have to be a bulk data. This could be just an individual row or it could be bulk data, a huge piece of data that is going to be moved.  Now, in the past, when the shard key of an existing record was going to be updated, we basically had to remove that row from the table, so moving it to a temporary table or moving it to another location. Basically, you're deleting the row, and then change the value and reinsert the row so the row would then be inserted into the proper location.  That causes a lot of work and requires specific code-writing and whatnot to manage those specific type of situations. And of course, if there is a lot of data, now, you're moving those bulk data in twice.  13:45 Lois: Yeah… you're moving it to one location and then moving it back in. That's a lot of double work, not to mention that it all needs to be managed manually, right? So, how has this process been improved? Ron: So now, basically, you can just go ahead and update the value of the partition key, and then data will then automatically move to the new location. So this gives you complete flexibility of the shard key values.  This is also completely transparent, and again, completely managed behind the scenes. All you do is identify what is going to be changed. Then the database will maintain the actual data location and movement behind the scenes.  14:31 Lois: And what are some of the specific benefits of this feature? Ron: Basically, it allows you to now be flexible, be able to update the shard key without having to worry about, oh, which location does this value have to exist? Do I have to delete it, reinsert it? And all of those different operations.  And this is done automatically by Oracle database, but it does require for you to enable row movement at the table level. So for tables that are expected to have partition key updates kind of without knowing when that happens, can happen, any time it happens by the clients directly or something, then we may need to enable row movement at the table level and leave it enabled. It does have tiny bit of overhead of maintaining these row locations behind the scenes when enabled, as it maintains some metadata behind the scenes.  But for cases that, let's say I know when the shard key is going to be changed, and we can use, let's say, a written procedure or something for that when the particular shard key is going to be changed. Then when the shard key is updated, the data will then automatically move to the new location based on that shard key operation. So we don't need to move the data manually in and out or to different locations.  16:03 Nikita: In our final segment, I want to bring up the update on splitting and moving a partition set, or basically subpartitioning tables and then being able to move all of the data associated with that in a bulk data move to a new location. Ron, can you explain how this process works?  Ron: This gives us a lot of flexibility for data management based on future requirements, size of the data, key changes, or key management requirements.  So generally when we use a composite sharding, remember, this is a combination of user-defined partitioning plus the system partitioning put together. That kind of defines a little bit more control over how the shards are, where the data is distributed evenly across the shards.  So sometimes based on this type of configuration, we may actually need to split partition and that can cause the shard key values to be now assigned to a new shard space based on the partitioning reconfiguration. So data, this needs to be automatically managed. So when you go ahead and split partition or partitionsets, then the data based on your configuration, based on your identification can automatically move to the new location automatically between those shard spaces.  17:32 Lois: What are some of the key advantages of this for clients? Ron: This provides a huge benefit to clients because it allows them flexibility of better managing their configuration, expanding both configuration servers, the structures for better management of the data and the load. Data is completely online during all of this data move. Since this is being done behind the scenes by the database, it does not impact the availability of the data for anyone who is actually using the data.  And then, data is generally moved using transportable tablespaces in big bulk and big chunks. So it's almost like copying portions of the files. If you remember in Oracle database, we could take a backup of big files as image copy in pieces. This is kind of similar where chunks of data can then be moved and then transported if possible depending on the organization of the data itself for those particular partitions.  18:48 Lois: So, what does it look like in practice? Ron: Well, clients now can go ahead and rearrange their data structure based on the adjustments of the partitioning that already exists within the sharded database. The bulk data move then automatically triggers once the customer execute the statement to go ahead and restructure the partitioning. And then all of the client, they're still accessing data. All of the data operation are completely maintained behind the scene.  19:28 Nikita: Thank you for joining us today, Ron. If you want to learn more about what we discussed today, visit mylearn.oracle.com and search for the Oracle Database 23ai New Features for Administrators course. Join us next week for a discussion on some more Oracle Database 23ai new features. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 19:51 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this two-part episode, hosts Lois Houston and Nikita Abraham are joined by Ron Soltani, a Senior Principal Database & Security Instructor, to discuss the ins and outs of database sharding. In Part 1, they delve into the fundamentals of database sharding, including what it is and how it works, specifically looking at Oracle Database Sharding and its benefits. They also explore the architecture of a sharded database, examining components such as shards, shard catalogs, and shard directors.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/137192/207062   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   --------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Nikita: Hello and welcome to the Oracle University Podcast. I'm Nikita Abraham, Principal Technical Editor with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi there! The last two weeks of the podcast have been dedicated to all things database security. We discussed why it's so important and looked at all the new features related to database security that have been released in Oracle Database 23ai, previously known as 23c.  00:55 Nikita: Today's episode is also going to be the first of two parts, and we're going to explore database sharding with Ron Soltani. Ron is a Senior Principal Database & Security Instructor with Oracle University. We'll ask Ron about what database sharding is and then talk specifically about Oracle Database Sharding. We'll look at the benefits of it and also discuss the architecture.  Lois: All this will help us to prepare for next week's episode when we dive into each 23ai new feature related to Oracle Database Sharding. So, let's get to it. Hi Ron! What's database sharding?  01:32 Ron: This is basically an architecture to allow you to divide data for better computing and scaling across multiple environments instead of having a single system performing the work. So this allows you to do hyperscale computing and other different technologies that are included that will allow you to distribute your queries and all other requests across these multiple components to be able to get a very fast response.  Now many times with this distributed segment across each kind of database that is called a shard allow you to have some geographical location component while you are not really sharing any of the servers or the components. So it allows you separation and data management for each of the shards separately. However, when it comes to the application, the sharded database is totally invisible. So as far as the application is concerned, they connect to a global service, submit their statements. Everything else is managed then by the sharded database underneath.  With sharded tables, basically it gets distributed across each shard. Normally, this is done through horizontal partitioning. And then the data depending on the partitioning scheme will be distributed across like server A, server B, server C, which are independent servers that are running independent databases.  03:18 Nikita: And what about Oracle Database Sharding specifically? Ron: The Oracle Database Sharding allows you to automate how the data is distributed, replicated, and maintain the kind of a directory that defines the complete sharding scheme, while everything is distributed across many servers with no sharing whether the hardware or software. It allows you to have a very good scaling to be able to scale based on this partitioning across all of these independent servers.  And based on the subset and the discrete data configuration, you can go ahead and distribute this data across these components where each shard is an independent data location or data component, a subset of data that can be used, whether individually on its own or globally across all of the shards together. And as we said to the application, the Oracle Database Sharding also looks as a single component.  04:35 Lois: Ron, what are some of the benefits of Oracle Database Sharding? Ron: With Oracle Database, you basically have linear scaling capability across as many shards as you like. And all of the different database configurations are supported with this. So you can have rack databases across the shards, Oracle Data Guard, GoldenGate. So all of the different components are still used to give you all of the high availability and every other kind of functionality that we generally used to having a single database with.  It provides you with fault toleration. So each component could be down. It could have its own replicated data. It doesn't affect other location and availability of the data in those other locations.  And finally, depending on data sovereignty and configuration, you could actually distribute data geographically across the different locations based on requirements and also data access to provide a higher speed for local data management.  05:46 Lois: I'd like to understand more about the architecture of Oracle Database Sharding. Ron, can you first give us a broad overview of how Oracle Database Sharding is structured? Ron: When it comes to dealing with Oracle Database architecture, the components include, first, your shards. The shards-- each one is an independent Oracle Database depending on the partitioning you decide on a partition key and then how the actual data is divided across those shards.  06:18 Nikita: So, these shards are like separate pieces of the database puzzle…Ok. What's next in the architecture? Ron: Then you have shard catalog. Shard catalog is a catalog of your sharding configuration, is aware of all of the components in the shard, and any kind of replicated object that master object exists in the shard catalog to be maintained from there.  And it also manages the global queries acting as a proxy. So queries can be distributed across multiple shards. The data from the shards returned back to the catalog to group together and then sent back to the client.  Now, this shard catalog is basically another version of an Oracle Database that is created independently of the shards that include the actual data, and its job is to maintain this catalog functionality.  07:19 Nikita: Got it. And what about the shard director? Ron: The shard director is like another form of a global service manager.  So it understands the sharding by being able to access the catalog, knows where everything exists. The client connection pool will hit the shard director. In general, communication and then whether it's being distributed to the shard catalog to be able to proxy it, or, if the key is available, then the director can send the query directly to the shard based on the key where the data exists. So the shard can then respond to the client directly. So all of the connection pool and the components for global administration, generally managed by the shard director.  08:11 Nikita: Can we dive into each of these components in a little more detail? Let's go backwards and start with the shard director. Ron: The shard director, as we said, this is like a global service manager. It acts as a regional listener where all of the connection requests will be coming to the shard director and then distributed from that depending on the type of connection that is being used.  Now the director understands the topology--maintains the complete understanding of the mapping of the data against the shards. And based on the shard key, if the request are specified on the specific key, it can then route the connection request directly to the shard that is appropriate where the data resides for the direct response.  09:03 Lois: And what can you tell us about the shard catalog? Ron: The shard catalog, this is another Oracle Database that is created for special purpose of holding the topology of the sharded database. And have all of the centralized information metadata about your sharded database. It also act as a proxy.  So, if a client request comes in without providing a shard key, then the request would go to the catalog. It can be distributed to all of the shards. So the shards that you actually have the data can respond, but the data can then be combined and sent back to the client. So, it also creates the master copy of all the duplicate tables that are created in the shard database.  09:56 Lois: Ok. I've got it. Now, let's talk more about the shards themselves. Ron: Each shard is basically a database. And data is horizontally partitioned to be placed on each of these shards. So, this physical database is called the shard. And depending on the topology of your sharding, there could be user sharding, for example, where multiple keys are in a single shard or could be a system sharding that based on the hash value data is distributed whether singly or multiple data components across each shard.  Now, this is completely transparent to the application. So, as far as application is concerned, this is a single database and the response everything that they do is generally just operating as a single database interaction.  However, when it comes to the administrators, each shard is a separate database. Each shard can be managed independently and can have its own standby and other components that is then set up for high availability and management of the data operations.  11:21 Do you have an idea for a new course or learning opportunity? We'd love to hear it! Visit the Oracle University Learning Community and share your thoughts with us on the Idea Incubator.  Your suggestion could find a place in future development projects! Visit mylearn.oracle.com to  get started.  11:41 Nikita: Welcome back! Let's move on to global services and the various sharding methods. Ron, can you explain what global services are and how they function in a sharded database? Ron: Global services is generally the service that is used for the application to be able to connect to the sharded database. This is provided and supported through the shard director. So clients are routed using this global service.  12:11 Lois: What are the different sharding methods that are available?  Ron: When it comes to sharding methods that were available, originally we started with the system sharding, which is a hash partition, basically data is distributed evenly across the shards. Then we needed to allow for the user-defined sharding because sometimes it's not about just distributing the data evenly, it's also about controlling where the data goes to be able to control individual query execution based on the keys. And even for data sovereignty and position of the data itself.  And then a composite sharding, which provides you kind of a combination of the user-defined sharding and the system hash sharding that gives you a little bit of a combination of the two to better distribute your data across the shard.  And finally, sub-partitioning all types of sub-partitionings are supported to provide a better structure of the data depending on the application schema design.  13:16 Nikita: Ron, how do clients typically connect to a sharded database? Ron: When it comes to the client connections, all the client connections are generally routed to the director and then managed from there. So there are multiple ways that clients can connect. One could be a direct connect. With a direct connect, they're providing the shard key in the request. Therefore, the director knowing the topology can route the client directly to the shard that has the data.  The proxy routing is done by the catalog. This is when generally a shard key is not provided or data is requested from many shards. So data will then request is then sent to the catalog. The catalog database will then distribute the query to the shards, collects the results, and then combine sending it back to the client acting as a proxy sitting in the middle.  And the middle tier routing, this is when you can expose the middle tier to the structure of your sharding. So when the middle tier send the request, the request identifies which shard the data is going to. So take advantage of that from the middle tier. So the data is then routed properly. But that requires exposing the structures and everything in the middle tier.  14:40 Lois: Let's dive a bit deeper into direct routing. What are the advantages of using this method? Ron: With the client request routing, as we talked about the direct routing, this allows the applications to get very quick data access when they know the key that is used for the distribution of the data. And that is used to access the data from the shard.  This provides you a direct connection to a shard from the shard director. And once the connection is established, then the queries can get data directly across the shard with the key that is supplied. So the RAC respond for that particular subset of data with the data request.  Now with the direct routing again, you get some advantages. The advantage is you have much better performance for capturing subset of the data because you don't have to wait for every shard to respond for a particular query.  If you want to distribute data geographically or based on the specific key, of course, all of that is perfectly supported. And kind of allows you to now distribute your query to actually the location where the data exists.  So for example, data that is in Canada can then be locally accessed in Canada through this direct access. And of course, when it comes to management of your client connection, load balancing of those connections. And of course, supporting all types of queries and application requests.  16:18 Nikita: And what about routing by proxy? Ron: The proxy routing is when queries do not supply the actual sharding key, where identifies which shard the data reside. Or the actual routing cannot be properly identified. Then the shard director will send the request to the catalog performing the work as the proxy.  So proxy will then send a request to all of the shards. If any shards can be eliminated, would be. But generally all of the shards that could have any portion of the data will then get the request. The requests are then sent back to the proxy. And then the proxy will then coordinate the data going back and forth between the client.  And the shard catalog basically hands this type of data access to the catalog to act as the proxy. And then the catalog is-- the shard director is no longer part of the connection management since everything is then handled by the shard catalog itself.  17:37 Lois: Can you explain middle tier routing, Ron? Ron: This generally allows you to use the middle tier to define which shard your data is being routed into. This is a type of routing that can be used where the data geographically have some sovereignty or the application is aware of the structure.  So the middle tier is exposed to the sharded database topology. So understand exactly what these components are based on the specific request on the shard key, then the middle tier can then route the application to the appropriate location for the connection.  And then the middle tier, and then the either one shard or the subset of shard will maintain those connections for the data access going back and forth since the topology is now being managed by the middle tier. Of course, all of the work that is done here still is known in the catalog, will be registered in the catalog. So catalog is fully aware of any operations that are going on, whether connection is done through middle tier or through direct routing.  18:54 Nikita: Ron, can you tell us how query execution and DDL operations work in a sharded database? Ron: When it comes to the query execution of the application, there are no changes, no requirement for identifying specifically how the data is distributed. All of that is maintained behind the scene based on your sharding topology.  For the DDL, most of your tables, most of the structures work exactly the same way as it did before. There are some general structures that are associated to the sharded database that we will originally create and set up with mapping. Once the mappings are configured, then the rest of the components are created just like a regular database.  19:43 Lois: Ok. What about the deployment process? Is it complicated to set up a sharded database? Ron: The deployment for the sharded database is fully automated using Terraform, Kubernetes, and scripts that are put together. Basically what you do is you provide some of your configuration information, structure of your topology through an input file, like a parameter file type of a thing.  And then you execute the scripts and then it will build everything else based on the structure that you have provided.  20:19 Nikita: What if someone wants to migrate from a non-sharded database to a sharded database? Is there support for that? Ron: If you are going to migrate from a regular database to a sharded database, there are two components that are fully shard aware.  First, you have the Shard Advisor. This can look at your current structure, the schema, how the data is distributed. And the workload and how the data is used to give you recommendation in what type of sharding would work best based on the workload.  And then Data Pump is fully aware of the sharding component. Normally, we use Data Pump and load into each of the databases individually on its own. So instead of one job having to read all the data and move data across many shards, data can be loaded individually across each shard using Data Pump for much faster operations.  21:18 Lois: Ron, thank you for joining us today. Now that we've had a good understanding of Oracle Database Sharding, we'll talk about the new 23ai features related to this topic next week. Nikita: And if you want to learn more about what we discussed today, visit mylearn.oracle.com and search for the Oracle Database 23ai New Features for Administrators course. Until next week, this is Nikita Abraham… Lois: And Lois Houston signing off! 21:45 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

In this episode, hosts Lois Houston and Nikita Abraham continue their exploration of Oracle Database 23ai's database security capabilities. They are joined once again by Ron Soltani, a Senior Principal Database & Security Instructor, who delves into the intricacies of the new hybrid read-only mode for pluggable databases, the flexibility of read-only users and sessions, and the newly introduced developer role. They also discuss simplified schema-level privileges and the integration of Azure Active Directory with Oracle Database.   Oracle MyLearn: https://mylearn.oracle.com/ou/course/oracle-database-23ai-new-features-for-administrators/137192/207062   Oracle University Learning Community: https://education.oracle.com/ou-community   LinkedIn: https://www.linkedin.com/showcase/oracle-university/   X: https://twitter.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode.   ---------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative  podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Lois: Hello and welcome to the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me today is Nikita Abraham, Principal Technical Editor. Nikita: Hi everyone! In our last episode, we discussed database security, why it is so important, and all its different components. Today, we're going to be continuing that conversation by looking at all the new features related to database security that have been released in Oracle Database 23ai, previously known as 23c.  00:59 Lois: And we're so happy to have Ron Soltani back as our guide. Ron is a Senior Principal Database & Security Instructor with Oracle University. Hi Ron! Thanks for joining us again! We have a list of the new features related to database security and we'd like to ask you about them one by one, starting with the new mode for pluggable databases. What's that about?  01:21 Ron: With the hybrid read-only mode for pluggable database, the database could be in the read/write mode or read-only mode, depending on the user that is actually connected. So one of the things we have to realize is the regular read-only mode has one major issue. The major issue is everything, including data dictionary, including SysAux and all of the other elements are also locked up read-only.  So we cannot do any database maintenance. We cannot collect statistics to monitor anything. So you pretty much have to hard tune everything for the load you want and maintain everything. And this happens in many warehouse environments, in environments where the data itself is generally loaded. And then just heavily read. So it requires to be in a read-only mode to protect it.  So with a hybrid read-only mode, if you are a local user in the PDB, even a PDB administrator-- so I can create a local user in the PDB as a PDB administrator. And grant that PDB administrator even sysdba privilege. But once the PDB is open hybrid read-only mode, even for that user, the PDB is read-only. However, if a common user connect, who is, as you know, is a CDB user. Generally, CDB-level privileges granted and considered CDB administrators.  If they connect to the PDB, then the PDB is actually in read/write mode. So now, they can take snapshots. They can use all of the database tools to monitor how things are going. They can perform maintenance. So this allows us to be able to perform patching, maintenance, and other database-related operation. 03:17 Nikita: So you don't have to flip back and forth between read-only, read/write, read-only, read/write…  Ron: Because you know if we have database read/write to go to read-only, generally, we would have to shut down the database, then go to read-only. Then from read-only, we can go to read/write. But then going back to read-only, we have to shut down again.  Lois: Which was the issue with the normal read-only on the pluggable database, right? I'm glad that's been made easier. Ok… Moving on to the next new feature, which is read-only users and sessions. What can you tell us about this one, Ron? 03:51 Ron: As we previously discussed, you can put the PDB in the hybrid read-only mode. But then now the PDB is read-only for all local application users. However, let's say we have an environment where you have multiple application users.  One needs to be able to perform maintenance and perform updates where other sessions who are just reading the data to protect against all security element, and then better performance and operation management. We are going to set up read-only.  So setting up read-only at the pluggable database, that can be very high level depending on the application need. So with the read-only users and session, this will give you capability of setting read-only either for a particular user.  So when the user connects, all the user can do is read-only process. We do a lot of testing, for example. And we have users that may have read/write privilege in the test environment, then we want to go ahead and perform other operation.  So we would have to take privileges away, set the read-only, then go back and change again to read/write. So performing all of those different type of tests and even with the development has always been an issue.  So having granular capability of managing at a user or a session level can give us a major benefit of better granularly managing all application needs without sacrificing either security or having extra components that would have to be done by administrators.  05:33 Nikita: Yeah, this gives you a lot of flexibility and you don't have to keep temporarily changing privileges or configuring specific types of sessions. It's also an easy way to control user behavior, right? Ron: An application, as we said, have the schema owner that today we want to have a schema-only user for the schema owner. That is usually nobody connect us.  But then we have multiple schema users that one may be used for performing updates, one is used for administration, and one can be used for read-only. So this can give me a mechanism to manage that, or if a particular operation needs to run and for security purposes, that particular session needs to be set to read-only. So that gives us major control over it.  And in the cloud environment, this can be a very, very good component for better managing all of the security levels, where you can enable very fine-grained control while supporting all functionality of the application.  06:39 Lois: Ok. So, can you tell us about this new developer role in the database?  Ron: If we think about application administration, usually we create a schema owner. And we start by giving that the schema owner privileges-- grant them a resource role. By having resource role, they can create simple objects. But when you design an application, you need to implement it, test that, and then deploy it.  Today, there are many, many complex objects that can be used at the application level to manage the application. So today, we grant the resource role to the schema owner. Then we wait until they complain. They don't have privilege for certain object they want to create. Then we're going to have to grant them privileges as needed, and that used to be the way the security had worked.  But today since we have a schema only account where we can only enable the account when we want to do any type of schema work, and then it's locked up so the schema is protected, giving the schema owner the application role, the DB application role, now that has all the privileges in it, should not cause any security issue when managed properly, and will provide them with all of the privileges that they need to perform their work, including there are many complex schema structure like analytical views, hierarchies, dimensions, data-specific types that you can create.  And many of these type of privileges are not just assigned through a regular privilege assignment. Some of them are assigned through procedures.  08:21 Lois: And could you give us some examples of how this feature could be used? Ron: So there are many different ways of granting all of these granule privileges. So at the time that we go ahead and perform development of the schema and all of that depending on what's available, we don't know really what privileges do we need.  And as we said, there are many packages that we may be able to use to create complex objects that then gradually have to go ahead and get privileges on executing those packages and to be able to use them. And as we said at the time we actually performed the application, many of these objects, we may not even know we're going to use them until later on becomes evident or it may be a better structure to represent what we want.  So having to add and continuously deal with these type of changes can become extremely kind of cumbersome and tedious. It also delays all of the operations, especially now that the application schema owner can be secured. So we can grant this developer role to the schema owner, give the schema owner all privileges that is needed very quickly that they can now manage their schemas and manage all complex objects for that schema operation.  So the role is called db developer role. And just like any other role, you would connect as an administrator, grant db developer role to the schema owner. Now, we don't need to grant the resource role and all other things, because everything here is included in the db developer role.  10:01 The Oracle University Learning Community is an excellent place to collaborate and learn with Oracle experts and fellow learners. Grow your skills, inspire innovation, and celebrate your  successes. All your activities, from liking a post to answering questions and sharing with others, will help you earn a valuable reputation, badges, and ranks to be recognized in the community. Visit www.mylearn.oracle.com to get started. 10:28 Nikita: Welcome back! Ron, how have schema-level privileges been simplified in 23ai?  Ron: To be able to understand this, first we can review the privilege assignment in Oracle Database. First, you can be granted a privilege at an object level, so you can perform certain work on a particular object.  However, let's say I have a user account that I'm going to use an app user who's going to have to read from multiple objects within a particular schema. Now this granting at the object level is too low because I have to go at each object and assign the privileges needed on that particular object to the user.  Or we had our system privilege, for example, grant create any table to a user. The problem with that is now you can create any table within the schema that I want you to work with. But that privilege goes across all the schemas in the database, of course, not the database schemas itself-- those are protected, but across all user schemas.  11:34 Lois: Right. So, you're getting that privilege on other schemas that you may not really need that privilege for... Ron: So now the gap is kind of met with creating a schema-level privilege that allows you to grant the same any privilege but on all objects of a particular schema and not granted across all the schemas.  So this now allows us to much better be able to manage schemas, have schema user accounts with different level privileges on all the objects that they need to perform the type of work that they need to, without having to granularly assign each one of those privileges as we used to create many different roles with different privileges needed, then try to control the users by granting them those roles. Here, these are much better simplified by going through the schema-level privilege.  12:34 Nikita: Ron, I want to ask you about the new feature on creating audit policies at the column level. Ron: So if you remember, in the past, we talked about we can create audit policies with the old system where you would identify what to audit. But then you had to manage a whole bunch of parameters and security. And protecting audit even from the administrator were major issues.  In 12, Oracle identified or added the unified audit, which gives you protection on the audit schema. Even administrators cannot access it. You manage it through privileges that are assigned specifically to users who are going to manage the audit.  And it also allow you to audit Oracle operations, tools like Data Pump, like RMAN. So you can create a really secure audit environment monitoring everything in the database using unified audit and then maintain and manage those audits.  One of the important aspect of auditing is generating the minimal amount of audits. So this way, audits can be reviewed because if you generate too much audit, it is very hard to automate either using an automated system to review the audits or having users to review those audits.  Furthermore, if we wanted to then audit specific columns and different operation like SELECT, DML, we would have had to use the row-level security and build additional policies to be able to then individually monitor those columns, which not very simple to use and manage. And then the audits are put in different tables. Having to maintain all of those, relate them has always caused major issue overall.  So the benefit of having now this column-level audit added to the normal unified audit policies is that you can go ahead and build now your audits instead of at the table level, only for a particular column. This is going to reduce the false positive results that are generated because if I'm going to put update on a table, not updating any column can generate an audit.  But if I put update on the column salary, then only if the salary is updated, the audit is generated. So that can give me just the audits that are needed without the additional false positive audits that are generally generated.  15:08 Lois: Ron, can you talk to us about the management of authorization for Unified Audit administration, especially when using Database Vault? Ron: So first as we know for the Unified Audit, you have audit admin privilege and audit viewer privilege.  If you want to be able to create and administer and manage all of the audit information, including the audit purging and time periods and all of that, you have to have audit admin privilege. If you want to be able to read and generate the reports or things like that from the audits that have been created, you have to have audit viewer privilege.  Now we also have Oracle Database Vault. Database Vault kind of uses a row level security, but not on the end user data. It applies this row level security and administration on Oracle data dictionary. And allows you to control when particular object can be used, at what level can they be used?  And give you complete control over how the actual database and the objects are used and become available to other users in the database, including other administrators, even schema owners. So when the Database Vault is then applied and enabled, in the past, we could have managed the Unified Audit, which was kind of very funky to put one of the major security functions outside the main security Administration utility of the database.  So now, the Unified Audit has been incorporated into the Database Vault. So you can now use Database Vault to go ahead and set up the privileges and configuration for the authorizations required for managing Unified Audit. This also controls all the high-level users, including SYS, SYSTEM, and anyone who may have DBA roles or other high-level privileges.  So this allows us to now enable the Database Vault, and then manage the authorizations for the Unified Audit through Database Vault. Therefore, all authorization administration is unified under the same security tool, which is Database Vault.  17:28 Nikita: The final new feature to discuss is the integration of Microsoft Azure Active Directory with the Oracle database environment. What can you tell us about it, Ron? Ron: This has been requested by many of the clients who use other platforms and active directories and then need to access either the Oracle OCI, Oracle Cloud where the databases are running or having Oracle databases even in a local environment.  So wanted to be able to now allow this to happen. So if you remember, originally we had capability of mapping users from the database into Oracle Active Directory. So this way the user's role privileges can be centrally managed and the user does not inherit any privileges in the database. So if the user directly connect to database, has no privileges. Connect properly through Active Directory, everything enabled.  Then in Database 18, they created the commonly managed users, the CMUs. Where we could now map a third party Active Directory and then be able to use that into connecting to Oracle database for authentication and user administration. However, many of our clients use Microsoft Azure Active Directory. And they wanted to be able to integrate that particular Active Directory into Oracle environment, especially in the Oracle OCI Database as a service environment.  So to be able to do that, Oracle has multiple components that they have built to allow this to be able to now be configured and used. So the client can use these Active Directory for their user administration centrally.  19:20 Lois: With that, I think we've covered all the new features related to database security in 23ai. Thanks so much for taking us through all of them and giving us some context. Nikita: Yeah, it's really been so helpful. To learn more about these new features and watch some demonstrations on them, visit mylearn.oracle.com and search for the Oracle Database 23ai New Features for Administrators course. Join us next week for a discussion on some more Oracle Database 23ai new features. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 19:54 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.