Mastering Database DevOps: Session 4 – Efficient Data Movement

In the last video, we talked about monitoring and the importance of monitoring in a DevOps pipeline. The reason monitoring is important, of course, is because change is happening faster. We need to keep track of that change and make sure that we're not introducing potential performance issues into a live environment. Because we've failed to capture that change and the effect of that change beforehand. Data movement challenges organizations, because the fact that the data is so critical. Organizations are very nervous, and DBA is particularly very nervous about any kind of data movement. Because there is the propensity there for data to become corrupt, to lose data, to lose the integrity of the data, the consistency of the data as your leaving from A to B. And moving it from A to B is important. Because otherwise, if you have all of your data in one single database, you don't have that data copied somewhere. Then you've got a single point of failure. So if that database goes down, that's it. That's your business is down, so it is important to do that. But I think companies are concerned about it because of the risk of something bad going wrong. So bad data movement would be where you don't have a reliable way of moving the data in a way that is consistent. What I mean by consistent is that if you bear in mind that transaction throughput in a production application, like a trading floor, thousands of thousands of transactions happening every second. So even if one of those is wrong, and you end up with corrupt data, then you've basically got a mistrust in the whole system. That's part of the problem as well is trust. How can I trust that whatever system I deploy is not going to cause any problem? Because I can't afford any problem whatsoever, because the downside is, I'm going to have an outage. And that could cost me thousands and thousands of pounds every minute that the system is down. So trust in whatever system you use is vitally important. Signs of bad data movement are where you've not got a reliable way to move data to another environment in a safe and effective way. What I mean by that is that you've got to think about re-consistency, moving data in a way which is immediate. So that whoever is using, remember that target database that you're moving that data to may be being used by somebody else. If you take, for example, an offline reporting database, which has been set up for the simple purpose that an analyst team who needs to generate reports at the end of the month. You'd want to have them put a lot of excessive load on your mainline database. So what you do is you create an offline reporting database to which those analysts would connect to and use the data there, thereby taking the load off the main system. The problem with that is that you have to have exactly the same data in that reporting database as your main line database. Beaing in mind, you've got thousands of thousands of transactions every second. That's a big challenge. There are lots of implications if you get this wrong. Data changes carry risk, and therefore, it's important that you've got data that is consistent when you move it. You've got to make sure you don't lose any data, so the methodology that you used to move that data from A to B has to be such that there is no chance of any data loss. You've got to have some sort of reconciliation process in there to make sure that you haven't lost any data. And also, you've got to make sure that when you move that data, it doesn't become corrupt. Because if any of those things happen, that's seriously bad. People will lose trust in it, and you're going to have to take the whole thing down. And you're going to have to restore that database from a backup, which is hugely disruptive and costly. In order to begin a replication or think about moving to a target database for data movement, for replication, or whatever it is, think about what the use cases are that you need to solve. Is it that you need to move data from A to B in order to have a reporting instance? Do you want to have that target database in the cloud? A lot of companies are actually taking the opportunity, like DevOps, as an opportunity to move some of their databases to the cloud. But they're thinking, well, if I move my database to cloud, how am I going to replicate move data from my data center out to a cloud service provider? How is that actually going to work? Once you think about what the use cases are that you need, the actual instantiation, if you will, of the replication process is very simple to do. SharePlex has its own monitoring capability. It monitors what's happening into the data. It lets you know whether or not the reconciliation process is working properly. It lets you know if something bad is happening. And if you have it set up properly, then it should just work fine. It'll replicate out to multiple environments simultaneously. And a lot of our customers actually take the opportunity to replicate to a cloud service provider and change to a different server operating system at the same time, or perhaps change to a different version of the database. You can actually do these things simultaneously with SharePlex. It's a very flexible tool, but ultimately, enables you to migrate data in a safe way. An example of a customer success story where they use SharePlex was actually in the case of a cloud migration, and this is a very popular use case these days as companies think about reducing their center costs by moving to a cloud service provider.