How often do you test your backups ? Whether it’s weekly, monthly, quarterly, or an even longer duration than that (if that is the case, stop reading this and test your backups now), every business and personal user needs the assurance that they can recover either entire systems, or an individual file in the event that a need arises. Whilst you may be thinking “yeah, obviously”, you’d be surprised at how often this critical process is the victim of oversight on the part of the operator, or forgotten completely in the rush to have a system ready by a particular deadline. We’ve all missed things on server builds like antivirus, or the latest patches, but no backup processes ?
No system should ever be considered ready for release to production (or even test in some cases) if there is no backup strategy in place. This is nothing more than unprecedented risk to the business - particularly if the service is considered critical to any particular function. Can you imagine having to explain to senior management or stakeholders that the system everyone has been using for months has just crashed, and you’ve just suddenly remembered that there isn’t a single backup ? You may as well start updating your CV or resume now. I’ve never had to explain to senior management that there is no backup, and I have no desire to start doing so now.
A backup strategy isn’t difficult to define, but even with one in place, there is still a risk that this may not function as expected in the event of needing to recover a system or individual file. Backups should be regularly tested to ensure both consistency and integrity, with the restore process being validated afterwards - in short, you are attesting to the fact that you are in a position to recover either an entire server image, or a single file (dependent on the need). Failure to comply with this simple task could see you being thrown to the lions unnecessarily. Backups themselves are not difficult to undertake, and should be running on a daily schedule at minimum. The actual strategy varies heavily between use cases, with some taking differential or incremental backups during the business week, and then executing a full backup over a weekend. This type of backup is still very popular as it has the huge benefit of backing up only files and folders that have changed - the downside is that in the event of a disaster, you’d need the last full backup to be restored first before you can play back each incremental. Another more common approach is to use delta based backups that leverage bits technology to make the most of compression in order to reduce storage requirements and costs. Usage of delta backups typically requires a concept called “safesets”. It’s more commonplace in today’s technology sphere to use an off-site vaulting mechanism where you create an initial “seed” of a backup target, then ship only the changes to that same system on a daily basis.
This type of backup is also referred to as a snapshot. In the case of (for example) Amazon Web Services, the underlying technology is hypervisor based. This means that an initial image can be made of the machine in question, then a daily / hourly snapshot performed which captures the changes since the last snapshot was taken. In the event of a restore or disaster recovery, the initial image is recovered first, with the snapshots being “layered on top” to provide the necessary recovery point - this itself really depends on your backup strategy. One thing to bear in mind with off-site vaulting is that the costs can easily spiral out of control with penalty charges being applied for exceeding your quota limit. Storage itself is relatively cheap and easy to acquire, but the backup process required to support it can often attract significant and unprecedented cost to any user. For this reason, the backup strategy should be well defined, with best practices playing a major role in determining both the strategy and the overall concept. It’s important to note at this point that although there are numerous product offerings that include cheaper storage options, these quite often become the Achilles heel for those managing systems - the storage is cheap, so you get to save a fortune. However, the RTO (Recovery Time Objective) shoots through the roof - that cheaper storage is often magnetic - meaning it could reside on a tape somewhere. The restore times are not just limited to the speed of the media. Tapes typically require a catalog to be built first before backup software can be in the required position to recover the data in question. Years ago, I was performing backups and recovery testing on DLT tapes via a single loader and low grade SCSI card. A restore of an Exchange database could take well over 24 hours to recover based on this aged technology, and a slow network made this even worse.
Thankfully, we are not bound by slow hardware in today’s modern technology era. However, if your backups remain untested, how do you know that they will serve the purpose they are designed for ? The short answer is that you don’t. Each backup taken should be subject to analysis in terms of the log files generated during the job - in other words, the backup log should be checked on a daily basis to determine whether it was successful or not, and any issues noted be rectified as soon as possible. With ransomware wreaking havoc across the world, it also makes perfect sense to perform sanity checks on servers and their associated backups to ensure that you are not actually taking copies of files and folders that have in fact been encrypted.
Hit with ransomware ? No problem - we’ll restore from backup
……until you discover that your backups are also full of encrypted content.
Obviously not a great situation to find yourself in. For this reason, it’s important to invest in decent anti malware endpoint tools and file monitoring capabilities to prevent this from happening in the first place. It’s also very important to frequently test your ability to recover files from backup.
The bottom line - It’s not possible to say with 100% certainty that your backups are in full working order unless they have actually been tested.