Backups and Updates

Backups

We don't have a "traditional" backup system (e.g. tape or dedicated disk backup system) in our department.  Backups (such as they are) are generally handled in the following manner.

• Workstations
  • Some of the Mac users use Time Machine to backup groups of endpoints to one location
  • Labs generally copy data files to a Research Drive or, in some cases, external or NAS hard drives.  We like to discourage external/NAS drives and encourage use of the Research Drive
    • Research Drive is provided and maintained by campus, and anyone with a research role can request one.
    • Researchers get 5 free terabytes of space
    • The data on Research Drive is itself backed up regularly
    • Start here for more information about Research Drive
    • Same goes for people who use Restricted Drive
  • For your average user workstation, we recommend keeping file copies in Box, Google Drive, or OneDrive in addition to (or in place of) the local hard drive.
    
    
• Web-Hosted Sites
  • This is easy - the web hosting team backs up your database and site files for you!  There is no need to install WordPress or other backup plugins/utilities on these sites.  Plesk has a backup utility, but you don't need it. It also has a tendency to not work right if the site has GBs of stuff to back up.
  • Web Hosting - Web Site Backup and Recovery (wisc.edu) has more information about the process, what they back up, how far back you can go, and how to restore data.
  • However, there are times when you may want to make your own backups.  For example, just before plugin or other major updates so that if you need to restore, you can restore the most current state of the data/files before the update.
    • The easiest way to accomplish this is to go into Plesk
    • For databases, go into each database in the site and click "Export Dump".  You can use the corresponding "Import Dump" to restore the data.
      • "Export Dump" will create a .zip file that contains the .sql file that contains your database schema and data.  It generally writes this to the site "Home Directory".  From there you can download to another storage medium.
      • Be aware the "Import Dump" requires a .sql file, not the .zip file.  So if you need to restore you will have to extract the .sql file first and then upload it somewhere into the site file structure.
      • Also be aware that "Import Dump" imports everything in the .sql file, OVERWRITING any existing data!  If your database restore needs to be a little more granular (e.g. you need to restore just one or two tables or perhaps specific records), you really need to modify the .sql file so that it contains only the table definitions or data you need to restore.  You should then be able to use "Import Dump" or another MySQL GUI tool to restore the data.
    • For files, you can use Plesk to add files/folders to an archive (it's a .zip archive).
      • Go into the folder containing the files/subfolders you want to back up and select the files/folders.
      • Click Archive -> Add to Archive.  Give the .zip file a name.
      • .zip file is created in the folder you are in when you create the archive. 
      • You can then download the .zip file to other storage, either via Plesk or your favorite SFTP tool.
      • To restore the files, simply upload the .zip file into the exact same folder used to create the file. 
      • Click Archive-> Extract Files
      • EVERYTHING in the .zip file is extracted into whatever folder you are currently in.  THEREFORE, be absolutely sure you are in the folder in which you want to extract these files! 
    • If you do any manual backups in this manner, please please please remember to DELETE any/all .zip files you no longer need.  They WILL count towards our storage costs.
      
      
• CCI/Virtual Machine Sites
  • CCI/VM gives you snapshot tools you can use to make backups of your VMs.
    • You can create a snapshot of the entire VM using the CCI portal/console.  You would want to do this before any updates/upgrades or any other time when you are doing things that could cause a major outage if you goof.
    • You can read all about VM snapshots at CCI Private Cloud - Managing VM Snapshots (wisc.edu) 
  • CCI also backs up the individual files in your VM!
    • Visit CCI Private Cloud - Virtual Machine (VM) Backup Overview (wisc.edu) for more information. 
    • If you click on the "Backup Service" link, be sure to select "SSO Signon" otherwise the backup service will not be able to find your VMs.
    • You can restore individual files using this portal.
• RTVM
  • Although this is a CCI/VM, I do my own backups of the database files for the same reason I back up web hosting databases - to have the most current version of data possible before an update.
  • Another reason is that sometimes restoring the MySQL files (the actual files as stored on disk, not the databases or sql files) containing your data can be a little tricky.  It's not impossible, but I always thought it was easier to run the mysqldump utility to create a backup file of the schema and data, and then import that if I need to restore.
  • To that end, I have a script file that creates a backup of critical files and databases and then copies the backup (via rsync) to the old snipe-it "server" sitting on my desk.
  • Given the tools available via CCI, I suppose that with better update/upgrade planning this is probably not necessary, but it was a very interesting exercise.  I suppose some day when I am comfortable I can disable the rsync backup and rely solely on the CCI tools.
    
    
• Learning Space
  This one has me worried.  As far as I know, nothing is backed up on this system and I have no login access to it whatsoever.  I do not know what we would do if there was ever a catastrophic failure resulting in file/data loss, but I suppose that's the risk we're taking.

Updates

• Workstations
  • Windows updates should happen per GPO settings (GPO "SOP-Windows Update Enhanced").  At the time of this writing it shouldn't force restarts, but that might change.
  • I also use TEM/BigFix to push updates of common applications (except for MS Office and Adobe stuff).  This generally happens on or shortly after the first of each month.  The details are a little too complex to document here, but basically I add updated components to the "Monthly Software Updates" baseline and then have it run the entire month.
    
    
• WordPress Sites
  • Most of our sites are configured to apply minor version updates automatically as they are released. 
  • I manually update plugins on all sites every Wednesday (or as needed for critical zero-day vulnerabilities).  This is usually harmless, but per above I do make a backup of the pharmacy.wisc.edu site before applying any updates.  That site is generally considered most important.  If a plugin update causes problems on any of the other sites, a request to restore from web hosting is usually sufficient.
  • It helps to keep some sort of record noting what what updated and on which site and date.  That way if something seems broken, you can go back and see what you updated.  Unless the reader knows of a better system, I manually kept most of these "logs" in a Word document, but feel free to use whatever works well for you.
    
    
• Linux Sites
  • These are pretty much all CCI/VM sites except for the occasional oddball one-off I might have around (the snipe-it box is one example)
  • I try to apply patches/updates on or shortly after the first of each month.
  • For the VMs, create a snapshot first in case you have to roll back!
  • Log into each site and run "apt update" (to update package managers), "apt upgrade" (to update individual packages).  In that order.
  • Sometimes for kernel updates you may also want to run "apt autoremove" to dispose of any old kernel packages.
  • Sometimes you will need to reboot.  You can do this immediately if you're feeling gutsy, or schedule a reboot (using the "shutdown" command) for early morning sometime.
  • Linux is usually pretty good about telling you if "apt autoremove" or a reboot is recommended.
  • It helps to keep some sort of record noting what what updated and on which site and date.  That way if something seems broken, you can go back and see what you updated.  Unless there is a better system, this is a manual process.  I kept most of these "logs" in a Word document, but feel free to use whatever works well for you.

• Learning Space
  Like the backup situation, I am not sure how or when CAE applies updates (if it ever does) to this system.  I have no login access to run updates or check on things.