Website Stuff

Things involving the website or any of the boot2docker VM's are handled by David Sibley, who is the staff web developer.  He should be looped into anything major happening like major vmware upgrades, storage migrations, etc...  He is also primarily responsible for CYCLONE, BASALT and probably a couple others.  David will reach out to us if he needs things like DHCP reservations, VM's created, etc...

Quarterly VMware/misc Patching Outline

This should be done on a quarterly basis, or if a critical patch is released.  This takes about 3 hours of mostly babysitting and waiting.  Here is a rough outline for the process:

1. Talk to Pete about scheduling a time for an outage.  Pete will send communication out to the WGNHS staff list.  I typically start this process around 8PM.
2. Manually update the handful of windows servers that require manual updates (m-s-storage02, WGSS-GISDATA)
3. When ready to begin vmware power down all VMs except vCenter.  Some servers might take a LONG time to shut down so be patient.
4. Upgrade the hypervisors that do not house the vCenter instance.  You can do this in vCenter.
5. Once the first 2 hosts are updated, log directly into the ESXI interface of the final machine and power down vCenter.  Proceed to patch the machine via CLI.
6. Once final host is updated power up vCenter.  Turn everything back on and make sure everything is happy.
7. Update vmware tools where needed
8. Check for vCenter updates.  Don't upgrade to a new major version unless you really mean it.

Windows Server Updates

The majority of the windows servers are setup to automatically update and reboot outside of business hours.  Fileservers M-S-STORAGE02, CUMULUS will NOT reboot automatically.  These machines should be manually checked for updates during quarterly maintenance or if critical security updates are needed.  Coordinate file server reboots with Pete so he can communicate the outage to staff.

ProxMox / Proxmox Backup Server

1. Update PEDIMENT.wgnhs.wisc.edu with apt
2. Update PBS.wgnhs.wisc.edu with apt
3. Verify backups are still completing and datastore usage is OK

Misc updates and tasks

1. M-S-UNIFI
   
   1. Server is set to automatically install security patches.  Occasionally its a good idea to ssh in to reboot and run a manual upgrade.
   2. Upgrade to new version using upgrade script from Glen R if available.
   3. Unifi device updates
      1. Mount horeb access point firmware updates happen automatically on the 1st of the month.
      2. Switches in the mineral point server rack do not update automatically.  These should be updated during the quarterly maintenance outage if necessary. 
2. Synology Active Backup System
   
   1. Log into DSM on GREENLAND and ensure Active Backup For Business is operating as expected for VM's and windows clients.  Make sure storage utilization is in a safe place and adjust retention if needed. 
   2. Ensure replication to ICELAND is taking place without error.
   3. Occasionally update system/packages on both synology devices
3. M-S-LICENSING01
   1. Licenses will need to be updated periodically.  Pete will generally provide a new license file, and depending on the software an update to the licensing service may also be required.  This should generally be worked on off-hours to avoid disruption to WGNHS staff.  ALWAYS SNAPSHOT THE VM BEFORE TOUCHING ANYTHING WITH LICENSES. TRUST ME.
   2. Verify licenses come back online after reboot
4. WGS-PETREL
   1. Same deal as M-S-LICENESING01 - update license file if needed (renews are in april I think) and verify licensing services works after reboot. 
   2. Petrel software is installed on WGS-CUTBANK for testing purposes.
5. Misc Network
   1. Periodically go through infoblox to remove old DHCP or DNS entries. 
   2. Remove firewall objects and rules if no longer needed.
6. PFsense (Site-To-Site wireguard VPN)
   
   1. You can update both the Mineral Point (VM at main office) and the Mount Horeb PFsense (Netgate3100) machines via the web interface from WGS-CUTBANK.  Can probably be done once or twice a year unless some horrible vulnerability is discovered.  
      1. MP is Pfsense CE and Mt Horeb is Pfsense+, so release schedule is not the same.
      2. Make sure to use the package manager page to update any packages, and apply any new system patches from the "System_Patches" package.
   2. See NETWORK for more specifics of this setup
7. UBUNTU_NUT
   1. Server is set to automatically install security patches.  Occasionally its a good idea to ssh in to reboot and run a manual upgrade.