I can’t believe I haven’t written about this yet. Unattended upgrades are a great way to keep your servers up to date, but there are a few things that didn’t work out of the box, so here is a summary of how my patch process is set up.
Why unattended-upgrades?
To be honest, running upgrades unattended can cause bad feelings with your colleagues if stuff breaks because of it. And it most likely will if you’re not doing it right. Unattended-upgrades is a feature available in Ubuntu, Debian and most likely other Linux derivatives, which allows you to control which updates should be installed and when you want to get notified about it.
From a security standpoint, unattended-upgrades is a no-brainer, you want to have the latest patches installed but you don’t want and can’t do it manually, unless you have near unlimited man power or really nothing better to do, which is pretty much never the case.
From the classic admin “keep things running” approach, doing any changes whatsoever is not really something you’d want, much less so doing them unattended, meaning “let the system do as it wants”.
This often leads to a discussion between security folks and administrators about whether or not this could actually be working. The problem, as so often in security, is that change is needed but for change not to end in disaster you need to know what you’re doing and more importantly you need to know how your network and your servers behave.
If you were to enable automatic updates on a Linux system without any knowledge of what that system is doing, it’s probably not going to end well.
So what then?
In order to get unattended-upgrades running without messing up your stuff, you need to
- understand what the host is doing
- implement the least needed upgrade process (usually security patches only)
- make test runs before you let it lose
- have backups ready, something which you should have always anyways!
There are two main problems that I have experienced so far with unattended-upgrades and that you should be aware of.
3rd party applications need specific package versions
The first one is easy to fix but a PITA to find out. Usually you find out by crashing the service, try to figure out what caused the problem and find out that package x has to be version y but the most recent version in your distros repository is newer than that.
Since you most likely can’t fix the applications dependencies, there is only one way – set the package on hold. Distros using the apt package manager allow you to set the hold flag for packages, which means that they won’t be upgraded along with the others. Be aware that this should only be done if no other way is possible as it can have side effects such as
- other applications can’t be updated because they require x to be updates with them
- which sometimes leads to apt dependency f-ups
- It’s possible that security patches won’t be installed either
Service restarts mess things up
This was really my own fault, though it took some time to fix. Some services, such as MySQL for instance, can be tweaked during runtime. If an upgrade process restarts the service after updating the binaries, these runtime tweaks are lost. This can be a problem if
- you don’t know about the restart
- you didn’t document your tweaks properly.
Another thing that can happen – it always can, no matter what you do, is that a service doesn’t restart properly. One reason, although that didn’t happen to me yet, is that a service won’t restart because the config file still uses a deprecated version that was finally kicked out for good with this upgrade. If that happens you just were to lazy to switch to the new one during the transition time where both, the new and the old feature, where still supported. One good way of avoiding this is by actually using your log files.
But monitoring, analyzing and getting the goods out of logs is a post for another day.
Needless to say that backups, documentation and the ability to restart your services without messing things up is something you should have already and is not a requirement that only comes with unattended-upgrades.
Installing unattended-upgrades
Installing the packages is as easy as it gets. Just run the following commands and you’re good to go.
sudo apt-get install update-notifier-common unattended-upgrades
The update-notifier-common package is an optional addition, that will create the file /var/run/reboot-required, which tells you that the system requires a reboot to apply the updated or newly installed packages, and the file /var/run/reboot-required.pkgs which tells you which packages require the reboot.
You can even add checks to your monitoring system or your message of the day (motd) to get notified about uninstalled packages or required reboots.
69 packages can be updated.
0 updates are security updates.
You can get these with the following update-motd config files
$ cat /etc/update-motd.d/90-updates-available
#!/bin/sh
if [ -x /usr/lib/update-notifier/update-motd-updates-available ]; then
exec /usr/lib/update-notifier/update-motd-updates-available
fi
$ cat /etc/update-motd.d/91-release-upgrade
#!/bin/sh
# if the current release is under development there won't be a new one
if [ "$(lsb_release -sd | cut -d' ' -f4)" = "(development" ]; then
exit 0
fi
if [ -x /usr/lib/ubuntu-release-upgrader/release-upgrade-motd ]; then
exec /usr/lib/ubuntu-release-upgrader/release-upgrade-motd
fi
$ cat /etc/update-motd.d/98-reboot-required
#!/bin/sh
if [ -x /usr/lib/update-notifier/update-motd-reboot-required ]; then
exec /usr/lib/update-notifier/update-motd-reboot-required
They should be included in Ubuntu by default and updated on login by the pam_motd module. If you change or add config files, you can either logout and login for the changes to take affect, or install the update-motd package and run it.
Configuring unattended-upgrades
To give you a quick overview, this is what we want the system to do stay updated.
- Fetch the newest package information (apt-get update)
- Install security updates only
- Notify us via email, at first always, later only on error
- Do not reboot automatically
- Do not overwrite config files
Let’s take a look at how to set these things up. Unattended-upgrades has three config files that are of interest to us.
/etc/apt/apt.conf.d/20auto-upgrades
This config file is pretty simple and straight forward.
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
With this we enable point one and, partly, two of our bullet list. But we haven’t configured point two yet. We enabled the installation of upgrades but haven’t defined which upgrades we would like to have.
/etc/apt/apt.conf.d/50unattended-upgrades
Here is where all the magic happens. The following shows only the options that I am using, the default config offers more that that though, so you might want to take a look at it.
// Automatically upgrade security packages from these (origin:archive) pairs
// Additional options are "-updates", "-proposed" and "-backports"
Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::MinimalSteps "true";
// Send report email to this address, 'mailx' must be installed.
Unattended-Upgrade::Mail "spam@hashtagsecurity.com";
// Set this value to "true" to get emails only on errors.
Unattended-Upgrade::MailOnlyOnError "true";
// Do automatic removal of new unused dependencies after the upgrade (equivalent to apt-get autoremove)
Unattended-Upgrade::Remove-Unused-Dependencies "true";
// Automatically reboot *WITHOUT CONFIRMATION* if a the file /var/run/reboot-required is found after the upgrade
// Unattended-Upgrade::Automatic-Reboot "true";
If you are really brave, you can enable the last option as well. It worked quite well for me on some servers but you have to make sure that all service are started properly after a reboot. I’ve read somewhere that unattended-upgrades can be timed to avoid redundant servers rebooting at the same time, but I haven’t looked into it so far.
For more information on the configuration, check out the official Ubuntu documentation here and here.
CRON
If you take a look at the file /etc/cron.daily/apt, you will see that we unattended-upgrades is already configured to run regularly.
1 #!/bin/sh
2 #set -e
3 #
4 # This file understands the following apt configuration variables:
5 # Values here are the default.
6 # Create /etc/apt/apt.conf.d/02periodic file to set your preference.
We created, or modified the files in /etc/apt/apt.conf.d/ and thus configured the /etc/cron/apt process to suit our needs, so there is no need to add a new cron job for it.
According to the Debian documentation, the 02periodic file is an alternative config file for the 20auto-upgrades, so we don’t need it.
Timing
The only problem I see with this, is that redundant servers might run updates or possibly even reboot themselves at the same time. The way it’s setup now is that the apt cron job is executed once daily.
Cron daily runs all scripts in /etc/cron.daily/ once a day, the start time for this is defined in /etc/crontab.
# cat /etc/crontab
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user command
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
As we can see, cron daily starts at 06:25 AM every day. In my case, apt is the first command to be executed as can be seen by the alphabetical order of scripts in /etc/cron.daily/.
# ls -1 /etc/cron.daily/
apt
bsdmainutils
creds
dpkg
logrotate
man-db
quota.dpkg-dist
sysklogd
This might be different on your system, depending on the cron jobs you have installed but it should be among the first to run. If not and you want to be sure, just rename it to 01_apt.
The reason why I care about the order of execution is because command running before apt could delay it’s execution. We can easily change the start of cron daily for redundant systems, but if the first system would have a huge delay the might still end up running at the same time. The chance is slim, but why take chances if you can be sure.
Here is an example for two redundant web servers.
web01: # grep daily /etc/crontab
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
web02: # grep daily /etc/crontab
25 8 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
The time difference of two hours should be more then enough, keep in mind that two much time difference might result in other problems. Such as two servers being out of sync because the dependencies changed on one system but hasn’t been updated on the other.
Logs
If you want to check on your recently applied updates, take a look at the following log files.
- /var/log/unattended-upgrades/unattended-upgrades.log
- /var/log/unattended-upgrades/*
- /var/log/dpkg.log
Config files change!
Here is one more add-on that I stumbled across recently. If you have done a few upgrades with apt-get, you should have seen this prompt at least once.
While this is a great thing if you’re doing the upgrade manually, it’s kind of a problem when you install them automatically.
The image above is actually a screenshot from a mail my server sent me, after the upgrade process stopped because of this dialog. If this happens, the updates are not installed completely, and you will receive this mail daily until you fix it!
So let’s fix it. Open the config file /etc/apt/apt.conf.d/local and add the following lines.
# keep old configs on upgrade, move new versions to <file>.dpkg-dist
# e.g. /etc/vim/vimrc and /etc/vim/vimrc.dpkg-dist
Dpkg::Options {
"--force-confdef";
"--force-confold";
}
This will tell unattended-upgrades, to keep the original config files and move the new versions to .dpkg-dist, so you can inspect them at a later point. What I was missing though, is a notification by mail that new .dpkg-dist files have been created.
To get this information, I whipped up this small script. If you know a better way to solve this, please let me know. In the meantime, this will get the job done.
#!/bin/bash
#
# This script will search for .dist-dpkg files and notify you if any are found
#
REPORT_MAIL="upgrades@yourmail.com"
find / -name *.dpkg-dist > /var/log/unattended-upgrades/unattended-upgrades-config-diff.log
confcount=$(wc -l /var/log/unattended-upgrades/unattended-upgrades-config-diff.log |awk {'print $1'})
if [ "$confcount" -ne "0" ];
then
echo -e "Subject: New Held-Back Config File Changes\nFor the following configs, changes have been held back during unattended-upgrade. \nPlease review them manually and delete the dpkg-dist file after you're done.\n\n $(cat /var/log/unattended-upgrades/unattended-upgrades-config-diff.log)\n\nRegards, \n$(hostname -f)" | sendmail $REPORT_MAIL
fi
Just save this script somewhere on your server, make it executable and add it to your daily cron jobs. Make sure to change the email address!
Also, make sure the script has write permissions to the /var/log/unattended-upgrades/ folder, or otherwise it will fail.
Summary
Unattended upgrades are not something you “just enable”. They have to be introduced into your environment carefully but it’s time well spent as they can be quite helpful later on.
Not only is security increased but you safe a lot of time when you finally have no other choice then moving on the the next distro release. Believe me, few things are more painful then having to perform full dist upgrades (e.g. 12.04 => 14.04) on way outdated production servers.
Being more secure not only means that the risk of loosing money is smaller, it also means for admins that the risk of running around in panic trying to figure out how it happened and what can be done to stop it, is smaller. That’s something you should keep in mind if you’re an admin, or that you should keep handy as an argument if your working with admins.