Hi,

I have a few questions related to server maintenance. I have inherited a Debian server from the previous sysadmin that was never updated. It's still running Debian 7. It's a pretty common setup: the server runs SSH, Apache with a handful of websites, MySQL, PHP, Postfix, SpamAssassin and Dovecot. I need to update it soon, and need some advice from seasoned admins.

1) What is the best way to get Debian up to date? Obviously running apt-get update && apt-get upgrade would kick things off, but somehow it doesn't give me comfort feeling this would go without a hitch when going from Debian 7 to 9. If it does, I'd be very happy, but what are the chances this upgrade goes without errors? (I know some websites will break with PHP 5.4 going to PHP 7)

2) What's my strategy when the upgrade goes bang? I have a backup of everything, so would it be smart to just wipe the box and start from scratch? Should I restore the image from before the upgrade and do something differently? Are there different strategies to perform an update/upgrade like this or is it running apt-get and keeping your fingers crossed?

3) How do I plan future maintenance? Should I create a cronjob that updates/upgrades the OS automatically? What happens if an update breaks something? On the other hand, I can't go and do all the updates manually every day. What is the preferred way of keeping Debian up to date?

Thanks in advance for your insights!

Upgrading the same machine would be done in three steps: Update Debian 7, upgrade to Debian 8, and finally upgrade to Debian 9. I would not try to go directly from Debian 7 to Debian 9 in one step unless you are building a new machine and transfer over the data and configurations, in which case you are really building from scratch. You'd have more knowledge of what's on the machine and how it works if you build a new machine or wipe it and do a fresh installation. So that would be the way I would go. Either way, you have about a month before Debian 7 reaches end-of-life.

Regarding question #2, my approach would be to set up a second box and install the applications, restore configurations from backup, and finally import the data.

You can get a list of packages with dpkg --get-selections and save that to a file. Configurations should be in the /etc/ hierarchy. The location of data depends, unfortunately, on the specific applications in questoins.

Regarding, question #3, you can automate some of the updates. There are several ways, using the package unattended-upgrades is one method. However, it is best if the machine in question is able to send e-mail notifications to you regarding failures. A warning upon login via SSH in the case of update failures would be good too.

Edit: Be sure to read the release notes for 8 and 9.

1 members found this post helpful.

There is quite a few changes between 7 and 9 (and yes you have to go through 8 on the way).

Debian upgrades are usually pretty reliable if you have a clean system (do you have documentation of everything the previous admin did?), but a bit of work can be involved migrating existing setups to new versions (as stated, PHP5 to 7, as well as MySQL to mariaDB, I think dovecot 1 to 2 as well).

If this is a live system, and you are in any way unsure of the current state of things, my advice would be to setup and test a new 9 system, migrate data, test again, then swap users to the new system.

I've done a few live dist-upgrades on production machines, but only ones I was very familiar with and had planned and tested first.

I second the suggestion to use a "swap" upgrade server.

While it is possible to install multiple versions of PHP on the same server, there can be complications. The better andswer is to upgrade your PHP scripts to the newer standard.
Greater complications are likely from an in-place step upgrade of Debian two full versions. That plan is still a great first plan, but have a backup plan in case it explodes on you.

A second plan would be a clean install of the latest Debian, followed by installation of the latest versions of your required software applications, followed by restore of the configuration and data that they require. With luck you will not need the backup plan, but having it ready is only due diligence.

The important thing is that you have time. While an upgrade is needed, it is not an urgent failure issue. Take your time and avoid the errors thrown by rushing things.

Much of the same problem here: https://www.linuxquestions.org/quest...ch-4175625894/ Although I don't think that thread is about a server as well.

If this is a live server I would not do dist-upgrades for the life of it. Debian dist-upgrades from one stable to the next are possible and should work. But I have seen to many times it did not work completely. If you do this on a live system, you want 100% certainty. Not 98%.

Setting up a new server is relatively cheap. Even cheaper when it is a VPS.

You don't have Virtualmin running for a chance? If you have, there is no question at all. Install a new server, install new Virtualmin and transfer the domains.

jlinkels

Thank you all very much for your helpful advice. It's much appreciated!

An in-place upgrade already felt risky, and with your suggestions, I am leaning towards the "swap upgrade server", or SUP as I will call it now The current server is VPS based, so setting up a new VPS with Debian 9 for the SUP is easy enough. And I figure I can migrate the configuration of various applications like Apache, Postfix and Dovecot quite easily too. There are MySQL dumps that I can load into MariaDB.

But how do you test if your www/smtp/imap services work properly? I have access to the DNS records, so should I add an A-record for the SUP and CNAME-records for testing the services? I want to use LetsEncrypt for all certificates instead of using the (ridiculously expensive) wildcard certificate from GD. How do I prevent conflicts in (sub)domainnames?

What about making the final switch? Considering the above, do I pull the plug on the old server, change the DNS records to point to the new server and then hope for business as usual? What about the propagation of the DNS information? I think the VPS comes with a fixed IPv4 address so changing that would be impractical, if not impossible.

What is the common practice that seasoned admins do in this case?

Leave the IP addresses the same and just change the CNAME and MX entries. The services should be set up in DNS already to be attached to a name which can be moved. If not, that should be done early in the planning.

If I recall correctly you can do a lot of testing of migrated services on the "SUP" using its A name alone. It will give a close enough approximation to test what works. Then when ready, take the zoo of different CNAME and MX entries which point to the old server and gradually reduce the TTL as you approach the changeover day. Then on that day reduce it until it's 1 minute. Then after some time make the swap in DNS. Plan the changeover for a day and hour where you can spend a lot of time if needed. You probably won't need it, but you don't want stress. I mean do it after lunch, not 5 minutes before you normally eat, or do it on Wednesday morning rather than late Friday afternoon. If there is a time when the services are least used, aim for that if it fits into a low-stress time. Then again, Sunday morning on the way home from the pubs would be a poor choice, too, even if you're feeling no stress at the moment.

It's probably best to make a list of old and new services along with the necessary DNS entries on paper and then plan out a sequence of events. Making a good checklist long in advance helps.

Those are my 2 cents, but others with much more recent experience will be able to provide fresher and maybe more accurate guidance.

When I migrate domains I install a test domain on the new server and point the MX records to the SUP. If this works for the test domain, chances are it will also work for the real domain.

Have a look at Virtualmin. But I am repeating myself.

jlinkels

The aptitude program is fairly apt at system maintenance if you'll pardon the pun. Here is a default debian /etc/apt/sources.list:

As you will note it rolls on the "stable" branch. Current "stable" is Debian 9 Stretch. To switch repositories you'd do a substitution with "testing" or "sid". However "stable" has been perfectly reasonable release cycle in my experience. As you mentioned all you need to stay updated is:

The issues you mention with the system breaking down following an upgrade usually tends to occur when switching to and fro "testing" or "sid". The sysadmin who is apt at their job can still mitigate such scenarios from arising in the first place by ensuring that the system being operated on is clean and in good order. Second, following a major upgrade, they can do a manual check of the bootloader, initramfs, kernel and init to resolve any errors report by aptitude before rebooting.

The worst case would be reinstalling a root system afresh and redeploying the backup data. It's time consuming but this ensures you have a clean system from the outset. At this point a snapshot image of your root system along with a series of good working order configurations might be a prudent thing to do. It goes without saying that periodic backups of the data (user data) should be made. This data must be system agnostic so that it could potentially be deployed on any system, be it a Debian, Red Hat or Shamware system.

is my advice.