IAM appliance migration to Rocky Linux 8

The IAM appliance uses CentOS 8 as its base operating system. Since 31/12/2021, CentOS 8 does not receive any further patches and it is vital to migrate the appliance to a supported operating system. IAM appliance offers the ability to migrate to Rocky Linux 8, which we have chosen as the platform we want to build on.

IAM appliance 1.2 is the last appliance version available for CentOS 8 platform. Further releases will support Rocky Linux only.

Performing migration

Migration is a one-way process. Once completed, your appliance will use Rocky Linux as its base operating system. Migrating back to CentOS is not supported. If something goes wrong during the migration, restoring the virtual machine from a backup (or snapshot) is the only way to recover it.

Migration consists of these steps:

  1. Prerequisites check.

  2. IAM appliance backup.

  3. Migration tool installation.

  4. Migration.

  5. RPM packages check.

  6. IAM appliance reboot.

  7. Post-migration checks.

  8. Post-migration IAM appliance backup.

  9. Migration tool uninstall and cleanup.

Prerequisites check

Before you attempt the migration, you have to ensure that:

  • You are running latest possible version of the operating system. For details, see here.

  • You are running latest possible IAM appliance packages.

    • You can get the list of appliance packages' updates by running dnf list updates | grep ^iam.

    • If you need to update some of your appliance’s packages, see here.

    • Exception: If you are using specific version of iam-app-czechidm, iam-app-cas or iam-app-web-proxy, you may leave those packages unupdated. This is a valid configuration for CzechIdM 10.x (LTS) deployments and migration process will handle it.

  • Your appliance can reach the repo.iamappliance.com content server on ports 443/tcp and 8443/tcp.

  • If you are using a proxy server in your appliance configuration, ensure that DNF can use it.

    • Configure the proxy server globally in the /etc/yum.conf using the proxy=http://address.of.your.proxy:port setting.

    • Warning: DNF supports proxy configuration inside particular .repo files. Do not use this feature. Always configure the proxy in the yum.conf.

    • Check that PackageKit can use the proxy by running pkcon repo-list as root. If you get an error instead of the list of repositories, try to logout and login to the server again.

  • DNF uses only official IAM appliance repositories.

    • List all active repositories using dnf repolist command. Each repository must have iam-centos8-* identifier.

  • Clean up all relicts after previous updates.

    • Find all .rpmnew and .rpmsave files, resolve their conflicts and delete those leftover files.

      • Find all .rpmnew files by running find / -name "*.rpmnew" 2>/dev/null as root.

      • Find all .rpmsave files by running find / -name "*.rpmsave" 2>/dev/null as root.

      • Do not touch any files in /var/lib/docker/overlay2/…​.

IAM appliance backup

The migration is a one-way process. This is the only backup you can return to!

  1. Power off your appliance the usual way.

  2. Create a snapshot or a full-backup of the virtual machine.

  3. Power on the virtual machine and wait until all appliance services start.

Migration tool installation

Install the migration tool.

[root@localhost ~]# dnf install iam-migration-centos2rocky

Migration

Now it is time to run the actual migration. We recommend you to stop and disable (systemctl stop/disable …​) appliance services. However, it is not necessary and you can perform the migration while all services are running. This is especially handy when you need to minimize the IAM appliance downtime. However, if you leave the services running and then you have to roll back, you may lose some data.

You can perform the migration while connected over SSH. However, make sure you can access local console of the virtual machine.

  1. Switch (or connect) as the root.

  2. Enter a screen terminal session.

    [root@localhost ~]# screen

  3. If you are using proxy, export the http_proxy and https_proxy environment variables. The proxy must also be set in /etc/yum.conf as the proxy=…​ setting.

    export http_proxy=http://address.of.your.proxy:port
export https_proxy=http://address.of.your.proxy:port

  4. Run the migration script. It will take about 30 minutes to complete.

    [root@localhost ~]# /opt/iam-migration-centos2rocky/migrate2rocky.sh
This script will convert your IAM appliance to Rocky Linux 8. There is no way back. Continue? [y/n] y

  5. Examine migration script log /opt/iam-migration-centos2rocky/migrate2rocky.log. If there are any errors and issues, evaluate them closely. If there is some unrecoverable error after the "Swapping repositories and importing Rocky Linux GPG key." text appeared in the log, then the migration failed and you have to recover your appliance from the virtual machine backup (or snapshot).

    • Following warnings and notices can be safely ignored.

      • Leftover CentOS repositories. Migration script will clean them up automatically.

        Erasing          : centos-linux-repos-8-3.el8.noarch                      6/8
warning: /etc/yum.repos.d/CentOS-Linux-Extras.repo saved as /etc/yum.repos.d/CentOS-Linux-Extras.repo.rpmsave
warning: /etc/yum.repos.d/CentOS-Linux-BaseOS.repo saved as /etc/yum.repos.d/CentOS-Linux-BaseOS.repo.rpmsave
warning: /etc/yum.repos.d/CentOS-Linux-AppStream.repo saved as /etc/yum.repos.d/CentOS-Linux-AppStream.repo.rpmsave

      • Module version specification for python 2. Has no effect on IAM appliance.

        Only module name is required. Ignoring unneeded information in argument: 'libselinux-python:2.8'

      • File shadow.rpmnew is created during reinstall of setup package.

        Reinstalling     : setup-2.12.2-6.el8.noarch                           2/1081
warning: /etc/shadow created as /etc/shadow.rpmnew

      • Cockpit WS is installed as a dependency but not used in current IAM appliance. Ignore this message.

        Running scriptlet: cockpit-ws-264.1-1.el8.x86_64                     353/1081
libsemanage.semanage_direct_install_info: Overriding cockpit module at lower priority 100 with module at priority 200.

      • Warning about changed unit file can be ignored. Server reboot will resolve it.

        Running scriptlet: rng-tools-6.14-4.git.b2b7934e.el8.x86_64          501/1081
Warning: The unit file, source configuration file or drop-ins of rngd-wake-threshold.service changed on disk. Run 'systemctl daemon-reload' to reload units.

      • Leftover CentOS EPEL repositories from epel-release package. Migration script will clean them up automatically.

        Upgrading        : epel-release-8-13.el8.noarch                      530/1081
warning: /etc/yum.repos.d/epel-modular.repo created as /etc/yum.repos.d/epel-modular.repo.rpmnew
warning: /etc/yum.repos.d/epel.repo created as /etc/yum.repos.d/epel.repo.rpmnew

  6. If the migration log does not contain any outstanding issues, proceed with RPM packages check.

RPM packages check

At its end, the migration script will tell you about four files which you need to compare. Those files contain lists of packages and their changes before and after the migration. Logfiles contain server hostname in their name, adjust examples below accordingly.

To quit vimdiff, press ESC to enter its "command mode". Then write :qa and press ENTER. If you accidentally press some other key, simply press ESC to enter "command mode" again.

  1. Check the full package list using vimdiff.

    [root@localhost ~]# vimdiff /opt/iam-migration-centos2rocky/convert/localhost.localdomain-rpm-list-begin.log /opt/iam-migration-centos2rocky/convert/localhost.localdomain-rpm-list-finish.log

    • This diff is a bit hard to read and vimdiff helps tremendously by coloring the output. Most of the output will be marked as changed (red color), but the names of packages will be the same (beginnings of lines will not be red). Some packages will be missing (any centos package) and some new ones will appear (rocky packages). This is expected.

  2. Check actual package changes using vimdiff.

    [root@localhost ~]# vimdiff /opt/iam-migration-centos2rocky/convert/localhost.localdomain-rpm-list-verified-begin.log /opt/iam-migration-centos2rocky/convert/localhost.localdomain-rpm-list-verified-finish.log

    • This diff is much shorter and should mark only a versionlock.list file change and change from CentOS to Rocky repo definition files inside /etc/yum.repos.d/ directory.

    • If there is a /etc/yum.repos.d/epel.repo file marked in the diff, see Known Issues.

IAM appliance reboot

If everything looks OK, force sync filesystem changes and reboot the IAM appliance. The IAM appliance may appear frozen for a while when shutting down.

[root@localhost ~]# sync
[root@localhost ~]# reboot

Post-migration checks

Perform the following post-migration checks. Resolve any issues. If you encounter an irrecoverable issue, recover the whole virtual machine from backup or snapshot.

  1. Check the operating system release file by running cat /etc/os-release. It must contain "Rocky Linux" identification.

  2. Check the dmesg for errors.

  3. Check the /var/log/messages for errors.

  4. Check configured DNF repositories by running dnf repolist. Only repositories with iam-rocky8-* identifier must show up.

  5. Run dnf makecache to check connectivity to software repositories.

  6. In case you are using CzechIdM 10.x (LTS) packages of IAM appliance, check they did not get upgraded. Run rpm -qa | grep ^iam to list packages.

  7. If you have disabled any services before the migration, enable and start them using systemctl start/enable …​.

  8. Wait for all appliance services to start. Check their start and overall functionality in the logs /data/logs/…​/…​.

  9. Access appliance services and do at least some basic testing (user login, user creation, CAS integration, etc.).

Post-migration IAM appliance backup

This backup is performed mainly for diagnostics purposes, in case you encounter issues some time after the migration when the recovery back to CentOS may not be possible.

  1. Power off your appliance the usual way.

  2. Create a snapshot or a full-backup of the virtual machine.

  3. Power on the virtual machine and wait until all appliance services start.

Migration tool uninstall and cleanup

Last thing to do is to uninstall the migration tool and clean up migration logs. Once uninstalled, it cannot be installed again.

We suggest you do this cleanup a week (or more) after the migration. Alternatively, save the logs somewhere safe in case you need them.

 
[root@localhost ~]# dnf rm iam-migration-centos2rocky

[root@localhost ~]# rm -rvf /opt/iam-migration-centos2rocky
removed '/opt/iam-migration-centos2rocky/migrate2rocky.log'
removed '/opt/iam-migration-centos2rocky/convert/localhost.localdomain-rpm-list-begin.log'
removed '/opt/iam-migration-centos2rocky/convert/localhost.localdomain-rpm-list-verified-begin.log'
removed '/opt/iam-migration-centos2rocky/convert/localhost.localdomain-rpm-list-finish.log'
removed '/opt/iam-migration-centos2rocky/convert/localhost.localdomain-rpm-list-verified-finish.log'
removed directory '/opt/iam-migration-centos2rocky/convert'
removed directory '/opt/iam-migration-centos2rocky'

Known Issues

EPEL repository remains enabled

During the migration’s cleanup phase, the following error occurs:

renamed '/etc/yum.repos.d/epel.repo.rpmnew' -> '/etc/yum.repos.d/epel.repo'
Transaction:    Modifying repository
Status:         Waiting in queue
Status:         Waiting for authentication
Status:         Waiting in queue
Status:         Starting
Status:         Querying
Percentage:     0
Status:         Finished
Results:
Fatal error: repo already disabled

Later on, when checking RPM package lists, you can confirm that there is a residual epel.repo difference between pre-migration and post-migration state:

[root@localhost ~]# vimdiff /opt/iam-migration-centos2rocky/convert/localhost-rpm-list-verified-begin.log /opt/iam-migration-centos2rocky/convert/localhost-rpm-list-verified-finish.log
... abbreviated ...
..5....T.  c /etc/yum.repos.d/epel.repo                      |  -----------------------------------------------

This issue occurs due to a race condition in the migration shell script when calling pkcon commands. To correct this issue, simply disable the epel repo:

[root@localhost ~]# pkcon -p repo-disable epel

There are some artifacts left after the migration

Following files are migration artifacts that were not correctly handled by the automatic process. They are mostly remains of various fixes to previous IAM appliance versions. You can safely remove any file listed here.

List of files that may be safely removed
/etc/yum.repos.d/iam-centos8-epel.repo.rpmsave