Release process¶

Foreword¶

This documentation is an extended version of the historical README.RELEASE, written by kibi when getting onboarded with debian-cd releases. The goal is to have a one-stop documentation giving context as well as ready-to-use snippets.

There are two kinds of releases:

d-i releases;
full releases.

Danger

Initially written when performing d-i releases, then extended to cover full releases. Those are less frequent, so don’t trust anything blindly!

Most operations happen on two machines of the Debian projects, operating under the debian-cd user:

casulana: image builder;
pettersson: image publisher — https://cdimage.debian.org/cdimage/

Some operations, like snapshot management on “remote machines”, might need to be conducted as the jigdo user on those machines instead. Some operations need to be conducted on the operator’s machine (e.g. signing images).

1. Planning¶

d-i releases

We’re talking about “D-I Codename Alpha N” and “D-I Codename RC N” releases here, which are organized by the installer team and built from testing.

The most important part is getting the right installer in place. There are two things to check:

The debian-installer package migrated to testing.
The installer-<ARCH>/ directories in testing contain the right installer version; after a debian-installer upload, and independently of its migration to testing, dak copy-installer must be used to copy those directories from unstable to testing. That can be done by the ftp team or by the d-i release manager.

After dinstall, a mirror pulse is triggered automatically and casulana should get all files quite quickly. The installer directories can be checked with the following command, which should return the number of release architectures followed by the desired debian-installer version:

➤ casulana

readlink /srv/mirrors/debian/dists/testing/main/installer-*/current | uniq -c

Tip

It is best to start the release process right after dinstall, to minimize the risks of having the archive change before everything is said and done.

full releases

Those are built from stable (initial release or point releases) or from oldstable (only point releases).

There’s more coordination needed here, involving both ftp and release teams to prepare the archive side. Once they give a green light, the archive shouldn’t change for a few weeks, so there’s plenty of time, ando no need for incremental snapshots.

2. Preparing¶

Take reboot locks¶

Both casulana and pettersson must be protected against reboots during the release process. This is achieved by taking a reboot lock. To make sure it doesn’t go away, it’s best to take it from a screen session. Let’s use a dedicated session on each machine.

➤ casulana ➤ pettersson

# Create a session, press Enter to validate:
screen -RD reboot_lock

# In the new session, take the lock:
cd ~/build
. common.sh
reboot_lock

It’s fine to detach (^A ^D) from those screen sessions, and to reattach at the very end.

Create shared sessions¶

Since there are many lengthy operations, it’s best to operate within screen sessions, to avoid interruptions if there are issues with the SSH connection. Creating shared screen sessions allow others to attach them, letting them keep an eye on what’s happening, or learn about the whole process!

➤ casulana ➤ pettersson

# Create a session, initially detached:
screen -d -m -S shared

# Now, everyone can join with:
screen -x shared

Disable regular builds¶

It doesn’t hurt to check there are no builds running. As of May 2023, we have daily builds plus weekly builds on Mondays.

➤ casulana

sed 's/^\(ABORT_.*\)=.*/\1=y/' -i ~/build/ftp.cron

Prepare variables¶

Neither casulana nor pettersson can push code. To avoid moving files around, it’s a good idea to commit on the operator’s machine, push to Salsa, and pull from the casulana and pettersson.

Tip

It’s a good idea to build a little file with the following variables, so that they can be pasted in the various shells, before pasting other snippets found in this document.

Those snippets use the ${VARIABLE?} syntax to make sure an error is generated if a variable isn’t set.

d-i releases

# Set these:
export CODENAME=bookworm
export RELEASE=bookworm_di_rc4
export OLD_RELEASE=bookworm_di_rc3

# Compute those:
export LOG=${RELEASE?}.log

Note: When preparing the first alpha in a release cycle, compare it against the last rc from the previous release cycle.

full releases

# Set these:
export CODENAME=bookworm
export MAJOR=12
export MINOR=0

# Compute those:
export VERSION=${MAJOR?}.${MINOR?}.0
export RELEASE=${CODENAME?}_release
if [ ${MINOR?} -gt 0 ]; then
  export OLD_RELEASE=${MAJOR?}.$((MINOR-1)).0
else
  export OLD_RELEASE=current
fi
export LOG=${CODENAME?}-r${MINOR?}.log

Note: When preparing the initial build, current points to the last point release of the previous stable (which just became oldstable), so setting OLD_RELEASE to current should be sufficient, at least until the very last sanity checks (current will have been updated by then).

Prepare configuration files¶

Caution

Most of the following happens on the operator’s machine.

1. Start by exporting the variables prepared above.

2. In setup.git, bootstrap a config file out of the previous release, and adjust as needed:

d-i releases

# This doesn't need frequent updates:
editor ${CODENAME?}/settings.sh

# At least the version number needs an update:
cp ${CODENAME?}/CONF.sh.${OLD_RELEASE?} ${CODENAME?}/CONF.sh.${RELEASE?}
editor ${CODENAME?}/CONF.sh.${RELEASE?}

full releases

When preparing the initial build (e.g. 12.0.0), copy the configuration file from the last d-i release as CONF.sh.${CODENAME?}_release, and tweak the DEBVERSION and OFFICIAL variables.

Compare this configuration file against the one from the previous codename, which should reflect changes that happened during the release cycle (e.g. dropping support for old checksums or tweaking support for firmware). Also compare settings.sh against the one from the previous codename.

When preparing a point release (e.g. 12.1.0), bumping DEBVERSION in CONF.sh.${CODENAME?}_release should be the only change needed.

3. In live-setup.git, ditto for the config file:

d-i releases

cp available/CONF.sh.${OLD_RELEASE?} available/CONF.sh.${RELEASE?}
editor available/CONF.sh.${RELEASE?}

full releases

When preparing the initial build (e.g. 12.0.0), copy the configuration file from the last d-i release as CONF.sh.${CODENAME?}_release, and tweak the VERSION variable.

When preparing a point release (e.g. 12.1.0), bump VERSION.

4. Review, commit, and push both repositories to Salsa…

5. Pull them from casulana.

➤ casulana

# Important: paste exported variables

cd ~/build.${CODENAME?} && git pull
cd ~/live               && git pull

6. Set variables on pettersson as well.

➤ pettersson

# Important: paste exported variables

3. Building¶

Ready, steady, go!

➤ casulana

cd ~/build.${CODENAME?}
script -c "RELEASE_BUILD=${RELEASE?} ./cronjob.weekly" ${LOG?}

Comments, as of May 2023:

Parallelism: up to 20 concurrent jobs.
Estimate run time: 3.5 hours for Bookworm, 5 hours for Bullseye.
Synchronization: files are transferred to pettersson on a per-arch basis, via iso_run.
Post-processing: some forced command deals with the transferred files and moves them into the temporary location on pettersson: /mnt/nfs-cdimage/.${RELEASE?}.
Log files: nice to keep around for later reference if needed, not important enough to be versioned.
Types of build:
- firmware: almost immediate.
- debian-cd: installer builds, first arch appears after 1.5 hour.
- live-free: live builds, amd64 and source appear last (plus i386 on Bullseye).
FIXME: extra unofficial/non-free for Bullseye builds.

Important

The build takes a long time but it’s important to stay on top of things! See below.

Next steps, while the build is going on:

Watch the temporary location for new directories: one directory per architecture under debian-cd, plus amd64 under live-free (there’s source too but that part is not getting tested).
When that happens, notify testers: 4. Testing.
When that happens, for d-i releases, poke snapshots: 5. Snapshotting.

Once the build has finished, for d-i releases and full releases:

Perform a final snapshot on remote machines, see 5. Snapshotting.

4. Testing¶

Do some testing, ideally get help!

5. Snapshotting¶

Context¶

There are 3 machines involved in snapshots:

free, behind ftp.uk.debian.org, operated by Phil Hands;
jack, operated by Steve McIntyre;
pettersson, the only debian.org machine.

The first two need to get files from pettersson, then running jigdo-related commands on them, so that all required files have safeguarded.

kibi has access to free, and automated rsync and jigdo-watcher2 into a script that can be used for incremental snapshots, and later on for final snapshots. It is important to have snapshots at least on free for redundancy. jack is an extra safety net, might also be useful for testers, but is less critical to have.

FIXME:

Rename and add this script to versioning.
Check whether it makes sense, and works, for full releases.

Incremental snapshots¶

d-i releases

This can take a while to process, it’s advised to run under screen as well.

➤ jigdo@free

# Important: paste exported variables

kibi-snapshot ${RELEASE?}

Ideally, run this command:

Every time new architectures are sync’d from casulana to pettersson.
One last time once the build has finished on casulana (which means everything has been sync’d to pettersson).

full releases

A final snapshot should be sufficient.

Final snapshot¶

d-i releases

This is similar to incremental snapshots, except for an extra final argument, so that files are fetched from their final location.

➤ jigdo@free

kibi-snapshot ${RELEASE?} final

full releases

FIXME:

Check/figure out the ordering between moving things into place and snapshotting. Might depend on test results? Or just snapshot in any case at the end of the build, from the .${RELEASE?} location?
If before moving things into place, pass final after $RELEASE, just like in the d-i case.
If after moving things into place, teach the script about full-final, and pass it after the $VERSION.

➤ jigdo@free

# Important: paste exported variables

# FIXME: see above.
kibi-snapshot ${RELEASE?} final
kibi-snapshot ${VERSION?} full-final # requires a little code change

6. More work for full releases¶

List of files used¶

d-i releases

Skip this step!

full releases

It’s very important to build and publish the list of files used. They are used when preparing update images, but they’re also used by other people. We have a list of all files (for each release) but also a diff against the first release for point releases.

Note that those will be generated on the build machine but need to be published to git, so a little dance is required.

1. Prepare files:

➤ casulana

# Work dir:
mkdir -p ~/lists/${CODENAME?}
cd ~/lists/${CODENAME?}

# Generate full list:
~/bin/make-cd-list ${CODENAME?} /srv/cdbuilder.debian.org/src/ftp/debian/ > r${MINOR?}list

# Generate diff against r0, if this is a point release:
if [ ${MINOR?} -gt 0 ]; then
    diff -u r0list r${MINOR?}list | grep ^+pool | cut -c2- > r0-r${MINOR?}.diff
fi

2. Sync locally, to the packages-lists.git repository, commit, and push:

➤ operator@machine

# Important: paste exported variables

rsync -av casulana.debian.org:~debian-cd/lists/${CODENAME?} .
git add ${CODENAME?}
git commit -m "Add data for ${VERSION?}"
git push

3. Check everything is in order, updating the repository:

➤ casulana

cd ~/lists
mv ${CODENAME?} ${CODENAME?}.local
git pull
# Compare and clean up if everything matches:
diff -urN ${CODENAME?} ${CODENAME?}.local && rm -rf ${CODENAME?}.local
git status

Note: Only .nobackup should show up.

Generate update DVD images¶

d-i releases

Skip this step!

full releases

Skip if performing an initial release.

FIXME: Check how Steve prefers doing the switch-over.

When performing a point release: First, enter and check the status of the right debian-cd repository. Second, tweak two variables in the update-cd script: VER (full version for the new release) and DIFF (path to the diff between r0 and r${MINOR?}). Third, generate the update images.

➤ casulana

# Sanity check: after the initial release, buildd/${CODENAME?} should be the current branch
cd /home/debian-cd/build.${CODENAME?}/debian-cd
git branch

# Tweak VER and DIFF:
editor update-cd

# Build!
./update-cd dvd

Since we’re only expecting the minor version number to change in two places, it’s customary not to commit those changes.

7. Releasing¶

Bittorrent clean-up¶

d-i releases

The d-i releases are confidential enough that making images available via bittorrent isn’t really worth it. Those files are prepared unconditionally, so we perform some clean-up.

➤ pettersson

cd /mnt/nfs-cdimage/.${RELEASE?}
find . -name 'bt*'                 # first check
find . -name 'bt*' | xargs rm -rvf # then delete

full releases

Skip this step!

Sanity checks¶

It’s a good idea to compare the old release and the tentative new one. The following script performs some sanity checks on the three target directories. We expect directory sizes and file lists to be “comparable”: either they are very similar or the differences can be explained.

d-i releases

➤ pettersson

~/bin/sanity-compare-releases ${OLD_RELEASE?} ${RELEASE?} ${CODENAME?}

Note: Lot of noise is to be expected when comparing -rc1 to the previous -alphaN. The script can be improved at this point, maybe by unversioning via -develN all the time (instead of -alphaN and -rcN).

full releases

➤ pettersson

~/bin/sanity-compare-releases --full ${OLD_RELEASE?} ${RELEASE?} ${CODENAME?}

Note: This doesn’t compare the unofficial/non-free part for Bullseye!

Merging checksums¶

d-i releases + full releases

For Bookworm+, we have a single debian-cd directory.

For Bullseye, we have an extra one (/mnt/nfs-cdimage/unofficial/non-free/images-including-firmware/.${RELEASE?}/debian-cd), and the following should be done in each directory.

As of May 2023, we have SHA256SUMS and SHA512SUMS, with a twist:

Under firmware and live-free directories, they appear normally.
Under debian-cd directories, there’s a .large suffix.

The merge-sums tool massages them into regular files, without the suffix.

➤ pettersson

cd /mnt/nfs-cdimage/.${RELEASE?}/debian-cd
~/build.${CODENAME?}/merge-sums $PWD

FIXME: Verify path for second run, needed for Bullseye’s full builds.

Collecting, signing, deploying checksums¶

d-i releases + full releases

Big picture:

checksums are bundled in a tarball;
they are copied by the operator on their machine;
the operator runs some signing machinery;
the operator uploads the updated tarball;
the new tarball is deployed.

1. Bundle checksums.

➤ pettersson

cd /mnt/nfs-cdimage/.${RELEASE?}
find . -name '*SUMS' | sort | xargs tar czvf ~/md5.tar.gz

2. to 4. In steve-scripts.git, copy, sign, and copy back:

➤ operator@machine

scp pettersson.debian.org:~debian-cd/md5.tar.gz . # copies from ~debian-cd
./sign-images md5.tar.gz                          # the original one gets replaced
scp md5.tar.gz pettersson.debian.org:             # copies to ~operator

5. Finally, deploy the checksums and their signatures.

Note: The following code excerpt relies on the sudo user having started the screen session.

➤ pettersson

cd /mnt/nfs-cdimage/.${RELEASE?}

# Either rely on the sudo user:
eval tar xvf ~${SUDO_USER?}/md5.tar.gz

# or explicitly extract from the operator's home:
tar xvf ~operator/md5.tar.gz

FIXME for Bullseye full releases: probably two dances instead of one.

Moving files into place¶

d-i releases

The d-i releases are stored at the top-level.

cd /mnt/nfs-cdimage
mv .${RELEASE?}/debian-cd ${RELEASE?}
mv .${RELEASE?}/live-free ${RELEASE?}-live
mv .${RELEASE?}/firmware  firmware/${CODENAME?}/${RELEASE?}
rmdir .${RELEASE?}

full releases

The full releases are stored under release/ and we finally use the VERSION variable that was computed from the major and minor version numbers.

# Move things, just like in the d-i case:
cd /mnt/nfs-cdimage
mv .${RELEASE?}/debian-cd release/${VERSION?}
mv .${RELEASE?}/live-free release/${VERSION?}-live
mv .${RELEASE?}/firmware  firmware/${CODENAME?}/${VERSION?}
rmdir .${RELEASE?}

FIXME:

Bullseye also has a separate unofficial/non-free directory.

Symlinks and mirroring¶

d-i releases

Skip this step!

full releases

➤ pettersson

# Update symlinks for release directories (installer and live):
cd /mnt/nfs-cdimage/release
ln -sfv ${VERSION?}      current
ln -sfv ${VERSION?}-live current-live

# Update/create symlinks for firmware directory:
cd /mnt/nfs-cdimage/firmware/${CODENAME?}
ln -sfv ${VERSION?} current
ln -sfv ${VERSION?} $(date +%Y%m%d)

FIXME:

When performing a point release, the previous release (installer and live) should be moved to the archive directory.
Clarify what happens for oldstable when preparing an initial release.
Clarify status of latest symlinks.

For the installer builds, create a directory /mnt/nfs-cdimage/release/project/build/${VERSION?}. In it, add a file containing cd dvd for each architecture in the release. The mirrors use this to work out what/how to mirror things.

FIXME:

Maybe cp -r from the previous release, and adjust as needed (e.g. dropping multi-arch).

Finally, trigger mirroring:

➤ pettersson

~/bin/trigger_seeders
~/bin/trigger_mirrors

Sanity checks¶

Same story as earlier, except checksums have been merged, signed, and everything has been moved into place. Passing an extra option makes sure we look in the right place, and that checkums are no longer treated specifically.

d-i releases

➤ pettersson

~/bin/sanity-compare-releases --in-place ${OLD_RELEASE?} ${RELEASE?} ${CODENAME?}

full releases

➤ pettersson

~/bin/sanity-compare-releases --full --in-place ${OLD_RELEASE?} ${RELEASE?} ${CODENAME?}

Note: For the initial build, having OLD_RELEASE set to current worked until this point, but since that’s just been updated, replace ${OLD_RELEASE?} with the full version for the last point release of (now) oldstable.

Final snapshot¶

See 5. Snapshotting for the remote machines.

Also put them in place on /org/cdbuilder.debian.org/dst/deb-cd/cdimage. That’s done by creating a snapshot once the release is published.

d-i releases

➤ pettersson

~/bin/release-snapshot ${RELEASE?}

full releases

➤ pettersson

~/bin/release-snapshot ${VERSION?}

8. Publishing¶

It’s time to update the header that appears on the https://cdimage.debian.org/cdimage/ landing page.

1. Update setup.git, adjusting at least version number and release date:

➤ operator@machine

editor publish-docs/images-HEADER.html

2. Review, commit, and push.

3. Pull from petterssson.

➤ pettersson

cd /mnt/nfs-cdimage/.git/setup
git pull

9. Cleaning up¶

Enable regular builds again¶

➤ casulana

sed 's/^\(ABORT_.*\)=.*/\1=n/' -i ~/build/ftp.cron

Close all shared screen sessions¶

Hit ^D in the right places!

Release reboot locks¶

Reattach specific screen sessions, then release the lock, and close the screen sessions as well.

➤ casulana ➤ pettersson

# Reattach session:
screen -RD reboot_lock

# Inside, release the lock, then close everything (shell, screen, ssh).
reboot_unlock
exit