Release process ############### .. The following makes it possible to use classes inline, used in replacements, but for the very few specific cases, to get custom styling applied. .. role:: machine-casulana .. role:: machine-pettersson .. role:: machine-jigdo .. role:: machine-other .. |casulana| replace:: :machine-casulana:`➤ casulana` .. |pettersson| replace:: :machine-pettersson:`➤ pettersson` .. Ditto for d-i vs. full releases headings. .. role:: release .. |d-i| replace:: :release:`d-i releases` .. |full| replace:: :release:`full releases` .. |d-i+full| replace:: :release:`d-i releases + full releases` Foreword ******** This documentation is an extended version of the historical `README.RELEASE `_, written by kibi when getting onboarded with debian-cd releases. The goal is to have a one-stop documentation giving context as well as ready-to-use snippets. There are two kinds of releases: - d-i releases; - full releases. .. Danger:: Initially written when performing d-i releases, then extended to cover full releases. Those are less frequent, so don't trust anything blindly! Most operations happen on two machines of the Debian projects, operating under the ``debian-cd`` user: - ``casulana``: image builder; - ``pettersson``: image publisher — https://cdimage.debian.org/cdimage/ Some operations, like snapshot management on “remote machines”, might need to be conducted as the ``jigdo`` user on those machines instead. Some operations need to be conducted on the operator's machine (e.g. signing images). 1. Planning *********** |d-i| We're talking about “D-I Codename Alpha N” and “D-I Codename RC N” releases here, which are organized by the installer team and built from ``testing``. The most important part is getting the right installer in place. There are two things to check: - The ``debian-installer`` package migrated to testing. - The ``installer-/`` directories in ``testing`` contain the right installer version; after a ``debian-installer`` upload, and independently of its migration to testing, ``dak copy-installer`` must be used to copy those directories from ``unstable`` to ``testing``. That can be done by the ftp team or by the d-i release manager. After ``dinstall``, a mirror pulse is triggered automatically and ``casulana`` should get all files quite quickly. The installer directories can be checked with the following command, which should return the number of release architectures followed by the desired ``debian-installer`` version: |casulana| .. code:: sh readlink /srv/mirrors/debian/dists/testing/main/installer-*/current | uniq -c .. Tip:: It is best to start the release process right after ``dinstall``, to minimize the risks of having the archive change before everything is said and done. |full| Those are built from ``stable`` (initial release or point releases) or from ``oldstable`` (only point releases). There's more coordination needed here, involving both ftp and release teams to prepare the archive side. Once they give a green light, the archive shouldn't change for a few weeks, so there's plenty of time, ando no need for incremental snapshots. ---- 2. Preparing ************ Take reboot locks ================= Both ``casulana`` and ``pettersson`` must be protected against reboots during the release process. This is achieved by taking a reboot lock. To make sure it doesn't go away, it's best to take it from a screen session. Let's use a dedicated session on each machine. |casulana| |pettersson| .. code:: sh # Create a session, press Enter to validate: screen -RD reboot_lock # In the new session, take the lock: cd ~/build . common.sh reboot_lock It's fine to detach (``^A ^D``) from those screen sessions, and to reattach at the very end. Create shared sessions ====================== Since there are many lengthy operations, it's best to operate within screen sessions, to avoid interruptions if there are issues with the SSH connection. Creating shared screen sessions allow others to attach them, letting them keep an eye on what's happening, or learn about the whole process! |casulana| |pettersson| .. code:: sh # Create a session, initially detached: screen -d -m -S shared # Now, everyone can join with: screen -x shared Disable regular builds ====================== It doesn't hurt to check there are no builds running. As of May 2023, we have daily builds plus weekly builds on Mondays. |casulana| .. code:: sh sed 's/^\(ABORT_.*\)=.*/\1=y/' -i ~/build/ftp.cron Prepare variables ================= Neither ``casulana`` nor ``pettersson`` can push code. To avoid moving files around, it's a good idea to commit on the operator's machine, push to Salsa, and pull from the ``casulana`` and ``pettersson``. .. Tip:: It's a good idea to build a little file with the following variables, so that they can be pasted in the various shells, before pasting other snippets found in this document. Those snippets use the ``${VARIABLE?}`` syntax to make sure an error is generated if a variable isn't set. |d-i| .. code:: sh # Set these: export CODENAME=bookworm export RELEASE=bookworm_di_rc4 export OLD_RELEASE=bookworm_di_rc3 # Compute those: export LOG=${RELEASE?}.log **Note:** When preparing the first ``alpha`` in a release cycle, compare it against the last ``rc`` from the previous release cycle. |full| .. code:: sh # Set these: export CODENAME=bookworm export MAJOR=12 export MINOR=0 # Compute those: export VERSION=${MAJOR?}.${MINOR?}.0 export RELEASE=${CODENAME?}_release if [ ${MINOR?} -gt 0 ]; then export OLD_RELEASE=${MAJOR?}.$((MINOR-1)).0 else export OLD_RELEASE=current fi export LOG=${CODENAME?}-r${MINOR?}.log **Note:** When preparing the initial build, ``current`` points to the last point release of the previous ``stable`` (which just became ``oldstable``), so setting ``OLD_RELEASE`` to ``current`` should be sufficient, at least until the very last sanity checks (``current`` will have been updated by then). Prepare configuration files =========================== .. Caution:: Most of the following happens on the **operator's machine**. **1.** Start by exporting the variables prepared above. **2.** In ``setup.git``, bootstrap a config file out of the previous release, and adjust as needed: |d-i| .. code:: sh # This doesn't need frequent updates: editor ${CODENAME?}/settings.sh # At least the version number needs an update: cp ${CODENAME?}/CONF.sh.${OLD_RELEASE?} ${CODENAME?}/CONF.sh.${RELEASE?} editor ${CODENAME?}/CONF.sh.${RELEASE?} |full| When preparing the initial build (e.g. 12.0.0), copy the configuration file from the last d-i release as ``CONF.sh.${CODENAME?}_release``, and tweak the ``DEBVERSION`` and ``OFFICIAL`` variables. Compare this configuration file against the one from the previous codename, which should reflect changes that happened during the release cycle (e.g. dropping support for old checksums or tweaking support for firmware). Also compare ``settings.sh`` against the one from the previous codename. When preparing a point release (e.g. 12.1.0), bumping ``DEBVERSION`` in ``CONF.sh.${CODENAME?}_release`` should be the only change needed. **3.** In ``live-setup.git``, ditto for the config file: |d-i| .. code:: sh cp available/CONF.sh.${OLD_RELEASE?} available/CONF.sh.${RELEASE?} editor available/CONF.sh.${RELEASE?} |full| When preparing the initial build (e.g. 12.0.0), copy the configuration file from the last d-i release as ``CONF.sh.${CODENAME?}_release``, and tweak the ``VERSION`` variable. When preparing a point release (e.g. 12.1.0), bump ``VERSION``. **4.** Review, commit, and push both repositories to Salsa… **5.** Pull them from ``casulana``. |casulana| .. code:: sh # Important: paste exported variables cd ~/build.${CODENAME?} && git pull cd ~/live && git pull **6.** Set variables on ``pettersson`` as well. |pettersson| .. code:: sh # Important: paste exported variables 3. Building *********** Ready, steady, go! |casulana| .. code:: sh cd ~/build.${CODENAME?} script -c "RELEASE_BUILD=${RELEASE?} ./cronjob.weekly" ${LOG?} Comments, as of May 2023: - Parallelism: up to 20 concurrent jobs. - Estimate run time: 3.5 hours for Bookworm, 5 hours for Bullseye. - Synchronization: files are transferred to ``pettersson`` on a per-arch basis, via ``iso_run``. - Post-processing: some forced command deals with the transferred files and moves them into the temporary location on ``pettersson``: ``/mnt/nfs-cdimage/.${RELEASE?}``. - Log files: nice to keep around for later reference if needed, not important enough to be versioned. - Types of build: - ``firmware``: almost immediate. - ``debian-cd``: installer builds, first arch appears after 1.5 hour. - ``live-free``: live builds, ``amd64`` and ``source`` appear last (plus ``i386`` on Bullseye). - **FIXME:** extra ``unofficial/non-free`` for Bullseye builds. .. Important:: The build takes a long time but it's important to stay on top of things! See below. Next steps, **while** the build is going on: - Watch the temporary location for new directories: one directory per architecture under ``debian-cd``, plus ``amd64`` under ``live-free`` (there's ``source`` too but that part is not getting tested). - When that happens, notify testers: `4. Testing`_. - When that happens, for d-i releases, poke snapshots: `5. Snapshotting`_. Once the build has finished, for d-i releases and full releases: - Perform a final snapshot on remote machines, see `5. Snapshotting`_. 4. Testing ********** Do some testing, ideally get help! 5. Snapshotting *************** Context ======= There are 3 machines involved in snapshots: - ``free``, behind ``ftp.uk.debian.org``, operated by Phil Hands; - ``jack``, operated by Steve McIntyre; - ``pettersson``, the only ``debian.org`` machine. The first two need to get files from ``pettersson``, then running jigdo-related commands on them, so that all required files have safeguarded. kibi has access to ``free``, and automated ``rsync`` and ``jigdo-watcher2`` into a script that can be used for incremental snapshots, and later on for final snapshots. It is important to have snapshots at least on ``free`` for redundancy. ``jack`` is an extra safety net, might also be useful for testers, but is less critical to have. **FIXME:** - Rename and add this script to versioning. - Check whether it makes sense, and works, for full releases. Incremental snapshots ===================== |d-i| This can take a while to process, it's advised to run under screen as well. :machine-jigdo:`➤ jigdo@free` .. code:: sh # Important: paste exported variables kibi-snapshot ${RELEASE?} Ideally, run this command: - Every time new architectures are sync'd from ``casulana`` to ``pettersson``. - One last time once the build has finished on ``casulana`` (which means everything has been sync'd to ``pettersson``). |full| A final snapshot should be sufficient. Final snapshot ============== |d-i| This is similar to incremental snapshots, except for an extra ``final`` argument, so that files are fetched from their final location. :machine-jigdo:`➤ jigdo@free` .. code:: sh kibi-snapshot ${RELEASE?} final |full| **FIXME:** - Check/figure out the ordering between moving things into place and snapshotting. Might depend on test results? Or just snapshot in any case at the end of the build, from the ``.${RELEASE?}`` location? - If before moving things into place, pass ``final`` after ``$RELEASE``, just like in the d-i case. - If after moving things into place, teach the script about ``full-final``, and pass it after the ``$VERSION``. :machine-jigdo:`➤ jigdo@free` .. code:: sh # Important: paste exported variables # FIXME: see above. kibi-snapshot ${RELEASE?} final kibi-snapshot ${VERSION?} full-final # requires a little code change 6. More work for full releases ****************************** List of files used ================== |d-i| **Skip this step!** |full| It's very important to build and publish the list of files used. They are used when preparing update images, but they're also used by other people. We have a list of all files (for each release) but also a diff against the first release for point releases. Note that those will be generated on the build machine but need to be published to git, so a little dance is required. **1.** Prepare files: |casulana| .. code:: sh # Work dir: mkdir -p ~/lists/${CODENAME?} cd ~/lists/${CODENAME?} # Generate full list: ~/bin/make-cd-list ${CODENAME?} /srv/cdbuilder.debian.org/src/ftp/debian/ > r${MINOR?}list # Generate diff against r0, if this is a point release: if [ ${MINOR?} -gt 0 ]; then diff -u r0list r${MINOR?}list | grep ^+pool | cut -c2- > r0-r${MINOR?}.diff fi **2.** Sync locally, to the ``packages-lists.git`` repository, commit, and push: :machine-other:`➤ operator@machine` .. code:: sh # Important: paste exported variables rsync -av casulana.debian.org:~debian-cd/lists/${CODENAME?} . git add ${CODENAME?} git commit -m "Add data for ${VERSION?}" git push **3.** Check everything is in order, updating the repository: |casulana| .. code:: sh cd ~/lists mv ${CODENAME?} ${CODENAME?}.local git pull # Compare and clean up if everything matches: diff -urN ${CODENAME?} ${CODENAME?}.local && rm -rf ${CODENAME?}.local git status **Note:** Only ``.nobackup`` should show up. Generate update DVD images ========================== |d-i| **Skip this step!** |full| Skip if performing an initial release. **FIXME:** Check how Steve prefers doing the switch-over. When performing a point release: First, enter and check the status of the right ``debian-cd`` repository. Second, tweak two variables in the ``update-cd`` script: ``VER`` (full version for the new release) and ``DIFF`` (path to the diff between ``r0`` and ``r${MINOR?}``). Third, generate the update images. |casulana| .. code:: sh # Sanity check: after the initial release, buildd/${CODENAME?} should be the current branch cd /home/debian-cd/build.${CODENAME?}/debian-cd git branch # Tweak VER and DIFF: editor update-cd # Build! ./update-cd dvd Since we're only expecting the minor version number to change in two places, it's customary not to commit those changes. 7. Releasing ************ Bittorrent clean-up =================== |d-i| The d-i releases are confidential enough that making images available via bittorrent isn't really worth it. Those files are prepared unconditionally, so we perform some clean-up. |pettersson| .. code:: sh cd /mnt/nfs-cdimage/.${RELEASE?} find . -name 'bt*' # first check find . -name 'bt*' | xargs rm -rvf # then delete |full| **Skip this step!** Sanity checks ============= It's a good idea to compare the old release and the tentative new one. The following script performs some sanity checks on the three target directories. We expect directory sizes and file lists to be “comparable”: either they are very similar or the differences can be explained. |d-i| |pettersson| .. code:: sh ~/bin/sanity-compare-releases ${OLD_RELEASE?} ${RELEASE?} ${CODENAME?} **Note:** Lot of noise is to be expected when comparing ``-rc1`` to the previous ``-alphaN``. The script can be improved at this point, maybe by unversioning via ``-develN`` all the time (instead of ``-alphaN`` and ``-rcN``). |full| |pettersson| .. code:: sh ~/bin/sanity-compare-releases --full ${OLD_RELEASE?} ${RELEASE?} ${CODENAME?} **Note:** This doesn't compare the ``unofficial/non-free`` part for Bullseye! Merging checksums ================= |d-i+full| For Bookworm+, we have a single ``debian-cd`` directory. For Bullseye, we have an extra one (``/mnt/nfs-cdimage/unofficial/non-free/images-including-firmware/.${RELEASE?}/debian-cd``), and the following should be done in each directory. As of May 2023, we have ``SHA256SUMS`` and ``SHA512SUMS``, with a twist: - Under ``firmware`` and ``live-free`` directories, they appear normally. - Under ``debian-cd`` directories, there's a ``.large`` suffix. The ``merge-sums`` tool massages them into regular files, without the suffix. |pettersson| .. code:: sh cd /mnt/nfs-cdimage/.${RELEASE?}/debian-cd ~/build.${CODENAME?}/merge-sums $PWD **FIXME:** Verify path for second run, needed for Bullseye's full builds. Collecting, signing, deploying checksums ======================================== |d-i+full| Big picture: #. checksums are bundled in a tarball; #. they are copied by the operator on their machine; #. the operator runs some signing machinery; #. the operator uploads the updated tarball; #. the new tarball is deployed. **1.** Bundle checksums. |pettersson| .. code:: sh cd /mnt/nfs-cdimage/.${RELEASE?} find . -name '*SUMS' | sort | xargs tar czvf ~/md5.tar.gz **2.** to **4.** In ``steve-scripts.git``, copy, sign, and copy back: :machine-other:`➤ operator@machine` .. code:: sh scp pettersson.debian.org:~debian-cd/md5.tar.gz . # copies from ~debian-cd ./sign-images md5.tar.gz # the original one gets replaced scp md5.tar.gz pettersson.debian.org: # copies to ~operator **5.** Finally, deploy the checksums and their signatures. **Note:** The following code excerpt relies on the sudo user having started the screen session. |pettersson| .. code:: sh cd /mnt/nfs-cdimage/.${RELEASE?} # Either rely on the sudo user: eval tar xvf ~${SUDO_USER?}/md5.tar.gz # or explicitly extract from the operator's home: tar xvf ~operator/md5.tar.gz **FIXME** for Bullseye full releases: probably two dances instead of one. Moving files into place ======================= |d-i| The d-i releases are stored at the top-level. .. code:: sh cd /mnt/nfs-cdimage mv .${RELEASE?}/debian-cd ${RELEASE?} mv .${RELEASE?}/live-free ${RELEASE?}-live mv .${RELEASE?}/firmware firmware/${CODENAME?}/${RELEASE?} rmdir .${RELEASE?} |full| The full releases are stored under ``release/`` and we finally use the ``VERSION`` variable that was computed from the major and minor version numbers. .. code:: sh # Move things, just like in the d-i case: cd /mnt/nfs-cdimage mv .${RELEASE?}/debian-cd release/${VERSION?} mv .${RELEASE?}/live-free release/${VERSION?}-live mv .${RELEASE?}/firmware firmware/${CODENAME?}/${VERSION?} rmdir .${RELEASE?} **FIXME:** - Bullseye also has a separate ``unofficial/non-free`` directory. Symlinks and mirroring ====================== |d-i| **Skip this step!** |full| |pettersson| .. code:: sh # Update symlinks for release directories (installer and live): cd /mnt/nfs-cdimage/release ln -sfv ${VERSION?} current ln -sfv ${VERSION?}-live current-live # Update/create symlinks for firmware directory: cd /mnt/nfs-cdimage/firmware/${CODENAME?} ln -sfv ${VERSION?} current ln -sfv ${VERSION?} $(date +%Y%m%d) **FIXME:** - When performing a point release, the previous release (installer and live) should be moved to the ``archive`` directory. - Clarify what happens for ``oldstable`` when preparing an initial release. - Clarify status of ``latest`` symlinks. For the installer builds, create a directory ``/mnt/nfs-cdimage/release/project/build/${VERSION?}``. In it, add a file containing ``cd dvd`` for each architecture in the release. The mirrors use this to work out what/how to mirror things. **FIXME:** - Maybe ``cp -r`` from the previous release, and adjust as needed (e.g. dropping ``multi-arch``). Finally, trigger mirroring: |pettersson| .. code:: sh ~/bin/trigger_seeders ~/bin/trigger_mirrors Sanity checks ============= Same story as earlier, except checksums have been merged, signed, and everything has been moved into place. Passing an extra option makes sure we look in the right place, and that checkums are no longer treated specifically. |d-i| |pettersson| .. code:: sh ~/bin/sanity-compare-releases --in-place ${OLD_RELEASE?} ${RELEASE?} ${CODENAME?} |full| |pettersson| .. code:: sh ~/bin/sanity-compare-releases --full --in-place ${OLD_RELEASE?} ${RELEASE?} ${CODENAME?} **Note:** For the initial build, having ``OLD_RELEASE`` set to ``current`` worked until this point, but since that's just been updated, replace ``${OLD_RELEASE?}`` with the full version for the last point release of (now) ``oldstable``. Final snapshot ============== See `5. Snapshotting`_ for the remote machines. Also put them in place on ``/org/cdbuilder.debian.org/dst/deb-cd/cdimage``. That's done by creating a snapshot once the release is published. |d-i| |pettersson| .. code:: sh ~/bin/release-snapshot ${RELEASE?} |full| |pettersson| .. code:: sh ~/bin/release-snapshot ${VERSION?} 8. Publishing ************* It's time to update the header that appears on the https://cdimage.debian.org/cdimage/ landing page. **1.** Update ``setup.git``, adjusting at least version number and release date: :machine-other:`➤ operator@machine` .. code:: sh editor publish-docs/images-HEADER.html **2.** Review, commit, and push. **3.** Pull from petterssson. |pettersson| .. code:: sh cd /mnt/nfs-cdimage/.git/setup git pull 9. Cleaning up ************** Enable regular builds again =========================== |casulana| .. code:: sh sed 's/^\(ABORT_.*\)=.*/\1=n/' -i ~/build/ftp.cron Close all shared screen sessions ================================ Hit ``^D`` in the right places! Release reboot locks ==================== Reattach specific screen sessions, then release the lock, and close the screen sessions as well. |casulana| |pettersson| .. code:: sh # Reattach session: screen -RD reboot_lock # Inside, release the lock, then close everything (shell, screen, ssh). reboot_unlock exit