Timelines Between New Releases and TUF Metadata Repository Updates

[sethfduke]

Does anyone happen to know what process or timeline is in place for when a new OS version is released and the release information making it's way into the TUF repository at https://updates.bottlerocket.aws ?

With the recent CVE effecting 1.18.0, I was eager to see the bottlerocket-update-operator having updated the nodes overnight in my clusters, but it sees no updates. After connecting to the node and executing apiclient update check, it still thinks that 1.18.0 is the latest version.

[yeazelm]

Hello @sethfduke, this is a great question. There are a couple things going on so I'll try to address both the general and the specific.

Generally how updates work

A wave schedule is chosen for each release and encoded to the repo. See https://github.com/bottlerocket-os/bottlerocket/tree/develop/sources/updater/waves for a bit more about these files. What this means is that each release has its own dates encoded into it about when waves start and finish. By default, tools like brupop (bottlerocket-update-operator) will respect the setting for waves since under the hood, they are calling apiclient update check like you called out.

Skipping waves

If you want to ignore this wave schedule, you can configure this in the api: https://bottlerocket.dev/en/os/1.18.x/api/settings/updates/#ignore-waves. This will have the update system ignore the waves and determine what the latest is from the TUF repo directly without accounting for the wave schedule for any given release. This can be useful to speed up an update so that all nodes will see the latest update immediately. Just keep in mind this will remain set after the update and the nodes will continue to ignore waves until the setting is removed.

Specifics about 1.19.0

We chose the accelerated wave schedule and it starts on 2024-02-05 to avoid folks being surprised by an aggressive update right before most people's weekend. You can find the wave schedule by downloading the manifest.json for your specific release (assuming you've downloaded tuftool and the root.json for Bottlerocket's repos:

ARCH="x86_64"
VERSION=$(curl -s https://api.github.com/repos/bottlerocket-os/bottlerocket/releases/latest | jq -r '.name')
VARIANT="aws-k8s-1.28"
DOWNLOAD="manifest.json"
OUTDIR="${VARIANT}-${VERSION}"

tuftool  download "${OUTDIR}" --target-name "${DOWNLOAD}" --root ./root.json \
   --metadata-url "https://updates.bottlerocket.aws/2020-07-07/${VARIANT}/x86_64/" \
   --targets-url "https://updates.bottlerocket.aws/targets/"

Now that you have the file, you can look at the data for a particular release (1.19.0 in this case) if you parse it with jq:

cat aws-k8s-1.28-v.1.19.0/manifest.json| jq '.updates | .[] | select(.version=="1.19.0")'
{
  "variant": "aws-k8s-1.28",
  "arch": "x86_64",
  "version": "1.19.0",
  "max_version": "1.19.0",
  "waves": {
    "0": "2024-02-05T19:00:00Z",
    "61": "2024-02-05T22:00:00Z",
    "245": "2024-02-06T02:00:00Z",
    "819": "2024-02-06T10:00:00Z",
    "1638": "2024-02-06T18:00:00Z"
  },
  "images": {
    "boot": "bottlerocket-aws-k8s-1.28-x86_64-1.19.0-2b1a7872-boot.ext4.lz4",
    "root": "bottlerocket-aws-k8s-1.28-x86_64-1.19.0-2b1a7872-root.ext4.lz4",
    "hash": "bottlerocket-aws-k8s-1.28-x86_64-1.19.0-2b1a7872-root.verity.lz4"
  }
}

As you can see, the wave starts at 2024-02-05T19:00:00Z and will move through quickly to provide the update to all nodes. So if you don't choose to skip the window, all the nodes will see 1.19.0 sometime over that window. So if you want to take 1.19.0 more quickly, you would need to set settings.updates.ignore-waves = true and then your nodes will update before Monday.

[sethfduke]

Thank you very much for the detailed explanation!

[landbaychrisburrell]

Hi

Thanks both for the question and the answer -I was wondering the same thing as was expecting an update also. I just had one more clarification to ask. Does the seed number that is shown when running apiclient -u /settings correspond to the indices in the waves element above? e.g. if seed < index, then the update will be updated on the next check in? As is there a 'calculation' as suggested in the docs here: https://github.com/bottlerocket-os/bottlerocket/blob/develop/sources/updater/README.md#update-wave

Cheers
Chris

[yeazelm]

Hello @landbaychrisburrell, that is a good question. I'll just paste the lines from the waves README because I don't think I can say it better 😄

Each Bottlerocket node generates a "seed" for itself which is simply a number between 0-2048 that determines where it falls in the update order. Nodes that have a seed within the current wave will update. All waves include the seeds of the prior wave, so if a node misses its wave for whatever reason, it still updates at a later time.

So to answer your question, yes, if the seed is below the index of the wave and that wave's time has started, that node will update when it checks. If it's beyond the wave's start and end time, it will still update. If you are curious and want to check out the code that does the wave checking, it can be found here!