zpool upgrade and bootcode on 13-RELEASE

[Michael Gmelin]

Quick question:

When upgrading a zpool, there used to be a message like this:

If you boot from pool 'zroot', don't forget to update boot code.
Assuming you use GPT partitioning and da0 is your boot disk
the following command will do it:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

After upgrading from 12.2-RELEASE to 13.0-RELEASE, there is no such
message:

# zpool upgrade zroot
This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':
userobj_accounting
encryption
project_quota
allocation_classes
resilver_defer
bookmark_v2
redaction_bookmarks
redacted_datasets
bookmark_written
log_spacemap
livelist
device_rebuild
zstd_compress
draid

#

Does this mean, re-installing the bootcode isn't necessary anymore - or
has the warning been removed by accident/as a side effect of merging
with OpenZFS?

Thanks,
Michael

[Ryan Moeller]

Post by Michael Gmelin
Does this mean, re-installing the bootcode isn't necessary anymore - or
has the warning been removed by accident/as a side effect of merging
with OpenZFS?

The code was lost. You still need to ensure your bootcode is up to date.

-Ryan

[Kurt Jaeger]

Hi!

Post by Michael Gmelin
Does this mean, re-installing the bootcode isn't necessary anymore - or
has the warning been removed by accident/as a side effect of merging
with OpenZFS?

On the contrary, because of the switch from FreeBSD ZFS to
OpenZFS, the bootcodes needs to be updated! It's unfortunate
that no message is displayed 8-(

The problem is, finding out which bootcode needs to go where etc.

It depends if your system boots from the EFI partition or
from the freebsd-boot partition. And how your system is
partitioned.

I have some notes for gpart filesystems:

- check with
gpart show
This system has both (!):
partition 1 is efi
partition 2 is legacy boot

=> 40 4000797280 ada0 GPT (1.9T)
40 409600 1 efi (200M)
409640 1024 2 freebsd-boot (512K)
410664 984 - free - (492K)
411648 18874368 3 freebsd-swap (9.0G)
19286016 3981510656 4 freebsd-zfs (1.9T)
4000796672 648 - free - (324K)

- To update the EFI:
mount -t msdos /dev/ada0p1 /mnt
cd /mnt/efi/boot
mv BOOTx64.efi BOOTx64.efi-old
cp /boot/loader.efi BOOTx64.efi
cd /
umount /mnt

- To update the legacy boot:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0

- If efi is not mountable ? create a msdosfs:
newfs_msdos /dev/ada0p1
mount -t msdos /dev/ada0p1 /mnt
mkdir /mnt/efi/boot
cp /boot/loader.efi /mnt/efi/boot/BOOTx64.efi
umount /mnt

[Michael Gmelin]

On Wed, 19 May 2021 19:09:06 +0200

Post by Kurt Jaeger
Hi!

Post by Michael Gmelin
Does this mean, re-installing the bootcode isn't necessary anymore
- or has the warning been removed by accident/as a side effect of
merging with OpenZFS?

On the contrary, because of the switch from FreeBSD ZFS to
OpenZFS, the bootcodes needs to be updated! It's unfortunate
that no message is displayed 8-(

That's too bad - maybe it would make sense to mention this in the
release errata?

Post by Kurt Jaeger
The problem is, finding out which bootcode needs to go where etc.

For the machines in question it was a straightforward legacy layout, so
that was easy enough (they came back up just fine - *phew*).

Do you think there is any chance to get the warning back in there?
Maybe in a more generic way, like:

In case you're booting from <poolname>, please make sure to update
the bootcode according to your partition layout. See `man zfsboot'
for details.

Fun fact: That man page already exists (I had no idea), but could use
some love - e.g., add the EFI examples you gave.

Best
Michael

[Graham Perrin]

Post by Michael Gmelin

Post by Kurt Jaeger
bootcodes needs to be updated! It's unfortunate
that no message is displayed 8-(

That's too bad - maybe it would make sense to mention this in the
release errata?

<https://www.freebsd.org/releases/13.0R/relnotes/#upgrade> there's a red
alert box referring to
<https://www.freebsd.org/releases/13.0R/relnotes/#boot>

[Michael Gmelin]

Post by Michael Gmelin

Post by Kurt Jaeger
bootcodes needs to be updated! It's unfortunate
that no message is displayed 8-(

That's too bad - maybe it would make sense to mention this in the
release errata?

<https://www.freebsd.org/releases/13.0R/relnotes/#upgrade> there's a red alert box referring to <https://www.freebsd.org/releases/13.0R/relnotes/#boot>

Given that I upgraded the zpool of a server that boots of a legacy boot partition, how would the red alert box saying

"Updating UEFI ESP partitions (the partition the firmware boots from) has changed. See Boot Loader Changes for important details."

have caught my attention? No UEFI involved at my end, no zpools involved in the warning (I noticed the absence of the warning after running "zpool upgrade", so no harm done, but other users might be less fortunate).

-m

[Graham Perrin]

Post by Michael Gmelin




Given that I upgraded the zpool of a server that boots of a legacy
boot partition, how would the red alert box saying
   "Updating UEFI ESP partitions (the partition the firmware boots
from) has changed. SeeBoot Loader Changes
<https://www.freebsd.org/releases/13.0R/relnotes/#boot>for important
details."
have caught my attention? No UEFI involved at my end, no zpools
involved in the warning (I noticed the absence of the warning after
running "zpool upgrade", so no harm done, but other users might be
less fortunate).
-m

In a lucky situation: a red alert for changes to something other than
the freebsd-boot partition might, obliquely, cause the reader to wonder
about changes to the freebsd-boot partition that are not classified as
changes for release note purposes :-)

That's with a smile, no personal criticism intended, because I am
<https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256024#c8> amongst
the people who misinterpreted part of the release notes.

For anyone who's not following that particular bug (for the command, the
action of upgrading a pool), there's a parallel bug for the FreeBSD
Handbook:

255318 – handbook: Document how to update the bootloader
<https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255318>

Add <https://man.freebsd.org/zpool-upgrade(8)> to the mix.

Eventually we should have a handbook, a manual page and an action that
are reasonably foolproof. Free from fortune, luck, obliqueness and
wonder :-)

[Michael Gmelin]

On Wed, 19 May 2021 19:32:43 +0200

Post by Michael Gmelin
On Wed, 19 May 2021 19:09:06 +0200

Post by Kurt Jaeger
Hi!

Post by Michael Gmelin
Does this mean, re-installing the bootcode isn't necessary anymore
- or has the warning been removed by accident/as a side effect of
merging with OpenZFS?

On the contrary, because of the switch from FreeBSD ZFS to
OpenZFS, the bootcodes needs to be updated! It's unfortunate
that no message is displayed 8-(

That's too bad - maybe it would make sense to mention this in the
release errata?

Post by Kurt Jaeger
The problem is, finding out which bootcode needs to go where etc.

For the machines in question it was a straightforward legacy layout,
so that was easy enough (they came back up just fine - *phew*).
Do you think there is any chance to get the warning back in there?
In case you're booting from <poolname>, please make sure to update
the bootcode according to your partition layout. See `man zfsboot'
for details.
Fun fact: That man page already exists (I had no idea), but could use
some love - e.g., add the EFI examples you gave.
Best
Michael

PR: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256024

-m

[Mathieu Arnold]

Post by Michael Gmelin
On Wed, 19 May 2021 19:09:06 +0200

Post by Kurt Jaeger
Hi!

Post by Michael Gmelin
Does this mean, re-installing the bootcode isn't necessary anymore
- or has the warning been removed by accident/as a side effect of
merging with OpenZFS?

On the contrary, because of the switch from FreeBSD ZFS to
OpenZFS, the bootcodes needs to be updated! It's unfortunate
that no message is displayed 8-(

That's too bad - maybe it would make sense to mention this in the
release errata?

Post by Kurt Jaeger
The problem is, finding out which bootcode needs to go where etc.

For the machines in question it was a straightforward legacy layout, so
that was easy enough (they came back up just fine - *phew*).
Do you think there is any chance to get the warning back in there?
In case you're booting from <poolname>, please make sure to update
the bootcode according to your partition layout. See `man zfsboot'
for details.
Fun fact: That man page already exists (I had no idea), but could use
some love - e.g., add the EFI examples you gave.

I have had a look at that man page, it gives ten different commands,
without saying which you need to use, if they are all needed, or not, or
when you actually have to used them.

Fun fact is that I absolutely never ever used any of them. All I ever
did was run:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

because `zpool upgrade` told me to. It would probably be great to have
that command in that man page.

[Michael Gmelin]

On Fri, 28 May 2021 20:37:14 +0200

Post by Mathieu Arnold

Post by Michael Gmelin
On Wed, 19 May 2021 19:09:06 +0200

Post by Kurt Jaeger
Hi!

Post by Michael Gmelin
Does this mean, re-installing the bootcode isn't necessary anymore
- or has the warning been removed by accident/as a side effect
of merging with OpenZFS?

On the contrary, because of the switch from FreeBSD ZFS to
OpenZFS, the bootcodes needs to be updated! It's unfortunate
that no message is displayed 8-(

That's too bad - maybe it would make sense to mention this in the
release errata?

Post by Kurt Jaeger
The problem is, finding out which bootcode needs to go where etc.

For the machines in question it was a straightforward legacy
layout, so that was easy enough (they came back up just fine -
*phew*).
Do you think there is any chance to get the warning back in there?
In case you're booting from <poolname>, please make sure to update
the bootcode according to your partition layout. See `man zfsboot'
for details.
Fun fact: That man page already exists (I had no idea), but could
use some love - e.g., add the EFI examples you gave.

I have had a look at that man page, it gives ten different commands,
without saying which you need to use, if they are all needed, or not,
or when you actually have to used them.
Fun fact is that I absolutely never ever used any of them. All I ever
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
because `zpool upgrade` told me to. It would probably be great to
have that command in that man page.

Hi Mathieu,

I opened a pull request upstream. It already has been reviewed and
accepted, but is yet to be merged:
https://github.com/openzfs/zfs/pull/12104

The bootcode warning is now only shown for pools that have bootfs set
(which should be fine on most "normal" installations) and reads like
this:

Pool 'testpool8' has the bootfs property set, you might need to update
the boot code. See gptzfsboot(8) and loader.efi(8) for details.

I figured that these are the most relevant man pages for current
systems. It would make sense to update these man pages to be more
helpful (latest, when this change lands in FreeBSD, but earlier
obviously won't hurt).

Best
Michael

[Michael Gmelin]

On Fri, 28 May 2021 20:50:52 +0200

Post by Michael Gmelin
On Fri, 28 May 2021 20:37:14 +0200

Post by Mathieu Arnold

Post by Michael Gmelin
On Wed, 19 May 2021 19:09:06 +0200

Post by Kurt Jaeger
Hi!

Post by Michael Gmelin
Does this mean, re-installing the bootcode isn't necessary anymore
- or has the warning been removed by accident/as a side effect
of merging with OpenZFS?

On the contrary, because of the switch from FreeBSD ZFS to
OpenZFS, the bootcodes needs to be updated! It's unfortunate
that no message is displayed 8-(

That's too bad - maybe it would make sense to mention this in the
release errata?

Post by Kurt Jaeger
The problem is, finding out which bootcode needs to go where etc.

For the machines in question it was a straightforward legacy
layout, so that was easy enough (they came back up just fine -
*phew*).
Do you think there is any chance to get the warning back in there?
In case you're booting from <poolname>, please make sure to
update the bootcode according to your partition layout. See `man
zfsboot' for details.
Fun fact: That man page already exists (I had no idea), but could
use some love - e.g., add the EFI examples you gave.

I have had a look at that man page, it gives ten different commands,
without saying which you need to use, if they are all needed, or
not, or when you actually have to used them.
Fun fact is that I absolutely never ever used any of them. All I
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
because `zpool upgrade` told me to. It would probably be great to
have that command in that man page.

Hi Mathieu,
I opened a pull request upstream. It already has been reviewed and
https://github.com/openzfs/zfs/pull/12104
The bootcode warning is now only shown for pools that have bootfs set
(which should be fine on most "normal" installations) and reads like
Pool 'testpool8' has the bootfs property set, you might need to
update the boot code. See gptzfsboot(8) and loader.efi(8) for details.
I figured that these are the most relevant man pages for current
systems. It would make sense to update these man pages to be more
helpful (latest, when this change lands in FreeBSD, but earlier
obviously won't hurt).

The pull request has been merged upstream:
https://github.com/openzfs/zfs/commit/65d9212aeeb531e9f987bb41a1ee11b526d2cdad

I'll continue tracking the issue (backporting it to 13?) at our end:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256024

Best
Michael

[Warner Losh]

Post by Michael Gmelin
On Fri, 28 May 2021 20:50:52 +0200

Post by Michael Gmelin
On Fri, 28 May 2021 20:37:14 +0200

Post by Mathieu Arnold

Post by Michael Gmelin
On Wed, 19 May 2021 19:09:06 +0200

Post by Kurt Jaeger
Hi!

Post by Michael Gmelin
Does this mean, re-installing the bootcode isn't necessary anymore
- or has the warning been removed by accident/as a side effect
of merging with OpenZFS?

On the contrary, because of the switch from FreeBSD ZFS to
OpenZFS, the bootcodes needs to be updated! It's unfortunate
that no message is displayed 8-(

That's too bad - maybe it would make sense to mention this in the
release errata?

Post by Kurt Jaeger
The problem is, finding out which bootcode needs to go where etc.

For the machines in question it was a straightforward legacy
layout, so that was easy enough (they came back up just fine -
*phew*).
Do you think there is any chance to get the warning back in there?
In case you're booting from <poolname>, please make sure to
update the bootcode according to your partition layout. See `man
zfsboot' for details.
Fun fact: That man page already exists (I had no idea), but could
use some love - e.g., add the EFI examples you gave.

I have had a look at that man page, it gives ten different commands,
without saying which you need to use, if they are all needed, or
not, or when you actually have to used them.
Fun fact is that I absolutely never ever used any of them. All I
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
because `zpool upgrade` told me to. It would probably be great to
have that command in that man page.

Hi Mathieu,
I opened a pull request upstream. It already has been reviewed and
https://github.com/openzfs/zfs/pull/12104
The bootcode warning is now only shown for pools that have bootfs set
(which should be fine on most "normal" installations) and reads like
Pool 'testpool8' has the bootfs property set, you might need to
update the boot code. See gptzfsboot(8) and loader.efi(8) for details.
I figured that these are the most relevant man pages for current
systems. It would make sense to update these man pages to be more
helpful (latest, when this change lands in FreeBSD, but earlier
obviously won't hurt).

https://github.com/openzfs/zfs/commit/65d9212aeeb531e9f987bb41a1ee11b526d2cdad
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256024

Great. To get into 13, it will need to be merged into their stable
branch...

The plan moving forward is to merge what's upstream based on the upstream
branch. The last technical issue with git should be resolved soon to let
that
get unstuck.

Warner

[Dominic Fandrey]

Post by Michael Gmelin
If you boot from pool 'zroot', don't forget to update boot code.
Assuming you use GPT partitioning and da0 is your boot disk
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

ports-mgmt/bsdadminscripts2 comes with a loader updater for GPT partitioned
drives, it has a --dry-run mode to show you what it does:

root# loaderupdate -D nvd0
gpart bootcode -b/boot/pmbr nvd0
gpart bootcode -p/boot/gptzfsboot -i1 nvd0
mkdir -p nvd0p2
mount -tmsdosfs -osync /dev/nvd0p2 nvd0p2
mkdir -p nvd0p2/efi/FreeBSD
cp /boot/loader.efi nvd0p2/efi/FreeBSD/bootamd64.efi
efibootmgr -Bb 0001
efibootmgr -cl nvd0p2:/efi/FreeBSD/bootamd64.efi -L 'FreeBSD 13.0-STABLE #11 stable/13-n245147-ee389afecf85: Fri Apr 9 10:16:57 CEST 2021 amd64 [nvd0p2]'
efibootmgr -ab 0001

[Michael Gmelin]

On Thu, 20 May 2021 10:14:00 +0200

Post by Dominic Fandrey

Post by Michael Gmelin
If you boot from pool 'zroot', don't forget to update boot code.
Assuming you use GPT partitioning and da0 is your boot disk
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

ports-mgmt/bsdadminscripts2 comes with a loader updater for GPT
root# loaderupdate -D nvd0
gpart bootcode -b/boot/pmbr nvd0
gpart bootcode -p/boot/gptzfsboot -i1 nvd0
mkdir -p nvd0p2
mount -tmsdosfs -osync /dev/nvd0p2 nvd0p2
mkdir -p nvd0p2/efi/FreeBSD
cp /boot/loader.efi nvd0p2/efi/FreeBSD/bootamd64.efi
efibootmgr -Bb 0001
efibootmgr -cl nvd0p2:/efi/FreeBSD/bootamd64.efi -L 'FreeBSD
13.0-STABLE #11 stable/13-n245147-ee389afecf85: Fri Apr 9 10:16:57
CEST 2021 amd64 [nvd0p2]' efibootmgr -ab 0001

Thanks, this looks useful, I might take a look at it for future use.

My question wasn't *how* to update the bootloader though (I've been
doing this for almost a decade now), but *if* it was still necessary
after running 'zpool upgrade', given that the warning message that used
to be there disappeared in 13. This has been been answered, thanks
again.

I opened a PR asking to bring a similar warning back in, let's see if
that's easily possible, given that we're using OpenZFS as the upstream
now.

Cheers,
Michael