Discussion:
GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable"
(too old to reply)
Radek Valášek
2009-10-15 12:08:54 UTC
Permalink
Hi,

I want to ask if there is something new in adding support to
gptzfsboot/zfsboot for reading gang-blocks?

From Sun's docs:

Gang blocks

When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.

Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ boot:
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
/>/ boot:
//
/I presume it's the same issue as talked in june-2009 current mailing
list
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html

Any success in that matter?

Thnx for answer.

vaLin
Robert Noland
2009-10-15 13:14:25 UTC
Permalink
This post might be inappropriate. Click to display it.
Robert Noland
2009-10-15 19:03:50 UTC
Permalink
Post by Radek Valášek
Hi,
I want to ask if there is something new in adding support to
gptzfsboot/zfsboot for reading gang-blocks?
Ok, I can't figure out any way to test this... beyond the fact that it
builds and doesn't break my currently working setup. Can you give this
a try? It should still report if it finds gang blocks, but hopefully
now will read them as well.

robert.
Post by Radek Valášek
Gang blocks
When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.
Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
//
/I presume it's the same issue as talked in june-2009 current mailing
list
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
Any success in that matter?
Thnx for answer.
vaLin
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Radek Valášek
2009-10-15 19:37:32 UTC
Permalink
Post by Robert Noland
Post by Radek Valášek
Hi,
I want to ask if there is something new in adding support to
gptzfsboot/zfsboot for reading gang-blocks?
Ok, I can't figure out any way to test this... beyond the fact that it
builds and doesn't break my currently working setup. Can you give this
a try? It should still report if it finds gang blocks, but hopefully
now will read them as well.
robert.
Big thanks for the patches Robert, I will definitely test them as soon
as possible (tomorrow) and report the results immediately to list. I can
repeat this issue probably at any time (up to cca 30 times tested with
the same result), so don't bother about the broken booting, I'm prepared
for it...

vaLin
Post by Robert Noland
Post by Radek Valášek
Gang blocks
When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.
Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
//
/I presume it's the same issue as talked in june-2009 current mailing
list
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
Any success in that matter?
Thnx for answer.
vaLin
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
Robert Noland
2009-10-21 17:38:29 UTC
Permalink
Post by Radek Valášek
Post by Robert Noland
Post by Radek Valášek
Hi,
I want to ask if there is something new in adding support to
gptzfsboot/zfsboot for reading gang-blocks?
I think that the gang block patch will work, though still haven't gotten
it tested. However, I'm fairly confident that the issue is not gang
block related. Right now, I have setup a disk like this:

=> 34 1953525101 ada1 GPT (932G)
34 128 1 freebsd-boot (64K)
162 8388608 2 freebsd-swap (4.0G)
8388770 648019968 3 freebsd-zfs (309G)
656408738 648019968 4 freebsd-zfs (309G)
1304428706 648019968 5 freebsd-zfs (309G)
1952448674 1076461 - free - (526M)

Note that this is not a raidz pool right now. It is just 3 toplevel
partitions setup as a single pool. I finally have this configuration
working reliably. At least in this case, the issue is due to all of the
partitions not being probed during early boot and so not being added to
the list of vdevs for the pool. When zio_read finds a dva that points
to a device it doesn't know about, it gives up and whines.

Can you detail for me how you have everything configured, so that I can
try to replicate it. gpart show, zpool status and zpool get all <pool>
would be good. I'm not sure that I have enough spare disks lying around
to do this properly, but maybe I can use virtual disks or something.

robert.
Post by Radek Valášek
Post by Robert Noland
Ok, I can't figure out any way to test this... beyond the fact that it
builds and doesn't break my currently working setup. Can you give this
a try? It should still report if it finds gang blocks, but hopefully
now will read them as well.
robert.
Big thanks for the patches Robert, I will definitely test them as soon
as possible (tomorrow) and report the results immediately to list. I can
repeat this issue probably at any time (up to cca 30 times tested with
the same result), so don't bother about the broken booting, I'm prepared
for it...
vaLin
Post by Robert Noland
Post by Radek Valášek
Gang blocks
When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.
Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
//
/I presume it's the same issue as talked in june-2009 current mailing
list
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
Any success in that matter?
Thnx for answer.
vaLin
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Radek Valášek
2009-10-24 17:44:25 UTC
Permalink
Post by Robert Noland
Post by Radek Valášek
Post by Robert Noland
Post by Radek Valášek
Hi,
I want to ask if there is something new in adding support to
gptzfsboot/zfsboot for reading gang-blocks?
I think that the gang block patch will work, though still haven't gotten
it tested. However, I'm fairly confident that the issue is not gang
=> 34 1953525101 ada1 GPT (932G)
34 128 1 freebsd-boot (64K)
162 8388608 2 freebsd-swap (4.0G)
8388770 648019968 3 freebsd-zfs (309G)
656408738 648019968 4 freebsd-zfs (309G)
1304428706 648019968 5 freebsd-zfs (309G)
1952448674 1076461 - free - (526M)
Note that this is not a raidz pool right now. It is just 3 toplevel
partitions setup as a single pool. I finally have this configuration
working reliably. At least in this case, the issue is due to all of the
partitions not being probed during early boot and so not being added to
the list of vdevs for the pool. When zio_read finds a dva that points
to a device it doesn't know about, it gives up and whines.
Can you detail for me how you have everything configured, so that I can
try to replicate it. gpart show, zpool status and zpool get all <pool>
would be good. I'm not sure that I have enough spare disks lying around
to do this properly, but maybe I can use virtual disks or something.
robert.
Sorry for not responding so long. Here are details you want from me:

# gpart show
=> 34 1953525101 ad6 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)

=> 34 1953525101 ad8 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)

=> 34 1953525101 ad10 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)

=> 34 1953525101 ad12 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)

# zpool status
pool: z
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
z ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad6p2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad10p2 ONLINE 0 0 0
ad12p2 ONLINE 0 0 0

errors: No known data errors

# zpool get all z
NAME PROPERTY VALUE SOURCE
z size 3.62T -
z used 4.62G -
z available 3.62T -
z capacity 0% -
z altroot - default
z health ONLINE -
z guid 17857007133862981114 -
z version 13 default
z bootfs z/system local
z delegation on default
z autoreplace off default
z cachefile - default
z failmode wait default
z listsnapshots off default

I've tested your patches but it seems that you're right and it's not
gang related issue. I was able to discover these things on a fully
functional zfs pool (system compiled with your patches):

1, If I overwrite the file /boot/loader.conf (with copy of itself, or
when upgrading kernel/world), next reboot comes with these messages:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(***@ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

Then I'm still able to boot the system, but I must set the boot
variables included in loader.conf by hand

2, Next I overwrite the file /boot/loader (with copy of itself, or when
upgrading kernel/world) and reboot comes with these messages:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(***@ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
Unable to load a kernel!

After that I'm no longer able to boot the system from zfs pool.

Hope you have some ideas...

vaLin
Post by Robert Noland
Post by Radek Valášek
Post by Robert Noland
Ok, I can't figure out any way to test this... beyond the fact that it
builds and doesn't break my currently working setup. Can you give this
a try? It should still report if it finds gang blocks, but hopefully
now will read them as well.
robert.
Big thanks for the patches Robert, I will definitely test them as soon
as possible (tomorrow) and report the results immediately to list. I can
repeat this issue probably at any time (up to cca 30 times tested with
the same result), so don't bother about the broken booting, I'm prepared
for it...
vaLin
Post by Robert Noland
Post by Radek Valášek
Gang blocks
When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.
Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
//
/I presume it's the same issue as talked in june-2009 current mailing
list
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
Any success in that matter?
Thnx for answer.
vaLin
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
Robert Noland
2009-10-25 03:45:52 UTC
Permalink
Post by Radek Valášek
Post by Robert Noland
Post by Radek Valášek
Post by Robert Noland
Post by Radek Valášek
Hi,
I want to ask if there is something new in adding support to
gptzfsboot/zfsboot for reading gang-blocks?
I think that the gang block patch will work, though still haven't gotten
it tested. However, I'm fairly confident that the issue is not gang
=> 34 1953525101 ada1 GPT (932G)
34 128 1 freebsd-boot (64K)
162 8388608 2 freebsd-swap (4.0G)
8388770 648019968 3 freebsd-zfs (309G)
656408738 648019968 4 freebsd-zfs (309G)
1304428706 648019968 5 freebsd-zfs (309G)
1952448674 1076461 - free - (526M)
Note that this is not a raidz pool right now. It is just 3 toplevel
partitions setup as a single pool. I finally have this configuration
working reliably. At least in this case, the issue is due to all of the
partitions not being probed during early boot and so not being added to
the list of vdevs for the pool. When zio_read finds a dva that points
to a device it doesn't know about, it gives up and whines.
Can you detail for me how you have everything configured, so that I can
try to replicate it. gpart show, zpool status and zpool get all <pool>
would be good. I'm not sure that I have enough spare disks lying around
to do this properly, but maybe I can use virtual disks or something.
robert.
# gpart show
=> 34 1953525101 ad6 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
=> 34 1953525101 ad8 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
=> 34 1953525101 ad10 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
=> 34 1953525101 ad12 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
# zpool status
pool: z
state: ONLINE
scrub: none requested
NAME STATE READ WRITE CKSUM
z ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad6p2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad10p2 ONLINE 0 0 0
ad12p2 ONLINE 0 0 0
errors: No known data errors
# zpool get all z
NAME PROPERTY VALUE SOURCE
z size 3.62T -
z used 4.62G -
z available 3.62T -
z capacity 0% -
z altroot - default
z health ONLINE -
z guid 17857007133862981114 -
z version 13 default
z bootfs z/system local
z delegation on default
z autoreplace off default
z cachefile - default
z failmode wait default
z listsnapshots off default
I've tested your patches but it seems that you're right and it's not
gang related issue. I was able to discover these things on a fully
1, If I overwrite the file /boot/loader.conf (with copy of itself, or
BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory
FreeBSD/i386 bootstrap loader, Revision 1.1
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
Then I'm still able to boot the system, but I must set the boot
variables included in loader.conf by hand
2, Next I overwrite the file /boot/loader (with copy of itself, or when
BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory
FreeBSD/i386 bootstrap loader, Revision 1.1
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
Unable to load a kernel!
After that I'm no longer able to boot the system from zfs pool.
Hope you have some ideas...
Ok, can you retest with -CURRENT? I just committed some fixes on
Friday. I'm having real difficulty in reproducing these issues. Most
of the problems that I've run into so far had to do with the system not
knowing about all of the vdevs when it wanted to read something. In
your case, it looks like you are making it to boot3 and it appears to be
seeing all 4 of your disks. Right now, I've been trying to track down
an issue wher the MOS can't be read, which basically means that we have
screwed up the root block pointer somehow. I haven't been able to
reproduce that issue in qemu, I have been able to reproduce it with
VirtualBox, but it is really time consuming trying to work in vbox since
I have to reconvert all of the disk images every time I make a change.
I'm actually a bit concerned that it hinges on how many drives are
visible to the bios at various points in time.

robert.
Post by Radek Valášek
vaLin
Post by Robert Noland
Post by Radek Valášek
Post by Robert Noland
Ok, I can't figure out any way to test this... beyond the fact that it
builds and doesn't break my currently working setup. Can you give this
a try? It should still report if it finds gang blocks, but hopefully
now will read them as well.
robert.
Big thanks for the patches Robert, I will definitely test them as soon
as possible (tomorrow) and report the results immediately to list. I can
repeat this issue probably at any time (up to cca 30 times tested with
the same result), so don't bother about the broken booting, I'm prepared
for it...
vaLin
Post by Robert Noland
Post by Radek Valášek
Gang blocks
When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.
Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
//
/I presume it's the same issue as talked in june-2009 current mailing
list
http://lists.freebsd.org/pipermail/freebsd-current/2009-June/008589.html
Any success in that matter?
Thnx for answer.
vaLin
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Merijn Verstraaten
2009-10-25 23:45:44 UTC
Permalink
Post by Robert Noland
Ok, can you retest with -CURRENT? I just committed some fixes on
Friday. I'm having real difficulty in reproducing these issues. Most
of the problems that I've run into so far had to do with the system not
knowing about all of the vdevs when it wanted to read something. In
your case, it looks like you are making it to boot3 and it appears to be
seeing all 4 of your disks. Right now, I've been trying to track down
an issue wher the MOS can't be read, which basically means that we have
screwed up the root block pointer somehow. I haven't been able to
reproduce that issue in qemu, I have been able to reproduce it with
VirtualBox, but it is really time consuming trying to work in vbox since
I have to reconvert all of the disk images every time I make a change.
I'm actually a bit concerned that it hinges on how many drives are
visible to the bios at various points in time.
robert.
I noticed this thread in the archives and I've been having the same
problems as Radek (see belw) with root on ZFS RAID-Z not booting the
machine. When I install world from the 8.0-RC1 USB image I can boot from
RAID-Z with no problems. But when I csup'ed to RELENG_8 yesterday
(Saturday, so this should be after you committed the new changes) and
build+installed world booting from RAID-Z broke with the exact same
problems as Radek describes below. I just tried again to be absolutely
sure (It just occured to me I perhaps should have done "gpart show",
"zpool status" and "zpool get all z" before breaking the system, but oh
well...).

I have 4 SATA disks ad4, ad6, ad8 and ad10 with three partitions each,
adXp1 is a freebsd-boot partition like Radek has, adXp2 is a 1GB
freebsd-swap partition and adXp3 has the rest of the disk as a freebsd-zfs
partition. These 4 p3 partitions are placed in a single RAID-Z vdev. zpool
status looks pretty much like that of Radek too. I have my root partition
in tank/root and mount things like tank/usr and tank/var inside tank/root.

After installing 8.0-RC1 (amd64) from USB stick this installation works
fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
booting from the ZFS fails. The initial installation I did just this, on
another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
adX" on all disks before rebooting to see if that had any effect. The end
result is the same. After rebooting the machine I get the following
prompt(s):

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
boot:

If I wait a while this prompt repeats itself. I've beent rying to test
inside VirtualBox to see if I could report more detail on what's going
wrong but I ran into the same problem as you with the "MOS can't be read"
error.

This machine and zpool don't (at this time) contain any valuable data, so
if you need someone to test anything for you I'd gladly volunteer my box
if it helps resolve things faster.

Kind regards,
Merijn Verstraaten
Post by Robert Noland
Post by Robert Noland
Hi,
I want to ask if there is something new in adding support to>>>>
gptzfsboot/zfsboot for reading gang-blocks?
Post by Robert Noland
I think that the gang block patch will work, though still haven't
gotten
Post by Robert Noland
it tested. However, I'm fairly confident that the issue is not gang
=> 34 1953525101 ada1 GPT (932G)
34 128 1 freebsd-boot (64K)
162 8388608 2 freebsd-swap (4.0G)
8388770 648019968 3 freebsd-zfs (309G)
656408738 648019968 4 freebsd-zfs (309G)
1304428706 648019968 5 freebsd-zfs (309G)
1952448674 1076461 - free - (526M)
Note that this is not a raidz pool right now. It is just 3 toplevel
partitions setup as a single pool. I finally have this configuration
working reliably. At least in this case, the issue is due to all of
the
Post by Robert Noland
partitions not being probed during early boot and so not being added to
the list of vdevs for the pool. When zio_read finds a dva that points
to a device it doesn't know about, it gives up and whines.
Can you detail for me how you have everything configured, so that I can
try to replicate it. gpart show, zpool status and zpool get all <pool>
would be good. I'm not sure that I have enough spare disks lying
around
Post by Robert Noland
to do this properly, but maybe I can use virtual disks or something.
robert.
# gpart show
=> 34 1953525101 ad6 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
=> 34 1953525101 ad8 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
=> 34 1953525101 ad10 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
=> 34 1953525101 ad12 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
# zpool status
pool: z
state: ONLINE
scrub: none requested
NAME STATE READ WRITE CKSUM
z ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad6p2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad10p2 ONLINE 0 0 0
ad12p2 ONLINE 0 0 0
errors: No known data errors
# zpool get all z
NAME PROPERTY VALUE SOURCE
z size 3.62T -
z used 4.62G -
z available 3.62T -
z capacity 0% -
z altroot - default
z health ONLINE -
z guid 17857007133862981114 -
z version 13 default
z bootfs z/system local
z delegation on default
z autoreplace off default
z cachefile - default
z failmode wait default
z listsnapshots off default
I've tested your patches but it seems that you're right and it's notgang
related issue. I was able to discover these things on a fullyfunctional
1, If I overwrite the file /boot/loader.conf (with copy of itself, or
BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory
FreeBSD/i386 bootstrap loader, Revision 1.1
(root at ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
Then I'm still able to boot the system, but I must set the bootvariables
included in loader.conf by hand
2, Next I overwrite the file /boot/loader (with copy of itself, or when
BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory
FreeBSD/i386 bootstrap loader, Revision 1.1
(root at ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
Unable to load a kernel!
After that I'm no longer able to boot the system from zfs pool.
Hope you have some ideas...
Robert Noland
2009-10-26 00:31:46 UTC
Permalink
Post by Merijn Verstraaten
Post by Robert Noland
Ok, can you retest with -CURRENT? I just committed some fixes on
Friday. I'm having real difficulty in reproducing these issues. Most
of the problems that I've run into so far had to do with the system not
knowing about all of the vdevs when it wanted to read something. In
your case, it looks like you are making it to boot3 and it appears to be
seeing all 4 of your disks. Right now, I've been trying to track down
an issue wher the MOS can't be read, which basically means that we have
screwed up the root block pointer somehow. I haven't been able to
reproduce that issue in qemu, I have been able to reproduce it with
VirtualBox, but it is really time consuming trying to work in vbox since
I have to reconvert all of the disk images every time I make a change.
I'm actually a bit concerned that it hinges on how many drives are
visible to the bios at various points in time.
robert.
I noticed this thread in the archives and I've been having the same
problems as Radek (see belw) with root on ZFS RAID-Z not booting the
machine. When I install world from the 8.0-RC1 USB image I can boot from
RAID-Z with no problems. But when I csup'ed to RELENG_8 yesterday
(Saturday, so this should be after you committed the new changes) and
build+installed world booting from RAID-Z broke with the exact same
problems as Radek describes below. I just tried again to be absolutely
sure (It just occured to me I perhaps should have done "gpart show",
"zpool status" and "zpool get all z" before breaking the system, but oh
well...).
I have 4 SATA disks ad4, ad6, ad8 and ad10 with three partitions each,
adXp1 is a freebsd-boot partition like Radek has, adXp2 is a 1GB
freebsd-swap partition and adXp3 has the rest of the disk as a freebsd-zfs
partition. These 4 p3 partitions are placed in a single RAID-Z vdev. zpool
status looks pretty much like that of Radek too. I have my root partition
in tank/root and mount things like tank/usr and tank/var inside tank/root.
After installing 8.0-RC1 (amd64) from USB stick this installation works
fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
booting from the ZFS fails. The initial installation I did just this, on
another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
adX" on all disks before rebooting to see if that had any effect. The end
result is the same. After rebooting the machine I get the following
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
Could you type "status" at this point and tell what it shows?

I'm convinced that almost all of these issues have to do with what
drives are visible at any given time. We actually probe for all the
drives twice, once during the early boot (gptzfsboot) and again in stage
3... They don't find the same set of drives in at least some cases. In
order to successfully boot, we need to see all of the drives at both
stages. The more drives that you have the more likely it is that we run
into issues.
Post by Merijn Verstraaten
If I wait a while this prompt repeats itself. I've beent rying to test
inside VirtualBox to see if I could report more detail on what's going
wrong but I ran into the same problem as you with the "MOS can't be read"
error.
So, something about the virtual box bios and our loader, don't get along
well. I finally have it booting from a 6 drive raidz2, but what drives
are detected during the early boot depend on what controllers I've told
vbox to use to host the drives. Basically in order to get it to work, I
had to tell it that the first two drives are on ide and the remaining 4
drives are on sata/scsi. Once we get to stage 3, the current code only
finds a single hard drive, so at best you have a 25% chance of
booting... As the blocks get spread around on the drives, we start
reading block pointers that point to devices that we don't know about.
My current hacked up code is now seeing 30 drives in stage 3, which is
also not correct... but it is booting...

robert.
Post by Merijn Verstraaten
This machine and zpool don't (at this time) contain any valuable data, so
if you need someone to test anything for you I'd gladly volunteer my box
if it helps resolve things faster.
Kind regards,
Merijn Verstraaten
Post by Robert Noland
Post by Robert Noland
Hi,
I want to ask if there is something new in adding support to>>>>
gptzfsboot/zfsboot for reading gang-blocks?
Post by Robert Noland
I think that the gang block patch will work, though still haven't
gotten
Post by Robert Noland
it tested. However, I'm fairly confident that the issue is not gang
=> 34 1953525101 ada1 GPT (932G)
34 128 1 freebsd-boot (64K)
162 8388608 2 freebsd-swap (4.0G)
8388770 648019968 3 freebsd-zfs (309G)
656408738 648019968 4 freebsd-zfs (309G)
1304428706 648019968 5 freebsd-zfs (309G)
1952448674 1076461 - free - (526M)
Note that this is not a raidz pool right now. It is just 3 toplevel
partitions setup as a single pool. I finally have this configuration
working reliably. At least in this case, the issue is due to all of
the
Post by Robert Noland
partitions not being probed during early boot and so not being added to
the list of vdevs for the pool. When zio_read finds a dva that points
to a device it doesn't know about, it gives up and whines.
Can you detail for me how you have everything configured, so that I can
try to replicate it. gpart show, zpool status and zpool get all <pool>
would be good. I'm not sure that I have enough spare disks lying
around
Post by Robert Noland
to do this properly, but maybe I can use virtual disks or something.
robert.
# gpart show
=> 34 1953525101 ad6 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
=> 34 1953525101 ad8 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
=> 34 1953525101 ad10 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
=> 34 1953525101 ad12 GPT (932G)
34 128 1 freebsd-boot (64K)
162 1953524973 2 freebsd-zfs (932G)
# zpool status
pool: z
state: ONLINE
scrub: none requested
NAME STATE READ WRITE CKSUM
z ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ad6p2 ONLINE 0 0 0
ad8p2 ONLINE 0 0 0
ad10p2 ONLINE 0 0 0
ad12p2 ONLINE 0 0 0
errors: No known data errors
# zpool get all z
NAME PROPERTY VALUE SOURCE
z size 3.62T -
z used 4.62G -
z available 3.62T -
z capacity 0% -
z altroot - default
z health ONLINE -
z guid 17857007133862981114 -
z version 13 default
z bootfs z/system local
z delegation on default
z autoreplace off default
z cachefile - default
z failmode wait default
z listsnapshots off default
I've tested your patches but it seems that you're right and it's notgang
related issue. I was able to discover these things on a fullyfunctional
1, If I overwrite the file /boot/loader.conf (with copy of itself, or
BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory
FreeBSD/i386 bootstrap loader, Revision 1.1
(root at ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
Then I'm still able to boot the system, but I must set the bootvariables
included in loader.conf by hand
2, Next I overwrite the file /boot/loader (with copy of itself, or when
BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory
FreeBSD/i386 bootstrap loader, Revision 1.1
(root at ztest, Thu Oct 22 22:27:22 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
ZFS: i/o error - all block copies unavailable
Unable to load a kernel!
After that I'm no longer able to boot the system from zfs pool.
Hope you have some ideas...
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Merijn Verstraaten
2009-10-26 09:23:36 UTC
Permalink
Post by Robert Noland
Post by Merijn Verstraaten
After installing 8.0-RC1 (amd64) from USB stick this installation works
fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
booting from the ZFS fails. The initial installation I did just this, on
another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
adX" on all disks before rebooting to see if that had any effect. The end
result is the same. After rebooting the machine I get the following
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
Could you type "status" at this point and tell what it shows?
If I type status at this point I get:

pool: tank
config:
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE

Which seems odd, since that's all the drives there are. So if it finds
these it's already found all drives. My optimistic "Oh! I'll try and boot
again" spirit was however crushed since it just results in the same error.

Kind regards,
Merijn
Robert Noland
2009-10-26 15:34:59 UTC
Permalink
Post by Merijn Verstraaten
Post by Robert Noland
Post by Merijn Verstraaten
After installing 8.0-RC1 (amd64) from USB stick this installation works
fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
booting from the ZFS fails. The initial installation I did just this, on
another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
adX" on all disks before rebooting to see if that had any effect. The end
result is the same. After rebooting the machine I get the following
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
Could you type "status" at this point and tell what it shows?
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Which seems odd, since that's all the drives there are. So if it finds
these it's already found all drives. My optimistic "Oh! I'll try and boot
again" spirit was however crushed since it just results in the same error.
Ok, that is both good and frustrating... I haven't produced any boot
failures with all of the drives visible. Do, note that I just added
support for reading gang blocks to the loader. (basically untested,
since I haven't managed to create them at will) You will need to update
your partition boot code for it to be supported during early boot. i.e.
gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>

The "all block copies unavailable" is a frustrating error, since all it
means is a failed read, but we don't get a clue what failed or why.
With the code that is in -CURRENT it will report gang blocks if found,
even if it fails to read them.

robert.
Post by Merijn Verstraaten
Kind regards,
Merijn
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Radek Valášek
2009-10-26 19:51:16 UTC
Permalink
Post by Robert Noland
Post by Merijn Verstraaten
Post by Robert Noland
Post by Merijn Verstraaten
After installing 8.0-RC1 (amd64) from USB stick this installation works
fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
booting from the ZFS fails. The initial installation I did just this, on
another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
adX" on all disks before rebooting to see if that had any effect. The end
result is the same. After rebooting the machine I get the following
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
Could you type "status" at this point and tell what it shows?
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Which seems odd, since that's all the drives there are. So if it finds
these it's already found all drives. My optimistic "Oh! I'll try and boot
again" spirit was however crushed since it just results in the same error.
Ok, that is both good and frustrating... I haven't produced any boot
failures with all of the drives visible. Do, note that I just added
support for reading gang blocks to the loader. (basically untested,
since I haven't managed to create them at will) You will need to update
your partition boot code for it to be supported during early boot. i.e.
gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
The "all block copies unavailable" is a frustrating error, since all it
means is a failed read, but we don't get a clue what failed or why.
With the code that is in -CURRENT it will report gang blocks if found,
even if it fails to read them.
robert.
So I switched to -CURRENT:

1, overwriting /boot/loader.conf results with:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(***@ztest, Mon Oct 26 14:01:44 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

so basically the same as in RELENG_8

2, + overwriting /boot/loader results with:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot:
\
int=00000001 err=00000000 efl=00000087 eip=0018d27d
eax=0018d2af ebx=18bf9925 ecx=540d8ef2 edx=00000000
esi=00009401 edi=000919d0 ebp=36571125 esp=80000000
cs=0008 ds=0010 es=0010 fs=0010 gs=0010 ss=0010
cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
BTX halted

3, I also try the 'status' as you told to Merijn before BTX halted:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot: status pool: z
config:
NAME STATE
z ONLINE
raidz1 ONLINE
ad6p2 ONLINE
ad8p2 ONLINE
ad10p2 ONLINE
ad12p2 ONLINE

radek.
Post by Robert Noland
Post by Merijn Verstraaten
Kind regards,
Merijn
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
Radek Valášek
2009-10-27 10:15:18 UTC
Permalink
Post by Robert Noland
Post by Merijn Verstraaten
Post by Robert Noland
Post by Merijn Verstraaten
After installing 8.0-RC1 (amd64) from USB stick this installation works
fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
booting from the ZFS fails. The initial installation I did just this, on
another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
adX" on all disks before rebooting to see if that had any effect. The end
result is the same. After rebooting the machine I get the following
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
Could you type "status" at this point and tell what it shows?
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Which seems odd, since that's all the drives there are. So if it finds
these it's already found all drives. My optimistic "Oh! I'll try and boot
again" spirit was however crushed since it just results in the same error.
Ok, that is both good and frustrating... I haven't produced any boot
failures with all of the drives visible. Do, note that I just added
support for reading gang blocks to the loader. (basically untested,
since I haven't managed to create them at will) You will need to update
your partition boot code for it to be supported during early boot. i.e.
gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
The "all block copies unavailable" is a frustrating error, since all it
means is a failed read, but we don't get a clue what failed or why.
With the code that is in -CURRENT it will report gang blocks if found,
even if it fails to read them.
robert.
So I switched to -CURRENT:

1, overwriting /boot/loader.conf results with:

BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory

FreeBSD/i386 bootstrap loader, Revision 1.1
(***@ztest, Mon Oct 26 14:01:44 CEST 2009)
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf

so basically the same as in RELENG_8

2, + overwriting /boot/loader results with:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot:
\
int=00000001 err=00000000 efl=00000087 eip=0018d27d
eax=0018d2af ebx=18bf9925 ecx=540d8ef2 edx=00000000
esi=00009401 edi=000919d0 ebp=36571125 esp=80000000
cs=0008 ds=0010 es=0010 fs=0010 gs=0010 ss=0010
cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
BTX halted

3, I also try the 'status' as you told to Merijn before BTX halted:

ZFS: i/o error - all block copies unavailable
Invalid format

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot: status pool: z
config:
NAME STATE
z ONLINE
raidz1 ONLINE
ad6p2 ONLINE
ad8p2 ONLINE
ad10p2 ONLINE
ad12p2 ONLINE

radek.
Post by Robert Noland
Post by Merijn Verstraaten
Kind regards,
Merijn
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
kickbsd kickbsd
2009-10-28 20:49:56 UTC
Permalink
Hi!

I've just tried to follow
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ2
with RC2
and got the same error as below.
Post by Radek Valášek
Post by Robert Noland
On Mon, 26 Oct 2009 01:31:46 +0100, Robert Noland
Post by Robert Noland
Post by Merijn Verstraaten
After installing 8.0-RC1 (amd64) from USB stick this installation works
fine. If I csup to RELENG_8 (amd64) and compile + install world and kernel
booting from the ZFS fails. The initial installation I did just this, on
another attempt I ran "gpart bootcode -b /boot/pbmr -p /boot/gptzfsboot
adX" on all disks before rebooting to see if that had any effect. The end
result is the same. After rebooting the machine I get the following
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
Could you type "status" at this point and tell what it shows?
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Which seems odd, since that's all the drives there are. So if it finds
these it's already found all drives. My optimistic "Oh! I'll try and boot
again" spirit was however crushed since it just results in the same error.
Ok, that is both good and frustrating... I haven't produced any boot
failures with all of the drives visible. Do, note that I just added
support for reading gang blocks to the loader. (basically untested,
since I haven't managed to create them at will) You will need to update
your partition boot code for it to be supported during early boot. i.e.
gpart bootcode -p /boot/gptzfsboot -i
The "all block copies unavailable" is a frustrating error, since all it
means is a failed read, but we don't get a clue what failed or why.
With the code that is in -CURRENT it will report gang blocks if found,
even if it fails to read them.
robert.
BTX loader 1.00 BTX version is 1.02
Consoles: internal video/keyboard
BIOS drive C: is disk0
BIOS drive D: is disk1
BIOS drive E: is disk2
BIOS drive F: is disk3
BIOS 627kB/3405248kB available memory
FreeBSD/i386 bootstrap loader, Revision 1.1
Loading /boot/defaults/loader.conf
ZFS: i/o error - all block copies unavailable
Warning: error reading file /boot/loader.conf
so basically the same as in RELENG_8
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
\
int=00000001 err=00000000 efl=00000087 eip=0018d27d
eax=0018d2af ebx=18bf9925 ecx=540d8ef2 edx=00000000
esi=00009401 edi=000919d0 ebp=36571125 esp=80000000
cs=0008 ds=0010 es=0010 fs=0010 gs=0010 ss=0010
cs:eip=1f 68 e2 c6 7d 75 0c 5d-45 58 c7 80 f5 99 bd 9e
fe 68 2d 3e 3c 35 5e 67-61 12 fe 50 c9 0b e4 70
ss:esp=00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00
BTX halted
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot: status pool: z
NAME STATE
z ONLINE
raidz1 ONLINE
ad6p2 ONLINE
ad8p2 ONLINE
ad10p2 ONLINE
ad12p2 ONLINE
radek.
Post by Robert Noland
Kind regards,
Merijn
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
--
Находчивая почта находится здесь: http://mail.yandex.ru/promo/new/search
Merijn Verstraaten
2009-10-27 10:17:40 UTC
Permalink
My apologies to Robert for the double message, mis-addressed the first
reply to him instead of the list.
Post by Robert Noland
Post by Merijn Verstraaten
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Which seems odd, since that's all the drives there are. So if it finds
these it's already found all drives. My optimistic "Oh! I'll try and boot
again" spirit was however crushed since it just results in the same error.
Ok, that is both good and frustrating... I haven't produced any boot
failures with all of the drives visible. Do, note that I just added
support for reading gang blocks to the loader. (basically untested,
since I haven't managed to create them at will) You will need to update
your partition boot code for it to be supported during early boot. i.e.
gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
I tried again yesterday evening by recompiling RELENG_8 and -CURRENT. I
somehow managed to boot into RELENG_8 the first time, but after that this
error comes up:

ZFS: i/o error - all block copies unavailable
ZFS: can't read object set for dataset ldd
Can't find root filesystem - giving up
ZFS: unexpected object set type ldd
ZFS: unexpected object set type ldd

FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
boot:
ZFS: unexpected object set type ldd

FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
boot: status

pool: tank
config:
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE

After recompiling world/kernel for -CURRENT I get roughly the same error:

ZFS: i/o error - all block copies unavailable
ZFS: can't read object set for dataset 30
Can't find root filesystem - giving up
ZFS: unexpected object set type 0
ZFS: unexpected object set type 0

FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
boot:
ZFS: unexpected object set type ldd

FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
boot: status

pool: tank
config:
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Post by Robert Noland
The "all block copies unavailable" is a frustrating error, since all it
means is a failed read, but we don't get a clue what failed or why.
With the code that is in -CURRENT it will report gang blocks if found,
even if it fails to read them.
robert.
I've seen no mention of gang blocks in the errors so far.

Kind regards,
Merijn
Robert Noland
2009-10-27 14:49:36 UTC
Permalink
Post by Merijn Verstraaten
My apologies to Robert for the double message, mis-addressed the first
reply to him instead of the list.
Post by Robert Noland
Post by Merijn Verstraaten
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Which seems odd, since that's all the drives there are. So if it finds
these it's already found all drives. My optimistic "Oh! I'll try and boot
again" spirit was however crushed since it just results in the same error.
Ok, that is both good and frustrating... I haven't produced any boot
failures with all of the drives visible. Do, note that I just added
support for reading gang blocks to the loader. (basically untested,
since I haven't managed to create them at will) You will need to update
your partition boot code for it to be supported during early boot. i.e.
gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
I tried again yesterday evening by recompiling RELENG_8 and -CURRENT. I
somehow managed to boot into RELENG_8 the first time, but after that this
ZFS: i/o error - all block copies unavailable
ZFS: can't read object set for dataset ldd
Can't find root filesystem - giving up
ZFS: unexpected object set type ldd
ZFS: unexpected object set type ldd
FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
ZFS: unexpected object set type ldd
FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
boot: status
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
ZFS: i/o error - all block copies unavailable
ZFS: can't read object set for dataset 30
Can't find root filesystem - giving up
ZFS: unexpected object set type 0
ZFS: unexpected object set type 0
FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
^^
This looks strange... Do you have bootfs set to /, or something in
loader.conf? Does it work if you just type "tank:/boot/kernel/kernel"
at this point?

robert.
Post by Merijn Verstraaten
ZFS: unexpected object set type ldd
FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
boot: status
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Post by Robert Noland
The "all block copies unavailable" is a frustrating error, since all it
means is a failed read, but we don't get a clue what failed or why.
With the code that is in -CURRENT it will report gang blocks if found,
even if it fails to read them.
robert.
I've seen no mention of gang blocks in the errors so far.
Kind regards,
Merijn
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Merijn Verstraaten
2009-10-27 15:02:58 UTC
Permalink
Post by Robert Noland
Post by Merijn Verstraaten
I tried again yesterday evening by recompiling RELENG_8 and -CURRENT. I
somehow managed to boot into RELENG_8 the first time, but after that this
ZFS: i/o error - all block copies unavailable
ZFS: can't read object set for dataset ldd
Can't find root filesystem - giving up
ZFS: unexpected object set type ldd
ZFS: unexpected object set type ldd
FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
ZFS: unexpected object set type ldd
FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
boot: status
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
ZFS: i/o error - all block copies unavailable
ZFS: can't read object set for dataset 30
Can't find root filesystem - giving up
ZFS: unexpected object set type 0
ZFS: unexpected object set type 0
FreeBSD/i386 boot
Default:/ tank:/boot/kernel/kernel
^^
This looks strange... Do you have bootfs set to /, or something in
loader.conf? Does it work if you just type "tank:/boot/kernel/kernel"
at this point?
robert.
I think this might be user error. I just checked and the leading / is
absent on my screen:

ZFS: i/o error - all block copies unavailable
ZFS: can't read object set for dataset 30
Can't find root filesystem - giving up
ZFS: unexpected object set type 0
ZFS: unexpected object set type 0

FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
boot:

I probably just typo'ed it this morning. As clarification I have
vfs.root.mountfrom="zfs:tank/root" in loader.conf with my root filesystem
installed on tank/root and then tank/usr, tank/var etc mounted on
tank/root. If you need more detail my current setup procedure is:

"gpart create -s GPT <disk>"
"gpart add -b 34 -s 128 -t freebsd-boot <disk>"
"gpart add -b 162 -s 1G -t freebsd-swap <disk>"
"gpart add -t freebsd-zfs <disk>"

"zpool create tank raidz <disk1> <disk2> <diskN>"

"zfs set checksum=fletcher4 tank"

"zfs create -o reserv=512m tank/root"

"zfs create -o mountpoint=/tank/root/usr tank/usr"
"zfs create -o mountpoint=/tank/root/tmp tank/tmp"
"zfs create -o mountpoint=/tank/root/var tank/var"
"zfs create -o mountpoint=/tank/root/home tank/home"
"zfs create tank/usr/obj"
"zfs create tank/usr/src"
"zfs create tank/usr/ports"

export DESTDIR=/tank/root
Run the ./install.sh scripts in the various directories of the dist


"mkdir /boot/zfs"
"zpool export tank && zpool import tank"
"cp /boot/zfs/zpool.cache /tank/root/boot/zfs/"

Set 'LOADER_ZFS_SUPPORT=yes' in /tank/root/etc/src.conf

"chroot /tank/root"
"mount -t devfs devfs /dev"
"unset DESTDIR"
"cd /usr/src/sys/boot/"
"make obj"
"make depend"
"make"
"cd i386/loader"
"make install"
"umount /dev"
"exit"
"export LD_LIBRARY_PATH=/dist/lib"

"gpart bootcode -b /tank/root/boot/pmbr -p /tank/root/boot/gptzfsboot -i 1
<disk>"

/boot/loader.conf:
zfs_load="YES"
vfs.root.mountfrom="zfs:tank/root"

/etc/rc.conf:
zfs_enable="YES"

"zfs umount -a"
"zfs set mountpoint=legacy tank"
"zfs set mountpoint=/tmp tank/tmp"
"zfs set mountpoint=/var tank/var"
"zfs set mountpoint=/usr tank/usr"
"zfs set mountpoint=/home tank/home"
"zpool set bootfs=tank/root tank"

Then reboot, csup sources to whatever version to test and build
world/kernel, install and watch things breaks.

Kind regards,
Merijn
Norikatsu Shigemura
2009-10-27 22:17:34 UTC
Permalink
Hi rnoland.

On Mon, 26 Oct 2009 10:34:59 -0500
Post by Robert Noland
Post by Merijn Verstraaten
Post by Robert Noland
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
Could you type "status" at this point and tell what it shows?
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Which seems odd, since that's all the drives there are. So if it finds
these it's already found all drives. My optimistic "Oh! I'll try and boot
again" spirit was however crushed since it just results in the same error.
Ok, that is both good and frustrating... I haven't produced any boot
failures with all of the drives visible. Do, note that I just added
support for reading gang blocks to the loader. (basically untested,
since I haven't managed to create them at will) You will need to update
your partition boot code for it to be supported during early boot. i.e.
gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
The "all block copies unavailable" is a frustrating error, since all it
means is a failed read, but we don't get a clue what failed or why.
With the code that is in -CURRENT it will report gang blocks if found,
even if it fails to read them.
I confirmed reproduce.

1. zpool list, and get SIZE and CAP.
$ zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 59.5G 48.4G 11.1G 81% ONLINE -

2. reduce AVAIL < 10% with creating dummy file like ...
$ dd if=/dev/zero of=$HOME/DUMMY.FILE bs=1m count=5632
5632+0 records in
5632+0 records out
5905580032 bytes transferred in 49.822200 secs (118533104 bytes/sec)
$ zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 59.5G 53.9G 5.61G 90% ONLINE -

3. cd /boot/; cp -pr kernel kernel.err
In this time, if reboot, we can get boot time error.

4. rm $HOME/DUMMY.FILE, and reboot

5. boot kernel.err on new-loader.
I can get "ZFS: gang block detected!" message and overrun:D.
Robert Noland
2009-10-27 23:19:15 UTC
Permalink
Post by Norikatsu Shigemura
Hi rnoland.
On Mon, 26 Oct 2009 10:34:59 -0500
Post by Robert Noland
Post by Merijn Verstraaten
Post by Robert Noland
ZFS: i/o error - all block copies unavailable
Invalid format
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
Could you type "status" at this point and tell what it shows?
pool: tank
NAME STATE
tank ONLINE
raidz1 ONLINE
ad4p3 ONLINE
ad6p3 ONLINE
ad8p3 ONLINE
ad10p3 ONLINE
Which seems odd, since that's all the drives there are. So if it finds
these it's already found all drives. My optimistic "Oh! I'll try and boot
again" spirit was however crushed since it just results in the same error.
Ok, that is both good and frustrating... I haven't produced any boot
failures with all of the drives visible. Do, note that I just added
support for reading gang blocks to the loader. (basically untested,
since I haven't managed to create them at will) You will need to update
your partition boot code for it to be supported during early boot. i.e.
gpart bootcode -p /boot/gptzfsboot -i <boot partition> <disk>
The "all block copies unavailable" is a frustrating error, since all it
means is a failed read, but we don't get a clue what failed or why.
With the code that is in -CURRENT it will report gang blocks if found,
even if it fails to read them.
I confirmed reproduce.
1. zpool list, and get SIZE and CAP.
$ zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 59.5G 48.4G 11.1G 81% ONLINE -
2. reduce AVAIL < 10% with creating dummy file like ...
$ dd if=/dev/zero of=$HOME/DUMMY.FILE bs=1m count=5632
5632+0 records in
5632+0 records out
5905580032 bytes transferred in 49.822200 secs (118533104 bytes/sec)
$ zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 59.5G 53.9G 5.61G 90% ONLINE -
3. cd /boot/; cp -pr kernel kernel.err
In this time, if reboot, we can get boot time error.
4. rm $HOME/DUMMY.FILE, and reboot
5. boot kernel.err on new-loader.
I can get "ZFS: gang block detected!" message and overrun:D.
Ok, so does it still boot? Or do you still get an error?

robert.
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Norikatsu Shigemura
2009-10-28 18:55:39 UTC
Permalink
Hi rnoland.

On Tue, 27 Oct 2009 18:19:15 -0500
Post by Robert Noland
Post by Norikatsu Shigemura
2. reduce AVAIL < 10% with creating dummy file like ...
$ dd if=/dev/zero of=$HOME/DUMMY.FILE bs=1m count=5632
5632+0 records in
5632+0 records out
5905580032 bytes transferred in 49.822200 secs (118533104 bytes/sec)
$ zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 59.5G 53.9G 5.61G 90% ONLINE -
3. cd /boot/; cp -pr kernel kernel.err
In this time, if reboot, we can get boot time error.
4. rm $HOME/DUMMY.FILE, and reboot
5. boot kernel.err on new-loader.
I can get "ZFS: gang block detected!" message and overrun:D.
Ok, so does it still boot? Or do you still get an error?
After reboot:
OK: boot kernel
NG: boot kernel.err

Sorry, kernel.err is ganged kernel. I should be called as
kernel.gang.
Krzysztof Dajka
2009-10-28 20:13:01 UTC
Permalink
I still can't boot. I don't know whether I missed it in first time,
but I'm getting
ZFS: i/o error - all block copies unavailable
x6 before seeing boot manager. I tried booting with verbose option,
but there was only laconic
Trying to mount root from zfs:zroot
ROOT MOUNT ERROR
When I execute lsdev I'm getting
....
disk devices:
disk0: BIOS drive C:
disk0p1: FreeBSD boot
disk0p2: FreeBSD swap
disk0p3: FreeBSD ZFS
disk1: BIOS drive D:
disk1p1: FreeBSD boot
disk1p2: FreeBSD swap
disk1p3: FreeBSD ZFS
disk2: BIOS drive E:
disk2p1: FreeBSD boot
disk2p2: FreeBSD swap
disk2p3: FreeBSD ZFS
pxe devices:
zfs devices:
zfs0: zroot

I can also do `ls` and I can browse whole zroot, and everything seems
to be there. At least for the boot loader.
Robert Noland
2009-10-28 20:43:07 UTC
Permalink
Post by Krzysztof Dajka
I still can't boot. I don't know whether I missed it in first time,
but I'm getting
ZFS: i/o error - all block copies unavailable
x6 before seeing boot manager. I tried booting with verbose option,
but there was only laconic
If you hit a key very early then you will land in boot2, where you can
type "status" and it will list the zfs pool.

robert.
Post by Krzysztof Dajka
Trying to mount root from zfs:zroot
ROOT MOUNT ERROR
When I execute lsdev I'm getting
....
disk0p1: FreeBSD boot
disk0p2: FreeBSD swap
disk0p3: FreeBSD ZFS
disk1p1: FreeBSD boot
disk1p2: FreeBSD swap
disk1p3: FreeBSD ZFS
disk2p1: FreeBSD boot
disk2p2: FreeBSD swap
disk2p3: FreeBSD ZFS
zfs0: zroot
I can also do `ls` and I can browse whole zroot, and everything seems
to be there. At least for the boot loader.
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
a***@gmail.com
2009-10-29 09:44:21 UTC
Permalink
Post by Robert Noland
If you hit a key very early then you will land in boot2, where you can
type "status" and it will list the zfs pool.
I haven't tried it yet. Would it allow me to mount root?
I tried to install from fixit once again, but I've made a script to eliminate
all typo's I could made. I amde script using most of:
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1
Post by Robert Noland
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type lld
ZFS: unexpected object set type lld
So in my opinion increasing number of sectors for first slice containing
bootcode, helped at least in my case.

I'll try to install FreeBSD in tuesday with new iso, as I suspect that
something went wrong during burning dvd, also I didn't check md5. So maybe
those problems are only related to defuncted dvd, I hope so.
Radek Valášek
2009-10-29 10:22:02 UTC
Permalink
Post by a***@gmail.com
Post by Robert Noland
If you hit a key very early then you will land in boot2, where you can
type "status" and it will list the zfs pool.
I haven't tried it yet. Would it allow me to mount root?
I tried to install from fixit once again, but I've made a script to eliminate
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1
Post by Robert Noland
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type lld
ZFS: unexpected object set type lld
So in my opinion increasing number of sectors for first slice containing
bootcode, helped at least in my case.
OK, so what's your current size of the first slice ? Could you paste
here the result of 'gpart show' on your instalation?
Post by a***@gmail.com
I'll try to install FreeBSD in tuesday with new iso, as I suspect that
something went wrong during burning dvd, also I didn't check md5. So maybe
those problems are only related to defuncted dvd, I hope so.
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
a***@gmail.com
2009-11-04 16:47:21 UTC
Permalink
Yesterday I tried installing FreeBSD once more, twice to be frank.
I downloaded amd64 8.0-RC2 checked md5 sum. First time I installed it on
physical machine.

I created script with commands pasted here:
http://pastebin.com/f72813264
After that I executed:

chroot /zroot
mount -t devfs devfs /dev
export DESTDIR=""
cd /usr/src/sys/boot/
make obj
make depend
make
cd i386/loader
make install
umount /dev
passwd
exit
cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache

Finally I launched this script:
http://pastebin.com/f1449f548
GPT boot with ZFS RAIDZ "ZFS: i/o error - all block copies unavailable
I created virtual machine on linux kvm, and added three drives. After starting
system boot loader failed to launch with same error.

I tried to reinstall FreeBSD under kvm, but it didn't help.

Anyone has succeed installing amd64 8.0-RC2 with raidz1 on gpt drives?
Robert Noland
2009-10-29 10:36:38 UTC
Permalink
Post by a***@gmail.com
Post by Robert Noland
If you hit a key very early then you will land in boot2, where you can
type "status" and it will list the zfs pool.
I haven't tried it yet. Would it allow me to mount root?
I tried to install from fixit once again, but I've made a script to eliminate
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1
This seems pretty accurate. I've never tried to do this with labeled
disks though. For my current setups, I create the pools using ada0p3,
etc... And once they get imported after boot, they end up using gpt
id's.

balrog% zpool status test
pool: test
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
raidz2 ONLINE 0 0 0
gptid/c56f3303-be79-11de-813c-002215ea6216 ONLINE 0 0 0
gptid/c70a07c6-be79-11de-813c-002215ea6216 ONLINE 0 0 0
gptid/c915d03c-be79-11de-813c-002215ea6216 ONLINE 0 0 0
gptid/0e4cf049-c0ce-11de-8c99-002215ea6216 ONLINE 0 0 0
gptid/0ff739a0-c0ce-11de-8c99-002215ea6216 ONLINE 0 0 0
gptid/10f4e873-c0ce-11de-8c99-002215ea6216 ONLINE 0 0 0

errors: No known data errors

This pool continues to boot fine under qemu, and with my hacks to the
BIOS drive detection, it boots under VBox as well.
Post by a***@gmail.com
Post by Robert Noland
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type lld
ZFS: unexpected object set type lld
Ok, if you are getting this... It may be helpful if you can run "zdb
-uuu zroot", so I can see what is going on with the root block pointer.
I think you should be able to do that from fixit. Also note that you
don't have the latest version of the loader if it is printing "lld",
though as long as all of the drives are detected it likely won't make a
difference. The "can't read MOS" error is the first attempt to read
from the pool after probing all of the devices to sort out the
configuration. Everything up to this point reads data directly from the
vdev labels on each drive, so this is the first time that it tries to
read from the pool.
Post by a***@gmail.com
So in my opinion increasing number of sectors for first slice containing
bootcode, helped at least in my case.
-rw-r--r-- 1 rnoland rnoland 25671 Oct 24 13:56 gptzfsboot

Even with all the debugging that I have enabled at the moment,
gptzfsboot still only requires 51 sectors, so the usual 128 (64k) should
be more than enough.

robert.
Post by a***@gmail.com
I'll try to install FreeBSD in tuesday with new iso, as I suspect that
something went wrong during burning dvd, also I didn't check md5. So maybe
those problems are only related to defuncted dvd, I hope so.
--
Robert Noland <***@FreeBSD.org>
FreeBSD
a***@gmail.com
2009-11-09 08:48:12 UTC
Permalink
Post by Robert Noland
Ok, if you are getting this... It may be helpful if you can run "zdb
-uuu zroot", so I can see what is going on with the root block pointer.
I think you should be able to do that from fixit. Also note that you
don't have the latest version of the loader if it is printing "lld",
though as long as all of the drives are detected it likely won't make a
difference. The "can't read MOS" error is the first attempt to read
from the pool after probing all of the devices to sort out the
configuration. Everything up to this point reads data directly from the
vdev labels on each drive, so this is the first time that it tries to
read from the pool.
Sorry I forgot about this part.
Here is zdb -uuu zroot
Uberblock

magic = 0000000000bab10c
version = 13
txg = 111
guid_sum = 14036990686815153767
timestamp = 1257636491 UTC = Sat Nov 7 23:28:11 2009
rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:110004a000:400>
DVA[1]=<0:40038400:400> DVA[2]=<0:1980041c00:400> fletcher4 lzjb LE contiguous
birth=111 fill=126 cksum=7e96f92e7:357c99e1eb9:b7d10db3a713:1ac2df05bd147a
Robert Noland
2009-11-09 11:09:23 UTC
Permalink
Post by a***@gmail.com
Post by Robert Noland
Ok, if you are getting this... It may be helpful if you can run "zdb
-uuu zroot", so I can see what is going on with the root block pointer.
I think you should be able to do that from fixit. Also note that you
don't have the latest version of the loader if it is printing "lld",
though as long as all of the drives are detected it likely won't make a
difference. The "can't read MOS" error is the first attempt to read
from the pool after probing all of the devices to sort out the
configuration. Everything up to this point reads data directly from the
vdev labels on each drive, so this is the first time that it tries to
read from the pool.
Sorry I forgot about this part.
Here is zdb -uuu zroot
Uberblock
magic = 0000000000bab10c
version = 13
txg = 111
guid_sum = 14036990686815153767
timestamp = 1257636491 UTC = Sat Nov 7 23:28:11 2009
rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:110004a000:400>
DVA[1]=<0:40038400:400> DVA[2]=<0:1980041c00:400> fletcher4 lzjb LE contiguous
birth=111 fill=126 cksum=7e96f92e7:357c99e1eb9:b7d10db3a713:1ac2df05bd147a
Ok, that all looks correct for a raidz1. The logical size is 1024 bytes
(400L) which is the uncompressed size of the data. The physical size is
512 bytes (200P) which is one sector on disk and the allocated size is
1024 bytes (512 bytes + 512 bytes of parity). For a raidz or mirror
pool, the vdev is always 0, which represents the pseudo vdev, but it
doesn't tell us for sure that all of the physical disks have been found.
We should be able to verify that all the disks are present by summing
the GUIDs. I'll have a look at that, so that we can at least warn if
that is the case. If all of the physical devices are accounted for,
then something is going wrong in the raidz read function, though I don't
understand why it isn't more prolific if that is the case. i.e. why my
raidz2 test is working fine.

robert.
--
Robert Noland <***@FreeBSD.org>
FreeBSD
a***@gmail.com
2009-10-28 07:42:50 UTC
Permalink
Hi,
I'm having same problem as Radek Valášek.
I tried to install FreeBSD-8.0-RC1 on 3 GPT disks.
I did step by step 'manual' from
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ2
I'd like to ask if this problem could be related to setting AHCI host
controller on?

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type lld
ZFS: unexpected object set type lld

FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
boot:
ZFS: unexpected object set type lld

FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
boot:

FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
boot:
Robert Noland
2009-10-28 11:10:10 UTC
Permalink
Post by a***@gmail.com
Hi,
I'm having same problem as Radek Valášek.
I tried to install FreeBSD-8.0-RC1 on 3 GPT disks.
I did step by step 'manual' from
http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ2
I'd like to ask if this problem could be related to setting AHCI host
controller on?
I think that is unlikely. If you type status at this point, does it
show all of your drives ONLINE?
Post by a***@gmail.com
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type lld
ZFS: unexpected object set type lld
I'm trying to find the "can't read MOS" issue now. This is pretty much
the first thing that we try to get once all of the drives are online. I
can't produce this issue locally, so I'm trying to get enough debugging
to see what is going on with the root block pointer. Basically, if we
can't read the MOS, we aren't going to read anything.

Thanks to ps@, we have a wrapper program that lets us debug this from
userland which helps a lot. I just wish that I could figure out how to
replicate the problem, but I guess if I could do that... we could figure
out how to fix it.

robert.
Post by a***@gmail.com
FreeBSD/i386 boot
Default: z:/boot/kernel/kernel
ZFS: unexpected object set type lld
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
FreeBSD/i386 boot
Default: tank:/boot/kernel/kernel
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
a***@gmail.com
2009-10-28 11:25:53 UTC
Permalink
Post by Robert Noland
I think that is unlikely. If you type status at this point, does it
show all of your drives ONLINE?
Yes, they we're online.
Post by Robert Noland
Post by Radek Valášek
ZFS: can't read MOS
I'm not quite sure about this line because I copy/pasted from Radek's post. I
will check it and post later when I'll get home.
Krzysztof Dajka
2009-10-28 18:34:16 UTC
Permalink
I managed to boot FreeBSD with raidz1 on gpt drives on my workstation.
Well I didn't quite finished booting yet...
But from the beginning.
I tried to install FreeBSD once again instead of raidz2 I chose
raidz1, but it's not the answer. I think that It could be related to
what Norikatsu Shigemura reproduced
http://lists.freebsd.org/pipermail/freebsd-current/2009-October/012935.html
When I used gpart to create my boot slices on 3 drives I increased
size of that slice to 400 sectors. Now my system boots and
ZFS: i/o error - all block copies unavailable
error is missing.
But I haven't really finish boot part.
Now I'm getting:
Trying to mount root from zfs:zroot
ROOT MOUNT ERROR

So I'll have to figure out what went wrong during fixit installation,
maybe some typo
Post by Robert Noland
I think that is unlikely. If you type status at this point, does it
show all of your drives ONLINE?
Yes, they we're online.
Post by Robert Noland
Post by Radek Valášek
ZFS: can't read MOS
I'm not quite sure about this line because I copy/pasted from Radek's post. I
will check it and post later when I'll get home.
Krzysztof Dajka
2009-10-28 18:32:01 UTC
Permalink
I managed to boot FreeBSD with raidz1 on gpt drives on my workstation.
Well I didn't quite finished booting yet...
But from the beginning.
I tried to install FreeBSD once again instead of raidz2 I chose
raidz1, but it's not the answer. I think that
When I used gpart to create my boot slices on 3 drives I increased
size of that slice to 400 sectors. Now my system boots and
ZFS: i/o error - all block copies unavailable
error is missing.
But I haven't really finish boot part.
Now I'm getting:
Trying to mount root from zfs:zroot
ROOT MOUNT ERROR

So I'll have to figure out what went wrong during fixit installation,
maybe some typo
Post by Robert Noland
I think that is unlikely. If you type status at this point, does it
show all of your drives ONLINE?
Yes, they we're online.
Post by Robert Noland
Post by Radek Valášek
ZFS: can't read MOS
I'm not quite sure about this line because I copy/pasted from Radek's post. I
will check it and post later when I'll get home.
Robert Noland
2009-10-28 20:25:04 UTC
Permalink
Post by Krzysztof Dajka
I managed to boot FreeBSD with raidz1 on gpt drives on my workstation.
Well I didn't quite finished booting yet...
But from the beginning.
I tried to install FreeBSD once again instead of raidz2 I chose
raidz1, but it's not the answer. I think that
When I used gpart to create my boot slices on 3 drives I increased
size of that slice to 400 sectors. Now my system boots and
if you mean the freebsd-boot partition, that isn't part of zfs. That is
just an empty partition to hold gptzfsboot. So, that isn't relevant.

This does work for me... I still haven't been able to produce a
non-working system. Other than the case where the BIOS doesn't pick up
all the drives. I've built booting systems with single and multiple
root devices as well as raidz1 or raidz2.

I'm not saying that there isn't an issue, just that I haven't been able
find a failure that I can reproduce.

robert.
Post by Krzysztof Dajka
ZFS: i/o error - all block copies unavailable
error is missing.
But I haven't really finish boot part.
Trying to mount root from zfs:zroot
ROOT MOUNT ERROR
So I'll have to figure out what went wrong during fixit installation,
maybe some typo
Post by Robert Noland
I think that is unlikely. If you type status at this point, does it
show all of your drives ONLINE?
Yes, they we're online.
Post by Robert Noland
Post by Radek Valášek
ZFS: can't read MOS
I'm not quite sure about this line because I copy/pasted from Radek's post. I
will check it and post later when I'll get home.
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Matt Reimer
2009-11-13 01:04:41 UTC
Permalink
Radek,
Try the attached patch (sponsored by VPOP Technologies). I found an
overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
causing my 6x1TB raidz2 array to fail to boot.
Apply the patch, build everything in /sys/boot, and then make sure you
update both gptzfsboot and /boot/loader.
Oops, here's the patch.

Matt
Matt Reimer
2009-11-13 00:54:58 UTC
Permalink
Post by Radek Valášek
Hi,
I want to ask if there is something new in adding support to
gptzfsboot/zfsboot for reading gang-blocks?
Gang blocks
When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.
Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
Radek,

Try the attached patch (sponsored by VPOP Technologies). I found an
overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
causing my 6x1TB raidz2 array to fail to boot.

Apply the patch, build everything in /sys/boot, and then make sure you
update both gptzfsboot and /boot/loader.

Robert, I'm guessing you couldn't replicate this because your array
was small enough not to result in block numbers overflowing an int.

The kernel source for the corresponding functionality is in
/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:vdev_raidz_map_alloc().
There all these variables are uint64_t, but I think unnecessarily. I
tried changing the boot loader's vdev_raidz_read() variables to all
uint64_t but then gptzfsboot would reboot itself, likely due to a
stack overflow. The attached patch just changes a few variables that,
after a quick analysis, seemed likely to overflow.

If this looks good, would someone commit it?

Matt
Robert Noland
2009-11-13 04:53:03 UTC
Permalink
Post by Radek Valášek
Hi,
I want to ask if there is something new in adding support to
gptzfsboot/zfsboot for reading gang-blocks?
Gang blocks
When there is not enough contiguous space to write a complete block, the ZIO
pipeline will break the I/O up into smaller 'gang blocks' which can later be
assembled transparently to appear as complete blocks.
Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
Radek,
Try the attached patch (sponsored by VPOP Technologies). I found an
overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
causing my 6x1TB raidz2 array to fail to boot.
Apply the patch, build everything in /sys/boot, and then make sure you
update both gptzfsboot and /boot/loader.
Robert, I'm guessing you couldn't replicate this because your array
was small enough not to result in block numbers overflowing an int.
This is likely, all of my raidz tests were with vnode backed 1GB memory
disks. So my largest configuration was a 6 x 1GB raidz2.
The kernel source for the corresponding functionality is in
/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:vdev_raidz_map_alloc().
There all these variables are uint64_t, but I think unnecessarily. I
tried changing the boot loader's vdev_raidz_read() variables to all
uint64_t but then gptzfsboot would reboot itself, likely due to a
stack overflow. The attached patch just changes a few variables that,
after a quick analysis, seemed likely to overflow.
If this looks good, would someone commit it?
ps@ grabbed it up already, but I may handle the MFC for him. I have
some other minor fixups in my tree right now... like teaching printf to
handle %llx. Thanks for finding this... It's been really frustrating
that I couldn't produce a failing system.

robert.
Matt
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Matt Reimer
2009-11-13 06:15:07 UTC
Permalink
Post by Robert Noland
Post by Radek Valášek
Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
Radek,
Try the attached patch (sponsored by VPOP Technologies). I found an
overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
causing my 6x1TB raidz2 array to fail to boot.
...
Post by Robert Noland
The kernel source for the corresponding functionality is in
/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:vdev_raidz_map_alloc().
There all these variables are uint64_t, but I think unnecessarily. I
tried changing the boot loader's vdev_raidz_read() variables to all
uint64_t but then gptzfsboot would reboot itself, likely due to a
stack overflow. The attached patch just changes a few variables that,
after a quick analysis, seemed likely to overflow.
If this looks good, would someone commit it?
some other minor fixups in my tree right now... like teaching printf to
handle %llx.  Thanks for finding this... It's been really frustrating
that I couldn't produce a failing system.
Is it possible for this patch to get into 8.0-RELEASE, or is it too
late? I suppose it doesn't matter that much since the loader isn't
built with LOADER_ZFS_SUPPORT by default anyway, so folks are going to
have to compile it themselves.

Matt
Robert Noland
2009-11-13 14:25:54 UTC
Permalink
Post by Matt Reimer
Post by Robert Noland
Post by Radek Valášek
Everything works fine for me, until I rewrite kernel/world after system
upgrade to latest one (releng_8). After this am I no longer able to boot
/ ZFS: i/o error - all block copies unavailable
/>/ ZFS: can't read MOS
/>/ ZFS: unexpected object set type lld
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: z:/boot/kernel/kernel
/>/ ZFS: unexpected object set type lld
/>/
/>/ FreeBSD/i386 boot
/>/ Default: tank:/boot/kernel/kernel
Radek,
Try the attached patch (sponsored by VPOP Technologies). I found an
overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
causing my 6x1TB raidz2 array to fail to boot.
...
Post by Robert Noland
The kernel source for the corresponding functionality is in
/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c:vdev_raidz_map_alloc().
There all these variables are uint64_t, but I think unnecessarily. I
tried changing the boot loader's vdev_raidz_read() variables to all
uint64_t but then gptzfsboot would reboot itself, likely due to a
stack overflow. The attached patch just changes a few variables that,
after a quick analysis, seemed likely to overflow.
If this looks good, would someone commit it?
some other minor fixups in my tree right now... like teaching printf to
handle %llx. Thanks for finding this... It's been really frustrating
that I couldn't produce a failing system.
Is it possible for this patch to get into 8.0-RELEASE, or is it too
late? I suppose it doesn't matter that much since the loader isn't
built with LOADER_ZFS_SUPPORT by default anyway, so folks are going to
have to compile it themselves.
I think we have missed the boat, but I'll talk to re@ and see if we can
get it in.

robert.
Post by Matt Reimer
Matt
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Stefan Esser
2009-11-13 20:22:19 UTC
Permalink
can get it in.
The patch fixed GPT/ZFS booting for me, too. It would be good to have
it in 8.0 (since it is definitely required to boot from ZFS pools with
non-trivial sizes), and does not affect anybody not trying to boot
this way. OTOH, it since you cannot just install FreeBSD on pure ZFS
from sysinstall, it might be sufficient to prominently warn about this
problem and point at the required patch, to prevent foot-shooting.

But having this patch that has been successfully tested by a number
of people that suffered from the GPT/ZFS boot problem looks highly
preferable to me ...

Regards, STefan
Radek Valášek
2009-11-14 09:05:21 UTC
Permalink
Post by Stefan Esser
can get it in.
The patch fixed GPT/ZFS booting for me, too. It would be good to have
it in 8.0 (since it is definitely required to boot from ZFS pools with
non-trivial sizes), and does not affect anybody not trying to boot
this way. OTOH, it since you cannot just install FreeBSD on pure ZFS
from sysinstall, it might be sufficient to prominently warn about this
problem and point at the required patch, to prevent foot-shooting.
But having this patch that has been successfully tested by a number
of people that suffered from the GPT/ZFS boot problem looks highly
preferable to me ...
Regards, STefan
I can confirm that the patch is working for me too, and I'm able now to
boot from raidz/raidz2 pool after rewriting loader.conf/loader/kernel. I
agree with Stefan, having it in 8.0-RELEASE would be good, catch the boat :)

Big thnx to Matt, great work from all.
Krzysztof Dajka
2009-11-14 09:15:52 UTC
Permalink
Thanks Matt for the patch. I used it with 8.0RC3 release. I installed
FreeBSD under Linux (KVM) my 3x500GB drives were mounted as a scsi
drives. Installation went smoothly but when I rebooted FreeBSD guest
it hang as usual ;) with "ZFS: i/o error - all block copies
unavailable", well it also spit out some LBA errors for the first
time. I was a little disappointed, because I've been trying for three
weeks to replace my Debian system with broken ext3 fs with FreeBSD on
raidz. But I thought to myself I'll give it a try, and run FreeBSD
native. To my suprise it welcomed me with login prompt. Once again
thanks for the patch. It would be good idea to merge it with final
release.
Post by Stefan Esser
can get it in.
The patch fixed GPT/ZFS booting for me, too. It would be good to have
it in 8.0 (since it is definitely required to boot from ZFS pools with
non-trivial sizes), and does not affect anybody not trying to boot
this way. OTOH, it since you cannot just install FreeBSD on pure ZFS
from sysinstall, it might be sufficient to prominently warn about this
problem and point at the required patch, to prevent foot-shooting.
But having this patch that has been successfully tested by a number
of people that suffered from the GPT/ZFS boot problem looks highly
preferable to me ...
Regards, STefan
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
Robert Noland
2009-11-14 18:44:21 UTC
Permalink
Post by Krzysztof Dajka
Thanks Matt for the patch. I used it with 8.0RC3 release. I installed
FreeBSD under Linux (KVM) my 3x500GB drives were mounted as a scsi
drives. Installation went smoothly but when I rebooted FreeBSD guest
it hang as usual ;) with "ZFS: i/o error - all block copies
unavailable", well it also spit out some LBA errors for the first
time. I was a little disappointed, because I've been trying for three
weeks to replace my Debian system with broken ext3 fs with FreeBSD on
raidz. But I thought to myself I'll give it a try, and run FreeBSD
native. To my suprise it welcomed me with login prompt. Once again
thanks for the patch. It would be good idea to merge it with final
release.
This was approved by re@ and has been merged to the release branch. It
should be included in 8.0-RELEASE.

robert.
Post by Krzysztof Dajka
Post by Stefan Esser
can get it in.
The patch fixed GPT/ZFS booting for me, too. It would be good to have
it in 8.0 (since it is definitely required to boot from ZFS pools with
non-trivial sizes), and does not affect anybody not trying to boot
this way. OTOH, it since you cannot just install FreeBSD on pure ZFS
from sysinstall, it might be sufficient to prominently warn about this
problem and point at the required patch, to prevent foot-shooting.
But having this patch that has been successfully tested by a number
of people that suffered from the GPT/ZFS boot problem looks highly
preferable to me ...
Regards, STefan
_______________________________________________
http://lists.freebsd.org/mailman/listinfo/freebsd-current
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Jonathan
2009-11-30 18:37:07 UTC
Permalink
I just wanted to thank all involved in getting RAIDZ boot working.

Thanks!

Jonathan

Stefan Bethke
2009-11-14 20:03:34 UTC
Permalink
Try the attached patch (sponsored by VPOP Technologies). I found an
overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
causing my 6x1TB raidz2 array to fail to boot.
I can confirm as well that the patch (as committed to -current as r199241) makes my loader happy. Now I just need to figure out why the kernel won't mount root...


Thanks,
Stefan
--
Stefan Bethke <***@lassitu.de> Fon +49 151 14070811
Stefan Bethke
2009-11-14 20:36:18 UTC
Permalink
Post by Stefan Bethke
Try the attached patch (sponsored by VPOP Technologies). I found an
overflow in /sys/cddl/boot/zfs/zfssubr.c:vdev_raidz_read() that was
causing my 6x1TB raidz2 array to fail to boot.
I can confirm as well that the patch (as committed to -current as r199241) makes my loader happy. Now I just need to figure out why the kernel won't mount root...
I was trying to boot off a raw ZFS pool. When using GPT partitions, it works just fine.


Stefan
--
Stefan Bethke <***@lassitu.de> Fon +49 151 14070811
Emil Smolenski
2009-11-16 16:26:10 UTC
Permalink
After installkernel/installworld my machine stops booting with the
following error message:

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type lld

FreeBSD/i386 boot
Default: pgpool:/boot/kernel/kernel
boot:
ZFS: unexpected object set type lld

This is 7.2-STABLE, amd64, zpool on single logical device (ciss(4),
hardware RAID5), root on ZFS (using zfsboot). After the failure I booted
the server from an external device with UFS and then I did rollback of
/usr and / datasets. The machine was still not bootable. Scrub went
without errors.
Then I read this thread and applied Robert Noland's and Matt Reimer's
patches -- and they didn't help. Then I grabbed following files from
-CURRENT (svn rev. 198420):

/sys/boot/i386/zfsboot/zfsboot.c
/sys/boot/zfs/zfs.c
/sys/boot/zfs/zfsimpl.c
/sys/cddl/boot/zfs/zfsimpl.h

and I did:

# cd /usr/src/sys/boot/
# make obj ; make depend ; make
# cd i386/loader
# make install
# cd /usr/src/sys/boot/i386/zfsboot
# make install
# sysctl kern.geom.debugflags=16
# dd if=/boot/zfsboot of=/dev/da0 count=1
# dd if=/boot/zfsboot of=/dev/da0 skip=1 seek=1024
# reboot

(is this procedure of updating zfsboot correct?)

After that, an error was slightly different (printf was fixed):

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type 0

FreeBSD/i386 boot
Default: pgpool:/boot/kernel/kernel
boot:
ZFS: unexpected object set type 0

Additional information:

# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
pgpool 4.06T 2.17T 1.89T 53% ONLINE -

# zpool status
pool: pgpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
pgpool ONLINE 0 0 0
da0 ONLINE 0 0 0

errors: No known data errors

# zfs list pgpool/ROOTFS
NAME USED AVAIL REFER MOUNTPOINT
pgpool/ROOTFS 568M 1.80T 55.3M legacy

# zpool get all pgpool
NAME PROPERTY VALUE SOURCE
pgpool size 4.06T -
pgpool used 2.17T -
pgpool available 1.89T -
pgpool capacity 53% -
pgpool altroot - default
pgpool health ONLINE -
pgpool guid 3920915583055727184 -
pgpool version 13 default
pgpool bootfs pgpool/ROOTFS local
pgpool delegation on default
pgpool autoreplace off default
pgpool cachefile - default
pgpool failmode wait default
pgpool listsnapshots off default

loader.conf:
usb_load="YES"
uplcom_load="YES"
umass_load="YES"
ugen_load="YES"
ukbd_load="YES"
random_load="YES"
loader_color="YES"
vfs.root.mountfrom="zfs:pgpool/ROOTFS"
zfs_load="YES"
autoboot_delay="2"

FreeBSD 7.2-STABLE #0: Fri Jun 19 13:27:29 CEST 2009
(as I mentioned above, there was the rollback)

ciss0: <HP Smart Array P400> port 0xe800-0xe8ff mem
0xdef00000-0xdeffffff,0xdeeff000-0xdeefffff irq 35 at device 0.0 on pci4
ciss0: [ITHREAD]
da0 at ciss0 bus 0 target 0 lun 0

I would rather not to upgrade the whole system to -CURRENT. What should I
do in this situation? Is there any other patch that I could apply or any
workaround for this issue? Is there possibility to switch from zfsboot to
gptzfsboot without loosing data? Or maybe I did something wrong?
--
am
Robert Noland
2009-11-16 16:59:44 UTC
Permalink
Post by Emil Smolenski
After installkernel/installworld my machine stops booting with the
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type lld
FreeBSD/i386 boot
Default: pgpool:/boot/kernel/kernel
ZFS: unexpected object set type lld
This is 7.2-STABLE, amd64, zpool on single logical device (ciss(4),
hardware RAID5), root on ZFS (using zfsboot). After the failure I booted
the server from an external device with UFS and then I did rollback of
/usr and / datasets. The machine was still not bootable. Scrub went
without errors.
Then I read this thread and applied Robert Noland's and Matt Reimer's
patches -- and they didn't help. Then I grabbed following files from
Matt's patch only effects raidz volumes.
Post by Emil Smolenski
/sys/boot/i386/zfsboot/zfsboot.c
/sys/boot/zfs/zfs.c
/sys/boot/zfs/zfsimpl.c
/sys/cddl/boot/zfs/zfsimpl.h
# cd /usr/src/sys/boot/
# make obj ; make depend ; make
# cd i386/loader
# make install
# cd /usr/src/sys/boot/i386/zfsboot
# make install
# sysctl kern.geom.debugflags=16
# dd if=/boot/zfsboot of=/dev/da0 count=1
# dd if=/boot/zfsboot of=/dev/da0 skip=1 seek=1024
# reboot
(is this procedure of updating zfsboot correct?)
This should be correct for updating the first stage bootstrap code. The
loader (boot/loader) is actually updated during installworld.
Post by Emil Smolenski
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type 0
This has my patch applied, which fixes the printf's so that they work
correctly among other things.
Post by Emil Smolenski
FreeBSD/i386 boot
Default: pgpool:/boot/kernel/kernel
ZFS: unexpected object set type 0
# zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
pgpool 4.06T 2.17T 1.89T 53% ONLINE -
# zpool status
pool: pgpool
state: ONLINE
scrub: none requested
NAME STATE READ WRITE CKSUM
pgpool ONLINE 0 0 0
da0 ONLINE 0 0 0
errors: No known data errors
# zfs list pgpool/ROOTFS
NAME USED AVAIL REFER MOUNTPOINT
pgpool/ROOTFS 568M 1.80T 55.3M legacy
# zpool get all pgpool
NAME PROPERTY VALUE SOURCE
pgpool size 4.06T -
pgpool used 2.17T -
pgpool available 1.89T -
pgpool capacity 53% -
pgpool altroot - default
pgpool health ONLINE -
pgpool guid 3920915583055727184 -
pgpool version 13 default
pgpool bootfs pgpool/ROOTFS local
pgpool delegation on default
pgpool autoreplace off default
pgpool cachefile - default
pgpool failmode wait default
pgpool listsnapshots off default
usb_load="YES"
uplcom_load="YES"
umass_load="YES"
ugen_load="YES"
ukbd_load="YES"
random_load="YES"
loader_color="YES"
vfs.root.mountfrom="zfs:pgpool/ROOTFS"
zfs_load="YES"
autoboot_delay="2"
FreeBSD 7.2-STABLE #0: Fri Jun 19 13:27:29 CEST 2009
(as I mentioned above, there was the rollback)
ciss0: <HP Smart Array P400> port 0xe800-0xe8ff mem
0xdef00000-0xdeffffff,0xdeeff000-0xdeefffff irq 35 at device 0.0 on pci4
ciss0: [ITHREAD]
da0 at ciss0 bus 0 target 0 lun 0
I would rather not to upgrade the whole system to -CURRENT. What should I
do in this situation? Is there any other patch that I could apply or any
workaround for this issue? Is there possibility to switch from zfsboot to
gptzfsboot without loosing data? Or maybe I did something wrong?
I don't think that you can switch to gptzfsboot as that would require
repartitioning the device.

A little more context though, was this working before? Or is this a new
install?

robert.
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Emil Smolenski
2009-11-16 18:33:34 UTC
Permalink
Post by Robert Noland
Post by Emil Smolenski
[...]
Then I read this thread and applied Robert Noland's and Matt Reimer's
patches -- and they didn't help. Then I grabbed following files from
Matt's patch only effects raidz volumes.
Oh, I see. Maybe there is similar bug in ZFS on single disk volumes also?
Post by Robert Noland
Post by Emil Smolenski
[cut: update procedure]
(is this procedure of updating zfsboot correct?)
This should be correct for updating the first stage bootstrap code. The
loader (boot/loader) is actually updated during installworld.
I'll try full build/installworld tomorrow.
Post by Robert Noland
Post by Emil Smolenski
[...]
I would rather not to upgrade the whole system to -CURRENT. What should I
do in this situation? Is there any other patch that I could apply or any
workaround for this issue? Is there possibility to switch from zfsboot to
gptzfsboot without loosing data? Or maybe I did something wrong?
I don't think that you can switch to gptzfsboot as that would require
repartitioning the device.
I thought so.
Post by Robert Noland
A little more context though, was this working before? Or is this a new
install?
This system has worked stable since Jun, but I've never done full
installworld after the initial installation. Now, after the installworld,
machine no longer boots. (Rollback did not help).
--
am
Emil Smolenski
2009-11-17 21:43:46 UTC
Permalink
Post by Emil Smolenski
Post by Robert Noland
Matt's patch only effects raidz volumes.
Oh, I see. Maybe there is similar bug in ZFS on single disk volumes also?
Post by Robert Noland
Post by Emil Smolenski
(is this procedure of updating zfsboot correct?)
This should be correct for updating the first stage bootstrap code. The
loader (boot/loader) is actually updated during installworld.
I'll try full build/installworld tomorrow.
It, unfortunately, didn't solve this issue. Should I file a PR? I would
like to help in debugging it (however my skills in low-level C aren't
strong enough to do it on my own).
--
am
Robert Noland
2009-11-17 22:33:41 UTC
Permalink
Post by Emil Smolenski
Post by Emil Smolenski
Post by Robert Noland
Matt's patch only effects raidz volumes.
Oh, I see. Maybe there is similar bug in ZFS on single disk volumes also?
Post by Robert Noland
Post by Emil Smolenski
(is this procedure of updating zfsboot correct?)
This should be correct for updating the first stage bootstrap code. The
loader (boot/loader) is actually updated during installworld.
I'll try full build/installworld tomorrow.
It, unfortunately, didn't solve this issue. Should I file a PR? I would
like to help in debugging it (however my skills in low-level C aren't
strong enough to do it on my own).
Ok, the first thing I would like to see is "zdb -uuu". I don't see an
obvious issue with single disk reads. My own setup uses 2 x 1TB
currently. Failing to read the MOS is basically the first read attempt
from the pool, in fact it is the read that attempts to mount the pool.

robert.
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Emil Smolenski
2009-11-18 00:17:20 UTC
Permalink
Post by Robert Noland
Post by Emil Smolenski
Should I file a PR? I would
like to help in debugging it (however my skills in low-level C aren't
strong enough to do it on my own).
Ok, the first thing I would like to see is "zdb -uuu".
# zdb -uuu pgpool
Segmentation fault: 11 (core dumped)

# zdb
pgpool
version=13
name='pgpool'
state=0
txg=439808
pool_guid=3920915583055727184
hostid=1642959122
hostname='unset'
vdev_tree
type='root'
id=0
guid=3920915583055727184
children[0]
type='disk'
id=0
guid=5859773264564918193
path='/dev/da0'
whole_disk=0
metaslab_array=23
metaslab_shift=35
ashift=9
asize=4500799356928
is_log=0
DTL=260
Post by Robert Noland
I don't see an
obvious issue with single disk reads. My own setup uses 2 x 1TB
currently. Failing to read the MOS is basically the first read attempt
from the pool, in fact it is the read that attempts to mount the pool.
--
am
Robert Noland
2009-11-18 13:50:47 UTC
Permalink
Post by Emil Smolenski
Post by Robert Noland
Post by Emil Smolenski
Should I file a PR? I would
like to help in debugging it (however my skills in low-level C aren't
strong enough to do it on my own).
Ok, the first thing I would like to see is "zdb -uuu".
# zdb -uuu pgpool
Segmentation fault: 11 (core dumped)
Ok, this is disturbing... It works fine for me on -CURRENT / amd64 and
reports the root block pointer, which is what we need to locate the MOS.

robert.
Post by Emil Smolenski
# zdb
pgpool
version=13
name='pgpool'
state=0
txg=439808
pool_guid=3920915583055727184
hostid=1642959122
hostname='unset'
vdev_tree
type='root'
id=0
guid=3920915583055727184
children[0]
type='disk'
id=0
guid=5859773264564918193
path='/dev/da0'
whole_disk=0
metaslab_array=23
metaslab_shift=35
ashift=9
asize=4500799356928
is_log=0
DTL=260
Post by Robert Noland
I don't see an
obvious issue with single disk reads. My own setup uses 2 x 1TB
currently. Failing to read the MOS is basically the first read attempt
from the pool, in fact it is the read that attempts to mount the pool.
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Emil Smolenski
2009-11-18 16:11:14 UTC
Permalink
Post by Robert Noland
Post by Emil Smolenski
Post by Robert Noland
Post by Emil Smolenski
Should I file a PR? I would
like to help in debugging it (however my skills in low-level C aren't
strong enough to do it on my own).
Ok, the first thing I would like to see is "zdb -uuu".
# zdb -uuu pgpool
Segmentation fault: 11 (core dumped)
Ok, this is disturbing... It works fine for me on -CURRENT / amd64 and
reports the root block pointer, which is what we need to locate the MOS.
Booting from 8.0-*-amd64-memstick.img (Fixit# console) makes "zdb -uuu"
happy:

Fixit# zdb -uuu pgpool
Uberblock

magic = 0000000000bab10c
version = 13
txg = 443448
guid_sum = 9780688847620645377
timestamp = 1258560175 UTC = Wed Nov 18 16:02:55 2009
rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:220000de400:200>
DVA[1]=<0:2a80008ee00:200> DVA[2]=<0:330000b9000:200> fletcher4 lzjb LE
contiguous birth=443448 fill=298
cksum=8a9775385:3935d6d58c7:c028430c00a8:1b58ac4ebf42ac
--
am
Robert Noland
2009-11-18 16:43:48 UTC
Permalink
Post by Emil Smolenski
Post by Robert Noland
Post by Emil Smolenski
Post by Robert Noland
Post by Emil Smolenski
Should I file a PR? I would
like to help in debugging it (however my skills in low-level C aren't
strong enough to do it on my own).
Ok, the first thing I would like to see is "zdb -uuu".
# zdb -uuu pgpool
Segmentation fault: 11 (core dumped)
Ok, this is disturbing... It works fine for me on -CURRENT / amd64 and
reports the root block pointer, which is what we need to locate the MOS.
Booting from 8.0-*-amd64-memstick.img (Fixit# console) makes "zdb -uuu"
Fixit# zdb -uuu pgpool
Uberblock
magic = 0000000000bab10c
version = 13
txg = 443448
guid_sum = 9780688847620645377
timestamp = 1258560175 UTC = Wed Nov 18 16:02:55 2009
rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:220000de400:200>
DVA[1]=<0:2a80008ee00:200> DVA[2]=<0:330000b9000:200> fletcher4 lzjb LE
contiguous birth=443448 fill=298
cksum=8a9775385:3935d6d58c7:c028430c00a8:1b58ac4ebf42ac
Ok, the offsets are definately up there... What is your normal
installation? 8.0 i386?

robert.
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Emil Smolenski
2009-11-18 16:46:06 UTC
Permalink
Post by Robert Noland
Post by Robert Noland
Post by Robert Noland
Post by Emil Smolenski
Post by Robert Noland
Post by Emil Smolenski
Should I file a PR? I would
like to help in debugging it (however my skills in low-level C
aren't
Post by Robert Noland
Post by Emil Smolenski
Post by Robert Noland
Post by Emil Smolenski
strong enough to do it on my own).
Ok, the first thing I would like to see is "zdb -uuu".
# zdb -uuu pgpool
Segmentation fault: 11 (core dumped)
Ok, this is disturbing... It works fine for me on -CURRENT / amd64
and
Post by Robert Noland
reports the root block pointer, which is what we need to locate the
MOS.
Booting from 8.0-*-amd64-memstick.img (Fixit# console) makes "zdb -uuu"
Fixit# zdb -uuu pgpool
Uberblock
magic = 0000000000bab10c
version = 13
txg = 443448
guid_sum = 9780688847620645377
timestamp = 1258560175 UTC = Wed Nov 18 16:02:55 2009
rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:220000de400:200>
DVA[1]=<0:2a80008ee00:200> DVA[2]=<0:330000b9000:200> fletcher4 lzjb LE
contiguous birth=443448 fill=298
cksum=8a9775385:3935d6d58c7:c028430c00a8:1b58ac4ebf42ac
Ok, the offsets are definately up there... What is your normal
installation? 8.0 i386?
7.2-STABLE, amd64.
--
am
Matt Reimer
2009-11-18 23:48:03 UTC
Permalink
Post by Robert Noland
Post by Emil Smolenski
Post by Robert Noland
Post by Emil Smolenski
Should I file a PR? I would
like to help in debugging it (however my skills in low-level C aren't
strong enough to do it on my own).
Ok, the first thing I would like to see is "zdb -uuu".
# zdb -uuu pgpool
Segmentation fault: 11 (core dumped)
Ok, this is disturbing...  It works fine for me on -CURRENT / amd64 and
reports the root block pointer, which is what we need to locate the MOS.
  Booting from 8.0-*-amd64-memstick.img (Fixit# console) makes "zdb -uuu"
Fixit# zdb -uuu pgpool
Uberblock
         magic = 0000000000bab10c
         version = 13
         txg = 443448
         guid_sum = 9780688847620645377
         timestamp = 1258560175 UTC = Wed Nov 18 16:02:55 2009
         rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:220000de400:200>
DVA[1]=<0:2a80008ee00:200> DVA[2]=<0:330000b9000:200> fletcher4 lzjb LE
contiguous birth=443448 fill=298
cksum=8a9775385:3935d6d58c7:c028430c00a8:1b58ac4ebf42ac
Ok, the offsets are definately up there... What is your normal
installation?  8.0 i386?
Robert's on to something. It looks like your LBAs are probably
overflowing 32 bits. This would affect all vdev regardless of type.

Try the attached patch.

Matt
Robert Noland
2009-11-19 03:57:37 UTC
Permalink
Post by Emil Smolenski
220000de400
This divided by 512 byte block size is 33 bits... At a glance, the patch
looks ok to me. I'll do a more thorough review of this tomorrow.

robert.
--
Robert Noland <***@FreeBSD.org>
FreeBSD
Emil Smolenski
2009-11-19 10:21:24 UTC
Permalink
Post by Matt Reimer
Robert's on to something. It looks like your LBAs are probably
overflowing 32 bits. This would affect all vdev regardless of type.
Try the attached patch.
Post by Emil Smolenski
220000de400
This divided by 512 byte block size is 33 bits... At a glance, the patch
looks ok to me. I'll do a more thorough review of this tomorrow.
Unfortunately it don't work. Error is the same as before:

ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type 0
ZFS: unexpected object set type 0

FreeBSD/i386 boot
Default: pgpool:/boot/kernel/kernel
boot:
ZFS: unexpected object set type 0


This is 7.2-STABLE, amd64. My test procedure:

1. I fully synchronized these zfsboot-related directories with -CURRENT:

src/sys/boot/i386/zfsboot
src/sys/boot/zfs
src/sys/cddl/boot/zfs

2. I applied Matt Reimer's zfsboot.c.patch3 patch:

# cd /usr/src/sys/boot/
# patch < /path/to/zfsboot.c.patch3

3. Then I did:

# make clean; make cleandir
# make obj ; make depend ; make
# cd i386/loader
# make install
# cd /usr/src/sys/boot/i386/zfsboot
# make install
# sysctl kern.geom.debugflags=16
# dd if=/boot/zfsboot of=/dev/da0 count=1
# dd if=/boot/zfsboot of=/dev/da0 skip=1 seek=1024
# reboot

4. Result: error shown above.

Thanks!
--
am
Robert Noland
2009-11-19 16:23:55 UTC
Permalink
Post by Matt Reimer
Robert's on to something. It looks like your LBAs are probably
overflowing 32 bits. This would affect all vdev regardless of type.
Try the attached patch.
Post by Emil Smolenski
220000de400
This divided by 512 byte block size is 33 bits... At a glance, the patch
looks ok to me. I'll do a more thorough review of this tomorrow.
Ok, I was concerned about the assembly code... So, I've been chatting
with jhb@ this morning. Please try this patch that jhb@ came up with
instead of Matt's latest patch.

robert.
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS
ZFS: unexpected object set type 0
ZFS: unexpected object set type 0
FreeBSD/i386 boot
Default: pgpool:/boot/kernel/kernel
ZFS: unexpected object set type 0
src/sys/boot/i386/zfsboot
src/sys/boot/zfs
src/sys/cddl/boot/zfs
# cd /usr/src/sys/boot/
# patch < /path/to/zfsboot.c.patch3
# make clean; make cleandir
# make obj ; make depend ; make
# cd i386/loader
# make install
# cd /usr/src/sys/boot/i386/zfsboot
# make install
# sysctl kern.geom.debugflags=16
# dd if=/boot/zfsboot of=/dev/da0 count=1
# dd if=/boot/zfsboot of=/dev/da0 skip=1 seek=1024
# reboot
4. Result: error shown above.
Thanks!
--
Robert Noland <***@FreeBSD.org>
FreeBSD
John Baldwin
2009-11-19 16:55:10 UTC
Permalink
Post by Robert Noland
Post by Matt Reimer
Robert's on to something. It looks like your LBAs are probably
overflowing 32 bits. This would affect all vdev regardless of type.
Try the attached patch.
Post by Emil Smolenski
220000de400
This divided by 512 byte block size is 33 bits... At a glance, the patch
looks ok to me. I'll do a more thorough review of this tomorrow.
Ok, I was concerned about the assembly code... So, I've been chatting
instead of Matt's latest patch.
Actually, I had missed updating one place, please use this instead. Also, I
think that this will fix using > 2TB volumes even in the GPT case as
zfsboot.c was always using 32-bit LBAs even for the GPT case.
--
John Baldwin
John Baldwin
2009-11-20 12:43:51 UTC
Permalink
Post by Robert Noland
Ok, I was concerned about the assembly code... So, I've been chatting
instead of Matt's latest patch.
Actually, I had missed updating one place, please use this instead.
Also, I
think that this will fix using > 2TB volumes even in the GPT case as
zfsboot.c was always using 32-bit LBAs even for the GPT case.
Thanks a million! Both patches works for me. Great work!
I know that we have missed the boat but maybe there is opportunity to
catch it up by swimming and commit these patches to 8-STABLE before
8.0-RELEASE? Thanks!
It is too late for 8.0 I'm afraid.
--
John Baldwin
Continue reading on narkive:
Loading...