Too many mails on this thread and I was away for the day.
Anyway to sum up :
indeed zfs is clever enough to balance the parity between all disks so that
even with 4 disks if you delete 1 disk you don't loose anything.
BUT
there is a bug at least in zfs-fuse-0.5.1 which makes it to say "can't
import the pool because 1 disk is currently unavailable" when you just
deleteed a disk.
This bug might very well be gone in 0.5.0.
Now there is a serious bug in the current mercurial repository when testing
with raidz1 pools and deleting one of the disks this way. When you try to
import the pool after that, the zpool command just hangs for ever with out
of memory errors related to put_nvlist in syslog.
So the culprit was easy to find : put_nvlist.
For some reason when you put back the line which was commented in an old
patch, it works again.
And the surprise is that this patch was supposed to fix the disappearing
volumes bug, but even with the line uncommented the test-datasets script is
still working.
And the good news is that it's working ok with 4 disks in this case, it
never complains.
I'll need more time to look into this...
BUT anyway, if you create a pool with 3 disks initially and then force the
addition of only 1 disk (you need the -f flag for this), then the disk is
added outside the raidz pool (as a normal disk), so if you fill your pool
enough to use this last disk and then remove the disk, the pool will become
impossible to import with the message "can't import pool 1 disk currently
unavailable".
So for this to work the pool must be created from the start with 4 disks,
which makes sense.
2009/8/26 ssc <steven.samuel.cole-***@public.gmane.org>
>
> Hey VL,
>
> just another piece of advice: When importing zpools, use import -d
> <folder>, e.g. sudo zpool import -d /dev/disk/by-id
> When I just rebooted from FreeBSD back into Kubuntu, zfs-fuse imported
> all zpools automatically (which it should not do), of course without
> the -d parameter, so all drives were listed in zpool status as sda,
> sdb, sdc, etc. Next time I booted, the IDE & SATA controllers were in
> a different order, so all sda sdb sdc etc drive names referred to
> different drives and all zpools were reported as this:
>
> pool: zpool01
> state: UNAVAIL
> status: The pool is formatted using an older on-disk format. The pool
> can
> still be used, but some features are
> unavailable.
> action: Upgrade the pool using 'zpool upgrade'. Once this is done,
> the
> pool will no longer be accessible on older software
> versions.
> scrub: none
> requested
> config:
>
> NAME STATE READ WRITE CKSUM
> zpool01 UNAVAIL 0 0 0 insufficient replicas
> raidz1 UNAVAIL 0 0 0 corrupted data
> sdi ONLINE 0 0 0
> sdj ONLINE 0 0 0
> sdk ONLINE 0 0 0
> sdl ONLINE 0 0 0
>
> pool: zpool02
> state:
> UNAVAIL
> status: The pool is formatted using an older on-disk format. The pool
> can
> still be used, but some features are
> unavailable.
> action: Upgrade the pool using 'zpool upgrade'. Once this is done,
> the
> pool will no longer be accessible on older software
> versions.
> scrub: none
> requested
> config:
>
> NAME STATE READ WRITE
> CKSUM
> zpool02 UNAVAIL 0 0 0 insufficient
> replicas
> raidz1 UNAVAIL 0 0 0 corrupted
> data
> sde ONLINE 0 0
> 0
> sdf ONLINE 0 0
> 0
> sdg ONLINE 0 0
> 0
> sdh ONLINE 0 0
> 0
>
> pool:
> zpool03
> state:
> UNAVAIL
> status: One or more devices could not be used because the label is
> missing
> or invalid. There are insufficient replicas for the pool to
> continue
>
> functioning.
> action: Destroy and re-create the pool from a backup
> source.
> see: http://www.sun.com/msg/ZFS-8000-5E
> scrub: none
> requested
> config:
>
> NAME STATE READ WRITE CKSUM
> zpool03 UNAVAIL 0 0 0 insufficient replicas
> raidz1 UNAVAIL 0 0 0 insufficient replicas
> sda FAULTED 0 0 0 corrupted data
> sdb FAULTED 0 0 0 corrupted data
> sdc FAULTED 0 0 0 corrupted data
> sdd FAULTED 0 0 0 corrupted data
>
> Solution: export all zpools and re-import them with the -d switch. Now
> it looks like this:
>
> pool: zpool01
> state: ONLINE
> status: One or more devices has experienced an error resulting in data
> corruption. Applications may be affected.
> action: Restore the file in question if possible. Otherwise restore
> the
> entire pool from backup.
> see: http://www.sun.com/msg/ZFS-8000-8A
> scrub: scrub in progress for 0h8m, 2,69% done, 4h50m to go
> config:
>
> NAME STATE READ
> WRITE CKSUM
> zpool01 ONLINE
> 0 0 4
> raidz1 ONLINE
> 0 0 4
> disk/by-id/scsi-1ATA_ST3500320AS_9QM3CPW7 ONLINE
> 0 0 0
> disk/by-id/scsi-1ATA_ST3500320AS_9QM3QNW7 ONLINE
> 0 0 0
> disk/by-id/scsi-1ATA_ST3500320AS_9QM3P9VB ONLINE
> 0 0 0
> disk/by-id/scsi-1ATA_ST3500418AS_9VM2LMCV ONLINE
> 0 0 0
>
> errors: Permanent errors have been detected in the following files:
>
> zpool01:<0x3736a>
>
> pool: zpool02
> state: ONLINE
> status: The pool is formatted using an older on-disk format. The pool
> can
> still be used, but some features are unavailable.
> action: Upgrade the pool using 'zpool upgrade'. Once this is done,
> the
> pool will no longer be accessible on older software versions.
> scrub: none requested
> config:
>
> NAME
> STATE READ WRITE CKSUM
> zpool02
> ONLINE 0 0 0
> raidz1
> ONLINE 0 0 0
> disk/by-id/scsi-1ATA_SAMSUNG_SP2514N_S08BJ1JL417664
> ONLINE 0 0 0
> disk/by-id/scsi-1ATA_SAMSUNG_SP2514N_S08BJ1JL417657
> ONLINE 0 0 0
> disk/by-id/scsi-1ATA_SAMSUNG_SP2514N_S08BJ10YB21736
> ONLINE 0 0 0
> disk/by-id/scsi-1ATA_SAMSUNG_SP2514N_S08BJ10YB21737
> ONLINE 0 0 0
>
> errors: No known data errors
>
> pool: zpool03
> state: ONLINE
> status: The pool is formatted using an older on-disk format. The pool
> can
> still be used, but some features are unavailable.
> action: Upgrade the pool using 'zpool upgrade'. Once this is done,
> the
> pool will no longer be accessible on older software versions.
> scrub: none requested
> config:
>
> NAME
> STATE READ WRITE CKSUM
> zpool03
> ONLINE 0 0 0
> raidz1
> ONLINE 0 0 0
> disk/by-id/scsi-1ATA_SAMSUNG_HM160JC_S0CMJ10L800059
> ONLINE 0 0 0
> disk/by-id/scsi-1ATA_SAMSUNG_HM160JC_S0CMJ10L800069
> ONLINE 0 0 0
> disk/by-id/scsi-1ATA_SAMSUNG_HM160JC_S0CMJ10L805620
> ONLINE 0 0 0
> disk/by-id/scsi-1ATA_SAMSUNG_HM160JC_S0CMJ10L805605
> ONLINE 0 0 0
>
> errors: No known data errors
>
> Let's see if I will ever get rid of that last error message...
>
> On Aug 26, 10:42 pm, Steven Samuel Cole <steven.samuel.c...-***@public.gmane.org>
> wrote:
> > OK, I solved my problem. I exported the zpool under Linux, booted an
> > ancient FreeBSD system, re-imported it there, only this time the
> > resilvering worked, the replacing was complete and I was able to
> > delete the file in question. Export under FreeBSD, reboot Linux,
> > eventually get rid of the remaining issues and reports about permanent
> > errors. Another scrub is running now, ETA 8 hours.
> >
> > Why does zfs-fuse import zpools upon system startup that had been
> > exported previously ? That's Micro$oft style.
> > Also, zfs-fuse seg-faulted a couple of times. That's Micro$oft style,
> too.
> >
> > My conclusion from this incident: zfs-fuse might just about work in an
> > _ideal_ world, In the real world of broken cables and flaky
> > harddrives, it is just useless when it is needed most. I will add a
> > dual-boot FreeBSD or OpenSolaris installation to my machine.
> >
> > 2009/8/26 VL <val.l...-***@public.gmane.org>:
> >
> >
> >
> > > On Aug 25, 7:18 pm, ssc <steven.samuel.c...-***@public.gmane.org> wrote:
> > >> Why do you export your zpool ? Just as a last resort ? AFAIK, you do
> > >> that only when you want to move a zpool to a different machine or use
> > >> it under a new OS.
> >
> > > I export the pool to "unmount" and or take it "offline", so it's not
> > > in use, so I can power down the drives. Is there a better way to
> > > offline/umount/suspend use of a pool so it's devices can be power
> > > cycled? "offline" command seems to be directed towards a device.
> >
> > >> > # zpool import rstore2
> > >> > cannot import 'rstore2': one or more devices is currently
> unavailable
> >
> > >> I think that's because zfs is still looking for the old hd (which you
> > >> replaced). Unfortunately, both old and new hd are running under the
> > >> same name.
> >
> > > Correct, ZFS is looking for the flaky HD which was temporarily
> > > removed. However, it worked just fine when dealing with disk images
> > > (created with dd). It tells me it can be imported, and works with
> > > disk images. With physical disk drives, I get the same "it can be
> > > imported" message, but, errors out with that error.
> >
> > >> How about
> > >> ***@host:~$ sudo zpool replace <pool name> disk/by-id/<old hd name>
> > >> disk/by-id/<new hd name>
> > >> At least this did not result in the problems you describe.
> > >> <old hd name> won't be in there as a file anymore as the hd has been
> > >> removed, you basically have to re-assemble the correct name from the
> > >> hd type and serial number using the still existing drives as a
> > >> guideline.
> >
> > > In order to do this, I need to reinsert the flaky "sdi" device, power
> > > it on, attempt to "import" (as that's the only way zfs-fuse is letting
> > > me import it to "online" status), then remove the flaky sdi device
> > > 'hot' (never a good idea, but hey, I'm game at this point) and slide
> > > in the new replacement drive 'hot'. Since sdi was in use, the new
> > > drive should show up as sdl, and I'll try the replace with the disk/by-
> > > id.
> >
> > > Process...
> > > * flaky drive reinserted, unit powered on
> > > * rstore2 pool brought online
> > > * flaky drive sdi yanked (dang those drives are hot)
> > > * new drive inserted, shows up as sdl as expected
> > > * did zpool replace with /dev/disk/by-id/.. Zpool hanging.
> > > * opened new ssh window, "zpool status" command also hanging.
> > > * ctrl-c out of zpool replace. All zpool command still hanging.
> > > * kill -9 the process /sbin/zfs-fuse ; remove the pid file /var/run/
> > > zfs-fuse.pid
> > > * start zfs: /etc/init.d/zfs-fuse start
> > > * rstore2 there, imported, online, corrupted, both sdi and sdh not
> > > happy. There is a chance that the live subtraction/addition of a
> > > drive in sdi's slot caused sdh to also cease functioning properly.
> >
> > > # zpool status
> > > pool: rstore2
> > > state: UNAVAIL
> > > status: One or more devices could not be opened. There are
> > > insufficient
> > > replicas for the pool to continue functioning.
> > > action: Attach the missing device and online it using 'zpool online'.
> > > see:http://www.sun.com/msg/ZFS-8000-3C
> > > scrub: none requested
> > > config:
> >
> > > NAME STATE READ WRITE CKSUM
> > > rstore2 UNAVAIL 0 0 0 insufficient replicas
> > > raidz1 UNAVAIL 0 0 0 insufficient replicas
> > > sdf ONLINE 0 0 0
> > > sdg ONLINE 0 0 0
> > > sdh UNAVAIL 0 0 0 cannot open
> > > sdi UNAVAIL 0 0 0 cannot open
> >
> > > * export the pool ; do a status. Back to square one... it sees a
> > > degraded pool that 'can be imported' with 3 good drives and 1 bad
> > > sdi. It can't actually be imported through:
> >
> > > # zpool import
> > > pool: rstore2
> > > id: 762967251253940714
> > > state: DEGRADED
> > > status: One or more devices are missing from the system.
> > > action: The pool can be imported despite missing or damaged devices.
> > > The
> > > fault tolerance of the pool may be compromised if imported.
> > > see:http://www.sun.com/msg/ZFS-8000-2Q
> > > config:
> >
> > > rstore2 DEGRADED
> > > raidz1 DEGRADED
> > > sdf ONLINE
> > > sdg ONLINE
> > > sdk ONLINE
> > > sdi UNAVAIL cannot open
> > > # zpool import rstore2
> > > cannot import 'rstore2': one or more devices is currently unavailable
> >
> > > Something about Linux (ubuntu?) hardware devices that zfs-fuse can't
> > > handle.
> >
> > > I'm game to try anything else. Hmm, what if I formatted these 4x200gb
> > > drives as reiser or xfs ... then created a 200gb image (using dd),
> > > then linked the 4 drives' images to /root/zfs/drive#.dat ... then
> > > created the zfs pool with those disk images /root/zfs/drive*.dat ... I
> > > wonder if it would then instead use the zfs code for disk images
> > > instead of the code for devices ..?
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "zfs-fuse" group.
To post to this group, send email to zfs-fuse-/***@public.gmane.org
To unsubscribe from this group, send email to zfs-fuse+unsubscribe-/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/zfs-fuse?hl=en
-~----------~----~----~----~------~----~------~--~---