All pastes #983222 Raw Edit

ZFS-Fuse nightmare

public text v1 · immutable
·published 2008-04-13 04:26 UTC
rendered paste body
First off: Thanks a lot, Ricardo, for this awesome work and your enthusiasm!

Unfortunately, I have a serious problem, though. I needed big storage for my MythTV files, added 4 new disks to my fileserver, tried ZFS-FUSE with RAID-Z on them. Works beautifully, like a snap, amazing how simple it is. Now I make a serious mistake: Because I am out of space already (thus new disks), I decide to *move*, not copy, my recordings. OK, takes a whole night (8.5 hours) to copy 280 GB (the first bunch). When it finished I had to restart the machine. I start up zfs-fuse again, no FS appears, and I see:

# zpool status
 pool: media
state: FAULTED
scrub: none requested
config:

       NAME        STATE     READ WRITE CKSUM
       media       UNAVAIL      0     0     0  insufficient replicas
         raidz1    UNAVAIL      0     0     0  corrupted data
           sda     ONLINE       0     0     0
           sdb     ONLINE       0     0     0
           sdc     ONLINE       0     0     0
           sdd     ONLINE       0     0     0

DANG. No, I don't have a copy of the files (see above).
The ZFS admin guide does not say anything about such a situation, because ZFS is so safe that this never occurs, RIIIIIGHT? Sounds almost like Titanic.

I tried to share the one FS via NFS, both via sharenfs=on and /etc/exports / Linux nfsserver, FWIW (while the copy was ongoing). It's a Gentoo system. ZFS-FUSE 0.4_beta1 from the ebuild.

5 (!) hours of searching on the web later, without any success, I realized that I had fuse compiled as module and loaded it manually, so it was of course gone after a reboot. Confirmed with lsmod, no fuse was loaded.

Well, at this time back then, I didn't realize the module yet, so I tried this:
# zpool export media   # <---
# zpool status
no pools available
# zpool import
 pool: media
   id: 18446744072804078091
state: FAULTED
action: The pool cannot be imported due to damaged devices or data.
config:
       media       UNAVAIL   insufficient replicas
         raidz1    UNAVAIL   corrupted data
           sda     ONLINE
           sdb     ONLINE
           sdc     ONLINE
           sdd     ONLINE
# zpool import media
cannot import 'media': pool may be in use from other system
use '-f' to import anyway
# killall zfs-fuse
# reboot
# zpool status
no pools available
# zpool import media
cannot import 'media': pool may be in use from other system
use '-f' to import anyway
# zpool import -f media
cannot import 'media': one or more devices is currently unavailable
# zpool status
no pools available
# cfdisk /dev/sda
                             Disk Drive: /dev/sda
                      Size: 500107862016 bytes, 500.1 GB
                           Pri/Log   Free Space                      500105.25
Same for sdb/c/d
This has always been like that, even when the pool was still online (I never rebooted, though, IIRC). I was first shocked, but then thought that ZFS is from Solaris and probably doesn't use the MSDOS partition tablem and I use the raw device, so the "no partition" seems normal, despite being unusual on PCs.

When I do less -f /dev/sda, I see "raidz", "/dev/sdb" etc, after lots of glibberish, and I can successfully do grep someknownfilename -a|less, so it seems that things were indeed written, and to this disk.

OK, back to present. I loaded fuse, but it didn't help. Restarted zfs-fuse, nothing. Rebooted the machine (with fuse autoloaded - confirmed! -, this time), no difference. I still get:
# zpool import
 pool: media
   id: 18446744072804078091
state: FAULTED
action: The pool cannot be imported due to damaged devices or data.
config:
       media       UNAVAIL   insufficient replicas
         raidz1    UNAVAIL   corrupted data
           sda     ONLINE
           sdb     ONLINE
           sdc     ONLINE
           sdd     ONLINE
# zpool import media
cannot import 'media': pool may be in use from other system
# zpool import -f media
cannot import 'media': one or more devices is currently unavailable

Of course, the ZFS admin guide again says nothing about this.

What can I do to get the thing online again? Again, the data seems fine, and it's my only copy of it.

Thanks in advance. But even though I of course need help to get my data back, it's even more important to change the software so that people won't run into such nightmare anymore.