{"id":140,"date":"2010-05-19T20:38:00","date_gmt":"2010-05-19T20:38:00","guid":{"rendered":"http:\/\/zensonic.dk\/?p=140"},"modified":"2017-04-09T10:03:56","modified_gmt":"2017-04-09T10:03:56","slug":"saved-by-zfs-a-disk-is-about-to-die","status":"publish","type":"post","link":"https:\/\/zensonic.dk\/?p=140","title":{"rendered":"Saved by ZFS &#8211; a disk is about to die"},"content":{"rendered":"<p>I have setup an NFS share under Solaris 10. It utilizes ZFS, which in turn uses\u00c2\u00a0 an IBM DS400 for backend storage. On top of that I have nagios running to monitor it. I got an alarm about the zfs pool being in a degraded state. I logged into the system and found this in the dmesg:<\/p>\n<pre>May 17 03:20:20 files DESC: The number of checksum errors associated with a ZFS device\r\nMay 17 03:20:20 files exceeded acceptable levels.\u00c2\u00a0 Refer to http:\/\/sun.com\/msg\/ZFS-8000-GH for more information.<\/pre>\n<p>To see more information I ran<\/p>\n<pre>-bash-3.00# zpool status\r\n pool: rz2pool\r\n state: DEGRADED\r\nstatus: One or more devices has experienced an unrecoverable error.\u00c2\u00a0 An\r\n attempt was made to correct the error.\u00c2\u00a0 Applications are unaffected.\r\naction: Determine if the device needs to be replaced, and clear the errors\r\n using 'zpool clear' or replace the device with 'zpool replace'.\r\n see: http:\/\/www.sun.com\/msg\/ZFS-8000-9P\r\n scrub: scrub completed after 5h21m with 0 errors on Wed May 19 08:41:49 2010\r\nconfig:\r\n\r\n NAME\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 STATE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 READ WRITE CKSUM\r\n rz2pool\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 DEGRADED\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n   raidz2\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd0\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd1\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd2\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd3\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd4\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd5\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd6\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n   raidz2\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 DEGRADED\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd7\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd8\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd9\u00c2\u00a0\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd10\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd11\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n     c3t21000000D12643DEd12\u00c2\u00a0 DEGRADED\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0 234\u00c2\u00a0 too many errors\r\n     c3t21000000D12643DEd13\u00c2\u00a0 ONLINE\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0\u00c2\u00a0 0\r\n\r\nerrors: No known data errors\r\n<\/pre>\n<p>And this is where ZFS is awsome. It may not be the fastest volume manager on the planet, or the smartest. But I trust the integrity of it (having read whitepapers on it).<\/p>\n<p>What is really cool here:<\/p>\n<ol>\n<li>It has detected that the underlying LUN is misbehaving.<\/li>\n<li>It has marked the LUN as degraded<\/li>\n<li>It has saved my data from silent corruption.<\/li>\n<\/ol>\n<p>There are not many volume managers out there, which does that. I have not lost data, the dataintegrity is still in place and I know what disk is about to fail. Kudos and thanks to the ZFS dev team!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have setup an NFS share under Solaris 10. It utilizes ZFS, which in turn uses\u00c2\u00a0 an IBM DS400 for backend storage. On top of that I have nagios running to monitor it. I got an alarm about the zfs pool being in a degraded state. I logged into the system and found this in [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,13],"tags":[],"class_list":["post-140","post","type-post","status-publish","format-standard","hentry","category-volume-managers","category-zfs"],"_links":{"self":[{"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/posts\/140"}],"collection":[{"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/zensonic.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=140"}],"version-history":[{"count":0,"href":"https:\/\/zensonic.dk\/index.php?rest_route=\/wp\/v2\/posts\/140\/revisions"}],"wp:attachment":[{"href":"https:\/\/zensonic.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=140"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zensonic.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=140"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zensonic.dk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=140"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}