{"id":595,"date":"2019-09-30T08:47:31","date_gmt":"2019-09-30T12:47:31","guid":{"rendered":"http:\/\/www.mbeckler.org\/blog\/?p=595"},"modified":"2019-09-30T08:48:49","modified_gmt":"2019-09-30T12:48:49","slug":"zfs-plans-and-logs-2x-mirroring-vdevs-with-future-expansion","status":"publish","type":"post","link":"https:\/\/www.mbeckler.org\/blog\/?p=595","title":{"rendered":"ZFS plans and logs: 2x mirroring VDEVs with future expansion"},"content":{"rendered":"<p>I have a small linux server at home, chiefly for file storage and running a plex server. In the past, I&#8217;ve used a Linux software RAID-1 (mirror) of two drives to provide more-robust storage for important files (documents, photos, etc), and a single drive for media files (that could be re-backed-up from their original CD\/DVD media, if lost). Recently I wanted to transition to a combined ZFS storage array for everything, for the following reasons:<\/p>\n<ul>\n<li>Filesystem checksums (during normal read\/write operation, plus weekly scheduled full-array scrubs) to ensure that there is no &#8220;bit rot&#8221; where files silently change over time.<\/li>\n<li>Full mirroring protection of all files<\/li>\n<li>In-place expansion capability<\/li>\n<\/ul>\n<p><strong>Research references<\/strong><\/p>\n<ul>\n<li><a href=\"https:\/\/www.freebsd.org\/doc\/handbook\/zfs-zpool.html\">FreeBSD has some of the best ZFS documentation<\/a><\/li>\n<\/ul>\n<p><strong>After discussions of my needs and capabilities with a ZFS &#8220;expert&#8221; friend, here&#8217;s the plan I decided to go with:<\/strong><\/p>\n<ul>\n<li>Create a single ZFS pool (top-level filesystem)<\/li>\n<li>This ZFS pool will consist of two virtual device (VDEVs in ZFS parlance), where each VDEV is a &#8220;mirroring&#8221; VDEV, where all drives in each VDEV are mirrored with each other, providing RAID-like drive redundancy within each VDEV.<\/li>\n<li>Set up <code>cron<\/code> jobs to periodically scrub the ZFS pool to verify the checksums and ensure the pool is in good health.<\/li>\n<li>Initially set up 2x 4TB drives in one VDEV, and 2x 2TB drives in the other VDEV, resulting in 6 TB total storage in the pool.<\/li>\n<li>When more space is required:\n<ul>\n<li>Add 2x larger drives to the smaller VDEV<\/li>\n<li>Wait for the VDEV to resilver [1]<\/li>\n<li>Remove the smaller drives from the VDEV<\/li>\n<li>Expand the VDEV to the size of the new larger drives<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><strong>To help plan this out, and learn the ZFS terminology, I created a series of statements about ZFS:<\/strong><\/p>\n<ol>\n<li>A ZFS pool is made from one or more virtual devices (VDEV), which are, in our cases, 2+ physical drives mirrored together.<\/li>\n<li>A ZFS pool expands when its existing VDEVs become larger, or by adding another VDEV, but you can never remove a VDEV from a pool (but you can *replace* a VDEV, which is swapping one VDEV out for another, using <code>zpool replace<\/code>).<\/li>\n<li>You can add a new drive to a mirroring VED and it&#8217;ll &#8220;re-silver&#8221; and add that new drive to the mirroring, slowly over time.<\/li>\n<li>You can remove a drive from a mirroring VDEV and it keeps going. Of course, if you remove the last drive of a mirroring VDEV, it can&#8217;t keep going.<\/li>\n<li>An <code>autoexpand=on<\/code> mirror VDEV expands when all the member drives are large enough. Or turn off autoexpand and do it manually using <code>online -e<\/code><\/li>\n<li><code>zpool replace<\/code> lets you switch the internal architecture of the VDEV (like switch from RAIDZx to\/from mirror). If you just add new big drives to the existing VDEV, let them recover, and then remove the old smaller drives, it of course stays as a mirroring VDEV.<\/li>\n<\/ol>\n<p><strong>Here are the commands used to build, maintain, and expand the ZFS pool:<\/strong><\/p>\n<ul>\n<li>Create the zpool from two mirrors of two drives each:\n<ul>\n<li>\n<pre>sudo zpool create mypool mirror \/dev\/sda \/dev\/sdb mirror \/dev\/sdc \/dev\/sdd<\/pre>\n<\/li>\n<li>Note: The command above uses the <code>\/dev\/sdX<\/code> device names, which may changed based on device initialization order at boot time, so it&#8217;s strongly suggested to instead use the device files in <code>\/dev\/disk\/by-id\/<\/code> which will not change.<\/li>\n<\/ul>\n<\/li>\n<li><span style=\"font-size: inherit;\">Create the ZFS filesystem mount point in the zpool:<\/span>\n<ul>\n<li>\n<pre><span style=\"font-size: inherit;\">sudo zfs create mypool\/zfs<\/span><\/pre>\n<\/li>\n<\/ul>\n<\/li>\n<li>Set the mountpoint:\n<ul>\n<li>\n<pre>sudo zfs set mountpoint=\/whatever mypool\/zfs<\/pre>\n<\/li>\n<li>Note that you don&#8217;t need to put anything into <code>\/etc\/fstab<\/code> for this ZFS pool mountpoint, it&#8217;ll be mounted automatically when ZFS starts up at boot.<\/li>\n<li>I don&#8217;t know how to use ZFS for your boot drive (<code>\/<\/code>) as I only use it for non-OS data.<\/li>\n<\/ul>\n<\/li>\n<li>Add an optional drive for ZFS intent log:\n<ul>\n<li>\n<pre>sudo zpool add -f mypool log \/dev\/nvme0n1<\/pre>\n<\/li>\n<li>A friend loaned me a nifty PCIe SSD to experiment with, so I added it to store my ZFS intent log (much like the EXT journal). I don&#8217;t think my typical &#8220;frequent-read with rare-writes&#8221; workload really take advantage of this cool device, it was mostly a fun experiment.<\/li>\n<\/ul>\n<\/li>\n<li><span style=\"font-size: inherit;\">Add these entries to root&#8217;s crontab with <code>sudo crontab -e<\/code>:<\/span>\n<ul>\n<li>\n<pre><span style=\"font-size: inherit;\"># Every Monday at 00:00, start a ZFS scrub<br \/><\/span>0 0 * * 1 \/sbin\/zpool scrub mypool<br \/># Every Monday at 18:00 (6pm), send the zfs zpool status email<br \/>0 18 * * 1 \/sbin\/zpool status<\/pre>\n<\/li>\n<\/ul>\n<\/li>\n<li>When we want to upgrade the 2 x 2 TB drives to 2 x 4 TB drives:\n<ul>\n<li>The syntax here is somewhat odd. We attach each new drive to one of the old smaller drives, which is how the new drives get added to the mirror.<\/li>\n<li>\n<pre>sudo zpool attach mypool -f oldDriveName1 newDriveName1<br \/>sudo zpool attach mypool -f oldDriveName1 newDriveName2<\/pre>\n<\/li>\n<li>Use <code>sudo zpool status<\/code> to monitor the resilvering process. You should see that the VDEV with the smaller drives now has the newly-added larger drives.<\/li>\n<li>Once resilver is complete, run a scrub (just in case), to confirm everything is working right and your data is safe.<\/li>\n<li>Remove the old drives:\n<ul>\n<li>\n<pre>sudo zpool detach mypool oldDriveName1<br \/>sudo zpool detach mypool oldDriveName2<\/pre>\n<\/li>\n<\/ul>\n<\/li>\n<li>Expand the new drives:\n<ul>\n<li>\n<pre>sudo zpool online -e mypool newDriveName1<br \/>sudo zpool online -e mypool newDriveName2<\/pre>\n<\/li>\n<\/ul>\n<\/li>\n<li>You can now use <code>sudo zpool list<\/code> or <code>df -h<\/code> to see that the pool has expanded in size, and now you can store more data!<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Hope this helps! Feel free to leave a comment if you notice any typos, or have any suggestions to add.<\/p>\n<p>[1]. To &#8220;re-silver&#8221; is what you&#8217;d do to an antique mirror when it became degraded and needed to be restored :-)<\/p>\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have a small linux server at home, chiefly for file storage and running a plex server. In the past, I&#8217;ve used a Linux software RAID-1 (mirror) of two drives to provide more-robust storage for important files (documents, photos, etc), and a single drive for media files (that could be re-backed-up from their original CD\/DVD [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-595","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p2BznB-9B","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/595","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=595"}],"version-history":[{"count":4,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/595\/revisions"}],"predecessor-version":[{"id":614,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/595\/revisions\/614"}],"wp:attachment":[{"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=595"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=595"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mbeckler.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=595"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}