I have been lucky. I have had very few critical hard drive failures in my life. But as time goes on rolling those dices will be more risky so.. Lets see how hard is to set up RAID array.
In this small write up I will show how to set up RAID6 array with 4 drives.
Hardware that we will use is:
RaspberryPi 3B+
USB HUB
5x USB memory sticks (Any size will work).
Before we start learn from my failures.
- Make sure that your USB hub provides enough power for your memory sticks.
- USB and/or USB hub is not reliable, in most cases, hardware to hold your RAID array.
- On RaspberryPi you may need to play around with different distributions.
The amount of failures while doing this small project was just crazy. Memory sticks stop responding, hub not working correctly and everything being finicky overall. So this setup is not recommended for real data in any shape or form.
But it is best setup to test and learn about RAID because everything and anything that can go wrong will go wrong. So you can learn to rebuild and recover your arrays.
Ok lets get started
- We will use ubuntu 18.04.3 server for raspberry pi. (ubuntu-18.04.3-preinstalled-server-arm64+raspi3.img). I started with version 19.10 but for some reason RPi hanged couple of seconds after starting to create array.
Install the os and set it up. Updates, ssh if you like, networking and the whole nine yards. - Insert 4 of the usb drives in to the hub.
- Now check that OS sees the drives. You should see devices sda, sdb, etc:
ubuntu@ubuntu:~$ sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 28.7G 0 disk └─sda1 8:1 1 13.7G 0 part sdb 8:16 1 29G 0 disk └─sdb1 8:17 1 13.7G 0 part sdc 8:32 1 28.7G 0 disk └─sdc1 8:33 1 13.7G 0 part sdd 8:48 1 14.6G 0 disk └─sdd1 8:49 1 13.7G 0 part sde 8:64 1 28.7G 0 disk └─sde1 8:65 1 13.7G 0 part mmcblk0 179:0 0 29.7G 0 disk ├─mmcblk0p1 179:1 0 256M 0 part /boot/firmware └─mmcblk0p2 179:2 0 29.5G 0 part /
- My drives are already partitioned but lets take a look how to partition drives. All the partitions you include in the array have to be same size. Because one of my drives was smaller (14.6GB) I decided to create 13.7GB partitions on all the drives.
Run:ubuntu@ubuntu:~$ sudo fdisk /dev/sda Welcome to fdisk (util-linux 2.31.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): n Partition type p primary (0 primary, 0 extended, 4 free) e extended (container for logical partitions) Select (default p): p Partition number (1-4, default 1): First sector (2048-60063743, default 2048): Last sector, +sectors or +size{K,M,G,T,P} (2048-60063743, default 60063743): 28674047 Created a new partition 1 of type 'Linux' and of size 13.7 GiB. Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks.you have to do it to all the drives (/dev/sda, /dev/sdb …) you want to include in the array.
- After that you get drives and partitions that look similar to this:
ubuntu@ubuntu:~$ sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 28.7G 0 disk └─sda1 8:1 1 13.7G 0 part sdb 8:16 1 29G 0 disk └─sdb1 8:17 1 13.7G 0 part sdc 8:32 1 28.7G 0 disk └─sdc1 8:33 1 13.7G 0 part sdd 8:48 1 14.6G 0 disk └─sdd1 8:49 1 13.7G 0 part sde 8:64 1 28.7G 0 disk └─sde1 8:65 1 13.7G 0 part mmcblk0 179:0 0 29.7G 0 disk ├─mmcblk0p1 179:1 0 256M 0 part /boot/firmware └─mmcblk0p2 179:2 0 29.5G 0 part /
- Command for creating array is quite simple. You need to indicate the name/device of the array (dev/md0) how many drives/devices you will have in the array (–raid-devices=4), raid level (–level=6) and partitions/devices (/dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1)
depending on your drive letters you may need to change them. If all looks good run:ubuntu@ubuntu:~$ sudo mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 512K Continue creating array? Continue creating array? (y/n) y mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started.
- Now the array is being built. To check the status rund:
ubuntu@ubuntu:~$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid6 sdd1[3] sdc1[2] sdb1[1] sda1[0] 43920384 blocks super 1.2 level 6, 512k chunk, algorithm 2 [5/5] [UUUU] [>....................] resync = 0.8% (118532/14640128) finish=82.7min speed=2924K/sec unused devices: <none>Or you can monitor the progress with the following command
watch -d cat /proc/mdstat
This will take time and depending on the size and count of drives the waiting can get long.
- If build has finished you can check the status:
ubuntu@ubuntu:/$ sudo mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Mon Dec 9 11:57:54 2019 Raid Level : raid6 Array Size : 28653568 (27.33 GiB 29.34 GB) Used Dev Size : 14326784 (13.66 GiB 14.67 GB) Raid Devices : 4 Total Devices : 4 Persistence : Superblock is persistent Update Time : Mon Dec 9 20:05:29 2019 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Consistency Policy : resync Name : ubuntu:0 (local to host ubuntu) UUID : 9970b426:1cd1ba78:7d4fd643:81a4753a Events : 17 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 49 3 active sync /dev/sdd1 - Now we need to create a file system on to the new array device (/dev/md0). I’m using ext4 in this example.
ubuntu@ubuntu:/media$ sudo mkfs.ext4 /dev/md0 mke2fs 1.44.1 (24-Mar-2018) Creating filesystem with 7163392 4k blocks and 1794048 inodes Filesystem UUID: b80f7697-902b-46ba-aa81-1190557bb7f0 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done
- Next we create a mound point for the array. I used my /media folder and used name “raid6” for mount folder. But you are free tho choose.
sudo mkdir /media/raid6
- Lets add the mount point to the fstab. Open fstab file:
sudo nano /etc/fstab
and add the following line:
/dev/md0 /media/raid6/ ext4 defaults 1
- Next lets mount the array with mount command:
sudo mount -a
- Now if you run:
ubuntu@ubuntu:/$ df -h Filesystem Size Used Avail Use% Mounted on udev 441M 0 441M 0% /dev tmpfs 92M 4.7M 87M 6% /run /dev/mmcblk0p2 29G 2.2G 26G 8% / tmpfs 457M 0 457M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 457M 0 457M 0% /sys/fs/cgroup /dev/mmcblk0p1 253M 95M 158M 38% /boot/firmware tmpfs 92M 0 92M 0% /run/user/1000 /dev/md0 27G 45M 26G 1% /media/raid6
You should see your mounted array.
Optional
- Right now your mound point is with root permissions so if you want to use with non root users run:
sudo chown -r ubuntu:ubuntu /media/raid6
And that’s it.
lmao I’d never think of building a RAID6 on a raspberry pi with 4 USB HDs to see how they fail! 😀
Hi. You are right. This would be a bad idea for critical stuff. But why I did this and the tutorial is because it is so finiky.
This will give valuable lessons how to handle hardware fails, disconnects, recoveries and so on.
So yeah please don’t use this in production xD