Filesystem Commands & Notes
October 25, 2019 | Comments
I’ve been digging into using AWS’ fairly new NVMe SSDs for some performance tests with the goal of tarring thousands of files and multiple gigabytes in under a second. Because these operations are fairly fast on my local laptop NVMe SSD, and it turns out the SSD EBS volumes, which are not local, are pretty slow compared to my local. Just documenting some commands here, like a cheat sheet that I can use later. Out of the box a c5d.2xlarge (‘d’ = enhanced (d)isk) EC2 instance has a locally attached 200GB NVMe SSD. This allows for the lowest possible latency, however, it does not come with a filesystem and therefore isn’t mounted on boot. The below commands are how to take a raw volume, put a filesytem on it and then mount it for testing.
First, let’s list out our filesystems, their size and mount path with df --human:
[root@ip-10-247-114-103 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.6G 64K 7.6G 1% /dev
tmpfs 7.6G 0 7.6G 0% /dev/shm
/dev/nvme0n1p1 985G 8.9G 976G 1% / We don’t see our superfast local SSD mounted above.
All gp2 and io1 EBS volumes are mounted as /dev/nvme* devices (some have pointers from /dev/sdb etc). Here we have our root gp2 general purpose SSD volume, /dev/nvme0n1p1 mounted at / for the entire system to run on. But, we don’t see our real NVMe SSD though, nor another io1 volume I attached for testing. These raw volumes both will need filesystems created before they can become useable. I will only be targeting nvme2 for these examples, the fast one.
Let’s list out our attached volumes with ls -l (-l = ‘long format/more info, no long-option):
[root@ip-10-247-114-103 ~]# ls -l /dev/nvme*
crw------- 1 root root 250, 0 Oct 24 23:45 /dev/nvme0 # This is the root gp2 volume.
brw-rw---- 1 root disk 259, 2 Oct 24 23:46 /dev/nvme0n1
brw-rw---- 1 root disk 259, 5 Oct 24 23:46 /dev/nvme0n1p1
brw-rw---- 1 root disk 259, 6 Oct 24 23:45 /dev/nvme0n1p128
crw------- 1 root root 250, 1 Oct 24 23:45 /dev/nvme1 # A io1 volume I have attached for testing.
brw-rw---- 1 root disk 259, 1 Oct 24 23:45 /dev/nvme1n1
crw------- 1 root root 250, 2 Oct 24 23:45 /dev/nvme2 # The _real_, local NVMe SSD.
brw-rw---- 1 root disk 259, 0 Oct 26 00:48 /dev/nvme2n1Next, blockdev --getsize64 dev/nvme2n1 to tell us how many blocks are on our attached volume, so we can verify that the 200GB volume that I provisioned is actually the nvme2 because of the 200000000000:
[root@ip-10-247-114-103 ~]# blockdev --getsize64 /dev/nvme2n1
200000000000 lsblk appears to be a bit better for checking attached volume size as it lists human-readable form by default. It also lists partitions if they exist.
[root@ip-10-247-114-103 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:2 0 1000G 0 disk
├─nvme0n1p1 259:5 0 1000G 0 part /
└─nvme0n1p128 259:6 0 1M 0 part
nvme2n1 259:0 0 186.3G 0 disk
nvme1n1 259:1 0 70G 0 disk mount --show-labels - This is the infamous mount -l, which I always thought meant --list but it doesn’t, and is the reason why all documentation should always use “—long-options”, they are self-documenting. Sample output:
[root@ip-10-247-114-103 ~]# mount --show-labels
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=7908016k,nr_inodes=1977004,mode=755)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime)
/dev/nvme0n1p1 on / type ext4 (rw,noatime,data=ordered) [/]
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)Next is the file --special-files /dev/nvme2n1 command that we use to tell us if there is a filesystem on the volume. Here is what it looks like if there is no filesystem.
[root@ip-10-247-114-103 ~]# file --special-files /dev/nvme2n1
/dev/nvme2n1: dataHere we make an XFS filesystem with mkfs --type xfs /dev/nvme2n1 (yum install xfsprogs --yes if the prior command throws an error):
[root@ip-10-247-114-103 ~]# mkfs --type xfs /dev/nvme2n1
meta-data=/dev/nvme2n1 isize=512 agcount=4, agsize=12207032 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=48828125, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=23841, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0Now we have a filesystem and we verify with file --special-files /dev/nvme2n1:
[root@ip-10-247-114-103 ~]# file --special-files /dev/nvme2n1
/dev/nvme2n1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)lsblk shows that even though we have a filesystem, there still isn’t a partition on /dev/nvme2n1, and we don’t actually need one:
[root@ip-10-247-114-103 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:2 0 1000G 0 disk
├─nvme0n1p1 259:5 0 1000G 0 part /
└─nvme0n1p128 259:6 0 1M 0 part
nvme2n1 259:0 0 186.3G 0 disk
nvme1n1 259:1 0 70G 0 disk [root@ip-10-247-114-103 ~]# fdisk -l
Disk /dev/nvme2n1: 200.0 GB, 200000000000 bytes, 390625000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/nvme1n1: 75.2 GB, 75161927680 bytes, 146800640 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.
Disk /dev/nvme0n1: 1073.7 GB, 1073741824000 bytes, 2097152000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt
# Start End Size Type Name
1 4096 2097151966 1000G Linux filesyste Linux
128 2048 4095 1M BIOS boot parti BIOS Boot PartitionBelow you can see that ebsnvme-id tells us if we have EBS devices or not:
[root@ip-10-247-114-103 ~]# /sbin/ebsnvme-id /dev/nvme2n1
[ERROR] Not an EBS device: '/dev/nvme2n1'
[root@ip-10-247-114-103 ~]# /sbin/ebsnvme-id /dev/nvme1n1
Volume ID: vol-03a0af9a8cad9be07
sdb
[root@ip-10-247-114-103 ~]# /sbin/ebsnvme-id /dev/nvme0n1
Volume ID: vol-0350248eed34cb730
xvda And, in the spirit of not publishing this without finishing it because if I don’t publish it right now then it may never get published because kids are begging me “how many minutes are you going to be done working” right this minute and taking my headphones off to ask it as this is the weekend coming up and it is Friday night I am going to publish this in its confusing imperfections and touch it up later.
@todo Fix it to be accurate
@todo Put cheatsheat at top