Formatting a Hard Drive with Linux, for Linux

I never thought I would struggle that much with an external Hard Drive (HD) formatting operation under Linux.
But I did, so here is this article, describing step-by-step what one shall do to smoothly format a HD under Linux, using the console only (it means no UI, neither a mouse. sigh)

However don't get too excited, the context of this article is quite narrow.

First of all it sticks to Linux Ext formats, Ext4 in the current use-case. Furthermore the Linux distrib is a light Raspbian (the one used by Raspbmc) on a Raspberry Pi. Finally this focuses on a single partition scheme only. I may explain partitionning  more into details in a further post if needed.

Before we start make sure you've got:

• A system running Linux (a Pi in this example)
• Basic knowledge in Linux bash command lines
• A USB HD that can be safely erased

First of all, open a bash prompt onto the machine which will be in charge of formatting the target HD, e.g. a remote ssh session, a command prompt from the Linux Graphical User Interface (GUI) such as X, or directly the command prompt given by your system like Arch Linux would do.

We'll assume we are logged as root. If you think it's bad practice, then simply prefix the following bash commands with sudo.

Plug your HD in and see how your system sees it:

fdisk -l

This should give you an output like:

root@raspbmc:~# fdisk -l
Disk /dev/mmcblk0: 15.9 GB, 15931539456 bytes
255 heads, 63 sectors/track, 1936 cylinders, total 31116288 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006e204
        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1               1      143360       71680    b  W95 FAT32
/dev/mmcblk0p2          151552    31115263    15481856   83  Linux

A few explanations about this output:

• The whole chunk between lines 2 and 10 would repeat for each disk detected by your system.
• Line 2 shows the actual Linux device against which your HD has been bound: it is know as mmcblk0. That's what we were looking for actually. We can also see the size of the device, which is almost 16 GB in this example. In this case, it is an SD card (already formatted).
  Let's call this device <myDevice> as a generic placeholder for the name you will find for your own device. However my output sample will still show mmcblk0 for the sake of the example.
• Lines 9 and 10 show the partitions that already exist, if any. Each of them has been bound to its own Linux device, respectively mmcblk0p1 and mmcblk0p2. It may happen if the disk you're using has been pre-formatted already.

We can now run the fdisk utility in interactive mode, i.e. the prompt now belongs to fdisk until you exit it (so no more need to prefix anything with sudo in this section):

fdisk /dev/<myDevice>

You should see something like this:

root@raspbmc:~# fdisk /dev/mmcblk0
Command (m for help):

As you can see, the prompt is not the bash one anymore, but the fdisk one.

Out of curiosity, have a look to the available commands:

m

You should see something like this:

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition’s system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

As we simply want to wipe out all the data on this HD, let's check if any partition exists already so we can get rid of it:

p

You should see something like this:

Command (m for help): p

Disk /dev/mmcblk0: 15.9 GB, 15931539456 bytes
255 heads, 63 sectors/track, 1936 cylinders, total 31116288 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006e204

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1               1      143360       71680    b  W95 FAT32
/dev/mmcblk0p2          151552    31115263    15481856   83  Linux

Which is exactly the output we had earlier, when running fdisk in non-interactive mode.
Delete existing partitions if needed:

d

You should see something like this:

Command (m for help): d
Partition number (1-4): 

Simply enter the number corresponding to the line of the partition you want to delete and hit enter. Make sure no partition remains (if needed) by printing the partition list again using the 'p' command.

Now we've wiped out the existing partitions (if any), let's create a new one:

n

You should see something like this:

Command (m for help): n
Partition type:
   p   primary (2 primary, 0 extended, 2 free)
   e   extended

We'll create a primary partition by default. Very roughly, we can only create 4 partitions, that's a hard limit. To work that around without breaking everything, some clever guys introduced the extended ones, which can be used to nest as many logical partitions as needed. This forum post gives a bit more details about this.
Once a partition type has been selected, simply follow the instructions to build your partition. Each prompt comes with a default value which all drive to a partition covering the whole disk space.

Now we've got our new partition, it will be automatically bound to a new device within your /dev folder. Let's see what device has been created (we'll name it <myDevicePartition>) by using the 'p' command again.
It is very important as from now on we are going to deal with the partition we've just created within the HD, not the HD itself. E.g. in this example case, /dev/mmcblk0 points to the HD, while /dev/mmcblk0p1 and /dev/mmcblk0p2 point to the respective existing partitions within the HD.

If you are happy with your work, ask fdisk to write changes and exit:

w

In case you've done rubbish, simply use the 'q' command so you don't save any changes on exit.

We are halfway from having a usable HD now. The disk is partitioned, meaning it has an internal structure able to host a file system (a single one in this example).
Let's format our partition:

mkfs.ext4 /dev/<myDevicePartition> 

In order to access our freshly formatted HD, we have to mount it, i.e. build an access point into the existing file system (the one hosting the bash you have used to partition your HD).

This access point is a simple directory. We'll create it into the /media directory for the sake of Linux good practices (assuming it does exist as it could also be named /mnt or /mount), but it could be any other really:

mkdir /media/<myMountPoint>

Where <myMountPoint> is a name of your choice.

We can now mount the device (which is the -only- partition of your HD)

mount -t ext4 /dev/<myFormatedDevice> /media/<myMountPoint>

Some systems automatically mount the HD they detect either at startup or on-the-fly. In case your system is not that clever, you can manually make this mount permanent.
You need to edit your /etc/fstab configuration file and add the following line:

/dev/<myFormatedDevice> /media/<myMountPoint> ext4 defaults 0 0

Other sources used for this article: http://superuser.com/a/643797 and fdisk help.

That's all folks!

Which SD cards for a Raspberry Pi?

Many types of Secure Digital (SD) cards exist to fit many different purposes. This article aims to help one spotting the differences and the way they are advertised to finally pick the SD card that will fit a given device (more specifically a Raspberry Pi).

Basics of SD cards

SD cards exist in various formats, namely regular, mini and micro.

From top to bottom: regular, mini and micro sizes

The format doesn't mean anything in terms of capacity (how much it can hold) and speed (how fast it reads and writes data).

Checking the capacity is trivial, the size is most of the time given in GB and may vary from a few GB to hundreds (and so does the price).

The trickier part is about speed: it is actually not precise enough, we should rather talk about:

• Reading speed
  How much data per second an SD card can read
• Writing speed
  How much data per second an SD card can write
• Read access latency
  How much time it takes an SD card before it can read data
• Write access latency
  How much time it takes an SD card before it can write data

Reading and writing speeds can be spotted via the following properties: 

1. Speed class
   It is simply the minimum writing speed an SD card can perform. The higher the faster, knowing it is expressed in MB, e.g. a class 6 SD card will never write non-fragmented data slower than 6 MB per second. This may change if the data written is fragmented, e.g. being many small files instead of a single one.
2. "x" rating
   Quoting wikipedia: 

The “×” rating, made obsolete by speed classes, is a multiple of the standard CD-ROM drive speed of 150 KB/s (approximately 1.23 Mbit/s) 

It is to be read carefully though: there is no rule regarding what it refers to, e.g. from a vendor to another it could be an average reading speed, the best reading speed, the best writing speed, or any combination of the above.
As most of the times (not to say always), a card will read data faster than it writes, it is safer to assume the "x" rating refers to the average reading speed rather than anything else.  

Read and write access latencies however cannot be spotted that easily, and you'll have to dig into the vendor documentation to check those details, if provided.
That's very unfortunate though, because this is what we need to pick the right SD card for our Pi, which leads to the next section.

Which SD card for my Pi?

Most of us would think "I'll take the fastest one" (commonly class 10 on the paper) to get the best results.

If you've read and understood the previous section then you may think this could be wrong:  the class (and in a certain way the "x" rating as well) gives an I/O rate, i.e. how fast the card is to read or write data.

However, before getting into this I/O you need to access the card and this takes a certain amount of time, and this time is not represented by the class neither the x speed.

This might not be big deal when it comes to read or write big chunks of data: the access latency would be negligible compared to the time it takes to write the data. However this latency would add up for each access the cards makes, so when it comes to access very small files, it may take ages even if the SD card is class 10.

Let's take a simple example with a class 10 SD card which would have a 1 second write access latency.
Class 10 means it can write 10 MB/s, i.e. it can write 10 GB (=10240 MB) of data in 10240/10=1024 s.
1 s latency means it will take the card 1 s each times it will begin to write a single file.
Write access latency adds up as follow:

• In the case of a single file (let's say a HD video), it is just a one off write access, so only 1 s latency added up to the write operation.
  It makes it 1024+1 seconds to write this 10 GB file, which is roughly half an hour.
• In the case of 10,240 files, each one 1 MB big , we still have 10 GB worth of data in total, but fragmented into 10,240 single files.
  Hence we would still have the same writing time, 1024 sec, but the added latency remains 1 s per file, i.e. 10,240 seconds.
  Total time for the 10GB write operations is now 1024+10240=11264 s, which is roughly more that 3 hours.

Conclusion: when it comes to manipulate small files, like Linux does, or like your MP3 collection is, the shorter the access latencies are, the better, regardless of the SD card class or "x" rating.

This very interesting article shows numbers for many different SD cards brands and classes. It also highlights the fact the highest class SD cards may not have the best latencies...

Regarding the Pi, it seems that a class 6 SD card is sufficient most of the times, according to the Pi capabilities. You may not need a higher class unless you're manipulating large files directly on the SD card instead of on an external HD for instance.

As a concrete example, this 8 GB class 6 SD card is fairly cheap and delivers good performances for most of the usages I've got for my Pi, e.g. media-center and games emulation.


-That's all folks!