Setting up a Dynamic Storage unit to the Data node in HDFS Cluster

Hadoop and LVM

Nowadays, data are generated and updated per second. In this Big data world, Some times We can’t even predict how much data we will get in the future. For storing these massive amounts of data, typically we are using Clustering technologies, so such technique is Hadoop Distributed File System (HDFS). Hadoop is an open-source Java-based framework used for storing data and running applications on clusters of commodity hardware.

Since the size of data is exponentially growing, to store these data efficiently and effectively, storage should possess some special characteristics in nature namely Dynamic Storage.

In this article, we are going to see how to set up a Dynamic Storage unit to the Data node in HDFS Cluster.

For those who are not familiar with Hadoop and HDFS cluster, I have already written some articles. You can find them in my profile.

To setting up a Dynamic Storage unit in HDFS Cluster, we use one important concept in Linux. Yes, it is none other than LVM.

In Linux, Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume.

OK, let’s go step by step. In this practical, I used Redhat Enterprise Linux (RHEL8) OS and Oracle VM VirtualBox which is cross-platform virtualization software that allows users to extend their existing computer to run multiple operating systems at the same time.

Step 1: Creation of new hard disk

We can’t increase the storage with the existing amount of storage in hand. So We have to create additional storage. For this, Oracle Virtualbox provided one feature namely Virtual Hard disk which looks the same as a real hard disk but it is not. I already created two Virtual hard disks which shown below.

I am clearly explained step by step about creating a virtual hard disk in my previous article. You can find this by clicking here.

You can also verify whether the hard disk is there or not by running the following command in the command line.

fdisk -l

In the above image, you can find our newly created virtual hard disk namely /dev/sdb and /dev/sdc which has a size of 50 GiB and 40 GiB.

Step 2: Creation of LVM

The creation of LVM involves the following steps namely

• Creation of Physical Volumes (PV)
• Creation of Volume Groups (VG)
• Creation of Logical Volumes (LV)
• Logical Volume formatting
• Mounting Logical Volumes

Creation of Physical Volumes

We have to run “pvcreate” command to initialize a block device to be used as a physical volume.

The following command initializes /dev/sdband /dev/sdcas LVM physical volumes for later use as part of LVM logical volumes. You can view the Physical volumes by running pvdisplay command.

In future, if you want to remove the physical volumes, you have to run the following command.

pvremove diskname

Creation of Volume Groups (VG)

To create a volume group from one or more physical volumes, use the vgcreate command. The vgcreate command creates a new volume group by name and adds at least one physical volume to it.

vgcreate myvg /dev/sdb /dev/sdc

The above commands created one Volume Group named “myvg” which comprises of /dev/sdb and /dev/sdc volumes. You can also view the further details by using vgdisplay vgnamecommand.

By default, the block size of Volume Groups is fixed as 4MiB but we can change according to our requirements.

In the future, if you want to remove the Physical Volumes from a Volume Group, you have to run the following command

vgreduce vgname pvname

Creation of Logical Volumes (LV)

We can create the Logical volume by using lvcreate command. We can also create one Logical volume with 88% of the total size of Volume Groups. When you create a logical volume, the logical volume is carved from a volume group using the free extents on the physical volumes that make up the volume group.

Here, we created one Logical Volume named “mylv1” with the size of “myvg” size.

You can also view the further details of the logical volume by running lvdisplay command.

Normally logical volumes use up any space available on the underlying physical volumes on a next-free basis. Modifying the logical volume frees and reallocates space in the physical volumes.

Format

If you take any hard disk, without done formatting, we can't use that space even it has free space. The format is also known as Filesystem, which has an Inode table that acts as an Index table for OS operations. Here, the format is ext4.

mkfs.ext4 /dev/myvg/mylv1

Mount

It is not possible to use the physical device directly. You have to mount to one folder to use.

mount /dev/myvg/mylv1 /lvm2

We mounted that Logical Volume into one directory named “lvm2".

You can view this mount by running df -hT .

Step 3: HDFS Cluster Configuration

We already discussed this configuration in my previous article. You can check this by clicking here.

Then we have to update this directory name in datanode’s hdfs-site.xml file.

Then start the name node and data node.

Step 4: Increase the size of the data node

We know that 10GiB space is available in our Volume group. We can utilize that space to increase the size of the data node in the HDFS Cluster.

Steps:

• lvextend
• format
• mount

We have to extend the Logical volume size by using lvextend command.

lvextend --size +5G /dev/myvg/mylv1

We extended logical volume by an extra 5GiB.

Then we have to format the remaining space (5GiB) by using resize2fs command because format command will format the total hard disk again, so there will be a data loss.

resize2fs /dev/vgname/lvname

The size of data node is increased by 5GiB.

Reduce the size of the Data node

You can also reduce the data node size. For this, you have to follow the below steps.

• Unmount
• Cleaning bad sectors
• Format
• Lvreduce
• Mount

First step is to unmount your mount point since it is online, somebody will using it. But before that you have to stop data node because it is busy.

Then we have to clean the bad sectors by running the following command

e2fsck -f /dev/mapper/vgname-lvname

Then we have to format the size you want. Here we want only 50GiB and we will reduce the remaining 5GiB space. This is done by running following command.

resize2fs /dev/mapper/vgname-lvname 50G

Then we have to reduce the 5GiB space by using lvreduce command.

lvreduce -f --size 50G /dev/mapper/vgname-lvname

Then start the data node in HDFS Cluster.

Finally, we reduced the data node size.

---

Thank you all for your reads. This article explaining the manual method for providing the Elasticity to data node in HDFS cluster. The Next article will be how to do these using Automated Python Scripts. Stay tuned.Will see you all in my next article. Have a good day.