Hadoop high availability through multiple active name nodes

  • Authors

    • P Vijaya Lakshmi
    • K V.S Ramesh
    • P Likhitha
    • M Pranay Kumar
    2017-12-21
    https://doi.org/10.14419/ijet.v7i1.1.9710
  • Name Node, Standby Nodes, Scalability, Availability, Hot Standby.

  • HDFS having only single dynamic name node, if that name node occur hardware or software failure, the entire HDFS model will be inac-tive position until the recovery of name node. So that to reduce that problem the standby name nodes are placed, which they are an inactive position. On failover occur to primary name node all its metadata will transfer to the standby name nodes. After primary name node fails remaining standby nodes elects one of the nodes to take the position of primary name node. But on transferring the metadata to remaining standby name nodes there will be heavy burden to the primary name node

    In this paper, we proposed a solution to reduce the load on the primary name node by transferring the metadata to remaining standby name nodes. We compress the entire metadata in the primary name node and sent that data into remaining all standby name nodes.

  • References

    1. [1] Wang, Z. and Wang, D., 2013, November. NCluster: Using Multiple Active Name Nodes to Achieve High Availability for HDFS. In High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing (HPCC_EUC), 2013 IEEE 10th International Conference on (pp. 2291-2297). IEEE.

      [2] Wang, F., Qiu, J., Yang, J., Dong, B., Li, X. and Li, Y., 2009, November. Hadoop high availability through metadata replication. In Proceedings of the first international workshop on Cloud data management (pp. 37-44). ACM. https://doi.org/10.1145/1651263.1651271.

      [3] Varghese, Lino Abraham, V. P. Sreejith, and S. Bose. "Enhancing NameNode fault tolerance in Hadoop over cloud environment." In Advanced Computing (ICoAC), 2014 Sixth International Conference on, pp. 82-85. IEEE, 2014.

      [4] Khan, Mohammad Asif, Zulfiqar A. Memon, and Sajid Khan. "Highly Available Hadoop NameNode Architecture." In Advanced Computer Science Applications and Technologies (ACSAT), 2012 International Conference on, pp. 167-172. IEEE, 2012. https://doi.org/10.1109/ACSAT.2012.52.

      [5] Foley, Matt. "High availability HDFS." In 28th IEEE Conference on Massive Data Storage, MSST, vol. 12. 2012.

      [6] Wan, J., Liu, M., Hu, X., Ren, Z., Zhang, J., Shi, W. and Wu, W., 2012, December. Dual-JT: Toward the high availability of JobTracker in Hadoop. In Cloud Computing Technology and Science (CloudCom), 2012 IEEE 4th International Conference on (pp. 263-268). IEEE.

      [7] Oriani, A. and Garcia, I.C., 2012, October. From backup to hot standby: High availability for hdfs. In Reliable Distributed Systems (SRDS), 2012 IEEE 31st Symposium on (pp. 131-140). IEEE.

      [8] Aung, Ohnmar, and Thandar Thein. "Enhancing NameNode Fault Tolerance in Hadoop Distributed File System." International Journal of Computer Applications 87, no. 12 (2014). https://doi.org/10.5120/15264-4020.

      [9] Kim, Y., Araragi, T., Nakamura, J. and Masuzawa, T., 2014, October. A Distributed NameNode Cluster for a Highly-Available Hadoop Distributed File System. In Reliable Distributed Systems (SRDS), 2014 IEEE 33rd International Symposium on (pp. 333-334). IEEE. https://doi.org/10.1109/SRDS.2014.61.

      [10] Bi, Kun, and Dezhi Han. "Scalable Multiple NameNodes Hadoop Cloud Storage System." International Journal of Database Theory and Application 8, no. 1 (2015): 105-110. https://doi.org/10.14257/ijdta.2015.8.1.12.

      [11] Devi, S., and K. Kamaraj. "Architecture for Hadoop Distributed File Systems." Architecture 3, no. 10 (2014).

      [12] Le, Hieu Hanh, Satoshi Hikida, and Haruo Yokota. "NameNode and DataNode Coupling for a Power-proportional Hadoop." Book name Database Systems for Advanced Applications Lecture Notes in (2013).

      [13] Shvachko, Konstantin, Hairong Kuang, Sanjay Radia, and Robert Chansler. "The hadoop distributed file system." In Mass storage systems and technologies (MSST), 2010 IEEE 26th symposium on, pp. 1-10. IEEE, 2010. https://doi.org/10.1109/MSST.2010.5496972.

      [14] Oriani, Andre, and Islene C. Garcia. "From backup to hot standby: High availability for hdfs." In Reliable Distributed Systems (SRDS), 2012 IEEE 31st Symposium on, pp. 131-140. IEEE, 2012. https://doi.org/10.1109/SRDS.2012.33.

      [15] Donvito, Giacinto, Giovanni Marzulli, and Domenico Diacono. "Testing of several distributed file-systems (HDFS, Ceph and GlusterFS) for supporting the HEP experiments analysis." In Journal of Physics: Conference Series, vol. 513, no. 4, p. 042014. IOP Publishing, 2014.

  • Downloads

  • How to Cite

    Vijaya Lakshmi, P., V.S Ramesh, K., Likhitha, P., & Pranay Kumar, M. (2017). Hadoop high availability through multiple active name nodes. International Journal of Engineering & Technology, 7(1.1), 311-313. https://doi.org/10.14419/ijet.v7i1.1.9710