Improving hadoop performance in heterogeneous big data environments by dynamic slot configurations in mapreduce hadoop programming model

  • Abstract
  • Keywords
  • References
  • PDF
  • Abstract

    Hadoop has been developed as a platform solution for processing a large scale of data in parallel for different applications in Cloud com-puting. A Hadoop system can be characterized based on three main factors: cluster, workload, and user. Each of these factors can be described in heterogeneous environment, which reflects the heterogeneity degree of the Hadoop system. This paper investigates the effect of heterogeneity in each of these factors on the performance of Hadoop for different schedulers. Three schedulers which consider different levels of Hadoop heterogeneity are used for the analysis: FIFO, Fair sharing, and COSHH (Classification and Optimization based Scheduler for Heterogeneous Hadoop). Performance issues are introduced for Hadoop schedulers and comparative performance analysis between different cases of jobs submission. These jobs are processed in heterogeneous data environments and, under fixed or reconfigurable slot between map and reduce tasks for Hadoop MapReduce java programming clustering model. The results showed that when assigning tunable knob between map and reduce tasks under certain scheduler like FIFO algorithm, the performance enhanced about 81.42% especially in cases of heterogeneity environment where the workload is decreased significantly and the utilization of computational resources in increased obviously.



  • Keywords

    Hadoop; MapReduce; Scheduling Algorithms; Workload; Heterogeneous Data.

  • References

      [1] Al-Ameen, Z., Sulong, G., & Johar, M. G. M. (2012). Enhancing the contrast of CT medical images by employing a novel image size dependent normalization technique. International journal of Bio-science and bio-technology, 4(3), 63-68.

      [2] Issa, J. A. (2015). Performance evaluation and estimation model using regression method for hadoop WordCount. IEEE Access, 3, 2784-2793

      [3] Zhao, Y., Wu, J., & Liu, C. (2014). Dache: A data aware caching for big-data applications using the MapReduce framework. Tsinghua science and technology, 19(1), 39-50.

      [4] Zhang, Z., Cherkasova, L., & Loo, B. T. (2013, April). Benchmarking approach for designing a mapreduce performance model. In Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (pp. 253-258). ACM.

      [5] Rizvandi, N. B., Taheri, J., & Zomaya, A. Y. (2011). Some observations on optimal frequency selection in DVFS-based energy consumption minimization. Journal of Parallel and Distributed Computing, 71(8), 1154-1164

      [6] Rizvandi, N. B., Zomaya, A. Y., Boloori, A. J., & Taheri, J. (2012). On modeling dependency between mapreduce configuration parameters and total execution time. arXiv preprint arXiv:1203.0651.

      [7] Jaiswal, N., & Bhatt, M. (2017). Big-data application using the mapreduce framework.

      [8] Lämmel, R. (2008). Google’s MapReduce programming model—revisited. Science of computer programming, 70(1), 1-30.

      [9] Ghemawat, S., Gobioff, H., & Leung, S. T. (2003). The Google file system (Vol. 37, No. 5, pp. 29-43). ACM.

      [10] Xiong, R., Luo, J., & Dong, F. (2014, November). Sldp: a novel data placement strategy for large-scale heterogeneous hadoop cluster. In advanced cloud and big data (CDB), 2014 second international conference on (pp. 9-17). IEEE.

      [11] Xiong, R., Luo, J., & Dong, F. (2015). Optimizing data placement in heterogeneous hadoop clusters. Cluster computing, 18(4), 1465-1480.

      [12] Park, D., Kang, K., Hong, J., & Cho, Y. (2016, April). An efficient hadoop data replication method design for heterogeneous clusters. In proceedings of the 31st annual acm symposium on applied computing (pp. 2182-2184). Acm.

      [13] Shaikh, T. A., Shafeeque, U. B., & Ahamad, M. (2018). An Intelligent Distributed K-means Algorithm over Cloudera/Hadoop.

      [14] Singh, S., & Liu, Y. (2016). A cloud service architecture for analyzing big monitoring data. Tsinghua Science and Technology, 21(1), 55-70.

      [15] Johar, M. G. M., & Awalluddin, J. A. A. (2011). The role of technology acceptance model in explaining effect on e-commerce application system. International Journal of Managing Information Technology, 3(3), 1-14.

      [16] Hamat, A., Embi, M. A., & Hassan, H. A. (2012). The use of social networking sites among Malaysian university students. International Education Studies, 5(3), 56.




Article ID: 26653
DOI: 10.14419/ijet.v7i4.26653

Copyright © 2012-2015 Science Publishing Corporation Inc. All rights reserved.