Techniques for handling error in user-estimated execution times during resource management on systems processing MapReduce Jobs
In our previous work, we described a resource allocation and scheduling technique for processing an open stream of MapReduce jobs with SLAs (characterized by an earliest start time, an execution time, and a deadline) called the Hadoop Constraint Programming based Resource Management technique (HCP-RM). Since the user-estimated job execution times are used to perform resource allocation and scheduling, error/inaccuracies in the execution times can hinder the ability of HCP-RM from making effective scheduling decisions, leading to a degradation in system performance. This paper focuses on improving the robustness of HCP-RM by introducing a mechanism to handle errors/inaccuracies in user estimates of job execution times that are submitted as part of the job's SLA. A Prescheduling Error Handling technique (PSEH) is devised to adjust the user-estimated execution times of the jobs to make them more accurate before they are used by the resource management algorithm. Results of experiments conducted on a Hadoop cluster deployed on Amazon EC2 demonstrate the effectiveness of the PSEH technique in improving system performance.
|Keywords||Handling error in user-estimated job execution times, MapReduce with SLAs, Resource allocation and scheduling|
|Conference||17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2017|
Lim, N. (Norman), Majumdar, S, & Ashwood-Smith, P. (Peter). (2017). Techniques for handling error in user-estimated execution times during resource management on systems processing MapReduce Jobs. In Proceedings - 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2017 (pp. 788–793). doi:10.1109/CCGRID.2017.70