This paper focuses on devising efficient resource management techniques used by the resource management middleware in clouds that handle MapReduce jobs with end-to-end service level agreements (SLAs) comprising an earliest start time, execution time, and a deadline. This research and development work, performed in collaboration with our industrial partner, presents the formulation of the matchmaking and scheduling problem for MapReduce jobs as an optimization problem using: Mixed Integer Linear Programming (MILP) and Constraint Programming (CP) techniques. In addition to the formulations devised, our experience in implementing the MILP and CP models using various open source as well as commercial software packages is described. Furthermore, a performance evaluation of the different approaches used to implement the formulations is conducted using a variety of different workloads. Copyright is held by the owner/author(s). Publication rights licensed to ACM.

Additional Metadata
Keywords Constraint programming (CP), MapReduce with deadlines, Mixed integer linear programming (MILP), Optimization, Resource management on clouds
Persistent URL
Conference 5th ACM/SPEC International Conference on Performance Engineering, ICPE 2014
Lim, N. (Norman), Majumdar, S, & Ashwood-Smith, P. (Peter). (2014). Engineering resource management middleware for optimizing the performance of clouds processing mapreduce jobs with deadlines. Presented at the 5th ACM/SPEC International Conference on Performance Engineering, ICPE 2014. doi:10.1145/2568088.2576796