Dynamic Hadoop Slot Allocation in Cloud File System

Abstract

Map Reduce could be a well-liked computing paradigm for large-scale processing in cloud computing supported slots. Map Reduce system (e.g., Hadoop MRv1) will suffer from poor performance because of its unoptimized resource allocation. To handle it, this paper identifies and optimizes the resource allocation from 3 key aspects. First, because of the pre-configuration of distinct map slots and Reduce slots that don't seem to be fungible, slots will be severely under-utilized. we have a tendency to propose an alternate technique known as Dynamic Hadoop Slot Allocation by keeping the slot-based model. It relaxes the slot allocation constraint to permit slots to be reallocated to either map or reduce tasks betting on their wants. Second, the speculative execution will tackle the dawdler problem that has shown to boost the performance for one job however at the expense of the cluster potency. Seeable of this, we have a tendency to propose Speculative Execution Performance equalization to balance the performance trade-off between one job and a batch of jobs. Third, delay programming has shown to boost the info neck of the woods, however at the price of fairness. Or else, we have a tendency to propose a way known as Slot PreScheduling that may improve the data locality however with no impact on fairness. Finally, by combining these techniques along, we have a tendency to type a stepwise slot allocation system known as DynamicMR. The experimental results show that our DynamicMR will improve the performance of Hadoop MRv1 considerably whereas maintaining the fairness, by up to 46% ~ 115% for single jobs and 49% ~ 112%  for multiple jobs. Moreover, we have a tendency to build a comparison with YARN by experimentation, showing that DynamicMR outperforms YARN by concerning 2% ~ 9% for multiple jobs because of its quantitative relation management mechanism of running map/reduce tasks.