Deadline-Aware Coflow Scheduling in a DAG

Data-intensive applications usually need to deal with huge volumes of data within their deadlines. These applications can be modelled as DAGs and require parallel computation frameworks such as MapReduce and Spark to enhance the performance. The network communication has a crucial impact on the performance of an application. Coflow is intended to address the application-specific network level Quality-of-Service (QoS) requirements in cloud-based data centres. However, existing works mainly focus on scheduling coflows in a single stage. How to schedule coflows in multi-stage applications (represented as DAGs) remains to be an open problem. In this paper we study the problem of scheduling coflows in a DAG to meet its deadline requirement. Single stage coflow scheduling has been proven to be NP-hard. Multiple stages in a DAG make our problem even more complex. Owing to the complexity of the problem, we propose a genetic algorithm-based method for solving the problem. The effectiveness of our solution is verified through numerical evaluation. Experimental results show that our solution can effectively guarantee the deadline of the DAGs compared with existing single stage coflow scheduling algorithms.