Resource management in multi-application data centres

(This project group is open to both computer science and computer engineering students. It is intended to start in the summer term 2018. Come to the presentation on January 29th!) 

Data centres are sometimes custom-built for a small set of applications, e.g., in high-performance computing scenarios. More typically, data centres host a large, very diverse set of applications, ranging from simple webservers to complex applications based on, for example, big-data frameworks like Apache Spark. Such applications are assigned to different servers, distributing computational, storage, and memory resources between concurrently running applications. This is the job of a cluster management system like Apache Spark. For simple applications consisting just of individual processes (or containers, or virtual machines), this is a perfectly adequate solution. 

But what about more complex applications? Applications that consist of multiple processes running on multiple servers, interacting with each other by exchanging data? The typical example for that would be such big-data applications. For such applications, just allocating computational resources is not good enough as their execution time also depends on networking resources. And indeed, managing networking resources is possible by using the software-defined networking (SDN) concept and SDN management frameworks like ONOS or OpenDaylight. 

This project group will hence develop an approach to jointly allocate computational and networking resources towards such distributed applications. We shall attempt to integrate such an approach between (likely) Mesos and ONOS or OpenDaylight (concrete choice to be decided during the project group).

A particular challenge will be to ensure fairness between multiple applications. This will require both conceptual work - fair resource allocation between applications, each with multiple data flows - as well as system design work. We also need to consider suitable APIs between applications and resource management systems.