HybridFlow: Towards intelligent video delivery and processing over hybrid infrastructures

In this paper we present the architecture and the prototype implementation of HybridFlow, a system for processing and delivering very high definition video. It is designed to handle the challenges that arise in hybrid distributed infrastructures. It does so by implementing a number of software components that select and manage storage, computation and network resources in order to meet the Quality of Service levels desired by the user. One advantage our our system that has facilitated its adoption by CineGrid, an interdisciplinary community focused on the digital cinema domain, is that it shields the end user from the details of the IT infrastructure and allows users from different member organizations to use any resource available in the community. The dynamic nature of the infrastructure makes it susceptible to bottlenecks and our prototype focuses on determining I/O hotspots. At the core of the system lie resource descriptions based on semantic web ontologies that provide information about the existing resources. By using them, the resource selection process is optimized and in line with the application's estimated demands. In this work we focus on the I/O performance as a bottleneck type. We show how HybridFlow can be used in the typical scenario of a video processing pipeline to identify bottlenecks and work towards achieving the desired QoS levels while keeping resource utilization optimal.