Multi-person localization and orientation estimation in volumetric scene reconstructions

Accurate localization of persons and estimation of their pose are important topics in current-day computer vision research. As part of the pose estimation, estimating the body orientation of a person (i.e. rotation around torso major axis) conveys important information about the person's current activity and focus of attention.
This thesis introduces methods for doing multi-person localization and tracking and for estimating the appearance and body orientation of persons in the scene. The possible application areas require the methods to work in complex, dynamic environments with cluttered backgrounds and changing illumination conditions. The proposed methods operate using a moderate number of cameras (3-5) with overlapping fields-of-view. A volumetric reconstruction of the scene is created to cope with occlusions between persons and get accurate real-world locations of all persons in the scene.
Multi-person localization and tracking is based on a novel two-step approach, jointly estimating the positions and track assignments of persons under occlusion while taking into account their appearances. Solving the assignment problem is kept tractable while taking into account how different assignments influence which features appear to be part of which persons.
Appearance modeling and orientation estimation use a 3D shape and texture model, represented using spherical harmonics. The estimation process alternates between the estimation of texture, orientation and shape. Texture is estimated by measuring image colors with the predicted 3D shape (i.e. torso and head) and the predicted orientation from the last time step. Orientation is estimated by minimizing the difference between a learned texture model in a canonical orientation and the current texture estimate. The newly estimated orientation allows to update the 3D shape estimate, taking into account the new 3D shape measurement obtained by volume carving.
Extensive evaluation on publicly available benchmark datasets as well as on novel datasets, which are made public for benchmarking, shows the effectiveness of the proposed methods with respect to the state-of-the-art.