A Data-Centric Approach to Distributed Tracing

Modern applications are often implemented as distributed systems consisting of multiple application layers and spread across many machines. Monitoring and diagnosing are fundamental challenges of such systems, and many solutions have been proposed. They all aim to abstract the distributed nature and offer a concise overview of the system. Distributed logging, monitoring and tracing solutions help with root cause analysis or assessing the performance of the system. In data processing situations, data is the main driver and it should be treated accordingly. Microservices are often used to implement highly distributed data processing systems, so troubleshooting from a data point of view is also difficult. Data should be monitored and traced across machines and applications in order to get reliable insights into how the data is stored and processed. However, existing approaches to distributed tracing are based on tracing requests that are propagated throughout the system instead on their content. In this work we take a data-centric perspective to tracing and data processing to investigate how content can be traced in a distributed system. We evaluate existing tracing approaches to see if they can be extended to the content itself, then we take into consideration new ones. We implement three different approaches to data-centric distributed tracing in a highly distributed data processing system built using microservices and discuss their advantages and disadvantages.