SecGrad-XAI: Secure and Accurate Explainable AI with Secure Multi-party Image Classification

Explainable AI (XAI) is critical for interpreting Neural Network behavior. Existing XAI methods require full access to model weights and/or inputs and outputs, which may be sensitive data. This transparency requirement creates a fundamental conflict in Secure Neural Network Inference (SNNI) environments, where model weights are protected intellectual property and client inputs/outputs are confidential. We present SecGrad-XAI, a novel framework that provides a fully secure input-to-explanation pipeline using Secure Multi-Party Computing (SMPC). SecGrad-XAI allows a client to receive both an inference result and a corresponding XAI explanation without revealing the input to the server or exposing the server’s model parameters to the client. Using SecGrad-XAI, we implement and evaluate three prominent XAI methods—Vanilla Gradient, Gradient × Input, and Integrated Gradients—across three convolutional architectures and three image classification datasets. Our results demonstrate that despite the constraints of SMPC, SecGrad-XAI maintains high utility; XAI outputs provided by SecGrad-XAI are nearly identical to the ones created in plaintext.