Did a John Doe make a sudden quick transaction of the entire procurement cycle from purchase requisition to purchase order to payment, very quickly than typical transactions? Did "John Doe" create the purchase requisition, while "J Doe" made the final invoice payment? The problem of finding and matching John Doe versus J.Doe is attempted in an end-to-end procurement process to detect internal fraud, based on data from ERP systems. We present an explainable AI (XAI) approach that attempts to explain to business users how certain features of data affect high-risk indicators in a procurement process. Various state-of-the-art XAI methods and novel contributions are empirically evaluated to provide an explainable AI-driven output that can be trusted by business users for actions.