The Research Department of Economics and Management of Tadbir Nikan Business, Marketing, and Finance Open 3092-6238 In Press 2026 05 16 A Comparative Analysis of XGBoost and DNN in Online Payment Fraud Detection under Extreme Class Imbalance: A Cost-Sensitive Optimization Approach Based on the F3-Score A Comparative Analysis of XGBoost and DNN in Online Payment Fraud Detection under Extreme Class Imbalance: A Cost-Sensitive Optimization Approach Based on the F3-Score 1 15 EN Journal Article 2026 03 11 <p><strong>Abstract: </strong>The rapid expansion of digital payment systems has introduced new security challenges, particularly in the domain of fraud detection. One of the primary obstacles in this field is the highly imbalanced nature of financial datasets, in which the proportion of fraudulent transactions is extremely small compared to legitimate transactions. This study evaluates and compares machine learning approaches for detecting suspicious transactions within the UPI platform. To address the challenge of data imbalance, two different approaches were implemented and compared: (1) a Deep Neural Network (DNN) with class weighting and (2) the XGBoost algorithm with scale-sensitive parameter tuning. For a more precise evaluation, and considering the high sensitivity required in fraud detection, the F3-Score metric was employed, as it assigns greater importance to recall and minimizing false negatives. Experimental results indicate that the XGBoost model, achieving an F3-Score of 0.7299 and an Area Under the Curve (AUC) value of 0.8730, demonstrates more stable performance than the neural network in distinguishing legitimate transactions from fraudulent ones. Furthermore, using the interpretable SHAP method, the key features influencing fraud detection were identified and analyzed.</p>