A Data-Driven Framework For Real-Time Fraud Detection In Financial Transactions Using Machine Learning And Big Data Analytics

Abstract

The rapid growth of electronic commerce and the gradual increase in customer confidence in the security of electronic payments have led to a persistent increase in the number of on-line transactions in the past years. Credit and debit cards account for the majority of the on-line payments. Consequently, the credit card financial ecosystem growth has been accompanied by a similar growth of illicit actions, by which con-men try to benefit from this huge financial exchange. Fraud detection is critical for credit institutions, merchants, and national services to minimize money losses. In recent years, several initiatives to enhance systems aimed at detecting fraudulent credit card transactions have been taken. Detecting fraud is very challenging since machine learning approaches rely on training sets limited to the observations that were available at the moment of the training. The newly triggered events, which were never observed in the training phase, can lead to severe issues such as alarm fatigue, in which frauds are detected only after a large amount of incurred losses. Detecting fraud requires applying scalable learning techniques able to analyze the huge amount of streaming data generated by the transactions and able to mitigate the two main situations complicating the problem: the class imbalance and the concept drift since the world evolves and consequently the fraud patterns change. The need to detect frauds in real-time gives rise to several challenges. Recent advances in analytics, and the availability of open source solutions for storage, processing, and analytics of Big Data, have opened new perspectives for the real-time detection of frauds in massive amounts of transactions. In this paper, the SCAlable Real-time Fraud Finder (SCARFF) framework is presented. SCARFF is a distributed (anomaly) detection machine learning approach for the fraud detection that integrates Big Data tools both for the massive storage and processing of the transactions and for the predictive analysis. SCARFF makes a contribution to the literature in four directions. First, the integration of the Hadoop and Spark ecosystems and of the sophisticated learning approach, addressing the inherent problems of imbalance, nonstationarity, and feedback latency, is a unique contribution. Second, the capability of handling a never-seen-before massive dataset of real credit card transactions is a unique achievement. Third, the formal description of the methods implemented to tackle data imbalance in real-time is presented. Fourth, the implementation in real-time of an ensemble learning engine capable of detecting credit card frauds at the rate of records−1, with large computational savings compared to batch implementations, is a further unique achievement.