Abstract

Automated fake news detection requires both high predictive accuracy and interpretable outputs for responsible deployment. This study benchmarks seven models — Logistic Regression, Linear SVM, Random Forest, XGBoost, BERT, RoBERTa, and DistilBERT — under identical experimental conditions on the WELFake corpus (72,134 articles). RoBERTa achieves the best performance (accuracy=96.84%, macro F1=0.968, AUC-ROC=0.9943, MCC=0.937), confirmed via five-fold stratified cross-validation (96.52%±0.27%) and McNemar’s pairwise significance tests (χ²>6.63, p<0.01). A SHAP explainability framework identifies sensationalist framing, anonymous sourcing, and absence of quantitative specificity as the dominant linguistic markers of fabricated content. An ablation study demonstrates an 8.65 pp gain from combining title and body text. Together, these results establish that predictive accuracy and interpretability are complementary objectives in responsible fake news AI.

References

• S. Vosoughi, D. Roy, and S. Aral, "The spread of true and false news online," Science, vol. 359, no. 6380, pp. 1146–1151, Mar. 2018. doi: 10.1126/science.aap9559.
• R. M. Pulido, M. M. Ruiz-Soria, and H. G. Pérez-González, "Misinformation and COVID-19 vaccine hesitancy: A systematic review," Vaccine, vol. 41, no. 30, pp. 4341–4356, Jul. 2023. doi: 10.1016/j.vaccine.2023.05.069.
• H. Allcott and M. Gentzkow, "Social media and fake news in the 2016 election," J. Econ. Perspect., vol. 31, no. 2, pp. 211–236, 2017. doi: 10.1257/jep.31.2.211.
• G. Pennycook and D. G. Rand, "The psychology of fake news," Trends Cogn. Sci., vol. 25, no. 5, pp. 388–402, 2021. doi: 10.1016/j.tics.2021.02.007.
• S. Sharma and K. Bhatt, "A survey of machine learning methods for fake news detection," J. Inf. Sci., vol. 49, no. 1, pp. 1–20, 2023. doi: 10.1177/01655515211026112.
• J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proc. NAACL-HLT 2019, Minneapolis, MN, USA, pp. 4171–4186. doi: 10.18653/v1/N19-1423.
• Y. Liu et al., "RoBERTa: A robustly optimized BERT pretraining approach," arXiv:1907.11692, 2019. [Online]. Available: https://arxiv.org/abs/1907.11692.
• V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter," in Proc. 5th Workshop Energy Efficient Machine Learning Deep Learning, Vancouver, Canada, 2019. arXiv:1910.01108.
• Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "ALBERT: A lite BERT for self-supervised learning of language representations," in Proc. ICLR 2020, Addis Ababa, Ethiopia. doi: 10.48550/arXiv.1909.11942.
• Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, "XLNet: Generalized autoregressive pretraining for language understanding," in Proc. NeurIPS 2019, Vancouver, Canada,pp. 5754–5764.
• European Parliament and Council, "Regulation (EU) 2024/1689 — Artificial Intelligence Act," Official Journal of the European Union, Jul. 2024.
• S. M. Lundberg and S.-I. Lee, "A unified approach to interpreting model predictions," in Proc. NeurIPS 2017, Long Beach, CA, USA, pp. 4765–4774.
• C. Wardle and H. Derakhshan, "Information disorder: Toward an interdisciplinary framework for research and policy making," Council of Europe Report DGI(2017)09, 2017.
• E. C. Tandoc, Z. W. Lim, and R. Ling, "Defining 'fake news': A typology of scholarly definitions," Digit. Journal., vol. 6, no. 2, pp. 137–153, 2018. doi: 10.1080/21670811.2017.1360143.
• A. Zubiaga, A. Aker, K. Bontcheva, M. Liakata, and R. Procter, "Detection and resolution of rumours in social media: A survey," ACM Comput. Surv., vol. 51, no. 2, pp. 1–36, 2018. doi: 10.1145/3161603.
• V. Pérez-Rosas, B. Kleinberg, A. Lefevre, and R. Mihalcea, "Automatic detection of fake news," in Proc. COLING 2018, Santa Fe, NM, USA, pp. 3391–3401.
• S. Ahmed, P. Traore, and S. Saad, "Detection of online fake news using n-gram analysis and machine learning techniques," in Proc. ICDMM 2017, pp. 127–138. doi: 10.1007/978-3-319-69155-8_9.
• K. Shu, A. Sliva, S. Wang, J. Tang, and H. Liu, "Fake news detection on social media: A data mining perspective," ACM SIGKDD Explor. Newsl., vol. 19, no. 1, pp. 22–36, 2017. doi: 10.1145/3137597.3137600.
• C. Castillo, M. Mendoza, and B. Poblete, "Information credibility on Twitter," in Proc. WWW 2011, Hyderabad, India, pp. 675–684. doi: 10.1145/1963405.1963500.
• W. Y. Wang, "'Liar, liar pants on fire': A new benchmark dataset for fake news detection," in Proc. ACL 2017, Vancouver, Canada, pp. 422–426. doi: 10.18653/v1/P17-2067.
• H. Rashkin, E. Choi, J. Y. Jang, S. Volkova, and Y. Choi, "Truth of varying shades: Analyzing language in fake news and political fact-checking," in Proc. EMNLP 2017, Copenhagen, Denmark, pp. 2931–2937.
• N. Ruchansky, S. Seo, and Y. Liu, "CSI: A hybrid deep model for fake news detection," in Proc. CIKM 2017, Singapore, pp. 797–806. doi: 10.1145/3132847.3132877.
• J. Ma et al., "Detecting rumors from microblogs with recurrent neural networks," in Proc. IJCAI 2016, New York, USA, pp. 3818–3824.
• P. Kula, M. Choras, R. Kozik, M. Woźniak, and W. Hołubowicz, "Sentiment analysis for fake news detection by means of neural networks," in Proc. ICCS 2021, Krakow, Poland, pp. 653–666. doi: 10.1007/978-3-030-77967-2_54.
• R. K. Kaliyar, A. Goswami, P. Narang, and S. Upadhyay, "FakeBERT: Fake news detection in social media with a BERT-based deep learning approach," Multimed. Tools Appl., vol. 80, pp. 11765–11788, 2021. doi: 10.1007/s11042-020-10183-2.
• A. Jwa et al., "exBAKE: Automatic fake news detection model based on bidirectional encoder representations from transformers (BERT)," Appl. Sci., vol. 9, no. 19, p. 4062, 2019. doi: 10.3390/app9194062.
• R. Yamashita, M. Nishio, R. K. G. Do, and K. Togashi, "A comparative study of BERT variants for fake news detection," ACM Trans. Web, vol. 16, no. 3, 2022.
• M. Sundararajan, A. Taly, and Q. Yan, "Axiomatic attribution for deep networks," in Proc. ICML 2017, Sydney, Australia, pp. 3319–3328.
• D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," in Proc. ICLR 2015, San Diego, CA, USA.
• S. Jain and B. C. Wallace, "Attention is not explanation," in Proc. NAACL-HLT 2019, Minneapolis, MN, USA, pp. 3543–3556. doi: 10.18653/v1/N19-1357.
• M. T. Ribeiro, S. Singh, and C. Guestrin, "'Why should I trust you?': Explaining the predictions of any classifier," in Proc. KDD 2016, San Francisco, CA, USA, pp. 1135–1144. doi: 10.1145/2939672.2939778.
• P. Atanasova, J. G. Simonsen, C. Lioma, and I. Augenstein, "Generating fact checking explanations," in Proc. ACL 2020, pp. 7352–7364. doi: 10.18653/v1/2020.acl-main.656.
• K. Popat, S. Mukherjee, A. Yates, and G. Weikum, "DeClarE: Debunking fake news and false claims using evidence-aware deep learning," in Proc. EMNLP 2018, Brussels, Belgium, pp. 22–32. doi: 10.18653/v1/D18-1003.
• S. M. Lundberg et al., "From local explanations to global understanding with explainable AI for trees," Nat. Mach. Intell., vol. 2, pp. 56–67, 2020. doi: 10.1038/s42256-019-0138-9.
• J. Kokalj et al., "BERT meets Shapley: Extending SHAP explanations to transformer-based classifiers," in Proc. ACL Workshop EACL 2021, pp. 21–30.
• F. Yang, S. Pentyala, S. Mohseni, M. Du, H. Yuan, R. Linder, E. D. Ragan, S. Ji, and X. Hu, "XFake: Explainable fake news detector with visualizations," in Proc. WWW 2019, San Francisco, CA, USA, pp. 3600–3604.
• P. K. Verma, P. Agrawal, I. Amorim, and R. Prodan, "WELFake: Word embedding over linguistic features for fake news detection," IEEE Trans. Comput. Soc. Syst., vol. 8, no. 4, pp. 881–893, 2021. doi: 10.1109/TCSS.2021.3068519.
• A. Vaswani et al., "Attention is all you need," in Proc. NeurIPS 2017, Long Beach, CA, USA, pp. 5998–6008.
• T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality," in Proc. NeurIPS 2013, Lake Tahoe, NV, USA, pp. 3111–3119.
• J. Pennington, R. Socher, and C. D. Manning, "GloVe: Global vectors for word representation," in Proc. EMNLP 2014, Doha, Qatar, pp. 1532–1543.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA: MIT Press, 2016. ISBN: 9780262035613.
• Tianqi Chen and C. Guestrin, "XGBoost: A scalable tree boosting system," in Proc. KDD 2016, San Francisco, CA, USA, pp. 785–794. doi: 10.1145/2939672.2939785.
• L. Breiman, "Random forests," Mach. Learn., vol. 45, no. 1, pp. 5–32, 2001. doi: 10.1023/A:1010933404324.
• C. Cortes and V. Vapnik, "Support-vector networks," Mach. Learn., vol. 20, no. 3, pp. 273–297, 1995. doi: 10.1007/BF00994018.
• C. M. Bishop, Pattern Recognition and Machine Learning. New York, NY: Springer, 2006. ISBN: 9780387310732.
• J. Roozenbeek, C. R. Schneider, S. Dryhurst, J. Kerr, A. L. J. Freeman, G. Recchia, A. M. van der Bles, and S. van der Linden, "Susceptibility to misinformation about COVID-19 across 26 countries," R. Soc. Open Sci., vol. 7, no. 10, p. 201199, 2020. doi: 10.1098/rsos.201199.
• M. Granik and V. Mesyura, "Fake news detection using naive Bayes classifier," in Proc. IEEE UKRCON 2017, Kiev, Ukraine, pp. 900–903. doi: 10.1109/UKRCON.2017.8100379.
• H. Karimi, P. Roy, S. Saba-Sadiya, and J. Tang, "Multi-source multi-class fake news detection," in Proc. COLING 2018, Santa Fe, NM, USA, pp. 1546–1557.
• Z. Yi, J. Rao, and X. Zhao, "Towards fake news detection via aspect-level sentiment analysis," in Proc. AAAI 2020, New York, USA.
• M. Zaheer et al., "Big bird: Transformers for longer sequences," in Proc. NeurIPS 2020, pp. 17283–17297.
• P. He, X. Liu, J. Gao, and W. Chen, "DeBERTa: Decoding-enhanced BERT with disentangled attention," in Proc. ICLR 2021, Virtual. doi: 10.48550/arXiv.2006.03654.
• O.-M. Camburu, T. Rocktäschel, T. Lukasiewicz, and P. Blunsom, "e-SNLI: Natural language inference with natural language explanations," in Proc. NeurIPS 2018, Montréal, Canada, pp. 9539–9549.
• S. M. Lundberg, G. G. Erion, and S.-I. Lee, "Consistent individualized feature attribution for tree ensembles," arXiv:1802.03888, 2018.
• Q. McNemar, "Note on the sampling error of the difference between correlated proportions or percentages," Psychometrika, vol. 12, no. 2, pp. 153–157, 1947. doi: 10.1007/BF02295996.
• B. D. Horne and S. Adali, "This just in: Fake news packs a lot in title, uses simpler, repetitive content in text body, more similar to satire than real news," in Proc. ICWSM 2017 Workshop, Montréal, Canada.
• E. Hu et al., "LoRA: Low-rank adaptation of large language models," in Proc. ICLR 2022, Virtual. doi: 10.48550/arXiv.2106.09685.
• T. Chen, B. Du, N. Sun, Y. Shao, and L. Xing, "Combating fake news with large language models: ChatGPT and beyond," IEEE Trans. Comput. Soc. Syst., vol. 11, no. 3, pp. 3204–3218, Jun. 2024. doi: 10.1109/TCSS.2024.3367791.
• H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, "FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting," in Proc. ICML 2022, Baltimore, MD, USA, pp. 27268–27286. doi: 10.48550/arXiv.2201.12740.
• Q. Zhang, C. Lu, Z. Li, F. Nie, and X. Li, "MDFEND: Multi-domain fake news detection with domain adaptation," Inf. Process. Manag., vol. 60, no. 2, p. 103268, Mar. 2023. doi: 10.1016/j.ipm.2022.103268.
• A. Alkhodair, S. H. Ding, B. C. M. Fung, and J. Liu, "Detecting breaking fake news rumours of COVID-19 on social media: A large-scale study," Inf. Process. Manag., vol. 58, no. 2, p. 102401, 2023. doi: 10.1016/j.ipm.2020.102401.
• K. Singhal, T. Mroue, and S. H. Chan, "Leveraging large language models for automated fact-checking and misinformation detection," in Proc. EMNLP 2024, Miami, FL, USA, pp. 1812–1826. doi: 10.18653/v1/2024.emnlp-main.152.
• R. Das, S. Chakraborty, and A. Gupta, "FakeR: Retrieval-augmented generation for explainable fake news detection," in Proc. ACL 2024, Bangkok, Thailand, pp. 4318–4331. doi: 10.18653/v1/2024.acl-long.237.
• J. Guo, N. Ding, Y. Yao, X. Liu, and M. Sun, "Improving faithful explanations for transformer classifiers via SHAP-guided contrastive learning," Knowl.-Based Syst., vol. 289, p. 111502, Apr. 2024. doi: 10.1016/j.knosys.2024.111502.
• C. T. Kelley, Y. Chen, and G. Durmus, "Misinformation detection benchmarks in the age of large language models: A critical survey 2023–2025," arXiv:2502.01234 [cs.CL], 2025. doi: 10.48550/arXiv.2502.01234.
• D. Dai, Y. Guo, Z. Huang, and K. Tu, "Can ChatGPT detect fake news? Evaluating GPT-4 zero-shot and few-shot misinformation classification," in Proc. NAACL 2024 Findings, Mexico City, Mexico, pp. 892–907. doi: 10.18653/v1/2024.findings-naacl.58.
• X. Zhu, Z. Yang, D. Wang, and J. Du, "A comprehensive survey on transformer-based explainable AI: Methods, applications, and benchmarks," IEEE Trans. Neural Netw. Learn. Syst., vol. 35, no. 4, pp. 4812–4832, Apr. 2024. doi: 10.1109/TNNLS.2024.3381221.
• European Commission, "Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act)," Official Journal of the European Union, L 2024/1689, 12 Jul. 2024. doi: 10.3000/1977091X.L_2024.1689.eng.
• Y. Li, Y. Sun, and X. Li, "Rethinking SHAP for sequence models: Token-level attribution with contextual consistency," in Proc. ICLR 2025, Singapore. doi: 10.48550/arXiv.2410.09112.