Abstract

Online misinformation calls for classifiers that are both accurate and open to inspection. This paper presents a text-only fake news detection system that combines WordPiece tokenization, a fine-tuned BERT encoder, a linear softmax classification head, and Local Interpretable Model-Agnostic Explanations (LIME). The model is trained on a merged corpus of two public files of full-length articles, 52.3% fake and 47.7% real, whose composition is consistent with the ISOT Fake News Dataset. On a held-out test subset of 500 articles (245 fake, 255 real), the classifier makes no errors: precision, recall, F1, accuracy, the Matthews correlation coefficient, and ROC-AUC all equal 1.00. Because a perfect score on a finite sample is not proof of a zero error rate, the results are analyzed further with exact 95% confidence intervals, a zero-error sensitivity analysis, a class-proportion check of the test subset, and a comparison with results reported on other benchmark datasets. The analysis shows that, under a binomial sampling assumption, the in-domain error rate is below 0.6% with 95% confidence, that no statistically significant difference between the class proportions of the test subset and the corpus was detected (p = 0.14), and that ceiling-level scores are typical of binary article corpora rather than of harder benchmarks. LIME explanations make individual predictions inspectable and can expose source-related cues. The system is therefore a transparent baseline for explainable news-text classification, and the paper defines the cross-dataset, source-disjoint, and temporal evaluations required before deployment.

References

• S. Vosoughi, D. Roy, and S. Aral, “The spread of true and false news online,” Science, vol. 359, no. 6380, pp. 1146–1151, 2018.
• K. Shu, A. Sliva, S. Wang, J. Tang, and H. Liu, “Fake news detection on social media: A data mining perspective,” ACM SIGKDD Explorations Newsletter, vol. 19, no. 1, pp. 22–36, 2017.
• W. Y. Wang, “‘Liar, liar pants on fire’: A new benchmark dataset for fake news detection,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017, pp. 422–426.
• K. Shu, D. Mahudeswaran, S. Wang, D. Lee, and H. Liu, “FakeNewsNet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media,” Big Data, vol. 8, no. 3, pp. 171–188, 2020.
• J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of NAACL-HLT, 2019, pp. 4171–4186.
• M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why should I trust you?’: Explaining the predictions of any classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1135–1144.
• R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, pp. 665–673, 2020.
• Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, “Hierarchical attention networks for document classification,” in Proceedings of NAACL-HLT, 2016, pp. 1480–1489.
• A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 5998–6008.
• R. K. Kaliyar, A. Goswami, and P. Narang, “FakeBERT: Fake news detection in social media with a BERT-based deep learning approach,” Multimedia Tools and Applications, vol. 80, pp. 11765–11788, 2021.
• S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Advances in Neural Information Processing Systems, vol. 30, 2017, pp. 4765–4774.
• H. Ahmed, I. Traore, and S. Saad, “Detection of online fake news using n-gram analysis and machine learning techniques,” in Intelligent, Secure, and Dependable Systems in Distributed and Cloud Environments (ISDDC 2017), Lecture Notes in Computer Science, vol. 10618, Springer, 2017, pp. 127–138.
• H. Ahmed, I. Traore, and S. Saad, “Detecting opinion spams and fake news using text classification,” Security and Privacy, vol. 1, no. 1, e9, 2018.
• C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning, 2017, pp. 1321–1330.
• M. H. Goldani, S. Momtazi, and R. Safabakhsh, “Detecting fake news with capsule neural networks,” arXiv preprint arXiv:2002.01030, 2020.
• I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proceedings of the International Conference on Learning Representations (ICLR), 2019.
• P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, “Mixed precision training,” in Proceedings of the International Conference on Learning Representations (ICLR), 2018.
• D. Chicco and G. Jurman, “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation,” BMC Genomics, vol. 21, art. 6, 2020.
• C. J. Clopper and E. S. Pearson, “The use of confidence or fiducial limits illustrated in the case of the binomial,” Biometrika, vol. 26, pp. 404–413, 1934.
• J. A. Hanley and A. Lippman-Hand, “If nothing goes wrong, is everything all right? Interpreting zero numerators,” JAMA, vol. 249, no. 13, pp. 1743–1745, 1983.