Abstract

The goal of this research is to develop a keyword extraction program using Term Frequency-Inverse Document Frequency (TF-IDF) in Python. The purpose of keyword extraction is to identify the set of words (keywords) that describe the content of the text. The TF-IDF method is used to measure the importance of words in the text. The basic steps of keyword extraction are explained: preprocessing text, creating list of words, creating bag of words, creating word frequency (TF), creating inverse document frequency (IDF), creating word frequency-inverse document frequency (TF-IDF), creating keywords, and sorting keywords. The developed program was tested on an experimental text from Wikipedia. The program successfully performed the basic steps of keyword extraction and provided the required results.

References

• Sammut, C., & Webb, G. I. (2011). "Encyclopedia of Machine Learning". Springer.
• Aggarwal, C. (2015). "Data Mining: The Textbook". New York: Springer.
• Lee, S., & Kim, H. J. (2008). "Automatic Keyword Extraction from News Articles using TF-IDF Model". Networked Computing and Advanced Information Management, 2.
• Rose, S., Engel, D., Cramer, N., & Cowley, W. (2010). "Automatic Keyword Extraction from Individual Documents". In Text Mining: Applications and Theory, 1-20.
• Kaur, J., & Gupta, V. (2010). "Effective Approaches for Extraction of Keywords". International Journal of Computer Science Issues, 7(6), 144-148.
• Hong, B., & Zhen, D. (2012). "An Extended Keyword Extraction Method". Physics Procedia, 24, 1120-1127.
• Beliga, S. (2014). "Keyword Extraction: A Review of Methods and Approaches". University of Rijeka, Department of Informatics, Rijeka, 1(9).
• Breitinger, C., Gipp, B., Langer, S. (2015). "Research-Paper Recommender Systems: A Literature Survey". International Journal on Digital Libraries, 17(4), 305-338.
• Siddiqi, S., & Sharan, A. (2015). "Keyword and Keyphrase Extraction Techniques: A Literature Review". International Journal of Computer Applications, 109(2), 18-23.
• Gupta, T. (2017). "Keyword Extraction: A Review". International Journal of Engineering Applied Sciences and Technology, 2(4), 215-220.
• Bharti, S. K., & Babu, K. S. (2017). "Automatic Keyword Extraction for Text Summarization: A Survey". arXiv preprint arXiv:1704.03242.
• Qaiser, S., & Ali, R. (2018). "Text Mining: Use of TF-IDF to Examine the Relevance of Words to Documents". International Journal of Computer Applications, 181(1), 25-29.
• Thushara, M. G., Mownika, T., & Mangamuru, R. (2019). "A Comparative Study on Different Keyword Extraction Algorithms". In 2019 3rd International Conference on Computing Methodologies and Communication (ICCMC) (pp. 969-973). IEEE.
• Firoozeh, N., Nazarenko, A., Alizon, F., & Daille, B. (2020). "Keyword Extraction: Issues and Methods". Natural Language Engineering, 26(3), 259-291.
• Xu, Z., & Zhang, J. (2021). "Extracting Keywords from Texts based on Word Frequency and Association Features". Procedia Computer Science, 187, 77-82.
• Li, J. (2021). "A Comparative Study of Keyword Extraction Algorithms for English Texts". Journal of Intelligent Systems, 30(1), 808-815.
• Luhn, H. (1958). "The Automatic Creation of Literature Abstracts". IBM Journal of Research and Development, 2(2), 159-165.
• Sparck Jones, K. (1972). "A Statistical Interpretation of Term Specificity and Its Application in Retrieval". Journal of Documentation. 28(1), 11–21.
• Sparck Jones, K. (2004). "IDF Term Weighting and IR Research Lessons". Journal of Documentation, 60(5), 521-523.
• Robertson, S. (1972). "Term Specificity". Journal of Documentation, 28(1), 164-165.
• Robertson, S. (1974). "Documentation Note: Specificity and Weighted Retrieval". Journal of Documentation, 30(1), 41-46.
• Robertson, S. (2004). "Understanding Inverse Document Frequency: On Theoretical Arguments for IDF". Journal of Documentation, 60(5), 503-520.
• Salton, G., Wong, A., & Yang, C. S. (1975a). "A Vector Space Model for Automatic Indexing". Communications of the ACM, 18(11), 613-620.
• Salton, G., Yang, C. S., & Yu, C. T. (1975b). "A Theory of Term Importance in Automatic Text Analysis". Journal of the American Society for Information Science, 26(1), 33-44.
• Salton, G. & McGill, M. (1983). "Introduction to Modern Information Retrieval". McGraw Hill Book Co, New York.
• Salton, G., & Buckley, C. (1988). "Term-Weighting approaches in Automatic Text Retrieval". Information Processing and Management, 24(5), 513-523.
• Salton, G. (1989). "Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer". Addison- Wesley Publishing Company, USA.
• Salton, G., Singhal, A., Mitra, M., & Buckley, C. (1997). "Automatic Text Structuring and Summarization". Information Processing & Management, 33(2), 193-207.
• Python: https://www.python.org
• Numpy: https://www.numpy.org
• Pandas: https:// pandas.pydata.org
• Matplotlib: https://www. matplotlib.org
• NLTK: https://www.nltk.org
• SK Learn: https://scikit-learn.org
• Wikipedia: https://en.wikipedia.org