Abstract

This paper presents a hybrid AI-based system for recognizing and converting handwritten medical prescriptions into digital text to address the widespread issue of illegible handwriting in healthcare. The system combines Optical Character Recognition (OCR) with deep learning techniques—specifically Convolutional Neural Networks (CNN) for visual feature extraction and Long Short-Term Memory (LSTM) networks for sequence modeling. Tesseract OCR is used as an initial pass, with the CNN-LSTM model refining the recognition results. A dataset of prescription images is preprocessed using OpenCV and used to train the model. The proposed system achieves a character-level accuracy of over 91.3%, an error rate below 8%, and an average processing time of 1.8 seconds per image. Unlike traditional OCR systems, this solution is optimized for medical handwriting, incorporating domain-specific terminology. It provides a scalable, real-time tool for use in hospitals, clinics, and pharmacies, reducing transcription errors and supporting the digitization of healthcare records.

References

• A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, "Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks," Proc. 23rd Int. Conf. on Machine Learning (ICML), pp. 369–376, 2006.
• R. Smith, "An Overview of the Tesseract OCR Engine," Proc. 9th Int. Conf. Document Analysis and Recognition (ICDAR), vol. 2, pp. 629–633, 2007.
• A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet Classification with Deep Convolutional Neural Networks," Commun. ACM, vol. 60, no. 6, pp. 84–90, 2017.
• Y. LeCun, Y. Bengio, and G. Hinton, "Deep Learning," Nature, vol. 521, no. 7553, pp. 436–444, 2015.
• T. Bluche, "Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition," Neural Information Processing Systems (NeurIPS), vol. 29, 2016.
• K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," arXiv preprint, arXiv:1409.1556, 2014.
• A. Vaswani et al., "Attention is All You Need," Advances in Neural Information Processing Systems (NeurIPS), vol. 30, 2017.
• Tesseract OCR, GitHub Repository. [Online]. Available: https://github.com/tesseract-ocr/tesseract
• TensorFlow, "An end-to-end open-source machine learning platform." [Online]. Available: https://www.tensorflow.org/
• OpenCV, "Open Source Computer Vision Library." [Online]. Available: https://opencv.org/
• A. Kharazmi, J. Puigcerver, A. Majumdar, and U.-V. Marti, "ICDAR 2019 Competition on Recognition of Handwritten Medical Forms," Proc. 15th Int. Conf. Document Analysis and Recognition (ICDAR), pp. 1500–1505, 2019.
• M. Liwicki and H. Bunke, "IAM-OnDB – An On-Line English Sentence Database Acquired from Handwritten Text on a Whiteboard," Proc. 8th Int. Conf. Document Analysis and Recognition (ICDAR), vol. 2, pp. 956–961, 2005.
• I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, MIT Press, 2016.
• J. Schmidhuber, "Deep Learning in Neural Networks: An Overview," Neural Networks, vol. 61, pp. 85–117, 2015.
• S. Hochreiter and J. Schmidhuber, "Long Short-Term Memory," Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
• G. Pavithiran et al., "Doctors Handwritten Prescription Recognition System in Multi Language Using Deep Learning," arXiv preprint, arXiv:2210.11666, 2022.
• U. Ali et al., "Leveraging Deep Learning with Multi-Head Attention for Accurate Extraction of Medicine from Handwritten Prescriptions," arXiv preprint, arXiv:2412.18199, 2024.
• V. Lobo, P. Sequeira, and A. Quadros, "Doctors’ Handwriting Interpretation Using Deep Learning," Proc. 3rd Int. Conf. on Optimization Techniques in Engineering (ICOFE-2024). [Online]. Available: https://ssrn.com/abstract=5091265
• T. Singh and B. Kaushik, "A Comprehensive Review on the Techniques Used for Recognising Handwritten Medical Prescriptions," Proc. 7th Int. Conf. on Computing Sciences (ICCS 2023). [Online]. Available: https://ssrn.com/abstract=4490364
• A. Bhuyar, R. Khatale, and S. Jadhav, "Review on Doctor’s Handwriting Recognition Using Deep Learning," Int. J. Eng. Res. Technol. (IJERT), vol. 12, no. 3, pp. 428–434, 2023.
• T. Mankash, A. Gandhi, and P. Sarawgi, "MIRAGE: Multimodal Identification and Recognition of Annotations in Indian General Prescriptions," arXiv preprint, arXiv:2410.09729, 2024.