Abstract

Neural rendering is arising as a leading tool at the intersection of computer vision, computer graphics, and artificial intelligence, and allows for generating high-quality, photorealistic images from 2D models, low-resolution images, or sparse data. The review offers a comprehensive outline of the state-of-the-art techniques in neural rendering, including neural radiance fields (NeRF), view synthesis and implicit surface representation models. However, the success of these models is strongly tied to the availability and quality of medical datasets, which often face challenges related to data scarcity, patient privacy, and modality diversity. The article also explores key application in areas such as virtual reality, autonomous systems, and medical imaging, where neural rendering has shown significant promise. This survey reviews state-of-the-art neural rendering methods in healthcare, discusses benchmark datasets, identifies open challenges, and outlines future research directions.

References

• Wang, X., Hu, S., Fan, H., Zhu, H., & Li, X. (2024). Neural Radiance Fields in Medical Imaging: Challenges and Next Steps. arXiv preprint arXiv:2402.17797.
• Yao, M., Huo, Y., Ran, Y., Tian, Q., Wang, R., & Wang, H. (2024). Neural radiance field-based visual rendering: a comprehensive review. arXiv preprint arXiv:2404.00714.
• Iddrisu, K., Malec, S., & Crimi, A. (2023, June). 3D reconstructions of brain from MRI scans using neural radiance fields. In International Conference on Artificial Intelligence and Soft Computing (pp. 207-218). Cham: Springer Nature Switzerland.
• Mahmood, F., Chen, R., Sudarsky, S., Yu, D., & Durr, N. J. (2018). Deep learning with cinematic rendering: fine-tuning deep neural networks using photorealistic medical images. Physics in Medicine & Biology, 63(18), 185012.
• Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., ... & Lungren, M. P. (2018). Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS medicine, 15(11), e1002699.
• Cai, Y., Wang, J., Yuille, A., Zhou, Z., & Wang, A. (2024). Structure-aware sparse-view x-ray 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11174-11183).
• Zhou, Y., Chia, M. A., Wagner, S. K., Ayhan, M. S., Williamson, D. J., Struyven, R. R., ... & Keane, P. A. (2023). A foundation model for generalizable disease detection from retinal images. Nature, 622(7981), 156-163.
• Singla, K., & Nand, P. (2024). Optimizing deep learning architectures for novel view synthesis: Investigating the impact of NeRF MLP parameters on complex scenes. International Journal of Information Technology, 16(4), 2295-2305
• Molaei, A., Aminimehr, A., Tavakoli, A., Kazerouni, A., Azad, B., Azad, R., & Merhof, D. (2023). Implicit neural representation in medical imaging: A comparative survey. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2381-2391).
• Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2021). Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1), 99-106.
• Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., De Mello, S., ... & Wetzstein, G. (2022). Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16123-16133).
• Wynn, J., & Turmukhambetov, D. (2023). Diffusionerf: Regularizing neural radiance fields with denoising diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4180-4189).
• Suhail, M., Esteves, C., Sigal, L., & Makadia, A. (2022, October). Generalizable patch-based neural rendering. In European Conference on Computer Vision (pp. 156-174). Cham: Springer Nature Switzerland.
• Montúfar, J., Romero, M., Muñoz-Jiménez, V., Scougall-Vilchis, R. J., & Jiménez, B. (2018). Perspective and orthogonal CBCT/CT digitally reconstructed radiographs compared to conventional cephalograms. In Conf. Biomedical Engineering and Sciences| BIOENG (Vol. 16, pp. 41-45).
• Link, “Mednerf dataset,” https://github.com/abrilcf/mednerf, 2024.
• E. B. Tsai, S. Lungren, M. P. Hershman, M. Roshkovan, L. Colak, et al., “Medical imaging data resource center (midrc) – rsna international covid open research database (ricord) release 1b -chest ct covid,” 2021.
• K. Clark, B. Vendt, K. Smith, J. Freymann, et al., “The cancer imaging archive (tcia): Maintaining and operating a public information repository,” 2013, vol. 26.
• Michael D. Harris, Adam J. Cyr, Azhar A. Ali, et al., “A combined experimental and computational approach to subject specific analysis of knee joint laxity,” 2016, vol. 138.
• S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNittGray, C. R. Meyer, A. P. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, E. A. Hoffman, et al. The lung image database consortium (LIDC) and image database resource initiative (IDRI): a completed reference database of lung nodules on CT scans. MP, 38(2):915–931, 2011.
• Zhao, W., Zhang, S., Guan, Z., Luo, H., Tang, L., Peng, J., & Fan, J. (2020). 6D object pose estimation via viewpoint relation reasoning. Neurocomputing, 389, 9-17
• Lin, C. Y., Fu, Q., Merth, T., Yang, K., & Ranjan, A. (2024). FastSR-NeRF: improving NeRF efficiency on consumer devices with a simple super-resolution pipeline. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 6036-6045)
• Chen, Y., Zhang, J., Xie, Z., Li, W., Zhang, F., Lu, J., & Zhang, L. (2025). S-nerf++: Autonomous driving simulation via neural reconstruction and generation. IEEE Transactions on Pattern Analysis and Machine Intelligence.
• Germain, H., Lepetit, V., & Bourmaud, G. (2021). Neural reprojection error: Merging feature learning and camera pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 414-423).. .
• Ding, W. and Marchionini, G. 1997 A Study on Video Browsing Strategies. Technical Report. University of Maryland at College Park.
• An, H., Khan, J., Kim, S., Choi, J., & Jung, Y. (2024). The Adaption of Recent New Concepts in Neural Radiance Fields and Their Role for High-Fidelity Volume Reconstruction in Medical Images. Sensors, 24(18), 5923. https://doi.org/10.3390/s24185923