Generative AI for VR: Creating Physically Realistic Models

Year : 2025 | Volume : 12 | Issue : 03 | Page : 14 22
    By

    Varun Bhilare,

  • Pranita Chaudhary,

  • Aayush Bodkhe,

  • Abhishek Bichare,

  • Shushant Bhat,

  1. Student, Department of Computer Science & Engineering (Artificial Intelligence and Machine Learning), Pimpri Chinchwad College of Engineering, Pune, Maharashtra, India
  2. Assistant Professor, Department of Computer Science & Engineering (Artificial Intelligence and Machine Learning), Pimpri Chinchwad College of Engineering, Pune, Maharashtra, India
  3. Student, Department of Computer Science & Engineering (Artificial Intelligence and Machine Learning), Pimpri Chinchwad College of Engineering, Pune, Maharashtra, India
  4. Student, Department of Computer Science & Engineering (Artificial Intelligence and Machine Learning), Pimpri Chinchwad College of Engineering, Pune, Maharashtra, India
  5. Student, Department of Computer Science & Engineering (Artificial Intelligence and Machine Learning), Pimpri Chinchwad College of Engineering, Pune, Maharashtra, India

Abstract

Virtual Reality has revolutionized the traditional learning system by creating and interactive and engaging environment. However, its ability to show precise real-world experiences is limited due to lack of physical realism. This study investigates the potential of Generative Adversarial Network (GAN) in creating physically realistic 3D models. Proposed system incorporates deep learning techniques along with physics-based constraints to enhance model’s accuracy and usability. To achieve this, experiments were conducted on Shapenet dataset, various preprocessing were done to standardize the point cloud dataset. A multicomponent GAN architecture was used to generate the 3D model. It included spatial transformer as well as a semantic segmentation network. The results demonstrated by the model included good Intersection-over-Union (IoU) score and the dice coefficient over various categories. Comparative evaluation included much improvement in accuracy and realism than conventional method. Moreover, the qualitative analysis showcases clear segmentation and improved interactivity. This highlighted the system’s potential in bridging the gap between the virtual and real-world dynamics. All these findings indicate that the GAN generated models can provide more effective and indulging VR based learning experience. The study concludes that including deep learning along with physics-based constraints offers a promising pathway for advance educational applications and providing deep understanding and engagement among learners.

Keywords: Point cloud, generative AI, virtual reality (VR), physics simulation, GAN, VAE

[This article belongs to Journal of Advancements in Robotics ]

How to cite this article:
Varun Bhilare, Pranita Chaudhary, Aayush Bodkhe, Abhishek Bichare, Shushant Bhat. Generative AI for VR: Creating Physically Realistic Models. Journal of Advancements in Robotics. 2025; 12(03):14-22.
How to cite this URL:
Varun Bhilare, Pranita Chaudhary, Aayush Bodkhe, Abhishek Bichare, Shushant Bhat. Generative AI for VR: Creating Physically Realistic Models. Journal of Advancements in Robotics. 2025; 12(03):14-22. Available from: https://journals.stmjournals.com/joarb/article=2025/view=228256


References

  1. Wu J, Zhang C, Xue T, Freeman B, Tenenbaum J. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Advances in neural information processing systems (29th conf). 2016; 1–9.
  2. Achlioptas P, Diamanti O, Mitliagkas I, Guibas L. Learning representations and generative models for 3d point clouds. In International conference on machine learning, PMLR. 2018 Jul 3; 40–49.
  3. Qi CR, Su H, Mo K, Guibas LJ. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2017; 652–660.
  4. Groueix T, Fisher M, Kim VG, Russell BC, Aubry M. A papier-mâché approach to learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2018; 216–224.
  5. Chheang V, Sharmin S, Márquez-Hernández R, Patel M, Rajasekaran D, Caulfield G, Kiafar B, Li J, Kullu P, Barmaki RL. Towards anatomy education with generative AI-based virtual assistants in immersive virtual reality environments. In 2024 IEEE international conference on artificial intelligence and eXtended and virtual reality (AIxVR). 2024 Jan 17; 21–30.
  6. Sung NJ, Ma J, Choi YJ, Hong M. Real-time augmented reality physics simulator for education. Appl Sci. 2019 Sep 25; 9(19): 4019.
  7. Vanfretti L, Laughman CR, Chakrabarty A. Integrating Generative Machine Learning Models and Physics-Based Models for Building Energy Simulation. In Modelica Conferences. 2024; 178–188.
  8. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y. Generative adversarial nets. Advances in neural information processing systems (NIPS27). 2014.
  9. Kingma DP, Welling M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013 Dec 20.
  10. Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun ACM. 2021 Dec 17; 65(1): 99–106.
  11. Raissi M, Perdikaris P, Karniadakis GE. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J Comput Phys. 2019 Feb 1; 378: 686–707.
  12. Zhang R, Isola P, Efros AA, Shechtman E, Wang O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2018; 586–595.
  13. Saito S, Huang Z, Natsume R, Morishima S, Kanazawa A, Li H. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF international conference on computer vision. 2019; 2304–2314.
Regular Issue Subscription Review Article
Volume 12
Issue 03
Received 16/05/2025
Accepted 24/06/2025
Published 19/09/2025
Publication Time 126 Days
224

Login


My IP

PlumX Metrics