New York University (NYU) is embarking on an ambitious project aimed at transforming the realm of dance education and potentially other fields through immersive 3D video technology. Fueled by a substantial $1.2 million grant from the National Science Foundation, this four-year endeavor will strive to make Point-Cloud Video (PCV) technology practical for streaming.
A point cloud comprises a collection of data points within a 3D space, representing the contours of an object or environment. NYU describes Point-Cloud Video as a concept that strings together these point-cloud frames to construct a dynamic and moving scene. However, the data demands associated with PCV have historically been too high for practical use, necessitating bandwidth beyond the capabilities of today’s connected devices.
Researchers from NYU are determined to surmount these challenges by “reducing bandwidth consumption and delivery latency, and increasing power consumption efficiency so that PCVs can be streamed far more easily,” as outlined in an NYU Engineering blog post. Project leader Yong Liu, a professor in NYU’s Department of Electrical and Computer Engineering, is optimistic about the possibilities, stating, “With recent advances in the key enabling technologies, we are now at the verge of completing the puzzle of teleporting holograms of real-world humans, creatures, and objects through the global Internet.”
Engadget reached out to the project leader to inquire whether generative AI tools, such as the one developed by OpenAI last year for creating 3D point clouds from text prompts, are part of the process. Further details will be provided if received.
To test this cutting-edge technology, the NYU team will collaborate with the NYU Tisch School of the Arts and the Mark Morris Dance Group’s Dance Center. Dancers from both institutions will take the stage on a volumetric capture platform, with their movements streamed live and available on-demand. This initiative not only offers educational content for aspiring dancers, providing access to high-level performances for study, but it also grants engineers an opportunity to experiment and refine their PCV technology.
Beyond dance education, the researchers envision this work as a gateway to more advanced virtual reality (VR) and mixed reality streaming content. According to Yong Liu, “The success of the proposed research will contribute towards the wide deployment of high-quality and robust PCV streaming systems that facilitate immersive augmented, virtual, and mixed reality experiences and create new opportunities in many domains, including education, business, healthcare, and entertainment.”
Jelena Kova?evi?, NYU Tandon Dean, expressed her enthusiasm for the project, emphasizing its potential to revolutionize multiple industries. She noted, “Point-Cloud Video holds tremendous potential to transform a range of industries, and I’m excited that the research team at NYU Tandon prioritized dance education to reap those benefits early.”
