Apple’s recently introduced augmented reality (AR) device, the Apple Vision Pro (AVP), has captured the attention of the robotics community with its advanced motion tracking capabilities. Developed by Younghyo Park and Pulkit Agrawal, the AVP offers precise tracking of human movements, particularly wrist and finger motions, enabling new possibilities in robot training and teleoperation.
Capturing Real-World Human Behavior for Robot Programming
One of the primary features of the AVP is its ability to accurately track various human movements using Apple’s ARKit technology. By recording head, wrist, and finger movements in real-time, the device obtains valuable data that can inform robot programming and decision-making for navigation and manipulation tasks in real-world environments.
Intuitive Teleoperation Through Natural Gestures
In addition to data collection, the AVP facilitates intuitive teleoperation of robots through natural human motions. The device streams tracking data over a network, allowing users to control robots remotely using their gestures and movements captured by the device’s sensors.
Tracking Streamer: A VisionOS App for Robotics Applications
To support developers in leveraging the AVP for robotics applications, Park and Agrawal have introduced the Tracking Streamer, a VisionOS app available on the App Store and open-sourced on GitHub. This app serves as a platform for tracking human movements and streaming the data to connected devices on the same network, utilizing Apple’s ARKit and gRPC communication protocol for reliable and efficient data transmission.
Limitations and Future Developments
While the AVP offers exciting potential, there are certain limitations to consider. The device may encounter challenges in environments with constrained movement, and hand positioning near the waist or hip can affect tracking accuracy. However, the developers are actively working to address these limitations and explore new possibilities, such as establishing a bilateral connection between simulations, real-world environments, and the AVP.
As Apple continues to refine and expand the capabilities of the Vision Pro, the robotics community eagerly anticipates the opportunities this innovative technology will bring to training, programming, and controlling robots through intuitive human interactions.
