VR Multicopter Teleoperation

Third-person VR control of a hexacopter, with onboard SLAM shown to the pilot as a live 3D mesh

Second author Published Feb 2026
WebXR Three.js ROS 2 SLAM Jetson Orin NX ZED

First-person-view (FPV) drone piloting suffers from limited spatial awareness - operators can't see the drone's relationship to nearby obstacles. This leads to collisions, especially in confined indoor environments.

The question: can a third-person virtual reality perspective improve obstacle clearance without slowing the operator down?

A live 3D view of the drone and the space around it

In the headset, the pilot sees the hexacopter from a third-person view (TPV), with its surroundings drawn around it as a live 3D mesh. That gives spatial context the first-person view does not.

A ZED camera and the ZED SDK do the SLAM. The pilot wears a Meta Quest 3, and the VR view is a WebXR site built with Three.js, hosted on the hexacopter's onboard Jetson Orin NX 16GB. A Vicon motion capture system provides the flight pose.

My part was the link between the two ends: from WebXR on the Meta Quest 3 to ROS 2 on the Jetson Orin NX, which also runs the ZED camera SLAM.

WebXR in the headset browser

The VR view is a website. It is built with Three.js on WebXR and hosted on the Jetson Orin NX that flies with the drone. The Meta Quest 3 loads it in its browser.

WHY → the headset needs a browser, not a separately built app
Third-person view of the drone

In first-person view the pilot sees through the drone's camera. In third-person view the pilot looks at the drone from outside, inside the 3D mesh of its surroundings, so the drone and the obstacle are in the same picture.

WHY → the pilot sees the drone and its surroundings together
Jetson Orin NX on board the hexacopter

The hexacopter carries a Jetson Orin NX 16GB. It runs ROS 2 and the ZED camera SLAM, and it hosts the WebXR site that the headset loads.

WHY → the drone itself runs ROS 2 and SLAM and hosts the VR site

Key features

SLAM as a Live 3D Mesh
The ZED camera and its SDK run SLAM on board the hexacopter. The pilot sees the result in the headset as a live 3D mesh of the surroundings. Flight pose comes from a Vicon motion capture system.
Joystick Velocity Control
The pilot commands velocity in the horizontal plane with a joystick. I built the link between the WebXR site on the Meta Quest 3 and ROS 2 on the Jetson Orin NX.
Tracking Latency
Mean one-way tracking latency over a full run was 29.8 ms after optimization, against 41.9 ms before. In steady state after optimization it was 24.2 ms.
First-Person vs Third-Person Trials
Two trained pilots flew matched trials. The numbers come from one pilot over 8 trials, four first-person and four third-person. The task was to approach a wall, stop, return and land. Minimum obstacle distance was 0.62 m in first-person and 0.82 m in third-person (+0.20 m). Task time was 55 ± 6 s against 57 ± 7 s, which the paper calls comparable. Contacts were 1 of 4 in first-person and 0 of 4 in third-person. The paper calls the sample small: evidence of technical feasibility, not a usability study.

By the numbers

+0.20 m Minimum obstacle distance: 0.82 m third-person vs 0.62 m first-person
57 s Task time in third-person (57 ± 7 s), 55 ± 6 s in first-person
29.8 ms Mean one-way tracking latency after optimization, 41.9 ms before
0 of 4 Contacts in third-person, 1 of 4 in first-person
8 Trials flown by one trained pilot: four first-person, four third-person
IEEE AESM IEEE Aerospace and Electronic Systems Magazine, 41(2):46-55, Feb 2026

In third-person view the pilot kept 0.20 m farther from obstacles: 0.82 m against 0.62 m in first-person view. Task time was comparable, 57 ± 7 s against 55 ± 6 s. Contacts were 0 of 4 in third-person and 1 of 4 in first-person.

That is evidence of technical feasibility, not a usability study: the numbers come from one trained pilot over 8 trials, and the paper itself calls the sample small. A larger study with more participants is the next step.

I'd still choose WebXR for the VR view. It's a Three.js site hosted on the drone's Jetson Orin NX, and the Meta Quest 3 loads it in a browser.

Hughes, J. T., Mazmanyan, G., Ghufran, M., and Rastgoftar, H. "VR-Based Control of Multicopter Operation." IEEE Aerospace and Electronic Systems Magazine, 41(2):46-55, Feb 2026.