VLA Autonomous Driving
Adapting SimLingo, a vision-language-action driving model, from the CARLA simulator to Quanser's QLabs digital twin of the QCar 2.
Three clips from the thesis experiments in QLabs: the pretrained model, the fine-tuned model on the baseline route, and one of the obstacle runs it passed (4 of 10). No sound.
SimLingo is a vision-language-action (VLA) driving model trained in the CARLA simulator. QLabs is Quanser's simulator, with a digital twin of the QCar 2. Without any fine-tuning, the pretrained model covered 4.7% of the route there: 1.94 m of 89.1 m. The two simulators look very different through the camera. CARLA is photorealistic and QLabs is flat-shaded.
The thesis asks how to adapt a model like this to a new domain. The constraints were limited data and one 16 GB desktop GPU, with the car steering from a single camera at 4 Hz. All of it ran in the simulator, not on the physical car.
The pipeline: data collection → fine-tuning → inference → evaluation
The vision encoder stays frozen and the language model gets LoRA adapters. The idea is to keep what the pretrained model already knows about driving and train only a small set of new weights on QLabs data.
Camera images go into InternViT-300M, the frozen vision encoder. The language model is Qwen2-0.5B with LoRA adapters at rank 32, about 17.6M adapter parameters. The model's driving output comes from learnable queries: 20 route-waypoint queries and 10 speed-waypoint queries.
The training set is 7,498 frames. Updating every weight on that little data risked overfitting and overwriting what the pretrained model already knows about driving. LoRA trains low-rank adapter matrices instead: rank 32, about 17.6M adapter parameters.
InternViT-300M stays frozen for all of training, so its visual features stay fixed. The adaptation is left to the LoRA adapters on the language model. With 7,498 training frames, fewer trainable weights was the safer choice.
The visual domain gap: CARLA's photorealistic rendering (left) vs QLabs' flat-shaded graphics (right)
Key features
By the numbers
The main thing I learned is that the pretrained model's 4.7% route coverage was a distribution gap, not a broken model. After LoRA fine-tuning on 7,498 frames from QLabs, route coverage reached 99.1%. That suggests the driving knowledge was already in the pretrained weights and the inputs were what had changed.
The limits are just as clear. The fine-tuned model follows the route, but it passed only 4 of 10 obstacle runs, where the LiDAR baseline passed every run. With 5 of 5 baseline runs, that is 9 of 15 runs overall, a 60% pass rate.
All of this is in simulation. The model never ran on the physical car, and transfer to the physical QCar 2 is future work. Quanser featured the project on its Community Showcase.
"Adapting Vision-Language-Action Models to New Domains: A Case Study with SimLingo on QCar 2." M.S. thesis, University of Arizona, 2026. Committee: Chicheng Zhang and Hossein Rastgoftar (co-chairs), Eduardo Blanco.