AeroVLA: A Vision-Language-Action Model for UAV Navigation via Minimalist End-to-End Control
🔥 Check out our Project Page for more demo videos and qualitative results!
📖 Abstract
Vision-Language Navigation (VLN) for Unmanned Aerial Vehicles (UAVs) demands complex visual interpretation and continuous control in dynamic 3D environments. Existing hierarchical approaches rely on dense oracle guidance or auxiliary object detectors, creating semantic gaps and limiting genuine autonomy. We propose AeroVLA, a minimalist end-to-end Vision-Language-Action framework mapping raw visual observations and fuzzy linguistic instructions directly to continuous physical control signals. First, we introduce a streamlined dual-view perception strategy that reduces visual redundancy while preserving essential cues for forward navigation and precise grounding, which additionally facilitates future simulation-to-reality transfer. To reclaim genuine autonomy, we deploy a fuzzy directional prompting mechanism derived solely from onboard sensors, completely eliminating the dependency on dense oracle guidance. Ultimately, we formulate a unified control space that integrates continuous 3-Degree-of-Freedom (3-DoF) kinematic commands with an intrinsic landing signal, freeing the agent from external object detectors for precision landing. Extensive experiments on the TravelUAV benchmark demonstrate that AeroVLA achieves state-of-the-art performance in seen environments. Furthermore, it exhibits superior generalization in unseen scenarios by achieving nearly three times the success rate of leading baselines, validating that a minimalist, autonomy-centric paradigm captures more robust visual-motor representations than complex modular systems.🚀 News & Updates
- [2026-07] 🔥 Training code and training dataset are officially released! You can now train your own AeroVLA models. - [2026-06] Our paper has been accepted to ECCV 2026! Please note that the project has been officially renamed from AerialVLA to AeroVLA. - [2026-04] AerialVLA evaluation code and pre-trained LoRA weights are officially released!
🛠️ Installation
1. Clone the repository:
git clone https://github.com/XuPeng23/AeroVLA.git
cd AeroVLA2. Create a Conda environment and activate it:
conda create -n aero_vla python=3.10 -y
conda activate aero_vla3. Install PyTorch:
⚠️ Note: The following command is for CUDA 11.8. Please adjust it according to your local CUDA version.
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118⚠️ Note: If you are using a Blackwell GPU, I recommend the following torch version:
pip install torch==2.7.0 --index-url https://download.pytorch.org/whl/cu128
pip install torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu1284. Install the required dependencies:
pip install -r requirements.txt5. Install AirSim & other packages
pip install [flash_attn.whl] (Download the version suitable for your experimental equipment)
pip install msgpack-rpc-python-fix-msgpack-dep.zip
pip install airsim --no-build-isolation⚠️ Note:
If you experience extremely slow evaluation speeds (e.g., simGetImages takes >4s per step), it is likely due to an AirSim RPC communication bug. Please refer to our Troubleshooting Guide for a quick fix.
🌍 Data & Simulation Environment
AeroVLA uses the Unreal Engine-based AirSim simulator and the dataset provided by the TravelUAV benchmark. You do not need to install AirSim separately; simply download the compiled environment binaries.
- Follow the TravelUAV Official Documentation to download the dataset (
dataset_raw) and the simulation environments (envs).
⚠️ Important Data Preprocessing Step:
The evaluation script relies on merged_data.json, which is not included in the raw download. You MUST generate it using the tool from the TravelUAV repository. Inside the cloned TravelUAV directory, run:
>> python tools/generate_merged_json.py --root_dir ./dataset_raw
- The
data/directory contains the evaluation JSON files. We have split the original test cases by map to facilitate granular, per-map performance analysis, while strictly adhering to the original TravelUAV validation splits.
📥 Checkpoints
- Base Model: AeroVLA is built upon
openvla-7b. You can download the base weights from Hugging Face and place them in the./openvla-7bdirectory. - AeroVLA LoRA Weights: Download our pre-trained LoRA weights from our Hugging Face Repository and place them in
./checkpoints/aerial_vla/. - Training Dataset: Download aerovla_train_dataset.json from the same Hugging Face repository and place it in the ./data/ directory for training.
📁 Project Structure
AeroVLA/
├── airsim_plugin/
├── checkpoints/
├── data/
│ ├── meta/
│ ├── uav_dataset/
│ └── aerovla_train_dataset.json
├── dataset_raw/
│ ├── BattlefieldKitDesert/
│ ├── BrushifyCountryRoads/
│ └── ...
├── envs/
│ ├── carla_town_envs/
│ ├── closeloop_envs/
│ ├── extra_envs/
│ └── ...
├── eval_results/
├── openvla-7b/
├── scripts/
│ ├── eval_aerovla.sh
│ └── metric.sh
├── src/
│ ├── model_wrapper/
│ ├── vlnce_src/
│ ├── aerovla_dataset.py
│ └── train_aerovla.py
└── utils/🏋️♂️ Training
1. Prepare the Dataset: Ensure you have downloaded aerovla_train_dataset.json from our Hugging Face repository and placed it inside the data/ directory.2. Configure Training Parameters: You can modify the training hyperparameters (e.g., learning rate, batch size, epochs) directly inside src/train_aerovla.py.
3. Start Training: Run the provided training script to start fine-tuning:
bash scripts/train.sh⚠️ Reproducibility Warning: Optimizer Precision:
To strictly reproduce our baseline metrics, please ensure your Optimizer States are kept in FP32 during fine-tuning. We observed that forcing pure BF16 globally (i.e., downgrading optimizer states to BF16) causes gradient underflow and noticeably degrades final performance.
(Note: In our experiments, we used DeepSpeed, which automatically maintains FP32 optimizer states under the hood when bf16 is enabled. If you are using custom setups or single-GPU training, please verify your mixed-precision configurations accordingly.)
🏃♂️ Evaluation
1. Run Closed-Loop Evaluation:
To evaluate AeroVLA on a specific map, modify the TASK_ID in scripts/eval_aerovla.sh and run:
bash scripts/eval_aerovla.shThe model wrapper (aerovla_wrapper_ui.py) includes a UI feature that displays real-time dual-view perception, dynamic prompts, and continuous action outputs during inference.
2. Calculate Metrics: Once the evaluation is complete, compute the aggregated metrics (SR, OSR, NE, SPL) by running:
bash scripts/metric.sh aero_vlaThe results will be saved in eval_results/aero_vla/evaluation_detailed.csv and evaluation_summary_aggregated.csv.
📋 TODO List
We are continuously working on improving AeroVLA and pushing it towards real-world applications.
- [x] Release inference code and pre-trained weights.
- [x] Release the training code and curated trainset.
- [ ] Hardware Deployment: Deploy AeroVLA on real-world UAVs for physical testing.
⭐ Star History
📄 License
This project is licensed under the Apache License 2.0. See the LICENSE file for more details.
✒️ Citation
If you find our work helpful for your research, please consider citing our paper:
@inproceedings{xu2026aerovla,
title={{AeroVLA}: A Vision-Language-Action Model for {UAV} Navigation via Minimalist End-to-End Control},
author={Xu, Peng and Deng, Zhengnan and Deng, Jiayan and Gu, Zonghua and Wan, Shaohua},
booktitle={European Conference on Computer Vision (ECCV)},
pages={289--305},
year={2026},
publisher={Springer}
}