Skip to content

README.md

Latest commit

 

History

History
226 lines (163 loc) · 5.74 KB

README.md

File metadata and controls

226 lines (163 loc) · 5.74 KB

DroidRun Evaluation with AndroidWorld

This module provides tools for benchmarking DroidRun using the AndroidWorld (upstream) task suite - a collection of 116 diverse tasks across 20 Android applications.

1. Manual Environment Setup

Reference: android_world official documentation.

1.1 Install Android Emulator

1.2 Launch Android Emulator

Launch the emulator from command line (not Android Studio UI) with the -grpc 8554 flag, which is required for accessibility forwarding.

# Emulator path is usually located at:
# ~/Android/Sdk/emulator/emulator
# ~/Library/Android/sdk/emulator/emulator

EMULATOR_NAME=AndroidWorldAvd
~/Library/Android/sdk/emulator/emulator -avd $EMULATOR_NAME -no-snapshot -grpc 8554

2. Clone and Install

2.1 Clone repository and initialize submodules

git clone https://github.com/droidrun/droidrun-android-world && \
cd droidrun-android-world

# initialize submodules (android_world + droidrun)
git submodule update --init

2.2 Install dependencies

# install dependencies using uv
uv sync
# build cli
uv build

3. Configuration

3.1 Set API Key

export GEMINI_API_KEY=your-key  # Or other provider keys

3.2 Start AndroidWorld Environment

Must be executed in droidrun-android-world/android_world directory, and keep running:

cd android_world
python -m server.android_server

3.3 Common Issues

  1. adb not found
    Locate adb path: 
    which adb
    Then update adb_path in android_server.py: 
    adb_path="/opt/android/platform-tools/adb"

4. Running Tasks

4.1 Verify Environment

Run in another terminal (keep server.android_server running):

droidworld check

4.2 Run a Task

Execute from droidrun-android-world directory:

# Example: add contact task
droidworld run --tasks ContactsAddContact

5. Summary

  • Android Emulator must be launched via command line and kept running
  • server.android_server must be kept alive in background
  • Run tasks only under droidrun-android-world directory

Usage

Basic Usage

Run a specific task by index:

droidworld run --min-task-idx 0 --max-task-idx 1

Run a specific task by name:

droidworld run --task ContactsAddContact

List Available Tasks

View all available tasks with their IDs:

droidworld list-tasks

Customizing the Benchmark

# Run with a different LLM provider and model
droidworld run --llm-provider Anthropic --llm-model claude-3-sonnet-20240229

# Set maximum steps per task: multiplier * task complexity
droidworld run --max-step-multiplier 15

# Run multiple parameter combinations per task
droidworld run --n-task-combinations 3

# Check all available configuration options with
droidworld run --help

Accessibility Service Notes

The benchmark script automatically manages the DroidRun accessibility service, which is required for proper interaction with Android UI elements: If you encounter issues with UI interaction:

  • Verify the DroidRun Portal app is installed correctly
  • Make sure both Droidrun Portal and Google Accessibility Forwarder are configured and enabled as accessiblity service

To diagnose run droidworld check