Skip to content

Commit

Permalink
update: badges & QA
Browse files Browse the repository at this point in the history
  • Loading branch information
QiushiSun committed Jan 24, 2025
1 parent 098ded4 commit 2324c1d
Show file tree
Hide file tree
Showing 4 changed files with 25 additions and 3 deletions.
Binary file modified .DS_Store
Binary file not shown.
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,12 @@
[![arXiv](https://img.shields.io/badge/arXiv-2412.19723-b31b1b.svg)](https://arxiv.org/abs/2412.19723)
![License](https://img.shields.io/badge/License-MIT-blue)
[![Paper page](https://huggingface.co/datasets/huggingface/badges/resolve/main/paper-page-sm.svg)](https://huggingface.co/papers/2412.19723)
[![Twitter Follow](https://img.shields.io/twitter/follow/qiushi_sun)](https://twitter.com/qiushi_sun)
[![Generic badge](https://img.shields.io/badge/WeChat-机器之心-green.svg?logo=wechat)](https://mp.weixin.qq.com/s/_gu3NSCpAbAE1A8mEhGD7Q)
<a href = "https://zhuanlan.zhihu.com/p/18229337790"><img src="https://img.shields.io/badge/-%E7%9F%A5%E4%B9%8E-%232f6be0" target="_blank"></a>
<!-- [![Twitter Follow](https://img.shields.io/twitter/follow/qiushi_sun)](https://twitter.com/qiushi_sun)
[![Twitter Follow](https://img.shields.io/twitter/follow/zichen_ding)](https://twitter.com/heroding77)
[![Twitter Follow](https://img.shields.io/twitter/follow/chuanyang_jin)](https://twitter.com/chuanyang_jin)
[![Twitter Follow](https://img.shields.io/twitter/follow/chuanyang_jin)](https://twitter.com/chuanyang_jin) -->


This repository contains the code and data for the paper [OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis](https://arxiv.org/abs/2412.19723).
> We are uploading the data and checkpoints. Due to bandwidth limitations, this will take some time. Stay tuned!
Expand Down
3 changes: 3 additions & 0 deletions evaluation/android_world/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# AndroidWorld Evaluation

https://github.com/google-research/android_world
18 changes: 17 additions & 1 deletion faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,20 @@ Thank you for your interest in OS-Genesis. Below are some questions we have coll

## When will the checkpoints and data be available?

We have already uploaded the checkpoints and evaluation code for AndroidControl. ~~The remaining checkpoints will be uploaded in the coming days~~ (done). Due to server bandwidth limitations, this may take some time. The data will also be available shortly.
We have already uploaded the checkpoints and evaluation code for AndroidControl. ~~The remaining checkpoints will be uploaded in the coming days~~ (done). Due to server bandwidth limitations, this may take some time. The data will also be available shortly.


## How About Desktop?

Q: Why haven’t you worked on PC/Desktop data? Is there a particular reason?

A:
We originally intended to cover PC, mobile, and web. In fact, our high-level reverse-synthesis process can also run on PC (we used [OSWorld](https://os-world.github.io/) as the dynamic environment). However, we decided not to continue on the PC side for the following reasons:
1. Data collection on PC is too difficult for a model-based approach.
For instance, in [OSWorld](https://os-world.github.io/), the success rate for GPT-4o across most scenarios is <10%, which means the proportion of high-quality trajectories would be low. Ensuring quality would require a massive amount of data and a more rigorous TRM, making costly.

2. Even after collecting trajectories, there are significant challenges in training:
1. Length of a11ytree:
We use a11ytree, and on desktop the a11ytree is much longer than the mobile or web DOM. In training that involves multimodal information, it exceeds the context window of models like InternVL and Qwen.
2. Instruction-following issues:
Currently, open-source VLMs face major problems with instruction-following on PC environments.

0 comments on commit 2324c1d

Please sign in to comment.