arkmodscraper

Extracts all subscribed Workshop mods for Ark Survival Evolved from the Steam Workshop, using the Steam Web API. Outputs in arkmanager instance config format by default.

Usage

usage: arkmodscraper [-h] [-v] [--config CONFIG] [--csv | --map MAP] [--top TOP]

Automatically extract subscribed workshop mods for Ark Survival Evolved

options:
  -h, --help       show this help message and exit
  -v, --version    show program's version number and exit
  --config CONFIG  Path to the configuration file (default: steam-credentials.conf)
  --csv            Output as CSV (name,id) instead of arkmanager config format
  --map MAP        Comma-separated instance name(s) to update under
                   /etc/arkmanager/instances (e.g. theisland,ragnarok).
                   Updates each config file and prints a bash script to stdout
                   to apply mod changes.
  --top TOP        Comma-separated mod name substrings to pin at the top of the
                   load order (requires --map). Case-insensitive substring match;
                   ambiguous matches abort.

Direct instance config update (--map)

--map is the primary workflow for maintaining a live server. It updates the mod list in each named arkmanager instance config file and prints a bash script to stdout that applies the changes:

arkmodscraper --map theisland --top "uncut" > apply_mods.sh
# review, then:
bash apply_mods.sh

The generated script stops the server, disables and uninstalls all current mods (clean slate), installs and enables the full new mod list (downloading updates), then restarts the server. set -euo pipefail and a run() helper ensure the script halts with a clear message if any step fails.

Each instance config file is backed up with a Melbourne-time timestamp before being written (e.g. theisland.cfg-20260328143000).

--top accepts a comma-separated list of case-insensitive substrings. Matched mods are placed at the top of the load order in the order given. A pattern that matches no subscribed mods produces a warning and is skipped; a pattern that matches more than one mod aborts the run before any file is touched.

Configuration

The configuration file uses TOML. You will need a Steam Web API key — get one at https://steamcommunity.com/dev/apikey — and your 64-bit Steam ID.

# Steam credentials configuration file

# Steam Web API key (mandatory)
apikey = "YOUR_API_KEY_HERE"

# 64-bit Steam ID (mandatory)
steamid = "76561197900000000"

Build and Installation

Build the wheel on your development machine:

make wheel

Then copy dist/arkmodscraper-*.whl to the target host and install with pipx:

apt install pipx        # if not already installed
pipx install arkmodscraper-*.whl

pipx installs the tool into an isolated virtualenv and wires up the arkmodscraper command automatically — no manual venv management or sudo required.

A PyPI release may happen at some point, which would make installation as simple as pipx install arkmodscraper — no wheel file required.

Development

make init      # install all dependencies (including dev)
make tests     # run the test suite
make coverage  # run tests with coverage report (80% minimum enforced)
make lint      # ruff check and format
make wheel     # build the distributable wheel

Development History

The original version of this tool was written by Michael Hoffmann in 2021, entirely without AI assistance. It used the steam Python library for authentication and BeautifulSoup to scrape the Steam Workshop subscription page.

When Valve changed their login API in late 2023 the steam library broke and was never fixed, taking the tool with it. Attempts to repair it using ChatGPT and JetBrains Junie were unsuccessful.

The tool was eventually brought back to life in 2026 with the help of Claude (Anthropic). The broken authentication and scraping approach was replaced entirely with a direct call to the Steam Web API (IPublishedFileService/GetUserFiles), which requires only a Web API key and Steam ID — no password, no 2FA, no screen scraping.