Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ work/
node_modules/
knnIndices/

async-profiler/

# emacs temporary files
*~
#*
86 changes: 85 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,4 +194,88 @@ python src/python/localrunFacets.py -source facetsWikimediumAll
```

Note that only comparison of taxonomy based facets is supported at the moment. We need to add SSDV facets support
to the sandbox facets module, as well as add support for other facet types to this package.
to the sandbox facets module, as well as add support for other facet types to this package.


# Profiling

Choose the profiler in **`src/python/localconstants.py`**:

```python
# 'JFR' – default (-XX:StartFlightRecording)
# 'ASYNC' – async-profiler agent + flame-graph
PROFILER_TYPE = 'JFR'
```

---

#### Quick setup — one command

#### Note: you will generally have to setup proper permissions for async-profiler to run, see Troubleshooting section below.

```bash
# from util/ directory
./gradlew setupAsyncProfiler
```

This creates

```
util/
└─ async-profiler/
├─ bin/ (profiler.sh, jfrconv, …)
└─ lib/ (libasyncProfiler.so | libasyncProfiler.dylib)
```

---

#### Manual install

```bash
wget https://github.com/async-profiler/async-profiler/releases/download/v4.0/async-profiler-4.0-linux-x64.tar.gz
tar -xzf async-profiler-4.0-linux-x64.tar.gz
mv async-profiler $LUCENE_BENCH_HOME/util/
# or set ASYNC_PROFILER_HOME in localconstants.py
```

---

#### Running a benchmark with async-profiler

Choose the profiler in **`src/python/localconstants.py`**:

```python
# enable the profiler
PROFILER_TYPE = 'ASYNC'
```

# run any benchmark
```bash
python src/python/localrun.py -source wikimedium10k
```

Output (in `logs/`):

* `*.jfr` — same format as JFR runs
* `*.html` — interactive flame-graph

---

#### Tuning (optional)

```python
ASYNC_PROFILER_OPTIONS = "interval=10ms,threads" # extra agent args
```

Keys `event, alloc, delay, file, log, jfr` are reserved and cannot be
overridden.

---

#### Troubleshooting

| Symptom | Fix |
|------------------------------------|--------------------------------------------------------------------------------------------------------------------------|
| **Library not found** | Ensure `async-profiler/lib/libasyncProfiler.so` (Linux) or `.dylib` (macOS) exists and `ASYNC_PROFILER_HOME` is correct. |
| **perf permission errors (Linux)** | `sudo sysctl kernel.perf_event_paranoid=1`<br/>`sudo sysctl kernel.kptr_restrict=0` |
| **No flame-graph generated** | Check console for warnings—usually `jfrconv` missing or agent failed to attach. |
64 changes: 63 additions & 1 deletion build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,66 @@ allprojects {
println "cmd: $commandLine"
}
}
}
}

task setupAsyncProfiler {
description = 'Download and unpack async-profiler unless its native library is already present.'
group = 'build setup'

// ─────────── platform detection ───────────
def version = '4.0'
def os = org.gradle.internal.os.OperatingSystem.current()
def classifier, libName

if (os.isLinux()) {
classifier = 'linux-x64'
libName = 'libasyncProfiler.so'
} else if (os.isMacOsX()) {
classifier = System.getProperty('os.arch').contains('aarch')
? 'macos-arm64'
: 'macos-x64'
libName = 'libasyncProfiler.dylib'
} else {
throw new GradleException("async-profiler: unsupported OS '${os.getName()}'")
}

def downloadUrl = "https://github.com/async-profiler/async-profiler/releases/" +
"download/v${version}/async-profiler-${version}-${classifier}.tar.gz"

// ─────────── up-to-date checks ───────────
def targetDir = project.file("${project.projectDir}/async-profiler")
def targetLibFile = project.file("${targetDir}/lib/${libName}")

outputs.file(targetLibFile)
onlyIf { !targetLibFile.exists() } // skip whole task once extracted

doLast {
println "▶ Installing async-profiler ${version} (${classifier})"

// 1) download
def dlFile = project.layout.buildDirectory
.file("tmp/async-profiler-${version}-${classifier}.tgz")
.get().asFile
dlFile.parentFile.mkdirs()

ant.get(src: downloadUrl, dest: dlFile, verbose: true)

// 2) extract (flatten leading dir)
copy {
from tarTree(resources.gzip(dlFile))
into targetDir
includeEmptyDirs = false
eachFile { f ->
f.path = f.path.replaceFirst('^async-profiler-[^/]+/', '')
}
}

// 3) cleanup
dlFile.delete()

if (!targetLibFile.exists()) {
throw new GradleException("Extraction finished, but ${libName} not found in ${targetDir}")
}
println "✔ async-profiler ready – native lib at ${targetLibFile}"
}
}
Loading
Loading