pgremapper performance limit?

[k0ste]

Hi, I find the cluster, where cancel-backfill is not super fast as usual (took minutes)
1423 osds | 37920 pgs - not super big. But have 38 rack buckets with 67 hosts

What I see is that pgremapper use only ~ 3 cores of 24 cores CPU, and that's suspicious (low) for golang. May be this some limit in code or on compiler? Or scalability milestone for an application?

[jbaergen-do]

Were any PGs unclean at the time? pgremapper will issue quite a few extra commands in that case, and would be limited in performance by command round-tripping. Increasing --concurrency can help with this.

If you run with --verbose it will spit out all of the Ceph commands being run; I'd be interested to know whether it's issuing commands the whole time or if it was sitting there thinking.

[k0ste]

If you run with --verbose it will spit out all of the Ceph commands being run; I'd be interested to know whether it's issuing commands the whole time or if it was sitting there thinking.

It was thinking. Everything I said concerns not what the pgremapper will do actually, but the moment when the pgremapper plan what to do, e.g. dry-run: cancel-backfill --verbose (without --yes)

[jbaergen-do]

Hmm, OK. The OSD and PG counts are consistent with some systems we've tested on in the past, and I don't think the rack/host count should affect cancel-backfill. Reviewing the code, I see that we do run the PG calculations in parallel, controlled by the concurrency setting, but depending on how much of the computation time is spent in mappingState.tryRemap(), the lock in that function might be a bottleneck...

[k0ste]

the lock in that function might be a bottleneck...

In perf top I see this

This guy?

[jbaergen-do]

Oh, interesting, that appears to be spending a bunch of time in garbage collection. Maybe there are things we can do to be more memory-allocation-efficient here.

[k0ste]

Maybe there are things we can do to be more memory-allocation-efficient here

We would be happy to test the patch if it is possible to do something in this place. According to my observations, the pgremapper can consume 7-8 cores in another places. In this cluster it is limited to 2-3 cores

[jbaergen-do]

Apologies - this is still on our list but it hasn't been able to bubble to the top yet