[Question][framework] Configuring DevLake for ~6.5k GitLab repos and company-wide metrics

[mfrancisc]

We are sizing a DevLake deployment for a large GitLab estate and want to know whether a single instance is expected to handle this, and how we should configure projects/blueprints if so.

Scale

• One product org already has ~6,500 GitLab repos; we need to plan for tens of thousands across the company.
• Teams are disjoint (no shared repos between team projects).
• We still need org- and company-wide metrics (e.g. Cycle Time) from one database. We do not use Grafana; a metrics API reads the same MySQL.
• Splitting into isolated DevLake+MySQL stacks would speed collection, but then we could not compute org- or company-wide Cycle Time with a single query against one database.

What we think is the intended setup (please correct us) :

• One lake process, one MySQL (we know a second instance on the same DB_URL hits the exclusive _devlake_locking_stub lock).
• Many team-sized projects/blueprints (tens to ~150 repos each), not one project with 6,500 scopes.
• PIPELINE_MAX_PARALLEL > 1, staggered crons, incremental sync, skip heavy gitextractor options if needed.

Questions

1. Has anyone run DevLake successfully at a few thousand GitLab repos on one instance? What project size, PIPELINE_MAX_PARALLEL, and sync policy actually worked?
2. Is the guidance above right, or is there a better project/blueprint layout for this?
3. At this scale, is the bottleneck expected to be the single runner (sequential blueprints / sequential GitLab stages) rather than MySQL?
4. If one instance cannot keep a daily incremental cycle, is the intended path still “more hardware on one process”, or is multi-instance sharing one DB something the project would consider?

Related: #8448, #8802, #8260

[mieliespoor]

The problems I see with DevLake is scaling. Given the data collection nature from DevLake, it hits api's fairly hard. A tool like GitLab would be hit hard and rate limits seen regularly, at least for me, and in deployments like in the OP above. With things like PGBouncer for PostgreSQL, scaling database side is achievable and fairly simple to achieve. That means we can scale up the processors that collect the data. Doing that, however, runs the risks or hitting rate limits or impacting tool performance where rate limits do not exist for a tool.

[mfrancisc]

I agree the GitLab API (or GitLab itself) is a hard ceiling. Extra DevLake parallelism on one token would only burn the same quota faster. GitLab’s authenticated limit is per user, not per PAT, and requests from one DevLake instance also share one IP, which can be a second cap.

What we could do on the GitLab side, rather than pushing one token as hard as possible:

• One GitLab connection per team/org, each with its own bot (or group access token)
• Set rateLimitPerHour on each connection below what GitLab actually allows (or 0 to follow RateLimit-Limit response header)
• Incremental collection after the first backfill, and keep PIPELINE_MAX_PARALLEL modest on a given connection

That is how we could try to avoid melting GitLab.
But I guess that this does not by itself get a daily incremental cycle done at ~6.5k repos if the limiter is still a single runner and sequential GitLab stages

So we would still want to understand:

1. Has anyone actually run a few thousand GitLab repos on one instance, and with what project size / PIPELINE_MAX_PARALLEL / sync policy?
2. If one process cannot finish a daily incremental cycle even when GitLab is paced correctly, is the intended answer still “bigger machine, one process”, or adding support for several collectors sharing one database with disjoint scopes, one scheduler and many workers could be a potential enhancement to the framework ?