How do you measure local dev loop comprehensively?

[pastelsky]

👋 Hi there,

Came across the DPH framework recently and am curious about the Developer build time (DBT) metric that LinkedIn uses.

The document defines DBT as — 

TL;DR: How much time do developers spend waiting for their build tool to finish
its work.
The intention of this metric is to measure the amount of time that human beings
spend waiting for their build tool to complete. This is measured as the
wall-clock time from when the build tool starts a "build" to when it completes.
This measures all human-triggered builds. At present, this includes all
human-triggered builds with the following tools:

• Gradle
• Bazel
• Ember 
• Xcode 

Do you measure tool times as an aggregate?

Or aggregate it basis lifecyles like —

1. Version control (git performance)
2. Application Build
3. Running unit tests
4. Running integration tests
   etc?

[mkanat]

So, mostly we don't aggregate, because it's hard to take action on the aggregate.

We have been experimenting with a metric that measures the time from the start of a code review to the time that a deployment completes. (Because from the moment a code review starts, that's where there's a clear expectation of the whole process completing within some reasonable period of time.) It has produced some useful insights, but only because we can slice it to see the times between every touchpoint in the process (like how long every review comment took to come in, for example), and we have lots of other metrics we can look at to figure out what's going on in the detailed areas.

[pastelsky]

I see. Post review journey is something we measure as well, but was curious on if there are effective ways that have been applied to measure the time between change start (first checkout) to pull request (the "active development time on a branch" perhaps?)

The timespan itself may be too high level of a metric, but it can capture things like —

— how easy it is to contribute to a codebase
— sufficiency of local (or remote) development hardware
— intuitiveness and speed of the local build toolchain

which are bits that can be meaningfully improved by dev prod.

[grantjenks]

The Developer Build Time metric at LinkedIn includes (2) application build and (3) running unit tests from your list. Different build systems provide different telemetry and granularity as to the individual steps executed as part of the build and their duration. The steps have been analyzed in different, targeted, studies but are not surfaced at the director+ level. For the most part, the combination of 2+3 is reported.

To measure from the start of development, we could connect the creation of remote development environments with the branch name used in the PR. Some companies also encourage pushing branches and creating "draft" PRs as early as possible to track the start of development. These can help measure how easy it is to contribute to a codebase.

We've also though a lot about the "ramp up" time for engineers in the context of onboarding, training, and switching teams. We've tried looking at things like "merge rate" compared to peers but it breaks down at the team level and works best at the company level. The easiest thing to measure "ramp up" time has been simply to ask developers with targeted surveys 30, 60, and 90 days after joining.