How to measure progress

  • Creating useful metrics is hard and dangerous

  • Metrics will define the incentives that people will use to game the embedded system

  • Poorly constructed metrics can miss vital things that we did not understand at the time

  • Metrics cannot be used as the be all and end all of measurement due to their limitations and risks

  • Yet metrics are vital to understand how we can measure our progress on our aims

  • For each aim I would aim to work with those impacted to iteratively define and refine various metrics over a regular cadence

  • For high level aims the metrics must also be at high levels

    • Accepting that such metrics will gloss over and oversimplify details from time to time

    • Many of these metrics would be used to help define and prioritize other work later instead of being used immediately

  • These high level aims and metrics will then be used to help define lower level acceptance criteria for various efforts

    • I would try to make efforts to encourage prioritiziation instead of using time as a metric.

    • Where quality, scope, and time clash we should first look at time and if it makes sense to take longer. Or if some of the scope can be encapsulated to be done at a later time

  • I would push council to adopt regular post implementation reviews with those involved with projects and to publish them

    • Such efforts would need leadership to show that they are no blame situations. That they exist to help our future selves better understand the unexpected.