The Should-I-Build Test
Balance intended benefits against harms before turning capability into reality
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 94%
The Should-I-Build Test inserts an ethical decision between technical capability and deployment. Begin with the intended benefit, then identify unintended environmental, social, safety, and privacy effects that may appear at scale. Evaluate those effects through multiple controls: engineering safeguards, organizational leadership, education, rules, and regulation. The central question is not merely whether a team can build something, but whether it should—and under what constraints. Deployment does not end the process because consequences often become visible only through use. Teams therefore monitor outcomes, revise safeguards, and apply further engineering when earlier technologies create new problems. The result is a conditional go, redesign, restricted deployment, or stop decision supported by explicit reasoning.
Origin
Extracted from Masters in Business through Lord Browne's discussion of engineering's intended and unintended consequences.
Core principles
- 01Technical possibility is not moral permission
- 02Every engineered system can create unintended consequences
- 03Benefits and harms must be evaluated together
- 04Rules, education, leadership, and design all shape outcomes
- 05New engineering can repair harms caused by earlier engineering
How to run it
- 1
Define the intended benefit
State what human problem the technology is meant to solve and who should benefit. Make the claimed improvement measurable.
Pro tip Describe the benefit without referring to the novelty of the technology itself.
Watch out A capability is not a benefit merely because it is technically impressive.
- 2
Map unintended effects
List plausible second-order effects across safety, environment, privacy, behavior, inequality, and misuse. Include effects that may emerge only at large scale.
Pro tip Invite critics and affected stakeholders to challenge the team's assumptions.
Watch out Do not dismiss a consequence simply because nobody intended it.
- 3
Ask whether to proceed
Compare the benefits, harms, reversibility, and uncertainty. Decide whether the system should be built, redesigned, limited, or stopped.
Pro tip Record the assumptions that would change the decision.
Watch out Do not let sunk costs turn the review into a ceremonial approval.
- 4
Design layered controls
Combine engineering safeguards with policies, education, leadership oversight, and regulation where appropriate. Assign ownership for every control.
Pro tip Prefer controls that prevent harm over those that only respond afterward.
Watch out Policy alone cannot compensate for an inherently unsafe design.
- 5
Monitor and remediate
Measure real-world effects after deployment and compare them with the original assumptions. Apply more engineering or withdraw the system when the balance changes.
Pro tip Set thresholds that automatically trigger review or suspension.
Watch out Do not treat approval as permanent when evidence is still evolving.
In the wild
Hydrocarbons delivered light, heat, and mobility, but their use also destabilized the climate through carbon emissions. Applying the test reveals both the historic benefit and the unintended harm, leading to safeguards and remediation such as efficiency improvements, carbon capture, renewable energy, and policy constraints.
→ The original technology is evaluated honestly while engineering effort shifts toward reducing its unintended damage.
Common mistakes
Equating feasibility with permission
Teams proceed because they can build the technology without making a defensible decision about whether they should.
Looking only at intended outcomes
A narrow business case hides effects on people or systems outside the immediate customer relationship.
Freezing the original decision
Consequences change with adoption and scale. Refusing to revisit deployment turns early uncertainty into unmanaged risk.
Is it for you?
Best for
It is best for engineers, product leaders, boards, and policymakers evaluating consequential technologies.
Not ideal for
It is not ideal for trivial, reversible changes with negligible effects beyond the immediate user.
From the transcript
“all engineering has great intended consequences that most de great, sure, but also unintended consequences.”
“it's not right that engineers should say, let's do it because I can do it, should stop for a moment and say should I do…”
“the way to fix it is not to stop in engineering, is to apply more engineering to solving the problem.”
From the episode
Lord Browne Discusses the Equilibrium of Engineering
Lord Browne