Experiment Design Kit Skill

When to Use

• Translating raw ideas into testable hypotheses with clear success metrics.
• Ensuring experiment briefs include guardrails, instrumentation, and rollout details.
• Coaching pods on best practices for multi-variant or multi-surface tests.

Framework

1. Problem Framing – define user problem, business impact, and north-star metric.
2. Hypothesis Structure – “If we do X for Y persona, we expect Z change” with assumptions.
3. Measurement Plan – primary metric, guardrails, min detectable effect, power calc.
4. Variant Strategy – control definition, variant catalog, targeting, and exclusion rules.
5. Operational Plan – owners, timeline, dependencies, QA/rollback steps.

Templates

• Experiment brief (context, hypothesis, design, metrics, launch checklist).
• Guardrail register with thresholds + alerting rules.
• Variant matrix for surfaces, messaging, and states.
• GTM Agents Growth Backlog Board – capture idea → sizing → prioritization scoring (ICE/RICE) @puerto/README.md#183-212.
• Weekly Experiment Packet – includes KPI guardrails, qualitative notes, and next bets for Marketing Director + Sales Director.
• Rollback Playbook – pre-built checklist tied to lifecycle-mapping rip-cord procedures.

Tips

• Pressure-test hypotheses with counter-metrics to avoid local optima.
• Document data constraints early to avoid rework during build.
• Pair with guardrail-scorecard to ensure sign-off before laun

…(truncated)

Guardrail Scorecard Skill

When to Use

• Setting non-negotiable metrics (stability, churn, latency, compliance) before launching tests.
• Monitoring live experiments to ensure guardrails stay within thresholds.
• Reporting guardrail status in launch packets and post-test readouts.

Framework

1. Metric Inventory – list guardrail metrics, owners, data sources, refresh cadence.
2. Threshold Matrix – define warning vs critical bands per metric / persona / region.
3. Alerting & Escalation – map notification channels, DRI, and decision timelines.
4. Exception Handling – document when guardrail overrides are acceptable and required approvals.
5. Retrospective Loop – log breaches, mitigations, and rule updates for future tests.

Templates

• Guardrail register (metric, threshold, owner, alert channel).
• Live monitoring dashboard layout.
• Exception memo structure for approvals.

Tips

• Tie guardrails to downstream systems (billing, support) to catch second-order impacts.
• Keep thresholds dynamic for seasonality but document logic.
• Pair with launch-experiment to ensure readiness before flipping flags.

Hypothesis Library Skill

When to Use

• Capturing new experiment ideas with consistent metadata.
• Referencing past wins/losses before prioritizing the backlog.
• Sharing reusable learnings across pods and channels.

Framework

1. Metadata Schema – hypothesis ID, theme, persona, funnel stage, metrics.
2. Assumptions Matrix – belief statements, supporting evidence, confidence rating.
3. Status Tracking – idea → scoped → running → decided → archived.
4. Learning Tags – impact summary, guardrail notes, follow-up ideas.
5. Governance Hooks – approvals, owners, review cadence.

Templates

• Intake form for new hypotheses.
• Learning card format (context, result, recommendation).
• Portfolio dashboard summarizing mix by theme/metric.

Tips

• Require at least one supporting data point before moving to prioritization.
• Use consistent tagging so search/filtering works across teams.
• Link to synthesize-learnings outputs to keep narratives fresh.