# AI Architecture Checklist

**Digital Solutions Delivery — AI Platform Architecture Studio**
digitalsolutiondelivery.com

---

## How to use this checklist

Use this before starting implementation — not after. Each section should be completed (or deliberately deferred with documented reasoning) before the corresponding implementation phase begins.

Mark each item: ✓ Done · ○ Deferred (with reason) · ✕ Not applicable

---

## Section 1: Knowledge Layer

Before any agents or AI features are built, verify:

- [ ] Domain entities are defined (what "things" exist in your domain)
- [ ] Entity relationships are typed and directed (not just "related to")
- [ ] Attributes are defined for each entity type
- [ ] Claims about entities are separated from raw data
- [ ] Claim provenance is tracked (source, timestamp, conditions)
- [ ] Knowledge retrieval format is defined for agent consumption
- [ ] There is a plan for updating the knowledge base when facts change
- [ ] The knowledge model is reviewed by a domain expert, not just a technical one

---

## Section 2: Agent Schema

Before writing any agent code:

- [ ] The agent's capability boundary is defined (what it does AND what it doesn't do)
- [ ] Each tool the agent uses has a precise name (unambiguous, no generic terms)
- [ ] Tool input schemas are typed — not accepting arbitrary text
- [ ] Tool output schemas are typed — returning structured, predictable formats
- [ ] Error conditions for each tool are defined and handled explicitly
- [ ] Side effects are documented (does this tool modify state? which state?)
- [ ] The agent's memory scope is defined (in-context, session, or long-term)
- [ ] Context window management strategy is defined for long conversations

---

## Section 3: Multi-Agent Design (skip if single-agent)

- [ ] Agent responsibilities are non-overlapping — no two agents do the same thing
- [ ] Handoff protocol is defined: what data transfers, in what format
- [ ] Receiving agent validates handoff payload before accepting the task
- [ ] Completion signaling is defined — sender knows when task is done or failed
- [ ] Orchestration decision logic is documented — how is the right agent selected?
- [ ] Fallback behavior is defined for when an agent fails mid-task
- [ ] Agent-to-agent communication is logged for debugging

---

## Section 4: Integration Contracts

Before connecting to external systems:

- [ ] Each external dependency is documented: what it provides, what format, what SLA
- [ ] API contracts are versioned — breaking changes are detected before deployment
- [ ] Authentication and authorization for each integration is implemented (not hardcoded)
- [ ] Rate limits and quotas for external APIs are known and handled
- [ ] Failure modes for each external dependency are defined and tested
- [ ] Data transformation between external formats and internal schema is explicit
- [ ] There is a circuit breaker or fallback for each external dependency

---

## Section 5: Validation

Before calling anything production-ready:

- [ ] Performance metrics are defined before measurement (not reverse-engineered from results)
- [ ] Evaluation dataset is separate from development dataset — no leakage
- [ ] If using historical data: lookahead bias has been audited in all feature calculations
- [ ] Walk-forward or holdout validation is used — not just in-sample performance
- [ ] Edge cases are tested: unusual inputs, empty responses, API failures
- [ ] Cost model is validated: token counts, API costs, scaling economics are realistic
- [ ] A human has reviewed a sample of outputs against the correctness definition

---

## Section 6: Deployment Readiness

Before production deployment:

- [ ] Monitoring exists for output quality (not just system uptime)
- [ ] There is an alerting threshold for output degradation
- [ ] A rollback plan exists and has been tested
- [ ] The system's behavior when the AI API is unavailable is defined
- [ ] PII and sensitive data handling in AI context windows is audited
- [ ] Rate limiting exists to prevent runaway API costs
- [ ] The deployment is documented — another developer can reproduce it

---

## Red flags — stop and reconsider if:

- Any tool schema accepts a string with no further validation
- Agent output is passed to another system without type checking
- Performance numbers come from the same data used for prompt development
- "We'll add validation later" appears anywhere in the plan
- No one has defined what "wrong output" looks like

---

*Use this checklist as a pre-implementation review. Deferred items should be explicitly documented with the reason and when they'll be addressed — not silently skipped.*

**Architecture review available at digitalsolutiondelivery.com/services**