docs(templates): email/message template module plan

Design + 8-task TDD plan for the IIOS template module. Self-reviewed: provenance-write path, nullable-scope unique index, PII minimization. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 12:18:26 +05:30
parent 6dc9e4ffee
commit cea0a27118
1 changed files with 277 additions and 0 deletions
@@ -0,0 +1,277 @@
 # Email / Message Template Module — Implementation Plan
 **Owner:** Maaz · **Repo:** `iios` (`packages/iios-service`) · **Target:** first templates sending by Friday go-live.
 ## Goal
 A reusable template module so **any** message the system sends — welcome, payment receipt,
 onboarding reminders, drip — is produced from a stored (or ad-hoc) template with variables filled
 in, then handed to the **existing** outbound pipeline. One render path, one send path, provenance
 recorded for every send.
 From the July 16 meeting: *"template का एक पूरा module बनाना है… template में value भरोगे, और वो
 outbound क्यू में डाल दोगे।"*
 ## What already exists (the substrate — do NOT rebuild)
 - `OutboundService.send(channelType, target, payload, idempotencyKey?, scopeId?, purpose?)` —
  idempotency + per-target/per-tenant rate limits + delivery ledger (`IiosOutboundCommand`).
 - `CapabilityBroker` — policy gate + obligations + provider selection for egress.
 - `EMAIL` is a registered channel; `EmailProvider` exists (**HTTP**, not SMTP — see Out of Scope).
 - `EMAIL` is a first-class `IiosInteractionKind`; `IiosMessagePartKind` has `HTML`/`TEXT`.
 - `IiosActorKind` includes `SERVICE`/`BOT` (system sender is first-class).
 - `InboxModule` uses `OnModuleInit` — copy that pattern for the template seeder.
 The module adds **content/rendering** in front of this. It never talks to a provider directly.
 ## Design decisions (locked)
 | # | Decision | Rationale |
 |---|---|---|
 | 1 | **Channel-generic**: one template per `(key, channel)`; channels `EMAIL` / `SMS` / `INTERNAL` | Vivek wants the same confirmation on email *and* SMS; INTERNAL = app-to-app, no SMTP |
 | 2 | **Seed-from-files, DB-is-truth**: templates are files in the repo, seeded into the DB on boot if absent | Copy is version-controlled + code-reviewed; a later admin UI can edit the DB with no deploy; Friday needs no UI |
 | 3 | **Handlebars** rendering | Auto-escapes HTML (customer names go into email → XSS risk), logic-less, no code execution, one dep |
 | 4 | **Global default + scope override**: `scopeId` nullable — `NULL` = platform default, set = tenant override; resolve scoped-first-else-global | Seeds cleanly at boot (IIOS scopes are created lazily, so a boot seeder has no scope to seed into); leaves room for white-label |
 | 5 | **Provenance on `IiosOutboundCommand`** (4 columns), not a new `template_snapshot` table | Rendered content is already in `payload`; only provenance is missing. Honours the SOT's intent at 4 columns |
 | 6 | **`TemplateSource` = stored `{key,version?}` OR `{inline:{subject,html,text}}`** | Marketing hands over finished HTML (*"Maaz, HTML भेज सकते हैं"*); inline still renders vars + records provenance |
 | 7 | **Integration pattern B**: pure `render()` + thin `sendTemplated()` composer | Single entry point for callers; provenance guaranteed by construction; `OutboundService` stays content-agnostic |
 | 8 | **Caller = a service** (be-crm's system token); **recipient = a `target` address, never a login** | System mail has no user on the sending side; the recipient may have no account yet |
 ## Caller & auth model
 Every send is authenticated as the **calling app/service** (e.g. be-crm via its `APP_SECRETS`
 entry), verified by `SessionVerifier` like every other IIOS endpoint. The recipient is a plain
 `target` string — **not** an IIOS principal and **not** required to be logged in or registered.
 Sending a welcome email to an anonymous payer is the normal case: the app is the sender, the
 address is data. `scopeId` for the send is derived from the caller's principal (`org/app/tenant`).
 ## Data model
 ### New table — `IiosMessageTemplate`
 ```prisma
 enum IiosTemplateChannel {
  EMAIL
  SMS
  INTERNAL
 }
 /// The template SOURCE. DB is runtime truth; platform defaults are seeded from repo files on boot.
 /// Versions are immutable: a change writes a new (higher) version, never edits in place.
 model IiosMessageTemplate {
  id        String              @id @default(cuid())
  /// NULL = platform default (seeded). Set = a tenant scope's override of the same key.
  scopeId   String?
  scope     IiosScope?          @relation(fields: [scopeId], references: [id], onDelete: Cascade)
  key       String              // "welcome", "payment.receipt", "onboarding.reminder"
  channel   IiosTemplateChannel
  locale    String              @default("en")
  version   Int                 @default(1)
  subject   String?             // EMAIL only
  bodyHtml  String?             // EMAIL / INTERNAL
  bodyText  String?             // SMS, and EMAIL plaintext fallback
  /// Declared variable names — render throws if a declared var is missing (fail loud).
  variables Json?
  active    Boolean             @default(true)
  createdAt DateTime            @default(now())
  updatedAt DateTime            @updatedAt
  @@unique([scopeId, key, channel, locale, version])
  @@index([key, channel, locale, active])
 }
 ```
 `IiosScope` needs the back-relation `messageTemplates IiosMessageTemplate[]`.
 **Resolution:** `resolve(key, channel, locale, scopeId?)` returns the highest-`version` `active`
 row, preferring `scopeId = <caller scope>` and falling back to `scopeId IS NULL`. No match → 404.
 ### Provenance columns on `IiosOutboundCommand`
 ```prisma
  templateKey     String?
  templateVersion Int?
  templateLocale  String?
  renderedHash    String?   // sha256 of the rendered content — replay/audit
 ```
 All nullable: non-templated sends (if any) leave them null.
 **How provenance is written (review finding):** `OutboundService.send` **creates the command row
 itself** (both the RATE_LIMITED and PENDING paths), so a caller cannot stamp provenance after the
 fact without a racy update. Therefore `send()` gains one optional trailing arg —
 `provenance?: { templateKey; templateVersion; templateLocale; renderedHash }` — written into the
 same `create()` in both paths. `OutboundService` stays content-agnostic: it does not render or
 resolve templates, it only persists four opaque strings it is handed. This is what makes
 decision 7's "provenance guaranteed by construction" true. **This is a change to an existing file**
 (`adapters/outbound.service.ts`) and its spec — call it out in the PR.
 ### Migrations (hand-written, non-destructive)
 1. `add_message_template` — the enum + table + `IiosScope` back-relation. **Plus a partial unique
   index for global rows (review finding):** the `@@unique([scopeId, key, channel, locale, version])`
   does **not** prevent duplicate *platform-default* rows, because Postgres treats `NULL` scopeId as
   distinct (`NULL != NULL`) — the same footgun handled in the inbox idempotency migration. Add:
   `CREATE UNIQUE INDEX "IiosMessageTemplate_global_key" ON "IiosMessageTemplate"("key","channel","locale","version") WHERE "scopeId" IS NULL;`
   so a double-seed or race cannot create two defaults for the same key.
 2. `add_outbound_template_provenance` — the 4 nullable columns on `IiosOutboundCommand`.
 ## Module structure
 ```
 packages/iios-service/src/templates/
  template.channel.ts        // IiosTemplateChannel re-export/helpers if needed
  template.model.ts          // TemplateSource, RenderedContent, CreateSendInput types
  template.repository.ts     // resolve(): scoped ?? global, highest active version
  template.renderer.ts       // Handlebars compile+render; escaping; missing-var throw
  template.service.ts        // render(source, vars) — pure; resolve + renderer
  templated-sender.ts        // sendTemplated(): render -> OutboundService.send -> stamp provenance
  template.seeder.ts         // OnModuleInit: seed file defaults into DB if (key,channel,locale,version) absent
  template.controller.ts     // POST /v1/templates/send, POST /v1/templates/preview
  template.dto.ts            // SendTemplateDto, PreviewTemplateDto (class-validator)
  template.module.ts
  seeds/
    welcome.email.ts         // { key, channel, locale, version, subject, html, text, variables }
    payment-receipt.email.ts
    payment-receipt.sms.ts
    onboarding-reminder.email.ts
  templates.spec.ts          // TDD, real Postgres (localhost:5434), mirrors inbox.spec.ts
 ```
 **Scope boundary:** `render()` is channel-generic (it can render an `INTERNAL` template to
 `{subject,html,text}`). `sendTemplated()` covers **external** channels (`EMAIL`/`SMS`) via
 `OutboundService`. `INTERNAL` *delivery* (render → create an in-app `Interaction`, no SMTP) reuses
 `render()` but is wired by the messaging/mirror spec — this module never imports the messaging layer.
 ## Contract
 ```ts
 type TemplateSource =
  | { key: string; version?: number }
  | { inline: { subject?: string; html?: string; text?: string } };
 interface RenderedContent { subject?: string; html?: string; text?: string }
 // template.renderer.ts — PURE (no I/O): given a template's raw strings + vars, produce content.
 // template.service.ts render() — resolves the template from the DB (I/O), then calls the pure renderer.
 // For an inline source there is no DB row, so declared-variable validation is skipped — inline
 // content is the caller's responsibility; only stored templates enforce their declared `variables`.
 render(source: TemplateSource, vars: Record<string, unknown>, opts: { channel: IiosTemplateChannel; locale?: string; scopeId?: string }): Promise<RenderedContent>
 // templated-sender.ts — external egress
 sendTemplated(input: {
  source: TemplateSource;
  channel: 'EMAIL' | 'SMS';
  target: string;                 // email address / phone
  vars: Record<string, unknown>;
  scopeId?: string;
  idempotencyKey: string;         // REQUIRED — e.g. "receipt:<stripe_session_id>"
  purpose?: string;
 }): Promise<IiosOutboundCommand>
 ```
 **HTTP** (`SessionVerifier`-auth'd; `scopeId` from the caller's principal):
 - `POST /v1/templates/send` → `sendTemplated`. OPA action `iios.template.send`;
  inline source additionally gated on `iios.template.send.inline` (arbitrary HTML to customers).
 - `POST /v1/templates/preview` → `render` only, returns `RenderedContent`. No send. For marketing/QA.
 ## Task-by-task (TDD)
 Each task: write the failing test → run it (confirm red) → implement → run (green) → commit.
 **T1 — Renderer (`template.renderer.ts`)**
 - Tests: substitutes `{{firstName}}`; **escapes `<script>` in a name**; `{{#if}}`/`{{#each}}`;
  a declared-but-missing variable throws; renders `bodyHtml` and `bodyText` independently.
 - Impl: Handlebars, `noEscape:false`; validate declared `variables` present.
 **T2 — Migrations + repository (`template.repository.ts`)**
 - Migration 1 (table+enum). Regenerate client.
 - Tests: `resolve` returns highest active version; **scoped overrides global**; unknown key → NotFound;
  inactive versions ignored.
 **T3 — `render()` service tying resolve+renderer, incl. inline source**
 - Tests: stored `{key}` resolves+renders; `{inline}` renders without a DB row; wrong channel → 404.
 **T4 — Provenance: `OutboundService.send` param + migration + `sendTemplated()`**
 - Migration 2 (4 columns).
 - Modify `OutboundService.send` to accept the optional `provenance` arg and write it into the command
  `create()` on both the RATE_LIMITED and PENDING paths. Extend `outbound.service.spec.ts`: a send
  with provenance persists all four fields; a send without leaves them null (no regression).
 - Then `sendTemplated()`. Tests: composes render→`OutboundService.send`; **stamps
  `templateKey/version/locale/renderedHash`** on the command; **idempotent per key** (replay → one
  `IiosOutboundCommand`); inline → `templateKey` null, `renderedHash` set.
 **T5 — Seeder (`template.seeder.ts`, `seeds/*`)**
 - Tests: boot seeds the file defaults as `scopeId NULL`; **re-seed is idempotent** (no dupes);
  a bumped file version inserts a new row, leaves the old.
 **T6 — Controller + DTOs**
 - Tests (HTTP, boot against sandbox provider so no real mail): `POST /send` renders+queues;
  unknown key → 404; bad body → 400; missing auth → 400/401; `POST /preview` returns content, sends nothing.
 **T7 — Whole-suite gate**
 - `vitest run` (all packages), `npm run boundary`, `npm run build` all green.
 - Manual: boot locally, `POST /v1/templates/send` with the `welcome` seed via sandbox, inspect the
  `IiosOutboundCommand` row for payload + provenance.
 **T8 — PII redaction of outbound commands (fast-follow; lever #2)**
 - Extend `RetentionService` so its sweep also redacts aged `IiosOutboundCommand` rows: raw `target`
  and rendered `payload` → redacted, while `templateKey/version/renderedHash` + `scopeId` are kept
  for audit. Reuse the existing redact-in-place pattern (currently applied to `iiosMessagePart`).
 - Tests: a command past its window has `target`/`payload` redacted but provenance intact; a command
  under compliance hold is skipped; audit row `retention.redacted` written.
 - Independent of T1–T7 — can land immediately after the module without blocking Friday.
 ## Error handling
 - Unknown key/channel/locale → `NotFoundException` (404).
 - Declared variable missing → throw (400) — never send a half-rendered receipt.
 - Transport failure → surfaced as `FAILED` on the command by `OutboundService`, never thrown to caller.
 - Idempotent replay (same key) → returns the existing command; never double-sends.
 ## Out of scope (separate specs — this module unblocks them)
 1. **`SmtpProvider`** (nodemailer, `accounts@`/`ceo@lynkeduppro.com`) in IIOS's capability registry.
   Today's `EmailProvider` is HTTP; SMTP is a new provider flipped on by env. Without it, sends land
   in the sandbox.
 2. **`POST /webhooks/stripe` in be-crm** — verify Stripe signature → mint a **service token** →
   call `POST /v1/templates/send` (welcome + receipt), keyed on the Stripe object id. This is the
   "backend" the frontend-only payment site lacks.
 3. **Inbox mirror** (post-registration): render an email → also create an `Interaction(kind=EMAIL)`
   so it appears in the customer's in-app inbox. `INTERNAL` delivery lives here. Mirror only *after*
   registration (no inbox exists before).
 ## PII minimization
 Sending email means IIOS must *touch* the recipient's address (you can't mail without it) and the
 rendered body holds their name — so `IiosOutboundCommand.target`/`payload` become PII. You cannot
 avoid IIOS touching it; you avoid it **accumulating** and being **cheaply reachable**. Three levers,
 by impact:
 1. **Rotate the `Qwerty@a2` signing secret — precondition for prod, highest impact, not code.**
   The stored PII is only dangerous because any leaked token can be brute-forced back to the secret,
   letting an attacker forge a token and read the outbound table. A strong random secret leaves the
   PII in place but **unreachable by forgery** — 90% of the risk closed by one env change. **Gate:
   do not point the real SMTP provider at production until this is rotated.**
 2. **Redact the raw address + body after delivery (fast-follow task — see T8).**
   IIOS keeps `templateKey/version/renderedHash` + the opaque `scopeId` for audit, but the raw
   `target` and rendered `payload` are redacted-in-place once past their retention window. The
   `RetentionService` already does exactly this for interaction message parts (`bodyText →
   '[redacted]'`); it just doesn't cover `IiosOutboundCommand` yet. Reuse the same sweep.
 3. **Tokenized recipient (future, when MDM ships).** The clean model is: callers pass a `userId`,
   IIOS resolves the address from MDM *at send time* and never persists it. Not available today
   (MDM isn't deployed; `externalId` is a UUID with no email). **Design accommodation now:** keep
   `target: string` for Friday, but treat it as an opaque "where to send" so it can later become
   `{ userId }` resolved via MDM without changing the `sendTemplated` contract shape.
 **Rejected shortcut:** having be-crm send SMTP directly so IIOS never sees the address. It "avoids"
 the PII but breaks the inbox mirror and the single-send-path (IIOS would have no record to copy into
 the inbox) — trading a solvable security problem for a broken feature.
 ## Other risks
 - **Friday, no test env:** first real sends go to paying customers from an unproven path. Insist on a
  sandbox target / 100%-off test coupon before wiring the real SMTP provider (compounds with lever #1
  above — don't send real mail from prod until the secret is rotated *and* a safe test target exists).