Skip to content

Architecture Decision Records

This page documents the key architectural decisions made for Imbi. We use Architecture Decision Records (ADRs) as described by Michael Nygard to capture important architectural choices and their rationale.

What is an ADR?

An Architecture Decision Record (ADR) is a document that captures an important architectural decision made along with its context and consequences. Each ADR describes:

  • Context: The issue or requirement that prompted the decision
  • Decision: The change or approach being proposed or adopted
  • Status: Whether the decision is proposed, accepted, deprecated, or superseded
  • Consequences: The positive and negative outcomes of the decision

ADR Index

Core Architecture

ADR 0001: Record Architecture Decisions

Date: 2025-12-30 | Status: Accepted

Establishes the practice of using Architecture Decision Records to document architectural decisions on this project, following Michael Nygard's format.

Key Points:

  • Lightweight ADR process adopted
  • ADRs stored in version control alongside code
  • Provides historical context for architectural choices

ADR 0002: Authentication and Authorization Architecture

Date: 2025-12-30 | Status: Accepted

Comprehensive design for Imbi's authentication and authorization system, including OAuth2/OIDC integration, JWT tokens, API keys, and permission management.

Key Decisions:

  • Graph database (originally Neo4j, since migrated to Apache AGE on PostgreSQL) for user and permission data (natural fit for permission inheritance)
  • ClickHouse for audit logs (time-series optimized with automatic TTL)
  • JWT tokens with revocation list (stateless with security)
  • Argon2id for password hashing (modern, memory-hard algorithm)
  • Hybrid RBAC + resource-level permissions (flexible and powerful)
  • FastAPI dependency injection for authorization (not middleware)
  • Dual authentication for services (JWTs and API keys)

Authentication Methods:

  • Local username/password
  • OAuth2 (GitHub, Google, Generic OIDC)
  • JWT tokens for API access
  • API keys for service-to-service communication

ADR 0003: Email Sending Architecture

Date: 2026-01-01 | Status: Accepted

Design for transactional email system supporting password reset, welcome emails, email verification, and security alerts.

Key Decisions:

  • Direct SMTP using Python's smtplib (no vendor lock-in)
  • Templates stored in source code (version control, code review)
  • FastAPI BackgroundTasks for async delivery (simple, no infrastructure)
  • Exponential backoff retry with dead letter queue
  • Mailpit for development testing (web UI, SMTP server in Docker)
  • ClickHouse for email audit logs (1-year retention)
  • Jinja2 with dual HTML/text templates (accessibility)
  • Table-based layout with inline CSS (email client compatibility)

Email Types:

  • Password reset
  • Welcome messages
  • Email verification
  • Security notifications

ADR 0004: Phase 5 Authentication Enhancements

Date: 2026-01-01 | Status: Accepted

Seven security and usability enhancements to the authentication system, addressing gaps identified after Phase 4 completion.

Key Enhancements:

  1. Token Revocation: Implement actual logout (complete Phase 2 TODO)
  2. OAuth Token Encryption: Encrypt provider tokens using Fernet
  3. Rate Limiting: Using slowapi library to prevent brute force
  4. API Keys: Full-featured API keys with scopes and usage tracking
  5. Session Management: Enforce max concurrent sessions setting (not implemented; the limit helpers were removed)
  6. MFA/2FA: TOTP-based multi-factor authentication with backup codes
  7. Token Rotation: Rotate refresh tokens on every use (security best practice)

Security Improvements:

  • OAuth tokens encrypted at rest (Fernet symmetric encryption)
  • Token revocation on logout (immediate invalidation)
  • Rate limiting on auth endpoints (5/min login, 10/min refresh, etc.)
  • MFA support with TOTP and backup codes
  • Refresh token rotation (detects token theft)
  • Session limits (prevent session hijacking); not implemented

API Keys:

  • Format: ik_<32 hex characters>_<secret> for easy identification
  • Scoped permissions (least privilege)
  • Usage tracking in ClickHouse
  • Rotation support
  • Expiration enforcement

ADR 0005: File Upload Storage Architecture

Date: 2026-02-06 | Status: Accepted

Design for general-purpose file upload system backed by S3-compatible storage, supporting icons, avatars, and documents for all Node-based entities.

Key Decisions:

  • S3-compatible object storage with aioboto3 for async operations
  • LocalStack for development (full AWS API emulation)
  • Apache AGE (PostgreSQL) for upload metadata (consistent with all other entities)
  • Presigned URL redirects (307) for efficient file serving
  • Pillow for thumbnail generation, filetype for magic-byte validation
  • WEBP thumbnails at 256x256 max, maintaining aspect ratio

Upload Features:

  • Content type validation with magic-byte verification
  • Configurable size limits (default 50 MB)
  • Automatic thumbnail generation for raster images
  • Permission-based access control (upload:create, upload:read, upload:delete)

ADR 0006: Project Identity and Multi-Type Support

Date: 2026-02-13 | Status: Accepted

Replaces single-type, slug-keyed project identity with a Nano-ID primary key and many-to-many project type relationships, enabling stable URLs and natural blueprint composition.

Key Decisions:

  • Nano-ID (21-char URL-safe) as the canonical project identifier
  • Many Project -[:TYPE]-> ProjectType relationships (was: single)
  • Blueprint aggregation across all of a project's types, merged by priority
  • API paths shift from /projects/{type_slug}/{slug} to /projects/{id}

ADR 0007: Relationship Blueprints

Date: 2026-02-20 | Status: Accepted

Extends the blueprint system to relationships (edges), so admins can add custom properties to [:DEPLOYED_IN] and other edges without code changes.

Key Decisions:

  • kind: 'node' | 'relationship' discriminator on Blueprint
  • Relationship blueprints declare (source, edge, target) triple
  • Edge properties become data-driven, mirroring the node-blueprint model

Plugin Platform

ADR 0008: Plugin System Architecture

Date: 2026-04-27 | Status: Accepted

Establishes Imbi as a SaaS IDP with a stable, versioned plugin model. Plugins are standalone Python packages discovered via entry points; capabilities are stored as Plugin graph nodes linked to a shared ThirdPartyService (now Integration).

Key Decisions:

  • imbi.plugins entry-point group; manifests are ClassVars on handler classes
  • ThirdPartyService + ServiceApplication + Plugin triplet in the graph
  • One service record can back multiple capabilities (aws-ssm, aws-cloudwatch-logs, aws-iam-ic)
  • Project-type and per-project USES_PLUGIN edges with explicit overrides
  • Plugin types grow incrementally (configuration, logs, then identity, deployment)

ADR 0009: Database-Driven OAuth Provider Configuration

Date: 2026-04-30 | Status: Accepted

Moves OAuth provider configuration (Google, GitHub, OIDC) from IMBI_AUTH_OAUTH_* environment variables into the graph database, manageable via the admin UI.

Key Decisions:

  • New OAuthProvider node with Fernet-encrypted client_secret_encrypted
  • 30-second TTL cache keyed by slug, invalidated on writes
  • New admin endpoints gated by oauth-providers:read|write permissions
  • Per-provider env vars deleted; oauth_auto_link_by_email / oauth_auto_create_users remain

ADR 0010: Identity Plugin Architecture

Date: 2026-05-05 | Status: Accepted

Adds a third plugin type — identity — for backends where caller identity matters (AWS IAM IC, GitHub, generic OIDC). Other plugin types declare identity_plugin_id; the API materializes per-user credentials at call time.

Key Decisions:

  • identity plugin type alongside configuration and logs
  • Per-user IdentityConnection node with Fernet-encrypted tokens, scoped to (User, Plugin)
  • Identity plugins can double as login providers (used_as_login: bool)
  • First-party identity entry points in imbi-plugin-aws, imbi-plugin-github, imbi-plugin-oidc

ADR 0011: Graph-Based Project Scoring

Date: 2026-04-15 | Status: Accepted

Replaces v1's fact-based scoring with a blueprint-aware, graph-driven scoring policy model. Attribute policies ship first; event policies are deferred until the integration ingestion path is settled.

Key Decisions:

  • ScoringPolicy node with value_score_map or range_score_map
  • Effective-attribute-set + optional TARGETS → ProjectType for policy selection
  • Materialized Project.score, history in ClickHouse (score_history + score_latest MV)
  • Valkey Streams queue with per-project debounce for async recomputation
  • CH-before-AGE write ordering for durable history

ADR 0012: Plugin Manifest Third-Party Service Template

Date: 2026-05-05 | Status: Accepted

Plugin manifests declare a third_party_service template and an embedded icon asset; install-time auto-provisions the ThirdPartyService, dedupes capabilities from the same package, and stops re-entering vendor boilerplate.

Key Decisions:

  • PluginManifest.third_party_service and PluginIcon (via importlib.resources)
  • One service record per template ID; multi-entry-point packages dedup
  • Manifests carry no secrets; ServiceApplication is still operator-supplied
  • Re-install preserves operator edits; opt-in "managed by manifest" mode

ADR 0013: Deployment Plugin Type

Date: 2026-05-08 | Status: Accepted

Adds a fourth plugin type — deployment — so the release-train UI can trigger workflows, draft AI release notes, and record deployments end-to-end against any plugin-supported backend.

Key Decisions:

  • deployment plugin type with a small protocol (list refs/commits, trigger, tag/release)
  • Separate github-deployment entry point alongside the github identity plugin
  • workflow_dispatch with {environment, ref} inputs as the v1 trigger
  • Deploys run as the human via identity_plugin_id
  • Anthropic client in imbi-common for server-side release-note drafting

ADR 0014: Generic Plugin-Entity CRUD Abstraction

Date: 2026-05-06 | Status: Accepted

Replaces hand-coded per-plugin entity routers (e.g., aws_accounts.py) with a generic CRUD surface driven by vertex_labels / edge_labels declared in plugin manifests.

Key Decisions:

  • Generic /admin/plugins/{slug}/entities/{label} routes mounted once
  • Edge endpoints for plugin-declared (source, edge, target) triples
  • Manifest-driven admin UI consuming JSON Schemas
  • The AWS-specific code was migrated in place; no URL compatibility was kept (pre-release)

Operations & Integrations

ADR 0015: CycloneDX 1.7 as the SBoM Standard

Date: 2026-05-27 | Status: Accepted

Standardizes the SBoM spec version accepted on PUT .../releases/{id}/sbom and the recommended producer for build CI.

Key Decisions:

  • Accept CycloneDX 1.7 on the wire (1.5/1.6 silently drop fields Imbi reads)
  • Single recommended producer that covers the polyglot service population and emits the per-component metadata needed for ReleaseComponentEdge scope/groups

ADR 0016: Delegated Execution for Scheduled Jobs

Date: 2026-06-29 | Status: Accepted

Adds an OAuth2 token-exchange (RFC 8693) delegation primitive plus a Consent model so the imbi-scheduler service (apps/scheduler) can run scheduled jobs as a user, and externalizes the score-recompute daily tick off the API process.

Not implemented. imbi-api has no token-exchange grant, no Consent node, and no auth:delegate permission. The scheduler refuses run-as-user tasks. See scheduler ADR 0002 and ADR 0003.

Key Decisions:

  • New urn:ietf:params:oauth:grant-type:token-exchange grant minting short-lived JWTs with sub=<user> and an act/azp claim naming the scheduler SA
  • Consent graph node recorded under the user's own session; scheduler stores no user secrets
  • AuthContext.actor field + actor_name; permission resolution uses the user's permissions (including user-only CAN_ACCESS ACLs)
  • New auth:delegate permission and imbi-scheduler service account; system jobs use the SA directly
  • Score-recompute daily tick (run_daily_tick) moves to a scheduler-owned cron; the Valkey-stream consumer stays in imbi-api

ADR 0017: Autonomous Deployment Principals

Date: 2026-09-02 | Status: Accepted

Lets an Imbi service account drive GitHub-backed deployment capabilities with the Integration's own GitHub App installation token — no per-user IdentityConnection — under Imbi-side controls that bound what an autonomous actor may do. The service-account half of the problem ADR 0016 frames.

Key Decisions:

  • The identity/service-credential fork keys on the principal (auth.user is None), not on a per-call-site best_effort_identity flag
  • New cross-cutting integration:act-as-service permission, additive to the capability's own and granted to no non-admin default role
  • Organization membership enforced in _resolve_and_context, so every deployment endpoint inherits it
  • New default-false Environment.allow_autonomous; human authority in an environment is unchanged
  • AuthContext.internal exempts Imbi's own in-process workers, which hold no permissions and no memberships
  • An autonomous principal cannot set acknowledge_ci_failure — it asserts a human review that did not happen
  • Installation tokens are minted per declared operation scope, with the scope in the cache key
  • Terminal failures are 403 with a discriminated detail.error, so a daemon branches on cause rather than spinning on a retried 5xx

ADR 0018: AI Model Catalog

Date: 2026-09-04 | Status: Accepted

Adds a per-organization catalog of configured providers and the models they serve alongside the existing environment-based LLM configuration. Credentials are encrypted at rest and guarded by their own permission.

Key Decisions:

  • Drivers (anthropic, openai, openai_compatible, bedrock, vertex) are static code in imbi.common.llm.drivers; an AIProvider node exists only once an admin configures one
  • AIModel carries an explicit access_scope plus ALLOWED_FOR team edges, so "every team" and "no team picked yet" stay distinguishable
  • Provider API keys are stored as ConfigEncryption ciphertext; responses carry has_credentials, a last-four credential_hint, and credential_updated_at, never the key
  • ai_model:credentials is separate from ai_model:update — editing a temperature must not imply the right to replace the production key
  • URLs address nodes by id, and every Cypher query re-matches the organization edge, so a valid id under the wrong org is 404
  • No cascade delete: a provider that still serves models is 409
  • monthly_spend_cap is advisory; Imbi has no spend tracking yet
  • Discovery calls the provider's own list-models API with the stored key, sanitizing every failure to a driver and status line

Messaging, Agents, and Workflows

ADR 0019: Adopt Apache Iggy as the Message Streaming Service

Date: 2026-09-04 | Status: Accepted

Adopts Apache Iggy as the durable message stream. The first use moves every ClickHouse write onto Iggy. The official ClickHouse sink connector writes the rows to ClickHouse.

Key Decisions:

  • One stream per subject; topics split a stream along a dimension chosen per stream
  • imbi.common.iggy publishes rows as JSON on the official apache-iggy SDK, with one partition per topic
  • The TOPICS mapping in code is the single source for the sink configuration
  • Read-after-write paths and duplicate rows on plain MergeTree tables get explicit handling

ADR 0020: Agent Task Contract

Date: 2026-10-07 | Status: Accepted

Defines the contract between Imbi, any agent harness, and the people who watch and steer agent work. It does not decide how the harness runs.

Key Decisions:

  • Task state is in the Postgres agent_runtime schema; ClickHouse, fed through Iggy, is the permanent event log
  • A task state change and its event commit in one transaction
  • A sweep moves the event log of a finished task out of Postgres after ClickHouse has every event
  • A task pins its agent and prompt versions at creation and has an immutable origin (human, schedule, webhook, or task)
  • A hard per-task budget, checked against a Postgres ledger

ADR 0021: Workflow Contract

Date: 2026-10-12 | Status: Accepted

Defines workflows: a trigger followed by steps, built on a canvas and run without a person. Definitions and runs are relational. The imbi-workflows service runs them.

Key Decisions:

  • Definitions are in the Postgres workflows schema as immutable versions; only the active version runs, and a run pins its version
  • The graph is a directed acyclic graph; activation requires a graph with no errors and no warnings
  • Step types are a registry shared by imbi-api and the engine; the UI builds its palette from it
  • Run state is in the Postgres workflow_runtime schema; ClickHouse is the durable run log
  • Runs execute as the imbi-workflows service account; the person who activates a workflow must hold the permissions of every step in it

ADR 0022: Agent Harness and Secrets Broker

Date: 2026-10-13 | Status: Accepted

Defines the agent harness that runs agent tasks, and a secrets broker in imbi-api that gives agents short-lived access to secrets that external systems hold.

Key Decisions:

  • Runner pods are in pools; each pool is a fixed-size Kubernetes Deployment
  • Dispatch is pull: a runner long-polls imbi-api and claims a whole task chain
  • One chain per pod; the pod deletes itself when the chain ends
  • Lease expiry fences the runner, revokes its credentials, and only then requeues the chain
  • The broker stores no secret values; a grant must allow each request, and the executor delivers the secret only to the command that needs it

ADR 0023: Realtime Channel Between the UI and the Services

Date: 2026-10-23 | Status: Proposed

Adds one WebSocket per browser tab so open pages learn about changes made by other people, agents, webhooks, and jobs, and so the UI can show presence.

Key Decisions:

  • Browsers subscribe to topics scoped by organization; each topic type names the permission its REST read needs
  • Events carry identifiers only; the browser refetches through REST, so every existing query can become live
  • Writers in any service publish through imbi.common.realtime to Valkey pub/sub; each imbi-api process holds one subscriber
  • No sticky sessions; reconnects resubscribe and refetch, and sockets close at token expiry and with 1012 on shutdown
  • Presence moves from the HTTP heartbeat onto the socket; one 30-second refresh replaces per-feature polling while the socket is down
  • External state such as GitHub deploy runs is watched once on the server, not by each browser

ADR Format

Our ADRs follow this structure:

# [Number]. [Title]

Date: YYYY-MM-DD

## Status

[Proposed | Accepted | Deprecated | Superseded]

## Context

[Description of the issue or requirement]

## Decision

[The change or approach being adopted]

## Consequences

[Positive and negative outcomes]

## References

[Links to relevant documentation]

Creating New ADRs

When making significant architectural decisions:

  1. Create a new ADR file in docs/api/adr/
  2. Use sequential numbering: Check existing ADRs and use the next number
  3. Follow the format: Use the template structure above
  4. Be thorough: Include context, alternatives considered, and trade-offs
  5. Update this index: Add a summary entry linking to your new ADR
  6. Update the navigation: Add the new ADR to the nav section of mkdocs.yml
  7. Submit for review: ADRs should go through the same PR process as code

ADR Guidelines

  • Be specific: Vague ADRs aren't helpful. Include concrete examples.
  • Explain trade-offs: Every decision has pros and cons. Document both.
  • Consider alternatives: Show what other options were considered and why they were rejected.
  • Update status: If a decision is superseded, update the status and link to the new ADR.
  • Link to code: Reference the files or modules that implement the decision.
  • Keep context: Future developers need to understand why decisions were made.

References